Why is the app with only index changed not working?

[mlahey]

Hello all,

I have been trying to get out environments/indexes organized and I'm running in to some trouble. I am trying to set up our different development environments in to different indexes. As such, I am creating apps for all the different sourcetypes and indexes necessary for this. My prod and dev apps have been working great, but when I am trying to create another app for a test environment I can not get the servers to report to splunk. I have gotten these servers to report using the existing prod app (but to the prod index) by adding them to the prod server class. The inputs.conf file is the only thing with a difference (different value for index), but this difference makes it so I can't get data back from the servers.

Is there something I need to do to set up an index to get servers reporting to it? Or should making an app with a different index name be sufficient.

Any help or ideas are greatly appreciated.

[ekost]

To modify an app and change the index, it'll require changes (or verification) in a few others places as well.
1. You'll need an updated indexes.conf to make the index.
2. You'll need to update the inputs.conf to get/route data into the new index.
3. You'll need to check your Role to see how tightly index access is set, and add the index to the appropriate Role(s) if needed.

Easiest way to check at the file level is to grep your app folder looking for the index name. An app is all text (.conf,) .js, and .py files after all.

Good luck!

[mlahey]

Thanks for the help. I was able to modify indexes.conf to make the indexes. My problem is coming when I try to make a new app for the new environment, using the existing one that is working for another environment, it doesn't seem to send anything to the new index. the only thing I am modifying in changing the app is the inputs.conf and changing the index value.

[ekost]

I feel as though I'm missing some critical detail here, but let's talk through it some more.
- There are forwarder nodes grabbing/tailing the data and sending it off to the indexers.
- There are Prod, Dev, and Test (UAT) Splunk environments.
- I need to synch the Splunk apps so that I can have the same apps (more or less) in all three environments.
- I want to change the default index(es) for my $custom_app depending upon the environment.
- I'm using a deployment server to manage the forwarders.

Is there any other base details missing?

Bear with me! It's essentially 3 copies of the $custom_app_$environ, one for each environment with:
1. Unique indexes.conf with the index name defined.
2. Unique inputs.conf with the index name defined.
3. Full review of the app (my grep comment above) looking for any string value matching the index name that might need changing.
4. Validation that the appropriate role has index access to my $custom_app_$environ app.

Once you've got $custom_app_test validated with the new index name, you'd need to deploy it to:
1. Forwarders, as they need that inputs.conf changes.
2. Indexers (possibly) as they need the indexes.conf changes. Your enviro might have a dedicated app that is only for managing index definitions. Use that "indexes only" app if you find/have one.
3. Search Head, as it'll need any search-time props/transforms, UI/Dashboard/Searches, and other knowledge objects.

This is where your comment "it doesn't seem to send anything to the new index" comes in.

Can you find those hosts/forwarders sending data into other indexes (even index=_internal?) Can you see the data source you're looking for being sent, but to a different index name?
If the forwarders are communicating with Splunk:
1. Did you add the new $custom_app_test to the Deployment Server?
2. Did you define a new "server class" for the test forwarders?
3. How are the forwarders being defined/whitelisted?
4. Check one of your chosen hosts/forwarders and validate that the $custom_app_test was successfully deployed there.