Unable to search/index the uploaded text file in the newly built test machine?

[Hemnaath]

Hi, I have recently setup an single instance test machine in our environment, with splunk version as 6.6.1 in Linux environment (VM Platform) The same test machine is connected with the license master.

My agenda was to test a upgrade Paloalto add-on app 6.0.2 in this machine before pushing the config to Prod env, so pushed the updated Paloalto add-on to /opt/splunk/etc/apps/Splunk_TA_Paloalto/. Then upload a raw text file taken from my production machine and uploaded in the test machine via Splunk web --> settings --> Add Data -- Uploaded text file -- selected the sourcetype -- assigned to newly created index called Firewall --review --submit -start searching. But unable to see any data being indexed.

Note: Index location is given default "/opt/splunk/var/lib/splunk/firewall/db --> I could see the file called Creationtime in this location other then this there is nothing present in this location.

Could you please guide me to troubleshoot this issue.

[Hemnaath · Accepted solution]

The above issue got fixed, on investigating the problem we found that outputs.conf file was configured in with below stanza, due to this all the data when we uploaded via splunk test portal, the data was being ingested in to production.

Steps :

1) Checked by executing the index="_internal" and found that there was no data being ingested. This showed that some thing is really going wrong as we could not see the splunk internal data.
2) Executed splunk btool command to find out outputs.conf list to check the configuration

./splunk btool outputs list --debug | less 

3) We found that in one of the app, the out puts.conf was configured to route the data to the production indexers.

[tcpout]
defaultGroup = all_indexers
maxQueueSize = 1GB

[tcpout:all_indexers]
server = splunk.test.com:9997 

4) we had disabled the app by editing the app.conf file in local

# Autogenerated file
[install]
state = disabled

5) Then restarted the splunk service and check whether we are getting the splunk internal data and found to be ingesting.