Recently, I have been handed the Splunk instance we have in our company. We are running Splunk 4.2.2. There's one indexer/search head, and about 75 forwarders.
As I've been learning about Splunk and checking out the various apps and configurations we currently use, I noticed that when I clicked any of the Forwarders in the Deployment Monitor App for statics, no data appeared. After looking at the code I found that the app was trying to pull the data from the _internal index.
I checked the Indexes in Manager on our indexer/search head, and found that the _internal index had been disabled. I enabled the _internal index, and restarted Splunk for good measure. When data still wasn't being written to the _internal index, I searched this site and found the post below:
http://splunk-base.splunk.com/answers/53848/why-is-no-data-being-written-to-the-_internal-index-for-my-search-head
This is why you cannot find any _internal events recorded by your search-head anywhere. To correct this, add the following configuration to $SPLUNK_HOME/etc/system/local/inputs.conf:
[tcpout]
forwardedindex.3.whitelist = _internal
I have added that as specified, restarted Splunk, and still no data is being written to _internal.
I also added the following to inputs.conf in the same folder:
[monitor://$SPLUNK_HOME/var/log/splunk]
disabled = 0
index = _internal
However, no data gets written to _internal. If I remove the index part, the logs are scanned and indexed, but they are placed in the "main" index.
Here's the full inputs.conf from $SPLUNK_HOME/etc/system/local/outputs.conf:
[script://$SPLUNK_HOME\bin\scripts\splunk-admon.path]
disabled = 0
[script://$SPLUNK_HOME\bin\scripts\splunk-perfmon.path]
disabled = 0
[script://$SPLUNK_HOME\bin\scripts\splunk-regmon.path]
disabled = 0
[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0
[monitor://$SPLUNK_HOME/var/log/splunk]
disabled = 0
And outputs.conf:
[tcpout]
defaultGroup =
disabled = false
forwardedindex.3.whitelist = _internal
Again, I am new to Splunk, so there may be other configurations I should be checking, so any help would be greatly appreciated. If you need additional information, please let me know.
... View more