Missing _introspection index on 5.x Indexer

[reed_kelly]

We are using a Splunk 6.1 heavy forwarder to process and send data to 5.x indexer. I see the following error from the Search Head when I perform a search. I am unable to add the _introspection index on the indexer. Is there another way to eliminate this error?

Search peer xxx has the following message: received event for unconfigured/disabled/deleted index='_introspection' with source='source::/opt/splunk/var/log/introspection/resource_usage.log' host='host::yyy' sourcetype='sourcetype::splunk_resource_usage' (1 missing total)

[jrodman · Accepted solution]

In general, Splunk does not recommend sending data from newer forwarders to older indexers because you can run into hangups like this. There was an exception clause for Universal Forwarders that created during the rollout of the Universal Forwarder functionality, but the utility there has long since passed.

Basically, when the instrospection functionality was built, we decided to place its data in its own index, because of concerns regarding storage, and the potential for introspection to overwhelm _internal, or because administrators were likely to have different opinions on the amount of space they would be wiling to allocate to one category or another.

Thus we added the introspection index to all versions of splunk after this feature. However, when forwarding data to older indexers, they do not have this index, and thus the data cannot be stored in the index. For about 5 years, we have produced highly visible messaging when indexers are receiving data for an index they do not have, because the indexer has no way to handle the data properly, and it usually indicates a big misconfiguration problem.

In this case, you can simply create an index called _introspection on your indexers, or upgrade them to 6.x. If you do choose to create your own index, you may wish to revisit your index when you do upgrade to 6.x+ to rationalize your settings vs the defaults.

I suppose an alternative would be to disable introspection data acquisition on all the forwarders, but introspection data is good troubleshooting information, and this sounds like a larger administrative burden than my first proposal. But it's your choice.

[wbfoxii]

Any guidelines on how long data should be kept in _introspection? The default size is 500,000. Is six months enough?

[jrodman]

Oh, that works too. Introspection is currently mainly used by the Splunk on Splunk app and Splunk technical support, but it's useful to a splunk administrator too.

[reed_kelly]

I remembered that I could also do it with routing from outputs.conf:

[tcpout]
defaultGroup = default-autolb-group
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_internal)

We have an upgrade plan, but it will take a bit longer. - thanks