I'm receiving the following message on my Splunk Indexer:
Received event for unconfigured/disabled/deleted index=wineventlog with source="source::WinEventLog:System" host="host::xxx" sourcetype="sourcetype::WinEventLog:System". So far received events from 1 missing index(es).
I've seen the same question posted and resolved in many forums by simply adding the wineventlog index since it doesn't exist by default. However, that does not seem to work for me and I'm sure I'm missing something obvious.
My forwarder is forwarding custom logs, it just will not forward Windows event logs because of the above error. This is what I have in C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf
:
[WinEventLog://Security]
disabled = 0
[monitor://C:\Program Files (x86)\Entrust\VerificationServer\logs\webservices.log]
disabled = false
If I modify it to explicitly use the main index as below, the event logs come through without any issues:
[WinEventLog://Security]
disabled = 0
index=main
[monitor://C:\Program Files (x86)\Entrust\VerificationServer\logs\webservices.log]
disabled = false
In both cases, my monitored log (webservices.log) gets forwarded successfully.
Using the GUI, I created a Search & Reporting index called wineventlog, restarted both the indexer and forwarder, but nothing comes through. It set the contents of my etc/apps/search/local/indexes.conf file to the following:
[wineventlog]
coldPath = $SPLUNK_DB/wineventlog/colddb
homePath = $SPLUNK_DB/wineventlog/db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB/wineventlog/thaweddb
I've also tried selecting Distributed Management Console, Home and App Browser under the "App" type when creating the index instead of Search & Reporting, but they all have the same behaviour.
My question is, is there anything else I need to do in order to get my indexer to use this index?
Ah, my guess is that your user role doesn't search the wineventlog
log index by default. Did you try searchingindex=weblog
specifically?
If the index exists on your indexers and If it is happening from clustered indexers, has worked fine until the message was found, just all of a sudden it complains about the error after an activity on the cluster.
Check if the index gets disabled by splunk instance. If then this must be the cause of the message. You may need to check if you have any bucket id conflicts like below which is caused by bucket replication ;
ERROR IndexerService - Error intializing IndexerService: idx=MyIndex bid=MyIndex~25~9B9D1F9-8EA5-4C73-BCC4-6C7C65E2AB5A
bucket=rb_1466615370_1466529185_25_9B9D1F9-8EA5-4C73-BCC4-6C7C65E2AB5A Detected directory manually copied into its database, causing id conflicts [path1='C:\splunk_indexes\MyIndex\db\db_1466615370_1466529185_25_9B9D1F9-8EA5-4C73-BCC4-6C7C65E2AB5A' path2='C:\splunk_cold_indexes\MyIndex\db\rb_1466615370_1466529185_25_9B9D1F9-8EA5-4C73-BCC4-6C7C65E2AB5A'].
You would need to figure out the bucket conflicting issue by removing or moving it to somewhere Splunkd doesn't know.
Ah, my guess is that your user role doesn't search the wineventlog
log index by default. Did you try searchingindex=weblog
specifically?
Well now I feel like an idiot, that was the problem! First time setting up Splunk and I've been doing everything as admin so I just assumed it was searching all indexes. Thanks for the quick response!
Don't feel bad! I have done this more than once. Now, I always change the admin role so that it set to search "all non-internal indexes" by default. It's part of the steps I go through whenever I install or update Splunk...
Your forwarder probably IS forwarding these logs - but there is no way for the forwarder to know whether or not the data was successfully indexed.
And it looks like you created the correct index (should be on all your indexers if you have more than one). So that should be fine as well.
However - once the forwarder has sent some data, it will not send the same data again.
If this is your problem, you will need to reset the forwarder's file pointers. You can reset all of them by deleting the fishbucket directory, $SPLUNK_HOME\var\lib\splunk\fishbucket
on the forwarder. Or you can use btprobe
to reset individual file pointers. Here is an answer that may help (even though it is older): btprobe and re-indexing data
That may be the case for old data, but if I'm creating new events shouldn't it be sending those going forward? I can produce new event logs by logging in and out of the server, but they still don't show up on the indexer unless I point to index main. I just tried deleting the fishbucket directory as you mentioned and restarting the forwarder, but the behaviour doesn't change.
Thanks!