I have noticed that every 20-30 minutes we see a hugh CPU spike on two of our indexers. And looking at the splunkd.log file, the CPU spike co-incides with a large number of errors as follows:
02-27-2012 17:33:14.197 +0000 ERROR ConfObjectManagerDB - Ignoring invalid database setting: [inputs/monitor%3A%2F%2F%2Fvar%2Flog] modtime = 1321526706.498255000
02-27-2012 17:33:14.197 +0000 ERROR ConfObjectManagerDB - Ignoring invalid database setting: [inputs/monitor%3A%2F%2F~%2FLibrary%2FLogs] modtime = 1321526706.628675000
02-27-2012 17:33:14.197 +0000 ERROR ConfObjectManagerDB - Ignoring invalid database setting: [inputs/script%3A%2F%2F.%2Fbin%2Fcpu.sh] modtime = 1321526706.896315000
02-27-2012 17:33:14.197 +0000 ERROR ConfObjectManagerDB - Ignoring invalid database setting: [inputs/script%3A%2F%2F.%2Fbin%2Fdf.sh] modtime = 1321526706.990800000
02-27-2012 17:33:14.197 +0000 ERROR ConfObjectManagerDB - Ignoring invalid database setting: [inputs/script%3A%2F%2F.%2Fbin%2Fhardware.sh] modtime = 1321526707.087146000
02-27-2012 17:33:14.197 +0000 ERROR ConfObjectManagerDB - Ignoring invalid database setting: [inputs/script%3A%2F%2F.%2Fbin%2Finterfaces.sh] modtime = 1321526707.179597000
perhaps 500 or so lines...
I am assuming this is related to the deployment manager or bundle replication, but a quick google hasn't given me much to go on...
Does anyone have any suggestions where to start looking?
After a bit of a wait, Splunk suggested that this could be a bundle issue between the 4.3 searchhead, and 4.2.x indexers.
Whilst this was not listed as a possible problem on the compatibility matrix, (here: http://docs.splunk.com/Documentation/Splunk/4.3.1/Deploy/Whatisdistributedsearch#Cross-version_compa...) since upgrading everything to 4.3.1 the problem has been resolved.
After a bit of a wait, Splunk suggested that this could be a bundle issue between the 4.3 searchhead, and 4.2.x indexers.
Whilst this was not listed as a possible problem on the compatibility matrix, (here: http://docs.splunk.com/Documentation/Splunk/4.3.1/Deploy/Whatisdistributedsearch#Cross-version_compa...) since upgrading everything to 4.3.1 the problem has been resolved.
We have a similar setup, running on 4.2.1, and 4.2.0 on our two indexer servers (Linux 64bit on search heads and indexers). I see that both servers threw this message but I cannot see any CPU problems on any of the servers. We use HW based servers. We around 80 forwarders connected to our system and it lost connections to many of them at the same time.
Do you have any update on this error? We are experiencing the same problems here.
No. I also have a ticket open with spunk support since the 27th, and have not heard anything back yet.
Can you post an overview of your environment, incase there are any similarities i can update the ticket with.
We have 1 search head, with a pool of two indexers.
We see the errors on BOTH indexers at the same time.
Indexer1 is a deployment server, which deploys to about 12 universal forwarders, and we have a few manually configured forwarders too.
Our entire deployment is hosted in AWS.
i'll update this ticket when i get an answer from support, but in the meantime.... any ideas????