Deployment Architecture

recover-padding-# from *nix app

chicodeme
Communicator

In using the *nix app, I had cpu.sh/iostat.sh/vmstat.sh and all was working fine across 300 rhel4/5 boxes.

I added in ps.sh & top.sh and now I have 219 recover-padding=# host entries. recover-padding-[1-5] as sources & sourcetypes.

Why did this happen and how can I make them go away?

I noticed this question but I couldn't find a Hosts.metadata file. http://answers.splunk.com/questions/10032/wierd-hosts-recover-padding-listed

1 Solution

chicodeme
Communicator

After I got everything cleaned up from above and my *nix app deployment. I was able to get rid of the recover-padding entries after working with support.

I had to stop Splunk and delete *.data, .*manifest file(s) from /opt/splunk/var/lib/splunk/os/db Then start Splunk..

I didn't have any .*manifest files but that cleared them up for me.

Also note it is NOT *.manifest.. It is .*manifest. I validated that by looking in some of my other db dirs.

After the restart it rebuilt the *.data files.

They said if the above didn't work then I should "run recover-metadata on all of your os index and identify the bad index" But only if the above didn't work.

View solution in original post

0 Karma

chicodeme
Communicator

After I got everything cleaned up from above and my *nix app deployment. I was able to get rid of the recover-padding entries after working with support.

I had to stop Splunk and delete *.data, .*manifest file(s) from /opt/splunk/var/lib/splunk/os/db Then start Splunk..

I didn't have any .*manifest files but that cleared them up for me.

Also note it is NOT *.manifest.. It is .*manifest. I validated that by looking in some of my other db dirs.

After the restart it rebuilt the *.data files.

They said if the above didn't work then I should "run recover-metadata on all of your os index and identify the bad index" But only if the above didn't work.

0 Karma

chicodeme
Communicator

I found the recover-padding entries in here:
/opt/splunk/var/lib/splunk/os/db/Hosts.data
Can i just delete them and restart?

0 Karma

chicodeme
Communicator

I think it might have happened because when I added ps.sh & top.sh to /opt/splunk/etc/deployment-apps/unix/local/inputs.conf I forgot to add them to /opt/splunk/etc/deployment-apps/unix/default/inputs.conf
I had removed the entries I didn't want at first. Anyways I put them back and deployed, but the recover-padding entries still didn't go away. Now i'm back to no ps.sh & top.sh but those darn recover-padding entries are all over even after a restart.

0 Karma

jrodman
Splunk Employee
Splunk Employee

You shouldn't need to make them go away, they're intended to be behavior that's essentially internal to the .data files in your system. They get added for cases where you experienced a problem that required recovery, ie crashes, hardware failings, or operating system failings. They're part of the recovery process for handling incompletely written out data.

This problem should be fixed in Splunk 4.1.2, as I read our bug database. What version are you using?

jrodman
Splunk Employee
Splunk Employee

Most likely you are using a modified dashboard which doesnt contain the workaround to hide the padding entries, but the issue seems resolved in any event.

chicodeme
Communicator

i upgraded the indexer to 4.1.7. didn't make the recover-padding go away

0 Karma

chicodeme
Communicator

Will they go away after they do their recovery thing? They are most annoying visually..

0 Karma

chicodeme
Communicator

4.1.6 for both forwarder and indexer

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to May Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars this month. This ...

They're back! Join the SplunkTrust and MVP at .conf24

With our highly anticipated annual conference, .conf, comes the fez-wearers you can trust! The SplunkTrust, as ...

Enterprise Security Content Update (ESCU) | New Releases

Last month, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...