Monitoring Splunk

how to determine what exactly is happening with rolling data to cold storage

jamesdon
Path Finder

Hello,

We recently experienced a problem where our storage for cold data went offline. We are interested in determining exactly what happened to the data at that time, as well as when running a splunk fsdsk.

Is there a search prebuilt for this? Which file in /opt/splunk/var/log/splunk/ would have this info?

Thank you,

Jim

Tags (1)
0 Karma

jamesdon
Path Finder

OK, it took us a bit to figure this one out, but the results may help someone out in the future. It ends out that we stopped rolling data over to cold after making changes to indexes.conf a while back. Lessons learned:

First, the logs are in /opt/splunk/var/log/splunk/splunkd.log.

Second, do not put trailing comments in your config files, it will cause confusion and delay. Something like:

[stan]
config = this # Don't put a comment here, its trouble

Jim

0 Karma

twkan
Splunk Employee
Splunk Employee

This sounds like a nasty issue to have, and may be similar to what I have encountered recently. Did your data somehow rolled over nto the frozen stage, and you do not have a coldToFrozen script defined?

0 Karma

jbsplunk
Splunk Employee
Splunk Employee

There isn't a prebuilt search for this, and I am not exactly sure what you mean by your cold data went offline. I assume its remote storage of some sort, such as nfs? I would review splunkd.log for errors about the buckets. I would imagine you'd see a lot of inflight buckets in warm that never made it to cold.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...