Alerting

Why real-time alerts can lead to insufficient disk space on a device and cause splunkweb to not start?

Michael
Contributor

Sharing a lesson learned...
Splunk 6.1.3 (but I think would apply to most) on RHEL 6.

I came in one morning to being unable to log into Splunk, and the web interface producing an error indicating that the drive was full. Upon checking the space, there was plenty, over 30 gigs. I have had it stop indexing once when it reached the 2 gig mark, as designed, but never saw this -- that did not prevent the web interface from working.

1 Solution

Michael
Contributor

hmm, forgot to "answer" this so it would be closed. Tks Rich,

Cheers!

Long story short, I had the previous day created an alert to fire off in "real time". Be very careful with these! Overnight, the alert fired off, but I had set the criteria up wrong, so it fired off over 10,000 times. The space that was filled up were the inodes. This can be checked with 'ls -i'. The place that fills up is in ../splunk/var/run/splunk/dispatch/ -- I removed all the alerts in this directory and went happily about my business -- oh, and removing that offending alert.

View solution in original post

Michael
Contributor

hmm, forgot to "answer" this so it would be closed. Tks Rich,

Cheers!

Long story short, I had the previous day created an alert to fire off in "real time". Be very careful with these! Overnight, the alert fired off, but I had set the criteria up wrong, so it fired off over 10,000 times. The space that was filled up were the inodes. This can be checked with 'ls -i'. The place that fills up is in ../splunk/var/run/splunk/dispatch/ -- I removed all the alerts in this directory and went happily about my business -- oh, and removing that offending alert.

yannK
Splunk Employee
Splunk Employee

realtime/alltime alert searches are like a loaded gun, handle with care.

0 Karma

ppablo
Retired

Hi @Michael

I just moved your content around to the appropriate spaces and also accepted the answer for you so this post will get more hits. Thanks for sharing this 🙂 very helpful.

Patrick

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Thanks for sharing, Michael. For the benefit of users searching for similar problems in future, answer this question and accept the answer. That will mark this as a solution.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Observability | Use Synthetic Monitoring for Website Metadata Verification

If you are on Splunk Observability Cloud, you may already have Synthetic Monitoringin your observability ...

More Ways To Control Your Costs With Archived Metrics | Register for Tech Talk

Tuesday, May 14, 2024  |  11AM PT / 2PM ET Register to Attend Join us for this Tech Talk and learn how to ...

.conf24 | Personalize your .conf experience with Learning Paths!

Personalize your .conf24 Experience Learning paths allow you to level up your skill sets and dive deeper ...