- Splunk Answers
- :
- Splunk Administration
- :
- Deployment Architecture
- :
- AggregatorMiningProcessor
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
AggregatorMiningProcessor
Any idea what this means or more to the point, why?? Came in to a dead Splunk...
WARN AggregatorMiningProcessor - Breaking event because limit of 256 has been exceeded
-0400 FATAL ProcessRunner - Unexpected EOF from process runner child!
-0400 ERROR ProcessRunner - helper process seems to have died (child killed by signal 9: Killed)!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Turns out we had some users running large, numerous and mixed with real-time searches. Evidently, this causes the OS to shut-down splunkd via OOM killer.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We see this:
Yes, the OS kernel decided to kill off splunkd, not a good choice.
Aug 27 16:46:35 splunk01 kernel: [58064] 10074 58064 911880 897929 0 0 0 splunkd
Aug 27 16:46:35 splunk01 kernel: [58065] 10074 58065 12792 1163 2 0 0 splunkd
Aug 27 16:46:35 splunk01 kernel: [58067] 10074 58067 12791 1163 2 0 0 splunkd
Aug 27 16:46:35 splunk01 kernel: [58070] 10074 58070 12792 1163 0 0 0 splunkd
Aug 27 16:46:35 splunk01 kernel: [63522] 0 63522 1014 17 3 0 0 sleep
Aug 27 16:46:35 splunk01 kernel: [67229] 0 67229 1014 17 2 0 0 sleep
Aug 27 16:46:35 splunk01 kernel: Out of memory: Kill process 58064 (splunkd) score 439 or sacrifice child
Aug 27 16:46:35 splunk01 kernel: Killed process 58067, UID 10074, (splunkd) total-vm:51164kB, anon-rss:4648kB, file-rss:4kB
Aug 27 16:46:35 splunk01 kernel: splunkd invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=0, oom_score_adj=0
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yep - Out of memory will do it 😞
If you could mark this question as answered, that will help other people who are searching for an answer to the same problem.
Good luck!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Right... after a bit of searching I've come up with the following.
WARN AggregatorMiningProcessor - Breaking event because limit of 256 has been exceeded
When parsing for multiline events, Splunk has an inbuilt limit of 256 lines. This has been exceeded, and this error was thrown, however I do not think this was the problem....
-0400 FATAL ProcessRunner - Unexpected EOF from process runner child!
I don't think we're seeing the full event here... signal 9 is a KILL signal from an external process. From the second reference below:
"It is likely that your OS has some
kind of monitor or other setting on it
that kills processes that do certain
things. Perhaps your administrator is
watching for memory usage, access to
certain files, or other things. You
should consult with your system admin
to find out what they have put in
place."
I hope this sets you on the right path.
REFERENCES:
Introducing the 2024 SplunkTrust!
Introducing the 2024 Splunk MVPs!
Splunk Custom Visualizations App End of Life
