Monitoring Splunk

Slow Splunk - splunkd and auditd over 100% CPU - conflict?

Jason
Motivator

I'm working on a box right now that seems to be unnecessarily slow at both searching as well as indexing from a batch folder (files have been in there for hours and haven't been deleted yet).

One odd thing I noticed is that auditd is taking 95-110% CPU, splunkd is taking 100-175% CPU, and there's another process, setroubleshootd around 90-95%.

This is an 8-3Ghz core box, so I'm not worried about the over 100% measurements from the top command, but with auditd taking so much CPU, is it perhaps conflicting with Splunk, similar to having an antivirus system on Windows monitoring Splunk's data directories?

Tags (1)
0 Karma
1 Solution

Jason
Motivator

There was a problem with auditd and setroubleshootd. I don't know what happened exactly, but I traced it to a misconfiguration with those spitting out 20MB+ logs every couple of seconds, and Splunk was trying to eat it. One of the system admins managed to fix it and unfortunately I don't know how.

I did up Splunk indexing performance by cleaning out tens of thousands of learned csv-nnnnn sourcetypes from the Learned app.

View solution in original post

0 Karma

Jason
Motivator

There was a problem with auditd and setroubleshootd. I don't know what happened exactly, but I traced it to a misconfiguration with those spitting out 20MB+ logs every couple of seconds, and Splunk was trying to eat it. One of the system admins managed to fix it and unfortunately I don't know how.

I did up Splunk indexing performance by cleaning out tens of thousands of learned csv-nnnnn sourcetypes from the Learned app.

0 Karma

dwaddle
SplunkTrust
SplunkTrust

I asked because of setroubleshootd, which is related to selinux. You might check to see if someone has dynamically re-enabled selinux. (It's a rare chance, but still...)

0 Karma

Jason
Motivator

I doubt it, I know Splunk checks for SELINUX on startup and it hasn't complained.

0 Karma

dwaddle
SplunkTrust
SplunkTrust

Are you running with selinux on or off?

0 Karma

Jason
Motivator

Ouch, I guess I hadn't noticed HOW slow it was at batch input: it seems like it's doing ~5 files per minute...

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...