Slow Splunk - splunkd and auditd over 100% CPU - conflict?

[Jason]

I'm working on a box right now that seems to be unnecessarily slow at both searching as well as indexing from a batch folder (files have been in there for hours and haven't been deleted yet).

One odd thing I noticed is that auditd is taking 95-110% CPU, splunkd is taking 100-175% CPU, and there's another process, setroubleshootd around 90-95%.

This is an 8-3Ghz core box, so I'm not worried about the over 100% measurements from the top command, but with auditd taking so much CPU, is it perhaps conflicting with Splunk, similar to having an antivirus system on Windows monitoring Splunk's data directories?

[Jason · Accepted solution]

There was a problem with auditd and setroubleshootd. I don't know what happened exactly, but I traced it to a misconfiguration with those spitting out 20MB+ logs every couple of seconds, and Splunk was trying to eat it. One of the system admins managed to fix it and unfortunately I don't know how.

I did up Splunk indexing performance by cleaning out tens of thousands of learned csv-nnnnn sourcetypes from the Learned app.

[dwaddle]

I asked because of setroubleshootd, which is related to selinux. You might check to see if someone has dynamically re-enabled selinux. (It's a rare chance, but still...)

[Jason]

I doubt it, I know Splunk checks for SELINUX on startup and it hasn't complained.

[dwaddle]

Are you running with selinux on or off?

[Jason]

Ouch, I guess I hadn't noticed HOW slow it was at batch input: it seems like it's doing ~5 files per minute...