All Apps and Add-ons

Metrics collecting bash scripts generating triggering too many auditd alerts

tzhang_gr
New Member

Hi,

We're using a simplified version of the Splunk_TA_nix app (basically just a subset of the bash scripts) to collect system metrics such as CPU memory disk usages;

It works well, but the problem is that these bash commands are constantly triggering our auditd rules on the end point (which is configured to monitor for execve syscalls), at an extremely high rate. E.g. every 30 seconds, one of these bash scripts executes, which triggers ~ 20 audit events, resulting in 100+ audit records, just from the sed, awk, tee, wc etc. commands within the script, which get fed back into splunk via the other auditd app we have (the amount of data generated from this is huge - larger than normal system activities in general).

One solution we're considering is to filter out audit records with auid=-1 from the audit rules side (with the tradeoff of losing some visibility);
Another approach is to rewrite all these bash scripts in python or some compiled languages so that it doesn't trigger that many execve syscalls;
Third option is to filter out the specific commands generated by these scripts;

Does anyone else have a similar problem? What's the best way to deal with this kind of issues? Is there another good app that serves a similar purpose but does not use bash (using python / C etc. instead)?

Any info is really appreciated!

0 Karma

hmaldonado
Explorer

Hello @tzhang_gr ,
If I correctly understand your problem, it seems that the scripts in the TA for Unix and LInux leave too much of a footprint on the system.
If that is the case, might it be possible for you to use collectd to get the metrics you seek?
https://collectd.org/

You could then visualize the data with the Splunk App for Infrastructure.
https://splunkbase.splunk.com/app/3975/

Regards,
Hans

0 Karma

tzhang_gr
New Member

Thank you very much! I'll take a look into this approach.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...