Monitoring Splunk

Search killing _audit

tsheets13
Communicator

Our _audit file keeps growing and growing. We have identified what is filling it up but cannot figure out what is causing it.

The user is stripa. If I search index=_audit stripa, I find 100's of thousands of events over a 15 minute period that look like this...

9/17/19
12:53:09.523 PM
Audit:[timestamp=09-17-2019 13:53:09.523, user=splunk-system-user, action=search, info=terminate, search_id='rt_scheduler_stripasearch_RMD55e845684aa67ede1_at_1558279620_18914'][n/a]
source = audittrailsourcetype = audittrail
9/17/19
12:53:09.523 PM
Audit:[timestamp=09-17-2019 13:53:09.523, user=splunk-system-user, action=search, info=cancel, search_id='rt_scheduler_stripasearch_RMD55e845684aa67ede1_at_1558279620_18914'][n/a]
source = audittrailsourcetype = audittrail
9/17/19
12:53:09.523 PM
Audit:[timestamp=09-17-2019 13:53:09.523, user=splunk-system-user, action=search, info=terminate, search_id='rt_scheduler_stripasearch_RMD52dc925e4d0d65765_at_1565488020_78337'][n/a]
source = audittrailsourcetype = audittrail
9/17/19
12:53:09.523 PM
Audit:[timestamp=09-17-2019 13:53:09.523, user=splunk-system-user, action=search, info=cancel, search_id='rt_scheduler_stripasearch_RMD52dc925e4d0d65765_at_1565488020_78337'][n/a]
source = audittrailsourcetype = audittrail
9/17/19
12:53:09.522 PM
Audit:[timestamp=09-17-2019 13:53:09.522, user=splunk-system-user, action=search, info=terminate, search_id='rt_scheduler_stripasearch_RMD52dc925e4d0d65765_at_1559222520_46294'][n/a]
source = audittrailsourcetype = audittrail

We only found two items under "Settings -> All Configurations" and these were unrelated reports, but we disabled them nonetheless.

How can I get to the bottom of what is causing this. I'm stumped.

Labels (1)
Tags (2)
0 Karma

tsheets13
Communicator

In this case it's our dev enviroment. One search head and one indexer.

That search provides no results.

0 Karma

adonio
Ultra Champion

in the _audit data, look for the host field value and splunk_server field value
this user might saves their search in private mode ...

0 Karma

tsheets13
Communicator

host is the hostname of the search head

splunk_server is the DNS name of the search head

0 Karma

adonio
Ultra Champion

looks like a real-time search of some sort
rt stands for real-time scheduler is the component that schedules the searches
what is stripa?
make sure to stop and disable all real-time search

tsheets13
Communicator

stripa is a user.

How can I determine where this realtime search is running? There are no searches or reports owned by that user that aren't disabled.

0 Karma

adonio
Ultra Champion

apparently there are ...
try this:
| rest /services/search/jobs | search eventSorting=realtime
find the user and teach her / him
if you have distributed / clustered environment, maybe that search runs on another search head or even worse, directly on a single indexer.
regardless, i will highly recommend to disable real-time searches across all environment
https://docs.splunk.com/Documentation/Splunk/7.3.1/Search/Restrictrealtimesearch

0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer Certification at ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...