Splunk Search

How to find out where a search is originating from?

hartfoml
Motivator

I have this process running on all my indexes:

[splunkd pid=7803] search --id=remote_SearchHead.local_scheduler__nobody__datasystems__RMD5e816c6f7615a1e8c_at_1470755400_14045 --maxbuckets=0 --ttl=60 --maxout=0 --maxtime=0 --lookups=1 --streaming --outCsv=true --user=splunk-system-user --pro --roles=admin:power:splunk-system-role:user

I can tell that it is coming from the search head and from the datasystems app, but I cannot find the search name or where it is scheduled to run to stop it. It is running as "nobody" or "splunk-system-user", so it is hard to ID who is running the search.

I can kill the pid but it comes right back.

0 Karma

somesoni2
Revered Legend

It's a scheduled search so you'd find it's traces in scheduler logs

Try this

index=_internal sourcetype=scheduler sid="Copy the id field from your process description e.g. remote_SearchHead.local_scheduler__nobody__datasystems__RMD5e816c6f7615a1e8c_at_1470755400_14045" 

The output will contain a field called savedsearch_id, which will include, owner;AppName;Saved search name.

Update#1
I'm guessing you're killing the search before it's completed, so there is no search completion records in above query.

Try this alternative method/place. You already have the owner (nobody) and app name. This will give your saved search name.

index=_audit action=search search="*" NOT search="'typeahead*" NOT search="'|history*" 
search_id="Copy the id field from your process description " OR id="Copy the id field from your process description"
0 Karma

hartfoml
Motivator

I did the search back 7 days and this is the result... Only one log entry

08-09-2016 11:28:19.963 -0500 INFO SavedSplunker - AlertNotifier::execute: queued sid=scheduler_nobodydatasystems_RMD5e816c6f7615a1e8c_at_1470755400_14045 for action execution

0 Karma

somesoni2
Revered Legend

Try the updated answer.

0 Karma

hartfoml
Motivator

Thanks @somesoni2 this did not yield any results. Here is my search

index=_audit action=search search="*" NOT search="'typeahead*" NOT search="'|history*" 
 search_id="*RMD5e816c6f7615a1e8c_at_1470755400_14045" OR id="*RMD5e816c6f7615a1e8c_at_1470755400_14045"

See you at dot conf

0 Karma

somesoni2
Revered Legend

May be try like this, ensure to select proper time range which will include the search execution time
(1470755400)

index=_audit action=search NOT (search="'typeahead*" OR search="'|history*" )  "*RMD5e816c6f7615a1e8c*"
0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer Certification at ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...