Splunk Search

How to find out where a search is originating from?

hartfoml
Motivator

I have this process running on all my indexes:

[splunkd pid=7803] search --id=remote_SearchHead.local_scheduler__nobody__datasystems__RMD5e816c6f7615a1e8c_at_1470755400_14045 --maxbuckets=0 --ttl=60 --maxout=0 --maxtime=0 --lookups=1 --streaming --outCsv=true --user=splunk-system-user --pro --roles=admin:power:splunk-system-role:user

I can tell that it is coming from the search head and from the datasystems app, but I cannot find the search name or where it is scheduled to run to stop it. It is running as "nobody" or "splunk-system-user", so it is hard to ID who is running the search.

I can kill the pid but it comes right back.

0 Karma

somesoni2
Revered Legend

It's a scheduled search so you'd find it's traces in scheduler logs

Try this

index=_internal sourcetype=scheduler sid="Copy the id field from your process description e.g. remote_SearchHead.local_scheduler__nobody__datasystems__RMD5e816c6f7615a1e8c_at_1470755400_14045" 

The output will contain a field called savedsearch_id, which will include, owner;AppName;Saved search name.

Update#1
I'm guessing you're killing the search before it's completed, so there is no search completion records in above query.

Try this alternative method/place. You already have the owner (nobody) and app name. This will give your saved search name.

index=_audit action=search search="*" NOT search="'typeahead*" NOT search="'|history*" 
search_id="Copy the id field from your process description " OR id="Copy the id field from your process description"
0 Karma

hartfoml
Motivator

I did the search back 7 days and this is the result... Only one log entry

08-09-2016 11:28:19.963 -0500 INFO SavedSplunker - AlertNotifier::execute: queued sid=scheduler_nobodydatasystems_RMD5e816c6f7615a1e8c_at_1470755400_14045 for action execution

0 Karma

somesoni2
Revered Legend

Try the updated answer.

0 Karma

hartfoml
Motivator

Thanks @somesoni2 this did not yield any results. Here is my search

index=_audit action=search search="*" NOT search="'typeahead*" NOT search="'|history*" 
 search_id="*RMD5e816c6f7615a1e8c_at_1470755400_14045" OR id="*RMD5e816c6f7615a1e8c_at_1470755400_14045"

See you at dot conf

0 Karma

somesoni2
Revered Legend

May be try like this, ensure to select proper time range which will include the search execution time
(1470755400)

index=_audit action=search NOT (search="'typeahead*" OR search="'|history*" )  "*RMD5e816c6f7615a1e8c*"
0 Karma
Get Updates on the Splunk Community!

Observability | Use Synthetic Monitoring for Website Metadata Verification

If you are on Splunk Observability Cloud, you may already have Synthetic Monitoringin your observability ...

More Ways To Control Your Costs With Archived Metrics | Register for Tech Talk

Tuesday, May 14, 2024  |  11AM PT / 2PM ET Register to Attend Join us for this Tech Talk and learn how to ...

.conf24 | Personalize your .conf experience with Learning Paths!

Personalize your .conf24 Experience Learning paths allow you to level up your skill sets and dive deeper ...