Splunk Search

How to find out where a search is originating from?

hartfoml
Motivator

I have this process running on all my indexes:

[splunkd pid=7803] search --id=remote_SearchHead.local_scheduler__nobody__datasystems__RMD5e816c6f7615a1e8c_at_1470755400_14045 --maxbuckets=0 --ttl=60 --maxout=0 --maxtime=0 --lookups=1 --streaming --outCsv=true --user=splunk-system-user --pro --roles=admin:power:splunk-system-role:user

I can tell that it is coming from the search head and from the datasystems app, but I cannot find the search name or where it is scheduled to run to stop it. It is running as "nobody" or "splunk-system-user", so it is hard to ID who is running the search.

I can kill the pid but it comes right back.

0 Karma

somesoni2
Revered Legend

It's a scheduled search so you'd find it's traces in scheduler logs

Try this

index=_internal sourcetype=scheduler sid="Copy the id field from your process description e.g. remote_SearchHead.local_scheduler__nobody__datasystems__RMD5e816c6f7615a1e8c_at_1470755400_14045" 

The output will contain a field called savedsearch_id, which will include, owner;AppName;Saved search name.

Update#1
I'm guessing you're killing the search before it's completed, so there is no search completion records in above query.

Try this alternative method/place. You already have the owner (nobody) and app name. This will give your saved search name.

index=_audit action=search search="*" NOT search="'typeahead*" NOT search="'|history*" 
search_id="Copy the id field from your process description " OR id="Copy the id field from your process description"
0 Karma

hartfoml
Motivator

I did the search back 7 days and this is the result... Only one log entry

08-09-2016 11:28:19.963 -0500 INFO SavedSplunker - AlertNotifier::execute: queued sid=scheduler_nobodydatasystems_RMD5e816c6f7615a1e8c_at_1470755400_14045 for action execution

0 Karma

somesoni2
Revered Legend

Try the updated answer.

0 Karma

hartfoml
Motivator

Thanks @somesoni2 this did not yield any results. Here is my search

index=_audit action=search search="*" NOT search="'typeahead*" NOT search="'|history*" 
 search_id="*RMD5e816c6f7615a1e8c_at_1470755400_14045" OR id="*RMD5e816c6f7615a1e8c_at_1470755400_14045"

See you at dot conf

0 Karma

somesoni2
Revered Legend

May be try like this, ensure to select proper time range which will include the search execution time
(1470755400)

index=_audit action=search NOT (search="'typeahead*" OR search="'|history*" )  "*RMD5e816c6f7615a1e8c*"
0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...