Seeing old splunk search pids

[DFresh4130]

We have a simple splunk installation where the indexer and server process reside on the same box. When I do a ps -ef | grep splunk I'm seeing a lot of old pids. Below are a couple examples. Are these something that I should kill? Is there a way for me to correlate if they're some kind scheduled search in the web UI?

root     17283     1  0 Jun12 ?        00:00:00 [splunkd pid=8743] search --id=1402598653.142 --maxbuckets=0 --ttl=10 --maxout=50 --maxtime=8640000 --lookups=0 --reduce_freq=10 --user=user1 --pro --roles=admin:can_delete:power:user

root     17284 17283  0 Jun12 ?        00:00:00 [splunkd pid=8743] search --id=1402598653.142 --maxbuckets=0 --ttl=10 --maxout=50 --maxtime=8640000 --lookups=0 --reduce_freq=10 --user=user1 --pro --roles=admin:can_delete:power:user [process-runner]

[drdosia]

Has this issue reappeared? By any chance do you have the *nix app?

The sourcetype=ps is used by the PS- All process info search in the *NIX app.
What version of *NIX app are you running? Check if you are current.
Cheers,
Doc

[lguinn2]

Wow - sounds like some sort of hung process or search. Personally, I haven't seen anything like this in years.

You could do the following:

1 - Kill the process(es)

2 - Delete the associated jobs from the Jobs menu

3 - Make sure the corresponding directories have disappeared from dispatch

I don't know how/why this showed up, but if it recurs, I would open a support ticket.

[DFresh4130]

Deleting the job didn't do it so I had to manually kill the pid on the server. Unfortunately that didn't get rid of the directory in the dispatch folder.

EDIT: just took a couple minutes for the directory to go away.

[lguinn2 · Accepted solution]

Look in $SPLUNK_HOME/var/run/splunk/dispatch for directories that have the same id as the searches in your process list. Within those directories, you should find a file named search.log which will give you more information.

I can tell from the process list that you are looking for searches that are being run by user1. You could also login to the Splunk GUI as an admin and look under Settings (or Manager in older Splunk versions) for Searches & Reports that are owned by user1. You should find searches that correspond to what you found in the search logs.

[DFresh4130]

Ah, that helps. I went into the jobs and found the corresponding job with the same SID as the PID. It shows a status of done though. When I inspect the job this is what it shows in the search field:

typeahead prefix="sourcetype=ps" max_time="1" count="50" use_cache=1

It's not related to any saved searches, report or alerts. Any idea what's going on?

[lguinn2]

You should also be able to go into the Jobs menu as an administrator and see any jobs that are currently running. That should give you the actual search.

[DFresh4130]

I found the corresponding search log, but it doesn't appear to be telling me what search is actually being run. The tail end of the log that tells me what sourcetype is being searched, I can't make heads or tails of the other output.

06-12-2014 14:44:13.187 INFO  SearchParser - PARSING: pretypeahead  prefix="sourcetype=ps" max_time="1" count="50" use_cache=1