My indexer is running at 97,8% CPU utalisation and there are about 18 splunkd threads running along with python and splunk_optimizer and ssl_test.pl
How can I tell what all the Splunkd threads are doing and if I can kill some of them to ease the load or at least look at what they are so that i can schedule them to run at different times.
Thanks for your help... 🙂
To check if they are search processes, install the SOS app, and enable the ps_sos scripts (for linux or windows with powershell)
then check the resource usage dashboard of the APP for the details.
To check the scheduler, use the SOS app too, there is a scheduled searches dashboard.
red-hat and swap, make me think of this known issue, http://docs.splunk.com/Documentation/Splunk/6.0.3/ReleaseNotes/SplunkandTHP
thanks Yann,
This was very good advice. Yesterday I had a busy day and the virtual memory went way high and then dropped down after the work was done. The SOS app does not map PID numbers to searches so I can not see witch Splunkd process is tied to which user/search.
Also the Swap file on my indexer was never released and I had to bounce splunk to get the swap file freed up. I find that when the swap file is close to 100% utilized the server is likely to crash. RHEL 5.x
run $SPLUNK_HOME/bin/splunk status
to get back a pid list of main splunkd process, splunkweb process and the splunkd helper processes (your searches)
Thanks this helps a bit. I was hoping for a way to know which pid was running which job. Something with btools or REST API or something that would tell me which splunkd process is running which job.
Thanks for the help I really appreciate it.
its not the indexer, the search head would initiate the searches.. other indexing activity would also be going on..
I logged into the indexer https://MyIndexer:8000 as admin opend the jobs window selected all apps All Owners Status Running ans even though Top shows several Splunkd processes running no jods or at time only a few jobs are running.
thanks this helps a little to see what scheduled or search jobs are running on the indexer
login as admin check the jobs which are running