Monitoring Splunk

What is Splunkd Doing

hartfoml
Motivator

My indexer is running at 97,8% CPU utalisation and there are about 18 splunkd threads running along with python and splunk_optimizer and ssl_test.pl

How can I tell what all the Splunkd threads are doing and if I can kill some of them to ease the load or at least look at what they are so that i can schedule them to run at different times.

Thanks for your help... 🙂

Tags (2)
0 Karma

yannK
Splunk Employee
Splunk Employee

To check if they are search processes, install the SOS app, and enable the ps_sos scripts (for linux or windows with powershell)
then check the resource usage dashboard of the APP for the details.

To check the scheduler, use the SOS app too, there is a scheduled searches dashboard.

0 Karma

yannK
Splunk Employee
Splunk Employee

red-hat and swap, make me think of this known issue, http://docs.splunk.com/Documentation/Splunk/6.0.3/ReleaseNotes/SplunkandTHP

0 Karma

hartfoml
Motivator

thanks Yann,

This was very good advice. Yesterday I had a busy day and the virtual memory went way high and then dropped down after the work was done. The SOS app does not map PID numbers to searches so I can not see witch Splunkd process is tied to which user/search.

Also the Swap file on my indexer was never released and I had to bounce splunk to get the swap file freed up. I find that when the swap file is close to 100% utilized the server is likely to crash. RHEL 5.x

0 Karma

MuS
SplunkTrust
SplunkTrust

run $SPLUNK_HOME/bin/splunk status to get back a pid list of main splunkd process, splunkweb process and the splunkd helper processes (your searches)

0 Karma

hartfoml
Motivator

Thanks this helps a bit. I was hoping for a way to know which pid was running which job. Something with btools or REST API or something that would tell me which splunkd process is running which job.

Thanks for the help I really appreciate it.

0 Karma

linu1988
Champion

its not the indexer, the search head would initiate the searches.. other indexing activity would also be going on..

0 Karma

hartfoml
Motivator

I logged into the indexer https://MyIndexer:8000 as admin opend the jobs window selected all apps All Owners Status Running ans even though Top shows several Splunkd processes running no jods or at time only a few jobs are running.

thanks this helps a little to see what scheduled or search jobs are running on the indexer

0 Karma

linu1988
Champion

login as admin check the jobs which are running

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...