Hello,
I do not have access to the OS machines of the Splunk but I suspect the CPU bottleneck because my alert jobs are having 3 min lag between scheduling and dispatching. I would like to investigate it further.
Is there any way to query the internal index for the CPU utilization of the SH or indexer?
Kind Regards,
Kamil
Another good place to look for bottlenecks are in the monitoring console -> indexing -> performance -> indexing performance:deployment... then look for the queues at 90th percentile and see how much they are utilized... these indexing queues will fill up and cause performance issues, which is a good place to start troubleshooting.
next steps would be to see which sourcetypes are consuming the most resources and optimize them using props/transforms.
additionally alerting is triggered by your search-heads so make sure your sh resources are ok too, you can see them in a similar fashion in the monitoring console under search -> activity
Another good place to look for bottlenecks are in the monitoring console -> indexing -> performance -> indexing performance:deployment... then look for the queues at 90th percentile and see how much they are utilized... these indexing queues will fill up and cause performance issues, which is a good place to start troubleshooting.
next steps would be to see which sourcetypes are consuming the most resources and optimize them using props/transforms.
additionally alerting is triggered by your search-heads so make sure your sh resources are ok too, you can see them in a similar fashion in the monitoring console under search -> activity