How to track memory/cpu usage per search execution (on Search Head/Indexer)?

[melonman]

Hi

I am looking for a way to track memory/cpu usage per search execution on search head and indexer.
I thought I could use _introspection index to track it, but I can not find process resource information in there.

I am currenly testing with splunk6.4.0 on MacOS, and I am trying to monitor searches on Search head Splunk 6.3.2 and Indexer Splunk 6.0.2.

Could anyone comment on this?

Thank you,

[gjanders]

There are a few dashboards in Alerts for Splunk Admins (splunkbase) or github that might help with tracking down the issues here. There are also alerts / reports to detect dashboard or saved searches with index=* or similar.

In particular for the dashboards:
troubleshooting_indexer_cpu
troubleshooting_resource_usage_per_user
detect_excessive_search_use

Saved searches:
SearchHeadLevel - Scheduled searches not specifying an index
SearchHeadLevel - User - Dashboards searching all indexes
SearchHeadLevel - Scheduled Searches without a configured earliest and latest time

[spunk311z]

Lots of great info and search queries in this thread (thanks),  splunk really is amazing!

One thing i can contribute is this search (below) that i often use to show all of my scheduled reports (it pairs nicely with some of the resource usage searches in this thread to help ID and modify your scheduled reports or their cron entry).

Also its nice to review this from time to time as its easy to loose track of cron scheduled reports you may no longer need to run (or run as frequently);

| rest /servicesNS/-/-/saved/searches  | search is_scheduled=1 | table author cron_schedule is_scheduled schedule_window title updated embed.enabled Search

 thanks

[VatsalJagani]

Hi @melonman,

If you want to search CPU and memory utilization per search execution with relevant information like which used executed and more.

index=_introspection host=* source=*/resource_usage.log* component=PerProcess data.process_type="search" 
 | stats latest(data.pct_cpu) AS resource_usage_cpu latest(data.mem_used) AS resource_usage_mem by data.pid, _time, data.search_props.type,data.search_props.mode, data.search_props.role,data.search_props.user, data.search_props.app, data.search_props.sid

[MuS]

Hi melonman,

Did you check out the Distributed Management Console http://docs.splunk.com/Documentation/Splunk/6.3.2/DMC/DMCoverview this should provide data for the search head.
Regarding the indexer try this search

host=YourHostNameHere sourcetype=splunk_resource_usage index=_introspection component=PerProcess "data.process_type"=search

Hope this helps ...

cheers, MuS