Knowledge Management

Can I get info about these components in one place in Splunk?

arpit_arora
Explorer

Hello, I am really struggling with trying to find a data source to get all the following pieces of information about the searches which are run on our splunk enterprise cluster. These are the fields I am interesting in finding.

search id
search command
user
time scheduled
time dispatched
total run time
search type (real time, scheduled, ad hoc, remote etc)
status (running, paused, finalized, completed etc)
messages (any error or warn messages)

Tags (1)
0 Karma

traxxasbreaker
Communicator

The scheduler logs are in the _internal index. If you use the Monitoring Console, you'll also find some good stuff that will point you to data sources that will be helpful for looking at scheduler behavior under Search -> Scheduler Activity.

By tweaking one of the Monitoring Console searches a little, I get something like this which covers most of the fields you are looking for:

index=_internal host= sourcetype=scheduler
| eval alert_actions = if(isnull(alert_actions) OR alert_actions == "", "none", alert_actions)
| eval window_time = if(isnotnull(window_time), window_time, 0)
| eval execution_latency = max(dispatch_time - (scheduled_time + window_time), 0)
| dedup sid
| table sid, user, app,scheduled_time, dispatch_time, run_time, result_count, status

However, that will only give you info for the scheduled searches and not the real time, ad hoc, or remote searches. For those, you'll want to take a look at the _audit index. If you do a transaction on the search_id field, you can look at other fields like search for the search string, is_realtime to determine if it's realtime. The _audit index logs will also tell you the start and end time ranges of the search and how many events were scanned while it was running.

mattymo
Splunk Employee
Splunk Employee

+1 for starting with search section in the monitoring console! Excellent place to open panels in search and see where it gets it's goodies! It also has a toggle for ad-hoc searches. Otherwise Activity > Jobs in the top right corner of your screen can be a good place if you are digging into particular jobs...

Also shout out to the search activity app https://splunkbase.splunk.com/app/2632/#/details

- MattyMo
0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...