Knowledge Management

Can I get info about these components in one place in Splunk?

arpit_arora
Explorer

Hello, I am really struggling with trying to find a data source to get all the following pieces of information about the searches which are run on our splunk enterprise cluster. These are the fields I am interesting in finding.

search id
search command
user
time scheduled
time dispatched
total run time
search type (real time, scheduled, ad hoc, remote etc)
status (running, paused, finalized, completed etc)
messages (any error or warn messages)

Tags (1)
0 Karma

traxxasbreaker
Communicator

The scheduler logs are in the _internal index. If you use the Monitoring Console, you'll also find some good stuff that will point you to data sources that will be helpful for looking at scheduler behavior under Search -> Scheduler Activity.

By tweaking one of the Monitoring Console searches a little, I get something like this which covers most of the fields you are looking for:

index=_internal host= sourcetype=scheduler
| eval alert_actions = if(isnull(alert_actions) OR alert_actions == "", "none", alert_actions)
| eval window_time = if(isnotnull(window_time), window_time, 0)
| eval execution_latency = max(dispatch_time - (scheduled_time + window_time), 0)
| dedup sid
| table sid, user, app,scheduled_time, dispatch_time, run_time, result_count, status

However, that will only give you info for the scheduled searches and not the real time, ad hoc, or remote searches. For those, you'll want to take a look at the _audit index. If you do a transaction on the search_id field, you can look at other fields like search for the search string, is_realtime to determine if it's realtime. The _audit index logs will also tell you the start and end time ranges of the search and how many events were scanned while it was running.

mattymo
Splunk Employee
Splunk Employee

+1 for starting with search section in the monitoring console! Excellent place to open panels in search and see where it gets it's goodies! It also has a toggle for ad-hoc searches. Otherwise Activity > Jobs in the top right corner of your screen can be a good place if you are digging into particular jobs...

Also shout out to the search activity app https://splunkbase.splunk.com/app/2632/#/details

- MattyMo
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...