Hi Guys:
We are facing some delays in getting firewall events getting indexed and displayed on Splunk Search Head. Is there a way to create alerts to identify the time when we are facing lags in getting the events generated.
We are adding an additional indexer, this should overcome the problem.
But in order to be notified as a precautionary step, I would like to know whether we can generate alerts when we face lag times.
For example for some firewall events the events from the network device reach and gets indexed on Splunk indexer after 10 -15 minutes delay.
Your help will be appreciated in this regard.
Thanks,
To elaborate on the answer provided by thomrs:
I've recently set up an alert to keep an eye on lag (including 'negative lag' caused by incorrect timezone setting or server time). Note that the query is looking only at the latest event for each source by using dedup and does not attempt to determine lag by indexer.
index=<index_name>
| dedup host source
| eval indexed_time=strftime(_indextime, "%+")
| eval file_timestamp=strftime(_time, "%+")
| eval lag = round(_indextime - _time)
| eval abs_lag=abs(lag)
| table host file_timestamp indexed_time lag abs_lag source
| sort -num(abs_lag)
| search abs_lag > 30
| rename lag as "lag in seconds"
| rename abs_lag as "lag in seconds (unsigned)"
| rename indexed_time as "log event indexed at"
| rename file_timestamp as "timestamp from log event (adjusted)"
Nice! Cant type that much on my tablet.
You can get index time like this
eval indextime = _indextime
Then compare against _time and create your alert.