Alerting

How to create an alert to identify if there is any lag between the _time and _indextime of firewall events?

OMohi
Path Finder

Hi Guys:

We are facing some delays in getting firewall events getting indexed and displayed on Splunk Search Head. Is there a way to create alerts to identify the time when we are facing lags in getting the events generated.

We are adding an additional indexer, this should overcome the problem.

But in order to be notified as a precautionary step, I would like to know whether we can generate alerts when we face lag times.

For example for some firewall events the events from the network device reach and gets indexed on Splunk indexer after 10 -15 minutes delay.

Your help will be appreciated in this regard.

Thanks,

0 Karma

tpflicke
Path Finder

To elaborate on the answer provided by thomrs:
I've recently set up an alert to keep an eye on lag (including 'negative lag' caused by incorrect timezone setting or server time). Note that the query is looking only at the latest event for each source by using dedup and does not attempt to determine lag by indexer.

index=<index_name> 
| dedup host source 
| eval indexed_time=strftime(_indextime, "%+") 
| eval file_timestamp=strftime(_time, "%+") 
| eval lag = round(_indextime - _time) 
| eval abs_lag=abs(lag) 
| table host file_timestamp indexed_time lag abs_lag source 
| sort -num(abs_lag) 
| search abs_lag > 30 
| rename lag as "lag in seconds" 
| rename abs_lag as "lag in seconds (unsigned)" 
| rename indexed_time as "log event indexed at" 
| rename file_timestamp as "timestamp from log event (adjusted)"

thomrs
Communicator

Nice! Cant type that much on my tablet.

0 Karma

thomrs
Communicator

You can get index time like this

eval indextime = _indextime 

Then compare against _time and create your alert.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...