Splunk Search

Need Help with a Splunk query

phularah
Communicator

So, I have data like this after I ran a query. 

phularah_1-1715356433010.png

For each aggregator, if the aggregator_status is Error and before15 minutes, the aggregator_status becomes Up, alert should not run. But, if the aggregator_status is still Error or no new event comes, alert should trigger. The Time field is epoch time which I am thinking can be used to find difference in Up and Error status times.

How do I create such a query for the alert? I am thinking of using foreach command or some sort of streamstats, but I am unable to resolve this issue. The alert needs to run once every 24 hours.

Labels (4)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

It might be doable with the transaction command but it's usually not a good idea (transaction is a relatively "heavy" command and has its limitations).

I'd go with streamstats and reset_before, reset_after and time_window options. (can't give you a ready-made answer at the moment since I'm away from my Splunk environment but that's the way I'd try)

0 Karma

phularah
Communicator

I tried something like this 

index=abc ("Aggregator * is Error" OR "Aggregator * is Up") NJ12GC102
| rex field=_raw "Aggregator\s(?<aggregator>[^\s]+)\sis\s(?<aggregator_status>\w+)\s"
| streamstats current=t global=f window=2 range(_time) as time_diff by aggregator,aggregator_status
| streamstats current=t global=f window=2 range(_time) as time_diff2 by aggregator
| table _time aggregator aggregator_status time_diff time_diff2
|



But the output is now what I needed. For that I would need to change the window=2, but it brings more issues.

phularah_1-1715358409563.png

 

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Try starting with something like this

| streamstats values(aggregator_status) as previous_aggregator_status by aggregator window=1 current=f global=f
| eval changetime=if((aggregator_status="Up" and previous_aggregator_status="Error") or (aggregator_status="Error" and previous_aggregator_status="Up"),_time,null())
| where isnotnull(changetime)
| streamstats current=t global=f window=2 range(_time) as time_diff2 by aggregator
| where aggregator_status="Error"
0 Karma
Get Updates on the Splunk Community!

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...

Adoption of Infrastructure Monitoring at Splunk

  Splunk's Growth Engineering team showcases one of their first Splunk product adoption-Splunk Infrastructure ...

Modern way of developing distributed application using OTel

Recently, I had the opportunity to work on a complex microservice using Spring boot and Quarkus to develop a ...