Hello -
I have a logging event like this one. We are searching on "Threshold Exceeded" AND "225"
9/26/17 13:45:18:690 EDT] 000215d9 SystemOut O 4580330012 [SIBJMSRAThreadPool **: 764**] ERROR com.hdx.routing.saf.SafUtils - ** SAF THRESHOLD EXCEEDED ** currently SAF count is: 100 for Node : BJH/BJC/225/302/4.0 and route info:
When we hit on this we need to search backwards over one min looking for the same ThreadPool ID in the error above it's : 764.
[9/26/17 13:45:18:675 EDT] 000215d9 SystemOut O 4580329994 [SIBJMSRAThreadPool **: 764**] WARN com.hdx.routing.delivery.DeliveryEventHandlerSafV1 - **SAF** Failed sending to node 840153625 at TCPfalsefalse**64.46.236.20****10202**03ACK with RLogPK
For this result we need to pull out the IP / Port and generate an alert. I have not extracted any fields yet.
We are still very new to Splunk. Thanks in advance for the help.
Carl
In splunk, it's almost always "wrong" to think in terms of taking one value and doing another search on that value.
Sometimes, when you have extracted fields, you can pull the values of the field in a subsearch and return it into a main search, but that way you are always going to the well at least twice for the water.
This way, you only have to go once, grab all the records you need, then copy what you need from the exceeded record to thelast record before it.
(index=foo "Threshold Exceeded" "225")
| eval TX = if(match(_raw,"THRESHOLD EXCEEDED") AND match(_raw,"225"),1,0)
| rex "SIBJMSRAThreadPool something something:\s+(?<ThreadPool>\d+)"
| rex "something something (?<MyIP>\d+\.\d+\.\d+\.\d+) something (?<MyPort>\d+)something"
| fields TX ThreadPool MyIP MyPort
| rename COMMENT as "Get rid of everything that is not one or the other"
| where (TX=1 AND isnotnull(ThreadPool)) OR (isnotnull(ThreadPool) AND isnotnull(MyIP) AND isnotnull(MyPort))
| rename COMMENT as "Remember that these record were in reverse time order"
| rename COMMENT as "Copy threshold time back one record, keep only if it is within a minute"
| streamstats current=f last(eval(case(TX=1,_time))) as lastTX window=1 by ThreadPool
| where (TX=0) AND (_time>= lastTX - 60)
| rename COMMENT as "Any record remaining is the last IP/Portrecord before a Threashold Exceeded record"
In splunk, it's almost always "wrong" to think in terms of taking one value and doing another search on that value.
Sometimes, when you have extracted fields, you can pull the values of the field in a subsearch and return it into a main search, but that way you are always going to the well at least twice for the water.
This way, you only have to go once, grab all the records you need, then copy what you need from the exceeded record to thelast record before it.
(index=foo "Threshold Exceeded" "225")
| eval TX = if(match(_raw,"THRESHOLD EXCEEDED") AND match(_raw,"225"),1,0)
| rex "SIBJMSRAThreadPool something something:\s+(?<ThreadPool>\d+)"
| rex "something something (?<MyIP>\d+\.\d+\.\d+\.\d+) something (?<MyPort>\d+)something"
| fields TX ThreadPool MyIP MyPort
| rename COMMENT as "Get rid of everything that is not one or the other"
| where (TX=1 AND isnotnull(ThreadPool)) OR (isnotnull(ThreadPool) AND isnotnull(MyIP) AND isnotnull(MyPort))
| rename COMMENT as "Remember that these record were in reverse time order"
| rename COMMENT as "Copy threshold time back one record, keep only if it is within a minute"
| streamstats current=f last(eval(case(TX=1,_time))) as lastTX window=1 by ThreadPool
| where (TX=0) AND (_time>= lastTX - 60)
| rename COMMENT as "Any record remaining is the last IP/Portrecord before a Threashold Exceeded record"