Hi,
We have gotten quite a complex search request, which we are not sure if is possible at all.
If the application log says "SITE X DOWN", and then within an five minute interval logs "SITE BACKUP X DOWN" it must raise an alert. If the applications logs SITE BAKCUP X DOWN outside the five minute interval everything is OK.
How, if at all possible, could this be accomplished?
--
Espen
You can use the transaction
command. This will join separate events together to a new combined event (a transaction) based on rules that you specify. You can then search for transactions that match multiple conditions.
In your case, you want to find cases where "SITE X DOWN" and "SITE BACKUP X DOWN" occur within a 5 minute interval. This can be accomplished using transaction
like this:
"SITE X DOWN" OR "SITE BACKUP X DOWN"
| transaction source startswith="SITE X DOWN" endswith="SITE BACKUP X DOWN" maxspan=5m
Any events returned by this search will match your condition. I used "source" as an argument to transaction
but any field identifier can be used. This specifies what field(s) Splunk should look for and use when grouping together events, so in this case Splunk will be looking to grouping events into transactions if they have the same value for the "source" field. The more unique the field value, the better.
More information on the transaction
command is available in the docs: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Transaction
This is perfect, thank you.
But is it best run with time range "rt-5 to rt" or just a normal search that ranges the last ten minutes or so?
You can use the transaction
command. This will join separate events together to a new combined event (a transaction) based on rules that you specify. You can then search for transactions that match multiple conditions.
In your case, you want to find cases where "SITE X DOWN" and "SITE BACKUP X DOWN" occur within a 5 minute interval. This can be accomplished using transaction
like this:
"SITE X DOWN" OR "SITE BACKUP X DOWN"
| transaction source startswith="SITE X DOWN" endswith="SITE BACKUP X DOWN" maxspan=5m
Any events returned by this search will match your condition. I used "source" as an argument to transaction
but any field identifier can be used. This specifies what field(s) Splunk should look for and use when grouping together events, so in this case Splunk will be looking to grouping events into transactions if they have the same value for the "source" field. The more unique the field value, the better.
More information on the transaction
command is available in the docs: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Transaction
Just don't use source as a field to run transaction
on.
How would you do it if you have two different source logs?
This solves the problem.
Thank you very much, Ayn
Excellent answer, clearly explained