Splunk Search

How to search for a missing word from an indexed log and alert if the word is not found in the last 15 minutes?

iggydolby2
Loves-to-Learn Lots

I will like to search for a missing word like "main" on an indexed log and alert if that word is not found in the last 15 minutes
I do not want to get any other results like "everything else but main"
We generate thread dumps and if the "main" thread is missing from the dump it will indicate that the process is up but not processing.

0 Karma
1 Solution

Vijeta
Influencer

@iggydolby2 - try something like below, based on thise search you can trigger alert if result.count >=1

index=<your index name> source=<your sourcename> |eval x= if(match(_raw,"main"), 1,0)| eventstats sum(x) as sum| where sum=0

View solution in original post

0 Karma

goodair
New Member

When you define an alert, you have the option of setting the trigger condition to fire when number of results is equal to 0. Then all you need to do for your search is try to find that word in your logs, which will trigger the alert if it doesn't find it:

index=where_you_are_looking "main"

https://docs.splunk.com/Documentation/Splunk/7.2.6/Alert/AlertTriggerConditions

0 Karma

iggydolby2
Loves-to-Learn Lots

Thank you for the advice. Actually it is working now for me with your original search query. I'm not sure if I was doing something wrong but it works for me and already setup the alerts.

0 Karma

goodair
New Member

No worries, glad it is working in the end. Looks like you accepted the wrong answer as accepted though!

0 Karma

rmjharris
Path Finder

A simple NOT "word" will return events that do not contain that string.

index=xxxhub host="xxxxx" source=/logs/SystemOut.log NOT "Exceptions"

Alert on Number of Results is greater than 0.

0 Karma

Vijeta
Influencer

@iggydolby2 - try something like below, based on thise search you can trigger alert if result.count >=1

index=<your index name> source=<your sourcename> |eval x= if(match(_raw,"main"), 1,0)| eventstats sum(x) as sum| where sum=0
0 Karma

iggydolby2
Loves-to-Learn Lots

This is not working.... I've tried a word that shows up on every JAVA app SystemOut.log like "Exception" and that search query shows me everything but "Exceptions" on the search results.
What I will like is a query that should only show any results if the word "Exceptions" is missing.
Here is the result of your suggested query:

index=xxxhub host="xxxxx" source=/logs/SystemOut.log |eval x= if(match(_raw,"exceptions"), 1,0)| eventstats sum(x) as sum| where sum=0

Results I only pasted a few lines but it's full of results.....
I should see NO results because the word "Exception" is NOT missing but appears multiple times....
Again, I want to see results if the "word" is missing from the log.

[5/21/19 13:03:08:696 PDT] 0000010d SAMLDefaultLo I org.springframework.security.saml.log.SAMLDefaultLogger log AuthNRequest;SUCCESS; xx.00.00.103;xxxx.com;;;
host = xxxxx.com source = /logs/SystemOut.log sourcetype = sysout
5/21/19
1:03:07.411 PM

[5/21/19 13:03:07:411 PDT] 000000fe LoggingOutInt I org.apache.cxf.interceptor.LoggingOutInterceptor Outbound Message

ID: 25179
Response-Code: 200
Content-Type: application/xml;charset=UTF-8
Show all 8 lines

0 Karma

Vijeta
Influencer

@iggydolby2 Please make sure you use the exact word with the proper case as match is case sensitive. This should work as per this logic it will assign value of 1 to x sum when any event contains Exception word that means sum will be >0 if there is any event which contains exact match for word Exception.

0 Karma

iggydolby2
Loves-to-Learn Lots

Thanks but I'm still getting results.....
it does not matter the case..... I should get NO results because "Exception" is NOT missing.
Try it yourself please and you will see..... find a word that is on every log during the time span analyzed.
Then do the query to see if you get a result if that word is missing........

0 Karma

Vijeta
Influencer

@iggydolby2 I tried and I do not get any results when searching on Error , if i change my condition sum > 0 only then I am getting events. See below example it returns no events as there are events with word Error and yes it expects to be an exact match.

index="_internal" | eval x= if(match(_raw,"Error"), 1,0) | eventstats sum(x) as sum | where sum =0
0 Karma

rmjharris
Path Finder

This won't work because sum(x) is only going to be 0 if the string, "Error" in this case, is not present in any event. If x=1 for any event then sum is going to be greater than 0.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...