Splunk Search

I need to alert when one value from last 24 hours multiplied by 2 differs from dedup of 2 fields from the past 60 minutes

dorgra
Path Finder

I have 36 servers that forward event sources with 2 distinct values. I need to compare the number of system names (from previous 24 hours) * 2 and compare that result to a dedup of system_name, and event_source (from past 60 minutes). If the results don't match, I need an alert. One more problem is, the number of system names is subject to change.

index=myIndex sourcetype=mySourcetype 
    [ search index=myIndex sourcetype=mySourcetype earliest=-1d
    | dedup system_name
    | stats count(eval(system_name*2)) as SysNmCount ]
    [ search index=myIndex sourcetype=mySourcetype earliest=-60m
    | dedup event_source, system_name
    | stats count as EvntCount ]
| stats count(eval(SysNmCount/EvntCount)) as FinalCount

This search returns 0. I was thinking I could use a

| Where 

clause to alert when the FinalCount doesn't equal 1 when I get the return I need. I would like to include the

dedup event_source, system_name
in the alert email so the system operators can start looking for the missing combination(s) problem.

The search that works right now has the value of 72, but it changes from time to time.
index=myIndex sourcetype=mySourcetype

| dedup event_source, system_name
| eventstats count as total 
| where (total < 72 OR total > 72)
| fields system_name event_source

Any suggestion would be appreciated.

0 Karma

dorgra
Path Finder

See final comment for the solution that works.

0 Karma

dorgra
Path Finder

Researching got me a little further. Since I have 2 subsearches, I use "set union". Now I need a table that is created ONLY when SysNmCount and EvntCount are different.

Any suggestions?

| set union
[search index=myIndex sourcetype=mySourcetype earliest=-60m@m
| dedup system_name event_source
| stats count as EvntCount]
[search index=myIndex sourcetype=mySource earliest=-24h
| dedup system_name
| stats count as total
| eval SysNmCount = total * 2]
| table SysNmCount EvntCount

0 Karma

dorgra
Path Finder

I finally figured it out. Instead of "diff" or "union" or "makeresults" there is a simple solution. Sorry, I'm still new to SPL.

index=myIndex sourcetype=mySourcetype earliest=-24h NOT
[search index=myIndex sourcetype=mySourcetype earliest=-1h
| dedup system_name event_source
| fields system_name event_source
| table system_name, event_source]
| dedup system_name event_source
| fields system_name event_source
| table system_name, event_source
| eventstats count as total
| where total > 0

Setting this to run every hour (at 8 minutes past for resource issues) as an alert returns the event sets that are there for 24 hours but not the previous hour.

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...