Splunk Search

What are your best anomaly finding searches?

carasso
Splunk Employee
Splunk Employee

Besides the obvious things of looking for rare field values...

  • what are all the list of anomaly searches you use to find
    unexpected events?

  • How do you limit false positives, or uninteresting anomalies?

the more specific and the more searches you can list out, the better.

.

prelert
Path Finder

Feedback from our Splunk customers showed the top anomalies they were looking for were:

  • increase/decrease in numeric value in an event
  • increase/decrease in event rate
  • increase/decrease in event rate of a field value
  • rare field values
  • unusual field values compared to other field values

These can be satisfied by running:

   | prelertautodetect metric_value by metric_name
   | prelertautodetect count
   | prelertautodetect count by field_name
   | prelertautodetect rare by field_name
   | prelertautodetect metric_value over field_value
   | prelertautodetect count over field_value

Some specific customer examples are shown here http://support.prelert.com/customer/portal/articles/1355584-examples-overview

To limit false positives, we've found that it is key to apply accurate statistical models to these data. In particular, modelling the tails of probability distributions accurately is key to reducing false positives. In addition, automatically modelling the periodic and seasonal components means that you can model the residuals, which again improves accuracy.

Finally, we've found that normalising the results allows the signal to noise ratio to be controlled, providing an accurate ranking of results in highly anomalous environments.

Happy to provide more customer examples as required.

0 Karma

kristian_kolb
Ultra Champion

blah blah | eval p = punct | stats c(p) as cpu values(_raw) by punct | sort cpu

will show the full _raw for uncommon punct. A bit like rare for the whole event so-to-speak.

/k

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...