All Apps and Add-ons

Anomalies detection using Splunk vs R

manishvrmv
New Member

Hi all,

I have a small numerical dataset to perform anomalies detection. My data contains 177 events and I have imputed 3 records to check anomalies using Splunk and R. I applied LOF algo. in R and find out all 3 events. In splunk, I applied anomalies command and generated unexpectedness score for each event and sorted in decreasing order. I found normal events as anomalous and imputed event comes into 6 place. Even all event found as with very less unexpectedness score.

I am just curious to know whether Splunk perform well with machine learning commands such as anomalies, outliers, cluster etc. one more query i have whether splunk work well with numerical data which contains timpstamp.

Tags (1)
0 Karma

prelert
Path Finder

If the dataset is something like:

_time, value
10:00:01, 3.22
10:00:04, 32.22
...

An effective approach to identify anomalies is to create a statistical model of the numerical values, and computing the probability of a specific data value. If the probability is low, then the value is anomalous.

Generally, to accurately model these data and avoid false positives these models needs to be more sophisticated than a simple Normal distribution. In addition, these data are generally periodic and so the models need to allow for daily and weekly patterns.

LOF methods can be effective on static low dimensional datasets, but suffer from similar issues to kernel density functions (overfitting, linear space complexity etc.).

Further details are available here:
http://www.ijmlc.org/papers/398-LC018.pdf

We have built an app to automatically identify anomalies in numeric and categorical data using these techniques:

http://apps.splunk.com/app/1306/

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...