Comments and answers for "How to represent normal distribution in a graph format using mean and stdev values?"
https://answers.splunk.com/answers/655436/how-to-represent-normal-distribution-in-a-graph-fo.html
The latest comments and answers for the question "How to represent normal distribution in a graph format using mean and stdev values?"Answer by aljohnson_splunk
https://answers.splunk.com/answering/655867/view.html
Hey @chandana204 - there are a few commands that use similar approaches to look for outliers in Splunk. Try checking out the `anomalydetection` command.
http://docs.splunk.com/Documentation/SplunkCloud/7.0.0/SearchReference/Anomalydetection
If you just want to find the mean and stdev, you can checkout the stats command, e.g.
| stats mean(field) as mean, stdev(field) as stdev
If you are just looking for numeric outliers, you can try using the "Detect Numeric Outliers" assistant in the MLTK.
https://docs.splunk.com/Documentation/MLApp/3.2.0/User/DetectNumericOutliers
As for fitting your values directly to a distribution and checking how well it fits, you could hypothetically try to use a custom algorithm in the machine learning toolkit to use a [kolomogorov-smirnoff test](https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test) or a [anderson-darling test](https://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test) to see if your samples match some distribution. You can read about custom algorithms [here](http://docs.splunk.com/Documentation/MLApp/3.2.0/API/Introduction).Thu, 03 May 2018 23:31:50 GMTaljohnson_splunkComment by chandana204 on chandana204's comment
https://answers.splunk.com/comments/655672/view.html
In my case, My whole concept is I want to find type of the distribution on my data. Based on the distribution type I want to find that the data has any outliers or not, if not I want to find the mean and stdev. First I am not sure hoe to represent distribution type in graphic mode using Splunk enterprise not using Machine learning toolkit. Can anyone help on this.
NOTE: Correct me if my concept is wrong cause I am new Data Science/Machine Learning concepts.
Thanks,
ChandanaWed, 02 May 2018 16:38:45 GMTchandana204Answer by aljohnson_splunk
https://answers.splunk.com/answering/655445/view.html
Here's a pure SPL version. Its kinda terrible and fun.
First, lets generate ten thousand values from a [standard normal distribution](https://en.wikipedia.org/wiki/Normal_distribution#Standard_normal_distribution) a using the [box-muller transform](https://en.wikipedia.org/wiki/Box%E2%80%93Muller_transform):
| makeresults count=10000
| eval standard_normal = sqrt(-2 * ln((random() / (pow(2, 31) -1)))) * cos((2*pi()*(random() / (pow(2, 31) -1))))
| fields x standard_normal
Gives us this - 10,000 samples from normal standard distribution
![alt text][1]
------
Since we might want to make multiple of these, we can put it into a macro (read more about making those [here](http://docs.splunk.com/Documentation/SplunkCloud/7.0.0/Knowledge/Definesearchmacros)), which I'll just call `makenormal(3)` - it is parameterized by the number of samples, the mean, and standard deviation. We just scale by the standard deviation, and add the mean to unstandardize our samples.
appendpipe [
| makeresults count=$count$
| eval normal_mu$mean$_sigma$stdev$ = (sqrt(-2 * ln((random() / (pow(2, 31) -1)))) * cos((2*pi()* . (random() / (pow(2, 31) -1)))) * $stdev$) + $mean$
| fields - _time
]
Now that we have our macro, we can use it to compare two values... however, getting them to share the same binned x values (for what you might expect of a standard graph) requires a little SPL trickery.
| stats count
| `makenormal(10000, 10, 2)`
| `makenormal(100000, 2, 10)`
| foreach normal* [bin <<FIELD>> span=0.5]
| untable count field value
| eventstats count by value field
| xyseries value field count
| sort value
which gives us a visualization like this:
![alt text][2]
------
To be frank, I really doubt any one would want to do this in Splunk in this exact fashion... but if you expand your question with more information about what kind of problems you want to solve, or how you intend to use these values from a normal distribution - there are lots of better ways I'm sure. For example, we could use custom search commands, a custom visualization, or even a custom algorithm in the machine learning toolkit to achieve the same thing in a much simpler fashion.
But I just wanted to show it could be done in SPL, too :)
[1]: https://i.imgur.com/nPOEkVs.png
[2]: https://i.imgur.com/Qo16oMG.pngTue, 01 May 2018 02:54:57 GMTaljohnson_splunkComment by aljohnson_splunk
https://answers.splunk.com/comments/655438/view.html
hi @chandana204 - there are a few ways you could do this, but depending on what you're trying to do, some might make more sense than others. what is the purpose of graphing the mean and stdev? Are you looking for a z-test of sorts, for example? It may be easier to calculate some of the statistics without a graph of the full distribution.
Could you tell us more about what you're trying to do?Mon, 30 Apr 2018 22:22:00 GMTaljohnson_splunk