Comments and answers for "Cannot reproduce Predict command confidence interval"
https://answers.splunk.com/answers/743580/cannot-reproduce-predict-command-confidence-interv.html
The latest comments and answers for the question "Cannot reproduce Predict command confidence interval"Comment by skoelpin on skoelpin's comment
https://answers.splunk.com/comments/743927/view.html
You should create your own limits so you understand exactly how it works and can adjust it as needed.. You'll ned to knock the dust off those old statistics books and find the equations for the UCL and LCL and apply it here. I've posted about this a lot, here's an example
This is the output of a query after the arguements have been passed into the macro, so some of the numerical values will represent a value you pass. These include, confidence interval
| eval upper=if((count > pred),count,pred), lower=if((count < pred),count,pred), lower=if((lower == 0),"",lower)
| eventstats avg(count) AS pred, stdev(count) as pred_stdev, by time, customer
| eval upper=if((upper > (pred + (1 * pred_stdev))),((pred_stdev * 0.5) + pred),upper), lower=if((lower < (pred - (1 * pred_stdev))),((pred_stdev * 0.5) + pred),lower)
| stats avg(count) AS pred, stdev(upper) AS ustdev, stdev(lower) AS lstdev stdev(count) as stdev by time, customer
| eval low=(pred - (lstdev * (exact(3.1622776601683795)))), low=if((low < 0),1,low), high=(pred + (ustdev * (exact(3.1622776601683795)))), _time=timeMon, 06 May 2019 13:27:12 GMTskoelpinComment by jaideeplamba on jaideeplamba's comment
https://answers.splunk.com/comments/743831/view.html
Thank you for your help. I will look into the book and get back.Fri, 03 May 2019 20:08:00 GMTjaideeplambaComment by nnguyen_splunk on nnguyen_splunk's comment
https://answers.splunk.com/comments/743810/view.html
The LL algorithm follows the book "Time Series Analysis by State Space Method" by Durbin and Koopman, chapter 2 "Local Level Method".Fri, 03 May 2019 15:43:55 GMTnnguyen_splunkComment by jaideeplamba on jaideeplamba's comment
https://answers.splunk.com/comments/743746/view.html
You are right in your count. But it is a small subset of data to illustrate the point. Is it possible to share some more details on the underlying calculations behind mean, variance and confidence interval for LL algorithm specifically.Fri, 03 May 2019 03:45:36 GMTjaideeplambaComment by jaideeplamba on jaideeplamba's answer
https://answers.splunk.com/comments/743745/view.html
This is sample dataset from the original dataset. Objective is not to predict responseTime but to detect anomaly and define confidence band. Bigger ask is to decide between splunk or python algorithm to move forward.
Hope that helps.Fri, 03 May 2019 03:38:42 GMTjaideeplambaAnswer by skoelpin
https://answers.splunk.com/answering/743928/view.html
Do you think you can realistically predict response time? Is there a known pattern to learn from? In my experience, you will have better luck setting this up as a mutli-variant model using multiple input variables to get your target valueThu, 02 May 2019 23:11:23 GMTskoelpinComment by nnguyen_splunk on nnguyen_splunk's comment
https://answers.splunk.com/comments/743738/view.html
This is too little data for me to really judge, but I counted 3 out of 10 Actual_RT that are greater than Python_RT_Upper95. That's not good for a 95% confidence interval.Thu, 02 May 2019 21:46:02 GMTnnguyen_splunkComment by jaideeplamba on jaideeplamba's comment
https://answers.splunk.com/comments/743731/view.html
Sorry for the typo. Reran the simulation and here are the results.
_time Splunk_RT Splunk_RT_Upper95 Python_RT Python_RT_Upper95 Actual_RT
1 29 39 29 30 27
2 30 41 31 32 33
3 30 41 31 32 30
4 29 39 29 30 26
5 31 41 31 32 33
6 31 41 32 33 32
7 30 40 30 32 28
8 32 42 32 34 35
9 32 42 33 34 32
10 31 41 31 32 28
p.s.: Cant upload any more screenshots.Thu, 02 May 2019 20:36:29 GMTjaideeplambaComment by nnguyen_splunk on nnguyen_splunk's answer
https://answers.splunk.com/comments/743726/view.html
Hmm, not sure why the Python Upper95 curve (blue one) is below the Python prediction (yellow)?Thu, 02 May 2019 20:01:25 GMTnnguyen_splunkAnswer by jaideeplamba
https://answers.splunk.com/answering/743725/view.html
I apologize for the confusion. Actually both the algorithms start similarly and then deviate. So here is a sample from the data where they have deviated significantly. I have attached a screenshot to explain better.
![alt text][1]
[1]: /storage/temp/272611-screen-shot-2019-05-02-at-25629-pm.pngThu, 02 May 2019 19:57:26 GMTjaideeplambaComment by Sukisen1981 on Sukisen1981's comment
https://answers.splunk.com/comments/744626/view.html
Having a bit of a difficulty in understanding the outputs from splunk and python, you say the predictions are quite simlar. Can you highlight from the output snippets what is simlar and what is widely divergent?
The numbers are very close to each other in your snap and its difficult to understandThu, 02 May 2019 19:33:48 GMTSukisen1981Comment by jaideeplamba on jaideeplamba's comment
https://answers.splunk.com/comments/743719/view.html
Thank you for your reply. I am using non-seasonal data for validation and hence selected LL.
I can understand that there is some difference between Kalman in Python (filterpy library) vs in Splunk. But predicted means are around the same values in both cases. Confidence interval are significantly apart which is puzzling me. My understanding is upper95 bound should be mean+1.96*variance. If means are similar then variance is different in two algorithms. More details around that are hight appreciated.
RegardsThu, 02 May 2019 19:24:42 GMTjaideeplambaComment by Sukisen1981 on Sukisen1981's comment
https://answers.splunk.com/comments/744587/view.html
The key point and from splunk docs - All the algorithms are variations based on the Kalman filter
https://docs.splunk.com/Documentation/Splunk/7.2.6/SearchReference/Predict
So, in essence there are bound to be differences when you compare this with an exact kalman implementation from python. If , for instance you would have used linear regression or k-means clustering for other use cases, the outputs would have tallied exactly with the python libraries.
Depends upon your use case but within the limitations of the kalman filter in general, splunk does a pretty good job of implementing the same.Thu, 02 May 2019 17:13:54 GMTSukisen1981Comment by nnguyen_splunk
https://answers.splunk.com/comments/743681/view.html
Hello,
Since you set the period=96, you may want to use algorithm=LLP5. THe 'LL' algorithm is for non-periodic data.Thu, 02 May 2019 16:39:24 GMTnnguyen_splunk