All Apps and Add-ons

How can I benchmark my Machine Learning forecasting procedure?

rosho
Communicator

Hi
I am forecasting the number of logins. My dataset has the number of logins by the hour (1 month).

I use a) and b) to clean the data (removing or transforming outliers).
With c) I forecast

(My Machine Learning alert)
a) Standard scaler
b) Detect outliers using DBSCAN
c) Forecast with Kalman filter or MLP

How can I benchmark my results besides from using a test set? Maybe setting an alert (I will call it SPL alert) and later compare it with my Machine Learning alert and check which one failed more?

Do you have any suggestions?

Thank you

0 Karma

gbenitezfelix
Engager

The easiest way to create a benchmark of your model is to use a naive forecasting. This prediction will always use the last period as predicted. You can read about it here https://otexts.com/fpp2/simple-methods.html.

once you implement it you can calculate the error of each forecasting model. If the naive forecast has lower error than your initial model you should consider to tune that model

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...