All Apps and Add-ons

Machine Learning Toolkit: Is there a way to import a trained model into Splunk?

Rinkesh88
Engager

Hi, I am training a random forest model with large amount of data outside of Splunk (on spark). I need to import this model into Splunk. I am not able to use the Splunk Machine Learning Toolkit as training a random forest with large data and n_estimators>100 results in memory errors. Is there a way to import a trained model into Splunk which can then be used to "apply" on new data? Thanks.

yangzd
Splunk Employee
Splunk Employee

Hi,

As of now, MLTK does not support importing a trained model from outside of Splunk.

There are several ways that you could try:

  1. Use random forest algorithm in MLTK to train a native MLTK model, you can just change the max_memory_usage_mb stanza in the mlspl.conf file to allow higher memory usage.

  2. Write a custom algorithm that reads your trained model and translate the parameters and pass into sklearn models, then run on your data. It may not be trivial in your use case due to the complexity of decision tree.

  3. Write a custom search command that sends Splunk data to your environment where you trained your model, make predictions and send back to Splunk. This requires good understanding of Splunk custom search command and extra integration work.

grana_splunk
Splunk Employee
Splunk Employee

Hi,

Currently we do not support importing of models in Machine Learning Toolkit but I have a workaround for memory errors

  1. Go to /home/splunk/splunk/etc/apps/Splunk_ML_Toolkit/default/mlspl.conf and increase memory limit for your algorithm
  2. Restart Splunk
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...