All Apps and Add-ons

Splunk Machine Learning Toolkit: error message when importing new algorithm

mhyipa
Engager

I followed the procedure in API Guide 2.4.0 to add the CorrelationMatrix algorithm. But an error resulted.

I am using Windows 10.

I first added [CorrelationMatrix] in algos.conf located at ~\etc\apps\Splunk_ML_Toolkit\default\algos.conf
Then, I added a file CorrelationMatrix.py at ~\etc\apps\Splunk_ML_Toolkit\bin\algos

The .py file was copied from the "Finished example" in the API Guide.

Finally, I restarted Splunk and searched "| inputlookup iris.csv | fit CorrelationMatrix petal* sepal*"

The error message was:

Error in 'fit' command: Error while initializing algorithm "CorrelationMatrix": Algorithm "CorrelationMatrix" cannot be loaded

The system log is:

10-08-2017 11:35:38.579 INFO dispatchRunner - Search process mode: freestanding
10-08-2017 11:35:38.580 INFO dispatchRunner - initing LicenseMgr in search process: nonPro=0
10-08-2017 11:35:38.580 INFO LicenseMgr - Initing LicenseMgr
10-08-2017 11:35:38.583 INFO ServerConfig - Found no hostname options in server.conf. Will attempt to use default for now.
10-08-2017 11:35:38.583 INFO ServerConfig - Host name option is "".
10-08-2017 11:35:38.604 INFO ServerConfig - SSL session cache path enabled 0 session timeout on SSL server 300.000
10-08-2017 11:35:38.606 INFO LMConfig - serverName=xxxxxxxx guid=xxxxxxxxxxxxxxxxxx
10-08-2017 11:35:38.606 INFO LMConfig - connection_timeout=30
10-08-2017 11:35:38.606 INFO LMConfig - send_timeout=30
10-08-2017 11:35:38.613 INFO LMConfig - receive_timeout=30
10-08-2017 11:35:38.613 INFO LMConfig - squash_threshold=2000
10-08-2017 11:35:38.613 INFO LMConfig - strict_pool_quota=1
10-08-2017 11:35:38.613 INFO LMConfig - key=pool_suggestion not found in licenser stanza of server.conf, defaulting=''
10-08-2017 11:35:38.613 INFO LMConfig - key=test_aws_metering not found in licenser stanza of server.conf, defaulting=0
10-08-2017 11:35:38.613 INFO LMConfig - key=test_aws_product_code not found in licenser stanza of server.conf, defaulting=0
10-08-2017 11:35:38.613 INFO LicenseMgr - Initing LicenseMgr runContext_splunkd=false
10-08-2017 11:35:38.613 INFO LMStackMgr - closing stack mgr
10-08-2017 11:35:38.613 INFO LMSlaveInfo - all slaves cleared
10-08-2017 11:35:38.613 INFO LMStackMgr - partial init only since node has remote master=https://splunk.xxxxxx:8089
10-08-2017 11:35:38.613 INFO LicenseMgr - StackMgr init complete...
10-08-2017 11:35:38.613 INFO LMTracker - Setting default product type='enterprise'
10-08-2017 11:35:38.613 INFO LMTracker - this is not splunkd, will perform partial init
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=Acceleration state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=AdvancedSearchCommands state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=AdvancedXML state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=Alerting state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=ArchiveToHdfs state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=Auth state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=CustomRoles state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=DeployClient state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=DeployServer state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=DistSearch state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=FwdData state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=GuestPass state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=KVStore state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=LDAPAuth state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=LocalSearch state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=MultifactorAuth state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=MultisiteClustering state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=NontableLookups state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=RcvData state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=RcvSearch state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=RollingWindowAlerts state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=SAMLAuth state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=ScheduledAlerts state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=ScheduledReports state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=ScheduledSearch state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=ScriptedAuth state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=SearchheadPooling state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=SigningProcessor state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=SplunkWeb state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=SubgroupId state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=SyslogOutputProcessor state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LMTracker - Setting feature=UnisiteClustering state=ENABLED (featureStatus=1)
10-08-2017 11:35:38.613 INFO LicenseMgr - Tracker init complete...
10-08-2017 11:35:38.618 INFO AdminManagerDispatch - added factory for admin handler: 'licenses'
10-08-2017 11:35:38.618 INFO AdminManagerDispatch - added factory for admin handler: 'pools'
10-08-2017 11:35:38.618 INFO AdminManagerDispatch - added factory for admin handler: 'stacks'
10-08-2017 11:35:38.618 INFO AdminManagerDispatch - added factory for admin handler: 'groups'
10-08-2017 11:35:38.618 INFO AdminManagerDispatch - added factory for admin handler: 'slaves'
10-08-2017 11:35:38.618 INFO AdminManagerDispatch - added factory for admin handler: 'localslave'
10-08-2017 11:35:38.618 INFO AdminManagerDispatch - added factory for admin handler: 'licensermessages'
10-08-2017 11:35:38.618 INFO AdminManagerDispatch - added factory for admin handler: 'scriptedwarning'
10-08-2017 11:35:38.618 INFO AdminManagerDispatch - added factory for admin handler: 'licenseusage'
10-08-2017 11:35:38.618 INFO dispatchRunner - Per-process handle limit is 512
10-08-2017 11:35:38.618 INFO dispatchRunner - Increasing per-process handle limit from '512' to '2048'
10-08-2017 11:35:38.618 INFO dispatchRunner - Successfully increased per-process handle limit from '512' to '2048'
10-08-2017 11:35:38.621 INFO dispatchRunner - registering build time modules, count=1
10-08-2017 11:35:38.621 INFO dispatchRunner - registering search time components of build time module name=vix
10-08-2017 11:35:38.621 INFO dispatchRunner - Splunkd starting (build e21ee54bc796).
10-08-2017 11:35:38.621 INFO dispatchRunner - System info: Windows, xxxxxx, 3, 6, x64.
10-08-2017 11:35:38.621 INFO dispatchRunner - Detected 4 (virtual) CPUs, 2 CPU cores, and 16267MB RAM
10-08-2017 11:35:38.621 INFO dispatchRunner - Maximum number of threads (approximate): 8133
10-08-2017 11:35:38.621 INFO dispatchRunner - Arguments are: "search" "--id=xxxxxxxx" "--maxbuckets=0" "--ttl=600" "--maxout=500000" "--maxtime=8640000" "--lookups=1" "--reduce_freq=10" "--user=admin" "--pro" "--roles=admin:can_delete:power:splunk-system-role:user"
10-08-2017 11:35:38.621 INFO dispatchRunner - Getting search configuration data from: C:\Program Files\Splunk\etc\modules\parsing\config.xml
10-08-2017 11:35:38.622 INFO BundlesSetup - Setup stats for C:\Program Files\Splunk\etc: wallclock_elapsed_msec=34, cpu_time_used=0.03125, shared_services_generation=1, shared_services_population=1
10-08-2017 11:35:38.625 INFO UserManagerPro - Load authentication: forcing roles="admin, can_delete, power, splunk-system-role, user"
10-08-2017 11:35:38.626 INFO SessionManager - auth tokens will be generated with shpooling shared secret
10-08-2017 11:35:38.626 INFO UserManager - Setting user context: splunk-system-user
10-08-2017 11:35:38.626 INFO UserManager - Done setting user context: NULL -> splunk-system-user
10-08-2017 11:35:38.626 INFO UserManager - Unwound user context: splunk-system-user -> NULL
10-08-2017 11:35:38.627 INFO UserManager - Setting user context: admin
10-08-2017 11:35:38.627 INFO UserManager - Done setting user context: NULL -> admin
10-08-2017 11:35:38.627 INFO dispatchRunner - search context: user="admin", app="Splunk_ML_Toolkit", bs-pathname="C:\Program Files\Splunk\etc"
10-08-2017 11:35:38.631 INFO dispatchRunner - Executing the DispatchThread.
10-08-2017 11:35:38.631 INFO SearchParser - PARSING: | inputlookup iris.csv | fit CorrelationMatrix petal* sepal*
10-08-2017 11:35:38.640 INFO ISplunkDispatch - Not running in splunkd. Bundle replication not triggered.
10-08-2017 11:35:38.644 INFO UserManager - Setting user context: admin
10-08-2017 11:35:38.644 INFO UserManager - Done setting user context: NULL -> admin
10-08-2017 11:35:38.652 INFO ChunkedExternProcessor - Running process: "C:\Program Files\Splunk\bin\python.exe" "C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\fit.py"
10-08-2017 11:35:38.919 INFO ChunkedExternProcessor - stderr: Running C:\Program Files\Splunk\etc\apps\Splunk_SA_Scientific_Python_windows_x86_64\bin\windows_x86_64\python.exe C:\Program Files\Splunk\etc\apps\Splunk_ML_Toolkit\bin\fit.py
10-08-2017 11:35:40.554 ERROR ChunkedExternProcessor - Error in 'fit' command: Error while initializing algorithm "CorrelationMatrix": Algorithm "CorrelationMatrix" cannot be loaded
10-08-2017 11:35:40.554 INFO UserManager - Unwound user context: admin -> NULL
10-08-2017 11:35:40.563 INFO UserManager - Setting user context: admin
10-08-2017 11:35:40.563 INFO UserManager - Done setting user context: NULL -> admin
10-08-2017 11:35:40.563 INFO UserManager - Unwound user context: admin -> NULL
10-08-2017 11:35:40.563 INFO DispatchManager - DispatchManager::dispatchHasFinished(id='xxxxxxxxx', username='admin')
10-08-2017 11:35:40.952 INFO UserManager - Unwound user context: admin -> NULL
10-08-2017 11:35:40.958 INFO ShutdownHandler - Shutting down splunkd
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_Begin"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_FileIntegrityChecker"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_JustBeforeKVStore"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_KVStore"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_Thruput"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_TcpInput1"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_ExecSendInitialSigterm"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_TcpOutput"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_UdpInput"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_FifoInput"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_WinEventLogInput"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_HttpInput"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_CacheManager"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_Scheduler"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_SyslogOutput"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_HTTPOutput"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_ArchiveAndOneshot"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_MainThread"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_Exec"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_Tailing"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_PeerManager"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_AuditTrailManager"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_AuditTrailQueueServiceThread"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_FSChangeMonitor"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_FSChangeManagerProcessor"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_DeploymentClient"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_HttpClientPollingThread"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_AsyncQueuedMessageDispatcherThread"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_OfflineFlusher"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_Slave"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_SlaveSearch"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_Captain"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_Select"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_IdataDO_Collector"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_TcpOutput2"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_IndexerService"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_Database1"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_LastIndexerLevel"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_AWSMetering"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_TcpInput2"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_SearchDispatch"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_LoadLDAPUsers"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_MetricsManager"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_Pipeline"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_Queue"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_CallbackRunner"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_HttpClient"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_DmcProxyHttpClient"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_Duo2FAHttpClient"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_ApplicationLicenseChecker"
10-08-2017 11:35:40.958 INFO ShutdownHandler - shutting down level "ShutdownLevel_S3ConnectionPoolManager"
10-08-2017 11:35:40.958 INFO ShutdownHandler - Shutdown complete in 0 microseconds
10-08-2017 11:35:40.959 ERROR dispatchRunner - RunDispatch::runDispatchThread threw error: Error in 'fit' command: Error while initializing algorithm "CorrelationMatrix": Algorithm "CorrelationMatrix" cannot be loaded

thanks for enlightenments!

Andy

0 Karma

mhyipa
Engager

Thanks very much aljohnson!

I downloaded curl and registered the algorithm using REST API, then it works.

Seems that manually editing the existing algos.conf is not working in my case.

My MLTK is 2.4.0 and PSC is 1.2 (64 bit)

aljohnson_splun
Splunk Employee
Splunk Employee

Interesting. I am glad to hear that it is working now!

0 Karma

aljohnson_splun
Splunk Employee
Splunk Employee

hi @mhyipa - likely the documentation example does not have perfect whitespace.

can you

1.) paste the file contents here in a source block

2.) try using the correlation matrix available in this example repo and see if you have the same problem?
https://github.com/metasyn/mlspl-extensibility-examples

3.) which version of MLTK are you using & which version of PSC (Python for Scientific Computing) ?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...