How can I programmatically monitor data model acceleration progress?

[Glenn]

I want to add some monitoring of data model acceleration to inform us when it is falling behind (as this can have quite serious effects on our dashboard performance). However, so far I have not been able to find a way to do this programmatically (ie. other than by physically looking at the details of the data model in the manager list.

I found the following REST API endpoint URI "datamodel/acceleration/{name}", but this is now deprecated (and seemed to not actually have a value for progress anyway).
http://docs.splunk.com/Documentation/Splunk/6.3.1/RESTREF/RESTknowledge#datamodel.2Facceleration_.28...

The completely undocumented "| datamodelinfo" command exists but only returns results for 2 of our approx. 100 data models for some unknown reason.

So three questions...
1) How can I programmatically monitor data model acceleration progress?
2) If the above REST API has been deprecated, what is the functionality replacement?
3) Has anyone successfully used the "| datamodelinfo" command for all datamodels ie. is the incomplete results issue affecting others or just me?

Cheers,

Glenn

[MuS]

Hi Glenn,

this the kind of question I like 🙂 Okay let me explain: the command datamodelinfo is not documented, because it is a custom search command provided by the App Splunk_SA_CIM so it is not a default Splunk command. But, if you look at the script of this command $SPLUNK_HOME/etc/apps/Splunk_SA_CIM/bin/datamodelinfo.py you can see what it does.

First it gets all accelerated data models form the app context or the ones globally exported and then uses the name to get further details from another REST end point. Adapted to a Splunk search you will get this nice use case for a map search (actually this is my first real useful use case for map!):

| rest /services/data/models 
| search acceleration="1" 
| table title eai:appName  
| rename eai:appName AS name 
| eval myDatamodel="DM_" . name . "_" . title 
| map search="| rest /services/admin/summarization/tstats:$myDatamodel$ | table eai:acl.app title summary.complete"

What happens here is the following:

• | rest /services/data/models | search acceleration="1" get all accelerated data models
• | table title eai:appName | rename eai:appName AS name a rename is needed because of the : in the title
• | eval myDatamodel="DM_" . name . "_" . title eval the new data model string to be used in the map search and finally
• | map search="| rest /services/admin/summarization/tstats:$myDatamodel$ | table eai:acl.app title summary.complete" run the second search to get the details.

If you need more details, simply adapt or remove the last table in the map search.
This can be run from Splunk UI, Splunk CLI or run a adapted copy of $SPLUNK_HOME/etc/apps/Splunk_SA_CIM/bin/datamodelinfo.py as cron job.

Hope this helps ...

cheers, MuS

[Glenn]

Thanks! Silly me for not realising that datamodelinfo was from an app! I can see that this should work in theory. However, in practice (on our 6.2.3 search head cluster as a user with admin role) I am still getting what I consider to be incomplete results from these rest calls, which means this solution is not working for me.

"| rest /services/data/models" is not returning all of our data models

"| rest /services/admin/summarization/tstats:DM_search_Web" (which has been constructed based on the title and name as per your logic above from one of few data models which are actually returned from the previous rest call) gets no results at all.

I'll have to conduct some more tests to see if these problems are due to issues with our cluster, but in any case this solution does not work for me at the moment 😕

[gjanders]

Did you resolve this ?
From what I can see the command only works for data stores that have global permissions on them, if they are application level or private the REST API does not expose them...

[Glenn]

Didn't resolve it no, sorry. I figured that it was probably a bug, and the client was unable to upgrade before I moved on.

[gjanders]

It is an open support ticket with splunk support now.

Interestingly, if you use the REST API it does work the way you would expect:
curl -k -u admin:changeme https://localhost:8089/servicesNS/admin/nmon/data/models

Assuming you query in the context of the app.

Does return a completely different list compared to what I see in the GUI by running:
| rest /services/data/models

Note that the above was run inside the nmon application in the GUI and I did not see the same results. I will update this if I do get a solution.

[gjanders]

|rest servicesNS/-/-/data/models

Works as expected! This is the solution provided by splunk support...
I have also sent this information into the author of the firebrigade app so it can be updated as well.

[gjanders]

Ticket logged on this one, it might be a bug in 6.4.0

[MuS]

Login to each of your SHC members and run the search from the cli to see if you get more information.