Splunk dashboard widget to display the state of service as on / off periodically

[amit_saxena]

Hi all,

Please suggest a dashboard widget through which I can display the state of service as on / off periodically.

Specifically, I have following data from CSV file indexed in Splunk indexer.

input.csv

DateTimeStamp,ServiceName,ServiceState
Mon Sep 26 10:58:00 2016,service1,Running
Mon Sep 26 10:59:26 2016,service1,Stopped
Mon Sep 26 11:00:30 2016,service1,Running

I have indexed the data through CSV sourcetype in Splunk indexer.

The requirement is to build a dashboard widget which would show the current state of the service as on / off value ( as an example, RED when state is Stopped and GREEN when state is Running ). Moreover, the historical state of the service should also be available as a chart.

Please help and suggest.

Regards,
Amit Saxena

[gcusello · Accepted solution]

I did this dashboard to verify the status of Forwarders and execution of application processes:
for Hosts:
| inputlookup perimeter.csv | count=0 | host=upper(host) | append [ search index=myindex | stats count by host ] | stats sum(count) AS Total by host | rangemap field=Total severe=0-0 low=1-1000000000 default=severe | table Status range
For Services:
| inputlookup services.csv | count=0 | append [ search index=myindex | stats count by services ] | stats sum(count) AS Total by services | rangemap field=Total severe=0-0 low=1-1000000000 default=severe | table Status range
(beware that the field that contains the services name must be the same in lookup and index, otherwise you have to rename it!)
perimeter.csv or services.csv are lookup tables that contain the hosts and services to check.
in addition you have to copy in $SPLUNK_HOME/etc/apps/yourapp/appserver/static the following files that you can find in the Dashboard Examples - Table Icon Set (Rangemap) (App that you can find in apps.splunk.com):
- table_icons_rangemap.js
- table_decorations.css
(remember that you have to restart Splunk after copy)
and modify the first row of your dashboard as this:

<form script="table_icons_rangemap.js, stylesheet="table_decorations.css">

About the graphic dashboard, see in Dashboard Examples the way to proceed.

Bye.
Giuseppe

[amit_saxena]

Hi Giuseppe,

Thanks for the reply. Give me some time, I will try and revert.

Note : I was assuming that it might be a simple search / visualization command that will do it. Unfortunately that was not the case :-).

Regards,
Amit Saxena

[amit_saxena]

Hi Giuseppe,

While working to replicate this in my environment, I just realized that I need a dashboard for each of the services which are present in the indexed data. For simplicity purposes, I have incorporated only one service in the example but I want separate indicators for each of the services and not for aggregated statistics for all of the services.

Can you please confirm this ?

It would really help if you could share sample CSV and the lookup file which I would use and modify for my scenario to test out the dashboard you have mentioned.

Regards,
Amit Saxena

[gcusello]

No, in your lookup, you have to list all the services to monitor.
you have to create a panel for each different type of monitoring (e.g.: services, hosts, errors, etc...).

About a lookup example, You need only a list of your services to monitor,
service
service1
service2
...
remember to put as first line the column name that is useful to put equal to the field name.

In addition, if you have something useful to show (as the service owner or a brief description, etc...) you can put it in your lookup and show in the panel: for example I was monitoring server status so I enriched my lookup with other information about my hosts (IP, ServerClass, etc...) and I showed them in the panel
for your case you could, for example use a lookup like this

service,owner,description
service1,Goofy,service to ........
service2,Donald, service to ...
service3,Mickey, service to ...

and modify your search
| inputlookup services.csv | count=0 | append [ search index=myindex | stats count by service ] | stats values(owner) AS "Service Owner" values(description) AS "Brief Description" sum(count) AS Total by host | rangemap field=Total severe=0-0 low=1-1000000000 default=severe | table Service "Service Owner" "Brief Description" range

If you want to rename the range field you have to modify the table_icons_rangemap.js file.

Bye.
Giuseppe

[amit_saxena]

Hi Giuseppe,

I created the lookup file as you have mentioned and used the search provided by you. However the search command is still giving an error. The error message is Unknown search command 'count' . I tried to prefix 'search" before 'count = 0' however in that case, even though the error went away, however there are no results to shown.

Please help and suggest.

Search query 1 ( giving error )

| inputlookup services.csv | count=0 | append [ search index=data3 | stats count by service ] | stats values(owner) AS "Service Owner" values(description) AS "Brief Description" sum(count) AS Total by host | rangemap field=Total severe=0-0 low=1-1000000000 default=severe | table Service "Service Owner" "Brief Description" range

Search query 2 ( no results )

| inputlookup services.csv | search count=0 | append [ search index=data3 | stats count by service ] | stats values(owner) AS "Service Owner" values(description) AS "Brief Description" sum(count) AS Total by host | rangemap field=Total severe=0-0 low=1-1000000000 default=severe | table Service "Service Owner" "Brief Description" range

Lookup contents

ServiceState,ServiceOwner,ServiceDescription
service1,owner1,description1
service2,owner2,description2

Regards,
Amit Saxena

[amit_saxena]

Hi Giuseppe,

Please help and suggest.

Regards,
Amit Saxena

[gcusello]

Sorry I forgot the eval command before count=0.
Try now
| inputlookup services.csv | eval count=0 | append [ search index=data3 | stats count by service ] | stats values(owner) AS "Service Owner" values(description) AS "Brief Description" sum(count) AS Total by host | rangemap field=Total severe=0-0 low=1-1000000000 default=severe | table Service "Service Owner" "Brief Description" range
In this way you'll have the list of your services with four columns:

• Service
• Owner
• Brief description
• range

range has two values, "severe" and "low":
- "severe" is when service isn't active,
- "low" is when service active
when you'll insert the search in a dashboard and active the graphic visualization, you'll see an icon in your table.

The strange thing is that I already answered to your comment but my answer isn't in the page!

Bye.
Giuseppe

[amit_saxena]

Hi Giuseppe,

Thanks for the reply.

I modified the search query given by you so as to include the latest status of the service and for that instead of aggregate function, I used "dedup" command. Please find below the query I used.

| inputlookup services.csv | append [ search index=data1 | dedup ServiceName | eval ServiceStateInBinary=if(ServiceState="Running", 1, (if(ServiceState="Stopped",0,-1))) | eval OverallServiceState=ServiceStateInBinary | table ServiceName, OverallServiceState ] | eval OverallServiceState=if(OverallServiceState=1,1,(if(OverallServiceState=0,0,(if(OverallServiceState=-1,-1,-999))))) | stats values(ServiceOwner) AS "Service Owner" values(ServiceDescription) AS "Brief Description" max(OverallServiceState) AS "OverallServiceState" by ServiceName | rangemap field=OverallServiceState severe=-999-0 low=1-1 default=low | table ServiceName "Service Owner" "Brief Description" "OverallServiceState" range

Through this, I am getting the "OverallServiceState" and "range" value as required.

Please let me know if this query is OK.

Also, I install Splunk dashboard examples for 6.x app and copied the files as you have mentioned. I did restarted Splunk after that. However after I re-executed the query, I didn't find the visualization I was looking for. Can you please tell me in which file, I have to update the following text ?

<form script="table_icons_rangemap.js, stylesheet="table_decorations.css">

Regards,
Amit Saxena

[gcusello]

go ahead step by step: before enlarge your search, test it as it is.
After, try to modify it: what is the result of your query, column range? it has sense or not?

About the graphic interface, see the example "Table Icon Set (Rangemap)" in Dashboard examples: after the table tag there is id="table1" that is the reference key for the javascript.

Bye.
Giuseppe

[amit_saxena]

Hi Giuseppe,

Finally I was able to get this working.

The search query I used was as follows.

| inputlookup services.csv | append [ search index=data1 | dedup ServiceName | eval ServiceStateInBinary=if(ServiceState="Running", 1, (if(ServiceState="Stopped",0,-1))) | eval OverallServiceState=ServiceStateInBinary | table ServiceName, OverallServiceState ] | eval OverallServiceState=if(OverallServiceState=1,1,(if(OverallServiceState=0,0,(if(OverallServiceState=-1,-1,-999))))) | stats values(ServiceOwner) AS "Service Owner" values(ServiceDescription) AS "Brief Description" max(OverallServiceState) AS "OverallServiceState" by ServiceName | rangemap field=OverallServiceState severe=-999-0 low=1-1 default=low | table ServiceName "Service Owner" "Brief Description" "OverallServiceState" range

Thank you so much for the help 🙂 .

I am going to accept the very first answer provided by you.

Regards,
Amit Saxena

[gcusello]

After the first time it becomes very easy: only cut & paste!

If the solution will solve your problem, please remember to accept my answer!
Bye.
Giuseppe