How to show Splunk log ingestion availability by sourcetype in a dashboard?

[Kieffer87]

I'm trying to build a dashboard that shows the log availability by sourcetype focusing on the ingestion of the logs from their source. We have a mix of sourcetypes coming from splunk forwarders, syslog, and other connection types such as opsec lea. At the end of the day I want to be able to provide management with a dashbaord that shows the Firewall management station was streaming logs to Splunk 99.999% of the time or the Cisco VPN devices only forwarded logs 89% of the time. Due to the size of our environment all sourcetypes have a constant stream of logs coming 24x7x365. I want to start with sourcetype metrics and then will get more granular by host with a seperate dashboard.

I've put together the following search which gets me closer to the end goal but what I really want is a search that builds a time chart and puts a 1 if an event exists per sourcetype during a 15m span or a 0 if one does not. The timechart would cover 30 days and I would be able to average the total per sourcetype which should effectively give me a percentage of log ingestion by sourcetype that I could display on a dashboard.

index=_internal sourcetype=splunkd splunk_server="ldxx90spkinx*" source="*metrics.log" group=per_sourcetype_thruput series!="splunk*" series!="dbx*" series!="audittrail" series!="exec" series!="kvstore" series!="mongod" series!="first_install-too_small" series!="pdfgen-too_small" series!="scheduler"
| timechart limit=50 useother=f span=15m count by series

Any suggestions on how to make this happen? Is there a way to reference the value's output by timechart so I can further manipulate them with an eval statement?

[mattymo · Accepted solution]

I recommend checking out the meta woot! app. https://splunkbase.splunk.com/app/2949/

It will provide you a great look at what is in your environment, and provide the base to assure your data feeds. It is a great example of the power of tstats and summaries.

It will probably get you most of the way there on a lot of the data integrity and compliance checks, you basically only need to write a few simple alert pipes on their searches and you will have a good eye on your data trends. I like to shove the summaries thru the Machine Learning Toolkit for that analysis.

This in conjunction with the Monitoring Console views on Forwarder Management you should be pretty nicely covered.

[Kieffer87]

Meta woot! looks very interesting. I will have to check that out.