Dashboards & Visualizations

How to show Splunk log ingestion availability by sourcetype in a dashboard?

Kieffer87
Communicator

I'm trying to build a dashboard that shows the log availability by sourcetype focusing on the ingestion of the logs from their source. We have a mix of sourcetypes coming from splunk forwarders, syslog, and other connection types such as opsec lea. At the end of the day I want to be able to provide management with a dashbaord that shows the Firewall management station was streaming logs to Splunk 99.999% of the time or the Cisco VPN devices only forwarded logs 89% of the time. Due to the size of our environment all sourcetypes have a constant stream of logs coming 24x7x365. I want to start with sourcetype metrics and then will get more granular by host with a seperate dashboard.

I've put together the following search which gets me closer to the end goal but what I really want is a search that builds a time chart and puts a 1 if an event exists per sourcetype during a 15m span or a 0 if one does not. The timechart would cover 30 days and I would be able to average the total per sourcetype which should effectively give me a percentage of log ingestion by sourcetype that I could display on a dashboard.

index=_internal sourcetype=splunkd splunk_server="ldxx90spkinx*" source="*metrics.log" group=per_sourcetype_thruput series!="splunk*" series!="dbx*" series!="audittrail" series!="exec" series!="kvstore" series!="mongod" series!="first_install-too_small" series!="pdfgen-too_small" series!="scheduler"
| timechart limit=50 useother=f span=15m count by series

Any suggestions on how to make this happen? Is there a way to reference the value's output by timechart so I can further manipulate them with an eval statement?

0 Karma
1 Solution

mattymo
Splunk Employee
Splunk Employee

I recommend checking out the meta woot! app. https://splunkbase.splunk.com/app/2949/

It will provide you a great look at what is in your environment, and provide the base to assure your data feeds. It is a great example of the power of tstats and summaries.

It will probably get you most of the way there on a lot of the data integrity and compliance checks, you basically only need to write a few simple alert pipes on their searches and you will have a good eye on your data trends. I like to shove the summaries thru the Machine Learning Toolkit for that analysis.

This in conjunction with the Monitoring Console views on Forwarder Management you should be pretty nicely covered.

- MattyMo

View solution in original post

mattymo
Splunk Employee
Splunk Employee

I recommend checking out the meta woot! app. https://splunkbase.splunk.com/app/2949/

It will provide you a great look at what is in your environment, and provide the base to assure your data feeds. It is a great example of the power of tstats and summaries.

It will probably get you most of the way there on a lot of the data integrity and compliance checks, you basically only need to write a few simple alert pipes on their searches and you will have a good eye on your data trends. I like to shove the summaries thru the Machine Learning Toolkit for that analysis.

This in conjunction with the Monitoring Console views on Forwarder Management you should be pretty nicely covered.

- MattyMo

Kieffer87
Communicator

Meta woot! looks very interesting. I will have to check that out.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...