Getting Data In

delta report for multiple hosts

mcafeesecure
Explorer

I have the following log messages coming from syslog-ng

Jun 14 10:32:04 sc4-cron.mcafeesecure.com syslog-ng[2775]: Log statistics; dropped='tcp(AF_INET(10.98.202.126:514))=1613614', processed='center(queued)=175194', processed='center(received)=109152', processed='destination(d_resin)=51369', processed='destination(d_boot)=0', processed='destination(d_auth)=439', processed='destination(d_cron)=362', processed='destination(d_mlal)=0', processed='destination(loghost)=109152', processed='destination(d_mesg)=93', processed='destination(d_cons)=0', processed='destination(d_snmp)=6782', processed='destination(d_spol)=0', processed='destination(d_mail)=6997', processed='source(s_sys)=66165', processed='source(s_tail)=42187', processed='source(s_raccess)=800', processed='source(s_tail_2)=0'

Jun 14 10:22:04 sc4-cron.mcafeesecure.com syslog-ng[2775]: Log statistics; dropped='tcp(AF_INET(10.98.202.126:514))=1613614', processed='center(queued)=173017', processed='center(received)=107903', processed='destination(d_resin)=50855', processed='destination(d_boot)=0', processed='destination(d_auth)=430', processed='destination(d_cron)=359', processed='destination(d_mlal)=0', processed='destination(loghost)=107903', processed='destination(d_mesg)=92', processed='destination(d_cons)=0', processed='destination(d_snmp)=6610', processed='destination(d_spol)=0', processed='destination(d_mail)=6768', processed='source(s_sys)=65231', processed='source(s_tail)=41872', processed='source(s_raccess)=800', processed='source(s_tail_2)=0'

Jun 14 10:12:04 sc4-cron.mcafeesecure.com syslog-ng[2775]: Log statistics; dropped='tcp(AF_INET(10.98.202.126:514))=1613614', processed='center(queued)=171101', processed='center(received)=106828', processed='destination(d_resin)=50405', processed='destination(d_boot)=0', processed='destination(d_auth)=416', processed='destination(d_cron)=338', processed='destination(d_mlal)=0', processed='destination(loghost)=106828', processed='destination(d_mesg)=87', processed='destination(d_cons)=0', processed='destination(d_snmp)=6428', processed='destination(d_spol)=0', processed='destination(d_mail)=6599', processed='source(s_sys)=64382', processed='source(s_tail)=41646', processed='source(s_raccess)=800', processed='source(s_tail_2)=0'

I am using the following query to generate a diff of the field in this message I want (in this case the dropped field)

syslog-ng "Log statistics" host="sc4-cron*" | rex "(?i)^(?:[^=]=){2}(?P[^'])(?=')" | search DROPPED="" | delta DROPPED as DROPPEDDIFF | search DROPPEDDIFF=""

this works fine on a single host, however, I would like to run this across all of my hosts. The problem I then encounter, is that the previous log message is not from the same host as the latest message, therefore the diffs that are created are not accurate.

I have hundreds of hosts, and would like to build a report on this data, however, I can't really go and do them all individually. In addition getting all of this into one report would be ideal.

Tags (2)
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

You can use the streamstats command, something like:

... | streamstats current=f global=f window=1 first(DROPPED) as NEXT_DROPPED by host | eval DROPPEDDIFF = NEXT_DROPPED-DROPPED

Something like that. You may have to reverse the subtraction, I'm not sure which way around it goes, and well, it might be off-by-one, i.e. the diff might be on the previous vs the current result, but you get the idea.

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

You can use the streamstats command, something like:

... | streamstats current=f global=f window=1 first(DROPPED) as NEXT_DROPPED by host | eval DROPPEDDIFF = NEXT_DROPPED-DROPPED

Something like that. You may have to reverse the subtraction, I'm not sure which way around it goes, and well, it might be off-by-one, i.e. the diff might be on the previous vs the current result, but you get the idea.

mcafeesecure
Explorer

Thanks.
For the record, the final query looks like this:

syslog-ng "Log statistics" host!="qa2-cron*" | rex "(?i)^(?:[^=]=){2}(?P[^'])(?=')" | search DROPPED="*" | streamstats current=f global=f window=1 first(DROPPED) as NEXT_DROPPED by host | eval DROPPEDDIFF = NEXT_DROPPED-DROPPED | search DROPPEDDIFF>0

morian
Engager

Thank you, you helped me solve my problem.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...