Getting Data In

automating the missing forwarder records query

umiotoko
New Member

Newbie to splunk, hello everyone...

I use the UniversalForwarder on a pool of windows IIS servers. Each server has 4 unique sites (IP's). I've had some problems with the universal forwarder ignoring some IIS logfiles, eventually this was fixed by adding both "crcSalt = " and "alwaysOpenFile = 1" to the inputs.conf on the IIS servers.

In order to validate that this fix is working, I want to automate a check.

The query "#Fields: | stats values(source) by host" for the last 7 days should return results like:

c:\inetpub\logs\site1\ex20120924.log
c:\inetpub\logs\site1\ex20120925.log
c:\inetpub\logs\site1\ex20120926.log
c:\inetpub\logs\site1\ex20120927.log
c:\inetpub\logs\site1\ex20120928.log
c:\inetpub\logs\site1\ex20120929.log
c:\inetpub\logs\site1\ex20120930.log
c:\inetpub\logs\site2\ex20120924.log
c:\inetpub\logs\site2\ex20120925.log
c:\inetpub\logs\site2\ex20120926.log
c:\inetpub\logs\site2\ex20120927.log
c:\inetpub\logs\site2\ex20120928.log
c:\inetpub\logs\site2\ex20120929.log
c:\inetpub\logs\site2\ex20120930.log

Above are good results.

Below are bad results:

c:\inetpub\logs\site1\ex20120924.log
c:\inetpub\logs\site1\ex20120925.log
<no entry for ex20120926.log>
c:\inetpub\logs\site1\ex20120927.log
c:\inetpub\logs\site1\ex20120928.log
c:\inetpub\logs\site1\ex20120929.log
c:\inetpub\logs\site1\ex20120930.log
c:\inetpub\logs\site2\ex20120924.log
c:\inetpub\logs\site2\ex20120925.log
c:\inetpub\logs\site2\ex20120926.log
c:\inetpub\logs\site2\ex20120927.log
c:\inetpub\logs\site2\ex20120928.log
c:\inetpub\logs\site2\ex20120929.log
<no entry for ex20120930.log>

Above has 2 entries missing.

In pseudo logic, I would run the query:
"#Fields: | stats values(source) by host" for the last 7 days

through a loop counter that returns an error if each site (site1/site2) fails to return 7 records.
If this is too complex, I suppose I could look for "at least XX rows" or such. Any suggestions would be appreciated.

Tags (2)
0 Karma

Ayn
Legend

Use the stats function dc to get a distinct count of the field values, then check if the count is 7. Then create an alert that triggers if 0 results are returned. You could also reverse that logic - check if count is NOT 7 and trigger alert if more than 0 results are returned.

#Fields: | stats dc(source) as uniquecount by host | where uniquecount!=7
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...