Newbie to splunk, hello everyone...
I use the UniversalForwarder on a pool of windows IIS servers. Each server has 4 unique sites (IP's). I've had some problems with the universal forwarder ignoring some IIS logfiles, eventually this was fixed by adding both "crcSalt =
In order to validate that this fix is working, I want to automate a check.
The query "#Fields: | stats values(source) by host" for the last 7 days should return results like:
c:\inetpub\logs\site1\ex20120924.log
c:\inetpub\logs\site1\ex20120925.log
c:\inetpub\logs\site1\ex20120926.log
c:\inetpub\logs\site1\ex20120927.log
c:\inetpub\logs\site1\ex20120928.log
c:\inetpub\logs\site1\ex20120929.log
c:\inetpub\logs\site1\ex20120930.log
c:\inetpub\logs\site2\ex20120924.log
c:\inetpub\logs\site2\ex20120925.log
c:\inetpub\logs\site2\ex20120926.log
c:\inetpub\logs\site2\ex20120927.log
c:\inetpub\logs\site2\ex20120928.log
c:\inetpub\logs\site2\ex20120929.log
c:\inetpub\logs\site2\ex20120930.log
Above are good results.
Below are bad results:
c:\inetpub\logs\site1\ex20120924.log
c:\inetpub\logs\site1\ex20120925.log
<no entry for ex20120926.log>
c:\inetpub\logs\site1\ex20120927.log
c:\inetpub\logs\site1\ex20120928.log
c:\inetpub\logs\site1\ex20120929.log
c:\inetpub\logs\site1\ex20120930.log
c:\inetpub\logs\site2\ex20120924.log
c:\inetpub\logs\site2\ex20120925.log
c:\inetpub\logs\site2\ex20120926.log
c:\inetpub\logs\site2\ex20120927.log
c:\inetpub\logs\site2\ex20120928.log
c:\inetpub\logs\site2\ex20120929.log
<no entry for ex20120930.log>
Above has 2 entries missing.
In pseudo logic, I would run the query:
"#Fields: | stats values(source) by host" for the last 7 days
through a loop counter that returns an error if each site (site1/site2) fails to return 7 records.
If this is too complex, I suppose I could look for "at least XX rows" or such. Any suggestions would be appreciated.
Use the stats
function dc
to get a distinct count of the field values, then check if the count is 7. Then create an alert that triggers if 0 results are returned. You could also reverse that logic - check if count is NOT 7 and trigger alert if more than 0 results are returned.
#Fields: | stats dc(source) as uniquecount by host | where uniquecount!=7