Since this weekend I suddenly have a bunch of hosts that don't exist. A script that is meant to alert if any host has not sent logs in 24 hours started reporting hosts named "recover-padding-1" through "recover-padding-84" as not checking in. When I look at the Summary they are also listed but no event entries are listed.
How and why did 84, "recover-padding-#" hosts, get added to my host list with out any events? Why do I still get output on my script after I "deleted" them? Usually when I "delete" something it changes the date on a "host" last contact to be the date when Splunk started collecting data.
Name: 'minor_host_not_reporting'
Query Terms: '| metadata index=main type=hosts | eval age = now()-lastTime | where (age > (1*86400)) AND (age < (7*86400)) | sort age d | convert ctime(lastTime) | fields age,host,lastTime'
Link to results: arthamm_search_bWlub3JfaG9zdF9ub3RfcmVwb3J0aW5n_at_1292864400_1115244169">http://nascsplunk:8000/app/search/@go?sid=scheduler_arthamm_search_bWlub3JfaG9zdF9ub3RfcmVwb3J0aW5n_at_1292864400_1115244169
Alert was triggered because of: 'Saved Search [minor_host_not_reporting]: number of hosts(84)'
age host lastTime
270603 recover-padding-67 12/17/2010 08:49:57
270603 recover-padding-66 12/17/2010 08:49:57
270603 recover-padding-65 12/17/2010 08:49:57
270603 recover-padding-64 12/17/2010 08:49:57
270603 recover-padding-63 12/17/2010 08:49:57
... View more