Alert when any hosts are added or removed from a sourcetype or source

[guru865]

We would like to monitor a few hosts which are logging errors as events across different sources .

Alert condition we would like to setup are :

All error logs from sources are getting monioted for all hosts.
Alert condition : When any Hosts added or removed, trigger an alert.
logs are getting monitored for all hosts.
Alert condition : When any Hosts added or removed, trigger an alert.

I have used | metadata queries nothing seems matching my requirement.

| metadata type=hosts index=* | where relative_time(now(), "-1d") > lastTime | convert ctime(lastTime) as Latest_Time | sort -lastTime | table host,Latest_Time

| inputlookup hoststomonitor.csv | append [ search index=myindexname| stats Count by host ] | table host Count | stats sum(Count) as Count by host | fillnull value=0 | eval Status=if(Count=="0","Stopped",if(Count>"0","Running","needtofill")) | sort - Status | search Status="Running" | eval Count=tostring(Count , "commas")

Appreciate your help in creating alert conditions for the above. Thanks

[gcusello · Accepted solution]

Hi guru865,
let me better understand. do you want to trigger if there is an addition host or if an host was removed from a list, or if a log is sending logs?
To know if an host a a list is sending logs or is missing it's easy:
you have to create a lookup (called e.g. perimeter.csv) with all the hosts in your perimeter to check (in the lookup there's one field called host) and run a search like this:

| metasearch index=* earliest=-300s latest=now
| eval host=upper(host)
| stats count by host
| append [ | inputlookup perimeter.csv | eval host=upper(host), count=0 | fields  host count ]
| stats sum(count) AS Total by host
| where Total=0

At the same time to know if there's a new host it's easy

| metasearch index=* earliest=-300s latest=now NOT [ | inputlookup perimeter.csv | fields  host ]
| table host

If instead you want to know if an host is added or removed from inputs.conf files, it's a different thing (and less easy!) because you have to ingest serverclass.conf file from your Deployment Server (if present) or inputs.conf from every host (if you haven't a Deployment Server) and create a lookup with the consolidated situation; after you'll be able to verify the logs you're receiving with the lookup.

Bye.
Giuseppe

[guru865]

Thanks Cusello. I have created similar search with a few additional business requirements .

[FritzWittwer_ol]

Depends on your intentions and environment. If you have deployed forwarders, and you want to be alerted if a host does not send any data, you could go for the events in the _internal index use a join with a query some (short) time ago to figure out which hosts came or have gone. In the internal index you have regular events from each host every second or so, so you can react quickly.
If you go for certain other indexes, you are bound by the maximum interval between two events per host, if your test period is shorter than the maximum time between two events, a host will appear as having gone.