How to create an alert when a new forwarder is added to deployment server?

[merrelr]

I'm looking for a way to report/alert anytime a new forwarder is added to my deployment server. I've tried searching on internal with the following search, but this isn't unique to when a forwarder is initially added to our environment.

index=_internal sourcetype=splunkd component=PubSubSvr host=<deploymentServer>

[jacobpevans]

Greetings @merrelr,

The current provided answers would work. This is how I would do it though:

| metadata type=hosts
| eval daysSinceFirstTime = round((now() - firstTime)/86400, 2)
| eval hoursSinceLastTime = round((now() - lastTime )/3600 , 2)
| convert ctime(firstTime) as firstTime
| convert ctime(lastTime)  as lastTime
| search daysSinceFirstTime < 1 OR (hoursSinceLastTime>24 AND hoursSinceLastTime<48)

In this example, you could run the alert search once a day to get all hosts that sent their first event to your indexer(s) in the past day and hosts that sent their last event to your indexer(s) between 24 and 48 hours ago. Run the search for all time as this is an extremely efficient command.

Refer here for the metadata command documentation.

[Tyler]

Edit: This one should be more accurate and will also let you use the date picker.

index=_internal host="SPLUNK_DS_NAME" component=PubSubSvr event_message="Subscribed*"
| eval hostname=mvindex(split(connectionId, "_"), 4)
| search hostname!="direct"
| eval guid=mvindex(split(connectionId, "_"), 5)
| dedup guid sortby +_time
| join guid type=inner [search index=_internal host="SPLUNK_DS_NAME" component=PubSubSvr event_message="Subscribed*" earliest=1
| eval hostname=mvindex(split(connectionId, "_"), 4)
| search hostname!="direct"
| eval guid=mvindex(split(connectionId, "_"), 5)
| eval h_time=_time
| dedup guid sortby +_time 
| fields guid, h_time]
| addinfo
| eval firstSeenDiff=_time-h_time
| eval pickerDiff=now()-info_min_time
| where pickerDiff>=firstSeenDiff
| eval lastSeen=strftime(_time, "%Y-%m-%d %H:%M:%S")
| eval firstSeen=strftime(h_time, "%Y-%m-%d %H:%M:%S")
| table lastSeen, firstSeen, guid, hostname

This seems to be the earliest date stamp of events, not the forwarder check-in.

index=_internal source=*metrics.log group=tcpin_connections
| dedup guid
| table _time, guid, hostname

(Simplified version of Solved: Re: Listing forwarders - Splunk Community)

I then realized this was just for forwarders sending data. Although it would be strange, you can have forwarders connected to just the DS.

index=_internal host="SPLUNK_DS_HOSTNAME" component=PubSubSvr event_message="Subscribed*"
| eval hostname=mvindex(split(connectionId, "_"), 4)
| search hostname!="direct"
| eval guid=mvindex(split(connectionId, "_"), 5)
| dedup guid sortby +_time 
| table _time, guid, hostname

[buckiboy]

| tstats count where index = _* by host | search NOT [| inputlookup forwarders.csv | fields+ host]
      | stats values(host) AS new_hosts

There should be a pipe before inputlookup.

[adonio]

Hello merrelr,
you can create a lookup table of all forwarders:
| tstats count where index = _* by host | table host | outputlookup forwarders.csv
and let it run as a schedule search. now you can search and compare results to lookup to check who is new on your forwarders list:

   | tstats count where index = _* by host | search NOT [inputlookup forwarders.csv | fields+ host]
     | stats values(host) AS new_hosts

you can also use the | metadata command and search for newly received data leveraging the firstTime field
more in this answer:
https://answers.splunk.com/answers/321024/compare-search-to-lookup-table-and-return-results.html

[jlh3dz]

I found the answer. "search" is missing. Use
| tstats count where index = _* by host | search NOT [inputlookup forwarders.csv | fields+ host]
| stats values(host) AS new_hosts

[bbraun]

This worked! Thank you!

[adonio]

@bbraun
modified the answer to reflect, indeed forgot to put the search after pie. this is the reason for your error, there is no NOT command

[bbraun]

This is very helpful!

Although when i enter the query to compare the results, I get the below error.

"Search Factory: Unknown search command 'not'."

I tried playing around with the query but did not have any luck.

[jlh3dz]

I get the same error, "Search Factory: Unknown search command 'not'."