Deployment Architecture

Finding forwarders that are not reporting in but have in the past

Robbie1194
Communicator

Hi guys,

I've been been given 2 tasks with regards to our Splunk forwarders.

1) Find out which forwarders are not checking in/do not have a heartbeat but have in the past.
2) Find out which forwarders do have a live heartbeat but have not sent any logs in over a specific period of time (probably going to make it 4 hours)

Could anyone give me advise on how I would go about finding this information? I have looked into creating a daily alert or report to inform me but I don't have a clue where I should be searching.

Any help at all would be appreciated as I don't even know where to start with this.

Thanks!

0 Karma

jbjerke_splunk
Splunk Employee
Splunk Employee

Hi Robbie1194

You can also run a search like this.

This would display a list of forwarders that haven’t reported in for a 2 hour window at any point. You can adjust accordingly to suit your needs.

index=_internal source=*metrics.log group=tcpin_connections 
| eval sourceHost=if(isnull(hostname), sourceHost,hostname) 
| table sourceHost _time 
| sort sourceHost, -_time 
| reverse 
| streamstats window=1 current=f global=false last(_time) as previous_time by sourceHost 
| eval d=_time-previous_time 
| fields - previous_time 
| search d>7200
0 Karma

ddrillic
Ultra Champion

If you can identify all the hosts which you need the forwarders on, then you are in a good shape. It should be the list from the serverclass.conf file.

You can then create a lookup with these hosts and do a left join on them using the metadata command.
This approach has been working for us very well.

We do something like -

| inputlookup <lookup>
| fields host, <rest of fields>
| join type=left host 
[ | metadata type=hosts index=<the corresponding indexes>
  | eval host=lower(host) ] 
  | eval RECENT=strftime(recentTime,"%a %m/%d/%Y-%T %Z(%z)") 
  | eval LAST=strftime(lastTime,"%a %m/%d/%Y-%T %Z(%z)") | sort host 
  | where <anything...>
  | table host, LAST
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...