Splunk Search

Forwarder not reporting more than 2 weeks ago.

sanju005ind
Communicator

I need help with a query to find the forwarders which stopped reporting for more than 2 weeks.

Tags (1)
0 Karma
1 Solution

thall79
Communicator

This is my search that I use track the last time a forwarder has responded.

| metadata type=hosts | eval age = now() - lastTime | sort -age | search age > 86400 | eval d=floor(age/86400) | eval h=floor((age-(d*86400))/3600) | eval m=floor((age-(d*86400)-(h*3600))/60) | eval s=floor(age-(d*86400)-(h*3600)-(m*60)) | eval age =d." days ".h." hours ".m." minutes ".s." seconds" | fields age,host | rename age as Time_of_Last_Response

travis.

View solution in original post

thall79
Communicator

This is my search that I use track the last time a forwarder has responded.

| metadata type=hosts | eval age = now() - lastTime | sort -age | search age > 86400 | eval d=floor(age/86400) | eval h=floor((age-(d*86400))/3600) | eval m=floor((age-(d*86400)-(h*3600))/60) | eval s=floor(age-(d*86400)-(h*3600)-(m*60)) | eval age =d." days ".h." hours ".m." minutes ".s." seconds" | fields age,host | rename age as Time_of_Last_Response

travis.

sanju005ind
Communicator

Thanks.This is what I was looking for.

0 Karma

rotten
Communicator

I check my forwarder status with a shell script. That way I can check the indexer status also, and if I have a system which doesn't send logs very often, I can still make sure the forwarder is alive.

All of my forwarders on on Unix systems, so my script, invoked from cron every few minutes, does something like the snippet below.

I do take a short cut because all of my forwarder's PID files are available via NFS. However, you should be able to get the idea:


for splunker in `cat my_list_of_splunk_forwarders`
do

  PIDList=`cat ${splunker}/var/run/splunk/splunkd.pid`
  PIDCount=`print ${PIDList} | wc -w`

  SplunkFwdr=`print ${splunker} | cut -d- -f1`

  SplunkdCounter=0
  for PID in `print ${PIDList}`
  do
     ProcName=`ssh ${SplunkFwdr} ps -o comm -p "${PID}" | tail +2`

     if [[ ${ProcName} == "splunkd" ]]
     then

       SplunkdCounter=$((SplunkdCounter + 1))


     fi

  done

  ## There should always be at least two processes running:
  if [[ ${SplunkdCounter} -lt 2 ]]
  then
      print "${splunker} -- splunkd count:  ${SplunkdCounter}" | mailx -s "[${splunker}][Splunk] Forwarder Co
unt Mismatch [Found: ${SplunkdCounter}][Expected:  ${PIDCount}]" ${ToList}
  fi

done
0 Karma

Chris_R_
Splunk Employee
Splunk Employee

If your on version 4 try a query such as this to see recently connected forwarders.

| metadata type=hosts | sort recentTime desc | convert ctime(recentTime) as Recent_Time
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...