Forwarders can be not forwarding for many reasons other than shutdown including a crash, which would not have a shutdown event or a network problem. Try this:
| metadata index=* type=hosts | eval latencySeconds=(recentTime-lastTime) | eval quietSeconds=(now()-recentTime) | fieldformat firstTime=strftime(firstTime, "%m/%d/%Y %H:%M:%S") | fieldformat lastTime=strftime(lastTime, "%m/%d/%Y %H:%M:%S") | eval indexTime=strftime(recentTime, "%m/%d/%Y %H:%M:%S")
The field quietSeconds
tells you how long it has been since that forwarder sent any data to any indexer.
Hi athorat,
forward the _internal
logs of the forwarder to the indexer and search like this:
index=_internal component=ShutdownHandler
This will list all shutdown
events.
Basics about _internal
forwarding can be found here http://docs.splunk.com/Documentation/Splunk/6.3.0/DistSearch/Forwardsearchheaddata
Yes, you can do the same on your forwarder.
Hope this helps ...
cheers, MuS