Getting Data In

Heavy Forwarder Slow to Start Forwarding Syslog

DBattisto
Communicator

Hello-

My current setup:
Device Syslog --> Syslog Server w/ Splunk HvyFwd --> Splunk Indexer

When I restart my Heavy Forwarder server or Splunkd, it takes up to 30 minutes to begin forwarding syslogs to the indexer. Is this due to the number of devices and folders stored within the syslog server, and is there a way to speed this process up?

Thanks,

0 Karma

broberg
Communicator

When you stop splunk it still keep getting the syslog meaning when you start again it is falling behind and can't catch up in time (for some reason)
I would look on the queues on on the HF and on the indexing tier to see where the bottleneck is.
I would also look in the internal log and see if there is any errors on the monitoring path (to big files or such error)

0 Karma

manjunathmeti
SplunkTrust
SplunkTrust

Check number of directories and files monitored by the forwarder.
$SPLUNK_HOME/bin/splunk list monitor

0 Karma

mydog8it
Builder

Can you share details of how you are determining the delay is on the HF? Why are you using an HF on the syslog server?

0 Karma

DBattisto
Communicator

Thanks for the response!

I run heavy forwarder on top of my syslog server and it is actively monitoring the device directories. This was recommended to me as best practice a few years back. It works wonderfully, it just takes a while to start again after restarting the heavy forwarder.

I've monitored the syslog file of a device using tail -f and see that it's being updated instantly. If I look in splunkd.log after a restart, it takes 15-20 minutes to go through and say that it's going to begin monitoring all of the directories specified. My file hierarchy is as follows:

DeviceType/DeviceName/date.log
example: CiscoSwitch/Switch69/20200212.log

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...