Getting Data In

Why did my heavy forwarder stopped forwarding data abruptly?

neelshah
Path Finder

The HF in our environment stops forwarding data abruptly, while the Splunk process is still running. Initial investigation shows no issues with HF. On Restarting the HF it enables the data forwarding again before abruptly stops again with no clues in splunkd.log. There is no specific time no network issues.

Anyone with similar issues? any solutions?

0 Karma

FrankVl
Ultra Champion

How exactly do you come to the conclusion that your HF has stopped forwarding data?

0 Karma

neelshah
Path Finder

Like anyone else, data from the particular host is more than 24hrs late. After restart, all that missing data appears and updated to latest time.

0 Karma

ranjitbrhm1
Communicator

Im pretty sure this is not exactly the solution but try turning on the "Store a local copy" for lets say 5 minutes to verify that data is actually injecting into your heavy forwarder. then also login to the splunk console and then try telnetting into the receiving indexer's receiver port to make sure its not a network issue. Normally if there is some errors, there used to be a popup complaining that the receiver downstream is not receiving data and the data flow is stopped or something down the line like that. If both of the things are working fine then it might be something else.

0 Karma

neelshah
Path Finder

The Web page for forwarder is disabled. hence no means to get that popup. I'll try your suggestion of store local copy

0 Karma

teunlaan
Contributor

Are you sure it stopped forwarding data?
If it is processing large files, it can could be running in "batchmode". What means it will try to send out the complete file before switching to the next.

And how many parallelIngestionPiplines are you runnig? It the system in in batchmode, nothing else will come though. When adding an extra pipleline, the "extra pipeline" can process the other data (will need more CPU prower)

0 Karma

neelshah
Path Finder

parallelIngestionPiplines =3

I understand batch mode, but then a couple of times I have waited for 1-2 days for data to arrive and data only starts arriving after restart

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...