Getting Data In

Why does forwarding stop until i restart splunk

capilarity
Path Finder

I have a 4.3.3 UF on a windows 2008r2 box that was forwarding windows event logs quite happily.
It's now stopped forwarding but, if I restart splunk on the forwarding server, the missing events are forwarded, but no new events until I restart splunk again.
Short of restarting splunk every 5 minutes, can any one suggest why this might be happening?

Tags (2)
0 Karma
1 Solution

capilarity
Path Finder

Accepted the answer by mistake - any way you can unaccept?

The issue went away when we upgraded, we are now splunk 6 and not an issue so far......

View solution in original post

0 Karma

jareddjenkins
New Member

I had the same symptoms, it was a configuration issue.
Make sure you fully understand ignoreOlderThan=
https://docs.splunk.com/Documentation/Splunk/6.6.1/Admin/Inputsconf

In my case logs were not written to for 7 + days and then splunk will no longer try to read from that file even when new events appeard in the file..

0 Karma

jareddjenkins
New Member

In my case, it was a misunderstanding of ignoreOlderThan= in inputs.conf.

ignoreOlderThan will completely ignore files that ever reach this threshold.
From the inputs.conf documentation.
"Do NOT select a time that files you want to read could reach in age, even temporarily"

My files wouldn't write to the logs for several weeks and then begin writing again. Splunk would not even try to ingest them.

0 Karma

capilarity
Path Finder

Accepted the answer by mistake - any way you can unaccept?

The issue went away when we upgraded, we are now splunk 6 and not an issue so far......

0 Karma

SarahBOA
Path Finder

We are experiencing a few issues with our windows forwarders and one of them sounds like it might be the same. We have this issue where we get splunk internal logs constantly, but monitored files are only sent on shutdown of the universal forwarder. This problem appears to be that the splunk forwarder was trying to "restart" too quickly. When we tried to restart the forwarder we received an error message that the process was taking too long - but it appeared stopped in the windows serverice listing so we started it up again. No error message was received on startup. We then received the internal logs as expected, but didn't receive the application log file we were monitoring. We then stopped the forwarder (received the error message again) and waited about 5 minutes. After 5 minutes we started the forwarder and both the splunk internal logs and the monitored log files were continuously coming through. It appears that the stop, pause for a longer time, then start appeared to fix this issue.

This did not fix the issue where the splunkd logs only came through when the agent was stopped and the monitored logs never came through.

0 Karma

lguinn2
Legend

On the forwarder, have you looked at the splunkd log? You will find it in the subdirectory $SPLUNK_HOME\var\log\splunk

Let us know what you find there...

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...