Getting Data In

Splunk stops recording events on hosts

gmachacek
Engager

Greetings,
Ever since upgrading to Splunk5 I've had an issue where after a random amount of time Splunk will quit logging events from some of my hosts. I have 15+ servers sending normal syslogs and running a custom app that sends it's events on 515. When this issue happens on these couple hosts, Splunk continues to receive syslog messages, but quits seeing any events from our app.

I've used tcpdump on hosts and the Splunk server; events are still being received by the server.

The port is still up and listening, because I'm still getting the app data from the other hosts.

I've looked in splunkd.log, don't see anything happening around the time I quit getting events.

It's like Splunk has decided to just ignore these particular hosts, but only on the 515 port.

Restarting splunk fixes it and everything works, for about 10-12 hours..

I did have some SEDCMD commands, but I have since commented those out and I am still see the issue.

I'm not sure where else to look....

Tags (2)
0 Karma

sdaniels
Splunk Employee
Splunk Employee

sdaniels
Splunk Employee
Splunk Employee

I would recommend contacting support, perhaps the fix does not quite cover all possible cases. Also it is a best practise to send your data into an rsyslog/syslog-ng first and then have splunk monitor those files.

0 Karma

gmachacek
Engager

Greetings,
Well it increased the time before it quit recording entries for those hosts. It made it 48 hours versus 8-10 hours.

Running splunk-5.0.1-143156

0 Karma

gmachacek
Engager

I have update Splunk. I'll see tonight if it has (hopefully) fixed the issue. 😉

Thanks!

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...