Getting Data In

Why did a Splunk forwarder stop sending to log packet?

lifekis
Explorer

It was working fine until 1 month ago.
There was no Splunk forwarder and network configuration change.
No packets from the forwarder to the indexer.
How can I solve the problem?

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @lifekis,
what's tha last date you received logs?
if you have logs until 31/12/2019 and not more after, probably the problem is the timestamp parsing: if you have a date in format dd/mm/yyyy, Splunk usually read as format mm/dd/yyyy, so it correctly read them after the 13th of each month, but take the wrong format until the 12th of each month.
You can check this problem verifying if you have logs of 7th of January on the 1st of july.

To solve the problem, you have to set the correct TIME_FORMAT in props.conf.

Ciao.
Giuseppe

View solution in original post

0 Karma

gnangia
Explorer

Here are the steps I would take to troubleshoot -
(1) Login to the server to make sure splunk process is running
(2) Check that your outputs.conf is still on the system
(3) Try to do a network connectivity test to your HF/Indexer
(4) Check splunkd.log to see if there are any errors

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @lifekis,
what's tha last date you received logs?
if you have logs until 31/12/2019 and not more after, probably the problem is the timestamp parsing: if you have a date in format dd/mm/yyyy, Splunk usually read as format mm/dd/yyyy, so it correctly read them after the 13th of each month, but take the wrong format until the 12th of each month.
You can check this problem verifying if you have logs of 7th of January on the 1st of july.

To solve the problem, you have to set the correct TIME_FORMAT in props.conf.

Ciao.
Giuseppe

0 Karma

lifekis
Explorer

I haven't received it since December 13, 2019 (GMT + 9). There is a log in the forwarder but I don't know why not forwarding it.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @lifekis,
ok, let start dubugging:
at first check if you have internal logs from that server (index=_internal host=your_host).
If you haven't them, the problem is on the connection, so test the route (from the UF telnet ip_splunk_indexer 9997).
if Nok verify firewall route between servers, if Ok check in outputs.conf on UF if the address of the Indexers is correct.

If instead you have Splunk internal logs but not the logs you want, check in inputs.conf if the path and the filename of the monitor stanza is still correct.

Ciao.
Giuseppe

0 Karma

lifekis
Explorer

Thank you for answer! I checked the disconnection log in splunkd.log.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...