Getting Data In

Forwarder not sending data to indexer

722624
Path Finder

Please check the splunkd.log

08-30-2017 21:03:32.004 -0400 INFO TcpOutputProc - Connected to idx=10.100.xxx.1:9997, pset=0, reuse=0.
08-30-2017 21:03:32.008 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/searchhistory.log'.
08-30-2017 21:03:32.009 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_access.log'.
08-30-2017 21:03:32.011 -0400 INFO WatchedFile - Will begin reading at offset=57592 for file='/opt/splunkforwarder/var/log/splunk/audit.log'.
08-30-2017 21:03:32.013 -0400 INFO WatchedFile - Will begin reading at offset=969 for file='/opt/splunkforwarder/var/log/splunk/conf.log'.
08-30-2017 21:03:32.014 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/mongod.log'.
08-30-2017 21:03:32.016 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/license_usage.log'.
08-30-2017 21:03:32.017 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/license_usage_summary.log'.
08-30-2017 21:03:32.019 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/remote_searches.log'.
08-30-2017 21:03:32.020 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/scheduler.log'.
08-30-2017 21:03:32.022 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_ui_access.log'.
08-30-2017 21:03:32.024 -0400 INFO WatchedFile - Will begin reading at offset=369 for file='/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log'.
08-30-2017 21:03:32.025 -0400 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/splunkd_stdout.log'.
08-30-2017 21:03:32.102 -0400 INFO WatchedFile - Will begin reading at offset=20365668 for file='/opt/splunkforwarder/var/log/splunk/metrics.log'.
08-30-2017 21:14:07.561 -0400 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/scripts/rsda.txt'.
08-30-2017 21:29:06.640 -0400 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/scripts/rsda.txt'.

I need the file /opt/scripts/rsda.txt to be indexed , this is file is recreated every 15 mins....
but this is not coming to indexer
both UF and Indexer are in Linux, ping is working both ways....

I have searched , there are so many posts but none is addressing this problem..

Thank you
AB

0 Karma

woodcock
Esteemed Legend

You need to modify CHECK_METHOD in props.conf to modtime (checks only modification time of file):
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf?utm_source=answers&utm_medium=in...

0 Karma

muszyngr
Observer

ok, so what did it what fixed it? this is so frustrating finding unanswered threads, not your fault, just the Splunk Documentation is so lacking and here I am three years later with a very similar issue

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi AB,
some questions, to better understand the situation:

when the file is recreated, it's different, the same or both the possibilities?

Surely Splunk don't index it when it's the same, but only when updated.

When you update file, do you modify first chars?

Bye.
Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

ok for different content, but the first 256 chars hare different or the same?

When you say that the only two servers aren't sending logs, do you mean that the problem is only on two UF and correctly runs on the other 13?
If yes, delete the first question.

Bye.
Giuseppe

0 Karma

722624
Path Finder

Hello Giuseppe,
Thanks for quick response
Yes..first 250 chars are also different
We have same version of UF installed on each of our 15 hosts...13 hosts are sending data to indexer..but 2 hosts are not sending the data

0 Karma

gcusello
SplunkTrust
SplunkTrust

Yes, the problem that you don't index updates there is only on two Forwarders or in all Forwarders?
if the first, you have to check if the two Forwarders send other logs to Indexer ( index=_internal host=your_host1 OR host=your_host2 ).
If the second, it's a different problem.
Bye.
Giuseppe

0 Karma

722624
Path Finder

only two forwrarders are not sending... index=_internal host=your_host1 ...is not giving any data

0 Karma

gcusello
SplunkTrust
SplunkTrust

This means that the problem isn't in the ingestion of the variation of the file, te problem in in connection!

at first check if firewalls rules are open, using telnet IP_Indexer 9997

if ok, check hostname in $SPLUNK_HOME/etc/system/local/inputs.conf and $SPLUNK_HOME/etc/system/local/server.conf (beware if you have the same hostname of another forwarder sometimes it happens!)

if ok, check if outputs.conf is correctly configurated (usually is in $SPLUNK_HOME/etc/system/local/ or in a dedicated App): you must have something like this:

[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = IP_Indexer:9997
disabled=false
[tcpout-server://IP_Indexer:9997]

otherwise see at http://docs.splunk.com/Documentation/Forwarder/6.6.3/Forwarder/Troubleshoottheuniversalforwarder

Bye.
Giuseppe

0 Karma

722624
Path Finder

Hello Giuseppe....
the file is created every 15 mins with same file name,,,,but with different content ,
I have total 15 hosts, same configuration , same OS, same UF...13 hosts are sending but 2 hosts are not sending...

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...