Getting Data In

Data is not indexed from a critical log file.

sylim_splunk
Splunk Employee
Splunk Employee

Data is not indexed from critical log file.
File /var/abcACSLog.txt rotates by its volume, like every 100MB and immediately moved to another directory. This has critical info that should not be missing but it happens. Please help.
File rotates like /var/abcACSLog.txt to /backup/abcACSLog_20180509.txt

Tags (2)
0 Karma
1 Solution

sylim_splunk
Splunk Employee
Splunk Employee

Here's a possible situation. Your indexers appear to be under stress and not performing fast enough for the inputs. Then due to the busy indexers, your forwarders also unable to send data and put logs like " TailReader - Could not send data to output queue " and at the same time your application rolls log data over to another directory where UF is not monitoring files.
Here's the suggestion - please share it with your application team.

i) Your application team has to change the rotation mechanism to like below;

> How to rotate:
Rotate files as before but keep the rotated file in the same directory as "/logs" so that splunk forwarder keeps reading the rest regardless of the name change.
> When to move rotated files to the other directory.
Use below command and retrieve "sptr=", this value has to be the same as the file size of "ls -l" or stat . Then you can move it to the other safely.

"command sample" > ./splunk cmd btprobe -d ../var/lib/splunk/fishbucket/splunk_private_db --file /logs/abcACSLog_20180517.txt
"command resutl" > key=0x340aa4a716415f07 scrc=0xac3413d846a972c9 sptr=14822 fcrc=0xc7cee9b9b8bf07ca flen=0 mdtm=1429060083 wrtm=1429060148

Or there are some other ways like, rotate file first and then move the previously rotated file later. But this is not safe way to move.

ii) Your forwarder input has to be changed to keep reading the rotated file.
---- input config
[monitor://logs/abcACSLog.txt] ===> [monitor://logs/abcACSLog*.txt] to cover the rotated files.

Or just add one more monitor to cover the backup directory.

View solution in original post

0 Karma

sylim_splunk
Splunk Employee
Splunk Employee

Here's a possible situation. Your indexers appear to be under stress and not performing fast enough for the inputs. Then due to the busy indexers, your forwarders also unable to send data and put logs like " TailReader - Could not send data to output queue " and at the same time your application rolls log data over to another directory where UF is not monitoring files.
Here's the suggestion - please share it with your application team.

i) Your application team has to change the rotation mechanism to like below;

> How to rotate:
Rotate files as before but keep the rotated file in the same directory as "/logs" so that splunk forwarder keeps reading the rest regardless of the name change.
> When to move rotated files to the other directory.
Use below command and retrieve "sptr=", this value has to be the same as the file size of "ls -l" or stat . Then you can move it to the other safely.

"command sample" > ./splunk cmd btprobe -d ../var/lib/splunk/fishbucket/splunk_private_db --file /logs/abcACSLog_20180517.txt
"command resutl" > key=0x340aa4a716415f07 scrc=0xac3413d846a972c9 sptr=14822 fcrc=0xc7cee9b9b8bf07ca flen=0 mdtm=1429060083 wrtm=1429060148

Or there are some other ways like, rotate file first and then move the previously rotated file later. But this is not safe way to move.

ii) Your forwarder input has to be changed to keep reading the rotated file.
---- input config
[monitor://logs/abcACSLog.txt] ===> [monitor://logs/abcACSLog*.txt] to cover the rotated files.

Or just add one more monitor to cover the backup directory.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

So you're missing some entries when the log file is rolling over?

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...