Splunk Search

Splunk skips or delays indexing of the log file during the rotation occassionaly

ankithnageshshe
Path Finder

Hello Splunkers,

I have an issue where Splunk some times skips to index the log file during the rotation or delays the indexing during the log rotation.

This issue is only for specific file.So we can rule out the blocked queue, timezone, network throughput or slow performing indexer/forwarder.
Sar report showed good iostat cpu and mem stats on the forwarder.

I don't see initcrclength(crcSalT) or file_descriptor related issue in the splunk log.
In fact there are no error in the splunk log during this issue.

Any guidance is highly appreciated.

Best Regards,
Ankith

1 Solution

jcrabb_splunk
Splunk Employee
Splunk Employee

There is a bug for this in various versions. When the file rotates, Splunk stops reading the file until the file rotates again. At that point it ingests both files (catches up). The work around is to configure time_before_close = 1 under the relevant input.

[monitor://<path>]
time_before_close = 1

If you are on 6.6.x, this is fixed in 6.6.4:

http://docs.splunk.com/Documentation/Splunk/6.6.4/ReleaseNotes/6.6.4#Data_input_issues

  • SPL-142334, SPL-143553, SPL-145370, SPL-145978 logs are delayed in reading after rotation

This particular version of the bug is also fixed in:

  • 7.1.0 (SPL-143553)
  • 7.0.1 (SPL-145978)

I have seen this in 6.4.x as well and the provided work around (listed above) resolved the issue.

Jacob
Sr. Technical Support Engineer

View solution in original post

jcrabb_splunk
Splunk Employee
Splunk Employee

There is a bug for this in various versions. When the file rotates, Splunk stops reading the file until the file rotates again. At that point it ingests both files (catches up). The work around is to configure time_before_close = 1 under the relevant input.

[monitor://<path>]
time_before_close = 1

If you are on 6.6.x, this is fixed in 6.6.4:

http://docs.splunk.com/Documentation/Splunk/6.6.4/ReleaseNotes/6.6.4#Data_input_issues

  • SPL-142334, SPL-143553, SPL-145370, SPL-145978 logs are delayed in reading after rotation

This particular version of the bug is also fixed in:

  • 7.1.0 (SPL-143553)
  • 7.0.1 (SPL-145978)

I have seen this in 6.4.x as well and the provided work around (listed above) resolved the issue.

Jacob
Sr. Technical Support Engineer

knielsen
Contributor

Is it possible that bugfix never made it to the 7.2 tree?

We experience this issue on forwarder 7.2.6, and setting time_before_close to 1 seems to help so far.

0 Karma

vinaykata
Path Finder

Hi, I am having the same issue with IIS logs and right now we are in 6.6.3. We are not able to upgrade our versions sooner because of some other issues we are having. So my question is, is this issue resolved for 6.6.3 or is it just above 6.6.4. we have tried adding "time_before_close = 10" in our inputs.conf, but we did not see any improvements or changes. Thanks everyone!!!

0 Karma

gjanders
SplunkTrust
SplunkTrust

Fixed in 6.6.4 and the other versions mentioned by jcrabb, you will need to upgrade to a newer version to avoid the issue

0 Karma

ankithnageshshe
Path Finder

Thanks jcrabb for the update.

0 Karma

sudosplunk
Motivator

Hi @ankithnageshshetty,

When you said log rotation, does the name of the file change? If so, then splunk doesn't monitor log rotations by default. However, you can adjust your inputs(.conf) to achieve this.

0 Karma

ankithnageshshe
Path Finder

Yes..the name of the file changes..
usually splunk continues to read the new file after the rotation. But some times splunk either skips the new file or delayes in indexing.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

How frequently the file roles? How much data does the file contains?

0 Karma

ankithnageshshe
Path Finder

Hello Somesoni,

File rotates every 30 minutes and size is 96MB.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Depending upon the spike in data logging during those 30mins, there can be slowness in logs being ingested. Could you please provide monitoring stanza (inputs.conf) ? Also, how many rolled files do you keep and how are they renamed? (e.g. if myapp.log is name of monitored file, does it roll to myapp.log.1 first, then myapp.log.1 is renamed myapp.log.2 and next myapp.log is rolled as myapp.log.1 etc.)

0 Karma

ankithnageshshe
Path Finder

Hi somesoni,

Thanks for the prompt replies.

monitoring stanza:
[monitor:///app/logs/.../access]
sourcetype=ldap_access
index=XXX
ignoreOlderThan=14d

rotated file name: access.20180815152048Z . Basically it appends date and 6 digit number and "Z" to the file name.

1435 rotated files are present on the FS and retention is 56 days.

There is no information about file descriptor issue in the logs.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

So, in your monitoring stanza, the last access (.../access) is the file name Or directory? Ideally your monitoring stanza should be able to include rolled logs (you'd be monitoring both regularly written file and the rolled log file) as you're not using crcSalt=<SOURCE> (don't monitor rolled logs and use above crcSalt setting as it'll cause whole rolled log to be ingested again). With that, since Splunk is monitoring both regular log (will get each entry as they're new) and rolled logs (will not get everything else Splunk would recognize that it has already read those content, but will ingest anything that wasn't read).

0 Karma

ankithnageshshe
Path Finder

Hi somesoni,

access is file name and not the directory.

The actual file name is just "access" as mentioned in the monitoring stanza.

After the rotation new "access" file is created and old file is renamed to access.20180815152048Z

So I believe Splunk is only monitoring access here.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Ok.. So try this for your monitoring stanza. You should see improvement. (restart after making change if making change directly)

[monitor:///app/logs/.../access*]
sourcetype=ldap_access
index=XXX
ignoreOlderThan=14d
0 Karma

ankithnageshshe
Path Finder

Will this not read the rotated file? creating duplicate indexing?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...