Latest log not showing in Splunk

[justindett]

Hi,

I have a weird issue where when a log rolls and a new log gets created, it takes about a day or so to actually show the new log in Splunk. Looking on the server, the new log exists. But Splunk is only showing the last log before the new one was created.

Any idea why this would happen?

Thanks

[justindett · Accepted solution]

Enabling the crcSalt seemed to have solved the issue. Logs seem to be up to date for the last couple days now.

Thanks for all the suggestions

[woodcock]

Probably when the log rolls, the new log is created with the wrong ownership or permissions so that user splunk cannot read it but then there is a housekeeping ( probably cron-based) job that comes around once a day and deleted old files and fixes ownership and permissions. This should be easy to check, just keep doing this until you see it rotate and look:

ls -altr /Your/Path/To/Files/Here

[woodcock]

You probably have too many co-resident files. At hundreds of files (whether or not Splunk is supposed to forward them or not, or whether it already has or not), things slow down (like you are seeing). At thousands of files, things pretty much completely stop. A good test is that if you get a significant surge just after restarting the forwarder and then it goes back to really, really slow, then this is your problem. Do proper OS-level housekeeping to move/archive/delete older files and things will go back to snappy again.

[justindett]

@woodcock There are only 30 logs in this directory. I have enabled the crcSalt now as well. Lets see if that makes a difference.

[Sahr_Lebbie]

That was going to be my suggestion(crcSalt). How did it work out for you?

When you say renamed, were there new log file names being created or were files moving to a new directory and the same log file being appended to but just new logs?

[justindett]

I found a similar issue here : https://answers.splunk.com/answers/680732/splunk-skips-or-delays-indexing-of-the-log-file-du.html

Made the change as specified : time_before_close = 1

But doesn't look like it helped. Forwarder version is 7.0.3

Unless I need to wait until the log rolls again at midnight tonight?

[justindett]

Below is the content of the inputs.conf The whole log directory is specified, but its always just picked up the original .log file which is fine.

[monitor:///WebSphere8/applications/dev/psiberworks/logs]
disabled = false
whitelist = .log$

crcSalt = SOURCE

index = ibm_was_app_psi-was8-dev-01
time_before_close = 1

[DavidHourani]

What happens if your crcSalt is enabled ? do you still have the issue ?

[DavidHourani]

Hi @justindett,

Which files are you using for your input ? The original one or the rolled one ?