How will followTail=1 work on a symlink that keeps changing

[beaumaris]

We are trying to monitor logs in a directory that are created by an application that does the following

1. Creates a new filename that has a date/time stamp in the file name
2. Points a symbol link (filename working.log) to the newly created file
3. At regular intervals (a period of minutes) it repeats steps 1 and 2

In essence we have two options for monitoring here: we can monitor the directory, which detects the newly created files, waits for the end of file to not change and then uploads the file contents; or we can use followTail=1 on the working.log file and not worry about all the individually-named files. For the second option, we're assuming that Splunk will be able to follow the "end" of the working.log file even as the symlink is changed from one physical file to the next. If Splunk does correctly follow the end of the working.log file, then the expectation is that this will reduce the time it takes to upload the logged data since we won't have to wait for the "end of file" to be detected.

Will Splunk work correctly on a symlink that is changes the target file every few minutes?

[Simeon · Accepted solution]

Splunk should follow the symlink once the timestamp/modtime is updated. So in your case, you can point Splunk at the file which the app is updating.

[beaumaris]

Thanks Simeon, that is exactly how we would expect the system to operate and it's great to have confirmation of that behavior.

[beaumaris]

The log files and the symlink are being managed by a 3rd party application, so it is unclear to us why they take this approach. However, given that the symlink exists, the thinking is that if Splunk does follow the symlink as it gets set to different targets, then we can use followTail=1 on the working.log file and the events will be delivered sooner, since they will be detected as soon as they are written to disk rather than waiting for Splunk to determine that the file is no longer being written to which can take many seconds.

[Simeon]

What is the purpose of the symlink? Splunk should automatically handle the new file unless all of the events are the same as the previous file.