Indexing a file name of which changes daily

[rvbalaji]

We need to index a file which has the day's date as part of its name. How to I configure Splunk to read this file? Indexing the folder is an option, but I want to eliminate this option before taking the other route.

[Lowell]

BTW, if you want to consolidate your log file names into a single source name (because it's annoying to see a different log file each and every day), then you may find some helpful resources on this page:

http://answers.splunk.com/questions/3470/consolidate-similarly-named-log-files-into-a-single-source/...

[jervin · Accepted solution]

We use blacklists/whitelists for this; for instance:

[monitor:///var/httpd]
_whitelist = (access$|access-)

That will pick up all logs under /var/httpd either ending in access or containing "access-" in the name, such as "access-20101013.log". In general I've found whitelisting to be better than blacklisting or simple wildcarding because new logs won't be picked up without administrator intervention, which helps keep license costs down if someone puts a large debug log into a directory that Splunk is indexing.

--James

[rvbalaji]

Hey Jervin,

I tried whitelisting the file we wanted to monitor and it worked like a beauty. Thanks.

[muebel]

You could monitor such a file by using a wildcard for the variable part of the file. I.E.

[monitor://<path>/daily_file*]

Presuming that the "daily_file" part is static, and what comes after that is the datestamp.

[rvbalaji]

I tried configuring as you have mentioned using the Splunk\Manager\
Data Inputs\File & Directories\Add New and it did not start indexing this file.

[dwaddle]

The easiest approach to this is to make a symlink to the daily file. Tell Splunk to index the symlink, and swap the symlink out daily.

[dwaddle]

That is a good point about the timing - you would have to wait a reasonable time interval (and reasonable depends on how busy your forwarder / indexer is) after the last write to "yesterday's" file. I think the Splunk 3.x issues with this approach have been mostly squashed - while I'm not using this approach myself I know it's been suggested before on #splunk and seemed to worked for others.

[Lowell]

Seems like there could be potential timing issues with this. For example, if you switch the symlink too soon then you miss the end of the file, and if you wait too long, then indexing gets delayed. And last time I tried indexing a symlink (managed by cronolog) back in Splunk 3.x days, this caused some indexing issues, if I'm not mistaken. Perhaps this has all be fixed now, but it never seemed worth trying again. That's my 2 cents.