Getting Data In

splunk indexes rotated logs

unix
New Member

Example of actual inputs.conf

[monitor:////data/example/server/example/log/*.log]
sourcetype=jboss
index=idx_sep_dev
disabled=false
crcSalt =

first we had an inputs.conf like this:

[monitor:////data/example/server/example/log]
sourcetype=jboss
index=idx_sep_dev
disabled=false
whitelist=(.log$)
crcSalt =

With both configuration splunk indexes rotated logfiles like server.log, server.log.1 etc.
How can I disable indexing rotated logs?

0 Karma
1 Solution

Ayn
Legend

Was there a reason why you added crcSalt = <source> in the first place? It will cause the exact behaviour that you're seeing with rotated files.

From the docs:

crcSalt = <string>
* Use this setting to force Splunk to consume files that have matching CRCs (cyclic redundancy checks). (Splunk only performs CRC checks against the first few lines of a file. This behavior prevents Splunk from indexing the same file twice, even though you may have renamed it -- as, for example, with rolling log files. However, because the CRC is based on only the first few lines of the file, it is possible for legitimately different files to have matching CRCs, particularly if they have identical headers.)
* If set, <string> is added to the CRC.
* If set to the literal string <SOURCE> (including the angle brackets), the full directory path to the source file is added to the CRC. This ensures that each file being monitored has a unique CRC.   When crcSalt is invoked, it is usually set to <SOURCE>.
* Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed after it has rolled. 
* Defaults to empty. 

View solution in original post

mikelanghorst
Motivator

Set a whitelist to only monitor files ending in ,log

0 Karma

Ayn
Legend

Was there a reason why you added crcSalt = <source> in the first place? It will cause the exact behaviour that you're seeing with rotated files.

From the docs:

crcSalt = <string>
* Use this setting to force Splunk to consume files that have matching CRCs (cyclic redundancy checks). (Splunk only performs CRC checks against the first few lines of a file. This behavior prevents Splunk from indexing the same file twice, even though you may have renamed it -- as, for example, with rolling log files. However, because the CRC is based on only the first few lines of the file, it is possible for legitimately different files to have matching CRCs, particularly if they have identical headers.)
* If set, <string> is added to the CRC.
* If set to the literal string <SOURCE> (including the angle brackets), the full directory path to the source file is added to the CRC. This ensures that each file being monitored has a unique CRC.   When crcSalt is invoked, it is usually set to <SOURCE>.
* Be cautious about using this attribute with rolling log files; it could lead to the log file being re-indexed after it has rolled. 
* Defaults to empty. 

Ayn
Legend

If you know that, you know the cause of your troubles. What checksum error are you referring to?

0 Karma

unix
New Member

yes I know that, but we need the crcSalt because of the checksum-error...
is there a way to use both?

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to May Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars this month. This ...

They're back! Join the SplunkTrust and MVP at .conf24

With our highly anticipated annual conference, .conf, comes the fez-wearers you can trust! The SplunkTrust, as ...

Enterprise Security Content Update (ESCU) | New Releases

Last month, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...