How can I configure Splunk to read in a log that is identical to another log int he same directory?

[daniel333]

All,

We have a report that runs every hour. Basically 10 line CSV. Might grow to 20 lines on an off week.
myreport-%date%.csv

It's populating a larger data set. 99% of the time the report will be identical the previous run how ever. Since it's only a few kilobytes in size I would just rather Splunk ingest the report every time. But I believe there is some mechanism in Splunk the prevents it from reading in a file identical to another file in the same directory. Anyway to disable this?

I tried crcSalt= and it only worked for one run.

[jkat54]

Are you using a monitor stanza?

I just saw your comment about tried crcSalt and it only worked for one run...

crcSalt should work fine as long as the %date% changes each time. If it's not changing... it needs to.

You could do something like this to add the hour and minutes to the file name via the date variable:

# loop through time /t output
for /f "tokens=1,2,3 delims=: " %%A IN ('Time /T') do set h=%%A & set m=%%B & set t=%%C
# remove space after hour
set h=%h: =%
# create the date variable out of tokens from above.
set %date%=%h%%m%%t%

[jkat54]

in your inputs.conf, add this to the input stanza:

crcSalt=<SOURCE>

crcSalt = <string>
* Use this setting to force Splunk to consume files that have matching CRCs
  (cyclic redundancy checks).
    * (Splunk only performs CRC checks against, by default, the first 256 bytes
      a file. This behavior prevents Splunk from indexing the same file twice,
      even though you may have renamed it -- as, for example, with rolling log
      files. However, because the CRC is based on only the first few lines of
      the file, it is possible for legitimately different files to have matching
      CRCs, particularly if they have identical headers.)
* If set, <string> is added to the CRC.
* If set to the literal string <SOURCE> (including the angle brackets), the
  full directory path to the source file is added to the CRC. This ensures that
  each file being monitored has a unique CRC.   When crcSalt is invoked, it is
  usually set to <SOURCE>.
* Be cautious about using this attribute with rolling log files; it could lead
  to the log file being re-indexed after it has rolled.
* In many situations, initCrcLength can be used to achieve the same goals.
* Defaults to empty.