Why can't Splunk index some files in a directory?

[edrivera3]

Hi

I am trying to index a file from different subdirectory but Splunk is not indexing some of those files for some weird reason. All subdirectories contain different files but I am just interested in a file with an specific extension (.tir) so I am using a whilelist (.tir$). Splunk indexed almost all files but there are some files that Splunk just didn't index. Do you know a reason why is this happening?

I examined the files and they are normal (Same type data, same type format, and same extension).

[edrivera3 · Accepted solution]

I found a solution to my own problem. I need to add:
crcSalt =
initCrcLength = 2000

[nawazns5038]

What exactly does the setting do ?

[bandit]

Essentially how many bytes in Splunk will check at the beginning of a file to try to uniquely identify it.

From: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf
initCrcLength =
* How much of a file, in bytes, that the input reads before trying to
identify whether it is a file that has already been seen. You might want to
adjust this if you have many files with common headers (comment headers,
long CSV headers, etc) and recurring filenames.
* Cannot be less than 256 or more than 1048576.
* CAUTION: Improper use of this setting will cause data to be re-indexed. You
might want to consult with Splunk Support before adjusting this value - the
default is fine for most installations.
* Default: 256 (bytes).

[harsmarvania57]

Please use below code in monitor stanza

whiltelist = (*.tir)$
recursive = true

[edrivera3]

I am already indexing recursively but Splunk is not indexing for some subdirectories.

[harsmarvania57]

Can you please check on forwarder with below command whether all files with (.tlr) is showing or not in command output?

 $SPLUNK_HOME/bin/splunk list monitor

[edrivera3]

I'm working with Windows OS.

[edrivera3]

I was examining the files and I noticed that there are very similar, almost the same. Is it possible that Splunk believed I am duplicating a file? If so, how could I change it so Splunk would index the file?

[edrivera3]

I didn't solve the problem. I checked the wrong index...

Anyway I tried to use CHECK_METHOD AND crcSalt but they didn't change anything.

[edrivera3]

But I found many errors in index=_internal which is weird because I tried using crcSalt = and the problem wasn't resolve. Looks like I have added these files previously which is wrong because I just created the index.

ERROR TailingProcessor - File will not be read, seekptr checksum did not match (file=C:\blah\blah\30343.tir). Last time we saw this initcrc, filename was different. You may wish to use a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

[edrivera3]

Well in Splunk>Data Input> Files & Directory appears that there 4835 files which contain the .tir extension and they are supposed to be indexed.

[harsmarvania57]

Then use below command

$SPLUNK_HOME/bin/splunk.exe list monitor

For example:

C:\Program Files\SplunkUniversalForwarder\bin>splunk.exe list monitor

[edrivera3]

I checked the current size of the index (36MB) and the event count (1,854) which look normal.