Getting Data In

md5, crcSalt, gzip, oh my!

stensonb
Engager

Hello Splunkers -

I'm having trouble figuring out how to make the following work.

  1. I get usage files from a popular CDN delivered to me via FTP. These files come in gzipped...but, Splunk is nice and handles all of that wonderfully.
  2. However, on rare occasions, we may have the usage files redelivered from said CDN. When they are redelivered, the contents of the gzip are identical, but the modified time of the gzip is different (bytes 9-12)...causing Splunk to re-index (which doubles my counts for that period)...which is bad.

So, I'm trying to get around splunk using the first/last 256 bytes to determine uniqueness...I'd like to use something like:

CHECK_METHOD=none
crcSalt =

...which would use filename as the ONLY factor when determining uniqueness, but "CHECK_METHOD=none" isn't an options.

Can anybody suggest an alternative approach?

yannK
Splunk Employee
Splunk Employee

update since 5.0 you can increase the length of the crc sample :
see
http://docs.splunk.com/Documentation/Splunk/5.0/Admin/Inputsconf


initCrcLength = <integer>
* This setting adjusts how much of a file Splunk reads before trying to identify whether it is a file that has
already been seen. You may want to adjust this if you have many files with common headers (comment headers,
long CSV headers, etc) and recurring filenames.
* CAUTION: Improper use of this setting will cause data to be reindexed. You may wish to consult with Splunk
Support before adjusting this value - the default is fine for most installations.
* Defaults to 256 (bytes).
* Must be in the range 256-1048576.

stensonb
Engager

Helpful info, but won't actually do what I'm looking for.

The problem with this proposed solution is that the timestamp of the gzip header is within the first 256 bytes of the file....so, this won't work 😞

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...