I have a Powershell script that is writing data about sessions for an application to a file every 5 minutes so the file timestamp is changing, but Splunk only pulls in changed or new sessions. I was hoping Splunk would pull in all the data every 5 minutes so that I could get a good point-in-time count of the number of sessions and what they looked like at that time. Does anyone know if I can force Splunk to ingest the entire file when it changes? Here is the code I use to write a powershell object to a Splunk friendly file in case anyone is interested. "Id" is where the line break should be.
($obj | fl | Out-String).Trim() -replace "n
rn","" -replace "
rn","," -replace " :","=" -replace " +"," " -replace "(?m)^Id","
nId" | Set-Content $file
Change the CHECK_METHOD
for the file in props.conf. Any other than the default method will cause Splunk to index the entire file if it's changed. From the docs:
CHECK_METHOD = [endpoint_md5|entire_md5|modtime]
* Set CHECK_METHOD endpoint_md5 to have Splunk checksum of the first and last 256 bytes of a
file. When it finds matches, Splunk lists the file as already indexed and indexes only new
data, or ignores it if there is no new data.
* Set CHECK_METHOD = entire_md5 to use the checksum of the entire file.
* Set CHECK_METHOD = modtime to check only the modification time of the file.
* Settings other than endpoint_md5 cause Splunk to index the entire file for each detected
change.
* Defaults to endpoint_md5.
So, you should do something like:
[source::<yoursource>]
CHECK_METHOD = modtime
Change the CHECK_METHOD
for the file in props.conf. Any other than the default method will cause Splunk to index the entire file if it's changed. From the docs:
CHECK_METHOD = [endpoint_md5|entire_md5|modtime]
* Set CHECK_METHOD endpoint_md5 to have Splunk checksum of the first and last 256 bytes of a
file. When it finds matches, Splunk lists the file as already indexed and indexes only new
data, or ignores it if there is no new data.
* Set CHECK_METHOD = entire_md5 to use the checksum of the entire file.
* Set CHECK_METHOD = modtime to check only the modification time of the file.
* Settings other than endpoint_md5 cause Splunk to index the entire file for each detected
change.
* Defaults to endpoint_md5.
So, you should do something like:
[source::<yoursource>]
CHECK_METHOD = modtime