I have a custom application that does incremental loads from an external resource. I maintain a file with the latest timestamps so that each run I can read the file and pull the new data. My struggle is with how do I perform application upgrades without rolling these files back to their initial value. Is there a configuration that can be set to ignore file types or directories during a custom application upgrade?
Please advise.
The "modular input way", which I also would suggest for other types of inputs, is to use the "checkpoint_dir" configuration. When A modular input is created, it is assigned a special checkpoint directory to store such files that keep checkpoints. Each file contains whatever information is needed for the input (either a single timestamp, or perhaps a json object with a more complex status). The location for modular input is /opt/splunk/var/lib/splunk/modinputs/$INPUTNAME
. This will do a few things for you:
default
and local
configs. You could remove the entire App, reinstall, and as long as the script looks in the checkpoint dir, you are fine.As always, you can contact me directly via email, IRC (#splunk on EfNet), or just shouting at the screen (ok, maybe not that one).
If you put your timestamp file in the 'local' directory, it won't be affected by upgrades. Apps should not deliver a 'local' directory.