Recently, my Splunk environment decided to re-index ALL of my IIS logs (which crushed my daily license quota). I have been tasked with finding the root cause of why that happened.
Is there anyway to find in the Splunk logs why it decided to re-index all these logs?
Search index=_internal host=an_iis_forwarder NOT component="Metrics"
for clues around the time of the reindex.
A place to start would be to look at timestamps on your fishbucket.. Fishbucket is responsible for keeping pointers of what's been indexed, so this would be a reasonable assumption to check
Search index=_internal host=an_iis_forwarder NOT component="Metrics"
for clues around the time of the reindex.
Looks like a new deployed was created that monitored the IIS log location and the old deployed app was removed.
Would that cause Splunk to re-index? I thought that data was separate from the app.
Usually not, but it depends on the old and new input configuration.
It would. Once the old app was removed, it will clear Splunk's monitoring list/_fishbucket which tracks the files being monitored (and till what point it has monitored the log file). When the new app was deployed, Splunk will treat that a new data monitoring and will read the file from start and can cause duplicates.