Splunk Enterprise

Forcing LWF to resend (and Indexer to re-index) segment of corrupted data

TR_Splunker
Engager

We recently rebuilt several endpoints and cloned the configs on them. Unfortunately, the input.conf file had the same [default] host= for all 18 servers because all the files were cloned from one server. While the data is present, it is all being lumped under one hostname.

We've fixed the input.conf file and now all the data is being handled correctly, but we want to re-import about 2 weeks worth that was pulled in with the wrong hostname.

Is there a way to delete a range of data that is corrupted on the index servers, and force the forwarder to re-send it?

Tags (1)

jrodman
Splunk Employee
Splunk Employee

It's possible to hide some data so that has been already indexed with the delete command, this makes it no longer searchable.

http://www.splunk.com/base/Documentation/4.1.5/SearchReference/Delete http://www.splunk.com/base/Documentation/4.1.5/Admin/RemovedatafromSplunk

It isn't reversible (and off by default), so measure twice, cut once.

You can force reindexing by a couple of different methods. You could reindex everything using a splunk clean eventdata on your forwarders. You could force reindexing of specific files by copying them to $SPLUNK_HOME/var/log/splunk, though the paths will be a bit different. You can tell splunk to index a particular file regardless of the duplication logic with the oneshot input method: splunk help add oneshot

Lastly, a bit dirty, you could get somewhat sneaky and defeat splunk's redundancy checking. If you modify the first 256 bytes of your logfiles, eg by inserting a single character of whitespace at the start of them, it will reindex those files, assuming they are totally new.

Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...