Getting Data In

How to reindex old data residing in one index into a new index?

Bliide
Path Finder

I corrected an error in an index where data was being consumed by 2 indexes. I created a new index (IndexB) and the data is being gathered correctly now. The new index has all the data in the log files after the creation of the index but it did not get the data that was already indexed in another index (IndexA). I need to pull the old log files into the new index. How do I tell the new index to gather all the old data from the log files?

Tags (2)
0 Karma

chanfoli
Builder

I think a good approach is to use oneshot to index your log files with the correct options. See the following for more info:

http://docs.splunk.com/Documentation/Storm/Storm/User/CLIcommandsforinput

More hints, and other approach here:

http://answers.splunk.com/answers/72562/how-to-reindex-data-from-a-forwarder.html

aakwah
Builder

You need to reprocess the old files by moving them to the new monitored directory but the issue now is that splunk forwarder will not index them because they are already processed, so you have 2 solutions to reprocess old files:

 -Splunk forwarder keep track of processed files through fishbucket directory "/opt/splunkforwarder/var/lib/splunk/fishbucket/", so if you remove all the contents of fishbucket directory splunk will process again all files under monitored directories which will process the required files to the new index "IndexB", but this also will cause duplicates because all files will be processed, so you should move all processed files to archive directory to not be processed again.

 -Second solution is make small edit on files you want to process by adding newline or space for example, as splunk will check the checksum of the file to identify if the file is processed or not, unfortunately changing file name is not enough.

Hope this answer your questions, please let me know if you still have issues.

Regards,
Ahmed

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...