In order to retain a small subset of events for a longer retention period, is it possible to have events copied to two indexes at index time?

[andrewgarvin]

I have a need to retain a small subset of events in an index for a longer retention period. I have all the Windows Event Logs from all my servers going to an index with a 90 day retention period. But, I have a few events types that I would like to retain for longer (2 to 3 years).

Is there a way to have events be copied to two indexes at index time? I want all Windows Event Logs to go into the main event log index and the special events for long term retention to go to the main event log index AND a separate long term retention event log index.

I have tried to use summary indexing as per the recommendation from other posts. But when doing that it causes three problems.
1. The host, source, and sourcetype are all changed, I need those preserved.
2. All the field extractions are gone. I believe this is because the sourcetype is changed to stash so if #1 is fixed, that may resolve this too.
3. The time of the events loses the time zone. I have events forwarded from multiple time zones. When the events are copied to the summary index, the times are all the raw time on the event, not the real time it was generated.

[lguinn2]

I would not clone the data. And you don't have to use summary indexing, either.

What you could do is this: instead of sending the data to just one index, divide the data into two indexes. One index could have a 90-day retention, and the other could be set for 1000 days (about 3 years). For this example, let's call the indexes win90 and win1000.
Now, to search both indexes, users could simply search index=win* Or perhaps the users will search both indexes by default.
Set up the indexes as you normally would, with the appropriate retention.

When you are bringing in the data, send it to the win90 index. But as you are parsing, pick off the "interesting events" and send them to the win1000 index instead. Now you will have only 1 copy of each event.

To do this, you will need to use props.conf and transforms.conf on the indexer(s):

props.conf

[yoursourcetype]
TRANSFORMS-routing = route_events

transforms.conf

[route_events]
SOURCE=_raw
REGEX = (?msi)^EventCode=560
DEST_KEY = _MetaData:Index
FORMAT = win1000

This will send any events that start with "EventCode=560" to the win1000 index. The remaining events will go to the index that was originally specified in inputs.conf (hopefully win90)
Of course, you will need to set the REGEX appropriately to capture all and only the events you want to retain.

HTH

[somesoni2]

See below link for instruction on data cloning. Please remember that this cause double license usage for those cloned events.

https://answers.splunk.com/answers/224060/cloning-set-of-data-to-specified-splunk-indexer.html