Getting Data In

Reindex Duplicates / Reindex duplicate data

omuelle1
Communicator

HI Splunkers,

I got a little complicated issue I cannot figure out.

Everyday I receive a host file that we index that contains 80% of that data is duplicate from the days before (same timestamp everything). Splunk goes ahead and only indexes the non duplicates, which I would usually appreciate.

However, for a specific report I need to find a setting where Splunk overwrites the old events and indexes a duplicate from the latest source.

For example:

I get event 1111 - timestamp 1-17  on 1/17 in logfile1_17.txt

on 1-18 I have the event 1111 - timestamp 1-17 in logfile1_18.txt 

Now when I look for this event 1111 , I find as source logfile1_17.txt but I would need it indexed again and have source logfile1_18.txt

Is there a setting that the event 1111 is shown in splunk with the latest indexed file as source or is indexed again ?

Thanks.

Oliver

0 Karma

omuelle1
Communicator

Thank you, I went crcSalt = which also seems to reindex duplicates.

0 Karma

nabeel652
Builder

If this is the behaviour you want then consider using batch instead of monitor.

[batch://<path>]
disabled = false
move_policy = sinkhole
index = yourindex
sourcetype = somesourcetype

This will index any file put in that directory and delete it once ingested (so keep that in mind that you may lose the file once indexed). However the move_policy set to sinkhole will reindex the same file if put in that folder with different timestamp instead of following the tail.

gurugv
Engager

Would this duplicate existing data? Would the summary indices be automatically updated?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...