Why do certain searches return duplicates for some events that were only indexed once?

[Murali2888]

Hi All,

I came across a weird behavior where a search head displaying duplicate events only in certain scenarios, even though the event is indexed only once. I confirmed the indexing part by checking the metadata and _indextime values for the events.

When I run the Base Search for a month period, a few events are being displayed twice resulting in invalid number of events. However, when I run Base Search | timechart span=1mon count the events are not duplicated and give the correct count.

Has anybody came across this sort of behavior and would like to understand how the search head would render events?

Thanks for your help.

[javiergn]

Hi,

Can you try appending the following to one of those searches returning duplicate events? More info here

yoursearch 
| eval myUniqueId = index + "_" + _cd + "_" + splunk_server 
| stats count by myUniqueId 
| where count > 1

It should return 0.

Then try the following too:

yoursearch 
| stats count by _raw 
| where count > 1

It should return 0 too.

If any of these searches is returning anything at all, can you please post how your search looks like so that we can investigate this further? If none of those events was indexed twice, they shouldn't show up twice.

If we manage to identify the duplicates we can delete them to avoid problems in future but we should try to find the root cause first.

Which version of Splunk are you running? Can you give us more information about your deployment? Is it distributed, multisite, any clustering, etc?

Thanks,
J

[Murali2888]

Hi javiergn,

I ran those two searches and both returned no results. This confirms that the data is not indexed twice.

We are running Splunk V6.3.0 in both Search Head and Indexers in a distributed environment. The Search Head and Indexers are in different sites. Indexers are in secured environment for data protection.

We do not have any clustering deployed in SH or in Indexers. It is a single Search Head talking to 4 Indexers in distributed manner.