Splunk Search

Why do certain searches return duplicates for some events that were only indexed once?

Murali2888
Communicator

Hi All,

I came across a weird behavior where a search head displaying duplicate events only in certain scenarios, even though the event is indexed only once. I confirmed the indexing part by checking the metadata and _indextime values for the events.

When I run the Base Search for a month period, a few events are being displayed twice resulting in invalid number of events. However, when I run Base Search | timechart span=1mon count the events are not duplicated and give the correct count.

Has anybody came across this sort of behavior and would like to understand how the search head would render events?

Thanks for your help.

0 Karma

javiergn
SplunkTrust
SplunkTrust

Hi,

Can you try appending the following to one of those searches returning duplicate events? More info here

yoursearch 
| eval myUniqueId = index + "_" + _cd + "_" + splunk_server 
| stats count by myUniqueId 
| where count > 1

It should return 0.

Then try the following too:

yoursearch 
| stats count by _raw 
| where count > 1

It should return 0 too.

If any of these searches is returning anything at all, can you please post how your search looks like so that we can investigate this further? If none of those events was indexed twice, they shouldn't show up twice.

If we manage to identify the duplicates we can delete them to avoid problems in future but we should try to find the root cause first.

Which version of Splunk are you running? Can you give us more information about your deployment? Is it distributed, multisite, any clustering, etc?

Thanks,
J

0 Karma

Murali2888
Communicator

Hi javiergn,

I ran those two searches and both returned no results. This confirms that the data is not indexed twice.

We are running Splunk V6.3.0 in both Search Head and Indexers in a distributed environment. The Search Head and Indexers are in different sites. Indexers are in secured environment for data protection.

We do not have any clustering deployed in SH or in Indexers. It is a single Search Head talking to 4 Indexers in distributed manner.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...