Splunk Search

Why is clustered indexers duplicating non-load balanced data?

scottrunyon
Contributor

When I run a simple query "index=syslog update sourcetype=fgt_event devname=xxxxx", it returns duplicate (2) events with the only difference being the splunk_server field. The device is sending syslog data to only one of the indexers. I am using the standard UDP:514 Data Input to receive this data.

Splunk setup
2 server indexing cluster
2 non-clustered search heads.

Question 1 - Is this affecting my licence quota? Syslog data my largest source.
Question 2 - How do I clean this up as it is affecting reporting?

Thank you in advance for any help provided.

0 Karma
1 Solution

jdonn_splunk
Splunk Employee
Splunk Employee

Can you double check the values for source and sourcetype? If they are truly duplicated events, I would think that your search head is not configured correctly.

http://docs.splunk.com/Documentation/Splunk/6.4.2/Indexer/Aboutclusters

More specifically - http://docs.splunk.com/Documentation/Splunk/6.4.2/Indexer/Configurethesearchhead

View solution in original post

0 Karma

jdonn_splunk
Splunk Employee
Splunk Employee

Can you double check the values for source and sourcetype? If they are truly duplicated events, I would think that your search head is not configured correctly.

http://docs.splunk.com/Documentation/Splunk/6.4.2/Indexer/Aboutclusters

More specifically - http://docs.splunk.com/Documentation/Splunk/6.4.2/Indexer/Configurethesearchhead

0 Karma

scottrunyon
Contributor

It turns out the the firewall that should be sending data to only one indexer was actually configured to send data to both indexers in the cluster. Splunk was performing as expected.

0 Karma

lycollicott
Motivator

Can you post your 2 events??

0 Karma

scottrunyon
Contributor

It turns out the the firewall that should be sending data to only one indexer was actually configured to send data to both indexers in the cluster. Splunk was performing as expected.

0 Karma

scottrunyon
Contributor

As there are multiple events returned with each search, what are you wanting me to post?

When the search completes and you look at the events tab, all fields are the same except the splunk_server field which has a name of one of the two indexers.

I tried to verify that the data is being indexed only once by using the following search -

sourcetype=fgt_event | eval dupfield=_raw | transaction dupfield maxspan=1s keepevicted=true | where mvcount(sourcetype) > 1

There were no duplicated raw values.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...