Deployment Architecture

Running the same search, why are different results showing up?

szymonledzinski
New Member

If I run the same search using the same time window I get sometimes different results.
I have added

| eval bkt=_bkt | stats count by splunk_server index bkt 

At the end of the search to check which buckets are being read.
For some reason splunk skips 1 or 2 buckets sometimes.
We are using indexer cluster (10 nodes, 2 search factor, 3 replication factor).
All Data is Searchable, Search Factor is Met and Replication Factor is Met.
I don't see any errors in search logs. Any ideas what could be a problem?

0 Karma

nickhills
Ultra Champion

Because buckets are uniquely named per indexer, it will depend which indexer in your cluster provides the results to your search.

Since you have a Search factor > 1, there are two or more copies of each bucket (which will have different names on each indexer since each indexer applies its GUID to the end)

Its not an exact science, but (if your data is well distributed) a search over a small time window should return ~10 buckets (ideally one from each indexer)
If you run that search later, it's conceivable that you could get 10 entirely different buckets returned (from different servers), whilst representing the exact same results. This is by design.

If my comment helps, please give it a thumbs up!
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...