Splunk Search

How to get a daily count of distinct users over a large time range without re-running from beginning of time

e_sherlock
Explorer

I have a simple "| stats dc()" command to get a cumulative sum of distinct users; however, I don't want to have to run this query from the beginning of time every day it runs for performance reasons.

These couple routes came to mind, but don't seem best...

  1. Summary indexes can write out the distinct users per day, but seems like couldn't compute distinct over the whole set of days.
  2. Write the distinct values to lookup and only adding the new values each day -- a kinda hacky incremental approach.

Any ideas on the optimal solution?

Tags (2)
0 Karma
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

The best way would be to use sistats and a summary, or alternatively, use report acceleration. With report acceleration, just set up your search and tell Splunk to accelerate, and that should do it.

If you summarize yourself, then on a daily basis (and you can backfill later), you run a;

... | sistats dc(user) by x,y,z 

and store that to a summary. Then to get your counts:

index=my_summary_index name=my_summary_job | stats dc(user)

(or ... | stats dc(user) by x,y,z or ... | stats dc(user) by x,y).

the sistats command will have saved the right data, and the stats command will know how to handle what sistats did. Yes, it is doing slightly clever things under the hood.

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

The best way would be to use sistats and a summary, or alternatively, use report acceleration. With report acceleration, just set up your search and tell Splunk to accelerate, and that should do it.

If you summarize yourself, then on a daily basis (and you can backfill later), you run a;

... | sistats dc(user) by x,y,z 

and store that to a summary. Then to get your counts:

index=my_summary_index name=my_summary_job | stats dc(user)

(or ... | stats dc(user) by x,y,z or ... | stats dc(user) by x,y).

the sistats command will have saved the right data, and the stats command will know how to handle what sistats did. Yes, it is doing slightly clever things under the hood.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...