Splunk Search

Problem using eventstats - not populating all fields

dbryan
Path Finder

I have the following search string (which I've obfuscated slightly):

sourcetype=NetworkImpression
| fields User_ID IP Advertiser_ID Buy_ID Ad_ID Creative_ID Site_ID Page_ID
| eval key = User_ID + IP + Advertiser_ID + Buy_ID + Ad_ID + Creative_ID + Site_ID + Page_ID
| eventstats count by key

If I run it over about fifteen minutes of my data (~45,000 events), it behaves as expected. I get a count field in 100% of them. See here:

alt text

However, anything much more and I start to see this:

alt text

That was over half an hour - ~90,000 events - I see this:

alt text

It continues to go down over a longer time frame.

Can anyone give me an indication of what's going on here? There are no errors that I can see in the internal index, and the search inspector doesn't show anything that stands out to me.

Tags (2)
0 Karma

itinney
Path Finder

What happens if you add the following to the end of your search: | search NOT count="*"

0 Karma

dbryan
Path Finder

Now that I've done some more googling it looks like you might be right - there's a [stats] stanza documented for limits.conf which has a maxresultrows option. It's still specified as only applying to searches that create additional rows, but if it's applying to my eventstats command that'd make sense why I stop seeing values after ~50,000 rows, as that's the universal default for maxresultrows.

0 Karma

dbryan
Path Finder

I just checked the docs, and it looks like this option only affects search commands that grow the result set, so hopefully this isn't it.

0 Karma

dbryan
Path Finder

The thing is that the search still returns all my results - just many of them are not decorated with the data from the eventstats command.

0 Karma

itinney
Path Finder

I would run the search more frequently (so less than 50K results) and save the results to a summary index if this is something you need to report often.

0 Karma

itinney
Path Finder

There is a "configurable" limit of 50,000 search results returned when you Submit a search:

[searchresults]
maxresultrows = 50000

Although the README says that you should not increase this figure. Can you sample smaller time-ranges?

0 Karma

dbryan
Path Finder

I can see all the events missing the count field, corresponding to how many are missing in the search in my question. I don't understand though - I can't find any limits options for eventstats, and 100% of them have count.

0 Karma

dbryan
Path Finder

Yes, I meant to mention- it's always on 100% of events.

0 Karma

Ayn
Legend

Does the "key" field exist in all the events?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...