Dashboards & Visualizations

HiddenPostProcess 10000 limit? Configurable? How to overcome? By summarising?

Alex_Megremis
Explorer

Hi all,

I am having difficulty working around what I understand to be a hard limit of 10000 events for the HiddenPostProcess module. I am looking for a way to reduce the granularity of the events going across, but without success.

Example problem code:

<module name="HiddenSearch" autoRun="False" layoutPanel="mainSearchControls">
    <param name="search">index=analysisindex01
        eventtype="EVENTS01" OR  
        eventtype="EVENTS02" OR 
        eventtype="EVENTS03" OR 
        eventtype="EVENTS04" OR
        eventtype="EVENTS05" OR 
        eventtype="EVENTS06" OR 
        eventtype="EVENTS07" OR
        eventtype="EVENTS08"
        timeformat="%d/%m/%Y" 
        starttime=10/11/2010 
        searchtimespandays=5
        | fields eventtype, _time
    </param>
    ... stuff ...
    <module name="HiddenPostProcess" layoutPanel="panel_row2_col1">
                    <param name="search">timechart span="1d" count(eventtype) by eventtype</parm>

This will fail. It will produce results, but they will be capped to 10000.

If I do

    timeformat="%d/%m/%Y" 
    starttime=10/11/2010 
    searchtimespandays=5
    | bucket _time span=1h
    | fields eventtype, _time

in an attempt to summarize the events a bit, I only get 1 row, for 1 day, in the reports generated, and the numbers in it make no sense.

Is there something I am missing?

Any and all help is greatly appreciated.

Tags (1)
1 Solution

steveyz
Splunk Employee
Splunk Employee

bucket doesn't actually summarize at all. bucket just discretizes the value, e.g. in your search, it means for each event, snap the value of _time to the previous hour boundary.

What you'd want to do is something like

timeformat="%d/%m/%Y" starttime=10/11/2010 searchtimespandays=5 | bucket _time span=1h | stats count by _time eventtype

That will give you 1 row per day per eventtype

then in your hidden post process, you can do

timechart span=1d sum(count) by eventtype

View solution in original post

steveyz
Splunk Employee
Splunk Employee

bucket doesn't actually summarize at all. bucket just discretizes the value, e.g. in your search, it means for each event, snap the value of _time to the previous hour boundary.

What you'd want to do is something like

timeformat="%d/%m/%Y" starttime=10/11/2010 searchtimespandays=5 | bucket _time span=1h | stats count by _time eventtype

That will give you 1 row per day per eventtype

then in your hidden post process, you can do

timechart span=1d sum(count) by eventtype
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...