are postprocess searches counted against the search quota for a given role?

[jguarini]

seems that the quota for searches includes postprocess searches, can someone confirm this? If, so can I get a link to the documentation that covers this? I didn't see anything specific about postprocess searches in the authorize.conf.

so given the following scenario, what is the actual count of searches?

1 main search
5 postprocess searches

is this 6 searches total or just 1?

if this is 6 searches and using the default of 3 concurrently running search jobs, the main search and the first 2 postprocess jobs will run. It seems the remaining searches don't queue up or run, they just fail. Is there a way to get the remaining searches to run after the first 3 searches complete?

splunk version 4.3.4

thanks

so here is a sample test view that demonstrates the 1main search and multiple postprocess logic

using a guest account that just has the 'user' role that has a limit of 2 concurrent searches results in a blue or red bar( depending on Splunk version) noting the quota has been reached

<view autoCancelInterval="90" isVisible="true" objectMode="SimpleDashboard" onunloadCancelJobs="true" refresh="-1" template="dashboard.html">

<label>test</label>

<module name="AccountBar" layoutPanel="appHeader" />

<module name="AppBar" layoutPanel="navigationHeader" />

<module name="Message" layoutPanel="messaging">

<param name="filter">*</param>

<param name="clearOnJobDispatch">False</param>

<param name="maxSize">1</param>

</module>

<module name="TitleBar" layoutPanel="viewHeader">

<param name="showActionsMenu">False</param>

</module>

<module name="HiddenSearch" autoRun="True" layoutPanel="panel_row1_col1">

<param name="search">index=foo | fields _time, oid,metric,count,str| fields *</param>

<param name="earliest">-4h</param>

<param name="latest">now</param>

<module name="GenericHeader" layoutPanel="panel_row1_col1">

<param name="label">panel1</param>

<module name="HiddenPostProcess">

<param name="search">| where oid=420 | where metric="status_code" | timechart span=1m avg(count) by str</param>

<module name="ViewstateAdapter">

<module name="HiddenFieldPicker">

<param name="strictMode">True</param>

<module name="EnablePreview">

<param name="enable">True</param>

<param name="display">False</param>

<module name="HiddenChartFormatter">

<param name="charting.axisTitleY.text"></param>

<param name="charting.axisTitleX.text"></param>

<param name="charting.axisTitleX.visibility">collapsed</param>

<param name="charting.chart">line</param>

<param name="charting.chart.nullValueMode">connect</param>

<param name="charting.legend.placement">top</param>

<param name="charting.chart.resultTruncationLimit">10000</param>

<module name="FlashChart">

<param name="maxResultCount">100000</param>

<param name="width">100%</param>

<param name="height">250px</param>

<module name="Gimp" />

</module>

</module>

</module>

</module>

</module>

</module>

</module>

<module name="GenericHeader" layoutPanel="panel_row1_col2">

<param name="label">panel2</param>

<module name="HiddenPostProcess">

<param name="search">| where oid=677 | where metric="status_code" | timechart span=1m avg(count) by str</param>

<module name="ViewstateAdapter">

<module name="HiddenFieldPicker">

<param name="strictMode">True</param>

<module name="EnablePreview">

<param name="enable">True</param>

<param name="display">False</param>

<module name="HiddenChartFormatter">

<param name="charting.axisTitleY.text"></param>

<param name="charting.axisTitleX.text"></param>

<param name="charting.axisTitleX.visibility">collapsed</param>

<param name="charting.chart">line</param>

<param name="charting.chart.nullValueMode">connect</param>

<param name="charting.legend.placement">top</param>

<param name="charting.chart.resultTruncationLimit">10000</param>

<module name="FlashChart">

<param name="maxResultCount">1000000</param>

<param name="width">100%</param>

<param name="height">250px</param>

<module name="Gimp" />

</module>

</module>

</module>

</module>

</module>

</module>

</module>

<module name="GenericHeader" layoutPanel="panel_row2_col1">

<param name="label">panel3</param>

<module name="HiddenPostProcess">

<param name="search">| where oid=706 | where metric="status_code" | timechart span=1m avg(count) by str</param>

<module name="ViewstateAdapter">

<module name="HiddenFieldPicker">

<param name="strictMode">True</param>

<module name="EnablePreview">

<param name="enable">True</param>

<param name="display">False</param>

<module name="HiddenChartFormatter">

<param name="charting.axisTitleY.text"></param>

<param name="charting.axisTitleX.text"></param>

<param name="charting.axisTitleX.visibility">collapsed</param>

<param name="charting.chart">line</param>

<param name="charting.chart.nullValueMode">connect</param>

<param name="charting.legend.placement">top</param>

<param name="charting.chart.resultTruncationLimit">10000</param>

<module name="FlashChart">

<param name="maxResultCount">1000000</param>

<param name="width">100%</param>

<param name="height">250px</param>

<module name="Gimp" />

</module>

</module>

</module>

</module>

</module>

</module>

</module>

<module name="GenericHeader" layoutPanel="panel_row2_col2">

<param name="label">panel4</param>

<module name="HiddenPostProcess">

<param name="search">| where oid=706 | where metric="status_code" | timechart span=1m avg(count) by str</param>

<module name="ViewstateAdapter">

<module name="HiddenFieldPicker">

<param name="strictMode">True</param>

<module name="EnablePreview">

<param name="enable">True</param>

<param name="display">False</param>

<module name="HiddenChartFormatter">

<param name="charting.axisTitleY.text"></param>

<param name="charting.axisTitleX.text"></param>

<param name="charting.axisTitleX.visibility">collapsed</param>

<param name="charting.chart">line</param>

<param name="charting.chart.nullValueMode">connect</param>

<param name="charting.legend.placement">top</param>

<param name="charting.chart.resultTruncationLimit">10000</param>

<module name="FlashChart">

<param name="maxResultCount">1000000</param>

<param name="width">100%</param>

<param name="height">250px</param>

<module name="Gimp" />

</module>

</module>

</module>

</module>

</module>

</module>

</module>

</module>

</view>

[sideview]

Yep, it's the first problem I guessed. I updated my answer. Also download Sideview Utils 2.X and read the framework docs page aka "Introduction to the Advanced XML". Use a throwaway email address if you're concerned that I'm going to spam you with emails (I wont). 😃

[jguarini]

just updated the post with a sample view that should demonstrate the logic, sorry the formatting sucks

I most likely have something wrong with my logic, thus the reason I reached out to the Splunk community

[sideview]

I recommend posting your view XML. One kind of mistake in postProcess config can cause multiple searches to dispatch inadvertently. Another kind of mistake can inadvertently queue up the dispatch behavior, but in a way that has nothing to do with the concurrent search quota.

[jguarini]

so this may be a non-issue if one is using 5.X code (specifically 5.0.3). I wanted to be sure that it wasn't a revision related issue, and it looks like it may be.

on 4.3.4 the searches just fail

on 5.0.3 they queue and the page renders.

this works around my issue, but leaves the question:
if post process searches don't count against the concurrent search quota, then why are they?

if they don't, then please provide a working example, as my simple test cases seem to prove they do.

[sideview · Accepted solution]

No, postProcess searches do not count towards your concurrent search limits.

One thought though, is that it's not hard to create a custom view in the Advanced XML where you think it's going to be using 1 search and 6 postProcess searches, and in fact it dispatches 6 identical searches, each with their own single postProcess search. I've seen a couple companies hit this problem, and sometimes it takes them a while to realize that it's even happening.

The best way to fill the knowledge gap and at the same time to figure out how the Advanced XML works, is to read the "Introduction to the Advanced XML" page in Sideview Utils 2.X You can get the app for free from the Sideview site - http://sideviewapps.com/apps/sideview-utils and it's worth it even if you just read the key docs pages and then throw it away.

the "Introduction to Advanced XML" page basically has two topics. the first explains all the nuances around the "upstream/downstream" concept, and modules talk to eachother. The second explains exactly how, where, why, when searches get dispatched at runtime.

If you only ever read one page in Sideview Utils, make it that one. (If you read two, read the PostProcess overview docs as well)

UPDATE - Looking at your XML, indeed, you've fallen into the first problem that I mentioned. Because you've gotten a little weird with the hierarchy, you've nested things such that you're going to be dispatching four identical searches.

Bost basic problem - don't nest anything in those GenericHeader modules. Have all your HiddenPostProcess modules right immediately under the HiddenSearch. Also there's a ton of config in here that's unnecessary cruft carried over from the conversion from SimpleXML. real quick - delete all the ViewStateAdapter, Gimp, HiddenFieldPicker modules. Deindent every module that is indented (aka downstream) from EnablePreview and GenericHeader.

That will take you a long way towards cleaning up your view. Also keep the indentation in the XML file cleaner and that will help you towards avoiding weirdness. If you're familiar with advanced tools like tidy, you can clean up your indentation that way. Another way is a little weird, and that's to download Sideview Utils, use the Sideview Editor to edit your view, open a single edit screen somewhere and change nothing, but submit it. The Editor will then write out a file to etc/users/admin/<appname> matching the filename of the view. If you go look at this view it will be the perfectly indented version of the same view. Strange trick, and you have to delete the local "user-specific" copy of the view or you'll get confused later, but it can bring a horrifically indented view back from the dead and doesn't require any command line mastery like other tools do.

http://sideviewapps.com/apps/sideview-utils

[sideview]

The only good explanation, including any advanced XML documentation on the Splunk website, is the explanation in the "introduction to the advanced XML" page of Sideview Utils 2.X. It's really quite long though and too long to fit here. You will understand once you read this documentation, trust me.

[jguarini]

so one thing I don't understand, is that why the GenericHeader module would kick off another search. According to the documentation in Sideview utils 2.x any module that can be considered a "dispatching module" would initiate a search if no search was currently running or no search results could be accessed. The GerneicHeader module "just displays the configured text as a header element on the page." thus no search would be needed. With that said, seperating out the search from the generciheader stuff makes for a much cleaner layout.

[jguarini]

thanks

yeh alot of the nesting is weird and not really waht I wanted to do, just needed something quickly to provide as an example. So removing the GenericHeader module results in not hitting the concurrent search quota.

It's rather interesting that the Splunk documentation doesn't say anything about the GenericHeader module spawning off a new search.

If I had the ability to use a throw-away email to get documentation from sideview utils 2.x, trust me I would, but I don't at the moment. Maybe I'llgrab a copy once I get home.

thanks

[sideview]

I am positive that postProcess searches aren't counted against the quota. They're not really "searches" at all. It's just search syntax that can be submitted when you're asking for search results. If they counted against the quota then you'd rack up dozens of them whenever UI elements rendered.

As far as entering your email address to download Sideview Utils - Besides the download link itself, the only time you'd ever get email from Sideview because of that is if there are changes to the license terms. There is a Sideview Utils mailing list but you aren't put on it automatically.

[jguarini]

In the Splunk module reference for the HiddenPostProcess there is no mention of the search being spawned off and/or counting against the concurrent search quota.

If you have a working example using one main search and a few post process searches that demonstrate that post process searches aren't counted against the quota I would love to see that.

thank you

[jguarini]

thanks for the response.
Unfortunately I don't care to have to submit my email address in-order to get something. I don't mind agreeing to the licensing, but the cost of having to submit my name and email is why I stay on the 1.X release.

If you have Splunk documentation that points to how/why the postprocess searching works in conjunction to the concurrent search quota that would be wonderful.

will continue it another reply due to character limit

[aholzer]

1) postprocessors don't count towards concurrent searches. @bmacias84 was correct.
2) the limit is applied as an equal statement. If the limit is 3 and the user has 2 concurrent searches running and tries to run 1 more, it will fail.
3) I'm running on version 5.0.1. This could mean that older versions don't have the same behaviour for post processors.
4) if you look on the indexer server there is only 1 search running for the dashboard; the base search.

Hope this helps.

[aholzer]

I was curious about this question so decided to test it. I set a test user with access to a dashboard with 1 base search and 3 post processors.

If I set the role for the test user to have a limit of 1 concurrent search, then it wouldn't load.

If the role had a limit of 2 concurrent searches (yes real-time searches) the dashboard would load correctly without a problem.

If the role had a limit of 3+ (I tested at 4 and 100) the dashboard worked properly as well.

This seems to indicate the following (see comment below for list, as I am running low on characters)

[bmacias84]

Here a post about simultaneous searches http://splunk-base.splunk.com/answers/6261/simultaneous-queriesjobs-limit. Though doesn't anwser the question if postprocess searches count against a users concurrent searches. I was under the impression that postprocess didn't rerun the basesearch, but instead processed the results from the basesearch feeding them into the module.