How does splunk work with multiple sub-searches? If I want to have two sub-searches which one is evaluated first? Is it the last sub-search followed by the second to last etc... followed by the main? If my second sub search is creating a new stats value of numbererrors does that field need to be passed to the first sub search or does everything get passed to the main search from both sub searches?
If a search has one subsearch, obviously it runs the subsearch first. This same pattern continues for nested subsearches too. Ie if a subsearch itself contains another subsearch, Splunk runs the innermost one first, and then works its way out:
index=foo error [ search index=bar baz [search index=* | stats count by user | search count>100]| stats count by host ]
(Note that this is an artificial case, and as with many searches that appear to embrace subsearches, the subsearch logic probably isn't necessary and could be rewritten without it to be faster)
On the other hand, if you have more than one search but they're all at the same level, ie no one subsearch is contained in another, that is a bit different. In that case Splunk 6.2 appears to run them one at a time starting with the leftmost one and working it's way toward the right.
You can see Splunk chewing through these by running a really nasty experimental search and then looking at the Jobs Activity page in another window. At any one time during the execution there will be one main job running, and then various little "| subsearch" jobs running one at a time. If you click "Inspect job" on these as they are running you can figure out which of the N subsearches they represent and verify that Splunk runs such subsearches one at a time.
btw I used as my test searches things like index=* OR index=_* [search index=* | stats count by component] [search index=* | stats count by user]
over all time, finalizing/cancelling when I had inspected enough subsearches going by
Could you give your full search example - I suspect that you might be able to make use of eventstats to count your errors.
Concerning the passing of fields part of your question, do you have an example query? Guessing what you mean from ambiguous natural language is hard, much easier in formal syntax 🙂
Good question.
You can look in the 'jobs' section to see what order the jobs were dispatched in. It seems to be first defined = first dispatched, but would be interested in a difinitive answer.
The example I used was
[ search * earliest=-1d | tail 100 | fields sourcetype | dedup sourcetype ] OR [search * earliest=-1d | tail 100 | fields source | dedup source ]
1st subsearch took 7 seconds, and the 2nd subsearch was dispatched 7 seconds after the 1st