How can I know when Splunk reaches the 10000 result limit in a search to output a certain message?

[nikkkc]

In my search, I calculate some values, but if I reach the 10000 result limit, I get wrong results. I would like change the wrong results to something like "its bigger than.....". I don't care about the real value, but I would like to show that something did not finish getting calculated.

Thanks in advance

[javiergn]

If you want to do this programatically focus on the messages you can extract from the Search/Jobs endpoint:

| rest /services/search/jobs
| table author, title, cursorTime, messages.*

For instance, if you run the following:

index=_internal | head 10001 | sort _time

That will trigger an Info message in your job inspector (icon becomes green next to the Job dropdown).
This is great when you are running manual queries but what if you are running scheduled ones.
The rest query above will return the following in that case:

author  title   cursorTime  messages.error  messages.fatal  messages.info   messages.warn
admin   "search index=_internal | sort _time"   2016-01-18T00:00:00.000+00:00           The sort command is truncating output to 10000 rows      

That you can parse later on and use it for your own needs.

[nikkkc]

mmmhhh, i got a list with a lot of records from my search query and i want to know if something is truncated for a value in every record separately!

[javiergn]

If you just want to display those with a message that contains the word "truncating" do this instead:

| rest /services/search/jobs
| search 
   messages.info = "*truncating*" OR 
   messages.warn = "*truncating*" OR
   messages.error = "*truncating*" 
| table author, title, cursorTime, messages.*

Keep in mind there are lots of other fields you might find useful. The query above is just an example.

[jeffland]

Which limit are you reaching, the subsearch limit? The limit when sorting? We need to know some more details.

Usually, you get a warning next to the "Job"-dropdown just next to the buttons where you pause or stop a search.

[nikkkc]

hi, this is what job inspector says: (first reaching is with mvexpand or is it just by happenstance?, then presort and so on...)

Execution costs
Duration (seconds) Component Invocations Input count Output count
0.00 command.dedup 3 1 1
0.46 command.eval 60 100,010 100,010
0.00 command.fields 10 16 16
0.14 command.mvexpand 3 1 10,000
0.00 command.prededup 2 7 1
0.08 command.presort 1 10,000 10,000
0.00 command.rename 3 10,000 10,000
0.01 command.search 2 - 7
0.00 command.search.index 2 - -
0.00 command.search.calcfields 1 7 7
0.00 command.search.fieldalias 1 7 7
0.00 command.search.filter 1 - -
0.00 command.search.index.usec_1_8 398 - -
0.00 command.search.index.usec_512_4096 1 - -
0.00 command.search.kv 1 - -
0.00 command.search.rawdata 1 - -
0.00 command.search.lookups 1 7 7
0.00 command.search.summary 2 - -
0.00 command.search.tags 1 7 7
0.00 command.search.typer 1 7 7
0.04 command.sort 1 10,000 10,000
0.16 command.table 1 10,000 20,000
0.00 dispatch.check_disk_usage 1 - -
0.00 dispatch.createdSearchResultInfrastructure 1 - -
0.10 dispatch.emit_prereport_files 1 - -
0.40 dispatch.evaluate 1 - -
0.40 dispatch.evaluate.search 1 - -
0.00 dispatch.evaluate.eval 20 - -
0.00 dispatch.evaluate.dedup 1 - -
0.00 dispatch.evaluate.fields 3 - -
0.00 dispatch.evaluate.mvexpand 1 - -
0.00 dispatch.evaluate.rename 1 - -
0.00 dispatch.evaluate.sort 1 - -
0.00 dispatch.evaluate.table 1 - -
0.45 dispatch.fetch 3 - -
0.01 dispatch.localSearch 1 - -
0.37 dispatch.preview 1 - -
0.17 dispatch.preview.command.table 1 10,000 20,000
0.08 dispatch.preview.command.presort 1 10,000 10,000
0.05 dispatch.preview.command.sort 1 10,000 10,000
0.01 dispatch.preview.write_results_to_disk 1 - -
0.00 dispatch.results_combiner 3 - -
0.01 dispatch.stream.local 2 - -
0.19 dispatch.timeline 3 - -
0.02 dispatch.writeStatus 7 - -
0.06 startup.configuration 1 - -
0.58 startup.handoff 1 - -

[jeffland]

I don't see a limit hit there.