How to timechart a map command with multiple input rows

[weidertc]

I have a map command whose input contains multiple rows. The input is responsible for collecting the names of macros, each containing a different search string. I need to aggregate the results. Normally, I would do this.

<input with variable number of rows>
| map search="search earliest=@d latest=now `$macroname$`" maxsearches=10000
| timechart span=1hr count by status

this gives the results in the format I need, but it truncates the results. The map will iterate through all rows, and run a single timechart command, but the limits.conf has a 10,000 limit, and my search returns more than 10k rows.

So I moved the timechart inside the map command

<input with variable number of rows>
| map search="search earliest=@d latest=now `$macroname$`
| timechart span=1hr count by status" maxsearches=10000

This gives me all the data since the timechart reduces the row count to less than 10k, but not in the format I want. For example, if the timespan is from 8am to 10am, and map iterates through 3 rows of them, then I get 8am to 10am and then it goes back to 8am in the timechart, as if it's going backwards. i see 8am to 10am 3 times if there's 3 rows, where the first set contains data from the first search query, and the second set contains data form the second search query, and so on. or 5 times if there's 5 rows. this won't work either.

I need a solution that doesn't truncate the data and gives me a single timespan of 8am to 10am with everything in it. Since the fields are not known until the search runs, I can't do a sum. I am not bound by map, and prefer not to use it, but i must maintain relationships with all the data.

here's a sample output from the Statistics tab that shows the issue. There are 3 sets of 12am, 1am, etc, not combined and looks awkward in the timechart.

_time, field1, field2
08 am, 10, 5
09 am, 10, 5
10 am, 10, 5
08 am, 2, 2
09 am, 2, 2
10 am, 2, 2

I need it to show this

_time, field1, field2
08 am, 12, 7
09 am, 12, 7
10 am, 12, 7

Again, the field names or number of fields are not know until the search runs, as the field name is the error message and it could be any number of them.

How can I, using map or not, get my chart to only show a single frame of time in the panel even with multiple input rows and avoiding the event limit in limits.conf?

-Chris

[woodcock]

Do one search like this:

index=YouShouldAlwaysSpecifyAnIndex AND sourcetype=AndSourcetypeToo earliest=@d latest=now AND
[|inputlookup YourLookup.csv | table macroname ]
| timechart span=1hr count by status

[weidertc]

Sorry for the late reply. The index and sourcetype come from the macro, which comes from the lookup table. Can this logic be reversed?

[somesoni2 · Accepted solution]

You can do another timechart outside map command to further aggregate those rows. Like this:

<input with variable number of rows>
 | map search="search earliest=@d latest=now `$macroname$`
 | timechart span=1hr count by status" maxsearches=10000
 | timechart span=1h sum(*) as *

There may be a way to even remove map so that the limitation of 10k doesn't apply, but it will depend upon what your macros are written as. Could you give example of the result that you'd get from your <input with variable number of rows>? Mask anything sensitive.

[weidertc]

Sorry for the late reply. Adding a second timechart works. Thanks! logically it seems like a bit much. I will try the other answer too, but this does what asked. In the meantime, how do i convert yours to an answer?

A typical input row has lots of fields, but the important ones are:

id1, name1, macro1
id2, name2, macro2

the macros have things like this:

macro1:

index=asdf sourcetype=qwerty field1=zxcv, this=12, status=success

macro2:

index=asdf sourcetype=qwerty field1=tyuiop, this=432, status=fail

[ppablo]

Just converted the comment by @somesoni2 to an answer. You can now click Accept on his answer to resolve your post @weidertc