Why does streamstats in a search stop working when the search is saved?

[asmithe]

Finally got a somewhat longish workaround to a shortcoming in transaction (vis-a-vis my data) working last night.

Saved the search and this morning back to square one.

It seems that the third streamstats command just isn't working and everything collapses from there.

Looked everywhere: docs, forums, answers, and google. No real clue as to why streamstats just chokes.

This happened after I reformatted code for readability.

index=flowspaces sourcetype="growl_log" application="growl" 
| dedup _time eventtype 
| eval firsttime=_time 
| transaction name=Active 
| eval secondtime=(tonumber(mvindex(firsttime,1)))  
| streamstats current=f window=1 global=f last(secondtime) as more_recent_end_time 
| reverse  
| streamstats current=f window=1 global=f last(secondtime) as previous_end_time 
| streamstats current=t count 
| reverse 
| eval latest_idle_time=((tonumber(mvindex(firsttime,0)))-(tonumber(previous_end_time))) 
| eval firstduration=duration 
| streamstats current=f window=1 global=f last(latest_idle_time) as more_recent_idle_time 
| eval merge=if(isnotnull(latest_idle_time) AND latest_idle_time!=0 AND latest_idle_time<16, 1, 0)  
| eval more_recent_merge=if(isnotnull(more_recent_idle_time) AND more_recent_idle_time!=0 AND more_recent_idle_time<16, 1, 0) 
| eval mergedir=if(merge=1 AND more_recent_merge=1, "upanddown",if(merge=1 AND more_recent_merge=0, "down",if(merge=0 AND more_recent_merge=0, "stop",if(merge=0 AND more_recent_merge=1, "up",0))))
| streamstats current=f window=1 global=f last(mergedir) as more_recent_mergedir 
| reverse  
| streamstats current=f window=1 global=f last(mergedir) as previous_mergedir 
| reverse 
| eval KILLME=if(mergedir="stop", count, 
if(mergedir="down" AND (previous_mergedir="up" OR previous_mergedir="upanddown") AND (more_recent_mergedir="stop" OR more_recent_mergedir="up"), count, 
if(mergedir="up" AND previous_mergedir="down" AND more_recent_mergedir="down", count+1, 
if(mergedir="up" AND (previous_mergedir="stop" OR previous_mergedir="down") AND more_recent_mergedir="down", count+1, 
if(mergedir="up" AND (previous_mergedir="stop" OR previous_mergedir="down") AND more_recent_mergedir="upanddown", count+2,
if(mergedir="up" AND more_recent_mergedir="upanddown", count+3, 
if(mergedir="upanddown" AND more_recent_mergedir="upanddown" AND previous_mergedir="upanddown", count+2, 
if(mergedir="upanddown" AND more_recent_mergedir="down" AND previous_mergedir="upanddown", count+1, 
if(mergedir="upanddown" AND more_recent_mergedir="down" AND previous_mergedir="up", count+1, 
if(mergedir="upanddown" AND previous_mergedir="down", count+2, 
if(mergedir="upanddown" AND previous_mergedir="upanddown", count+2, 
if(mergedir="upanddown" AND previous_mergdir=up, count, 
if(mergedir="upanddown" AND more_recent_mergedir="upanddown" AND previous_mergedir="up", count+3, 
if(mergedir="down" AND (previous_mergedir="up" OR previous_mergedir="upanddown"), count,  
if(mergedir="stop", count, "othercondition"))))))))))))) ))
| transaction KILLME  keeporphans=true keepevicted=true mvlist=true 
| eval mergedduration=((tonumber(duration))+(tonumber(mvindex(firstduration,1)))) 
| eval active_sec=case(mergedduration > 30, mergedduration-30, mergedduration < 30, mergedduration) 
| eval sec=active_sec 
| eval active_min=active_sec/60 
| eval min=active_min 
| eval active_hrs=active_min/60 
| eval hours=active_hrs 

[asmithe · Accepted solution]

I spent hours on this. But the solution is simple: rename all my [streamstats produced] fields and regress formatting back to one long line.

Not sure if it was the multiline formatting or the longish evaled field names or maybe the underscore in the field names.

UPDATE: EACH?! time I update or tweak the query, I have to change the name of the field produced by the third streamstats command. This has also happened for the other streamstats fields, but not as regularly.

If anyone can explain this weirdness, please do.

index=flowspaces sourcetype="growl_log" application="growl" | dedup _time eventtype | eval firsttime=_time | transaction name=Active | eval this_endtime=(tonumber(mvindex(firsttime,1))) | streamstats current=f window=1 global=f first(this_endtime) as recent_endtime | reverse | streamstats current=f window=1 global=f last(this_endtime) as previous_endtime | streamstats current=t count | reverse | eval previous_idletime=((tonumber(mvindex(firsttime,0)))-(tonumber(previous_endtime))) | eval previous_idletime=if(isnull(previous_idletime),0,previous_idletime) | eval firstduration=duration | streamstats current=f window=1 global=f last(previous_idletime) as recent_idletime | eval merge=if(isnotnull(previous_idletime) AND previous_idletime!=0 AND previous_idletime<16, 1, 0) | eval recent_merge=if(isnotnull(recent_idletime) AND recent_idletime!=0 AND recent_idletime<16, 1, 0) | eval direction=if(merge=1 AND recent_merge=1, "upanddown",if(merge=1 AND recent_merge=0, "down",if(merge=0 AND recent_merge=0, "stop",if(merge=0 AND recent_merge=1, "up",0)))) | streamstats current=f window=1 global=f last(direction) as recent_direction | reverse | streamstats current=f window=1 global=f last(direction) as previous_direction | reverse | eval KILLME=if(direction="stop", count, if(direction="down" AND (previous_direction="up" OR previous_direction="upanddown") AND (recent_direction="stop" OR recent_direction="up"), count, if(direction="up" AND previous_direction="down" AND recent_direction="down", count+1, if(direction="up" AND (previous_direction="stop" OR previous_direction="down") AND recent_direction="down", count+1,if(direction="up" AND (previous_direction="stop" OR previous_direction="down") AND recent_direction="upanddown", count+2,if(direction="up" AND recent_direction="upanddown", count+3, if(direction="upanddown" AND recent_direction="upanddown" AND previous_direction="upanddown", count+2, if(direction="upanddown" AND recent_direction="down" AND previous_direction="upanddown", count+1, if(direction="upanddown" AND recent_direction="down" AND previous_direction="up", count+1, if(direction="upanddown" AND previous_direction="down", count+2, if(direction="upanddown" AND previous_direction="upanddown", count+2, if(direction="upanddown" AND previous_mergdir=up, count, if(direction="upanddown" AND recent_direction="upanddown" AND previous_direction="up", count+3, if(direction="down" AND (previous_direction="up" OR previous_direction="upanddown"), count, if(direction="stop", count, "othercondition"))))))))))))))) | transaction KILLME  keeporphans=true keepevicted=true mvlist=false | eval mergedduration=((tonumber(duration))+(tonumber(mvindex(firstduration,1)))) | eval active_sec=case(mergedduration > 30, mergedduration-30, mergedduration < 30, mergedduration) | eval sec=active_sec | eval active_min=active_sec/60 | eval min=active_min | eval active_hrs=active_min/60 | eval hours=active_hrs

[asmithe]

Thanks martin! Ill give both a look.

[martin_mueller]

For future development, take a look at the case() expression for eval, that may shorten your nested if-expressions a bit.

Additionally, for debugging complex searches you can give the (experimental) Search Exploder in SideviewUtils 3.0.1 a try.