Eval fields not making it to Summary index when stats is used.

[richard_whiffen]

I'm still sifting through the 'realated questsions' proposed in "Ask a Question" (great feature btw), but I don't think my senario is covered.

I have a search set to run every 24 hours to sumarize the previous 24 hours stats.

host="HOSTSBLAH*"  (source="/usr/local/tvs/apache-tomcat/logs/qlogger/*" NOT source="*.gz") 
| lookup Market_by_Controller_ID Controller_ID as Controller_ID OUTPUT Market as Market 
| eval QueryFirstTwo=substr(TextQuery,1,2) 
| transaction MAC, QueryFirstTwo maxspan=5m maxpause=1m delim="," mvlist=TextQuery 
| eval LastQuery=mvindex(TextQuery, -1) 
| fillnull value=0 forward
| eval MAC="salt".MAC 
| eval MAC=md5(MAC) 
| stats count(LastQuery) by Market, Controller_ID, StreamingServerID, forward 
| fields count LastQuery, Controller_ID StreamingServerID Market forward MAC 
| collect addtime=true index=dashboard_summarize

Data is getting into the index, but none of the fields that have been 'EVAL'ed at some stage. So in this example, LastQuery, forward and MAC are NULL in the summary index. If I take the stats stanza out, the data is collected. It seems odd/bug-like to me that STATS would some how null out EVAL'ed fields.

I've taken the stats stanza out and am moving on with my life. I'll generate the stats in a separate query, but again, I'm puzzled why they'd be NULL in the summary. Are there other cases like this that I need to watch out for?

Cheers, Rich

[gkanapathy · Accepted solution]

It is not a bug, but is how stats is intended to work. The only things to come out after stats are the actual stats you ask for, aggregated by the by fields, so it will be exactly those fields. stats does not keep any other values because it is not otherwise told how to aggregate them (i.e., what to do with multiple values per by field combination). I'm guessing what you need is just ... | stats first(MAC) as MAC ... but whether that's correct or not depends on your data.

[richard_whiffen]

Yeah, I found my mistake by adding and removing stanzas and then talking with Chris Olson from Splunk. In hind sight its obvious but didn't click when I was looking at it the last two days. I think what tripped me up the most is that 'LastQuery' wasn't going through and it was in the stats command, but was in the stats command (but as a count not the actual value DOH!). Anyway, Chris set me on the same path. Thanks gkanapathy