I am running a query to alert me if the sum of a particular property < 400000. I get alert most times saying the count < 400000. I go and run the query manually for that specific time describred in alert email.. I see that the count is well over 400000. What am I missing here?
index=app host="aws-service-ip*" sourcetype="aws-fht-service-transactions" SVCPushCounter earliest="08/07/2019:19:00:00" latest="08/07/2019:19:30:00"| stats sum(SVCPushCounter) as totalBySvc by Partner| where (Partner=CVS AND totalBySvc<4000000) )
You may be getting events arriving later into the instance, what could explain the variation in numbers you;re seeing
Check the _indextime field in those events to verify that