Why do I have events missing from search results depending on the time range?

[tomburnell]

I have a search that is returning 27 events within a 10 minute window. If I increase the window to 40 minutes, pulling in 30 minutes before the original timezone, I get 54 events, but some of the original events have gone missing.

This is the search:

ns="article.load"  | where geoCity=="LONDON" | transaction txid maxspan=30m maxpause=30m | eventstats perc95(duration) as perc | where duration < perc

The UI doesn't seem to suggest I'm hitting any limits and the numbers of events are very low.

Why might this be happening?

Thanks, Tom

[Richfez]

Some thoughts...

It could be that your snapping is confusing the issue (e.g. it 'snapping' to minute or other time boundaries). Could you try

ns="article.load"  earliest=-40m@m latest=-15m@m | where geoCity=="LONDON" | transaction txid maxspan=30m maxpause=30m | eventstats perc95(duration) as perc | where duration < perc

Then vary earliest and latest and compare results.
earliest=-35m@m latest=-10m@m
earliest=-30m@m latest=-5m@m
etc... As long as you run them within a minute or so, they should all return reasonable results - check the timeline to see how they fall and if they're fairly consistent.

Another thing that may be happening is that 40 minutes is longer than your maxspan and maxpause, whereas 10 minutes is shorter. This can subtly (and not so subtly) alter results. If you always run all searches older than 30 minutes, you should always get consistent results so try checking various timeframes older than 30 minutes.

Note, you should be able to skip | where geoCity=="LONDON" | ... because you should be able to

ns="article.load"  earliest=-2h@h latest=-1h@h geoCity=="LONDON" | transaction txid maxspan=30m maxpause=30m | eventstats perc95(duration) as perc | where duration < perc

Probably, anyway. 🙂

[tomburnell]

ok, doh, problem here is the query that is removing based on duration being lower than perc95.