"Gaps" in Timechart

[twkan]

Hello everybody,

I'm trying to do a timechart using a 3 day timeframe, for example from Jul 17 2011 00:00:00 to Jul 20 2011 00:00:00. The search querty is simple, and it is:

 sourcetype=access_combined | timechart count by uri useother=f usenull=f

The timechart will start rendering in reverse chronological order, which is the normal behavior. During the rendering of the timechart, there seems to be some "buffer" limit which prevents the entire graph to be rendered, and you would actually see 'gaps' in-between where the graph is supposed to be.

If I change the timeframe to e.g. 1 week, the "gaps" would actually expand accordingly as well. I've tried playing around with span and bins as well, but it doesn't seem to help in this case.

If I set the timeframe to 1 day, everything works well. I know Summary Indexing may help me to get around this, but the question is really to shed some light on this.

This was tested on 4.2.1, 4.2.2 and 4.2.3 under Linux 64-bit.

Thanks for any suggestions.

link text

[eamuncal]

Hi:

We resolve this by doing two things.

1. Using a bigger span(span=1h for a 24H timerange). No set of rules here. Just playing with the span to lessen the gap.
2. Adjusting values in limits.conf. 50000 were changed to 50,000,000, for example. As per steveyz suggestion.

Thanks twkan for the post.

Good day!

[steveyz · Accepted solution]

how many distinct uri's are you expecting to get in those 3 days?

try increasing in limits.conf

[stats]
maxresultrows = 50000

try upping it to 500,000 and see what happens.

[sloshburch]

I'm seeing the same issue but using (over last 24 hours):
timechart span=30min avg(field1) avg(field2).

Just like in this case, driving into the gap periods I see the data does exist. Updating the limits has not fixed it.

Anyone have any other ideas?

[twkan]

Hello Steve,

Thanks, I wil pulling about 420,000 events for the past 3 days and increasing the limits to 500,000 does help to solve the issue.

[twkan]

Thanks Nick for the help. Yes, it did occur that there could be a possibility the values of the "uri" are not in the Top 10, and hence we are getting 0 results for it.

I have tried to omit out usenull=f and useother=f and re-ran the search, and you can see that the gap still persists.

Enlarge Image

Enlarge Image

In my sample data set, the gap exists from 17 Jul 12:00:00 till 18 Jul 11:00:00. I have tried to create another search with that specific timeframe, and the results can be seen below with no gaps in between.

So the issue seems to compound itself when we tried to timechart thru a longer timeframe for some reasons.

[sideview]

I'm not sure if this is causing it, but 'useother' and 'usenull' can lead to problems if you dont understand what they do. Setting those to 'f' will just omit NULL and OTHER from the output. However if a given time bucket has ONLY null values or only 'other' values, then you'll just get no data at all during those buckets. In other words if those timebuckets have relatively low volume, and they only contain values of uri that are not in the top 10 overall, then you'd get exactly what you're seeing here.

If you havent already tried it, I would try it again without those useother/usenull arguments, or generally look for something distinctive about the data in the badly-behaving time buckets.

Also, be aware that during the 'preview' stage, the granularity of the timechart will change several times. It's maybe possible that what you saw was just the buckets of a lower-level granularity.

But if it's neither of those, then I would send it into support@splunk.com so they can help you diagnose the system more deeply.

[twkan]

Hmm, interesting. If i tried to use ...timechart count by clientip, or count by status, or count by eventtype the graphs are totally alright with no gaps. So it seems like the field uri and perhaps others (haven't had the chance to try all of them yet) is causing the issue...