Splunk Search

Collect bins time?

larryp
Explorer

OK, this is driving me crazy. I have a normal time in _time (displayed as yyyy-mm-dd HH:MM:SS). I collect it into an index without any bin. I search on the index, and the times are all yyyy-mm-dd HH:00:00 or yyyy-mm-dd HH:00:01. In addition, the number of events with the time yyyy-mm-dd HH:00:00 are various multiples of 100,000, as if there's a limitation being reached, but the multiple isn't the same.

I can copy the _time variable to another one, and it will be in the index exactly as I expect (displayed as epoch time with microsecond precision) from the index. But that AFAIK can't be used in, e.g., timechart. (And as this index is for users to run statistics on, copying it back to _time isn't an option.)

As I was researching this I came across a comment about sub-second times possibly creating a need for too much memory, but (a) I can't find it again, and (b) I would have thought that one would lose subsecond precision, not (in my case) hourly.

Any help would be appreciated!

Tags (4)
0 Karma
1 Solution

woodcock
Esteemed Legend

There is a limit on the number of events that can exist at the same second. There is an error that is generated when this is hit.

https://answers.splunk.com/answers/303/whats-max-events-i-can-have-timestamped-with-a-particular-sec...

View solution in original post

0 Karma

larryp
Explorer

Looks like I can't edit the question but only add new information via comment.

Turns out when I had search … | table … | collect … the _time gets collapsed to hourly increments (with rollover to subsequent seconds as @woodcock pointed out from https://answers.splunk.com/answers/303/whats-max-events-i-can-have-timestamped-with-a-particular-sec...). but search … | fields … | collect … keeps _time with millisecond accuracy in the index.

Unfortunately table is only one requirement; stats is also desired.

0 Karma

woodcock
Esteemed Legend

There is a limit on the number of events that can exist at the same second. There is an error that is generated when this is hit.

https://answers.splunk.com/answers/303/whats-max-events-i-can-have-timestamped-with-a-particular-sec...

0 Karma

larryp
Explorer

That's also interesting, but in my case why did Splunk ‘bin’ events into hourly buckets, where 100000 or 200000 events got put in second 0 (HH:00:00) and the rest (>100000) got put in second 1 (HH:00:01)? In fact, I don't want _time reduced to one arbitrary second out of 3600; I'd rather keep the events in the 3600 seconds for each hour they occur in.

I'm going to update the description of the problem with some clarifying information.

0 Karma

larryp
Explorer

OK, I'm accepting this answer. It looks like the secondary effect is created by the ‘earliest’ unit, not the value or the span, nor any bin value (e.g, 5m).

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Here's the apparent answer and workarounds...

https://answers.splunk.com/answers/218702/why-are-collected-events-in-a-summary-index-losing.html

If you do need the millisecond-level accuracy for timecharting, then you can always recalculate _time before the timechart command, either inline in the search or in props.conf the way @krdo suggests in the comments on that question.

0 Karma

larryp
Explorer

That's interesting, but apparently not the cause of my problem - I'm losing minute and second as well as millisecond granularity (where seconds are either 0 or 1). The work-around would only work for milliseconds.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...