Getting Data In

JSON timestamps not parsed via HTTP Event Collector

bradserbuddy
Engager

Here are some of the values I am using for my JSON source type:

MAX_TIMESTAMP_LOOKAHEAD = 1000 (as we have long JSON input)
TIME_FORMAT = %FT%T.%3Q
TIME_PREFIX = Timestamp\"\s:\s\"

I've successfully imported the JSON from a file with the above source type values, but, for some reason, when coming in through my HTTP Event Collector, the timestamp isn't picked up (that is, _time is not set to the timestamp).

I've restarted the server, tried different values for TIME_PREFIX (for instance not encoding the quotes, and dropping the \s regex) and TIME_FORMAT (for instance %Y-%m-%dT%H:%M:%S), and removed the KV_MODE=json to no avail.

Am I misunderstanding the relationship between timestamp parsing and _time? Is there something else I need to do to get my source type to work with my HTTP Event Collector? Are there additional troubleshooting steps/tools I can use to help track down what's going on?

Thanks,
Brad

1 Solution

gblock_splunk
Splunk Employee
Splunk Employee

Our JSON endpoint for HTTP Event Collector does not do timestamp extraction. Our JSON endpoint expects events sent using a lightweight structure, our JSON Event Protocol. The plus side is your "event" payload can be any JSON (or not) / can have whitespace, newlines, etc and it will still come as a single event. For example you could also have a Java stack trace and it will easily go in as a single event. For the timestamp, you have to specify "time" explicitly outside the payload in the event envelope in epoch format i.e.

{
"time": 1426279439,
"host": "localhost",
"source": "datasource",
"sourcetype": "txt",
"index": "main",
"event": { "hello": "world" }
}

In Splunk Cloud / Splunk 6.4 you have a different option, our new raw endpoint. With raw you can send an arbitrary payload to HEC (/services/collector/raw) and we will honor breaking rules and do timestamp extraction. It should I believe meet your need.

View solution in original post

gblock_splunk
Splunk Employee
Splunk Employee

Our JSON endpoint for HTTP Event Collector does not do timestamp extraction. Our JSON endpoint expects events sent using a lightweight structure, our JSON Event Protocol. The plus side is your "event" payload can be any JSON (or not) / can have whitespace, newlines, etc and it will still come as a single event. For example you could also have a Java stack trace and it will easily go in as a single event. For the timestamp, you have to specify "time" explicitly outside the payload in the event envelope in epoch format i.e.

{
"time": 1426279439,
"host": "localhost",
"source": "datasource",
"sourcetype": "txt",
"index": "main",
"event": { "hello": "world" }
}

In Splunk Cloud / Splunk 6.4 you have a different option, our new raw endpoint. With raw you can send an arbitrary payload to HEC (/services/collector/raw) and we will honor breaking rules and do timestamp extraction. It should I believe meet your need.

bradserbuddy
Engager

I switched to using the Splunk-provided generic_single_line source type, and that doesn't work either. I am testing by POSTing to the collector, which has no TIME_PREFIX, and only posting the timestamp itself. It appears that I can only set the time via the "time" property on the enclosing JSON itself as detailed here: http://dev.splunk.com/view/SP-CAAAE6P. Is that true?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...