Getting Data In

Custom timestamp detection for a sourcetype with some event w/o timestamp

D2SI
Communicator

Hello there,

For a particular sourcetype there are events with a timestamp and events without timestamp.

As Splunk cannot detect a timestamp in the events without timestamp, it generates plenty of errors saying 'Defaulting to timestamp of previous event'.

I am trying to clean this up.

I see that I can configure a local datetime.xml to take into account various timestamp formats.

What I would like to do is :

for event with a timestamp -> assign the correct timestamp

for events without timestamp -> assign the indexing time as the timestamp and don't throw errors

Is there a way in datetime.xml to say : for this format, apply indextime timestamp?

Thanks in advance for any hint!

0 Karma

jarizeloyola
Path Finder

Another way to solve this is to use INGEST_EVAL in transforms.conf.

I tried this config and works for me.

props.conf
[test]
LINE_BREAKER = ([\r\n]+)(\d+|\w+)
SHOULD_LINEMERGE = false
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3N
MAX_TIMESTAMP_LOOKAHEAD = 30
TRANSFORMS = modtimestamp


transforms.conf

[modtimestamp]
INGEST_EVAL = _time[float]:=if(match(_raw,"^(?!\d+-\d+-\d+).*"),time(),_time)

it filters the events that doesnt start with the year and apply the condition

time() is more expensive and acts on every event separately. now() appears to only be valid in search and is the time of the search itself.

Note: This configuration might be very impactful on the indexer's performance.

0 Karma

adonio
Ultra Champion

imho the workaround i suggested in the comment directly under the question performs better.
and only being used if needed (search time)

0 Karma

jarizeloyola
Path Finder

Depends on his requirements , either at the index or search time. whatever that helps him. 🙂

0 Karma

D2SI
Communicator

Thank both of you for your suggested solutions!

@adonio Good idea, thank you, I will test this with eval in the props.conf!

@jarizeloyola Thanks! I am using ingest_eval for another source which timestamp can only be found in the filename, and it works great! However, I see splunkd timestamp errors for that source, so I have to check if inges_eval works in a way there will still have timestamp issues generated even if OK, or if it is something else!

0 Karma

mloyola_splunk
Splunk Employee
Splunk Employee

I would suggest to use the big 6/8 of props.conf
Your props should contain these "TIME_PREFIX, TIME_FORMAT, MAX_TIMESTAMP_LOOKAHEAD" and
add "DATETIME_CONFIG = current".

DATETIME_CONFIG is when Splunk can't find something to match TIME_* .

D2SI
Communicator

Thanks for your answer.

Your are saying that with TIME_* parameters configured and DATETIME_CONFIG set to current, events with a timestamp will get timestamped according to TIME_* parameters while events from the same sourcetype without timestamp will have the indextime as the timestamp.

To my understanding setting DATETIME_CONFIG to current overrides all the TIME_* parameters and set the indextime as timestamp but I will test this.

0 Karma

adonio
Ultra Champion

would be very interesting, to know the result of your test @D2SI as i had the same understanding you had.
couple links for reference:
https://docs.splunk.com/Documentation/Splunk/8.0.1/Admin/Propsconf#Timestamp_extraction_configuratio...
https://wiki.splunk.com/Community:HowIndexingWorks

0 Karma

D2SI
Communicator

Alright I have tested it and it does not work, I mean if DATETIME_CONFIG is set to CURRENT, it overrides any other TIME_* parameter and set timestamp to indextime.

Thanks anyhow for the suggestion.

adonio
Ultra Champion

probably doesnt answer your question directly, but here is a workaround.
index all events with current timestamp
in search extract the timestamp for the field that contains time, lets say you will call it event_time
now in search (if need to use time other then current), run something like this:
... | eval _time = if(isnull(event_time),_time,event_time) ... or something along those lines
hope it helps a little bit

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...