Getting Data In

Changing timestamp and sourcetype based on record type

tnkoehn
Path Finder

I have two types of records - a START record and a STOP record. I want to be able to change the timestamp based on what type of record it is. Below is a sample START and STOP record.

Mar 15 16:48:32 tailer.pl: START|den01gsx03|28340923|03/15/2013|21:48:15.0|||6509|16|VoIP

Mar 15 16:50:22 tailer.pl: STOP|den01gsx03||03/15/2013|21:48:44.0|03/15/2013|21:49:49.9|6509|16|VoIP

For the START and STOP records, I want the timestamp to be the bolded date/times, and then I want to perform some field transactions that are the same for each record type. I've read a lot of posts on how to do this but I'm still struggling. My input.confs looks like:

[udp://514]
connection_host = ip
sourcetype = test

My props.conf looks like:

[test]
TRANSFORMS-set_sourcetype = set_start, set_stop

[start]
TIME_PREFIX = tailer.pl:\sSTART
MAX_TIMESTAMP_LOOKAHEAD = 150
TZ = UTC
EXTRACT-call_start = ^(?:[^\|]*\|){3}(?P<Call_Start>\d{2}\/\d{2}\/\d{4}\|\d{2}:\d{2}:\d{2})
EXTRACT-call_end = ^(?:[^\|]*\|){5}(?P<Call_End>\d{2}\/\d{2}\/\d{4}\|\d{2}:\d{2}:\d{2})
REPORT-parsefields = Parse


[stop]
TIME_PREFIX=(\d{1,2}\/\d{1,2}\/\d{4}\|\d{1,2}:\d{1,2}:\d{1,2}\.\d)
MAX_TIMESTAMP_LOOKAHEAD = 150
TZ = UTC
EXTRACT-call_start = ^(?:[^\|]*\|){3}(?P<Call_Start>\d{2}\/\d{2}\/\d{4}\|\d{2}:\d{2}:\d{2})
EXTRACT-call_end = ^(?:[^\|]*\|){5}(?P<Call_End>\d{2}\/\d{2}\/\d{4}\|\d{2}:\d{2}:\d{2})
REPORT-parsefields = Parse

The pertinent part of my transforms.conf looks like:

[set_start]
REGEX = tailer.pl:\sSTART
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::start

[set_stop]
REGEX = tailer.pl:\sSTOP
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::stop

However, this doesn't work at all and I've restarted Splunk several times. The timestamp is always the date/time at the beginning of the record. It's driving me crazy because I really need this work. What am I doing wrong?

Tags (1)
0 Karma
1 Solution

Paolo_Prigione
Builder

Have you tried assigning the timestamp in the [test] stanza?

TIME_PREFIX = tailer.pl:\s(START\|([^\|]*\|){2}|STOP\|([^\|]*\|){4})
TIME_FORMAT= %m/%d/%Y|%H:%M:%S.%1N

View solution in original post

sowings
Splunk Employee
Splunk Employee

When the data arrives, it has sourcetype test. Because this is the sourcetype branded on the data as it arrives over the wire, this sourcetype must define the rules for time parsing and line breaking. (These are the rules that are in play when time parsing and line breaking are done.)

Setting TIME_FORMAT, TZ or MAX_TIMESTAMP_LOOKAHEAD in separate sourcetype stanzas will not work; these rules apply at parse time, and the data has already been parsed. The answer given by Paolo Prigione on this same question is the correct one. These format rules have to go in the [test] sourcetype.

Note that it's OK to set EXTRACT and REPORT rules for the [start] and [stop] stanzas, as those apply at search time, and when the data is at rest in the index, it will have these new types, based on the set_start and set_stop rules.

0 Karma

Paolo_Prigione
Builder

Have you tried assigning the timestamp in the [test] stanza?

TIME_PREFIX = tailer.pl:\s(START\|([^\|]*\|){2}|STOP\|([^\|]*\|){4})
TIME_FORMAT= %m/%d/%Y|%H:%M:%S.%1N

tnkoehn
Path Finder

This worked perfectly. Thanks to everyone for all the great assistance.

0 Karma

kristian_kolb
Ultra Champion

Hi, have you tried to look at the TIME_FORMAT directive in props.conf? That let's you specify HOW the timestamp is formatted, in case Splunk does not understand it.

I would probably keep the original timestamps, and try to make a field extraction to create a new field instead of trying to modify _time. I haven't looked to closely at your regexes for the call_start and call_end, but assuming that they work, you should be able to use them. With eval's strftime/strptime functions you can make them into epoch, which is what splunk uses internally - and is very suitable for duration calculations.

Whether you need to set them as different sourcetypes is beyond me, but you may have your reasons.

/K

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...