I have two types of records - a START record and a STOP record. I want to be able to change the timestamp based on what type of record it is. Below is a sample START and STOP record.
Mar 15 16:48:32 tailer.pl: START|den01gsx03|28340923|03/15/2013|21:48:15.0|||6509|16|VoIP
Mar 15 16:50:22 tailer.pl: STOP|den01gsx03||03/15/2013|21:48:44.0|03/15/2013|21:49:49.9|6509|16|VoIP
For the START and STOP records, I want the timestamp to be the bolded date/times, and then I want to perform some field transactions that are the same for each record type. I've read a lot of posts on how to do this but I'm still struggling. My input.confs looks like:
[udp://514]
connection_host = ip
sourcetype = test
My props.conf looks like:
[test]
TRANSFORMS-set_sourcetype = set_start, set_stop
[start]
TIME_PREFIX = tailer.pl:\sSTART
MAX_TIMESTAMP_LOOKAHEAD = 150
TZ = UTC
EXTRACT-call_start = ^(?:[^\|]*\|){3}(?P<Call_Start>\d{2}\/\d{2}\/\d{4}\|\d{2}:\d{2}:\d{2})
EXTRACT-call_end = ^(?:[^\|]*\|){5}(?P<Call_End>\d{2}\/\d{2}\/\d{4}\|\d{2}:\d{2}:\d{2})
REPORT-parsefields = Parse
[stop]
TIME_PREFIX=(\d{1,2}\/\d{1,2}\/\d{4}\|\d{1,2}:\d{1,2}:\d{1,2}\.\d)
MAX_TIMESTAMP_LOOKAHEAD = 150
TZ = UTC
EXTRACT-call_start = ^(?:[^\|]*\|){3}(?P<Call_Start>\d{2}\/\d{2}\/\d{4}\|\d{2}:\d{2}:\d{2})
EXTRACT-call_end = ^(?:[^\|]*\|){5}(?P<Call_End>\d{2}\/\d{2}\/\d{4}\|\d{2}:\d{2}:\d{2})
REPORT-parsefields = Parse
The pertinent part of my transforms.conf looks like:
[set_start]
REGEX = tailer.pl:\sSTART
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::start
[set_stop]
REGEX = tailer.pl:\sSTOP
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::stop
However, this doesn't work at all and I've restarted Splunk several times. The timestamp is always the date/time at the beginning of the record. It's driving me crazy because I really need this work. What am I doing wrong?
Have you tried assigning the timestamp in the [test] stanza?
TIME_PREFIX = tailer.pl:\s(START\|([^\|]*\|){2}|STOP\|([^\|]*\|){4})
TIME_FORMAT= %m/%d/%Y|%H:%M:%S.%1N
When the data arrives, it has sourcetype test. Because this is the sourcetype branded on the data as it arrives over the wire, this sourcetype must define the rules for time parsing and line breaking. (These are the rules that are in play when time parsing and line breaking are done.)
Setting TIME_FORMAT, TZ or MAX_TIMESTAMP_LOOKAHEAD in separate sourcetype stanzas will not work; these rules apply at parse time, and the data has already been parsed. The answer given by Paolo Prigione on this same question is the correct one. These format rules have to go in the [test] sourcetype.
Note that it's OK to set EXTRACT and REPORT rules for the [start] and [stop] stanzas, as those apply at search time, and when the data is at rest in the index, it will have these new types, based on the set_start and set_stop rules.
Have you tried assigning the timestamp in the [test] stanza?
TIME_PREFIX = tailer.pl:\s(START\|([^\|]*\|){2}|STOP\|([^\|]*\|){4})
TIME_FORMAT= %m/%d/%Y|%H:%M:%S.%1N
This worked perfectly. Thanks to everyone for all the great assistance.
Hi, have you tried to look at the TIME_FORMAT
directive in props.conf
? That let's you specify HOW the timestamp is formatted, in case Splunk does not understand it.
I would probably keep the original timestamps, and try to make a field extraction to create a new field instead of trying to modify _time
. I haven't looked to closely at your regexes for the call_start
and call_end
, but assuming that they work, you should be able to use them. With eval
's strftime/strptime
functions you can make them into epoch, which is what splunk uses internally - and is very suitable for duration calculations.
Whether you need to set them as different sourcetypes is beyond me, but you may have your reasons.
/K