My events have two different times in them, one from when the dns server processed them, and then another is added to the beginning of the events from what I assume is splunk. I want splunk to treat the second time as the events timestamp. I have manually assigned the sourcetype for the port as dns_data. Here is part of my props.conf file from $SPLUNK_HOME/etc/sys/local:
props.conf--
[sourcetype::dns_data]
TIME_PREFIX = \w+ \d+ \d\d:\d\d:\d\d foo\.bar\.com
TIME_FORMAT = %b %d %H:%M:%S %Y
Here are some events:
Jun 23 16:29:19 foo.bar.com Wed Jun 23 16:27:15 2010: 123.123.123.123 -> 321.321.321.321: 52826 NOERR 'something.somewhere.com.' A IN (n#5) (x#9)
Jun 23 16:29:19 foo.bar.com Wed Jun 23 16:27:15 2010: 123.123.231.321 -> 25.321.321.31: 1572 NOERR 'something.somewhere.com.' AAAA IN (x#1)
Jun 23 16:29:19 foo.bar.com Wed Jun 23 16:27:15 2010: 213.213.21.231 -> 123.123.123.123: 25373 NOERR 'something.somewhere.com.' A IN (a#1) (n#6) (x#11) ANS 'something.somewhere.com. A IN 21.231.231.21
Any help would be appreciated.
Thanks.
I you didn't have a TIME_PREFIX
I think you would be ok (because you have %b
in your time format, splunk shouldn't match your first date timestamp because it doesn't have a day of the week shown). That said, I using TIME_PREFIX
is better.
I see a couple minor but potentially problem-causing issues with your current config:
sourcetype::
prefix is allowed, as suggested by Siegfried.TIME_PREFIX
doesn't allow for a trailing space between the host name and the date that you want to use.TIME_PREFIX
doesn't allow for a single-digit day (I'm assuming this is a syslog file, in which case the standard format for July 4 would be "Jul__4 2010" (had to use an underscore to get the formatting to show up on this site.) Your current regex only allows for a single space.^
(start of line) in your TIME_PREFIX
to let the regex engine do less work.Here is what I would suggest you put in your props.conf
file:
[dns_data]
TIME_PREFIX = ^\w+ .\d \d\d:\d\d:\d\d \S+\s+
TIME_FORMAT = %b %d %H:%M:%S %Y
If you want, you can also use a SEDCMD entry to drop that first date out of there entirely so you consume less bytes in your index. I have some log sources that do this and it makes it easily to review the logs when there aren't two sets of dates for your eyes to scan over.
SEDCMD-drop_extra_date = s/^\w+ .\d \d\d:\d\d:\d\d (.*)$/\1/
Keep in mind the timestamp extraction happens first, so you don't have to worry about your SEDCMD getting in the way of your TIME_PREFIX
.
You could also move the second date (the one your are keeping) to the front if you want to. (To be more consitent with your other syslog mesages.
SEDCMD-use_2nd_date = s/^\w+ .\d \d\d:\d\d:\d\d (\S+) (\w+ \w+ .\d \d\d:\d\d:\d\d \d{4})(.*)$/\2 \1 \3/
(I haven't tested these SEDCMD
expressions, so they may not work right away. They are mostly just to give you and idea of what you can do.)
Still not working huh? Here are a few more things to double check:
dns_data
on your indexed events and not syslog
or something like that. (You aren't using any sourcetype renaming transformers are you? If you don't know what I'm talking about then you probably are not.)$SPLUNK_HOME/etc/system/local/props.conf
(You noted "sys" instead of "system" in your original question, but I'm assuming that was just a shortcut notation which I get, but splunk would not.)See this link for more props debugging techniques
I you didn't have a TIME_PREFIX
I think you would be ok (because you have %b
in your time format, splunk shouldn't match your first date timestamp because it doesn't have a day of the week shown). That said, I using TIME_PREFIX
is better.
I see a couple minor but potentially problem-causing issues with your current config:
sourcetype::
prefix is allowed, as suggested by Siegfried.TIME_PREFIX
doesn't allow for a trailing space between the host name and the date that you want to use.TIME_PREFIX
doesn't allow for a single-digit day (I'm assuming this is a syslog file, in which case the standard format for July 4 would be "Jul__4 2010" (had to use an underscore to get the formatting to show up on this site.) Your current regex only allows for a single space.^
(start of line) in your TIME_PREFIX
to let the regex engine do less work.Here is what I would suggest you put in your props.conf
file:
[dns_data]
TIME_PREFIX = ^\w+ .\d \d\d:\d\d:\d\d \S+\s+
TIME_FORMAT = %b %d %H:%M:%S %Y
If you want, you can also use a SEDCMD entry to drop that first date out of there entirely so you consume less bytes in your index. I have some log sources that do this and it makes it easily to review the logs when there aren't two sets of dates for your eyes to scan over.
SEDCMD-drop_extra_date = s/^\w+ .\d \d\d:\d\d:\d\d (.*)$/\1/
Keep in mind the timestamp extraction happens first, so you don't have to worry about your SEDCMD getting in the way of your TIME_PREFIX
.
You could also move the second date (the one your are keeping) to the front if you want to. (To be more consitent with your other syslog mesages.
SEDCMD-use_2nd_date = s/^\w+ .\d \d\d:\d\d:\d\d (\S+) (\w+ \w+ .\d \d\d:\d\d:\d\d \d{4})(.*)$/\2 \1 \3/
(I haven't tested these SEDCMD
expressions, so they may not work right away. They are mostly just to give you and idea of what you can do.)
Still not working huh? Here are a few more things to double check:
dns_data
on your indexed events and not syslog
or something like that. (You aren't using any sourcetype renaming transformers are you? If you don't know what I'm talking about then you probably are not.)$SPLUNK_HOME/etc/system/local/props.conf
(You noted "sys" instead of "system" in your original question, but I'm assuming that was just a shortcut notation which I get, but splunk would not.)See this link for more props debugging techniques
Note that Lowell's TIME_PREFIX skips over the day of the week, and yours doesn't. That's pretty key.
I've added a few more config testing ideas to my answer. Best of luck.
I entered in the suggested code for props.conf and it still does not extract the correct time. I really don't know why it will not take it. It acts like it doesn't even read the [dns_data] stanza. Would there be any other reason why it would not read it or act upon it?
I guess you should use [dns_data]
as your stanza name instead of [sourcetype::dns_data]
. The other part of your config looks quite reasonable to me.