Might be related to https://answers.splunk.com/answers/168995/how-to-write-regex-to-identify-and-use-time-field.html
I have some RADIUS accounting logs that i send into splunk and this is my current props.conf
[radius_acct_detail]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = ^\w{3} \w{3} \d\d \d\d:\d\d:\d\d \d\d\d\d$
TIME_PREFIX = Event-Timestamp = "
TIME_FORMAT = %b %d %Y %H:%M:%S %z
MAX_TIMESTAMP_LOOKAHEAD = -1
Now most but not all radius accounting events have the Event-Timestamp field in them, for example:
Thu Jun 30 04:24:27 2016
Acct-Status-Type = Accounting-Off
Acct-Authentic = RADIUS
User-Name = "XX-XX-XX-XX-XX-XX"
NAS-IP-Address = 172.20.dd.ddd
NAS-Identifier = "XX-XX-XX-XX-XX-XX"
Called-Station-Id = "XX-XX-XX-XX-XX-XX:ZOLspot"
Ruckus-SSID = "ZOLspot"
Ruckus-Attr-14 = 0xXXXXXXXXXXXX
Ruckus-WlanID = 362
Ruckus-Attr-9 = 0xXXXXXXXX
Ruckus-SCG-CBlade-IP = XXXXXXXXXX
Ruckus-SCG-DBlade-IP = XXXXXXXXX
Proxy-State = 0x31
Acct-Unique-Session-Id = "6716f5d57c72963e"
Realm = "NULL"
Timestamp = 1467253467
as a result i get various regular parsing timestamp errors in the splunk log that i'd like to clean up. ideally what i'd want to happen in this case is for it to fall back to the Timestamp field (epoch time) which is present in every event or the first line of the event which is the time the packet is logged by the radius server (these are the same in the case of these packets but the Event-Timestamp field can and does in cases differ from this timestamp
reading the other answer it seems i could do some greedy regex to capture the first timestamp field if the Event-Timestamp field is not present but the format of the field is different:
Event-Timestamp = "Jun 30 2016 08:29:11 CAT"
specifically day is omitted and TZ is added and Year is in a different spot. anyone have any thoughts on some regex foo that might work or should we somehow look down the custom datetime.xml route?
So i ended up cheating on this with something thats probably not the best approach but works for me
Rather than try to solve the problem in splunk i looked at solving this at the source, and turns out with a little bit of freeradius logic i can solve this.
I defined the following function that i call within the accounting section of freeradius
append_timestamp_to_acct {
if (Acct-Status-Type == "Accounting-On" || Acct-Status-Type == "Accounting-Off") {
if (!Event-Timestamp) {
update request {
Event-Timestamp = "%l"
}
}
}
}
what this function does is set the Event-Timestamp attribute if its not present and this is an Accounting-On or Accounting-Off request
it sets it to the current timestamp (which happens to be the correct format for the Event-Timestamp attribute to start with) which then gets matched in my event in splunk - %l is a freeradius runtime variable for timestamp - http://wiki.freeradius.org/config/run_time_variables
I probbaly don't need the acct-status-type as the check if Event-Timestamp is not present (or empty string) would probably suffice on its own, just being explicitly careful as these are the only packets i've found where the attribute is not set so far
As far as preferential field selection for time, see this:
https://answers.splunk.com/answers/168995/how-to-write-regex-to-identify-and-use-time-field.html
When dealing with multiple time formats (not positions), you really have no choice but to use datetime.xml
:
http://www.function1.com/2013/01/oh-no-splunking-log-files-with-multiple-formats-no-problem
The question is, does custom datetime.xml accept first rule match? the default timestamp exists for every entry but if the Event-Timestamp exists i want to prefer it over the default timestamp
So i ended up cheating on this with something thats probably not the best approach but works for me
Rather than try to solve the problem in splunk i looked at solving this at the source, and turns out with a little bit of freeradius logic i can solve this.
I defined the following function that i call within the accounting section of freeradius
append_timestamp_to_acct {
if (Acct-Status-Type == "Accounting-On" || Acct-Status-Type == "Accounting-Off") {
if (!Event-Timestamp) {
update request {
Event-Timestamp = "%l"
}
}
}
}
what this function does is set the Event-Timestamp attribute if its not present and this is an Accounting-On or Accounting-Off request
it sets it to the current timestamp (which happens to be the correct format for the Event-Timestamp attribute to start with) which then gets matched in my event in splunk - %l is a freeradius runtime variable for timestamp - http://wiki.freeradius.org/config/run_time_variables
I probbaly don't need the acct-status-type as the check if Event-Timestamp is not present (or empty string) would probably suffice on its own, just being explicitly careful as these are the only packets i've found where the attribute is not set so far