UDP Syslog input, current timestamp changes to a year in the past.

[cbdick]

We use splunk with a single UDP syslog input.

Between July 13 and 14, we have found that after a certain set of events, the current timestamp gets set to a year in the past until splunkd is restarted.

From the show source, this event gets logged as 3 seperate events. These events get indexed with the proper current timestamp (ie: 2011/07/14 04:43:53).

Jul 14 04:43:53 4.3.2.1 Jul 14 04:38:16 hostname CSCOacs_TACACS_Accounting 0010775426 2 0 2011-07-14 04:38:16.930 -08:00 0032807669 3301 NOTICE Tacacs-Accounting: TACACS+ Accounting START, ACSVersion=acs-5.1.0.44-B.2347, ConfigVersionId=4, Device IP Address=1.2.3.4, RequestLatency=19, NetworkDeviceName=LDBRR110_2911, Type=Accounting, Privilege-Level=1, Service=None, Authen-Method=NotSet, AVPair=task_id=1, AVPair=timezone=PST, AVPair=event=sys_acct, AVPair=reason=reload, AVPair=reload-reason=power-on, AVPair=ios-version=Cisco IOS Software\, C2900 Software (C2900-UNIVERSALK9-M)\, Version 15.0(1)M3\, RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2010 by Cisco Systems\, Inc.

This next event and all subsequent events get logged a year in the past (ie: 2010/07/14 04:43:53... NOTE: 2010 INSTEAD OF 2011) until splunkd is restarted. This only started happening between the 13th and 14th of July and timestamps worked fine previous to that.

Jul 14 04:43:53 4.2.3.1 Jul 14 04:38:16 hostname CSCOacs_TACACS_Accounting 0010775426 2 1 NetworkDeviceGroups=Location:All Locations:Store ISRs, Response={Type=Accounting; AcctReply-Status=Success; }

It was my understanding that UDP syslog inputs should default to DATETIME_CONFIG=CURRENT and the date should not be parsed from the event. I can't understand what is causing splunk to set the current index timestamp to a year in the past. What can be added to props.conf (or other) to work around this issue.

I am also interested in what needs to be added to transforms.conf to group the initial event into a single event instead of indexing as 3 separate events.

Please be gentle, I am fairly new to splunk and this will be my first foray into tweaking the props and transforms etc....

[cbdick · Accepted solution]

Turns out UDP syslog does not default to DATETIME_CONFIG=CURRENT and the multiline syslog input that contained a year in one of the lines that did not contain a timestamp was causing issues. Splunk detected this year and reset the index year to 2010 (from 2011) so all subsequent inputs were indexed a year in the past. Restarting splunkd reset the year back to CURRENT, until the multiline input was encountered again....

The solution was an \etc\system\local\props.conf entry for the host exhibiting the multiline syslog input:

[host::4.3.2.1]
LINE_BREAKER = ([\r\n]+(?=\w{3}\s+\d{1,2}\s+\d{2}:\d{2}:\d{2}))

This told splunk not to process the multiline input as multiple events but instead to merge them together until the next properly formatted date was detected.

What we don't understand is why is started happening out of the blue... or why the issue did not resolve on subsequent multiline syslog inputs that contained 2011 in their non-timestamped lines. Sounds like we may never know and have decided to just fix the issue and move on.