Syslog over TCP-TLS not linebreaking correctly

[BryanBerry]

Hey guys,

I've setup our Linux hosts to send syslog using rsyslog over TCP encrypted with TLS. Data's being consumed, but the linebreaks aren't working. I have assigned syslog as the sourcetype for the input. I'd really like to avoid altering the syslog sourcetype, but it's fine if I have to. Any idea what might be causing this behavior and how to fix it?

My inputs looks like this:

[tcp-ssl:8514]
sourcetype=syslog

Current event:

84 <86>Sep 26 11:45:40 *censored* sshd[8218]: Invalid user bryan.berry from *censored* <86>Sep 26 11:45:40 *censored* sshd[8219]: input_userauth_request: invalid user bryan.berry86 <84>Sep 26 11:45:44 *censored* sshd[8218]: pam_unix(sshd:auth): check pass; user unknown143 <85>Sep 26 11:45:44 *censored* sshd[8218]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=*censored* <82>Sep 26 11:45:44 *censored* sshd[8218]: pam_succeed_if(sshd:auth): error retrieving information about user bryan.berry120 <86>Sep 26 11:45:47 *censored* sshd[8218]: Failed password for invalid user bryan.berry from *censored* port 57820 ssh275 <86>Sep 26 11:45:50 *censored* sshd[8219]: Connection closed by *censored*

Should be:

Sep 26 11:45:40 *censored* sshd[8218]: Invalid user bryan.berry from *censored*
Sep 26 11:45:40 *censored* sshd[8219]: input_userauth_request: invalid user bryan.berry
Sep 26 11:45:44 *censored* sshd[8218]: pam_unix(sshd:auth): check pass; user unknown
Sep 26 11:45:44 *censored* sshd[8218]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=*censored*
Sep 26 11:45:44 *censored* sshd[8218]: pam_succeed_if(sshd:auth): error retrieving information about user bryan.berry
Sep 26 11:45:47 *censored* sshd[8218]: Failed password for invalid user bryan.berry from *censored* port 57820 ssh
Sep 26 11:45:50 *censored* sshd[8219]: Connection closed by *censored*

Thanks!

[Ayn · Accepted solution]

Improper line breaking is very often due to improper timestamp parsing, and it seems likely this is the case here as well. Which transport method was used to get the data into Splunk doesn't really matter. Your log events do not have any year specified. Splunk has a collection of default time formats it tries to use for parsing event timestamps, but these default formats expect a year to be present. Have a look at the TIME_FORMAT in props.conf, which you can use to define a time format that Splunk can use for parsing the timestamps properly. By default Splunk will break and create a new event whenever it encounters a valid timestamp on a new line, so if you get your timestamp parsing setup correctly you will also get correct line breaking.

[kristian_kolb]

Ayn is right, but you might also have to set TIME_PREFIX (to skip the priority/facility stuff within angle brackets) and MAX_TIMESTAMP_LOOKHEAD to make sure splunk does not try find the missing year in the pid or some other numeric part of the event.

That being said, the fact that you have the sourcetype set to 'syslog' could mean that Splunk expects a certain formatting of the timestamp.

Also, I seem to recall seeing strange stuff with rsyslog/tcp in certain scenarios.

Sorry.

[BryanBerry]

Jumped the gun on the selected answer...
The timeformat for syslog matches: %b %d %H:%M:%S

Any other ideas?