Why is each line of syslog prepended with a number in angle brackets like <123>?

[stefanlasiewski]

Our central syslog server forwards syslog data to my Splunk server, using TCP (secure syslog).

In the Splunk web GUI, each line is prepended with a number contained in angle brackets (<>), like this:

<166>Feb 28 17:52:31 192.168.22.192 httpd[27331]: www.example.org 192.168.33.18 - - [28/Feb/2012:17:52:30 -0800] "GET /somepage HTTP/1.0" 200 6838 "-" "gsa-crawler"
<132>Feb 28 17:52:31 192.168.33.182 slapd[12312] LDAP message
<38>Feb 28 17:52:31 192.168.72.67 DROP packet on firewall X

What do the numbers in angle brackets mean? Are they related to syslog facilities and priorities?

This syslog data is also forwarded to another syslog server which does not run Splunk.

[dshpritz]

Here is an example SEDCMD config that I am using. This is in a props.conf:

[source::tcp:514]
SEDCMD-strip-tcp-priority=s/^<\d+>//

HTH,
Dave

[stefanlasiewski]

I only have one Splunk server. I'll have to play with this some more. I made sure the port matched up with what I have for the input source.

[stefanlasiewski]

I'm not sure how I managed to fix this, but this works for me now. I have tried both of the following patterns:

dshpritz's solution:

SEDCMD-tcp-header-stripper-priority=s/^<\d+>//

And the same solution, but with a sed capture group, which seems to be a bit more explicit:

SEDCMD-tcp-header-stripper-priority=s/^<\d+>(.*)/\1/

[dshpritz]

This should be in the props.conf. You will need to make sure the stanza matches your input (so if the port is different you would have to change it). What is the value for the "source" fields for the events?
Also, make sure that the setting is configured on the systems doing the initial parsing, so Indexers or Heavy Forwarders. This is an index-time transform, so the first parsing system needs to run the SEDCMD.

[stefanlasiewski]

Thanks. Doesn't seem to be working for me. Do I need to add the SEDCMD anywhere else, or make any changes to inputs.conf?

[dwaddle · Accepted solution]

This is the syslog priority / facility data. Syslog sources send this along with messages so the receiving syslog server has some context about the event. For UDP syslog, Splunk provides an option for priority stripping no_priority_stripping=[true|false] in inputs.conf. Based on the docs this isn't supported for TCP inputs. Regardless, I would give it a try - it may work anyway. But if not, don't be surprised ... afterall the docs didn't say it would.

If it doesn't work, you have some options

1. File an enhancement request asking for the option to work with TCP inputs
2. Use a SEDCMD to filter it out
3. Drop a forwarder on your central syslog server and use Splunk forwarding instead of syslog over TCP (This would be my personal preference)

[stefanlasiewski]

Thanks for the answer. "no_priority_stripping" does not work for TCP inputs. Splunk prints an error like "Possible typo in stanza [tcp://601] in /opt/splunk/etc/system/local/inputs.conf, line 11: no_priority_stripping = true". Bummer.

[stefanlasiewski]

Subject should say 'number like <123>', but the website keeps munging that text. parse-fail.