Getting Data In

Why is the Host IP value from udp:514 syslog input incorrect for one device?

evgenyv
Explorer

Hi,

I'm collecting syslog events sent by different network equipment. For all devices, the host value is recorded as expected - source IP address of syslog message. However, for one of devices - the host value is "2015".
tcpdump shows the correct source IP. All other fields are extracted as expected.
What can be wrong there?

I have defined the input as following:

./etc/apps/search/local/inputs.conf

[udp://514]
connection_host = ip
sourcetype = syslog
no_priority_stripping = true

./etc/apps/syslog_priority_lookup/default/props.conf

[source::udp:514]
EXTRACT-extract_syslog_priority = ^<(?<syslog_priority>\d+)>
LOOKUP-lookup_syslog_priority = syslog_priority_lookup syslog_priority OUTPUTNEW syslog_facility, syslog_severity
1 Solution

nnmiller
Contributor

First off, separate the notion of the sourcetype syslog from the protocol syslog. Any log event ingested into Splunk can be given the sourcetype syslog provided it conforms to the expected event format.

Data that you configure to have the sourcetype syslog passes through a transformation process, where the hostname is extracted from each event. Splunk extracts the source IP from the log message itself, it does not use the IP that sends the syslog message. Syslog messages may be forwarded from a syslog host that didn't create the log event.

Generally, a syslog event starts with the date and is immediately followed by the host name or IP that created the event. The events you shared that do not have the source host parsed correctly are not formatted as a normal syslog message.

Parsed:

Oct 4 05:29:41 10.20.4.209 CMD_ACCT <...>
Oct 4 05:29:41 10.20.4.209 CMD_ACCT <...>

Not parsed:

Oct  4 05:44:06 2015 HP %%10SHELL/<...>
Oct  4 05:44:06 2015 HP %%10SNMP/<...>

The second message has the year in the place that Splunk is expecting the host IP address. Is this actually a syslog event, or is it a SNMP event?

There are several methods to resolve the issue:

  • Modify the syslog output format on the hosts which are not including the host IP address in their syslog messages.
  • Modify your Splunk inputs.conf per the docs for inputs.conf for UDP/TCP inputs. From inputs.conf doc page:

    connection_host = [ip|dns|none]

    • "ip" sets the host to the IP address of the system sending the data.
    • "dns" sets the host to the reverse DNS entry for IP address of the system sending the data.
    • "none" leaves the host as specified in inputs.conf, typically the splunk system hostname.
    • Defaults to "ip".

Your inputs.conf could be written like:

[udp://514]
connection_host=ip
sourcetype = not_syslog
  • Send the events to a central syslog server rather than a UDP/TCP port on your Splunk server. Best practice when ingesting syslog is to have a syslog process (I recommend syslog-ng) write the logs to disk, then have a Splunk heavy forwarder or universal forwarder monitor those files. Using syslog in this manner gives you a great deal of flexibility. For instance, in this case, you could write the syslog messages out into a directory structure where the source IP is the directory name. Splunk can then use the directory name as the source. Take a look at George Starcher's blog post if you are interested in this method.

View solution in original post

muebel
SplunkTrust
SplunkTrust

It seems that the Splunk syslog processor is mixing up part of the timestamp with the ip of the sending device in this case. I think that we'd need to see example _raw events for the correct and incorrect messages, but I think a better answer would be to switching to a syslog server for this data source.

There are variety of drawbacks to using the network inputs. Check out this page for more info : http://www.georgestarcher.com/splunk-success-with-syslog/

nnmiller
Contributor

First off, separate the notion of the sourcetype syslog from the protocol syslog. Any log event ingested into Splunk can be given the sourcetype syslog provided it conforms to the expected event format.

Data that you configure to have the sourcetype syslog passes through a transformation process, where the hostname is extracted from each event. Splunk extracts the source IP from the log message itself, it does not use the IP that sends the syslog message. Syslog messages may be forwarded from a syslog host that didn't create the log event.

Generally, a syslog event starts with the date and is immediately followed by the host name or IP that created the event. The events you shared that do not have the source host parsed correctly are not formatted as a normal syslog message.

Parsed:

Oct 4 05:29:41 10.20.4.209 CMD_ACCT <...>
Oct 4 05:29:41 10.20.4.209 CMD_ACCT <...>

Not parsed:

Oct  4 05:44:06 2015 HP %%10SHELL/<...>
Oct  4 05:44:06 2015 HP %%10SNMP/<...>

The second message has the year in the place that Splunk is expecting the host IP address. Is this actually a syslog event, or is it a SNMP event?

There are several methods to resolve the issue:

  • Modify the syslog output format on the hosts which are not including the host IP address in their syslog messages.
  • Modify your Splunk inputs.conf per the docs for inputs.conf for UDP/TCP inputs. From inputs.conf doc page:

    connection_host = [ip|dns|none]

    • "ip" sets the host to the IP address of the system sending the data.
    • "dns" sets the host to the reverse DNS entry for IP address of the system sending the data.
    • "none" leaves the host as specified in inputs.conf, typically the splunk system hostname.
    • Defaults to "ip".

Your inputs.conf could be written like:

[udp://514]
connection_host=ip
sourcetype = not_syslog
  • Send the events to a central syslog server rather than a UDP/TCP port on your Splunk server. Best practice when ingesting syslog is to have a syslog process (I recommend syslog-ng) write the logs to disk, then have a Splunk heavy forwarder or universal forwarder monitor those files. Using syslog in this manner gives you a great deal of flexibility. For instance, in this case, you could write the syslog messages out into a directory structure where the source IP is the directory name. Splunk can then use the directory name as the source. Take a look at George Starcher's blog post if you are interested in this method.

Richfez
SplunkTrust
SplunkTrust

Nice, thorough answer, nnmiller!

0 Karma

nnmiller
Contributor

Could you provide examples of the events that are being extracted properly as well as those that are not, making sure to include the original line break characters? A pastebin URL or gist link would also work.

0 Karma

evgenyv
Explorer

I see that good events contain the source IP as a part of the message, when bad events don't. Please find examples here: http://pastebin.com/HskAQirf
I figured out that /etc/system/default/props.conf has the following section.

[syslog]
pulldown_type = true
maxDist = 3
TIME_FORMAT = %b %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 32
TRANSFORMS = syslog-host
REPORT-syslog = syslog-extractions
SHOULD_LINEMERGE = False
category = Operating System
description = Output produced by many syslog daemons, as described in RFC3164 by the IETF

I have commented out the TRNSFORMS and restarted the splunk server. It still didn't help

Still not clear why splunk does not set host value from source IP

0 Karma

Richfez
SplunkTrust
SplunkTrust

This appears to be an issue on the device side. You can see how the actual syslog message does not have the same information in the same places.

The best thing is, if it is possible, to change the format of the string the device is sending in. I just had this problem with Websense and had to change the syslog string to custom and rearrange some fields.

I have no idea how you would do so, and it is possible to fix this on the input side in splunk but if you can change it on the device I think that would be best.

Try that and let us know!

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...