Hi,
I'm collecting syslog events sent by different network equipment. For all devices, the host value is recorded as expected - source IP address of syslog message. However, for one of devices - the host value is "2015".
tcpdump shows the correct source IP. All other fields are extracted as expected.
What can be wrong there?
I have defined the input as following:
./etc/apps/search/local/inputs.conf
[udp://514]
connection_host = ip
sourcetype = syslog
no_priority_stripping = true
./etc/apps/syslog_priority_lookup/default/props.conf
[source::udp:514]
EXTRACT-extract_syslog_priority = ^<(?<syslog_priority>\d+)>
LOOKUP-lookup_syslog_priority = syslog_priority_lookup syslog_priority OUTPUTNEW syslog_facility, syslog_severity
First off, separate the notion of the sourcetype syslog
from the protocol syslog. Any log event ingested into Splunk can be given the sourcetype syslog
provided it conforms to the expected event format.
Data that you configure to have the sourcetype syslog
passes through a transformation process, where the hostname is extracted from each event. Splunk extracts the source IP from the log message itself, it does not use the IP that sends the syslog message. Syslog messages may be forwarded from a syslog host that didn't create the log event.
Generally, a syslog event starts with the date and is immediately followed by the host name or IP that created the event. The events you shared that do not have the source host parsed correctly are not formatted as a normal syslog message.
Parsed:
Oct 4 05:29:41 10.20.4.209 CMD_ACCT <...>
Oct 4 05:29:41 10.20.4.209 CMD_ACCT <...>
Not parsed:
Oct 4 05:44:06 2015 HP %%10SHELL/<...>
Oct 4 05:44:06 2015 HP %%10SNMP/<...>
The second message has the year in the place that Splunk is expecting the host IP address. Is this actually a syslog event, or is it a SNMP event?
There are several methods to resolve the issue:
Modify your Splunk inputs.conf per the docs for inputs.conf for UDP/TCP inputs. From inputs.conf doc page:
connection_host = [ip|dns|none]
Your inputs.conf
could be written like:
[udp://514]
connection_host=ip
sourcetype = not_syslog
It seems that the Splunk syslog processor is mixing up part of the timestamp with the ip of the sending device in this case. I think that we'd need to see example _raw events for the correct and incorrect messages, but I think a better answer would be to switching to a syslog server for this data source.
There are variety of drawbacks to using the network inputs. Check out this page for more info : http://www.georgestarcher.com/splunk-success-with-syslog/
First off, separate the notion of the sourcetype syslog
from the protocol syslog. Any log event ingested into Splunk can be given the sourcetype syslog
provided it conforms to the expected event format.
Data that you configure to have the sourcetype syslog
passes through a transformation process, where the hostname is extracted from each event. Splunk extracts the source IP from the log message itself, it does not use the IP that sends the syslog message. Syslog messages may be forwarded from a syslog host that didn't create the log event.
Generally, a syslog event starts with the date and is immediately followed by the host name or IP that created the event. The events you shared that do not have the source host parsed correctly are not formatted as a normal syslog message.
Parsed:
Oct 4 05:29:41 10.20.4.209 CMD_ACCT <...>
Oct 4 05:29:41 10.20.4.209 CMD_ACCT <...>
Not parsed:
Oct 4 05:44:06 2015 HP %%10SHELL/<...>
Oct 4 05:44:06 2015 HP %%10SNMP/<...>
The second message has the year in the place that Splunk is expecting the host IP address. Is this actually a syslog event, or is it a SNMP event?
There are several methods to resolve the issue:
Modify your Splunk inputs.conf per the docs for inputs.conf for UDP/TCP inputs. From inputs.conf doc page:
connection_host = [ip|dns|none]
Your inputs.conf
could be written like:
[udp://514]
connection_host=ip
sourcetype = not_syslog
Nice, thorough answer, nnmiller!
Could you provide examples of the events that are being extracted properly as well as those that are not, making sure to include the original line break characters? A pastebin URL or gist link would also work.
I see that good events contain the source IP as a part of the message, when bad events don't. Please find examples here: http://pastebin.com/HskAQirf
I figured out that /etc/system/default/props.conf has the following section.
[syslog]
pulldown_type = true
maxDist = 3
TIME_FORMAT = %b %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 32
TRANSFORMS = syslog-host
REPORT-syslog = syslog-extractions
SHOULD_LINEMERGE = False
category = Operating System
description = Output produced by many syslog daemons, as described in RFC3164 by the IETF
I have commented out the TRNSFORMS and restarted the splunk server. It still didn't help
Still not clear why splunk does not set host value from source IP
This appears to be an issue on the device side. You can see how the actual syslog message does not have the same information in the same places.
The best thing is, if it is possible, to change the format of the string the device is sending in. I just had this problem with Websense and had to change the syslog string to custom and rearrange some fields.
I have no idea how you would do so, and it is possible to fix this on the input side in splunk but if you can change it on the device I think that would be best.
Try that and let us know!