Problems with syslog and splunk

[krusty]

We currently testing the splunk software and it looks very nice. But now i have a problem with the hostname and syslog entries. I configured 3 TCP "Data Inputs". With one TCP input the splunk server received syslog messages. For example: <78>Dec 7 15:01:01 srv123 crond[16492]: (root) CMD (run-parts /etc/cron.hourly)

With another TCP input the splunk server received weblogic messages. For example: <174>Dec 7 16:11:06 srv123 WebLogicLog: ####<07.12.2010 16:11 Uhr MEZ> <> <> <> <1291734660734> <60% of the total memory in the server is free>

You see that both messages are from SRV123. Into splunk i see two hosts. One with the name SRV123 and another with the name srv123.test.it. The messages from weblogic are stored into host srv123.test.it and the message from syslog are stored into the host SRV123.

Can someone explain me how to configure incoming syslog messages to get the messages to the host srv123.test.it?

Regards

[gfriedmann]

Is it possible that your DNS server has multiple reverse records for the source IP?

maybe try running nslookup a half dozen times and make sure it is always returning the fully qualified domain name.

[krusty]

Hi gfriedmann,

no we only have one record configured with the DNS Server. And if I run nslookup several times I always get the full hostname like srv123.text.it.

[adamw]

Krusty,

What about configuring syslog-ng to write files out with hostname, then have splunk index that with regex on path for hostnames.

This way you also don't lose syslog entries when you restart splunk.

[southeringtonp]

See also - http://answers.splunk.com/questions/8912/syslog-ng-filter-by-ip

[adamw]

We use the host_segment option in inputs.conf, in order to pull the hostname straight off of the path on the file system, as written by syslog-ng.

in inputs.conf:

[monitor:///opt/syslog]
disabled = true
host_segment = 3
index = default

and syslog-ng configuration:
destination hosts {
file("/opt/syslog/$HOST/$YEAR/$MONTH/$DAY/$FACILITY$YEAR$MONTH$DAY")

[krusty]

Hi adamw,

could you please tell me how to configure splunk to get the hostname correctly.

With kind regards

[krusty]

Hi @all,

I tested something new.

If I edit the two config files transforms.conf and props.conf with the following entries and switch the sourcetype of my data input I get the complete hostname.

transforms.conf
[rsyslog-host]
disabled = 0
FORMAT = host::$1

[rsyslog-host-full]
disabled = 0
FORMAT = host::$1

props.conf
[rsyslog]
pulldown_type = true
TRANSFORMS-host = rsyslog-host

Is it possible to change any entry for syslog to get the full hostname? I've try some changes but nothing happens. 😞

Regards

[Genti]

When you create a data input for the TCP or UDP incoming syslog events at Manager » Data inputs » UDP » Add New you can also set the host you desire the data to come in. The options are:
. ip -sets the UDP input processor to rewrite the host with the ip address of the remote server
. dns - sets the host to the dns entry of the remote server
. Custom - Sets it to a custom value.

You either need to fix the dns for the two types of data, or you can set them both to the ip, and then tag the ip with the host name you desire, or you can set up a custom name for these two hosts.

Either of the three choices above should fix your issue.

Make sure to restart the server after you make any changes to these configurations.
Cheers

[krusty]

Thanks for you declaration but I still have all the three Data Inputs set to dns.
So I don't understand why syslog data will set the host srv123 and all other data will set srv123.test.it. You know what I mean?

I think the only possibility which works is to set for each host own TCP Data Inputs. But this is not really a smart solution.

If anyone have ideas to fix my problem, please let me know.

Thanks.