Splunk syslog event count too low compared to kiwi syslog server on Windows platform

[pdherndon]

I have configured approx. 100 access points to send syslog events to both Splunk and to a kiwi syslog server I have set up on a Windows 7 PC. (Splunk is installed on a fairly high powered Linux server). When I compare the events logged in Splunk to the events captured in the kiwi server and on the access points themselves, I see a huge difference. I can have over 2100 events from an access point captured in my kiwi server, (verified by looking at the AP itself), while I see 4 events in Splunk.

I have looked at the Manager>Systems Settings>System Logging logging levels, and they are set to INFO, and I looked at the configuration in $SPLUNK_HOME/etc/log.cfg. I see rootCategory=INFO,A1. I have scanned through these questions, but could not find one posted that seemed to help with this.

What else should I look at regarding Splunk's configuration? Even though I see a huge discrepancy between Splunk's syslog events and kiwi's, I still see a great number of events in Splunk's Search window.

[Ayn]

log.cfg relates to Splunk's own operational logs, not the logs it receives from other systems.

You should tell us more about your setup, my initial guess is that you're having problems with improper timestamp parsing (if you search over all time, do you still not see the number of messages you expect to see?) but depending on how you've configured your syslog input and how you're verifying whether logs are coming in or not, there may be other things that cause the problems you're having. So, please tell us more about your setup, in particular it would be interesting to hear how you're verifying that events are coming in.

[pdherndon]

OK, can someone give me a suggestion as to how I should validate the syslog configuration on this Linux server? (Ubuntu 10.04.2 LTS) I've tried looking for docs on this, but wasn't able to get very far.

[Ayn]

It certainly sounds like a network issue rather than a Splunk issue, yes.

[pdherndon]

I admit that I'm a Linux user, not an admin. I run ifconfig and I see eth0 (IP xx.xxx.xxx.3), eth1 (.34), and eth2 (.2). It is eth2's IP that I've configured the APs to use. When I run tcpdump -i eth2 "udp[2:2] = 514", I don't see any input. When I run tcpdump -i eth0 "udp[2:2] = 514", I do see some input. (I did config 2-3 APs to use the .3 syslog address as a test). Is it possible that syslog is not configured appropriately on this Linux server for eth2 to be the correct IP address? I should have seen many hundreds of lines on eth2 while tcpdump ran.

[Ayn]

OK, thanks for the info. If you run tcpdump and listen on port UDP/514 on the Splunk server, do you see all syslog data you expect to see arriving?

[pdherndon]

(Part 3)
I have kiwi installed on a much less powerful Windows 7 PC using 1 100Mb interface set up for UDP 514 (like Splunk).

[pdherndon]

I'll try to respond to your questions/theories in order: (Part 2):
- I have set up only one data input --> syslog on UDP 514. I have five different sites (IP subnets define which sites the events are coming from). I have both Splunk's and kiwi's syslog IPs configured on the access points I'm monitoring. Comparing line-by-line the Splunk server with kiwi and device, Splunk shows far too few events.

I've have verified that both Ethernet interfaces (Gig) on the Linux server and the switch port are running error-free with no drops.

[pdherndon]

(These comment fields are too short to answer your questions in one response, so I'll try to use two comments to respond:)

I'll try to respond to your questions/theories in order: (Part 1)

• Searching over all time still shows event counts that are probably .01% of the events I see from the same source over the same time period in kiwi syslog server.
• I don't know how to verify what events are reaching the splunkd process on my Linux server, but I verify what should be logged by Splunk by what I see in the kiwi syslog server and on the devices themselves.