I have the following setup:
Distributed Splunk Enterprise deployment with 2 clustered indexers, 1 cluster master, 1 search head.
Separate server configured with an instance of Kiwi Syslog server (listening on UDP 514). Syslogs are being successfully written to disk based on sending device category (e.g switch, firewall etc). This server also has an instance of Universal Forwarder installed, which is monitoring the log file and forwarding this data on to the Index cluster.
The above seems to be working ok. I can see syslogs being received by the syslog server and being written to log file successfully. I can also log into my Splunk Search head under the basic "Searching & Reporting" app and I can search on the custom index which I am sending these syslogs to and can see the syslogs appearing on the Indexer.
My issue however is three-fold:
Firstly, the Splunk Indexers don't seem to be getting the host field correct. Without any sourcetype defined on my Universal Fowarder, it was giving the host field as the Facility and severity level (Local6.Notice) of the syslog message. I changed the sourcetype defined for this syslog file monitor on the Universal forwarder to cisco:ios. Now it has labelled the host field as the hostname of the syslog server rather than the hostname of the device originating the syslog message. How do I get it to correctly pick out the correct hostname out of the syslog message?
Secondly, in a bid to solve this issue I installed the Cisco Networks Add-on (TA-cisco_ios) on my Indexers and my Search Head and then installed the Cisco Networks App (cisco_ios) on my search head. I believed that the add-on would help to interpret the incoming cisco syslog messages, so that the syslog fields would get interpreted correctly, however the syslogs are still being displayed with the host = the syslog hostname.
Finally the newly installed Cisco Networks app, although it appears to have installed correctly, is not showing any received data, even though I can see the syslog messages using the basic "Search & Reporting" app. (if my syslogs are being placed in a custom index, do I need to tweak the app to be looking into the correct index?).
Thanks in advance 🙂
... View more