Getting Data In

How to verify on the device (forwarder) end that it is indeed sending logs to a Splunk sever?

progress101
New Member

I am currently in a situation where I don't have access to the actual Splunk server but have been provided the Splunk VIP to send logs.
I entered the following command on the device:

logging host x.x.x.x transport udp port 514

I am able to ping the servers that are behind the VIP. The show log commands shows that logs are being logged at the server. Is there anything else I can do on the device end to prove that logs are being sent correctly to the Splunk Server? If I have missed or done something incorrectly be let me know as well.

v/r

0 Karma

woodcock
Esteemed Legend

I would push VERY HARD for migrating away from syslog-direct-to-Indexer by adding a syslog-ng node and either doing either this:
http://www.georgestarcher.com/splunk-success-with-syslog/
Or this:
https://www.splunk.com/blog/2017/03/30/syslog-ng-and-hec-scalable-aggregated-data-collection-in-splu...

If you are trying to syslog directly to splunk (you should not) on port 514 and you are running splunk as non-root, Splunk will show the port is enabled through the GUI but you will see error logs that it cannot open the port because it is not root. Use this is justification to use one of the other designs I mentioned.

If you are sending UDP, switch to TCP temporarily so that you can trace the failure more easily.

It is very easy to to a tcpdump just about anywhere to validate that traffic is getting here. Use your normal networking tools/skillset to validate/debug.

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi progress101,

This sounds like a firewall/routing issue and not really like a Splunk problem 😉

On your Cisco device increase the logging and send a syslog message to the VIP

send log <Put Any Text Here> 

This will send a syslog message to the configured syslog receiver. Should you not get any message in the console, try reading the log of the device using show log.

Hope this helps ...

cheers, MuS

PS: My Cisco brain is a bit rusty and commands may have changed by now 😉

0 Karma

mydog8it
Builder

Are you trying to send syslog data from a network device? If so is the receiver a syslog server with a UF forwarding logs or a Splunk server with a UDP listener? the standard UDP port for syslog is 514, are you sure you should be using port 154?

0 Karma

progress101
New Member

Sorry typo, yes 514.

0 Karma

mydog8it
Builder

It sounds like you have access to the network devices in your environment. If that is so, figure out where the syslog server receiving your traffic is attached to the network and span the port to capture interesting packets. If the data is making it to the network interface that the syslog server is attached to the Splunk Engineer should be responsible for onboarding the data from there.

0 Karma

mydog8it
Builder

If you do not have the ability to capture the mirrored data from the spanned port suggested above, look for a device in the path that you have access to to perform a tcp dump or packet capture. You mentioned this was a cisco switch, if you have a cisco ASA in the path a packet capture can be performed there. You also mentioned a VIP, if there is an F5 in the path a tcp dump can be performed there. The bottom line is there are many ways to look at the data flow on the wire to validate the data has been sent from the source, without knowing your environment it becomes difficult to suggest the "best" way for your situation.

Revisiting what you have provided though, the successful ping (if it is from the network device that you are trying to send logs from) validates the network path is available and the device sends the data destined for that network out the correct interface. It does not guarantee that syslog data is permitted. Can you verify there are no firewall denies along the way(for src_ip --> dest_ip udp port 514) . If you log firewall permitted traffic, and you have access to that data, you can verify the two end points are trying to talk on the correct protocol/port.

Do you have another syslog server in the environment? If so, does it have the syslog data for this device? If that server also has a UF installed you might suggest collecting the data from that server in the interim.

0 Karma

progress101
New Member

I appreciate you taking the time out to provide this information. I just sent an email to our senior networking engineer to gather more information related to the FW rules currently in place. I will circle back once he responds to provide an update.

0 Karma

progress101
New Member

During the conference call, we ensure the ports on the FW were opened for port 514 and that the ACLs were configured properly. The Splunk engineer did a TCPdump multiple times, but no logs showed up. We sent "send log" messages and brought an interface down then back up but still no logs being shown. At this point, we have escalated the issue.

0 Karma

codebuilder
SplunkTrust
SplunkTrust

You can ensure secure/verified access between your forwarders and indexers by using Indexer Discovery, setting a pass4SymmKey, and or using SSL.

https://docs.splunk.com/Documentation/Splunk/7.3.0/Indexer/indexerdiscovery

For verification of log consumption on the forwarder side, you can simply check the splunkd.log files which are generally located at /opt/splunkforwarder/var/log/splunk/

That log file will contain entries about what files were consumed, where they were sent, acks, issues, etc...

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

progress101
New Member

This can be done on a Cisco switch?

0 Karma

marycordova
SplunkTrust
SplunkTrust

maybe tcpdump?

@marycordova
0 Karma

progress101
New Member

I don't have access to the Splunk server, it is managed by a different team. They had already done a tcpdump, but no success in locating logs from the specified host. What can I do/provide on my end regarding the switch?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...