All Apps and Add-ons

Why does Splunk sometimes not recognize Scapy generated packets in Web Page Monitor?

dejuan
Explorer

Before the question, a bit of background.
I have a setup in which I have two machines.
The first collects data from various devices and sends it directly to the second over UDP, targeting port 5005.
The second is running Splunk and has been configured to listen on port 5005 for UDP messages, and record them on a "sandbox" index.
These machines are isolated from the internet and are connected by Ethernet cables to a hub switch right next to them.
For the sake of brevity, let's call them M1 and S, short for Machine 1 and Splunk Machine.
S has been assigned IP 192.168.0.5 while M1 has 192.168.0.6.

There are three ways I can transmit packets from M1 to S.
The first is by running from the terminal on S1:

echo -n "{'Message':'hello'}" > /dev/udp/192.168.0.5/5005

This message is successfully sent from M1 to S and shows up in the sandbox index.

The second is by running some scripts that emulate our desired behavior and form packets using Scapy, again on M1 targeting S.
This is Scapy, not Scipy.
This process also successfully completes the full loop and shows up in the sandbox index.

The final method, and the one this question centers on, is to open scapy on M1 and generate packets and send them that way. Assuming we want to emulate sending packets from a docker container on M1 with an an IP of 10.10.12.9, the command used to generate these packets is as follows, with some slight editing, namely substituting text for the actual mac addresses and placing each field on its own line:

sendp(Ether(dst="<mac address of container here>",src="<mac address of S here>")
            /IP(src="10.10.12.9",dst=192.168.0.5")
            /UDP(dport=5005,sport=33017)
            /Raw(load="{'Message':'Hello.'}"), iface="veth201")

If I execute this command in Scapy, I'm told that it sends the packet.
If I run a tcpdump on S, I can see that the packet generated by the command does in fact travel from M1 to S and is received.
However, this packet is entirely ignored by Splunk.
When I say "entirely ignored" I mean that, if I open the Splunk Web Search in a browser on S, and start a real time search with a 5 minute window for all events on the "sandbox" or "main" indexes, sending the scapy packet as described above does not cause an event, while the other two methods trigger events on the "sandbox" index as expected.
How do I make Splunk recognize this packet?

EDIT 1: I've found that if I run

send(IP(src="192.168.0.6",dst=192.168.0.5")
          /UDP(dport=5005,sport=33017)
          /Raw(load="{'Message':'Hello.'}"), iface="veth201")

Then the packet will be received successfully in TCP dump and is recognized by Splunk.
If I then change the command to

send(IP(src="10.10.12.9",dst=192.168.0.5")
          /UDP(dport=5005,sport=33017)
          /Raw(load="{'Message':'Hello.'}"), iface="veth201")

Then it will fail once again to be recognized by Splunk, but the packet is successfully delivered according to tcpdump.

0 Karma
1 Solution

dejuan
Explorer

I figured out the answer.
If you send a packet with a given source address, Splunk must be able to actually reach that address. If it is unable to reach the address, even if a packet is received, Splunk will not consider it a valid event.

I discovered this by trying to ping 10.10.12.9 from S and found it was unreachable. I added a route on S then checked by pinging that the 10.10.12 address was now reachable. After doing so, I re-sent the Scapy-created packet, and it showed up in the sandbox index as a valid event.

View solution in original post

dejuan
Explorer

I figured out the answer.
If you send a packet with a given source address, Splunk must be able to actually reach that address. If it is unable to reach the address, even if a packet is received, Splunk will not consider it a valid event.

I discovered this by trying to ping 10.10.12.9 from S and found it was unreachable. I added a route on S then checked by pinging that the 10.10.12 address was now reachable. After doing so, I re-sent the Scapy-created packet, and it showed up in the sandbox index as a valid event.

elliotproebstel
Champion

Can you share your inputs.conf configuration for that port? If you've defined a <remote server> as per the docs, then Splunk will only pick up data from a particular remote host and ignore data from other hosts.
https://docs.splunk.com/Documentation/Splunk/7.1.0/Data/Monitornetworkports#Configure_a_UDP_input

0 Karma

dejuan
Explorer

I did all of my configuration through Splunk Web, and did not restrict anything when I defined the input. I went to /opt/splujnk/etc/system/local to look at inputs.conf anyway, and it is only two lines long. The file is copied below, verbatim.

[default]
host = docker-swarm-mgr

docker-swarm-mgr is the name of the host running Splunk Enterprise.

0 Karma

dejuan
Explorer

Oh, and I forgot to mention. Changing the mac and src IP in the scapy packet to those of M1 itself instead of the container that is running on M1 makes no difference; Splunk still does not recognize the packet.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...