All Apps and Add-ons

Trying to get SNMP data into Splunk...

cpt12tech
Contributor

I'm having problems getting any SNMP data into Splunk using the SNMP Modular Input. I've set up a Polling Input that is polling a Windows 2008 Server. So far no data has shown up in Splunk.

I've checked for errors using:
index=_internal ExecProcessor error snmp.py
and don't see any results.

[snmp://Archive]
communitystring = public
destination = 192.168.1.101
do_bulk_get = 1
do_get_subtree = 1
host = Archive
ipv6 = 0
snmp_mode = attributes
snmp_version = 2C
sourcetype = snmp_ta
split_bulk_output = 0
trap_rdns = 0
v3_authProtocol = usmHMACMD5AuthProtocol
v3_privProtocol = usmDESPrivProtocol

0 Karma
1 Solution

cpt12tech
Contributor

I'm now receiving traps into Splunk.

I ran into an issue while working on SNMP where I couldn't see the Data Inputs in Splunk 6.4.2.
https://answers.splunk.com/answers/427846/clicking-on-data-inputs-shows-500-internal-server.html
This has been fixed in 6.4.3.

Just now getting back to working on getting SNMP traps into Splunk. I'm not sure what fixed the problem, however the server I'm running Splunk on has three NICs attached to 3 different subnets. The traps in the Data Input were set to localhost. Using wireshark I could see what address the SNMP traps were being addressed to, added that IP to my SNMP trap in the Data Inputs and data began to flow into splunk. Haven't gotten polling working, but that's a different issue.

Pretty sure I tried those different IP addresses in the past, but not 100% sure.

Many thanks to jkat54 & pmeyerson for their help!

View solution in original post

0 Karma

cpt12tech
Contributor

I'm now receiving traps into Splunk.

I ran into an issue while working on SNMP where I couldn't see the Data Inputs in Splunk 6.4.2.
https://answers.splunk.com/answers/427846/clicking-on-data-inputs-shows-500-internal-server.html
This has been fixed in 6.4.3.

Just now getting back to working on getting SNMP traps into Splunk. I'm not sure what fixed the problem, however the server I'm running Splunk on has three NICs attached to 3 different subnets. The traps in the Data Input were set to localhost. Using wireshark I could see what address the SNMP traps were being addressed to, added that IP to my SNMP trap in the Data Inputs and data began to flow into splunk. Haven't gotten polling working, but that's a different issue.

Pretty sure I tried those different IP addresses in the past, but not 100% sure.

Many thanks to jkat54 & pmeyerson for their help!

0 Karma

monteirolopes
Communicator

Thank you jkat54!
It works!

0 Karma

jkat54
SplunkTrust
SplunkTrust

Your error indicates that port 162 can't be bound to on your Splunk server.

[Errno 10013] An attempt was made to access a socket in a way forbidden by its access permissions

My best guess is that you have SNMP Agent or other SNMP service running on the Splunk server and it has already bound to the port. Start -> run -> services.msc [return/click ok]. Look for SNMP services. You can also netstat -ano at command prompt to see if the port is already in use when splunkd is stopped.

To fix, disable whatever is bound to the port.

Other possibility includes splunkd running as underprivileged user (a best practice), and the user needs a Group Policy setting to allow it to bind to 162.

Oh and the easiest solution. Edit the SNMP modular input settings and use TRAP port 1162 or any other port instead of the default 162. or just disable the TRAP by U checking the box alt text

cpt12tech
Contributor

Getting back into this to see if I can get it to work. Thanks for the help jkat.

Ran the command 'nstat -ano' and there was a service listening on 127.0.0.1. Stopped Splunkd and the listener went away, also went away when I disabled snmp_ta. I wanted to try changing the snmp ip that was listening on 161. Tried going into the app properties to check my settings but I don't see the GUI? Just a standard app GUI. Did something change or am I looking in the wrong place.

Also tried deleting snmp_ta app and reinstalling, but don't see the config gui.

0 Karma

pmeyerson
Path Finder

There isn't a seperate GUI component for SNMP Modular Input. You change the settings by adjusting the data inputs: Settings -> Data Inputs -> SNMP. The settings are defined per Input you define.

cpt12tech
Contributor

Ahh.. thank you. That started me off on a rabbit trail. I get a 500 internal error when I go to data inputs... while trying to solve that issue, now the Splunk web service is refusing to start and I cannot connect to Splunk via a web browser. Oui.

0 Karma

cpt12tech
Contributor

got the web browser working by typing in
splunk enable webserver
but haven't figured out why I get an internal server error when I go to the data inputs link...

0 Karma

cpt12tech
Contributor

This was a bug in the current Splunk version. Waited for Splunk update, data inputs are working again. Finally getting back to SNMP implementation.

0 Karma

pmeyerson
Path Finder

I can give a couple of things that helped me with the issues I had with SNMP Modular Input. I've found the process pretty cumbersome and required a lot of troubleshooting, at least from Windows.

  • check the splunkd log, errors will be reported there: \var\log\splunk\splunkd.log
  • if possible, you can install wireshark on the splunk server, and filter for UDP, this will allow you to see the SNMP packets leaving the splunk server and reply, if any, being received.
  • I was using Symantec Endpoint Protection 12.1.2100.2093. Even with the "Proactive Threat Protection" and "Network Threat Protection" disabled via the Symantec GUI, I found that outbound UDP was being denied. I installed on a testbed system without Symantec and the generic network error message in the Splunk log was fixed.
  • I also found that troubleshooting was a lot easier when going for a standard OID, like sysContact. This ruled out issues with getting the oid number to poll correct.

As an aside, if anyone has issues compiling their MIB files into python, you can actually drop an empty .py file into the custom mib directory, specify that filename in the SNMP Modular Input data input setup, and you will be able to poll. You just don't see the translation of oid values to key name. If the MIB file is specified in your input setup but not present polling will abort for that input.

Does your remote device report any errors like bad community name received or login failure, etc? It seems odd that you are specifying snmp_version = 2C but also including v3 auth settings?

cpt12tech
Contributor

Installed Wireshark, another great tool, and I do see SNMP data coming into the Splunk server, but don't see any SNMP messages leaving the server. I've set up trap notifications on the server's I'm monitoring and assuming those are the SNMP messages being received. Polling should be sending out messages? Searched in the Splunkd log (index=_internal sourcetype=splunkd snmp) and did find some messages.

I set up two trap receivers in Splunk as well. One with localhost as the trap destination, and the other with the IP I'm using to send the SNMP messages to.

03-17-2016 15:44:53.661 -0600 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\snmp_ta\bin\snmp.py"" Failed to register transport and run dispatcher: bind() for ('localhost', 162) failed: [Errno 10013] An attempt was made to access a socket in a way forbidden by its access permissions snmp_stanza:snmp://Kaleido-Trap

03-17-2016 15:24:32.816 -0600 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\snmp_ta\bin\snmp.py"" Failed to register transport and run dispatcher: bind() for (u'192.168.1.102', 162) failed: [Errno 10013] An attempt was made to access a socket in a way forbidden by its access permissions snmp_stanza:snmp://Archive-Trap

I've noticed all the trap messages are coming in on port 161. For windows servers I'm using the Microsoft SNMP service. For other devices I have I use whatever they provide. Will the SNMP Modular Input receive from port 161?

I didn't specify an MIB, just wanted to see if I could get any data into the Splunk database. I haven't been able to compile custom MIB's using python and get command not found when trying to make the egg.

How did you find out that the Symantec Endpoint Protection was blocking the UDP port?

0 Karma

pmeyerson
Path Finder

Hi cpt12tech.

When you poll SNMP from splunk, you should see SNMP packets leaving your server get-request and snmp packets returning get-response I believe they show as. I can find a packet capture when I am at office tomorrow. Of particular note, drill into those SNMP packets and verify the UDP port number, etc is as expected.

You are running splunk as a service? Are you running it as LocalSystem or some other user account?
Does your organization push any Microsoft Group Policy, or other firewall/antivirus policy that would block outgoing UDP connections from the machine? Another thing to check would be to try disabling any SNMP service running directly on the server you have splunk installed on, it almost sounds like they may be conflicting?

SNMP Modular Input can listen on any UDP port - specify the port number when you are in the settings page for the Data Input.

This: http://www.alvestrand.no/objectid/1.3.6.1.2.1.1.html is a good set of OIDs to use for test polls. I don't think you have to specify the MIB file name, just pick one or more of those OIDs. They are standard MIBS and I believe included with SNMP Modular Input.

As to SEP - I have an unfortunate long history from previous work with SEP not behaving as it should. I took a chance and repeated my setup on a windows VM that did not have it installed - I put Microsoft Security Essentials on it instead. (use at your own risk). I also had problems with SEP client blocking command line API calls needed to setup splunk clustering.

HTH

jkat54
SplunkTrust
SplunkTrust

Are your snmp ports open? Defaults to 161 on the source (called destination in the a snmp app) and 162 on the Splunk server.

If your splunk server is linux and you're following best practices, you will not be able to open port 162 because only root can open ports below 1025.

cpt12tech
Contributor

Thanks for the reply. I'm running Splunk on a Windows server. I'm monitoring the same source with another SNMP management app on a different server and am able to connect and poll the data. The firewall is turned off on the Windows server running Splunk and the Windows server I'm attempting to poll.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Any firewalls between the two on the network?

Use the following command to test:

start -> run -> cmd [enter/click ok]
telnet [ipaddress] [portnumber]

If you get a blinking cursor on a blank page, the port is open to the ipaddress you provided. Else you'll get a timeout error. If you do get the blinking cursor press ctrl + ] to escape out of the telnet session. Try from both servers to the other using the correct ports and ips.

Telnet might not be installed on the server, if not run this at command prompt

pkgmgr /iu:"TelnetClient"

0 Karma

cpt12tech
Contributor

Tried the telnet, but there was no response. I searched around the internet and it seems that Telnet doesn't support UDP. I downloaded nmap, and scanned port 161 and it reports as open.

Really great tool.
https://nmap.org

0 Karma

jkat54
SplunkTrust
SplunkTrust

Oh... Duh!!! Sorry. Netcat supports udp but it's Linux based... I'll be in the dunce corner

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...