All Apps and Add-ons

No port_scan data

richgalloway
SplunkTrust
SplunkTrust

We're running Splunk for Asset Discovery 6.0 under Splunk 6 on an Ubuntu system. The app has been running for a week, but we have no data in the asset_discovery index. There are several input scripts defined and enabled, including '$SPLUNK_HOME/etc/apps/asset_discovery/bin/nmap.sh -A -O 192.168.100.0/24'. If I run this command manually, I see data for all of the hosts in that subnet. However, a search of 'index=asset_discovery' returns no events. nmap is owned by root. I assume it is running as root also since all of Splunk does so.

I see nothing in splunkd.log other than "INFO ExecProcessor - New scheduled exec process:
/opt/splunk/etc/apps/asset_discovery/bin/nmap.sh -A -O 192.168.100.0/24."

Where is my port_scan data going?

---
If this reply helps you, Karma would be appreciated.
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Redirecting the nmap.sh output to a file showed nmap was failing because of a missing OpenSSL library.

nmap: /opt/splunk/lib/libcrypto.so.1.0.0: version `OPENSSL_1.0.0' not found (required by nmap)
nmap: /opt/splunk/lib/libssl.so.1.0.0: version `OPENSSL_1.0.0' not found (required by nmap)

Adding unset LD_LIBRARY_PATH to nmap.sh fixed the problem.

Thanks to Splunk tech support for their help with this.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

AnilPujar
Path Finder

03-10-2019 13:23:14.272 +0400 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\etc\apps\asset_discovery\bin\nmap.cmd" -A -O" The system cannot find the file C:\Program Files\Splunk\etc\apps\asset_discovery\bin\nmap.path.

0 Karma

AnilPujar
Path Finder

installed nmap, set env variables and cmd result also

C:\Windows\System32>nmap -v -A scanme.nmap.org
Starting Nmap 7.70 ( https://nmap.org ) at 2019-03-10 13:26 Arabian Standard Time
NSE: Loaded 148 scripts for scanning.
NSE: Script Pre-scanning.
Initiating NSE at 13:26
Completed NSE at 13:26, 0.00s elapsed
Initiating NSE at 13:26
Completed NSE at 13:26, 0.00s elapsed
Initiating Ping Scan at 13:26
Scanning scanme.nmap.org (45.33.32.156) [4 ports]
Completed Ping Scan at 13:26, 0.95s elapsed (1 total hosts)
Initiating Parallel DNS resolution of 1 host. at 13:26
Completed Parallel DNS resolution of 1 host. at 13:26, 0.00s elapsed
Initiating SYN Stealth Scan at 13:26
Scanning scanme.nmap.org (45.33.32.156) [1000 ports]
Discovered open port 80/tcp on 45.33.32.156
Discovered open port 22/tcp on 45.33.32.156
Completed SYN Stealth Scan at 13:27, 14.48s elapsed (1000 total ports)
Initiating Service scan at 13:27
Scanning 2 services on scanme.nmap.org (45.33.32.156)
Completed Service scan at 13:27, 6.56s elapsed (2 services on 1 host)
Initiating OS detection (try #1) against scanme.nmap.org (45.33.32.156)
Retrying OS detection (try #2) against scanme.nmap.org (45.33.32.156)
Initiating Traceroute at 13:27
Completed Traceroute at 13:27, 6.04s elapsed
Initiating Parallel DNS resolution of 6 hosts. at 13:27
Completed Parallel DNS resolution of 6 hosts. at 13:27, 0.00s elapsed
NSE: Script scanning 45.33.32.156.
Initiating NSE at 13:27
Completed NSE at 13:27, 8.52s elapsed
Initiating NSE at 13:27
Completed NSE at 13:27, 0.00s elapsed
Nmap scan report for scanme.nmap.org (45.33.32.156)
Host is up (0.27s latency).
Not shown: 993 filtered ports
PORT STATE SERVICE VERSION
21/tcp closed ftp
22/tcp open ssh OpenSSH 6.6.1p1 Ubuntu 2ubuntu2.11 (Ubuntu Linux; protocol 2.0)
| ssh-hostkey:
| 1024 ac:00:a0:1a:82:ff:cc:55:99:dc:67:2b:34:97:6b:75 (DSA)
| 2048 20:3d:2d:44:62:2a:b0:5a:9d:b5:b3:05:14:c2:a6:b2 (RSA)
| 256 96:02:bb:5e:57:54:1c:4e:45:2f:56:4c:4a:24:b2:57 (ECDSA)
|_ 256 33:fa:91:0f:e0:e1:7b:1f:6d:05:a2:b0:f1:54:41:56 (ED25519)
23/tcp closed telnet
80/tcp open http Apache httpd 2.4.7 ((Ubuntu))
|http-favicon: Unknown favicon MD5: 156515DA3C0F7DC6B2493BD5CE43F795
| http-methods:
|
Supported Methods: POST OPTIONS GET HEAD
|_http-server-header: Apache/2.4.7 (Ubuntu)
|_http-title: Go ahead and ScanMe!
443/tcp closed https
5061/tcp closed sip-tls
8080/tcp closed http-proxy
Device type: general purpose
Running (JUST GUESSING): Linux 4.X|3.X|2.6.X (99%)
OS CPE: cpe:/o:linux:linux_kernel:4.9 cpe:/o:linux:linux_kernel:3 cpe:/o:linux:linux_kernel:2.6.32
Aggressive OS guesses: Linux 4.9 (99%), Linux 3.10 - 3.16 (96%), Linux 3.10 (93%), Linux 2.6.32 (93%), Linux 3.10 - 3.12 (93%), Linux 4.4 (93%), Linux 2.6.39 (90%)
No exact OS matches for host (test conditions non-ideal).
Uptime guess: 14.541 days (since Sun Feb 24 00:28:30 2019)
Network Distance: 19 hops
TCP Sequence Prediction: Difficulty=261 (Good luck!)
IP ID Sequence Generation: All zeros
Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel

TRACEROUTE (using port 23/tcp)
HOP RTT ADDRESS
1 1.00 ms ******************
2 1.00 ms 172.30.7.1
3 1.00 ms 172.30.6.78
4 1.00 ms 172.30.3.45
5 1.00 ms 10.192.116.66
6 ... 18
19 260.00 ms scanme.nmap.org (45.33.32.156)

NSE: Script Post-scanning.
Initiating NSE at 13:27
Completed NSE at 13:27, 0.00s elapsed
Initiating NSE at 13:27
Completed NSE at 13:27, 0.00s elapsed
Read data files from: C:\Program Files (x86)\Nmap
OS and Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 59.31 seconds
Raw packets sent: 2124 (97.044KB) | Rcvd: 151 (11.090KB)

C:\Windows\System32>

0 Karma

mario_traf
New Member

Hi,
I also have a similar problem. I can see data within a splunk search "index=asset_discovery sourcetype=port_scan", but the eventtype port_scan (index=asset_discovery sourcetype=port_scan "Host:" "Ports:" "Ignored State:" ) doesn't produce anything as my script isn't generating any "Ignored State:"

I am running the following script:
/opt/splunk/etc/apps/asset_discovery/bin/nmap.sh -A -O -t 172.20.32.0/24 --max-retries 1 --osscan-guess --system-dns
and I have added "unset LD_LIBRARY_PATH" to the nmap.sh script as well as ensuring that nmap is chmod'ed so the splunk user can use it.
Have a missed something and argument when calling the script?
Mario

0 Karma

mario_traf
New Member

I wasn't sure if the ignored state was needed or not.
anyway, I have done as suggested.
turns out that the version of nmap I am using doesn't generate the "Ignored State:" text anymore

0 Karma

mw
Splunk Employee
Splunk Employee

You can edit the eventtype to remove that portion.

richgalloway
SplunkTrust
SplunkTrust

Redirecting the nmap.sh output to a file showed nmap was failing because of a missing OpenSSL library.

nmap: /opt/splunk/lib/libcrypto.so.1.0.0: version `OPENSSL_1.0.0' not found (required by nmap)
nmap: /opt/splunk/lib/libssl.so.1.0.0: version `OPENSSL_1.0.0' not found (required by nmap)

Adding unset LD_LIBRARY_PATH to nmap.sh fixed the problem.

Thanks to Splunk tech support for their help with this.

---
If this reply helps you, Karma would be appreciated.

tednaleid
Engager

Thanks! I hit this when running curl in my shell command to post an alert to a slack channel. Doing ldd /usr/lib/x86_64-linux-gnu/libcurl.so.4 in my script and piping that to a file showed that when splunk executed the script it was adding a prefix to the location of libssl as you saw. Unsetting the LD_LIBRARY_PATH it fixed it.

0 Karma

mw
Splunk Employee
Splunk Employee

Rich, I'm not sure what's up here. Could you shoot me an email when you have a chance and we can try doing some debugging? I'm curious about what's happening here as well, particularly since I really haven't changed the scanning stuff in the latest version. mwilson at splunk dot com.

0 Karma

mikaelbje
Motivator

I figured it out. I had to chmod +s the nmap binary. I also had to chsnge ifconfig in nmap.sh to /sbin/ifconfig. This was in Ubuntu

Restarting Splunk shouldn't be necessary AFAIK as you are only modifying Linux permissions for a binary that's called. However I have Splunk running as the "splunk" user, so if you're running as root it should absolutely work. My permissions for reference:

-rwsr-s--- 1 root adm 756464 Dec 14  2011 /usr/bin/nmap

groups splunk
splunk : splunk adm

Running this from command line works fine, also as a scripted input in Splunk:

/opt/splunk/etc/apps/asset_discovery/bin/nmap.sh -A -O -t 172.24.201.0/24

Since this is Ubuntu sh is a symbolic link to dash, not bash, but it should work in bash too.

asset_discovery/local/inputs.conf


[script://./bin/nmap.sh -A -O -t 172.24.201.0/24]
disabled = false
index = asset_discovery
interval = 3600
sourcetype = port_scan
source = nmap

0 Karma

jrodman
Splunk Employee
Splunk Employee

The reason setting the executable setUID works (which I don't recommend!) is that Linux sanitizes the environment when jumping through a setuid gate, specifically dropping LD_LIBRARY_PATH and other linker controls, so that a user cannot execute arbitrary code as root trivially.

Thus this indirectly requests the action the app should have taken in the first place, to strip the library path when running a system binary.

0 Karma

JoeIII
Path Finder

For NMAP to do OS detection it requires root privileges. This is why setUID works if Splunk is not running as root already.

I recommend configuring sudoers to allow splunk to run nmap without a password, then modifying the script to "sudo nmap", in addition to adding unset LD_LIBRARY_PATH in the event of that issue.

my /etc/sudoers.d/splunknmap (use visudo so you don't botch the file rights (my splunk instance runs as "splunk":

splunk ALL = (root) NOPASSWD: /usr/bin/nmap

Still not an ideal solution, but the best available, in my opinion

0 Karma

dmartinez_splun
Splunk Employee
Splunk Employee

I'm running this on my Mac for testing and I had to:

[as root]
sh-3.2# type nmap
nmap is /usr/local/bin/nmap
sh-3.2# cd /usr/local/bin/
sh-3.2# chmod 6711 nmap
sh-3.2# ls -lrt
-rws--s--x 1 root wheel 6059116 23 Sep 01:17 nmap

[as daniel]
192-168-1-5:bin Daniel$ ./nmap.sh -A -O

Nmap 6.49BETA5 scan initiated Fri Oct 16 12:17:08 2015 as: nmap -oG - -A -O 192.168.1.5/24

[it works]... data coming into Splunk

0 Karma

richgalloway
SplunkTrust
SplunkTrust

[script://./bin/nmap.sh]
disabled = 0

[script://./bin/nmap.sh -A -O]
disabled = 0

[script:///opt/splunk/etc/apps/asset_discovery/bin/nmap.sh -p14147 -t 172.16.42.
64 172.16.42.220 172.16.42.230]
disabled = false
index = asset_discovery
interval = 60
source = nmap
sourcetype = port_scan

---
If this reply helps you, Karma would be appreciated.
0 Karma

mikaelbje
Motivator

Can you paste your inputs.conf?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I restarted Splunk and still am getting no port_scan data. In fact, my asset_discovery index contains nothing at all.

---
If this reply helps you, Karma would be appreciated.
0 Karma

mikaelbje
Motivator

I updated my answer. Not sure if it's any help. You might try restarting Splunk just in case.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I made the same changes and still no data. Did you have to restart Splunk?

---
If this reply helps you, Karma would be appreciated.
0 Karma

mikaelbje
Motivator

I'm having the same problem. Running nmap through nmap.sh for a port scan works in the bash shell, even as the splunk user, but nothing is added for port_scan to Splunk. Tried running Splunk as root and splunk

0 Karma

richgalloway
SplunkTrust
SplunkTrust

-rwxr-xr-x 1 root root 1972032 Jan 4 2013 /usr/bin/nmap*

Splunk is running as root so wouldn't nmap also run as root?

---
If this reply helps you, Karma would be appreciated.
0 Karma

lukejadamec
Super Champion

Is the owner:group of the nmap binary root:root

And the permissions set to 4755?
It could be a suid bit problem that you're not seeing when you run it yourself as root.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...