Hello Everyone. I've found the issue, let me walk you through:
I've logged in the DCN and...
[root@splunkcollector ]# cd /home/splunkadmin/opt/splunk/var/log/splunk/
[root@splunkcollector splunk]# tail -f splunkd.log
05-05-2016 13:40:56.109 -0400 WARN TcpOutputFd - Connect to 10.x.xx.11:9997 failed. Connection refused
05-05-2016 13:40:56.109 -0400 ERROR TcpOutputFd - Connection to host=10.x.xx.11:9997 failed
... (repeatedly)...
NICE! Finally, So apparently my splunk is not listening on por 9997, lets check:
SSH on splunk box, and run:
root@splunkserver:/# netstat -anp | grep 9997
root@splunkserver:/#
So, nothing. Fine... log in to splunk, go to:
"Settings > Forwarding and Receiveing"
look on "Receiving Data" and click on "configure receiving.
You'll see that there is nothing there (I had nothing there, you might have it and it's disabled). Click in "NEW".
Type in the port number, in my case 9997, click save.... DONE! 🙂
Check:
root@splunkserver:/# netstat -anp | grep 9997
tcp 0 0 0.0.0.0:9997 0.0.0.0:* LISTEN 1095/splunkd
tcp 0 0 10.X.X.11:9997 10.B.XX.12:52947 ESTABLISHED 1095/splunkd
tcp 0 0 10.X.X.11:9997 10.A.XX.12:54437 ESTABLISHED 1095/splunkd
tcp 0 0 10.X.X.11:49997 10.X.XX.12:8008 TIME_WAIT -
root@splunkserver:/#
Note: Those were my internal IP's. B and A are different subnets on different sites/datacenters.
Sure enough data started pouring in. 🙂
Hope this help anyone in the same situation.
Cheers.
... View more