All Apps and Add-ons

Splunk Enterprise Trial & Splunk App for VMware Trial: "Forwarding to indexer group default-autolb-group blocked for 100 seconds."

geraldomagella
Explorer

I've deployed Splunk Enterprise, entered no license, and then I started deploying the Splunk App for VMware. Everything went just fine. I managed to follow every instruction on the documentation (deployed with the OVA). When I run the command list forward-server I get:

[root@myserver bin]# ./splunk list forward-server
Your session is invalid.  Please login.
Splunk username: admin
Password:
Active forwards:
        None
Configured but inactive forwards:
        10.1.99.11:9997

I can ping that IP, its on the same network, so no firewall between them. No firewall on Splunk server. Yep, I've clicked start scheduler.
Every time I set the configuration on the Splunk Server VMware app changing some detail on the DCN, I get the following MESSAGE on splunk web interface of the forwarder:

Forwarding to indexer group default-autolb-group blocked for 100 seconds.

I have NO idea WHY this is happening.
The result is NO DATA on my Splunk Enterprise. No data whatsoever is being sent to my Splunk indexer.

Any help?

0 Karma
1 Solution

geraldomagella
Explorer

Hello Everyone. I've found the issue, let me walk you through:
I've logged in the DCN and...

[root@splunkcollector ]# cd /home/splunkadmin/opt/splunk/var/log/splunk/
[root@splunkcollector splunk]# tail -f splunkd.log
05-05-2016 13:40:56.109 -0400 WARN  TcpOutputFd - Connect to 10.x.xx.11:9997 failed. Connection refused
05-05-2016 13:40:56.109 -0400 ERROR TcpOutputFd - Connection to host=10.x.xx.11:9997 failed
... (repeatedly)...

NICE! Finally, So apparently my splunk is not listening on por 9997, lets check:

SSH on splunk box, and run:

root@splunkserver:/# netstat -anp | grep 9997
root@splunkserver:/# 

So, nothing. Fine... log in to splunk, go to:

"Settings > Forwarding and Receiveing"
look on "Receiving Data" and click on "configure receiving.
You'll see that there is nothing there (I had nothing there, you might have it and it's disabled). Click in "NEW".
Type in the port number, in my case 9997, click save.... DONE! 🙂
Check:

root@splunkserver:/# netstat -anp | grep 9997
tcp        0      0 0.0.0.0:9997            0.0.0.0:*               LISTEN      1095/splunkd
tcp        0      0 10.X.X.11:9997         10.B.XX.12:52947        ESTABLISHED 1095/splunkd
tcp        0      0 10.X.X.11:9997         10.A.XX.12:54437        ESTABLISHED 1095/splunkd
tcp        0      0 10.X.X.11:49997        10.X.XX.12:8008         TIME_WAIT   -
root@splunkserver:/#

Note: Those were my internal IP's. B and A are different subnets on different sites/datacenters.

Sure enough data started pouring in. 🙂

Hope this help anyone in the same situation.
Cheers.

View solution in original post

0 Karma

geraldomagella
Explorer

Hello Everyone. I've found the issue, let me walk you through:
I've logged in the DCN and...

[root@splunkcollector ]# cd /home/splunkadmin/opt/splunk/var/log/splunk/
[root@splunkcollector splunk]# tail -f splunkd.log
05-05-2016 13:40:56.109 -0400 WARN  TcpOutputFd - Connect to 10.x.xx.11:9997 failed. Connection refused
05-05-2016 13:40:56.109 -0400 ERROR TcpOutputFd - Connection to host=10.x.xx.11:9997 failed
... (repeatedly)...

NICE! Finally, So apparently my splunk is not listening on por 9997, lets check:

SSH on splunk box, and run:

root@splunkserver:/# netstat -anp | grep 9997
root@splunkserver:/# 

So, nothing. Fine... log in to splunk, go to:

"Settings > Forwarding and Receiveing"
look on "Receiving Data" and click on "configure receiving.
You'll see that there is nothing there (I had nothing there, you might have it and it's disabled). Click in "NEW".
Type in the port number, in my case 9997, click save.... DONE! 🙂
Check:

root@splunkserver:/# netstat -anp | grep 9997
tcp        0      0 0.0.0.0:9997            0.0.0.0:*               LISTEN      1095/splunkd
tcp        0      0 10.X.X.11:9997         10.B.XX.12:52947        ESTABLISHED 1095/splunkd
tcp        0      0 10.X.X.11:9997         10.A.XX.12:54437        ESTABLISHED 1095/splunkd
tcp        0      0 10.X.X.11:49997        10.X.XX.12:8008         TIME_WAIT   -
root@splunkserver:/#

Note: Those were my internal IP's. B and A are different subnets on different sites/datacenters.

Sure enough data started pouring in. 🙂

Hope this help anyone in the same situation.
Cheers.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...