port connectivity issue "Connection refused"

[seema2502]

Hi Team,

We are facing port connectivity issue since 5th Sep 2014 between indexer and forwarder :-

$ telnet forwarder IP port
Trying forwarder IP...
telnet: connect to address forwarder IP: Connection refused
telnet: Unable to connect to remote host: Connection refused

It is throwing same error while checking connection from forwarder to indexer :-

$ telnet indexer port
Trying indexer IP...
telnet: connect to address indexer IP: Connection refused
telnet: Unable to connect to remote host: Connection refused

Please suggest.

Thanks

[seema2502]

Hi Ayn,

Please refer below link for what changed on September 5th.
http://answers.splunk.com/answers/169028/licensing-window-alerts-on-my-indexer-caused-splun.html

[grijhwani]

Not wishing to seem dismissive, but that does not sound like a fault within the Splunk realm. It is a network infrastructure problem of some sort, but beyond that any assistance that might be suggested is of a network and systems administration nature, and requires a lot more knowledge of your infrastructure as a whole.

Your example commands and responses indicate a Unix type platform, and if that is a common Linux distribution, then the command

sudo netstat -pant | grep -i listen

on the indexer and on the forwarder should at least give you some indication of the port statuses on each. Really, though, the topic of network fault disagnosis is well outside this forum. Provided Splunk is running all other questions really fall to matters of administration like changes of IP addresses, on-server firewalling - iptables - and network firewalls or faults. You would be better served treating it, in the first instance, as a generic service connection fault and taking your question to a more appropriate board.

[seema2502]

Hi grijhwani,
We also thought that this issue is related to network but we contacted network team they have responded like :-
"Connection refused" means that destination server is not listening on particular port. This is not a NW/FW issue. Application that would normaly respond on those ports is not running or malfunctioning."

when we tried the mentioned command sudo netstat -pant | grep -i listen we received "xyz is not in the sudoers file. This incident will be reported."

Please suggest.

[grijhwani]

I was assuming that you as you were attempting to administer Splunk that you were also an administrator of the system running Splunk. If you are not then you need to hand the problem to whoever is, since they should have the experience and the authority needed on the servers to investigate it.

As for the network group's reply, I find that tends to be the stock answer from any network admin until you are waving empirical evidence under his nose.

[dchima]

hi grijhwani yes i am the adminstrator for our splunk sandbox and do have command line access as well.

The System Admin who manages the linux VM that will push into my Splunk instance said he was seeing firewall issues.

On my splunk instance, do i need to start any listeners or anything like that on ports 8088 and 9997?

[Ayn]

Well uh...firewall problems? It's really impossible to say anything more without more details. What happened on September 5th that caused this issue to start occurring?