How do we connect to a splunk server sitting behind an elastic load balancer in aws ?

[nandikr]

Hi All

I am trying to connect to a local instance of Splunk using the following code in java splunk sdk
loginArgs.setUsername( "qa" );
loginArgs.setPassword( "myTest" );
loginArgs.setHost( "myhost" );
loginArgs.setPort( 8089 );

    Service service = new Service( loginArgs );
    service.login();

This works fine
But the moment i try to connect to an aws instance sitting behind an elb we get connection time outs

Any particular conf settings that we need to ensure ?
Any help is appreciated

[nandikr · Accepted solution]

Finally i found out a way to connect

Steps 1) Open ports 8089 from your organisation
2) Open ports 8089 at both elb and instance level
3) Configure splunk as allowRemoteLogin = always in server.conf
4) Add enableSplunkdSSL = false in server.conf

[nandikr]

Hi All

I had checked my open ports and it seemed that 8089 was not open
However i was running into other issues

for eg.
if i use loginArgs.setScheme( "http" ); for the same 8089 port i get the following
Exception in thread "main" java.lang.RuntimeException: Unexpected end of file from server
at com.splunk.HttpService.send(HttpService.java:365)
at com.splunk.Service.send(Service.java:1268)
at com.splunk.HttpService.post(HttpService.java:243)
at com.splunk.Service.login(Service.java:1099)
at com.splunk.Service.login(Service.java:1079)
at splunk.SplunkMain.main(SplunkMain.java:47)
Caused by: java.net.SocketException: Unexpected end of file from server
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:772)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
at com.splunk.HttpService.send(HttpService.java:362)

[nandikr]

Also if i use loginArgs.setScheme("https")

i get this
Exception in thread "main" java.lang.RuntimeException: Received fatal alert: handshake_failure
at com.splunk.HttpService.send(HttpService.java:345)
at com.splunk.Service.send(Service.java:1268)
at com.splunk.HttpService.post(HttpService.java:243)
at com.splunk.Service.login(Service.java:1099)
at com.splunk.Service.login(Service.java:1079)
at splunk.SplunkMain.main(SplunkMain.java:47)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1959)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1077)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:563)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.jav...
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
at com.splunk.HttpService.send(HttpService.java:338)
... 5 more

[matthieu_araman]

Can you reach the ip from the box where your code run ? (for example ping it)
While you do the test, what are the results of netstats -an | grep 8089 ?
if you see a SYN_SENT, you've got a firewall blocking between you and the LB.
If you see ESTABLISHED, that could be between the LB and your splunk instance that there's a problem

At a last resort try telnet host 8089 from a shell

Hope that helps

[nandikr]

Hi Matthieu

I tried to do telnet through my machine

i got this
telnet: Unable to connect to remote host: Connection timed out

is there any specific configuration i need to do on aws side as i have a public ip and also i have a 8089 port forwarding.

[linu1988]

First try disabling the firewall. if not then ask your network team to look into it

telnet IP_Add 8089

it has to be successful to make it work further

[alacercogitatus]

Make sure that your ELB is public, with a public IP. If it is internal only, it won't work. I don't know your AWS settings, but you need to make sure it is reachable from your location. Verify that the ELB is set to forward 8089->8089 as well.