Why can't I see my new forwarder in the Splunk UI?

[emechling]

Hello, I'm a new Splunk user.

I have configured a Splunk server with 2 Windows forwarders.

Now, I want to set up a Linux forwarder in order to send log messages from /var/log/syslog.

I downloaded and installed the Splunk forwarder to the client.

Then i use the commands :

[root]# /opt/splunkforwarder/bin/splunk start --accept-license
[root]# /opt/splunkforwarder/bin/splunk add forward-server 192.168.107.15:9997[root]# /opt/splunkforwarder/bin/splunk add monitor /var/log/syslog/
Added monitor of '/var/log/syslog'.
[root]# /opt/splunkforwarder/bin/splunk list forward-server
Active forwards:
        192.168.107.15:9997
Configured but inactive forwards:
        None

I restarted both the server and the client, however I can't see any new forwarder in the Splunk UI.

Here my client's splunkd.log :

12-12-2018 12:21:13.435 +0100 INFO  TcpOutputProc - Connection to 192.168.107.15:9997 closed. Connection closed by server.
12-12-2018 12:21:13.436 +0100 WARN  TcpOutputFd - Connect to 192.168.107.15:9997 failed. Connection refused
12-12-2018 12:21:13.436 +0100 ERROR TcpOutputFd - Connection to host=192.168.107.15:9997 failed
12-12-2018 12:21:13.436 +0100 WARN  TcpOutputProc - Applying quarantine to ip=192.168.107.15 port=9997 _numberOfFailures=2
12-12-2018 12:21:33.742 +0100 INFO  TcpOutputProc - Removing quarantine from idx=192.168.107.15:9997
12-12-2018 12:21:33.745 +0100 INFO  TcpOutputProc - Connected to idx=192.168.107.15:9997, pset=0, reuse=0.

root@srv-virtuel2:~# netstat -ano | grep 9997

tcp 0 0 192.168.107.2:49408 192.168.107.15:9997 ESTABLISHED off (0.00/0/0)
inputs.conf (client):

[default]
host = srv-virtuel2
[monitor:///var/log/syslog]
_TCP_ROUTING = default-autolb-group

outputs.conf (client):

[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = 192.168.107.15:9997
[tcpout-server://192.168.107.15:9997]

Can you help me ?
Thanks in advance.

[Rhin0Crash]

Echoing dkeck, you'll want to make sure your network connections are properly configured to allow traffic between the systems. You'll also want to run a forwarder command to manage your forwarder from your deployment server (why you aren't seeing anything in Settings --> Forwarder Management)

If you'd like some example commands for installations, you can reference some that I've built:
https://github.com/ryanadler/splunkInstalls

FYI also, best practice is not to run the forwarder as root on the Linux box, but instead to make a non-priv user for Splunk to run as.

(assumes the same IP is being used):

./splunk set deploy-poll 192.168.107.15:8089

[dkeck]

Hi,

looks like your Indexer 192.168.107.15:9997 is refusing the connection, make sure that port 9997 is not blocked and recieving is enabled,

[emechling]

Sorry, I was wrong. My clients do not appear by going to splunk-> settings -> forwarders.
But the sources are well listed by going over the summary of the data in the search part of splunk.
Thanks.

[dkeck]

please accept the answer if it was helpful 🙂

[emechling]

I can telnet to that port from client from the network and from that linux client. Receiving is enabled and the port not blocked.
Otherwise 2 Windows forwarders works and are connected to the server.
The connection is established in TCP.