Getting Data In

Index from old Splunk Heavy Forwarder

emccaslin
Path Finder

Setup currently I have the newest version of Splunk (6.0) running as my main Splunk server with several universal forwarders v 6.0 sending logs to the server to be indexed.

I have another box that the v 6.0 forwarders are incompatible with so I need to install Splunk version 3.14 onto the box. I see in the documentation that I can make the full installation a heavy forwarder to push to my regular indexer, but it is not working for me.

Steps Taken:

  1. I Installed the full Splunk v 3.14 on the box I want to use a forwarder
  2. Then enabled the forwarder: ./splunk enable app SplunkForwarder -auth <username>:<password>
  3. Started forwarding activity: ./splunk add forward-server <host>:<port> -auth <username>:<password>
  4. Added deploy server: ./splunk set deploy-poll <host>:<port>
  5. Retarted splunk: ./splunk restart
  6. Waited but the forwarder never appears in the list under Forwarder Management on the Splunk Server

I assume this has something to do with the different versions of Splunk that I am using, but the documentation says:

"All indexers are backwards compatible
with any forwarder and can receive
data from any earlier version
forwarder."

Anyone else have this problem or know how to better implement this?

Documentation:

1 Solution

sciurus
Path Finder

Start with the assumption that it's compatible, and something else is broken. Check basic TCP - can you see the connection in netstat? Is it successfully connecting? If so, check splunkd.log, if not, check routes and firewalls, etc.

If it ISN'T compatible, then you've got something which is being rejected by the v6 server - in which case it will show in logs somewhere. If it IS compatible but it's being rejected due to a configuration issue, that will also show up, etc. Also deploy-poll is different to forwarding, so troubleshoot that separately.

View solution in original post

Lowell
Super Champion

Any chance that this is your issue?

http://answers.splunk.com/answers/115495/i-upgraded-my-distributed-environment-to-splunk-60-and-now-...

Basically, try negotiateNewProtocol = false

0 Karma

sciurus
Path Finder

Start with the assumption that it's compatible, and something else is broken. Check basic TCP - can you see the connection in netstat? Is it successfully connecting? If so, check splunkd.log, if not, check routes and firewalls, etc.

If it ISN'T compatible, then you've got something which is being rejected by the v6 server - in which case it will show in logs somewhere. If it IS compatible but it's being rejected due to a configuration issue, that will also show up, etc. Also deploy-poll is different to forwarding, so troubleshoot that separately.

emccaslin
Path Finder

So I eventually got this working and now I am able to get it working on multiple Windows 2000 servers. One of the main differences I noticed it working is when I enabled the SplunkLightForwarder instead of SplunkForwarder.

Also, because a compatibility issue, Splunk cannot send the configurations through a deployment app as the Universal Forwarders do. So I have to manually put the configurations in $SPLUNK_HOME/etc/system/local and restart the forwarder. Seems to be working well now.

0 Karma

sciurus
Path Finder

If it's receiving back HTML, are you sure you're pointing it to the Splunk log port (default 9997), not the management (default 8089) or user interface (default 8000)? I'm not sure why you'd get HTML back from the log port.

0 Karma

emccaslin
Path Finder

On the v6 Server in splunkd.log I am getting the following about the v3 forwarder: "DEBUG RPCDispatcher - Request from 3.x deployment client : <ip address> received. <some html code>"

I believe the forwarder is connecting to the server. I'm not seeing anything in logs on the sever that indicates incompatibility, but on the forwarder I see a message along the lines of "possible server compatibility issue". I have tried getting the forwarder to monitor a log by placing the configuration in ./etc/system/local instead of having it pull the config from the server but this is still not working.

0 Karma

emccaslin
Path Finder

Great suggestions for me to start looking for a solution.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...