Getting Data In

6.2 Forwarder Configuration on Linux: Why am I getting error "TcpInputProc - Message rejected. Received unexpected 369295616 byte message!" in server's splunkd.log?

drodman29
Path Finder

Forwarder splunklog extraction -
First time manual config for a Linux box. The server is set up to listen on 9997 and makes the connection but can't complete the data forwarding. Forwarder Management App on Server lists 0 forwarders have phoned home.

Any ideas?

11-24-2014 13:21:35.851 -0500 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake me
ssage to DS; err=not_connected
11-24-2014 13:21:36.903 -0500 WARN  HttpPubSubConnection - Unable to parse message from PubSubSvr:
11-24-2014 13:21:36.903 -0500 INFO  HttpPubSubConnection - Could not obtain connection, will retry after=37 seconds.
11-24-2014 13:21:47.689 -0500 INFO  TcpOutputProc - Connected to idx=[MyServerIP]:9997
11-24-2014 13:21:47.851 -0500 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake me
ssage to DS; err=not_connected
11-24-2014 13:21:59.852 -0500 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake me
ssage to DS; err=not_connected
11-24-2014 13:22:11.852 -0500 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake me
ssage to DS; err=not_connected
11-24-2014 13:22:13.905 -0500 WARN  HttpPubSubConnection - Unable to parse message from PubSubSvr:
0 Karma
1 Solution

drodman29
Path Finder

OK, I'm answering my own question, which just makes me look dumb. But the Deployment client config got wired to the server data receiving port instead of the server management port.

View solution in original post

Rocket66
Communicator

This error message also occurs, when enable SSL on the forwarder, but have Non-SSL inputs (splunktcp://9997 instead of splunktcp-ssl:9997) on the indexer. Don't forget to add the SSL stanza in the inputs.conf on the indexer side.

http://wiki.splunk.com/Community:Splunk2Splunk_SSL_DefaultCerts

woodcock
Esteemed Legend

This should be the accepted answer.

0 Karma

drodman29
Path Finder

OK, I'm answering my own question, which just makes me look dumb. But the Deployment client config got wired to the server data receiving port instead of the server management port.

joel_roberts
Engager

don't feel dumb! I had the exact same problem and you helped me!

hall_ronald
Explorer

setup the deploy poll functionality:

splunk set deploy-poll <host>:<port>

ensure the port is the management port on the server (default is 8089) not the receiver listening port (default 9997). Check this in $SPLUNK_HOME/etc/system/local/deployment-client.conf.

forwarder still goes to 9997 (or whatever the port you have set the receiver to):

splunk add forward-server <host>:<port> -auth <username>:<password>

... provided by gethyn85, Problem in setting up forwarder and reciever ( Received unexpected 369295360 byte message)

delcorral
New Member

I experience this issue too.

My current setup is:
1 Centos: NGINX Loadbalancer + Splunk Universal forwarder 6.5
Configurations locations $SPLUNK_HOME/etc/apps//local/
inputs.conf

[monitor:///var/log/nginx/acc*]
disabled = false
index = internal_loadbalancer
sourcetype = nginx:plus:access
whitelist = access.log(-|\.)\d*
ignoreOlderThan = 30h


[monitor:///var/log/nginx/e*]
disabled = false
index = internal_loadbalancer
sourcetype = nginx:plus:error
whitelist = error.log(-|\.)\d*
ignoreOlderThan = 30h

output.conf

[tcpout]
defaultGroup=indx

[tcpout:indx]
disabled=false
server=<indexIP>:9997  
compressed=true
sendCookedData=ttue
autoLB=true

1 Centos: Splunk Enterprise 6.5

netstat -an | grep 9997
tcp        0      0 0.0.0.0:9997                0.0.0.0:*                   LISTEN      
tcp        0      0 10.0.10.6:9997              10.0.10.5:56079             ESTABLISHED 
0 Karma

ppablo
Retired

Hey @drodman29

It doesn't make you look dumb. Answering and accepting your own answer after finding the solution helps other users on here that are coming across similar/identical issues. It's better than just leaving it open without a possible troubleshooting point, so thanks for resolving this post 🙂

Patrick

drodman29
Path Finder

Additional info:
Server side splunkd.log has this:
11-24-2014 13:48:10.962 -0500 ERROR TcpInputProc - Message rejected. Received unexpected 369295616 byte message! from src={MyClientIP]:36189. Maximum message allowed: 67108864. (::)

boyanmilushev
Explorer

I was getting the same error message when I tried to ingest data from CheckPoint LogExporter Log Server to Intermediate Heavy Forwarder.

I tried to use port numbers below 1024 and 9997.

It is not mentioned anywhere in the documentation that you can't use these ports to ingest data from LogExporter to Splunk.

It turned-out that you can't use port numbers below 1024 if you are not running as "root" or "root privileges". You can't also use port 9997, because it's reserved for "cooked" data ingestion from Splunk Forwarder to indexer or heavy forwarder.

In the end, I chose port 18188 and it worked.

I hope that this info helps someone who runs into the same problem as I did.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...