Monitoring Splunk

TcpInputProc - Received unexpected message

joonradley
Path Finder

This error keeps repeating in the error logs, but I have no idea what is causing it.

02-15-2011 14:55:31.161 ERROR TcpInputProc - Received unexpected 68021378 byte message! from hostname=tchuxxx.xxxx.com, ip=10.xx.xx.xx, port=50563

Is this related to the size of the message?

Thx

Tags (1)

jrodman
Splunk Employee
Splunk Employee

Essentially yes, it's saying that you got a big message. Since a 68MB data item is highly unlikely, there was probably some breakage in the datastream.

The protocol for splunk->splunk forwarding includes a length indicator number, which causes the receiving code to allocate memory. To avoid breaking the receiving Splunk, it does not blindly trust the size, but for cases of very large length numbers logs the problem and does not allocate the memory.

This could be a case where the forwarder is encountering some kind of memory corruption bug, where something is communicating to a splunktcp:// socket which is not quite conformant (hard to imagine, but possible), or when the stream of bytes in the tcp socket is getting messed up via some other means.

We had a known problem with early versions of 4.0.x and late versions of 3.4.x where the forwarder would sometimes inject 'heartbeat' pseudo-messages in the middle of other messages, corrupting the datastream. You may want to evaluate if tchuxxx.xxxx.com may be running an older version of splunk.

0 Karma

sf_user_199
Path Finder

Quick old-issue CPR...

We have this issue with a search head summarizing data & sending it back to our indexers. All the indexers are 5.0.2, as is the search head.

0 Karma

joonradley
Path Finder

The oldest version on the forwarders are 4.1.3.

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to May Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars this month. This ...

They're back! Join the SplunkTrust and MVP at .conf24

With our highly anticipated annual conference, .conf, comes the fez-wearers you can trust! The SplunkTrust, as ...

Enterprise Security Content Update (ESCU) | New Releases

Last month, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...