Docker Splunk logging driver per line 4K byte limit.

[kyaparla]

We are using splunk logging driver to send docker container logs and some containers are sending log messages >4K bytes in a single line (like stack traces). We see them getting truncated in Splunk and want to check if any one aware of where the 4K limit is coming from?

Docker sends all logs to stdout and they get picked by logging driver, I saw in some linux forums that stdout pipe buffer limit is 4K, does anyone have more details on this how to resolve it?

[outcoldman · Accepted solution]

Hi @kyaparla

Author of the Splunk Logging Driver here.

The messages are getting split inside the copier in the docker daemon itself. https://github.com/moby/moby/blob/dfc2d62632d32f9d38166ea477f0ca033a5c91c2/daemon/logger/copier.go

Seems like change is coming which will increase the limit much higher (80k) https://github.com/moby/moby/pull/35617

Meanwhile, I would suggest you look at the solutions which we have built for monitoring Docker. It is a certified application https://splunkbase.splunk.com/app/3723/ and our custom built collector for docker (kubernetes and openshift), which can collect logs, deal with multiline events and have a lot of other benefits. You can see a list of benefits compared to the Splunk Logging Driver https://www.outcoldsolutions.com/docs/collectorfordocker/

[kyaparla]

Thanks @outcoldman, This helps a lot, we will wait new version of docker. Will also look at the app, it seems to resolve many of our issues.

Does the app require any docker volumes to be mounted to read log files?

[kyaparla]

Hi @outcoldman,

I am just reaching to you, if you are aware of recent updates with splunk logging driver? We are seeing that 4K char limit is not an issue any more when splunk-format is inline, but the raw format still has this issue. Appreciate, if you could share any updates or provide some pointers.

[outcoldman]

I have seen that the limit has been increased for all logging drivers. But not sure, why do you see different behavior on different types of logging formats, as this is handled by the driver itself.
As I have mentioned in my previous reply - you can try our solution, that does not have this problem.

[outcoldman]

@kyaparla, that is correct, you can take a look on the manual how to install the application and our collector https://www.outcoldsolutions.com/docs/monitoring-docker/ , our collector is built on top of json-file driver, so you just map the containers folder and some other for metrics. The best part, that you will be able to use docker logs, while you will be able to forward all logs. Also, in case of network failures, your logs will be kept on disk, so when the network issues will be resolved - you will index all the logs from the right place, without losing any information (at least once delivery).