Getting Data In

ERROR ExecProcessor - message from "/opt/splunk/etc/apps/ta-dockerstats/bin/docker_stats.sh" Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

eygtmbot
Engager

Hi All,

We are using Docker Swarm on Ubuntu 16.04
I'm starting my forwarder stack with below YML file.

version: '3'

services:
  splunk-forwarder:
    image: splunk/universalforwarder:7.0.0-monitor
    ports:
      - "514:1514/udp"
    deploy:
      replicas: 2
    environment:
      SPLUNK_START_ARGS: --accept-license --answer-yes
      SPLUNK_FORWARD_SERVER: "server:9997"
      SPLUNK_USER: root
    volumes:
      - opt-splunk-etc:/opt/splunk/etc
      - opt-splunk-var:/opt/splunk/var

volumes:
  opt-splunk-etc:
  opt-splunk-var:

After running stack deploy

sudo docker stack deploy -c stack-file.yml kiran

All the containers are propagated with the following error, can you please take a look?

04-07-2018 00:52:17.485 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/ta-dockerstats/bin/docker_service.sh" /opt/splunk/etc/apps/ta-dockerstats/bin/docker_service.sh: line 10: docker: command not found
04-07-2018 00:52:17.505 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/ta-dockerstats/bin/docker_events.sh" Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
04-07-2018 00:52:17.506 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/ta-dockerstats/bin/docker_top.sh" Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

What I'm missing? I'm running this on an Ubuntu 16.4 machine.

Is that a permission issue?

I already added ENV SPLUNK_USER: root

I tried sudo usermod -aG docker $USER but that didn't help.

Thanks in advance
Kiran

0 Karma

outcoldman
Communicator

You do not mount /var/run/docker.sock to the container from the host, also few more folders, you can find them in the docs https://hub.docker.com/r/splunk/universalforwarder/

 --volume /var/lib/docker/containers:/host/containers:ro \
 --volume /var/log:/docker/log:ro \
 --volume /var/run/docker.sock:/var/run/docker.sock:ro \

I see that this page also has a docker-compose example, but seems like this example is for general forwarder, not the forwarder with the scripts to collect docker metrics.


If you don't mind paid solutions, I would also recommend taking a look at our solution for Monitoring Docker https://www.outcoldsolutions.com. We have a free 30 days trial, instructions on how to install it https://www.outcoldsolutions.com/docs/monitoring-docker/

0 Karma

eygtmbot
Engager

I didn't try this but I made it work through the Splunk driver instead of syslog. May I know which one is the splunk recommendation. Splunk Forwarder or HTTP listener?

Thanks,
Kiran

0 Karma

outcoldman
Communicator

Splunk Docker Driver is great for a start, but it has some limitations and its issues. As an example, it is not resistant to failures, in case of network failures, it will retry several times, but will give up at some point.
But if you will install Heavy Weight Forwarder as a side container on every host and use it as a target for your logs - you will reduce the possibility of network issues between the logging driver and Splunk. And Splunk itself knows how to buffer data before forwarding.
Using syslog in case if you can use Splunk Logging Driver does not make a lot of sense, because in case of Splunk logging driver you get more Splunk-friendly format, also can add pre-indexed fields from container labels. But there can be situations, where you want to use syslog - if you want to reassemble multiline messages into one.

Saying that - still recommend to look on our solution https://www.outcoldsolutions.com, as we deal with all of these problems, and give you application monitoring on top of that.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...