Getting Data In

Filtering events from Docker

JenWun
New Member

Hello,

we are successfully logging events from Docker hosts via Splunk adapter (HEC) into splunk.
Problems arise when we try to filter some unwanted events.
I tried to send those events to the NULL-Queue

props.conf

[host::<somehost>]
TRANSFORMS-nullQueueFilterSpecificEvents = filter_setNull_4_specific_events

transforms.conf

[filter_setNull_4_specific_events]
REGEX = <someregex>
DEST_KEY = queue
FORMAT = nullQueue

It doesn't work. Still getting all events unfiltered. One possible reason could be, that depending on the HEC endpoint the events may go through a "structured parsing queue" instead of the normal route and that transformations aren't possible.
Is this a possbiel explanation?
How can I detemine what endpoint is used?

Perhaps the log config in the docker-compose.yml can give some clarification:

logging:
  driver: splunk
  options:
    tag: "ct={{.Name}}"
    splunk-token: <sometoken>
    splunk-url: https://<splunkhost-fqdn>:<port>;
    splunk-format: raw
    splunk-insecureskipverify: "true"

The events are not json but unstructured.
Can someone give me a hint or did anyone already successfully filter docker events with transformations in props.conf/transforms.conf.
Thanks in advance!

Regards,
Jens

0 Karma
1 Solution

outcoldman
Communicator

Docker Splunk Logging driver sends events to the /event (https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTinput#services.2Fcollector.2Fevent) endpoint. And only /raw (https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTinput#services.2Fcollector.2Fraw) HEC endpoint supports indexing pipeline (see http://dev.splunk.com/view/event-collector/SP-CAAAE6P).
Event endpoint expects that you will control the delivery on your end. Unfortunately Splunk Logging Driver does not have support for /raw endpoint.

With the collectord (Monitoring Docker application https://splunkbase.splunk.com/app/3723/) we support filtering out the events base on the specific regexp pattern, see example https://www.outcoldsolutions.com/docs/monitoring-docker/v5/annotations/#example-2-dropping-messages

View solution in original post

0 Karma

outcoldman
Communicator

Docker Splunk Logging driver sends events to the /event (https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTinput#services.2Fcollector.2Fevent) endpoint. And only /raw (https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTinput#services.2Fcollector.2Fraw) HEC endpoint supports indexing pipeline (see http://dev.splunk.com/view/event-collector/SP-CAAAE6P).
Event endpoint expects that you will control the delivery on your end. Unfortunately Splunk Logging Driver does not have support for /raw endpoint.

With the collectord (Monitoring Docker application https://splunkbase.splunk.com/app/3723/) we support filtering out the events base on the specific regexp pattern, see example https://www.outcoldsolutions.com/docs/monitoring-docker/v5/annotations/#example-2-dropping-messages

0 Karma

JenWun
New Member

Just as I thought. Thank you outcoldman!

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...