Hello,
we are successfully logging events from Docker hosts via Splunk adapter (HEC) into splunk.
Problems arise when we try to filter some unwanted events.
I tried to send those events to the NULL-Queue
props.conf
[host::<somehost>]
TRANSFORMS-nullQueueFilterSpecificEvents = filter_setNull_4_specific_events
transforms.conf
[filter_setNull_4_specific_events]
REGEX = <someregex>
DEST_KEY = queue
FORMAT = nullQueue
It doesn't work. Still getting all events unfiltered. One possible reason could be, that depending on the HEC endpoint the events may go through a "structured parsing queue" instead of the normal route and that transformations aren't possible.
Is this a possbiel explanation?
How can I detemine what endpoint is used?
Perhaps the log config in the docker-compose.yml can give some clarification:
logging:
driver: splunk
options:
tag: "ct={{.Name}}"
splunk-token: <sometoken>
splunk-url: https://<splunkhost-fqdn>:<port>;
splunk-format: raw
splunk-insecureskipverify: "true"
The events are not json but unstructured.
Can someone give me a hint or did anyone already successfully filter docker events with transformations in props.conf/transforms.conf.
Thanks in advance!
Regards,
Jens
Docker Splunk Logging driver sends events to the /event (https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTinput#services.2Fcollector.2Fevent) endpoint. And only /raw (https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTinput#services.2Fcollector.2Fraw) HEC endpoint supports indexing pipeline (see http://dev.splunk.com/view/event-collector/SP-CAAAE6P).
Event endpoint expects that you will control the delivery on your end. Unfortunately Splunk Logging Driver does not have support for /raw endpoint.
With the collectord (Monitoring Docker application https://splunkbase.splunk.com/app/3723/) we support filtering out the events base on the specific regexp pattern, see example https://www.outcoldsolutions.com/docs/monitoring-docker/v5/annotations/#example-2-dropping-messages
Docker Splunk Logging driver sends events to the /event (https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTinput#services.2Fcollector.2Fevent) endpoint. And only /raw (https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTinput#services.2Fcollector.2Fraw) HEC endpoint supports indexing pipeline (see http://dev.splunk.com/view/event-collector/SP-CAAAE6P).
Event endpoint expects that you will control the delivery on your end. Unfortunately Splunk Logging Driver does not have support for /raw endpoint.
With the collectord (Monitoring Docker application https://splunkbase.splunk.com/app/3723/) we support filtering out the events base on the specific regexp pattern, see example https://www.outcoldsolutions.com/docs/monitoring-docker/v5/annotations/#example-2-dropping-messages
Just as I thought. Thank you outcoldman!