Are any Fluentd apps Splunk vetted/supported? Or is there a preferred cloud-native solution for logging Kubernetes logs?

[mcluver]

We’re looking to get our Kubernetes logs into Splunk and it appears the best (most cloud native) way to do that is to forward the logs from Fluentd to Splunk HEC (HTTP Event Collector). With that being said, we see where there are a number of plugins that people have developed for Fluentd for this use-case, see: Fluentd Plugins Could you guys please tell us if any of these were developed by Splunk employees or are officially vetted/supported?

Does Splunk have another cloud native solution that they recommend instead? Don’t say the UF (Splunk Universal Forwarder). I also found this Splunk Answers post regarding the same topic for a bit of background on what others were doing cloud natively. Thanks for any assistance with this question.

Thanks & Best regards,
Matt

[epeterfi_splunk]

To monitor Kubernetes please go here and follow the steps: https://github.com/splunk/splunk-connect-for-kubernetes

[outcoldman]

One solution to monitor Kubernetes and collect all logs is to use our collector

SplunkBase: https://splunkbase.splunk.com/app/3743/
Youtube Demo https://www.youtube.com/watch?v=C2zOO2XX5TI
Installation/configuration instructions https://github.com/outcoldsolutions/collector/tree/master/kubernetes

With provided configuration it will automatically pick up all logs, enrich them with kubernetes metadata and ship it to Splunk.

Let me know if you will have any questions.

Edited (2017-10-05): posted link to published application on splunkbase

[mcluver]

That looks like a really nice solution for Kubernetes logging and metrics. Probably the best solution to date for Splunk I've seen. Is Splunk not planning on releasing their own TA for this platform at some point?

Being a Splunk Enterprise Security customer I think we'd be interesting in the collection being CIM compliant as well.

[epeterfi_splunk]

See below my comment. Connect for K8 was launched months ago and works great. Also comes with an app.

[outcoldman]

CIM compliant is on our radar for "Monitoring Kubernetes" and "Collector for Kubernetes".

[sduff_splunk]

Why are you dismissing the Universal Forwarder? It is as "cloud native" as anything else you might find?

[mcluver]

We feel the Splunk Forwarder is more for host/node level data collection and that's not the way we were planning to log our Kubernetes infrastructure. If one were to want to write all logs back to the host/node level, then yes the Splunk Forwarder would work fine and we use it in the cloud at that level significantly already for many other workloads. In our opinion, the Splunk HEC is much more well suited to the task of collecting logs from something like Kubernetes/Docker which should be more directly from the logging driver or container engine level.

The official k8s logging documentation mentions several different logging approaches, with node-level and cluster-level being the two main parent categories. Cluster-level is the one that we believe is the better approach and why we're looking for a different solution than the Splunk Forwarder in this space.