All Apps and Add-ons

Splunk Connect for Kubernetes multiline events,Kubernete Splunk Connect multiline logs

KarunK
Contributor

Hi,

I am trying to use the Splunk connect fro Kubernetes to eextarct the kube logs. However i cannot get the multiline event parsing working. Config i have used is as below.

  multiline:
    firstline: "\d+"
    flushInterval 5s

When i check the fluentd config i can see the following related config with an extra "\".

format_firstline "/^\\d+/"

Not sure why there is an extra "\". This is causing issues for me as the regex is not getting applied.

Anyone had faced similar issues ?

Thanks

Regards

KKN

0 Karma

mattymo
Splunk Employee
Splunk Employee

did you solve this?

- MattyMo
0 Karma

mattymo
Splunk Employee
Splunk Employee

HI KarunK!

we have some examples of applying filters to specific pods in the manifest located here:

https://github.com/splunk/splunk-connect-for-kubernetes/blob/45a0be058f503d415149de4fdd39a2a365f6920...

in the output.conf file in the configmap, we created some sample multiline configs using the concat filter ( https://github.com/fluent-plugins-nursery/fluent-plugin-concat -- see the multiline plugin also) for internal Kubernetes logs, for example:

   <filter tail.containers.var.log.containers.dns-controller*dns-controller*.log>
            @type concat
            key log
            timeout_label @SPLUNK
            stream_identity_key stream
            multiline_start_regexp /^\w[0-1]\d[0-3]\d/
            flush_interval 5s
          </filter>

The important part is to filter on the tag, which in the case of Splunk Connect for Kubernetes is pre-pended with tail.containers + the path of the in_file log you are trying to filter. (see https://docs.fluentd.org/v1.0/articles/life-of-a-fluentd-event for more on how tags are created)

in this example anything in /var/log.containers/dns-controller.

See this article about how to use wildcards:

https://docs.fluentd.org/v0.12/articles/config-file#how-match-patterns-work

Give this a try and let me know if you get it working!

- MattyMo

KarunK
Contributor

The reason I explained above is not the cause of the issue. Even in my local fluentd, the logs shows that an extra "\" will get applied. So not the cause of it.

After further ;looking into the config maps of the Kubernet connect I have noticed that all the /var/log/containers/*.log configs are hard coded and does not support the multiline. The multiline option is only for other logs which are not part of the /var/log/containers/ folder.
If anyone else have any ideas of doing it please let me know.

Thanks

kin

0 Karma

mattymo
Splunk Employee
Splunk Employee

I am doing multiline successfully using the latest charts, please ping me if you are still having trouble.

- MattyMo
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...