Getting Data In

Extract index from filename in inputs.conf

lindsaylandry
Engager

We have a splunkforwarder DaemonSet in Kubernetes, which is forwarding node logs to our splunk server.

We want to take the STDOUT logs from each container, located in /var/log/containers/*.log, and index by the namespace specified in the filename. Is there a way to do this?

Filenames look as follows:

/var/log/containers/<pod-name>_<namespace>_<some-hash>.log

We'd like to set the index in inputs.conf by extracting the middle namespace from these files. I know there is a host_regex that will dynamically set the host, but I haven't found an equivalent for index.

0 Karma
1 Solution

micahkemp
Champion

transforms.conf:

[indexfromsource]
SOURCE_KEY = MetaData:Source
DEST_KEY = _MetaData:Index
REGEX = /var/log/containers/<pod-name>_(<namespace>)_<some-hash>\.log
FORMAT = $1

props.conf:

[<sourcetype name>]
TRANSFORMS-indexfromsource = indexfromsource

Note: the regex is not valid, as I don't know how <pod-name>, <namespace>, <some-hash> will be formatted.

View solution in original post

micahkemp
Champion

transforms.conf:

[indexfromsource]
SOURCE_KEY = MetaData:Source
DEST_KEY = _MetaData:Index
REGEX = /var/log/containers/<pod-name>_(<namespace>)_<some-hash>\.log
FORMAT = $1

props.conf:

[<sourcetype name>]
TRANSFORMS-indexfromsource = indexfromsource

Note: the regex is not valid, as I don't know how <pod-name>, <namespace>, <some-hash> will be formatted.

lindsaylandry
Engager

this is good for the server when it gets the data, but is there a way to change the index on the universal forwarder side?

0 Karma

micahkemp
Champion

There is not. You would need to place this configuration on the first heavy forwarder or indexer that sees the data.

0 Karma

gwalford
Path Finder
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...