Splunk Search

What would be the best approach to write a regex for dynamic sourcetypes?

fisuser1
Contributor

We recently instrumented our OpenShift environment to index data into Splunk. I'm looking for the best approach for extracting fields with dynamic sourcetype names out of ocp. the data format will remain consistent per sourcetype, but I just need to key on two set's of sourcetypes, batch and soap. Once I define the structure of each, we can apply this field extraction to any new sourcetypes created out of openshift (which may not happen often, but still want to prepare). I figure I can key on kube:container: in the beginning and -soap-app at the end of each?

Soap app Sourcetypes:
kube:container:organization-soap-app
kube:container:customer-soap-app

kube:container:our-us-app-internal-soap-app
kube:container:internal-regulations-soap-app

Batch app Sourcetypes:
kube:container:our-us-app-internal-batch-app
kube:container:customer-batch-app
kube:container:organization-batch-app
kube:container:compliance-regulations-batch-app
kube:container:internal-regulations-batch-app

0 Karma

jkat54
SplunkTrust
SplunkTrust

props.conf:

[mysourcetype]
TRANSFORMS-mytransforms = mytransforms

transforms.conf:

[mytransforms]
SOURCE_KEY = MetaData:Source
REGEX = kube:container:(\S+)-(soap|batch)-app
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::$2-$1

Should take something with sourcetype of "mysourcetype" and source of "kube:container:organization-soap-app" and give it a final sourcetype of "soap-organization". You could add more "|KEYWORDS" in the 2nd capture group of the regex.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

IMO, if you have two or more sources with the same structure and same field extractions, then they're all of the same sourcetype. If you need to distinguish them for some reason, use source or the presence of a certain string (like "-soap-app").

---
If this reply helps you, Karma would be appreciated.
0 Karma

fisuser1
Contributor

maybe this?

kube:container:([^\s]+)-soap-app
and
kube:container:([^\s]+)-batch-app

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...