Splunk Enterprise

How to extract fields at index time?

paulmilbank
New Member

We have .net logs from SeriLog and we would like to break it down into key value pairs at index time and extract some fields.
I have tried to follow the splunk guides and blog posts, but my indexed fields are not available. I can't post links yet unfortunately.
transforms.conf:

[SerilogKVPairs]
DELIMS = "{,}", ":"

[LogLevel]
REGEX = ^(?:[^ \n]* ){3}(?P<LogLevel>[^ ]+)

props.conf:

# Extract fields from Serilog log inputs
TRANSFORMS-KVPairs= SerilogKVPairs
TRANSFORMS-LogLevel= LogLevel

fields.conf:

[SerilogKVPairs]
INDEXED=true

[LogLevel]
INDEXED=true

if I search with a pipe to kv SerilogKVPairs it all works, I have searchable values from my Serilog files.
But the fields are not available in the UI unless I pipe it through kv SerilogKVPairs.
We would like them to be available on all logs without having to pipe through the KV command.
Loglevel does not seem to be extracted either.

Is there a log which shows what is going on here?
Thanks
Paul

Tags (1)
0 Karma
1 Solution

paulmilbank
New Member

That is the guide I followed today and it has enabled me to sort this out now. Thanks very much.

0 Karma

paulmilbank
New Member

Final config looks like this if anyone ever needs it:
transforms.conf:

[SerilogKVPairs]
DELIMS = "{,}", ":"

[LogLevel]
REGEX = ^(?:[^ \n]* ){3}(?P<loglevel>[^ ]+)
WRITE_META = true

fields.conf:

[LogLevel]
INDEXED=true

props.conf:

# Extract fields from Serilog log inputs
REPORT-SerilogKVPairs= SerilogKVPairs
TRANSFORMS-LogLevel= LogLevel
0 Karma

paulmilbank
New Member

I have managed to get loglevel working as a field now. I am still unsure how to get the KV pairs to be extracted and available without needing to pipe search through KV.

0 Karma

paulmilbank
New Member

This is now resolved from the look of things. I changed TRANSFORMS to REPORTS for the KV Delimiter, Removed it from fields.conf and it seems to be working now.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...