How to split a field into multiple fields via sourcetype?

[AnujaJadhav2]

I have a a huge message field with the format: field1=value1,field2=value2......fieldn=valuen. This field is not getting extracted by Splunk automatically.

Is there a solution on how to get this field extracted into multiple fields with these values? I tried to edit sourcetype for my message field with this regex (\w+)=([^,]+)* but it didnt work.

I want to write a regex that captures value1 and names it as field1 and so on for all fields. I want the field names to be dynamically used as I do not know the names of all fields.

I do not want to use rex (as I saw in some other solutions) as this is a permanent requirement and not limited to the search query.

I also looked if I can do some things from transforms.conf. So far no luck. 😞

I am using a Splunk Kafka plugin to ingest logs from Kafka to Splunk. I am also using REGEX filter on logs to get only purchase events.

Props.conf
[kafka:topicEvent]
SHOULD_LINEMERGE = true
NO_BINARY_CHECK = true
CHARSET = AUTO
MAX_TIMESTAMP_LOOKAHEAD = 24
disabled = false
TIME_FORMAT = %FT%T.%3N%Z
TIME_PREFIX = "@timestamp":"
INDEXED_EXTRACTIONS = json
TRANSFORMS-changesourcetype = changesourcetype_to_ kafka
TRANSFORMS-override_kafka_host = override_kafka_host
TRANSFORMS-set = setnull,purchase,extraction
TRUNCATE = 15000

Transforms.conf
[changesourcetype_to_kafka]
REGEX = "@timestamp":"
FORMAT = sourcetype::kafka
DEST_KEY = MetaData:Sourcetype

[override_kafka_host]
REGEX = "host":"([^"]+)
FORMAT = host::$1
DEST_KEY = MetaData:Host

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[purchase]
REGEX = api=\"(ActionMenu|Buy|Create)\".*LIVE
DEST_KEY = queue
FORMAT = indexQueue

[extraction]
SOURCE_KEY = message
REGEX = (\w+)=([^,]+),
FORMAT = $1::$2

[AnujaJadhav2]

I am using a Splunk Kafka plugin to ingest logs from Kafka to Splunk. I am also using REGEX filter on logs to get only purchase events.

Props.conf
[kafka:topicEvent]
SHOULD_LINEMERGE = true
NO_BINARY_CHECK = true
CHARSET = AUTO
MAX_TIMESTAMP_LOOKAHEAD = 24
disabled = false
TIME_FORMAT = %FT%T.%3N%Z
TIME_PREFIX = "@timestamp":"
INDEXED_EXTRACTIONS = json
TRANSFORMS-changesourcetype = changesourcetype_to_ kafka
TRANSFORMS-override_kafka_host = override_kafka_host
TRANSFORMS-set = setnull,purchase,extraction
TRUNCATE = 15000

Transforms.conf
[changesourcetype_to_kafka]
REGEX = "@timestamp":"
FORMAT = sourcetype::kafka
DEST_KEY = MetaData:Sourcetype

[override_kafka_host]
REGEX = "host":"([^"]+)
FORMAT = host::$1
DEST_KEY = MetaData:Host

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[purchase]
REGEX = api=\"(ActionMenu|Buy|Create)\".*LIVE
DEST_KEY = queue
FORMAT = indexQueue

[extraction]
SOURCE_KEY = message
REGEX = (\w+)=([^,]+),
FORMAT = $1::$2

[mstjohn_splunk]

Thanks for the extra information @AnujaJadhav2. @FrankVl I'm going to go ahead and move that information to the question, so that it is better consolidated.

[mstjohn_splunk]

Hi @AnujaJadhav2. Did the answer below solve your question? If yes, please click “Accept” directly below the answer to resolve the post. If not, please comment with more information if you are still having issues. Thanks!

[FrankVl]

In props.conf (naming is free to choose of course)

[your-sourcetype]
REPORT-message-extraction = my-message-extraction

in transforms.conf:

[my-message-extraction]
SOURCE_KEY = message
DELIMS = ",", "="

or alternatively (see transforms.conf spec for some notes on which one to choose):

[my-message-extraction]
SOURCE_KEY = message
REGEX = (\w+)=([^,]+),
FORMAT = $1::$2

[AnujaJadhav2]

I tried to add these stanzas in props.conf and transforms.conf but I saw no difference. Not sure what is wrong. I also went through the documentation and tried a few more config like adding dest_key but no effect.

[FrankVl]

Can you share the exact props and transforms you have now? And perhaps also show a screenshot of what your data looks like now?