Splunk Search

How to extract a field that returns after a line_breaker

jorambokma
Explorer

Hello,

We are trying to split a nested json message into seperated events.
As we not wish to use the spath function it would be nice to split the events;
Our biggest challange right now is that we want to add the kId field in every event.

Example of the json;

{
    "kId": 47,
    "mId": 96,
    "resultStatus": "OK",
    "results": [{
        "creationDate": 1560509581000,
        "name": "test",
        "resultStatus": "OK",
        "duration": 0.858
    }, {
        "creationDate": 1560509581000,
        "name": "test2",
        "resultStatus": "OK",
        "duration": 0.858
    }]
}

we managed to get seperated events by using the following configurations:

props.conf
[measurements]
TRANSFORMS-kId = extract_kId
LINE_BREAKER = (},){|([){|(])}
SHOULD_LINEMERGE = false

transforms.conf
[extract_kId]
REGEX = kId[\"]:[,]
FORMAT = kId::$1
WRITE_META = true

events now look like this;

{"creationDate":1560509581000,"name":"test","resultStatus":"OK","duration":0.858}
{"creationDate":1560509581000,"name":"test2","resultStatus":"OK","duration":0.858}

How can we add the kId into this events?

Thanks in advance!

0 Karma
1 Solution

FrankVl
Ultra Champion

I don't think you can with Splunk. Event breaking is one of the first things that happens, so even if you could come up with a transforms or so to copy that kld to all the 'subsections', that all happens after line breaking, so the information is no longer available since Splunk processes each event on its own without knowledge of previous events.

You'll probably have to look at writing a script that pre-processes the logs before the are read by Splunk.

View solution in original post

FrankVl
Ultra Champion

I don't think you can with Splunk. Event breaking is one of the first things that happens, so even if you could come up with a transforms or so to copy that kld to all the 'subsections', that all happens after line breaking, so the information is no longer available since Splunk processes each event on its own without knowledge of previous events.

You'll probably have to look at writing a script that pre-processes the logs before the are read by Splunk.

jorambokma
Explorer

Thanks for your answer. Would sedcmd an option? Is it possible to use sedcmd with variables?

0 Karma

FrankVl
Ultra Champion

sedcmd also happens after event breaking. But anyway that is in essence just a search&replace and while you can use the /g flag to repeat the sedcmd, you cannot carry that kId found in the first part forward to later matches.

I think the only solution (other than preprocessing) would be to ingest the event as a whole and process it using SPL (using a combination of spath and multivalue tricks).

0 Karma

jorambokma
Explorer

Thanks. We will investigate our preprocessing options.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...