Getting Data In

Can I create a field transformation using a JSON source key?

Jordan_Brough
Path Finder

I don't seem to be able to set up a field transformation using a Source Key that comes from a JSON event field.

I have events like this:

{
  "time": "2013-06-23T13:55:37+00:00",
  "handler": "UsersController#index"
}

And I'd like to extract "UsersController" and "index" from the "handler" field. I have props.conf configured with KV_MODE=json

I added this transform via the GUI:

[controller_action_transform]
CLEAN_KEYS = 1
MV_ADD = 0
REGEX = (?<controller>.*)#(?<action>.*)
SOURCE_KEY = handler

and this field extraction:

[json]
REPORT-controller_action_extraction = controller_action_transform

however, when I do a query like this:

sourcetype=json | table handler controller action

I do get results for "handler" but don't get anything for "controller" or "action":

| handler               | controller | action |
-----------------------------------------------
| UsersController#index |            |        |

If I change the transform SOURCE_KEY to "_raw" then I do get results for controller & action (though not exactly correct).

Also, I can do an inline "rex" field extraction using the "handler" field and get the correct results. That is, this works just fine:

sourcetype=json | rex field=handler "(?<controller>.*)#(?<action>.*)" | table handler controller action

Am I doing something wrong with the transform? Are JSON-extracted fields not available for use in transforms or something?

(NOTE: The above is just some sample data I created for testing this out. The real logs that I need to use this on have more data and nested keys and so forth, so a workaround that involves not using the extracted JSON fields would be pretty non-ideal.)

Tags (3)
1 Solution

Jordan_Brough
Path Finder

I got an answer via Splunk Support.

They said:

I was able to repro. And this is a bug.

If not using the SOURCE_KEY then extractions are working.

When referring a field that was extracted using KV_MODE = json, it is not working.

I was able to find an already existing issue for that bug.

It's SPL-61046 and will probably fix in the next major release. (not maintenance release)

But there is a workaround to get it working.

Use the search like:

sourcetype=json | kv reload=t | table handler controller action

I've confirmed that the workaround does solve the problem.

View solution in original post

0 Karma

Jordan_Brough
Path Finder

I got an answer via Splunk Support.

They said:

I was able to repro. And this is a bug.

If not using the SOURCE_KEY then extractions are working.

When referring a field that was extracted using KV_MODE = json, it is not working.

I was able to find an already existing issue for that bug.

It's SPL-61046 and will probably fix in the next major release. (not maintenance release)

But there is a workaround to get it working.

Use the search like:

sourcetype=json | kv reload=t | table handler controller action

I've confirmed that the workaround does solve the problem.

0 Karma

apringle
Explorer

I was just curious if the referenced SPL-61046 issue was ever resolved? I am trying to do something very similar to the OP and having the same issue. I'm able to get around it by using a regex match on the _raw data, but it would be nice to be able to define the SOURCE_KEY for the JSON data.

(Also, if there is somewhere that I can view details about the referenced SPL issue, please let me know)

phoenixdigital
Builder

3 years on I am still seeing this issue.

Does anyone know at what point JSON fields are extracted?

It appears to be after custom transforms.conf configs.

0 Karma

vliggio
Communicator

I ping'ed my splunk support rep today and was told:

"The bug was closed with "cannot reproduce" this past October. The original issue was reported for 4.1.3 and 5.0.1.

That said, Splunk is particular about the JSON. Extraneous and/or incorrect delimiters will cause extraction to fail. Additionally, you should set KV_MODE=none if you are using INDEXED_EXTRACTIONS=json, otherwise data will be duplicated."

0 Karma

mikaelbje
Motivator

I'm also trying to get this working. The only way I got it working was using the | kv reload=t trick. I tried both with KV_MODE = json and with the default. Data is ingested using the HTTP Event Collector.

Splunk 7.1.0. SPL-61046 should be reopened IMHO

0 Karma

mikaelbje
Motivator

I was wrong. KV_MODE = json solved it. No need for the| kv reload=t trick

0 Karma

regiteric
Engager

Got the same issue, doing the transformation on the _raw seems to be the only solution. But is is not perfect as the escaped character in the JSON value are not displayed correctly.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...