Getting Data In

Extracting fields from JSON file format

abhayneilam
Contributor

Hi,
I want to extract the fields from JSON File format and after extrating the fileds. I want to remove one of the value of a particular key, such as :

My input data :

{"TID":"111222","EID":"44567","XXX":"00000x<?xml?>aasewl85765"}

Now, I want to remove the "XXX":"00000x<?xml?>aasewl85765" .. This should not be indexed other parts should be indexed like {"TID":"111222","EID":"44567" }

In my Splunk only two fields should come "TID" and "EID".

Please suggest the best possible way !!

Tags (2)
0 Karma
1 Solution

dshpritz
SplunkTrust
SplunkTrust

Your order is messed up. First you want to filter out the data (prior to indexing) and then (on search) you extract the fields.

To filter, in your props.conf:

[mysourcetype]
SEDCMD-removefields = s/{"TID":"\d+","EID":"\d+"(.*)/}/g

To extract the fields, you might be able to use Splunk's automagical field extraction. In the same props.conf:

[mysourcetype]
KV_MODE = json

So, all together:

[mysourcetype]
KV_MODE = json
SEDCMD-removefields = s/{"TID":"\d+","EID":"\d+"(.*)/}/g

This will need to be done on your search heads and indexers.

View solution in original post

dshpritz
SplunkTrust
SplunkTrust

Your order is messed up. First you want to filter out the data (prior to indexing) and then (on search) you extract the fields.

To filter, in your props.conf:

[mysourcetype]
SEDCMD-removefields = s/{"TID":"\d+","EID":"\d+"(.*)/}/g

To extract the fields, you might be able to use Splunk's automagical field extraction. In the same props.conf:

[mysourcetype]
KV_MODE = json

So, all together:

[mysourcetype]
KV_MODE = json
SEDCMD-removefields = s/{"TID":"\d+","EID":"\d+"(.*)/}/g

This will need to be done on your search heads and indexers.

abhayneilam
Contributor

"Note that that is the start of the line, not the end of it." This clue helped me to get a solution for my problem.

Thanks a lot !!

0 Karma

dshpritz
SplunkTrust
SplunkTrust

Your linebreaker doesn't have a capture, which may be the problem. I would set the linebreaker to:

([\r\n]){

Note that that is the start of the line, not the end of it.

0 Karma

abhayneilam
Contributor

I want to remove one of the field and its value from the file to be indexed. Please help me

Regards,
Abhay

0 Karma

laserval
Communicator

Have you tried without setting a LINE_BREAKER? When I've indexed JSON, it has worked well out of the box as long as the file has the format

{...}
{...}

i.e. each line is a separate JSON object (which it looks liek you have)

0 Karma

abhayneilam
Contributor

I want to have a separate events for each line of JSON file. I am getting all the lines merged with a single events. Please let me know how can I separate the events:

My Props.conf is :

[json]
KV_MODE = json
LINE_BREAKER = "}$"
NO_BINARY_CHECK = 1
TRUNCATE = 0
SHOULD_LINEMERGE = false

Please help

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...