I have some log files where some entries contain json content, and some entries do not. I want to extract the json content from between the { } characters and drop everything else
Sample log data:
2015 Jan 27 18:10:35:530 GMT +8 PCM_ARPI-PCM_BE_ARPI_AGENTA1 Info [/Channels/ARPI/Inbound/ChargeSet.Thread.9] - [user] [PCM_ARPI_A11] ChartSet UPD -->1447307
2015 Jan 27 18:10:35:543 GMT +8 PCM_ARPI-PCM_BE_ARPI_AGENTA1 Info [/Channels/ARPI/Inbound/ChargeSet.Thread.9] - [user] [PCM_ARPI_A11] PI-->PEN New N1:1447307|BL_CHRGSET_SETTLED|7fb9467d-2965-4da8-89de-77564ed5aecd|Pending
2015 Jan 27 18:10:35:543 GMT +8 PCM_ARPI-PCM_BE_ARPI_AGENTA1 Info [/Channels/ARPI/Inbound/ChargeSet.Thread.9] - [user] [PCM_ARPI_A11] {"component":"PCM","dateTime":"2015-01-27 18:10:35.543","bKeys":["inv_1447307"],"startMs":1422353435530,"ms":1422353435543,"host":"hklp321p","fail":false,"dest":"IR4.PCM.CHRGSET.UPDATE","domain":"PCM","incomingDomain":"ARPI","destType":"Q","platform":"BE","xcid":"6cf21017-c228-4f17-a289-fb3aed0d5f0c"}
The first 2 lines should be discarded (never indexed) and the 3rd line should have the json content extracted and indexed using the regular json parsing.
Props.conf
[json_mixed]
NO_BINARY_CHECK = true
category = Custom
disabled = false
pulldown_type = true
TRANSFORMS-set = extractjson,setnull
Transforms.conf
[extractjson]
REGEX=(.*)(\{.*\}$)
FORMAT= $2
DEST_KEY=_raw
[setnull]
REGEX=[^}]+
DEST_KEY=queue
FORMAT=nullQueue
I was able to extract the json portion and index it, but when I attempt to drop the rest of the entries, everything is dropped.
To test, I edit the above 2 files, restart splunk, use the 'upload data' feature and select 'json_mixed' as the source type.
With TRANSFORMS-set = extractjson
the log entries with json content show up as expected, but the non-json entries are still present in the preview pane.
With TRANSFORMS-set = extractjson,setnull
the preview pane is empty.
With TRANSFORMS-set = setnull, extractjson
the preview pane gives an error saying no events were found.
I have tried many different examples on discarding events, and discarding events that do not contain a character, but nothing works right.
What am I missing here?
Could you change the order as below:
TRANSFORMS-set = setnull, extractjson
From Document
the order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue.
Still does not work. Instead of a blank preview pane I get the message "No results found. Please change Sourcetype, adjust Sourcetype settings, or check your source file."
Wondering after changes, if you have added more json files in monitored folder. Changes will apply to new events only.
This is using the 'Upload files from my computer' feature, so it always uses new events.