I have some log files where some entries contain json content, and some entries do not. I want to extract the json content from between the { } characters and drop everything else
Sample log data:
2015 Jan 27 18:10:35:530 GMT +8 PCM_ARPI-PCM_BE_ARPI_AGENTA1 Info [/Channels/ARPI/Inbound/ChargeSet.Thread.9] - [user] [PCM_ARPI_A11] ChartSet UPD -->1447307
2015 Jan 27 18:10:35:543 GMT +8 PCM_ARPI-PCM_BE_ARPI_AGENTA1 Info [/Channels/ARPI/Inbound/ChargeSet.Thread.9] - [user] [PCM_ARPI_A11] PI-->PEN New N1:1447307|BL_CHRGSET_SETTLED|7fb9467d-2965-4da8-89de-77564ed5aecd|Pending
2015 Jan 27 18:10:35:543 GMT +8 PCM_ARPI-PCM_BE_ARPI_AGENTA1 Info [/Channels/ARPI/Inbound/ChargeSet.Thread.9] - [user] [PCM_ARPI_A11] {"component":"PCM","dateTime":"2015-01-27 18:10:35.543","bKeys":["inv_1447307"],"startMs":1422353435530,"ms":1422353435543,"host":"hklp321p","fail":false,"dest":"IR4.PCM.CHRGSET.UPDATE","domain":"PCM","incomingDomain":"ARPI","destType":"Q","platform":"BE","xcid":"6cf21017-c228-4f17-a289-fb3aed0d5f0c"}
The first 2 lines should be discarded (never indexed) and the 3rd line should have the json content extracted and indexed using the regular json parsing.
Props.conf
[json_mixed]
NO_BINARY_CHECK = true
category = Custom
disabled = false
pulldown_type = true
TRANSFORMS-set = extractjson,setnull
Transforms.conf
[extractjson]
REGEX=(.*)(\{.*\}$)
FORMAT= $2
DEST_KEY=_raw
[setnull]
REGEX=[^}]+
DEST_KEY=queue
FORMAT=nullQueue
I was able to extract the json portion and index it, but when I attempt to drop the rest of the entries, everything is dropped.
To test, I edit the above 2 files, restart splunk, use the 'upload data' feature and select 'json_mixed' as the source type.
With TRANSFORMS-set = extractjson the log entries with json content show up as expected, but the non-json entries are still present in the preview pane.
With TRANSFORMS-set = extractjson,setnull the preview pane is empty.
With TRANSFORMS-set = setnull, extractjson the preview pane gives an error saying no events were found.
I have tried many different examples on discarding events, and discarding events that do not contain a character, but nothing works right.
What am I missing here?
... View more