Getting Data In

LINE_BREAKER doesn't seem to work for new add-on

tomawest
Path Finder

Hi,

I've been trying to create a new add-on to ingest some data into a new sourcetype within splunk via a REST API service. Unfortunately the api returns JSON as a full array so I am unable to use the add-on creators function to automatically separate arrays into separate events.

I have been doing a lot of reading into event breaking and looked at a number of solutions to this using LINE_BREAKER and BREAK_BEFORE_ONLY however in both instances I haven't succeeded in breaking the event.

The JSON I have been using is [{"idOrg":"abc123","name":"Joe Smith","active":true,"id":"xyz789"},{"idOrg":"efg456","name":"Michael Thomas","active":true,"id":"uvw456"},{"idOrg":"hij789","name":"Craig Lease","active":true,"id":"rst123"}]

I'm trying to get the data into splunk so each nested json is it's own entry. I.e.
{"idOrg":"abc123","name":"Joe Smith","active":true,"id":"xyz789"}
{"idOrg":"abc123","name":"Michael Thomas","active":true,"id":"uvw456"}
{"idOrg":"abc123","name":"Craig Lease","active":true,"id":"rst123"}

I have managed to get this to work as intended when importing this data via the sourcetype editor using the following in props.conf

  • SHOULD_LINEMERGE = false
  • LINE_BREAKER = }(,){

However when running this outside the add-on creator on my machine, I cannot get the same level of success.

It's worthwhile noting that I have the following commands in my props.conf in addition to the ones above:

  • KV_MODE = json
  • SEDCMD-remove_header = s/[//g
  • SEDCMD-remove_footer = s/]//g
  • pulldown_type = 1

So far I have taken the following actions in order to attempt to resolve this. All of which have had no benefit.

  1. Change LINE_BREAKER to \}(,)\{
  2. Add a new SEDCMD to replace }, with }NEWLINE and change LINE_BREAKER to NEWLINE. The SEDCMD worked as expected but no luck with LINE_BREAKER
  3. Set SHOULD_LINEMERGE to true and LINE_BREAKER to BREAK_BEFORE_ONLY

Any advice at this stage would be very gratefully received.

Regards

Tom

0 Karma
1 Solution

tomawest
Path Finder

Hi,

Thankyou very much for your suggestion, however unfortunately it made no difference.

The answer in the end was as simple as editing the data input within the add-on creator and specify the JSON path as$.[*] within the event extraction settings.

Thanks

Tom

View solution in original post

0 Karma

tomawest
Path Finder

Hi,

Thankyou very much for your suggestion, however unfortunately it made no difference.

The answer in the end was as simple as editing the data input within the add-on creator and specify the JSON path as$.[*] within the event extraction settings.

Thanks

Tom

0 Karma

Yod_ssoni
Explorer

Hi @tomawest,

I am facing similar situation where I am trying to extract events from newrelic addon and unable to perform line break for JSON lines from single event to multiple events. Could you please help me explain with stanza configuration and how it is done like which conf file and what stanza i need to add JSON path $.[*] ?

Thanks,
Shashank Soni

0 Karma

sudosplunk
Motivator

Hi there,

Did you try below combination,

BREAK_ONLY_BEFORE = (?m)\{\"idOrg\"\:
SHOULD_LINEMERGE = true
0 Karma
Get Updates on the Splunk Community!

Index This | Forward, I’m heavy; backward, I’m not. What am I?

April 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

A Guide To Cloud Migration Success

As enterprises’ rapid expansion to the cloud continues, IT leaders are continuously looking for ways to focus ...

Join Us for Splunk University and Get Your Bootcamp Game On!

If you know, you know! Splunk University is the vibe this summer so register today for bootcamps galore ...