All Apps and Add-ons

REST API Modular Input: Is there a limit on JSON size before parsing?

DanielFordWA
Contributor

Hi,

Is there any limit on the size of the JSON the REST API Modular Input add-on can handle before the responsehandlers.py goes to work?

I have written a custom response handler to parse a large JSON file into separate events before indexing in Splunk.

In UAT and NFR environments everything works fine. The JSON is around 16.5k lines and 400K characters.

In PROD the data fails to index. The JSON is around 100k lines and 3M characters.

The sourcetype is set to the below.

[RESTMAR]
TRUNCATE = 0
TIME_PREFIX = lastModifiedDate

I can hit the endpoint with POSTMAN for both UAT /NFR and PROD, with the separate consumer secrets and keys, and get a successful response for each Environment.

I have edited the rest.py line 42 to put the logger on debug

#set up logging
logging.root
logging.root.setLevel(logging.DEBUG)

I will continue to investigate but has anyone else had this issue?

Thanks,

Dan

0 Karma
1 Solution

Damien_Dallimor
Ultra Champion

There is no imposed limits in the code.
Any error messages ?

Search : index=_internal ExecProcessor error rest.py

View solution in original post

0 Karma

Damien_Dallimor
Ultra Champion

There is no imposed limits in the code.
Any error messages ?

Search : index=_internal ExecProcessor error rest.py

0 Karma

DanielFordWA
Contributor

Hi Damien,

I think the issue was caused by moving the responsehandlers.py file from a windows box to a Linux box, or editing a previously created input rather than creating a new one.

The solution.

  • Test on NFR Linux box
  • Copy files from NFR box
  • Reinstall REST Modular input app in PROD
  • Add python files copied from NFR Linux box
  • Restart Splunk
  • Create New REST Modular input (editing a previously created input did not work)
  • Test data

Thanks for the response!

Cheers,

Dan

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...