Getting Data In

Which is the recommended way to input heavy load JSON data to Splunk?

wpoch
Engager

Hi everyone, I'm developing an integration with Splunk, and right now I'm using the Splunk Java SDK with the REST API and sending the events batch per documentation; the events are JSON encoded .

Now I'm considering if this is the best way to accomplish this, or should I use the TCP data inputs, or directly the Forwarders.

Any guidance will be welcome.

Thanks in advance.

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

While that is a perfectly good way to do it, you loose some granularity in the searches in Splunk due to the data being late coming in. The optimal way is to output to a file, and use the UF to send to Splunk. The benefits of using UF are outstanding:

  1. In the event of network communication errors, data will be backed up on the system until the connection between UF and Indexers is restored.
  2. Control over metadata for each file

That being said, you could also send directly to the new HTTP inputs in Splunk 6.3, but will be limiting your customer base until 6.3 is adopted widely.

0 Karma

wpoch
Engager

First of all, thanks for the answer.

So, in case we wan't to be able to send high throughput data over internet we keep using the REST API, and eventually migrate it to the new HTTP Event Collector. Also on controlled or on-prem environments we should use a TCP input with a previously created input on Splunk for better performance.

Finally recommend our customers, if they can install an UF use that.

We wan't to keep the UX as simple as possible, by not installing additional software (Universal Forwarder).

Kind regards,

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...