Deployment Architecture

Universal / Heavy Forward Vs Http Event Collector

nks
New Member

hi,
My requirement is to push a continuous stream of data into Splunk for search and reporting. Data will be provided to me in form of CSV files by a batch job.
these CSV files will be placed in a folder as input and once i am done inserting them into Splunk, these files needs to be deleted from folder for new files to come in.

I am wondering which route should i pick, namely :- A) setup Universal or Heavy Forwarders. They will push the data from CSV to Splunk.. Or B), write my own custom java code to push data from CSV files to HEC endpoints.

Tags (1)
0 Karma
1 Solution

Richfez
SplunkTrust
SplunkTrust

If you installed a Universal Forwarder and set the files up as a batch input, as long as you have your output configured and have set your timestamping and whatnot right, you are done.

I mean, sure you COULD write custom java code, but why?

Now, if it's REALLY high volume (by that, I mean hundreds of GB/day or more, definitely unlikely for this to be a problem even at dozens of GB/day), HEC may be beneficial.

In the docs I linked, ignore the warnings about doing it only for large uploads of historical data, and also ignore the part about how it works in Splunk Web. They don't matter.

View solution in original post

Richfez
SplunkTrust
SplunkTrust

If you installed a Universal Forwarder and set the files up as a batch input, as long as you have your output configured and have set your timestamping and whatnot right, you are done.

I mean, sure you COULD write custom java code, but why?

Now, if it's REALLY high volume (by that, I mean hundreds of GB/day or more, definitely unlikely for this to be a problem even at dozens of GB/day), HEC may be beneficial.

In the docs I linked, ignore the warnings about doing it only for large uploads of historical data, and also ignore the part about how it works in Splunk Web. They don't matter.

Richfez
SplunkTrust
SplunkTrust

nks,

If this resolved your issue, could you please mark it Accepted?

If it did not, please post back with more information or what's not working right so we can help finish this up!

Happy Splunking,
Rich

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...