Deployment Architecture

data from remote hosts

pezcrap
Explorer

I wish to ingest events from a large number of remote hosts. I cannot install any Splunk infrastructure on these hosts.

I have looked into the various remote interfaces for splunk and none seem appropriate for my needs. It seems likely that I will need to build my own service to collect events from these hosts.

My question is: what is the best way to get data from my service into Splunk? I would like to be able to guarantee that once I have sent an 'ACK' to the remote host, that the data will make its way into splunk. I would also like to be able to scale the infrastructure horizontally.

I could have the server write to a monitored file, but I don't really want to create huge log files just to get data into Splunk.

I could use a FIFO queue, but that would not provide the guarantee I was talking about.

Perhaps I should use a Splunk SDK from within my service?

Can a splunk forwarder help here?

Tags (1)
0 Karma

yannK
Splunk Employee
Splunk Employee

I cannot install any Splunk infrastructure on these hosts.

So you have to bring the logs to splunk somehow, to the indexer or to a forwarder.

  • for log file, use a shared folder monitored remotely by forwarder
  • a script to copy the files up to a forwarder
  • use a syslog server to send the logs ( to another syslog that will write to disk, then monitor with splunk), but avoid UDP of course.
0 Karma

pezcrap
Explorer

This doesn't really answer my question. I already noted I will need to implement my own service to collect events. I was asking how to go about implementing that service.

edit: Thanks a lot for responding though, I sounded a bit ungrateful there 🙂

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...