Getting Data In

Monitor a FTP server?

Jason
Motivator

FTP download is the only way this particular system is allowing us to access its logs. Files are dumped into the FTP area and then never changed, so they are perfect for a Splunk batch folder. But, we need to get them over to the Splunk box first.

Does anyone have a scripted input for monitoring a FTP site? I assume it would have to keep a listing of files already downloaded, but also keep an eye on that listing so it doesn't get too large.

Tags (3)
0 Karma

dwaddle
SplunkTrust
SplunkTrust

If you must access it via FTP, a couple of solution may work. Both of these share in common that they "mount" an FTP server as local disk via a "drive" or "mountpoint"

http://curlftpfs.sourceforge.net/ (for linux)

http://www.webdrive.com/products/webdrive/index.html (for windows/mac)

It is unknown whether these are sufficient to Splunk in order to support a monitor:// input - but I think they would work with a batch input just fine.

0 Karma

kdenton
Path Finder

Anthony solution works well if you have the ability to install the spunk universal forwarder on the server. It keeps track of the files it has sent you for indexing.

Another option to to monitor the log files remotely and basically achieve the same functionality with out installing additional software.

As for the other part of your question with monitoring what's been downloaded, splunk will keep track of whatcit ingested. As for watching the size, I had a similar problem that I solved with a small script that I kicked off every night at one minute passes midnight to delete previous days files.

0 Karma

kdenton
Path Finder

That is actually what I do. I ready the files and with a script delete everything that is over 1 day old at one past midnight every day.

0 Karma

Jason
Motivator

I had been planning on using a Splunk Batch directory to immediately index and delete downloaded files, but I guess I could use a Monitor directory instead and remove old files periodically.

I still need to have a script that doesn't re-download every file every time. Does anyone have any examples?

0 Karma

treinke
Builder

Why not put a splunk application on the box sending the data back to the central splunk server? Install the application, turn it to a light forwarder, and configure it to watch the folder.

There are no answer without questions
0 Karma

Jason
Motivator

I am a fan of the Splunk Forwarder. However, the data is generated on a closed system (no OS access) and FTP download is the only way we have to gather this data.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...