I wouldn't use the qualifier "all" in (1).
Not all machines will necessarily have the capacity or network architecture that allows you to do it, however, you may still want to install a forwarder on as many machines as you can.
Even if you have a standard build, and include a forwarder as part of your standard build, you'll still have systems which end up as "exceptions" for whatever reason.
RE: (3) splunk can listen for syslog inputs directly. You don't need to do this manually.
Missing: You can scp the data back to the indexer with a scripted input, or a job scheduler, or cron. Then pick the data up with the local file system. This can be handy when you want to grab some data off a production server without having to submit a change control to install something new, or change anything on that server. 😉 Also, sometimes the log files are created that way - the result of batch jobs, and it hardly seems worthwhile running a full time splunk forwarder when you only need the output from a batch job once per day.
Another architecture option:
Sometimes you might want to forward the splunk data to another forwarder, and then send it to the indexer from there.
I can think of two reasons off the top of my head:
You want to minimize the number
of holes in a firewall that you have
to open, so you designate one relay
box to pull everything from the
other side of the firewall and send
it through.
You want to run lightweight
forwarders on your production
systems, but need a "full fledged"
forwarder to do filtering before the
data ends up on the indexer.
A Note on NFS:
It is slow. If you have a lot of log files, you will quickly run into latency issues. On the other hand, it is usually quick and easy to set up, and can dramatically shorten the deployment time for many low volume sources.
I would argue which solution is "best" depends on what you are trying to do and what kind of data you are working with. That is why there are so many possible solutions.
... View more