Simple scenario with intermediate forawarder to syslog - any help appreciated

[zizzencs]

I would like to achieve the following:

• run Splunk on some Windows (2003, 2008, 2008R2) hosts and:
  • send all event logs to a Splunk intermediate forwarder
  • monitor some directories and send all logs from these directories to the same intermediate forwarder
• run Splunk on a Linux (RHEL6) host and:
  • receive all logs from the Windows hosts
  • forward all received logs to our central Splunk server
  • forward all received logs to a syslog server as well.

I feel awkward for not being able to set it up in a day... 😞

So on the Windows hosts the best option is to install and run Universal Forwarders. I set them up easily, no problem here.

On the Linux host the first problem is to choose the appropriate forwarder type. The documentation states that syslog output is not available in Light and in Universal Forwarders so they are not viable candidates. This leaves me with a Heavy Forwarder.

I deployed Splunk to our central repository and installed it from the .rpm file. I turned on the SplunkForwarder application and restarted Splunk. So far so good.

At this stage I noticed that many of the features of Splunk are enabled, mainly:

• the web interface
• some applications:
  • gettingstarted
  • launcher
  • learned
  • search
  • splunk_datapreview

It also handles some databases: _audit _blocksignature _internal _thefishbucket history main summary

Now, I don't really know why these are required and what these do. However, I can't seem to be able to disable them, their config file say in some cases that the app can't be disabled.

Additionally, I have this message in scheduler.log every minute:

08-08-2012 17:59:18.838 +0200 ERROR SavedSplunker - Scheduler will not start searches for the next 60 seconds. The minimum free disk space (2000MB) reached for /opt/splunk/var/run/
splunk/dispatch.

I know about this, but I really don't need/want searchers to run.

All these observations led me to the question: is there a way to install a Splunk instance that has light footprint and still able to accomplish my goals? If yes, how can it be set up? Any advice/details/etc. is welcome.

[mhouse]

@zizzencs Was this ever resolved. I am seeing the same error you referenced above and would like to know how you resolved it. If this is a new install why would you be getting this error since no searches are being run yet?

[mwhite_splunk]

These:

_audit _blocksignature _internal _thefishbucket history main summary

are default indexes and are used for many things Splunk specific. main is where it dumps things by default, _internal is Splunk's index for Splunk, so on, so forth, etc. Don't worry about them, they are either necessary or harmless. They will not take up any space unless Splunk is writing events them to and if Splunk is writing events to them, you should be messing with them anyway. I would also recommend creating some other source/sourcetype specific indexes for the different types of logs you have coming in.

On your Linux server, unless it is your intermediate forwarder, you should run a Universal Forwarder there as well. You can do one-off data routing from your Indexer, link below. From Splunk to Splunk, you do not need to do syslog out. Just let the forwarders do all the work.

Why would you keep a second copy of your syslog events? Splunk does a wonderful job of keeping all the events from your entire network in their original event format and then compressing them down to save ample amounts of disk space; erego, no need to keep a second copy.

Router and Filter Data