routing remote syslog data to different indexes

[colinj]

We have a number of machines set up with rsyslog to collect data from various systems. Rsyslog all the data is sent to /var/log/remote/ in to individual log files for the sending host. I've got each of these machines also set up as heavy forwarders so we can do whatever we need to with the incoming data before we send it along to the indexers.

Here's the inputs.conf file that I'm using at the moment:

[monitor:///var/log/remote]
disabled = 0
index = syslog
sourcetype = syslog

What I want to do eventually is change the index that the data is going to based on the host that is sending the initial syslog stream. What I'm looking for is some guidance on going about doing this. Is there a right way and a wrong way?

Thoughts, comments, questions?

colin j.

[yannK]

Here is the exact answer you need :

http://splunk-base.splunk.com/answers/8531/routing-to-index-based-on-host-etc

This is called : Index time field transformation.
You want to setup this props/transforms rules on the indexer (AND heavy forwarder in your case)

• The props.conf will contain the sourcetype and the transforms to call
  

  
  
  [syslog]
  
  TRANSFORMS-changeindex=routemyhosttomynewindex
  
  

• the transforms.conf will use a regex to match the host and change the index accordingly.
  

  
  
  [routemyhosttomynewindex]
  
  SOURCE_KEY = MetaData:Host
  
  REGEX = myhost
  
  DEST_KEY = MetaData:Index
  
  FORMAT = mynewindex
  
  

[rajanala]

I am trying to use the props and transforms as provided in the answer here. In my case, the server, where the HFWD, is contains logs from several sources.
Will this work ? ( the following files will be on the HFWD )

inputs.conf

[monitor:///var/log/*]
sourcetype = syslog
host_segment = 3

props.conf
[syslog]
TRANSFORMS-changeindex=routemyhosttomynewindex

transforms.conf
[routemyhosttomynewindex]
SOURCE_KEY = MetaData:Host
REGEX = myhost
DEST_KEY = MetaData:Index
FORMAT = mynewindex

[mikeschneider]

Hi Yann,

Shouldnt that be "_MetaData:Index" instead of "MetaData:Index" ?