We need to route data to specific indexes based on the file names being monitored. We are trying to get the data from the file(s) /app_qa/vodn_01/logfiles/current/bipc9996.stdout to route to the index named "RATABASE_SIT2" With our current settings all of this data is going to index "main" We have several other files that need to be routed accordingly, we are just looking for a good example we can use.
inputs.conf:
[monitor:///app_qa/vodn_01/logfiles/current/.]
followSymlink=true
crcSalt =
props:
[source::bipc9996.stdout]
TRANSFORMS-routing_SIT2_bipc9996 = index_redirect_to_RATABASE_SIT2_bipc9996
transforms:
[index_redirect_to_RATABASE_SIT2_bipc9996]
REGEX = bipc9996.stdout
sourcetype = bipc9996_SIT2
DEST_KEY = _MetaData:Index
FORMAT = RATABASE_SIT2
we assume these files (props/transforms) should be in the etc/app/local folder on the deployment server/forwarder. We have also deployed these changes to the indexer. Is this setup incorrect?
Easiest thing would be to set the correct index directly in inputs.conf on the forwarder.
[monitor:///app_qa/vodn_01/logfiles/current/bipc9996.stdout]
index = RATABASE_SIT2
...
Otherwise if you don't want that for some reason, this can be done almost like what you've pasted but NOT on the forwarders - it needs to go on the full Splunk instance that performs parsing of the logs, which is often your indexers.
props.conf:
[source::/app_qa/vodn_01/logfiles/current/bipc9996.stdout]
TRANSFORMS-routing_SIT2_bipc9996 = index_redirect_to_RATABASE_SIT2_bipc9996
transforms.conf:
[index_redirect_to_RATABASE_SIT2_bipc9996]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = RATABASE_SIT2
As you can see I'm simply using "." as regex - this is because the source matching was already performed in props.conf, so no further matching should be needed.
I assume the issue is with source. The source that you have set in props.conf might not be right
You need to set sourcetype in your inputs.conf file. Something like
[monitor:///app_qa/vodn_01/logfiles/current/bipc999*]
followSymlink=true
crcSalt = <source>
sourcetype = bipc999
props.conf can contain
[bipc999]
TRANSFORMS-routing_SIT2_bipc9996 = index_redirect_to_RATABASE_SIT2_bipc9996
It depends on what kind of forwarder you are using. If you use light forwarder then props and transforms should be on indexer.
Splunk doesn't recommend overriding source, hence i work with sourcetypes mostly 🙂
http://docs.splunk.com/Documentation/Splunk/6.1.2/admin/Inputsconf
are you using heavy forwarder or light forwarder?
Easiest thing would be to set the correct index directly in inputs.conf on the forwarder.
[monitor:///app_qa/vodn_01/logfiles/current/bipc9996.stdout]
index = RATABASE_SIT2
...
Otherwise if you don't want that for some reason, this can be done almost like what you've pasted but NOT on the forwarders - it needs to go on the full Splunk instance that performs parsing of the logs, which is often your indexers.
props.conf:
[source::/app_qa/vodn_01/logfiles/current/bipc9996.stdout]
TRANSFORMS-routing_SIT2_bipc9996 = index_redirect_to_RATABASE_SIT2_bipc9996
transforms.conf:
[index_redirect_to_RATABASE_SIT2_bipc9996]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = RATABASE_SIT2
As you can see I'm simply using "." as regex - this is because the source matching was already performed in props.conf, so no further matching should be needed.
The path did not quite work for me, I ended up have to use this:
[source::/app_qa/vodn_01/logfiles/.../bipc9996.stdout]
I found that the current directory was a symbolic link and this was the only way to get it to index the log. If I used the below it would not work correctly:
[source::/app_qa/vodn_01/logfiles/*/bipc9996.stdout]