Why is my configuration using regex to assign sourcetypes with props.conf and transforms.conf not working?

[Volto]

Hello,

I am trying to get dynamic sourcetype working for a set directories under Splunk. The intention is that the folder that contains the logs will be the sourcetype, and there are many folders underneath the Splunk directory that we are performing this on. Below are the configs I am using to perform the monitoring and sourcetyping of the logs. The issue that I'm having is that the configs do not to seem to work at all, I know the inputs.conf works by itself, but adding the props and transforms doesn't seem to work at all.

inputs.conf:
[monitor:///var/log/splunk/.../*.log]
disabled = false
recursive = true
index = dev_test

props.conf:
[source::/var/log/splunk/.../*log]
TRANSFORMS-sourcetype = override-sourcetype

transforms.conf
[override-sourcetype]
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Sourcetype
REGEX = source::([^/]+)/[^/]+.log$
FORMAT = sourcetype::$1

Any help would be greatly appreciated. I've been banging my head against the wall trying to figure this out.

[woodcock · Accepted solution]

Like this:

inputs.conf

[monitor:///var/log/splunk/.../*.log]
disabled = false
recursive = true
index = dev_test
sourcetype=replace_sourcetype_with_containing_directory

props.conf

[replace_sourcetype_with_containing_directory]
TRANSFORMS-replace = replace_sourcetype_with_containing_directory

transforms.conf

[replace_sourcetype_with_containing_directory]
SOURCE_KEY = MetaData:Source
REGEX = .*/([^/]+)/[^/]+$
FORMAT = sourcetype::$1
DEST_KEY = MetaData:Sourcetype

You will have to restart every Splunk instance on your Indexers (or Heavy Forwarders) before this will take effect.

[Volto]

Making your changes seemed to have worked, but everything is coming under the sourcetype of replace_sourcetype_with_containing_directory.

[woodcock]

So you restarted the Splunk instances on the Forwarders where you put inputs.conf but did you restart the Splunk instances on the Indexers where you put props.conf and transforms.conf? Double-check this list:

• The sourcetype matches replace_sourcetype_with_containing_directory exactly (casing, punctuation, etc.).
• The props.conf and transforms.conf configuration files are deployed to the Indexers or Heavy Forwarders (or Universal Forwarders in some cases, such as INDEXED_EXTRACTIONS = CSV).
• The inputs.conf configuration file is deployed to the Forwarder.
• You must restart/bounce all Splunk instances on the servers where you deploy these files.
• There are no configuration errors during restart (watch the response text during startup on one server of each type).
• You are verifying proper current function by looking at NEW data (post-deploy/post-bounce), not previously indexed data (which is immutable).

[Volto]

Running through your check list, we are using Universal Forwarders and not using Heavy Forwarders. I forgot to add the props and transforms to the indexers props.conf and transforms.conf files. It looks like it's working now. Thanks!