Getting Data In

conditional index and sourcetype name inputs.conf by file-naming-convention

hiddenkirby
Contributor

So my goal is to be able to pass a file to a splunk-monitored directory.. and have splunk apply it to the appropriate index and sourcetype...by a sort of naming convention.

file would come in as "indexname_sourcetype_filename.txt" or whatever... and my inputs.conf would do the appropriate thing.

is this possible?

Tags (1)
1 Solution

southeringtonp
Motivator

Yes, this should be possible. It would be something along these lines:

In transforms.conf:

[override-sourcetype]
SOURCE_KEY = source
DEST_KEY = MetaData:Sourcetype
REGEX = \w+_(\w+)_\w+\.txt$
FORMAT = sourcetype::$1

[override-index]
SOURCE_KEY = source
DEST_KEY = _MetaData:Index
REGEX = (\w+)_\w+_\w+\.txt$
FORMAT = index::$1

And then in props.conf:

[source::/var/log/inputdir/*]
TRANSFORMS-sourcetype = override-sourcetype
TRANSFORMS-index = override-index

View solution in original post

cssmdi
Explorer

Hi
I had some trouble defining an indexname out of the path and filename of the sourcefile.
In MetaData:Source, the sourcename is prefixed with 'source::', so you have to consider this in the regex. Further, in the index name, defined in 'FORMAT = ' , there is no prefix. The configuration in the example above has to be 'FORMAT = $1' and not 'FORMAT = index::$1'.

0 Karma

southeringtonp
Motivator

Yes, this should be possible. It would be something along these lines:

In transforms.conf:

[override-sourcetype]
SOURCE_KEY = source
DEST_KEY = MetaData:Sourcetype
REGEX = \w+_(\w+)_\w+\.txt$
FORMAT = sourcetype::$1

[override-index]
SOURCE_KEY = source
DEST_KEY = _MetaData:Index
REGEX = (\w+)_\w+_\w+\.txt$
FORMAT = index::$1

And then in props.conf:

[source::/var/log/inputdir/*]
TRANSFORMS-sourcetype = override-sourcetype
TRANSFORMS-index = override-index

platform_pie
New Member

BTW - I was doing this for just the sourcetype override, and needed to change SOURCE_KEY = MetaData:Source in the transforms.conf [override-sourcetype] stanza to get this to work.

0 Karma

hiddenkirby
Contributor

ok so no way to create the index dynamically. thanks for this. 🙂

0 Karma

southeringtonp
Motivator

No, configure inputs.conf as normal, and create each destination index ahead of time via indexes.conf or the Manager. The settings above will override the sourcetype and destination index as the file is indexed. Not sure what happens if the index doesn't exist - it will probably throw an indexing error but it might revert back to the default index.

0 Karma

hiddenkirby
Contributor

and no inputs.conf at all?

hiddenkirby
Contributor

That is cool. What happens if the index doesnt exist yet?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...