Getting Data In

How to assign syslog file to specific index based on file name in transforms.conf?

dhavamanis
Builder

Can you please provide sample configuration for the below, We have multiple forwarding sources and they are using syslog-ng. how to assign index based on log filename at indexer side.

Example :

mynbc-syslog-2015-07-30.log pattern files to go "mynbc" index
msnbc-syslog-2015-07-29.log patter files to go "msnbc" index
bravotv-syslog-2015-08-01_1.log patter files to go "bravotv" index

0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

Filename-based transforms.conf use the source key from the metadata like this:

[send_to_index_by_source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::/path/to/files/(\w+)-syslog
DEST_KEY = _MetaData:Index
FORMAT = $1

Reference that in props.conf with TRANSFORMS-foo = send_to_index_by_source.

View solution in original post

martin_mueller
SplunkTrust
SplunkTrust

Filename-based transforms.conf use the source key from the metadata like this:

[send_to_index_by_source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::/path/to/files/(\w+)-syslog
DEST_KEY = _MetaData:Index
FORMAT = $1

Reference that in props.conf with TRANSFORMS-foo = send_to_index_by_source.

dhavamanis
Builder

Forwarder- outputs.conf

[tcpout:splunkssl]
server = splunk.abc.com:9997
compressed = true

[tcpout-server://splunk.abc.com:9997]
sslCertPath = $SPLUNK_HOME/etc/certs/forwarder.pem
sslPassword = password
sslRootCAPath = $SPLUNK_HOME/etc/certs/cacert.pem

0 Karma

dhavamanis
Builder

trying with the below config, seems its not working for me, Can you please correct this if anything wrong,

props.conf

[syslog]
TRANSFORMS-idx_routing = generic_idx_routing

transforms.conf (created index abc)

[generic_idx_routing]
SOURCE_KEY = MetaData:Host
REGEX = abc.xyz.\nbcu.com
DEST_KEY = _MetaData:Index
FORMAT = $1

inputs.conf

[splunktcp-ssl:9997]
compressed = true
sourcetype = syslog

[SSL]
password = password
requireClientCert = false
rootCA = $SPLUNK_HOME/etc/certs/cacert.pem
serverCert = $SPLUNK_HOME/etc/certs/splunknode.pem

0 Karma

dhavamanis
Builder

Can you please provide the sample REGEX while using the the file name pattern like this ${sitename}.${logname} ${Existing MSG}. we want to parse the sitename and assign to index name.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...