Getting Data In

How can I pull out values in a log path and route it to a specific index?

bport15
Path Finder

I am looking to route logs to different indexes based on a specific value identified in the log path. For example:

/var/appl/logs/prod/server.log >> should go to index=prod
/var/appl/logs/dev/server.log >> should go to index=dev
/var/appl/logs/imp/server.log >> should go to index=imp
/var/appl/logs/test/server.log >> should go to index=test

The log path will always be the same except for the unique value in the 4th sub-directory, which is where our developers are modifying the path to tell us what index they'd like the data in.

We have around 180 indexes in our environment so I don't want to create a unique transforms.conf to accommodate for each one. I'd like to have one transforms.conf that can pull the unique value from the path and place that log in the corresponding index.

I haven't been able to find great documentation to help me set this up. Any assistance would be greatly appreciated!

0 Karma
1 Solution

DalJeanis
SplunkTrust
SplunkTrust

It's going to look something like this.

 # transforms.conf 
  [overrideindexbylogpath]
  DEST_KEY =_MetaData:Index
  REGEX =\/var\/appl\/logs\/(.*)\/server\.log
  SOURCE_KEY= ... name of extracted log path field ...
  FORMAT = $1




  #props.conf 
  [ ...definition of which records you want to change...]
  TRANSFORMS-index = overrideindexbylogpath

I'm a little vague on what level of escaping will be needed on the regex... conf stanza versus search time versus saved search etc have little tweaky variations... but I think that got it correctly.

Here's a link to a version that works based on the host name field.

https://answers.splunk.com/answers/507081/rename-index-based-on-host-and-index-name.html#comment-506...

View solution in original post

DalJeanis
SplunkTrust
SplunkTrust

It's going to look something like this.

 # transforms.conf 
  [overrideindexbylogpath]
  DEST_KEY =_MetaData:Index
  REGEX =\/var\/appl\/logs\/(.*)\/server\.log
  SOURCE_KEY= ... name of extracted log path field ...
  FORMAT = $1




  #props.conf 
  [ ...definition of which records you want to change...]
  TRANSFORMS-index = overrideindexbylogpath

I'm a little vague on what level of escaping will be needed on the regex... conf stanza versus search time versus saved search etc have little tweaky variations... but I think that got it correctly.

Here's a link to a version that works based on the host name field.

https://answers.splunk.com/answers/507081/rename-index-based-on-host-and-index-name.html#comment-506...

bport15
Path Finder

Thanks @DalJeanis. The actual path we have includes several wildcards since we're pulling logs from containers. How would the format $1 know to pull from the (.*) if we have the following (where prod is in the index)?

Actual log path located in inputs.conf:

[monitor:///var/lib/origin/openshift*/pods/*/volumes/kubernetes*/log-dir/prod/dataportal-deployment*/*/catalina*.log]

This is what the transforms.conf would look like using wildcards, but not sure if it would actually work:

[override_index_by_log_path]
SOURCE_KEY = MetaData:Source
REGEX = \/var\/lib\/origin\/openshift*\/pods\/*\/volumes\/kubernetes*\/log-dir\/(.*)\/*\/*\/*.log
DEST_KEY = _MetaData:Index
FORMAT = $1

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Can you be sure how close to the final name the prod or dev etc will be? is it always two before the last slash? Here's one you can try...

   REGEX=.*\/(prod|dev|imp|test)\/[^\/]*\/[^\/]*\/.*\.log

It works over at regex101.com against this input...

/GEORGE/HENRY/var/lib/origin/openshiftWILMA/pods/LUKE/volumes/kubernetes/log-dir/prod/dataportal-deploymentDANA/BETTY/catalina.log
0 Karma

bport15
Path Finder

I can definitely try your suggestion above and try to catch all of the various environments we have. I'll keep you posted!

0 Karma

bport15
Path Finder

I was able to request our developers add "index=" into the path, so this is what my transforms.conf looks like. I'm pushing that out today and will update this thread if everything works as expected.

Transforms.conf
[override_index_by_log_path]
SOURCE_KEY = MetaData:Source
REGEX = .*\/index=.*\/.*
DEST_KEY = _MetaData:Index
FORMAT = $1

Props.conf
TRANSFORMS-route_index_to_log = override_index_by_log_path

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

I'm not seeing any parenthesis in the regex to mark the capture group for the index value. That should probably be something like...

REGEX =  .*\/index=([^\/]*)\/.*

... the value of the index needs to stop at the first slash, so we define a character class [^\/] which is any character EXCEPT a slash.

You also need to specify/validate what is going to happen to the index name when no capture is found.

0 Karma

bport15
Path Finder

Good catch, thank you. Modifying now.......

0 Karma

bport15
Path Finder

I finally got this working!

This is what I had to put in my transforms.conf. I ended up having the developers put "index=" in the log path and this is the final result:

TRANSFORMS.CONF

    [override_index_by_log_path_2]
    SOURCE_KEY = MetaData:Source
    REGEX = .*\/log-dir\/index=([^\/]*)\/.*
    DEST_KEY = _MetaData:Index
    FORMAT = $1

PROPS.CONF

[servicelog]
TRANSFORMS-route_index_to_log_servicelog = override_index_by_log_path_2

bport15
Path Finder

So, our indexes don't have a standard naming convention. Some have "prod", "test", "non prod" in the name, depending on the application (online_prod, online_test, appname_prod, appname_nonprod, appname_n, etc) and it depends on the environment as well. We do, however, have a standard log file path that we've implemented so the name of the index is included in the file path and will always be in the 9th sub-directory.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...