How to route an index based on a search-time extracted field?

[mspetrovic]

I wish to use a search-time extracted field as the basis for routing to a specific index.

In my props.conf file I have a search-time extraction of fields. These fields are taken from the monitored log file name, which is a name that follows a known pattern but whose exact name cannot be predicted. Here is a typical log file name:

/var/log/containers/coredns-845cddccfc-prm27_kube-system_coredns-4a676a5795e68f4f2bacaed884bc3811de019083801b803ad67974b4ac3c221d.log

The set of log files are of a common sourcetype=kubernetes, evidenced in the LWF inputs.conf

[monitor:///var/log/containers/*.log]
sourcetype = kubernetes
multiline_event_extra_waittime = true

In props.conf I have

[kubernetes]
EXTRACT-sourcefields = /var/log/containers/(?<pod>.*)_(?<namespace>.*)_(?<containerName>.*)-(?<containerID>.*)\.log in source

To route to an index of interest, I add this to my props.conf

TRANSFORMS-theindex = theindex

In transforms.conf I add the associated stanza for this index, with the intent of routing to myindex when containerName=aContainerName. I note that the index myindex does in fact exist.

[theindex]
SOURCE_KEY = field:containerName
REGEX = aContainerName
DEST_KEY = _MetaData:Index
FORMAT = myindex

However, I find that the myindex never accrues events.

I suspect I am not allowed to use search-time extracted fields in transforms. If that is true, and my desire to route based on an extracted field, what approach should I take?

[FrankVl · Accepted solution]

Search time extractions are exactly that: search time. So that information is not available at index time.

But since you are using the source field, which of course is available at index time, you should simply be able to re-use a very similar expression as used for the search time extraction, to write your transforms. Just use the source field as the source_key.

Alternatively, since you're using the source field, you might even be able to write this much simpler:

props.conf

[source::/var/log/containers/*_*_aContainerName-*]
TRANSFORMS-theindex = theindex

transforms.conf

[theindex]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = myindex

[mspetrovic]

Thank you. Using your guidance, this stanza in transforms.conf meets my need:

[theindex]
SOURCE_KEY = MetaData:Source
# source sample: /var/log/containers/foo_default_containerName-f443343eb91ab92b1ed477bdc8b313727171214c0d7f3e1f13222ea786aaec5f.log
REGEX = /var/log/containers/?.*_?.*_containerName-.*\.log
DEST_KEY = _MetaData:Index
FORMAT = myindex