I wish to use a search-time extracted field as the basis for routing to a specific index.
In my props.conf file I have a search-time extraction of fields. These fields are taken from the monitored log file name, which is a name that follows a known pattern but whose exact name cannot be predicted. Here is a typical log file name:
/var/log/containers/coredns-845cddccfc-prm27_kube-system_coredns-4a676a5795e68f4f2bacaed884bc3811de019083801b803ad67974b4ac3c221d.log
The set of log files are of a common sourcetype=kubernetes, evidenced in the LWF inputs.conf
[monitor:///var/log/containers/*.log]
sourcetype = kubernetes
multiline_event_extra_waittime = true
In props.conf I have
[kubernetes]
EXTRACT-sourcefields = /var/log/containers/(?<pod>.*)_(?<namespace>.*)_(?<containerName>.*)-(?<containerID>.*)\.log in source
To route to an index of interest, I add this to my props.conf
TRANSFORMS-theindex = theindex
In transforms.conf I add the associated stanza for this index, with the intent of routing to myindex when containerName=aContainerName. I note that the index myindex does in fact exist.
[theindex]
SOURCE_KEY = field:containerName
REGEX = aContainerName
DEST_KEY = _MetaData:Index
FORMAT = myindex
However, I find that the myindex never accrues events.
I suspect I am not allowed to use search-time extracted fields in transforms. If that is true, and my desire to route based on an extracted field, what approach should I take?
Search time extractions are exactly that: search time. So that information is not available at index time.
But since you are using the source field, which of course is available at index time, you should simply be able to re-use a very similar expression as used for the search time extraction, to write your transforms. Just use the source field as the source_key.
Alternatively, since you're using the source field, you might even be able to write this much simpler:
props.conf
[source::/var/log/containers/*_*_aContainerName-*]
TRANSFORMS-theindex = theindex
transforms.conf
[theindex]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = myindex
Search time extractions are exactly that: search time. So that information is not available at index time.
But since you are using the source field, which of course is available at index time, you should simply be able to re-use a very similar expression as used for the search time extraction, to write your transforms. Just use the source field as the source_key.
Alternatively, since you're using the source field, you might even be able to write this much simpler:
props.conf
[source::/var/log/containers/*_*_aContainerName-*]
TRANSFORMS-theindex = theindex
transforms.conf
[theindex]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = myindex
Thank you. Using your guidance, this stanza in transforms.conf meets my need:
[theindex]
SOURCE_KEY = MetaData:Source
# source sample: /var/log/containers/foo_default_containerName-f443343eb91ab92b1ed477bdc8b313727171214c0d7f3e1f13222ea786aaec5f.log
REGEX = /var/log/containers/?.*_?.*_containerName-.*\.log
DEST_KEY = _MetaData:Index
FORMAT = myindex