filter data by source field wildcard

[Shakira1]

I've use case that I need to filter data by source field, that always changes.

in the transforms.conf I use:

[foo]
REGEX = MY REGEX
DEST_KEY = queue
FORMAT = nullQueue 

and in the props.conf I use:

[source::process_events]
TRANSFORMS-01= foo

The source always contains process_events and there is more data like date and info that changed. 

any way its possible to filter data by source wildcard?

thanks!

[Shakira1]

I still need your help pls.

its not working..

[Shakira1]

for example:

for example: 

s3://XXXXprocess_events/upt_day=20210627/part-00053-a08c751e-1c05-4725-bdea-740009db640e.c000.json.g...

[Shakira1]

which mean I can use 

[source::.*process_events.*] and it's should be working? 

[venkatasri]

@Shakira1 how is your complete source looks like, if you have /\ in source that might not work.

[Shakira1]

for example: 

s3://XXXXprocess_events/upt_day=20210627/part-00053-a08c751e-1c05-4725-bdea-740009db640e.c000.json.g...

[venkatasri]

@Shakira1 

Can you try this props.conf

[source::s3:\/\/*process_events...]

--

An upvote would be appreciated and Accept Solution if it helps!!

[Shakira1]

I'm still getting results...

any ideas why?

[venkatasri]

@Shakira1 REGEX in your transforms conf might be not correct. Can you share sample event and your transforms, props config?

[Shakira1]

I can't share sample because it's with PII. 

but what I want to exclude is some path in the raw data 

[foo]

so I just put the regex like that: XXX\/XXX\/XXX\/XXXX 

and in the props.conf I just add: TRANSFORMS = foo

[venkatasri]

@Shakira1  can you try this, REGEX matches 4 segment dir structure in your _raw event. The following conf shall be deployed to HF/indexers.

# props.conf
[source::s3:\/\/*process_events...]
TRANSFORMS-nullq = sendtonull

#transforms.conf
[sendtonull]
REGEX = [\w-]+\/[\w-]+\/[\w-]+\/[\w-]+
FORMAT = nullQueue
DEST_KEY = queue

 ---

An upvote would be appreciated and Accept solution if it helps!

[venkatasri]

@Shakira1 

If i understand correctly you have source= <values> having multiple combinations which you want to use in props.conf the source might always contains process_events?

Example. process_events_26062021,  process_events_27062021, log_process_events_26062021

It is possible to match them using Regex style- 

#your props.conf can be - 

[source::<regex>] 

TRANSFORMS-01= foo

Docs says , - Refer - https://docs.splunk.com/Documentation/Splunk/latest/Admin/PropsConf

When setting a [<spec>] stanza, you can use the following regex-type syntax:
... recurses through directories until the match is met
    or equivalently, matches any number of characters.
*   matches anything but the path separator 0 or more times.
    The path separator is '/' on unix, or '\' on Windows.
    Intended to match a partial or complete directory or filename.
|   is equivalent to 'or'
( ) are used to limit scope of |.
\\ = matches a literal backslash '\'.

Example: [source::....(?<!tar.)(gz|bz2)]

---

An upvote would be appreciated and Accept Solution if it helps!