Splunk Search

How to write the regex to filter events by contents of a specific field in transforms.conf?

pjohnson1
Path Finder

I am creating a filter to only keep certain events which contain a specific country code (they are actually hostnames which contain the country code).

props.conf

[log*]
TRANSFORMS-keep-LOG = setnull,keep-LOG_transform

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[keep-LOG_transform]
REGEX = SERVER01:\s+(KR|SG|IN|PH|TW|TH)
DEST_KEY = queue
FORMAT = indexQueue

How can I create a REGEX on a specific field?

Field extraction is complete with this data source, but I would like to filter all events with KR,SG,IN,PH,TW,TH in a specific field like host.

Thanks.

0 Karma

jeffland
SplunkTrust
SplunkTrust

Not sure if it's a typo in your question or in your settings, but there is a space in your transforms stanza name which is not there in your props.conf.

Where did you place these settings in your environment? It should be either directly on your indexer or on a heavy forwarder, depending on where your events are parsed.

Also, how does the setting fail? Is every event still indexed regardless of country code, or are none of them?

My guess is that these settings (even when applied properly) don't work for you because the transform is applied to _raw, and your raw data either doesn't contain SERVER01:, or that all of them do (assuming that SERVER01 is the value of your host field). So can we please see a small sample of an event? (Mask any sensitive data).

0 Karma

pjohnson1
Path Finder

Thanks for the input...

This is on the indexer under ..\system\local\ and it indexes everything and not filtering by the country codes.

Nov 10 10:42:23 SymantecServer SERVER01: KR-PC-098763,Continue,,File Read
Nov 10 14:22:23 SymantecServer SERVER01: CN-PC-012345,Continue,,File Read
Nov 10 15:32:23 SymantecServer SERVER01: SG-PC-054323,Continue,,File Read

I tested with the following and it appeared to work fine.
index=av | regex _raw="SERVER01:\s+(KR|SG|IN|PH|TW|TH)"

0 Karma

jeffland
SplunkTrust
SplunkTrust

If that's the raw data, that should work, so it would have to be something before actually applying the regex that isn't working as indended in that case. I'm curious, if you leave out the keep-LOG_transform stanza in the list of transforms applied in your props.conf, are events still indexed? That would mean changing your props.conf to

[log*]
TRANSFORMS-keep-LOG = setnull

If yes, then these settings aren't applied to the sourcetype at all. What exactly is your sourcetype named? Is using an asterisk in your stanza required, or could you temporarily use the precise sourcetype name to see if that helps? Also, is there another setting that may override this setting (settings for sourcetype are overridden by settings for host:: and source::)?

0 Karma

pjohnson1
Path Finder

The events are still indexed if just using TRANSFORMS-keep-LOG = setnull.

I have changed to the precise sourcetype and check via search but events are still being indexed.

[host:123.456.789]

and

[sep12:behavior]
[sep12:traffic]
[sep12:agt_system]
[sep12:scan]    
[sep12:risk]    
[sep12:log]

I have another props/transform which works perfectly for something else but this one is baffelling...

0 Karma

jeffland
SplunkTrust
SplunkTrust

Hi, sorry for taking so long to come back to you.
I see you are using [host:123], which I believe is supposed to be [host::123]. And the other stanzas, is your sourcetype sep12:behavior, and other settings are applied properly to this stanza?

0 Karma

pjohnson1
Path Finder

I referenced this https://answers.splunk.com/answers/47982/extracting-field-from-a-field-other-than-raw-in-props-conf....

[keep-LOG_transform]
SOURCE_KEY = host
REGEX = (KR|SG|IN|PH|TW|TH)
DEST_KEY = queue
FORMAT = indexQueue

But still no joy. Any guidance please...

Thanks.

0 Karma

krish3
Contributor

Please check this link and let me know how it goes.

Click Me

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...