Will someone help me with my REGEX in this Transforms.conf?

[haoban]

8/30/18 9:38:51.000 AM **rec_type=71** dns_query=s3.amazonaws.com dns_record_name=A src_tos=0 ssl_expected_action=Unknown ...
8/30/18 9:14:19.000 AM **rec_type=109** id=185 rec_type_desc="Web Application" name="Google Translate" rec_type_simple=PAYLOAD
8/30/18 9:13:55.000 AM **rec_type=520** id=600 rec_type_desc="Geolocation Data" name=paraguay rec_type_simple=GEOLOCATION
8/30/18 9:13:55.000 AM **rec_type=63** id=1033 rec_type_desc="Server Metadata" name=Youku rec_type_simple=SERVICE

props.conf
[cisco:estreamer:data]
TRANSFORMS-send-data-to-null-queue = setnull_2

transforms.conf
[setnull_2]
REGEX = (rec_type=(?<!71))
DEST_KEY = queu
FORMAT = null Queue

I'd like all rec_type not equal "71" send to nullQueue. But seems my REGEX doesn't work. Can anyone help? Thanks!

[horsefez]

Hi,

props.conf

 [cisco:estreamer:data]
 TRANSFORMS-cisco = throw_away, index_this

transforms.conf

[throw_away]
 REGEX = rec_type=(?!71)
 DEST_KEY = queue
 FORMAT = nullQueue

 [index_this]
 REGEX = rec_type=(?=71)
 DEST_KEY = queue
 FORMAT = indexQueue

[haoban]

[cisco:estreamer:data]
TRANSFORMS-send-data-to-null-queue = setnull_2, setnull_1

[cisco:estreamer:log]
TRANSFORMS-drop-data = setdrop

[cisco:estreamer:status]
TRANSFORMS-drop-data = setdrop

[setnull_1]
REGEX = ^fw_rule_reason\=N\/A$
DEST_KEY = queue
FORMAT = nullQueue

[setnull_2]
REGEX = ^((?!rec_type\=71).)*$
DEST_KEY = queu
FORMAT = null Queue

[setdrop]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

Seems the issue is not about the REGEX, it's about how to make multi-REGEX to work. Every time when I restart the service to test it, I only can have one correct logs. rec_type not equal "71" works or fw_rule_reason=N/A works.

[sudosplunk]

Hi there,

Try this and see if it works. Below combo routes all events to nullQueue except events containing rec_type=71

props.conf:

[cisco:estreamer:data]
TRANSFORMS-cisco = setnull, setparsing

transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = rec\_type\=71
DEST_KEY = queue
FORMAT = indexQueue

[haoban]

Thanks for replying. But this splunk is just a forwarder, don't have any license in it. Can I still using the "indexQueue"?

[sudosplunk]

If you installed the Universal Forwarder, and told it to get the data and forward it to Splunk Indexer, your configuration should go to Indexers (or heavy forwarder if you have one).

The incoming data passes through a couple of phases; Input, Parsing, Indexing and Search. These type of nullQueue routing operations take place during the Parsing phase. Universal Forwarder cannot do parsing, and will just ignore any such settings

The below link is really old but explains basics well.
http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F

[haoban]

the regex pass the "https://regex101.com/" online test, but still doesn't work. Does anyone know why?

[haoban]

[setnull_2]
REGEX = ^((?!rec_type\=71).)*$
DEST_KEY = queu
FORMAT = null Queue

This website is really helpful "https://regex101.com/"