Fortigate TA Config Queries

[gvmorley]

This is one for the folks at Fortigate regarding their TA.

I had a couple of queries on the latest version (1.6.0), specifically around the transforms.conf

Looking at the force_sourcetype_fgt_traffic stanza:

[force_sourcetype_fgt_traffic]
DEST_KEY = MetaData:Sourcetype
REGEX = devid=\"?F[G|W|6K].+type=\"?traffic
FORMAT = sourcetype::fgt_traffic

I'm 99% sure that you can't do alternation inside a character class. So [G|W|6K] actually means any character from the set GW6K or the literal |.

Was the intention that it should actually be (?:G|W|6K), which would be a non-capturing group of either G, W or 6K?

Secondly, you might want to reconsider the structure of these a bit. Taking that same stanza as above, it would actually match a utm event like this:

date=2019-03-15  time=05:12:28 logver=52 timestamp=1552626748 tz="UTC+0" devname="example-dev" devid="FG600C9999999999" logid="12345" type="utm" subtype="app-ctrl" eventtype="app-ctrl-all" level="information" vd="root" appid=12345 user="" srcip=10.1.1.1 srcport=64224 srcintf="INTERNAL" dstip=8.8.8.8 dstport=80 dstintf="EXTERNAL" proto=6 service="HTTP" policyid=42 sessionid=158758809 applist="standard" appcat="Web.Others" app="HTTP.BROWSER" action="pass" hostname="test.example.com" url="/net/WebService.aspx?type=traffic&Password=fred8" msg="Web.Others: HTTP.BROWSER," apprisk="medium"

It's the first of the force transforms tried against fgt_log. So even though the event has type="utm", it matches later in the event due to data in the url field. So the result would be that this event would be sourcetype=fgt_traffic. Now, as the remaining force transforms continue to be applied, it would probably then match the force_utm one later and then get set to that. 🙂 But it would probably be better to get it set correctly once.

One approach might be to combine your 3 force stanzas into one. So look to do this:

default/props.conf
[fgt_log]
TRANSFORMS-force_sourcetype_fgt = force_sourcetype_fgt

default/transforms.conf
[force_sourcetype_fgt]
SOURCE_KEY = _raw
DEST_KEY = MetaData:Sourcetype
REGEX = devid=\"?F(?:G|W|6K).+type="?(traffic|utm|event)
FORMAT = sourcetype::fgt_$1

You also get the advantage that Splunk only has to try the one stanza / regex per event, as opposed to three.

Finally, although it's a bit beyond the scope here, so may also want to look to anchor the regex and consider some backtracking control.

This proved more performant for a lot of my testing:

[force_sourcetype_fgt]
SOURCE_KEY = _raw
DEST_KEY = MetaData:Sourcetype
REGEX = ^.+?\sdevid=(*COMMIT)\"?F(?:G|W|6K).+?\stype=(*COMMIT)\"?(traffic|utm|event)
FORMAT = sourcetype::fgt_$1

The (*COMMIT) is a way of saying, "It you get this far in the match, but fail later, don't backtrack past this".

So the regex above is saying, "Going from the start of the line ( ^ ), look for any character 1 or more times (lazily). When you find devid=, then we know we're in the right place. If our F(?:G|W|6K) doesn't match, then don't go back and try to find it anywhere else. Once we got to devid= we were committed - there's no going back now..."

Basically, it's a handy way to stop the regex being tried later in the event. But I'm not regex guru, so do some testing and see how it works. It just proved useful for us in our environment.

The main one is really to see if you need to fix-up the alternation in the character class and possibly combine the stanzas in a future TA release.

[jerryzhao · Accepted solution]

Hi gvmorley
Thanks for pointing out the bug and providing a better approach.
We sure will verify it and include the fix in future release.
Thanks again!

[gvmorley]

Thanks for the quick response. And thanks again for supporting Splunk (and Splunkers) with your TA. It's greatly appreciated. G.