Sourcetype Override Not Working!

[allidoiswinboom]

Hi All!

Hope all is well. I am about to pull my hair out trying to override a sourcetype for a specific set of tcp network events. The event starts with the same string of 'acl_policy_name' and it is currently being labeled with a sourcetype of 'f5:bigip:syslog'. I want to override that sourcetype with a new one labeled 'f5:bigip:afm:syslog' however, even after modifying the props and transforms conf files: still no dice.

I used regex101 to ensure that the regex for the 'acl_policy_name' match is correct but I've gone through enough articles and Splunk documentation to no avail. Nothing in the btools outputs for it looks out of place or as though it could be interfering with the settings below. Any thoughts or suggestions would be greatly appreciated before I throw my laptop off a cliff. Thanks in advance!

Event Snippet:

Inputs.conf
[tcp://9515]
disabled = false
connection_host = ip
sourcetype = f5:bigip:syslog
index = f5_cs_p_p

Props.conf
[f5:bigip:syslog]
TRANSFORMS-afm_sourcetype = afm-sourcetype
*Note I also tried [source::tcp:9515] as a spec instead of the sourcetype but no dice either way.

Transforms.conf
[afm-sourcetype]
REGEX = ^acl_policy_name="$
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::f5:bigip:afm:syslog
WRITE_META = true

[gcusello]

Hi @allidoiswinboom ,

where do you located these conf files?

they must be located in the first full Splunk instance that data passing through.

In other words, if you're using an Heavy forwarder to take these logs, you have to put them in the HFs, if instead you're using a Universal Forwarder, you have to locate them on Indexers or on an eventual intermediate HF.

Ciao.

Giuseppe

[allidoiswinboom]

Hi @gcusello Thanks for the reply. We are using UFs and I have the confs files on the deployment server and the indexers. We use a CM to manage all the Indexers so I deploy the updated files from the CM to ensure consistent hashing across the files. Thank you! 

[gcusello]

Hi @allidoiswinboom ,

if you haven't any intermediate HF, you must locate them on Indexers.

Ciao.

Giuseppe

[allidoiswinboom]

Hello @gcusello  Yes these files are already on the indexers.

[gcusello]

Hi @allidoiswinboom,

have you, on Indexers, also the Splunk_TA_F5?

because it transforms logs and maybe your transformation isn't effective because the f5:bigip:syslog is already transformed in something else.

using the regex, which sourcetype do your events have?

Ciao.

Giuseppe

[allidoiswinboom]

Hi @gcusello 

No, the 'Splunk_TA_f5-bigip' app is on the DS but not the IDXs/CM. Is that something that ought to be pushed out to the CM/IDX? 

We have a local app that is specific to our program with a ruleset for that f5:bigip:syslog but that is just specifying the hosts that route data to that index. 

How can I check via the regex what sourcetypes the data has? 

Thank you! 

[gcusello]

Hi @allidoiswinboom,

please, which sourcetype ha your running this search:

index = f5_cs_p_p
| regex "^acl_policy_name=\"$"

i addition, in the REGEX in transforms.conf, you should escape the quotes:

REGEX = ^acl_policy_name=\"$

Ciao.

Giuseppe

[allidoiswinboom]

Hi @gcusello 

I ran that search and no results were found. 😞 So is the regex incorrect? I was just trying to match that event referenced above. 

Thank you!

[gcusello]

Hi @allidoiswinboom,

I used your regex, if you haven't any result in the search, the issue is in the regex.

Ciao.

Giuseppe

[allidoiswinboom]

Hi @gcusello 

Ok great, if that's the case, I just want to match events that start with "acl_policy_name" so I can transform the sourcetype to something else. All the events start with that so I'm not sure what else I need to add to the REGEX? 

Thank you!

[gcusello]

Hi @allidoiswinboom ,

search the correct regex using regex101.com or, please, share some event so we can help you.

Ciao.

Giuseppe

[allidoiswinboom]

Hi @gcusello 

I have to black out certain information but see below: 

Thank you for your help!

[gcusello · Accepted solution]

Hi @allidoiswinboom,

to help you, I need them in text format to use in regex101.com, I cannot use a screenshot|

Anyway, why in your regex there's the dollar char ($)?

the correct regex should be 

REGEX = ^acl_policy_name\=\"

Ciao.

Giuseppe

[allidoiswinboom]

@gcusello  Sorry for the late reply but this helped with the creation of the sourcetype. Thank you for all your help! 🙂 

[gcusello]

Hi @allidoiswinboom ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉