rex field command in props.conf file

[pgabo66]

The event.url field stores all the urls found in the logs, I want to create a new field called url_domain that only captures the domain of the urls stored in event.url, temporarily what I do is from the search write the following:

| rex field=event.url "^(?:https?:\/\/)?(?:www[0-9]*\.)?(?)(?<url_domain>[^\n:\/]+)"

What should I add in the props.conf so that this instruction is fixed for the sourcetype "sec-web"?

[gcusello]

Hi @pgabo66 ,

you have to create a new field associating it to your sourcetype and using this rule:

^(?:https?:\/\/)?(?:www[0-9]*\.)?(?)(?<url_domain>[^\n:\/]+) in event.url

in the field extraction.

Ciao.

Giuseppe

[pgabo66]

Do you have an example of how the props.conf would look like with that rule?

I've tried several sentences but it still doesn't take it.

[bowesmana]

You can do this in the UI - go to Settings->Fields-Field Transformations and add the regex and the field you want to extract from and then in Field Extractions add a new Extraction using transforms and reference the Field Transformation.

This will translate to something like this in props/transforms conf files

In transforms.conf you will need

[url_domain]
CLEAN_KEYS = 0
REGEX = ^(?:https?:\/\/)?(?:www[0-9]*\.)?(?)(?<url_domain>[^\n:\/]+)
SOURCE_KEY = event.url

 In props.conf 

[sec-web]
REPORT-file_name = url_domain

[pgabo66]

In the props.conf example, when it says "REPORT-file_name = url_domain", what should I replace file_name with?

I'll stay tuned, thank you very much.

[bowesmana]

REPORT-url_domain

It's the name of the field you want to assign the result to.

[pgabo66]

So with the "SOURCE_KEY = event.url" what I do is call the field where I want to get the information from?
 In my case it would be the urls stored there.

[bowesmana]

In your rex example you said

| rex field=event.url ...

that is why SOURCE_KEY is event.url - as that is where the urls are coming from right?

Your rex example indicated you are extracting the url into a field called url_domain, which is also what is in the transforms.

[pgabo66]

I did what you explained to me but it still doesn't work, when I check the zscaler logs apun the url_domain field does not appear.

It is important to mention that I am implementing this from a custom app for zsacaler.

[bowesmana]

It is probably because your field looks like it has come from JSON and based on the link provided by @isoutamo , means that the field extractions are happening at stage 4, whereas your REPORT extraction is happening at stage 3, therefore the field does not exist.

You could try creating a calculated field using an eval replace expression to remove the non-domain part.

You can try this in standard SPL by experimenting with your regex using

| eval domain=replace('event.url', "(?:https?:\/\/)?(?:www[0-9]*\.)?(?)([^\n:\/]+)", "\1")

That is NOT correct above, as I am not sure what the replacement token \1 should be with all the brackets and capturing/non-capturing groups, but you can experiment with regex101.com

[isoutamo]

Hi

here is order how those are managed in search time https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Searchtimeoperationssequence

You should ensure that this field has defined before you can use those e.g. in transforms.conf. E.g. if you are using ALIAS-field1 on props.conf you cannot use that field1 as  a SOURCE_KEY in props.conf. In this kind of situation you should extract that information from _raw instead of field which has defined on later phase of input sequence.

I'm not sure about your event.url field is same as this TA has defined or not. If it's then you can see in props.conf that it has defined like

EVAL-url = Host+URL

and if this is your event.url field then it didn't  exists yet when you try to use it on transforms.conf.

r. Ismo

[pgabo66]

Hi @isoutamo and @bowesmana,

I have tried the ways shared by you but it still doesn't work it's like Splunk doesn't read the transforms.conf I checked the logs of the index=_internal but I don't see any errors related to it.