regex/props.conf - How to set properly the entries in .conf files?

[skender27]

Hi,

I extracted from the default source field, in search-time, a new field called 'domain':
| rex field=source "^(\/home)\/(?P<domain>\w+[^\/])"

(Practically it takes the directory which follows /home in a linux path)

Now I'd like to have this field extraction active by default, everytime I search the same sourcetype or index (and when I see the field extracted list from Settings).
I should put it correctly in the props.conf and transform.conf, but do not know exactly the syntax...

Any suggestions?

Thanks,
Skender

[alacercogitatus · Accepted solution]

The answer provided by woodcock is partially correct. The props.conf configuration would look like that. The transforms.conf is wrong.

Since you are dealing with the metadata field, you have to specify it as such, or it might not pick up the field, or it might pick up a different field "source" if it is in the event.

[source_to_domain]
SOURCE_KEY = MetaData:Source
REGEX = ^/home/(?<domain>[^/]+)

You could even do this inline on props.conf (only if you don't plan on repeating it on another sourcetype)

[mySourceType]
EXTRACT-domain = ^/home/(?<domain>[^/]+) in source

Check the docs if you have deeper questions:

http://docs.splunk.com/Documentation/Splunk/6.2.3/admin/Propsconf
http://docs.splunk.com/Documentation/Splunk/6.2.3/admin/Transformsconf

[skender27]

Hi,
I made a treasure from your discussion.
It hopefully worked and now it is ok.

Thanks again,
Skender K.

[woodcock]

Actually, we have the option to use it from metadata AND this is sometimes necessary but only in very early stage processing (where source does not exist yet, but MetaData:Source does) or where source has been overridden (which OP does not state). However, for search-time processing, it is totally valid (and less complicated for newbies) to use source instead of MetaData:Source, although either will work just fine. Also, you should use the at-syntax for user names, like @woodcock and @alacercogitatus, that way people are less likely to miss when they have been called out.

[alacercogitatus]

@skender27 specifically mentioned default source field, so to prevent confusion, we should, as a best practice, use the MetaData call to explicitly define the source field. Either will work, but as required by the question, we should limit the scope of the parsing to exactly what is required. Making assumptions on the level of skill of a user is not something I typically do, I don't assume newbie, unless the OP states "I am a noob". Either way, @woodcock, your regex is still wrong and won't work as typed. Perhaps a typo? But since I don't assume as to a user's skill level, I just answered the question to completeness, and to cover an additional format if so desired by OP (EXTRACT also works well here) along with posted references.

[woodcock]

Making assumptions on the skill-level of posters is something I almost always do and it is beneficial to do so because it gives answerers more freedom to "interpret" what is being "meant" by the way it is asked. Based on the full character of OP's phrasing, everyone but the most contrary (even @skender27) will admit newbie status (which is not pejorative) so I think you are over-literalizing OP's default source phrasing. Even so, I have to give you credit: you make a good point if that is what he meant. As far as my answer being "wrong", it is not "wrong" for any reason that you described. It is "right" for all the reasons that the OP should care about. If it is "wrong" in any other sense, it has to be "wrong" because the RegEx that he gave is wrong (I notice that you have different RegEx than OP gave). I also notice that the original question has been re-edited to fix markdown mistakes by OP and that is why my RegEx looks (is) wrong; because I blindly copied it from OPs (newbie) text. OP was not asking about fixing his RegEx, he was asking about automating it and my answer is entirely correct (if perhaps sub-optimal in your opinion) for that purpose. But again, I admit my failure to test and to examine possibly markdown-damaged RegEx.

[woodcock]

I have re-edited my answer to fix the RegEx but I deliberately did not test it just to spite @alacercogitatus in my own heart by not doing everything that he would like me to do just as he would like me to do it. SO THERE! 😆

[woodcock]

In 'props.conf(same path that you putinputs.conf`) add something like this:

[mySourceType]
REPORT-source_to_domain = source_to_domain

In 'transforms.conf:

[source_to_domain]
SOURCE_KEY=source
REGEX=^/home/(?<domain>\w+[^/])

Then save this to your Search Heads

[skender27]

Hi,

I have a stand-alone server.
Should I put these stanzas under my_app/default or under my_app/local directory?
I have inputs.conf in the local one and transforms.conf only in the default one.

Thanks a lot,
Skender

[woodcock]

The person who is the author of the app should put his changes in default and publish them out. So you, as author, should put all of your files in default (including inputs.conf). It will work either way you do it so it isn't that big a deal (until somebody comes along and overwrites your stuff in local accidentally). Just know that local has precedence over default so be sure there is nothing in local after you move to default.