extract field from the source's file path and make it the host field

[nmohammed]

Hi,

I am trying to extract the field from the log file path which includes the actual host. currently, the host field is populated with the third segment of the log file path that is the clientid field. But what we want is the actual host name . we are currently indexing from a shared mount which is the reason not able to capture the actual host name

Example source field with log file name

/emp_logs-sc9/loaner/DE123456/EmpServer.DE123456.SC9VEABE1092.2014-04-13-11.log. 

current host field - DE123456

expected host field - SC9VEABE1092

can someone guide me, how to achieve this?

Thanks

[cramasta]

on your indexer you can try something like this. these changes require a restart of the indexer

#props.conf
[source::/emp_logs-sc9/loaner/DE123456/*]
TRANSFORMS-hostFromSource=hostFromSource


#transforms.conf
[hostFromSource]
SOURCE_KEY = MetaData:Source
REGEX=.*\/.*?\..*?\.(\w+)
FORMAT = $1
DEST_KEY= MetaData:Host

The regex really just needs to be something that has a capturing group of what you want the hostname to be when ran against the source. There may be a better regular expression depending on what the rest of your logs source paths look like.

[nmohammed]

tried this.. still doesn't work

[hostFromSource]
SOURCE_KEY = MetaData:Source
REGEX=EmpServer.[A-Za-z0-9]*.([^.]+)
FORMAT = host::$1
DEST_KEY= MetaData:Host

[cramasta]

and you put this on the indexers where the data is being collected?

Can you also try hardcoding in a specific current source in your props.conf to make sure we its not the wildcards/ ... thats throwing it off.
[source::/emp_logs-sc9/loaner/DE123456/EmpServer.DE123456.SC9VEABE1092.2016-04-30-X.log]

[nmohammed]

HI Cramasta,

Sorry was out for while, But I did try this, still not working.

[nmohammed]

I tried this out, but did not work in props and transforms settings with regex. It works with rex in search though.

Props.conf:

[source::/emp_logs*/.../*.log]
TRANSFORMS-hostFromSource=hostFromSource

transforms.conf :

[hostFromSource]
SOURCE_KEY = MetaData:Source
REGEX=EmpServer.\w+.(\w+)
FORMAT = $1
DEST_KEY= MetaData:Host

and here is the rex search which worked in over-riding the host field

index="emp_logs" | rex field=source "EmpServer.[A-Za-z0-9]*.(?P[^.]+)"

[cramasta]

Did you put this setting on your indexers?

Does using the regex that you are putting in your props.conf work in search with rex?

[cramasta]

Try this, didnt think its needed buy maybe it is

Props.conf:

[source::/emp_logs*/.../*.log]
TRANSFORMS-hostFromSource=hostFromSource

transforms.conf :

[hostFromSource]
SOURCE_KEY = MetaData:Source
REGEX=EmpServer.\w+.(\w+)
FORMAT = host::$1
DEST_KEY= MetaData:Host

[nmohammed]

I tried this.. did not work.
transforms.conf

[hostFromSource]
SOURCE_KEY = MetaData:Source
REGEX=EmpServer.[A-Za-z0-9]*.([^.]+)
FORMAT = $1
DEST_KEY= MetaData:Host

Let me try the updated one

[cramasta]

Ok try it with the last update I provided which adds
FORMAT = host::$1

[nmohammed]

Thanks cramasta,

all my log files are of the same format. the clientid and hostname in the source changes.

/emp_logs-sc9/loaner/DE123456/EmpServer.DE123456.SC9VEABE1092.2014-04-13-11.log.

DE123456 is actually the clientid

but in the fourth segment "EmpServer.DE123456.SC9VEABE1092.2014-04-13-11.log" the hostname is SC9VEABE1092. This is what we want to replace the host field with.

[cramasta]

did this end up working? if so please mark this as accepted. thanks

[cramasta]

so it seem like you get the basic idea here, i just captured the wrong part of the group. All you have to do is change the regex to capture that group. Also updated my original answer

.*\/.*?\..*?\.(\w+)

[cramasta]

or something like this for your regex. theres more than one way to write the regex depending on what the sources look like.
EmpServer\.\w+\.(\w+)

[cramasta]

one day Ill eventually figure out how to get formatting to work on this site.