Splunk Search

Way to insert/create field based on segment of source?

mfrost8
Builder

As a sort of followup to my earlier question at

Way to insert/create field based on source?

we're interested in doing a lookup or mapping based on part of the source name and generating a field.

The use case is for IIS. It's not particularly useful to see an error in W3SVC123456789 when you don't have a clue what site that error actually corresponds to. It would be great if we could create (manually, I assume) a mapping from the W3SVC####### name to an actual site name. For the directory above, if the site is named "FOO", we'd do something to create a new field "site" with that name.

I'm guessing this also has to be an index-time extraction. It's a lot like the question I asked above, except that it would be a mapping from part of the source name rather than creating the field based on the source name itself.

Thanks

Tags (2)
0 Karma
1 Solution

Paolo_Prigione
Builder

You can't use lookups or tags to INDEX a new field. You can, however, index the segment of the source that is going to be used in lookups.

(I use lookups straight on source for a use case similar to yours and they work very well. I also generate the lookup dinamycally every 15 minutes through a search.)

Back on topic... You'll need props, transforms and fields.conf

props.conf

[your source type]
TRANSFORMS-site = iss_site_code
LOOKUP-site = iss_site_code_lookup site_code OUTPUT site

transforms.conf

[iss_site_code]
SOURCE_KEY = MetaData:Source
REGEX = ...whathever regex applies...W3SVC(\d+)  
FORMAT = site_code::$1
WRITE_META = true

[iss_site_code_lookup]
filename = mysites.csv
default_match = n.a.
max_matches = 1

fields.conf

[site_code]
INDEXED = true

View solution in original post

0 Karma

Paolo_Prigione
Builder

You can't use lookups or tags to INDEX a new field. You can, however, index the segment of the source that is going to be used in lookups.

(I use lookups straight on source for a use case similar to yours and they work very well. I also generate the lookup dinamycally every 15 minutes through a search.)

Back on topic... You'll need props, transforms and fields.conf

props.conf

[your source type]
TRANSFORMS-site = iss_site_code
LOOKUP-site = iss_site_code_lookup site_code OUTPUT site

transforms.conf

[iss_site_code]
SOURCE_KEY = MetaData:Source
REGEX = ...whathever regex applies...W3SVC(\d+)  
FORMAT = site_code::$1
WRITE_META = true

[iss_site_code_lookup]
filename = mysites.csv
default_match = n.a.
max_matches = 1

fields.conf

[site_code]
INDEXED = true
0 Karma

mfrost8
Builder

Thanks, Paolo. I'm intrigued about how you generate the lookup table dynamically. Is that some kind of scripted input that watches some registry keys to map IIS sitenames to crazy W3* dirs? Would you mind sharing? Thanks.

0 Karma

mfrost8
Builder

Thanks. I'm familiar with lookup tables, but I'm still concerned that there's a piece missing here. I had thought that I'd extract the W3SVC### foldername from the source into a new field (let's call it the 'siteid' field) somehow in inputs.conf. Then I could make a lookup table that mapped that siteid into a friendly sitename.

I guess I could make lookup table whose input field was the full source path, but that seems a little odd.

I can't really see how I can extract only one part of teh sourcename and stick it into a new field in DEST_KEY for transforms.conf.

0 Karma

lguinn2
Legend

I think that what you want is a lookup table. A lookup table is a static CSV file that you upload to Splunk; it can provide the mapping that you want. I assume that the CSV file would have one column for the id (######) and one column for the actual site name. See the tutorial on lookups and I think you will be able to set up a lookup pretty quickly.

You will need an input field (key) for the lookup. This is the field that you refer to in your other question, so I will comment there also.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...