Getting Data In

Field extraction from information in field=source

zindain24
Path Finder

My webserver logs are sent to my indexers through a Universal Forwarder.

*Snippet from inputs.conf on the Universal Forwarder

[monitor:///path/to/apache/2.2/web/.../logs/*access_log]

disabled = false

sourcetype = access_combined

index = internet

followTail=0

With this configuration, we properly set the following fields
index = internet,
host = unixservername,
sourcetype = access_combined

The problem is, we need a field with the webserver name in segment 6 of the source:
/path/to/apache/2.2/web/.../logs/*access_log

We tried adding host_segment = 6 to the forwarder stanzas, but then we lose our true "host = unixservername" which is also necessary. Unfortunately, this information is NOT available anywhere but the source field.

So....

We can easily create a search time |rex for Splunk to process to pull the information:
|rex field=source "\/path\/to\/apache\/[0-9].[0-9]\/\w+\/(?.*?)\/"

This works well... however, I don't want my users to have to run this every time they search.

I would like the ability to add this as a Index time or Search time extraction through props and transforms -- preferably at the forwarder or indexer level. Any suggestions? Thanks for your help, ideas, and input! I'm stuck...

Jeremy

0 Karma
1 Solution

Ayn
Legend

Just set up a field extraction as usual - are you familiar with how to do this in props.conf / transforms.conf? If so, it's just like a regular field extraction but you specify SOURCE_KEY = (yourfieldhere) (for REPORT style extractions that reference a transforms.conf entry) or EXTRACT = <yourregex> in <yourfieldhere> (for EXTRACT style extractions directly in props.conf).

So for the first case it'd be something like...props.conf:

[yoursourcetype]
REPORT-getfieldfromsource = getfieldfromsource

transforms.conf:

[getfieldfromsource]
SOURCE_KEY = source
REGEX = /path/to/apache/[0-9]\.[0-9]/\w+/(.*?)/
FORMAT = webserver::$1

Or, in the second case, just throw your rex statement in almost unaltered into an EXTRACT extraction in props.conf:

[yoursourcetype]
EXTRACT-getfieldfromsource = /path/to/apache/[0-9]\.[0-9]/\w+/(?<webserver>.*?)/ in source

View solution in original post

Ayn
Legend

Just set up a field extraction as usual - are you familiar with how to do this in props.conf / transforms.conf? If so, it's just like a regular field extraction but you specify SOURCE_KEY = (yourfieldhere) (for REPORT style extractions that reference a transforms.conf entry) or EXTRACT = <yourregex> in <yourfieldhere> (for EXTRACT style extractions directly in props.conf).

So for the first case it'd be something like...props.conf:

[yoursourcetype]
REPORT-getfieldfromsource = getfieldfromsource

transforms.conf:

[getfieldfromsource]
SOURCE_KEY = source
REGEX = /path/to/apache/[0-9]\.[0-9]/\w+/(.*?)/
FORMAT = webserver::$1

Or, in the second case, just throw your rex statement in almost unaltered into an EXTRACT extraction in props.conf:

[yoursourcetype]
EXTRACT-getfieldfromsource = /path/to/apache/[0-9]\.[0-9]/\w+/(?<webserver>.*?)/ in source

zindain24
Path Finder

Fantastic Ayn, Thank you for the detailed response!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...