All Apps and Add-ons

How do you define custom field at index time

kfleming
Explorer

I am seeing many answers, but they seem conflicting.
sample of log file

TimeStamp : 10/24/2014 11:50:01 PM

FullName : Microsoft.Practices.EnterpriseLibrary.ExceptionHandling, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
AppDomainName : /LM/W3SVC/3/ROOT-1-130586652039568763

ThreadIdentity : joeblow

WindowsIdentity : xxxx\xxxx

Category: DBLog

Priority: 0

EventId: 100

Severity: Error

Title:Enterprise Library Exception Handling

Machine: machinename

Application Domain: /LM/W3SVC/3/ROOT-1-130586652039568763

Process Id: 6108

Process Name: c:\windows\system32\inetsrv\w3wp.exe

Win32 Thread Id: 5336

Thread Name:

Extended Properties: ActorsClientIP - x.x.x.x

ActorsCompany - mycompanyname

ActorsFirstName - Joe

ActorsLastName - Blow

ActorsUserName - jblow

ActorsUserEmail - jblow.v@gmail.com

ActorsBrowserVersion - IE,8.0
RequestURL - https://mysecure.site.net

CustomMessage -

so, for this specific example:
How do may ActorsUserEmail = jblow.v@gmail.com work
have several approaches for search time, but need index time
does the field have to be defined in fields.conf on search head or indexer or forwarder
if so what is the syntax (complete please)

seen props.conf for indexer with just sourcetype and name of transform e.g.
[extrace]
TRANSFORMS-set = extrace_actorsuseremail

and transform.conf for indexer
[extrace_actorsuseremail]
REGEX = ActorsUserEmail\s+-\s+(?P.+?)\s*\n
Format = ActorsUserEmail::$1
WRITE_META = true
DEST_KEY = _meta

but this does not work.

so, to restate original question. extract and index name/value pair
name = ActorsUserEmail
value= email address to be extracted from log file that comes right after ActorsUserEmail -

Sure hope that was clearer than some of the answers I've seen

Tags (2)
0 Karma

s2_splunk
Splunk Employee
Splunk Employee

Attitude check, please.

Of course you can specify fieldname=value in a search using fields extracted at search time. That's one of the biggest values of Splunk to NOT define schema (fields) at index time.

And if you feel the documentation is lacking, I encourage you to post a comment on the page to point that out.

0 Karma

kfleming
Explorer

I will try your answer and see if I can get buy in
thanks

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

I am also challenging you on the need to add an indexed field. If you care to explain: why do you think you need it and what benefits do you expect?
In any case, I would recommend reading this, specifically the "Caution:" section at the top of the page. If you still feel you have a requirement for index-time fields, the doc contains all details on how to go about it; including examples.

,I would also challenge the need for an index-time field here. But if you insist and have a good reason, this has all the caveats, warnings and detailed instructions on how to go about ignoring those warnings; along with examples.

0 Karma

kfleming
Explorer

and if you guys don't know how
just say so

0 Karma

kfleming
Explorer

Thank you
I have already read it several times

0 Karma

vbumgarner
Contributor

On the search head, you will need this in fields.conf:

[ActorsUserEmail]
INDEXED=true

And you DON'T want DEST_KEY = _meta in your transform.

That said. Why do you think this needs to be an indexed field? There are really only a few cases where it is advantageous:
1. The thing you are extracting is in a metadata field (source, sourcetype, host)
2. The thing you need to search on is not an indexed term already, for instance a portion of a string.
3. The term you are looking for is very common, and creating the indexed field will drastically reduce the number of matched items.

So, unless #3 is true, this indexed field isn't actually going to speed things up for you, and will really just make your index bigger.

kfleming
Explorer

because is has been requested that users be able to type fieldname=value
and you cannot do that with search time field

0 Karma

kfleming
Explorer

sorry about the post
it seems to have stripped out crlf
i.e. there should be line breaks on the example above that are missing

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...