I'm colleting WinEventLog:Application from windows machine using UniveralForwarder, the collected data is been sent to a HeavyForwarder cluster where the data is analised and if reach my criteria the data is changed using some SEDCMD and transformation, after this the data is sent to a Splunk Indexer.
Doing this, when data arrive to Splunk Indexer the automatic extraction field doesn't work. You can check that my events has well formatted messsages, like:
18/10/2013 01:25:28.7974852 PM AppName="vtex.requestcapture.windowsservice" AppVersion="7.0.1.0" Env="local" AZ="us-east-1e" WorkflowType="events processing" WorkflowInstance="lojaadcos" Level="Warn" LoggerVersion="7.0.0.0" OperationId="1789f3b5-d82a-4aa0-b6de-60ac693f5fb2" ExceptionType="applicationexception" VtexLog
18/10/2013 01:32:07.1257497 PM AppName="vtex.requestcapture.windowsservice" AppVersion="7.0.1.0" Env="local" AZ="us-east-1b" PageLoadTime="30986" Count="5" Browser="Chrome" TimeType="NetworkingTime" Account="EletronicaSantana" VtexMetric
As you can see, my messages only has a Timestamp and a sequence of Key/Values. If I do a simples search using extract auto=t like index=main | extract auto=t all fields are extracted correctly, but honestly I need it done to all data.
Now let me show my configs, to see if someone can help me to have this automatic extraction working again.
HeavyForwarder - props.conf
[WinEventLog:Application]
SEDCMD-remove_eol = s/[\n\r]/ /g
SEDCMD-remove_non_logger_event = s/(.)(VtexLog|VtexMetric|VtexMachineMetric)(.)/\1\2\3\2/g
SEDCMD-remove_primary = s/(.*)(\d\d\/\d\d\/\d\d\d\d\s\d\d:\d\d:\d\d.\d\d\d\d\d\d\d)(.+?$)/\2\3/1
SEDCMD-remove_exception_stacktrace = s/(.+)(ExceptionStack=)(.*)/\1/g
REPORT-field_extraction = logger_field_extraction
KV_MODE = none
TRANSFORMS-set = setNull,setLog,setMetric,setMachineMetric,setLogSource,setMetricSource,setMachineMetricSource,setRequestCaptureIndex
HeavyForwarder - transform.conf
[logger_field_extraction]
DELIMS = " "
[setNull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setLog]
REGEX = VtexLog
DEST_KEY = queue
FORMAT = indexQueue
[setMetric]
REGEX = VtexMetric
DEST_KEY = queue
FORMAT = indexQueue
[setMachineMetric]
REGEX = VtexMachineMetric
DEST_KEY = queue
FORMAT = indexQueue
[setLogSource]
REGEX = VtexLog
FORMAT = sourcetype::VtexLog
DEST_KEY = MetaData:Sourcetype
[setMetricSource]
REGEX = VtexMetric
FORMAT = sourcetype::VtexMetric
DEST_KEY = MetaData:Sourcetype
[setMachineMetricSource]
REGEX = VtexMachineMetric
FORMAT = sourcetype::VtexMachineMetric
DEST_KEY = MetaData:Sourcetype
[setRequestCaptureIndex]
REGEX = (AppName="vtex.requestcapture.windowsservice"|AppName="vtex.requestcapture.processwebsite")
FORMAT = rc
DEST_KEY = _MetaData:Index
As you guys can see on my HF if data matches what I'm looking for I change the source type and finally and chosse index.
Reading yestarday about how to use DELIMS to extract fields I set the DELIMS on HeavyForwarder, but I also have this on my Indexer:
Indexer - transforms.conf
[logger_field_extraction]
DELIMS = " "
Indexer - props.conf
[WinEventLog:Application]
REPORT-field_extraction = logger_field_extraction
KV_MODE = none
[source::WinEventLog:Application]
REPORT-field_extraction = logger_field_extraction
KV_MODE = none
[VtexLog]
REPORT-field_extraction = logger_field_extraction
KV_MODE = none
[VtexMetric]
REPORT-field_extraction = logger_field_extraction
KV_MODE = none
[VtexMachineMetric]
REPORT-field_extraction = logger_field_extraction
KV_MODE = none
As you guys can see I tried to apply this Delim anyway but it isn't working.
Again, I need to have all my Key/Values avaliable on my Indexer, and I'm losing this because I use a HF.
Any ideias?
First I would like to say thanks to alacercogitatus and emiller42, because of your anwsers I knew that I was going on right direction.
So, the reason why my props and transforms conf files weren't working on my indexer/search server was the missing of metadata file exporting props to be used by system.
As soon I added:
[]
export = system
to default.meta.conf file to my app and redeployed it, the extraction started to work.
Thanks God now it's 100% now ...
First I would like to say thanks to alacercogitatus and emiller42, because of your anwsers I knew that I was going on right direction.
So, the reason why my props and transforms conf files weren't working on my indexer/search server was the missing of metadata file exporting props to be used by system.
As soon I added:
[]
export = system
to default.meta.conf file to my app and redeployed it, the extraction started to work.
Thanks God now it's 100% now ...
With the way your events are formatted, Splunk should have no issue automatically extracting the fields with zero configuration. I think the primary problem with your setup is the
KV_MODE = none
setting. This tells Splunk not to do any automatic key-value pair extraction, which is the opposite of what you want.
Get rid of that setting in all of your props.conf stanzas, and get rid of the [logger_field_extraction] stanza in transforms. I would expect your fields to work as expected after that.
If you insist on keeping KV_MODE = none, then change your transforms stanza as follows:
[logger_field_extraction]
DELIMS = " ", "="
Keep in mind that may get inconsistent where your values can contain spaces. Auto-parsing handles this when the values are quoted, as yours are.
Hi emiller42, I tried first without KV_MODE and DELIM, since it didn't worked I tried with DELIM and them with KV_MODE. I will be re-doing my tests withou KV_MODE and with DELIM as you told. I will be back telling the results.
DELIMS won't help you in this case. Make sure your searching with Verbose mode, and not fast or smart. If you have key="value", splunk will pick it up. Additionally, DELIMS is search-time so it needs to be on a search head.
http://docs.splunk.com/Documentation/Splunk/6.0/Search/Changethesearchmode
alacercogitatus, I had tried it before, and now I tried again but it doesn't work too.
DELIMS is best used with csv, but can be used with other values. If you want to use DELIMS set it equal to " ", "="
and see if that helps.
Since I'm using my Indexer server as a search server, why DELIMS aren't working?
alacercogitatus, even using verbose mode the fields aren't been extracted.