The data below has NO key-value pairs, but the bold data is very important. If I could somehow extract the bold data below into the following fields, it would be very helpful:
(in order of boldness, and separated by a space. the bold data can obviously be different with each log entry, but could also be repeated)
Any suggestions?
Thanks, Sean
BTW - this is a Cognos log
192.168.27.13:9081 9892 2010-05-13 04:59:28.439 -6 ECD46F7C56E2D4F6E57931FA643C78980F4264F2 Ch9v29jvh228Glw8w8jd9M2jwC8M9wMvj448j8s2 Ch9v29jvh228Glw8w8jd9M2jwC8M9wMvj448j8s2 3104 RSVP 1198 3 Audit.RTUsage.RSVP Execute Report /content/folder[@name='Vendor Reports']/folder[@name='Mars']/report[@name='DSVND05 - DISTRIBUTION REPORT XLS'] Success
You can extract this at search time (interactively) using rex
. Once you have a good regular expression defined, then you can stick it into your props.conf
file to match against either a source or sourcetype that this data is found in. Or, you can use the interactive field extractor.
Option 1: Interactive extraction
your search ... | rex "^\S+ (?P<pid>\d+) \S+ \S+ \S+ (?P<session_id>\S+) (?P<request_id>\S+)"
Options 2: Setup and entry in props.conf:
[my_source_type]
EXTRACT-key_fields = ^\S+ (?P<pid>\d+) \S+ \S+ \S+ (?P<session_id>\S+) (?P<request_id>\S+)
Option 3: Use the interactive field extractor. See the Fields extraction tutorial.
Hope that helps you get started.
Just thought I would add the way I did this which can be found in my question here: http://answers.splunk.com/answers/112243/delimited-field-extractions-for-cognos-data
sometime you can just see the data is separated by tabs and can name the columns rather than using regex!
Build a transforms as follows:
[cognos-fields]
DELIMS="t"
FIELDS="Host ID","Process ID","Time","Time Zone","Session ID","Request ID","SubRequest ID","Step ID","Thread","Component ID","Build Number","Level","Logger","Operation","Object Type","Object Path","Status","Message","Log Data"
much easier I find 🙂
Presumably this data can be identified by a regular expression or delimiters between fields. In that case, you have a variety of ways to perform extractions. Please see: http://www.splunk.com/base/Documentation/latest/Knowledge/Addfieldsatsearchtime and http://www.splunk.com/base/Documentation/latest/Knowledge/Createandmaintainsearch-timefieldextractio...
You can assign an arbitrary field name as part of a search time extraction. Try something similar to this...
In ${SPLUNK_HOME}/etc/system/local/transforms.conf:
[cognosfields]
REGEX=^[^\s]+ ([0-9]+) [^\s]+ [^\s]+ [^\s]+ ([^\s]+) ([^\s]+)
FORMAT= pid::$1 sessionid::$2 requestid::$3
In ${SPLUNK_HOME}/etc/system/local/props.conf:
[cognos_type]
REPORT-cognos=cognosfields
Reload the configs by doing a search in the splunkweb UI of "| kv reload=true" - and see if your fields show up.
As of 4.1.x, the reload command is not necessary - just run another search.
Right you are... I use REPORT- in all of my configs, but goofed here... /me fixes
Dwaddle, I think you mean REPORT-cognos=cognosfields
, not EXTRACT
I think they used to be equivalent in Splunk 3.x, but they aren't in 4.0
You can extract this at search time (interactively) using rex
. Once you have a good regular expression defined, then you can stick it into your props.conf
file to match against either a source or sourcetype that this data is found in. Or, you can use the interactive field extractor.
Option 1: Interactive extraction
your search ... | rex "^\S+ (?P<pid>\d+) \S+ \S+ \S+ (?P<session_id>\S+) (?P<request_id>\S+)"
Options 2: Setup and entry in props.conf:
[my_source_type]
EXTRACT-key_fields = ^\S+ (?P<pid>\d+) \S+ \S+ \S+ (?P<session_id>\S+) (?P<request_id>\S+)
Option 3: Use the interactive field extractor. See the Fields extraction tutorial.
Hope that helps you get started.
Splunkerific! Thanks for the help - IFX worked like a charm.