Splunk Search

How to extract fields from a multi line messages

jdagenais
Explorer

We have a multi line message that looks like this:

11/30/10 16:28:34 Verifying pricing env CLOSE,FX_CLOSE,XLA_ENV,INTRADAY,CPTY_CREDIT
   No exceptions for CLOSE, loaded in 0.05 secs
   Messages for FX_CLOSE
      PricerConfigRefresh: No item found for 1246892/CurveZero (CLOSE)  before Tue Nov 30 16:28:34 EST 2010
   No exceptions for FX_CLOSE, loaded in 0.17 secs
   No exceptions for XLA_ENV, loaded in 0.05 secs
   No exceptions for INTRADAY, loaded in 0.10 secs
   No exceptions for CPTY_CREDIT, loaded in 0.40 sec

I would like to create the following fields and assign the numeric values as response time, so I can search and chart on the RT values.

  close_pe_rt=0.05
  fx_close_pe_rt=0.17
  xla_env_pe_rt=0.05
  intraday_pe_rt=0.10
  cpty_credit_pe_rt=0.40

What is the best way to handle such extraction and assignments?

Thanks very much for your help, Jean

Tags (1)
1 Solution

hulahoop
Splunk Employee
Splunk Employee

These are the 2 options I would try

  • configuration files
  • rex command in the search bar

The easiest, but also most transient, option is to use rex command inline in your search. For example:

sourcetype="multiline" | rex "CLOSE, loaded in (?<close_pe_rt>\S+)" | rex "FX_CLOSE, loaded in (?<fx_close_pe_rt>\S+)" | rex "XLA_ENV, loaded in (?<xla_env_pe_rt>\S+)" | rex "INTRADAY, loaded in (?<intraday_pe_rt>\S+)" | rex "CPTY_CREDIT, loaded in (?<cpty_credit_pe_rt>\S+)"

Maybe there's a way to do this in one rex invocation, but I tried several things which didn't work.

The other option is to add a few stanzas to props.conf and transforms.conf. For example,

in props.conf:

[multiline] REPORT-foo = mlFields

in transforms.conf:

[mlFields] REGEX = CLOSE, loaded in (\S+).* FX_CLOSE, loaded in (\S+).* XLA_ENV, loaded in (\S+).* INTRADAY, loaded in (\S+).* CPTY_CREDIT, loaded in (\S+) FORMAT = close_pe_rt::$1 fx_close_pe_rt::$2 xla_env_pe_rt::$3 intraday_pe_rt::$4 cpty_credit_pe_rt::$5

You could also try using the Interactive Field Extractor (IFX).

View solution in original post

hulahoop
Splunk Employee
Splunk Employee

These are the 2 options I would try

  • configuration files
  • rex command in the search bar

The easiest, but also most transient, option is to use rex command inline in your search. For example:

sourcetype="multiline" | rex "CLOSE, loaded in (?<close_pe_rt>\S+)" | rex "FX_CLOSE, loaded in (?<fx_close_pe_rt>\S+)" | rex "XLA_ENV, loaded in (?<xla_env_pe_rt>\S+)" | rex "INTRADAY, loaded in (?<intraday_pe_rt>\S+)" | rex "CPTY_CREDIT, loaded in (?<cpty_credit_pe_rt>\S+)"

Maybe there's a way to do this in one rex invocation, but I tried several things which didn't work.

The other option is to add a few stanzas to props.conf and transforms.conf. For example,

in props.conf:

[multiline] REPORT-foo = mlFields

in transforms.conf:

[mlFields] REGEX = CLOSE, loaded in (\S+).* FX_CLOSE, loaded in (\S+).* XLA_ENV, loaded in (\S+).* INTRADAY, loaded in (\S+).* CPTY_CREDIT, loaded in (\S+) FORMAT = close_pe_rt::$1 fx_close_pe_rt::$2 xla_env_pe_rt::$3 intraday_pe_rt::$4 cpty_credit_pe_rt::$5

You could also try using the Interactive Field Extractor (IFX).

jdagenais
Explorer

Thank you very much.

This solution worked quite well and I implemented it with the props.conf and transforms.conf.

0 Karma

hulahoop
Splunk Employee
Splunk Employee

Note: I did not have to use the (?m) regex modifier in the REGEX field for transforms.conf. Somewhere along the way, Splunk automatically knows how to deal with multiline events.

0 Karma

ziegfried
Influencer

You can either use multiple regular expressions (multiple rex command calls or multiple EXTRACT-* options in props.conf) or a single regex with the (?m) multiline flag (maybe as well the (?s) DOTALL flag).

Someting like this might work:

... | rex "(?ms)No exceptions for CLOSE, loaded in (?<close_pe_rt>[\d\.]+).+FX_CLOSE, loaded in (?<fx_close_pe_rt>[\d\.]+).+XLA_ENV, loaded in (?<xla_env_pe_rt>[\d\.]+).+INTRADAY, loaded in (?<intraday_pe_rt>[\d\.]+).+CPTY_CREDIT, loaded in (?<cpty_credit_pe_rt>[\d\.]+)"
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...