Splunk Search

How to extract fields from a multi line messages

jdagenais
Explorer

We have a multi line message that looks like this:

11/30/10 16:28:34 Verifying pricing env CLOSE,FX_CLOSE,XLA_ENV,INTRADAY,CPTY_CREDIT
   No exceptions for CLOSE, loaded in 0.05 secs
   Messages for FX_CLOSE
      PricerConfigRefresh: No item found for 1246892/CurveZero (CLOSE)  before Tue Nov 30 16:28:34 EST 2010
   No exceptions for FX_CLOSE, loaded in 0.17 secs
   No exceptions for XLA_ENV, loaded in 0.05 secs
   No exceptions for INTRADAY, loaded in 0.10 secs
   No exceptions for CPTY_CREDIT, loaded in 0.40 sec

I would like to create the following fields and assign the numeric values as response time, so I can search and chart on the RT values.

  close_pe_rt=0.05
  fx_close_pe_rt=0.17
  xla_env_pe_rt=0.05
  intraday_pe_rt=0.10
  cpty_credit_pe_rt=0.40

What is the best way to handle such extraction and assignments?

Thanks very much for your help, Jean

Tags (1)
1 Solution

hulahoop
Splunk Employee
Splunk Employee

These are the 2 options I would try

  • configuration files
  • rex command in the search bar

The easiest, but also most transient, option is to use rex command inline in your search. For example:

sourcetype="multiline" | rex "CLOSE, loaded in (?<close_pe_rt>\S+)" | rex "FX_CLOSE, loaded in (?<fx_close_pe_rt>\S+)" | rex "XLA_ENV, loaded in (?<xla_env_pe_rt>\S+)" | rex "INTRADAY, loaded in (?<intraday_pe_rt>\S+)" | rex "CPTY_CREDIT, loaded in (?<cpty_credit_pe_rt>\S+)"

Maybe there's a way to do this in one rex invocation, but I tried several things which didn't work.

The other option is to add a few stanzas to props.conf and transforms.conf. For example,

in props.conf:

[multiline] REPORT-foo = mlFields

in transforms.conf:

[mlFields] REGEX = CLOSE, loaded in (\S+).* FX_CLOSE, loaded in (\S+).* XLA_ENV, loaded in (\S+).* INTRADAY, loaded in (\S+).* CPTY_CREDIT, loaded in (\S+) FORMAT = close_pe_rt::$1 fx_close_pe_rt::$2 xla_env_pe_rt::$3 intraday_pe_rt::$4 cpty_credit_pe_rt::$5

You could also try using the Interactive Field Extractor (IFX).

View solution in original post

hulahoop
Splunk Employee
Splunk Employee

These are the 2 options I would try

  • configuration files
  • rex command in the search bar

The easiest, but also most transient, option is to use rex command inline in your search. For example:

sourcetype="multiline" | rex "CLOSE, loaded in (?<close_pe_rt>\S+)" | rex "FX_CLOSE, loaded in (?<fx_close_pe_rt>\S+)" | rex "XLA_ENV, loaded in (?<xla_env_pe_rt>\S+)" | rex "INTRADAY, loaded in (?<intraday_pe_rt>\S+)" | rex "CPTY_CREDIT, loaded in (?<cpty_credit_pe_rt>\S+)"

Maybe there's a way to do this in one rex invocation, but I tried several things which didn't work.

The other option is to add a few stanzas to props.conf and transforms.conf. For example,

in props.conf:

[multiline] REPORT-foo = mlFields

in transforms.conf:

[mlFields] REGEX = CLOSE, loaded in (\S+).* FX_CLOSE, loaded in (\S+).* XLA_ENV, loaded in (\S+).* INTRADAY, loaded in (\S+).* CPTY_CREDIT, loaded in (\S+) FORMAT = close_pe_rt::$1 fx_close_pe_rt::$2 xla_env_pe_rt::$3 intraday_pe_rt::$4 cpty_credit_pe_rt::$5

You could also try using the Interactive Field Extractor (IFX).

jdagenais
Explorer

Thank you very much.

This solution worked quite well and I implemented it with the props.conf and transforms.conf.

0 Karma

hulahoop
Splunk Employee
Splunk Employee

Note: I did not have to use the (?m) regex modifier in the REGEX field for transforms.conf. Somewhere along the way, Splunk automatically knows how to deal with multiline events.

0 Karma

ziegfried
Influencer

You can either use multiple regular expressions (multiple rex command calls or multiple EXTRACT-* options in props.conf) or a single regex with the (?m) multiline flag (maybe as well the (?s) DOTALL flag).

Someting like this might work:

... | rex "(?ms)No exceptions for CLOSE, loaded in (?<close_pe_rt>[\d\.]+).+FX_CLOSE, loaded in (?<fx_close_pe_rt>[\d\.]+).+XLA_ENV, loaded in (?<xla_env_pe_rt>[\d\.]+).+INTRADAY, loaded in (?<intraday_pe_rt>[\d\.]+).+CPTY_CREDIT, loaded in (?<cpty_credit_pe_rt>[\d\.]+)"
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...