Getting Data In

Extracting many "long key name" = "value" pairs from DB2's log files

Ricapar
Communicator

Working on extracting some Key/Value pairs out of DB2's log files. I have a file like this:

  [...snip...]
  Buffer pool xda writes                   = 0
  Asynchronous pool xda page writes        = 0
  Total buffer pool read time (millisec)   = 66
  Total buffer pool write time (millisec)  = 0
  Total elapsed asynchronous read time     = 46
  Total elapsed asynchronous write time    = 0
  Asynchronous data read requests          = 3
  Asynchronous index read requests         = 0
  [...snip...]

While I can go and use EXTRACT commands/regexes for only the specific ones I want, it would be extremely tedious to do so. Especially since this spans across various sourcetypes.

Ideally, I'd like to be able to make use of Splunk's "CLEAN_KEYS" setting and have things come out extracted so I can do a search like this without having to configure anything else:

search {stuff} | timechart avg(Total_buffer_pool_read_time_millisec)

I have this so far.. (copied some values from other pre-packaged transforms.conf files)

props.conf:

[db2dynsql]
BREAK_ONLY_BEFORE=Number of executions
SHOULD_LINEMERGE=true
KV_MODE=none
REPORT-kv = db2_kv

transforms.conf:

[db2_kv]
CAN_OPTIMIZE = True
CLEAN_KEYS = True
DEFAULT_VALUE =
DEST_KEY =
FORMAT = $1::$2
KEEP_EMPTY_VALS = False
LOOKAHEAD = 4096
MV_ADD = False
REGEX = ([^=]+)\s+=\s+(.*?)
SOURCE_KEY = _raw
WRITE_META = False

Right now.. all that is getting me is this error message when I'm running a search:

[splunkhost] Field extractor name=db2_kv is unusually slow (max single event time=1522ms, probes=14 warning max=1000ms)

lguinn2
Legend

You have specified a number of things that are only valid for index-time field extraction. Search time field extraction is faster. Yes, I know it is counter-intuitive, but it is true - especially for DB2 logs.
Try this for transforms.conf - it may not solve the problem, but we can start there.

[db2_kv]
CLEAN_KEYS = true
FORMAT = $1::$2
KEEP_EMPTY_VALS = false
MV_ADD = false
REGEX = (.+?)\s*=\s*(.*)

Also, what is the maximum number of characters in an event? The maximum number of lines in an event? And where is the timestamp?

0 Karma

kristian_kolb
Ultra Champion

Do you need to take the newlines into account, or specify it as a multi-line regex?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...