Getting Data In

Split multi-line events

lcasey001
Explorer

I have a file that has multiple multi line events. Each event is broken up into "INFO: ---" or "ERROR: ---"

ERROR: ---
blahNewsLetter: N
birthdate: 1947-10-25
countryId: 1
createdOn: 2011-05-31 13:40:46
...
INFO: ---
blahNewsLetter: ~
birthdate: 0000-00-00
countryId: ~
createdOn: 2011-05-31 13:40:48
...

My props.conf

[dblog]
SHOULD_LINEMERGE = true
# force splunk to detec multiline events
BREAK_ONLY_BEFORE = (.*)(INFO|ERROR):

My inputs.conf

[monitor:///var/log/dblog.log]
disabled = false
index = blah
sourcetype = dblog
blacklist = (\.(gz|bz2|z|zip)$)
followTail = 1

This does not work. It is splitting the file up into events that have lines similar to the "createdOn: 2011-05-31 13:40:48" lines.

modifiedOn: 2011-05-31 13:40:48
postalCode: 1111
promoCode: ~
requestDetail: |-
  <request>
  <billToEmail>null</billToEmail>
  <billToFirstName>Name</billToFirstName>
  <billToLastName>Name</billToLastName>
  <billToStreet1>null</billToStreet1>
  <billToStreet2>null</billToStreet2>
...

and

createdOn: 2011-05-31 13:40:48
email: myemail@email.com

Any help would be appreciated. And I have also tried using BREAK_ONLY_BEFORE_DATE = false with same result. This is currently on the forwarder.

1 Solution

hazekamp
Builder

lcasey001,

I would recommend using LINE_BREAKER.

[dblog]
SHOULD_LINEMERGE = false
# force splunk to detec multiline events
LINE_BREAKER = ([\r\n]+)(?:INFO|ERROR):

Update: These configurations need to be on your indexer. I would also recommend modifying Splunk's date/time properties since there is not a timestamp in first 150 characters of your event.

[dblog]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?:INFO|ERROR):
TIME_PREFIX = createdOn:\s+
TIME_FORMAT = %Y-%m-%d %H:%M:%S

View solution in original post

hazekamp
Builder

lcasey001,

I would recommend using LINE_BREAKER.

[dblog]
SHOULD_LINEMERGE = false
# force splunk to detec multiline events
LINE_BREAKER = ([\r\n]+)(?:INFO|ERROR):

Update: These configurations need to be on your indexer. I would also recommend modifying Splunk's date/time properties since there is not a timestamp in first 150 characters of your event.

[dblog]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?:INFO|ERROR):
TIME_PREFIX = createdOn:\s+
TIME_FORMAT = %Y-%m-%d %H:%M:%S

hazekamp
Builder

Not sure if there is a specific list, but any props related to line breaking, time parsing, or of type TRANSFORMS- are performed @ the indexing/full forwarder layer of your Splunk infrastructure.

0 Karma

lcasey001
Explorer

Where can one obtain that list for future reference? I did not see anything in the props.conf.spec file.

0 Karma

lcasey001
Explorer

Thanks !!! That was it. It needed to be done on the indexer and not forwarder. So is there a list of things that needs to be done on the indexer vs forwarder regarding props.conf ?

0 Karma

hazekamp
Builder

Line breaking and time parsing are done @ the indexer or Full forwarder, not Universal Forwarder/Light Forwarder. Please move these configurations to your indexer(s).

0 Karma

lcasey001
Explorer

These changes have been on the Universal Forwarder where the log file is hosted. I have other settings on the forwarder for other log files, mostly setting sourcetypes in the props.conf file. Should this all be done on the indexer?

0 Karma

lcasey001
Explorer

Made your recommended change, restarted the forwarder, and got the same result.

5/31/11
3:31:17.000 PM  

    modifiedOn: 2011-05-31 15:31:17
    postalCode: 11111
    promoCode: ''
    requestDetail: |-
      <request>
      <email>email@email.net</email>
      <userName>username</userName>
      <firstName>firstname</firstName>
      <lastName>lastname</lastName>
      <language>en</language>
Show all 18 lines
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...