Getting Data In

How to edit my props.conf to correctly line break my sample log?

dperry
Communicator

I want Splunk to break every time I see Event logged at *}:

Event logged at {1492205898958;2}:
  ID: com.innovision.ofx.ofxsgml.ncph.Response
  Title: Event logged at Fri Apr 14 14:38:18 PDT 2017
  Severity: INFO
  Timestamp: {1492205898958;2}
  Date-Time: Fri Apr 14 14:38:18 PDT 2017
  Source: com.innovision.ofx.ofxsgml.ncph.OfxResponse
  Transaction-ID: 0A044F3301D3BFED0000015B6E2BF39BD246
  Host-Name: XXXXXXX

  HTTP result code (so far) is 200.
  OFX output response message:
    Header version:     100
    Content type:       OFXSGML
    DTD version:        102
    Security type:      NONE
    Character encoding: USASCII
    Character set:      1252
    Compression type:   NONE
    Old file UID:       NONE
    New file UID:       NONE
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Event logged at {1492205898958;4}:
  ID: 0A044F3301D3BFED0000015B6E2BF39BD28D
  Title: System.out
  Severity: INFO
  Timestamp: {1492205898958;4}
  Date-Time: Fri Apr 14 14:38:18 PDT 2017
  Host-Name: XXXXX

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Event logged at {1492205898974;2}:
  ID: 0A044F3301D3BFED0000015B6E2BF39BD28E
  Title: System.err
  Severity: WARNING
  Timestamp: {1492205898974;2}
  Date-Time: Fri Apr 14 14:38:18 PDT 2017
  Host-Name: XXXXXXXX

  AccountsServant - returned error 2000
Event logged at {1492205898974;3}:
  ID: com.innovision.nepal.controller.XactOut
  Title: Event logged at Fri Apr 14 14:38:18 PDT 2017
  Severity: INFO
  Timestamp: {1492205898974;3}
  Date-Time: Fri Apr 14 14:38:18 PDT 2017
  Source: com.innovision.nepal.controller.TransactionServant
  Transaction-ID: 0A044F3301D3BFED0000015B6E2BF39BD246
  Host-Name: XXXXXXXX

  Transaction 0A044F3301D3BFED0000015B6E2BF39BD246 is complete.
  Total processing time: 00:00:01.407 (1.407s).
  Heap memory used: 135977728 of 518979584 bytes (26%).

here is my props.conf:

LINE_BREAKER = ([\r\n])Event logged at *}:
MAX_EVENTS = 2000
NO_BINARY_CHECK = true
disabled = false
pulldown_type = true
0 Karma
1 Solution

lguinn2
Legend

UPDATE: Ooops - typos, so editing - and I liked @cpetterborg's suggestion re: improving the regex. Also, as mentioned, this props.conf must go wherever the data is being parsed - regardless of whether you use the LINE_BREAKER or BREAK_ONLY_BEFORE. Normally, parsing happens on the indexer. Parsing does not happen on a Universal Forwarder, but the data is parsed on a Heavy Forwarder.

I am not great with the line breaker, so I would do it this way

BREAK_ONLY_BEFORE=^Event logged at .*\}
MAX_EVENTS = 2000

If you want to do it your way, you should set SHOULD_LINEMERGE=false. And make sure that you escape the } in your regular expression (as I did) because it is a special character.

View solution in original post

0 Karma

gvmorley
Contributor

Hi,

You should just be able to do a positive lookahead and an empty capture group. So:

[ofx]
SHOULD_LINEMERGE = False
LINE_BREAKER = (?m)(?=^Event logged at {[\d;]+})()

I did a quick test and it looked fairly successful:

alt text

Hopefully this gets you closer to what you're looking for.

lguinn2
Legend

UPDATE: Ooops - typos, so editing - and I liked @cpetterborg's suggestion re: improving the regex. Also, as mentioned, this props.conf must go wherever the data is being parsed - regardless of whether you use the LINE_BREAKER or BREAK_ONLY_BEFORE. Normally, parsing happens on the indexer. Parsing does not happen on a Universal Forwarder, but the data is parsed on a Heavy Forwarder.

I am not great with the line breaker, so I would do it this way

BREAK_ONLY_BEFORE=^Event logged at .*\}
MAX_EVENTS = 2000

If you want to do it your way, you should set SHOULD_LINEMERGE=false. And make sure that you escape the } in your regular expression (as I did) because it is a special character.

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

Though this could work in this case, making the BREAK_ONLY_BEFORE be more exact, add a caret, in case the other instance of "Event logged at" within the Title has a curly brace in it. Also, since the BREAK_ONLY_BEFORE uses regular expressions, you don't need to escape the asterisk, but you should have a period (.) before it:

BREAK_ONLY_BEFORE=^Event logged at .*\}

dperry
Communicator

I have the prop.conf on the UF should I place it on the indexer?

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

Props.conf is usually on the indexers, but under certain circumstances can be on the forwarder. In this case, the indexer is the place for this.

0 Karma

dperry
Communicator

Thanks.....I will let you know the outcome!

0 Karma

dperry
Communicator

Once I placed the props.conf in the indexers....BOOM...works, thanks!

0 Karma

dperry
Communicator

reason I ask this is this work with my local Splunk instance (laptop).

My enterprise env is:

1 SH
1 DS
4 indexers

0 Karma

dperry
Communicator

No dice I changed the props.conf to show:

[ofx]
BREAK_ONLY_BEFORE = Event logged at *}:
MAX_EVENTS = 2000
SHOULD_LINEMERGE= false
NO_BINARY_CHECK = true
disabled = false
pulldown_type = true

Still not breaking at this line

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...