Cannot get custom sourcetype to do line breaks correctly

[bobryant]

We have Splunk Enterprise with SH, Clustered IX (2), HF and many UFs. I have created an app in the deployment apps folder (with inputs.conf and props.conf) on deployment manager and deployed to server running UF. Ingestion begins as expected but does not line break as desired.

Log looks like this:
...........................
ipro Trace started on Thursday, July 12, 2018 at 8:16:12 PM Central Daylight Time(en-US)
Machine: P-XXXXXXX, Culture: en-US, UI Culture: en-US
Ini Settings:

20:16:12.672 Tid=4,Log file created.

20:16:12.679 Tid=4,Running module as Windows Service

20:16:12.680 Tid=4,Product version: 7.99.999.9331

20:16:12.813 Tid=9,Conn=1,ElapseMs=0,ipro:Received
RequestOnly, PingToClient, HeaderSize=4, DataSize=0

20:16:12.813 Tid=4,Conn=1,ElapseMs=1,ipro:Sent
RequestResponse, Connect, HeaderSize=43, DataSize=269
............................

Splunk appears to get that the date stamp is in the first row of the log file and that the time stamps appear at the beginning of each row. the problem is line breaking. I want it to break at each time stamp allowing for multi-line log entries to merge into one event.

I have tried a number of different options in the props.conf file and specified several different regex. Nothing I do seems to change the outcome. I wonder if I am deploying this correctly. It seems to randomly break lines, where most of the them time there are two or more log entries in each Splunk event. the number log entries in each event is not consistent so I do not know what it is breaking on.

Here is inputs.conf
...........................
[monitor://c:\ProgramData\XXX\XXXXXXX\ipro\XXXX\]
disabled = false
index = ipro
followtail = 0
sourcetype = appx:ipro
whitelist = \.txt$
ignoreOlderThan = 0d
...................................

here is props.conf
..................................
[appx:ipro]
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = [0-9]{2}:[0-9]{2}:[0-9]{2}\.
DATETIME_CONFIG =
MAX_TIMESTAMP_LOOKAHEAD = 80
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = true
category = Custom
description = ipro logs
disabled = false
pulldown_type = true
........................................

Any suggestions would be helpful.

[markusspitzli]

Hi @bobryant

You said that you put the inputs.conf and props.conf on the UF. is that correct?

If yes it's explainable why the linebreaking doesnt work. your props.conf should be located on the first Splunk Enterprise Instance. This instance is responsible for the parsing and will do the linebreaking and timestamp extraction. This can be either a HF or a Indexer.

I just noticed that you use the BREAK_ONLY_BEFORE parameters. I would exchange it with LINE_BREAKER and SHOULD_LINEMERGE=false . This way you get more performance out of Splunk and is also best practice doing that.

The config could look like this:

[appx:ipro]
LINE_BREAKER = ([\r\n]+)\d{2}:\d{2}:\d{2}\.\d{3}
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false