Splunk Search

Why is my configuration causing a search for field=value to return only single line events, not multiline?

avilandau
Path Finder

I have a log, broken to fields, where the free text field is the last field and can be multiline. After defining the fields in transforms.conf (i.e. rather than in props.conf) and making sure that every field, including the multiline text, is wrapped with the delimiter, it now works perfectly i.e. If I search by sourcetype I see all logs entries, each one with its fields. However, if I add to the search field=value (where field is one of the fields in the event) I get back only the single line events and none of the multiline ones.

Any idea what causes that?

Props.conf

[gpp_error_log]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+####)
LINE_BREAKER_LOOKBEHIND=32000
REPORT-fields = gpp_error_log_fields

transforms.conf

[gpp_error_log_fields]
DELIMS = "~"
FIELDS = timestamp,profile,severity,code,message

The log

####2015-01-27 11:16:23.370~POC4~9~29245~Exception Report: null
java.lang.NullPointerException
        at com.fundtech.core.paymentprocess.data.tx.IncomingFormattingContext.getParseObject(IncomingFormattingContext.java:48)
        at backend.paymentprocess.interfaces.handlers.FEDInterfaceHandler.processABMS(FEDInterfaceHandler.java:411)
        at backend.paymentprocess.interfaces.handlers.FEDInterfaceHandler.parseMsg(FEDInterfaceHandler.java:88)
        at backend.paymentprocess.businessflowselector.businessobjects.BOBusinessFlowSelector.executeBusinessFlowInner(BOBusinessFlowSelector.java:130)
        at com.ibm.ws.asynchbeans.WorkWithExecutionContextImpl.go(WorkWithExecutionContextImpl.java:199)
        at com.ibm.ws.asynchbeans.CJWorkItemImpl.run(CJWorkItemImpl.java:188)
        at com.ibm.ws.util.ThreadPoolWorker.run(ThreadPool.java:1613)
        .... more stack lines will be here......
,  File:sun.reflect.NativeMethodAccessorImpl,  Line:48,  User ID:,  IP Address:192.168.168.121~
####2015-01-27 11:16:46.516~POC4~4~22557~Exception Report: null
java.lang.NullPointerException
        at com.fundtech.core.paymentprocess.data.tx.IncomingFormattingContext.getParseObject(IncomingFormattingContext.java:48)
        at backend.paymentprocess.interfaces.handlers.FEDInterfaceHandler.processABMS(FEDInterfaceHandler.java:411)
        at backend.paymentprocess.interfaces.handlers.FEDInterfaceHandler.parseMsg(FEDInterfaceHandler.java:88)
        at backend.paymentprocess.businessflowselector.businessobjects.BOBusinessFlowSelector.executeBusinessFlowInner(BOBusinessFlowSelector.java:130)
        at com.ibm.ws.asynchbeans.WorkWithExecutionContextImpl.go(WorkWithExecutionContextImpl.java:199)
        at com.ibm.ws.asynchbeans.CJWorkItemImpl.run(CJWorkItemImpl.java:188)
        at com.ibm.ws.util.ThreadPoolWorker.run(ThreadPool.java:1613)
        .... more stack lines will be here......
,  File:sun.reflect.NativeMethodAccessorImpl,  Line:48,  User ID:,  IP Address:192.168.168.121~
####2015-01-27 11:17:11.723~POC4~4~33772~Simulated GPP error from POC4 of severity 4 from code 33772~
####2015-01-27 11:17:46.877~POC4~7~26846~Exception Report: null
java.lang.NullPointerException
        at com.fundtech.core.paymentprocess.data.tx.IncomingFormattingContext.getParseObject(IncomingFormattingContext.java:48)
        at backend.paymentprocess.interfaces.handlers.FEDInterfaceHandler.processABMS(FEDInterfaceHandler.java:411)
        at backend.paymentprocess.interfaces.handlers.FEDInterfaceHandler.parseMsg(FEDInterfaceHandler.java:88)
        at backend.paymentprocess.businessflowselector.businessobjects.BOBusinessFlowSelector.executeBusinessFlowInner(BOBusinessFlowSelector.java:130)
        at com.ibm.ws.asynchbeans.WorkWithExecutionContextImpl.go(WorkWithExecutionContextImpl.java:199)
        at com.ibm.ws.asynchbeans.CJWorkItemImpl.run(CJWorkItemImpl.java:188)
        at com.ibm.ws.util.ThreadPoolWorker.run(ThreadPool.java:1613)
        .... more stack lines will be here......
,  File:sun.reflect.NativeMethodAccessorImpl,  Line:48,  User ID:,  IP Address:192.168.168.121~
####2015-01-27 11:17:57.979~POC4~5~36591~Simulated GPP error from POC4 of severity 5 from code 36591~
####2015-01-27 11:18:20.050~POC4~6~29483~Simulated GPP error from POC4 of severity 6 from code 29483~
0 Karma

bmacias84
Champion

@avilandau,

I would recommend using the data previewer in Splunk to make sure you data is being broken correctly. You really should be using SHOULD_LINEMERGE=true=, which combines several lines of data into a single multiline event. The MAX_TIMESTAMP_LOOKAHEAD will also speed up indexing by telling Splunk to only lookup to 32 char instead of scanning the event. The other stuff can be ignored. Make sure these settings are applied to your indexer. The settings below have been tested with your sample data.


[gpp_error_log]
 SHOULD_LINEMERGE=true
 MAX_TIMESTAMP_LOOKAHEAD=32
 REPORT-fields = gpp_error_log_fields

Cheers,

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...