Why am I unable to extract fields from multiline events with my current props.conf configuration?

[kiranamex]

Hi All,

I am trying to extract fields from multiline events which were injected from our server to Splunk.

We have our events as below where each event starts with time stamp and all the below events are logged as a single event. When we tried to extract fields, we were thrown with below error.

The extraction failed. If you are extracting multiple fields, try removing one or more fields. Start with extractions that are embedded within longer text strings. 

But I need them to get logged as multiple events where the actual event message should start with time stamp first.

17:31:32.753 DEBUG [DatabasePoll] Database Poller-0
nextStateEvent = 1:EVENT_BEGIN_POLL 
17:31:32.753 DEBUG [TraceLog] Database Poller-0
Polling database ... Database Poller-0
17:31:32.754 DEBUG [DatabasePoll] DatabasePoll
nextStateEvent = 2:EVENT_BEGIN_LOGIN_TIMER 
17:31:32.754 DEBUG [TraceLog] Database Poller-0
Performing database poll ... timeout seconds=5
17:31:32.755 DEBUG [DatabasePoll] DatabasePoll
nextStateEvent = 3:EVENT_BEGIN_QUERY_TIMER 
17:31:32.755 DEBUG [TraceLog] Database Poller-0
Database poll ... completed
17:31:32.755 DEBUG [DatabasePoll] DatabasePoll
nextStateEvent = 0:EVENT_READ_NEXT_STATE_EVENT 
17:31:32.755 DEBUG [LocalController] Database Poller-0
Process active [Failover_Controller] [started] monitor running is true

I expect the data should be logged as below:

Event 1:

17:31:32.753 DEBUG [DatabasePoll] Database Poller-0
nextStateEvent = 1:EVENT_BEGIN_POLL 

Event 2:

17:31:32.753 DEBUG [TraceLog] Database Poller-0
Polling database ... Database Poller-0

Event 3:

17:31:32.754 DEBUG [DatabasePoll] DatabasePoll
nextStateEvent = 2:EVENT_BEGIN_LOGIN_TIMER 

Editing the props.conf with below fields did not help.

[Source_type]
BREAK_ONLY_BEFORE_DATE = true
TIME_FORMAT = %b %d %T
SHOULD_LINEMERGE = false
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 20

[jkat54]

Try this please:

 [Source_type]
 BREAK_ONLY_BEFORE = \d{2}:\d{2}:\d{2}\.\d{3}
 SHOULD_LINEMERGE = false
 MAX_TIMESTAMP_LOOKAHEAD = 12

Only I'm not seeing where the date of month, month of year, and year are in your data. I'm not sure if this will work as expected or not.

[kiranamex]

Thanks for your answer. However, I have lines containing exceptions as well which should be included in an event within a time stamp but are getting ignored.
Each log statement which starts with time stamp should go as a single event and the next event which starts with another time stamp again should go as a next event.

18:52:06.047 INFO  [JMSConnectionManager] Default Callback Handler[0]
    Connecting to JMS Provider [xxx:ActiveMQ]
18:52:06.047 DEBUG [JMSConnectionManager] Default Callback Handler[0]
    Changing connection state from NOT CONNECTED to CONNECTING
18:52:06.047 DEBUG [TcpTransport] Default Callback Handler[0]
    Stopping transport tcp://null:0
18:52:06.048 INFO  [JMSConnectionManager] Default Callback Handler[0]
    Failed to connect to JMS Provider [xxx:ActiveMQ]

javax.jms.JMSException: Could not connect to broker URL: tcp://xxx.com:63623. Reason:          java.net.ConnectException: Connection refused
        at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:35)
        at     org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:280)
        at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:224)
        at org.apache.activemq.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:172)
        at com.authentic.sysmon.jms.JMSConnectionManager.startConnection(JMSConnectionManager.java:682)
        at com.authentic.sysmon.jms.JMSConnectionManager$3.run(JMSConnectionManager.java:738)
        at com.authentic.callback.CallbackHandler$Processor.run(CallbackHandler.java:219)
        Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:435)
        at org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:401)
        at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:53)
        at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
        at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
        at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
        at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
        at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
        at    org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:260)
        ... 5 more