Why am I having a line breaking problem with syslog like data over TCP?

[las]

Hi.

I'm having some issues with a datasource comming from TCP:514 (Syslog like).

239 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern Aqp4 rO0ABXQAKHV1aWQ0ZDM0OTU5MS0wMTYyLTFmYWQtYTQ1YS1kMjBkZWMxMDI5NDc= itfim/distributedmaps/ssops_plugins -1 UPDATE DMAP_ENTRIES SET DMAP_VALUE = ? WHERE (DMAP_KEY = ? AND DMAP_PARTITION = ? )264 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:742 CET] 000850a8 id=         com.tivoli.am.fim.distributed.jdbc.JDBCDBTransaction         > prepareStatement ENTRY UPDATE DMAP_ENTRIES SET DMAP_VALUE = ? WHERE (DMAP_KEY = ? AND DMAP_PARTITION = ? )181 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:742 CET] 000850a8 id=         com.tivoli.am.fim.distributed.jdbc.JDBCDBTransaction         < prepareStatement RETURN187 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:743 CET] 000850a8 id=         com.tivoli.am.fim.distributed.jdbc.JDBCDBHelper              < executeUpdateStatement RETURN179 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:743 CET] 000850a8 id=         com.tivoli.am.fim.distributed.jdbc.JDBCDBHelper              < insertOrUpdate RETURN169 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:743 CET] 000850a8 id=         com.tivoli.am.fim.distributed.jdbc.JDBCDBHelper              > close ENTRY170 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:743 CET] 000850a8 id=         com.tivoli.am.fim.distributed.jdbc.JDBCDBTransaction         > commit ENTRY171 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:743 CET] 000850a8 id=         com.tivoli.am.fim.distributed.jdbc.JDBCDBTransaction         < commit RETURN169 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:743 CET] 000850a8 id=         com.tivoli.am.fim.distributed.jdbc.JDBCDBTransaction         > close ENTRY170 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:743 CET] 000850a8 id=         com.tivoli.am.fim.distributed.jdbc.JDBCDBTransaction         < close RETURN170 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:743 CET] 000850a8 id=         com.tivoli.am.fim.distributed.jdbc.JDBCDBHelper              < close RETURN3343 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:743 CET] 000850a8 id=         com.tivoli.am.fim.distributed.jdbc.JDBCDistributedMap        < put RETURN SAML20Session [sessionId = uuid4d349591-0162-1fad-a45a-d20dec102947 signOutInfo = null Session Attributes = [ ; key = SAMLSESS-UserInteractionActionList value = [com.tivoli.am.fim.saml20.protocol.actions.sso.SAML20LocalLoginAction]; key = SAMLSESS-ContextState value = {RequestContext.RelayState=uuid4d349591-0162-1fad-a45a-d20dec102947, RequestContext.ProtocolBinding=Protocol Binding:HTTPPost urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST, RequestContext.Target=null, RequestContext.InitialSAMLRequestProtocolBinding=null, ResponseContext.RelayState=uuid4d349591-0162-1fad-a45a-d20dec102947, ResponseContext.ProtocolEndpoint=null, ProtocolRole=REQUESTER, RequestContext.Artifact=null, ResponseContext.SessionOP=STORE, ContextAttributes={SAMLCTX-ProtocolPartner=http://pilotam.prod.atp.local/sso/sps/AtpLoginSpecialistFederationIdp/saml20, SAMLCTX-SamlTokenElement=[saml:Assertion: null], SAMLCTX-IvcredTokenElement=[wss:BinarySecurityToken: null], SAMLCTX-AssertionNameID=null, SAMLCTX-NeedToFederate=false, SAMLCTX-SamlToken=com.tivoli.am.fim.saml.assertion.Saml20AssertionImpl@6df87a89(Version: 2.0, ID: Assertion-uuid4d3495d5-0162-18c0-98ea-d20dec102947, IssueInstant: 2018-03-22T10:15:14Z, Issuer: com.tivoli.am.fim.saml.assertion.Saml20IssuerImpl@8fcdaa76[NameQualifier: null, SPNameQualifier: null, format: urn:oasis:names:tc:SAML:2.0:nameid-format:entity, SPProvidedID: null, value: http://pilotam.prod.atp.local/sso/sps/AtpLoginSpecialistFederationIdp/saml20], Subject: com.tivoli.am.fim.saml.assertion.Saml20SubjectImpl@27fd901a)}, RequestContext.InitialSAMLRequestReturnProtocolBinding=null, ResponseContext.SAMLMessageProtocolBinding=Protocol Binding:HTTPPost urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST, ResponseContext.SignMessage=false, RequestContext.RequestURL=https://privat-pilo.atp.dk/sso/sps/AtpSpecialistPrivatFederation01/saml20/login, RequestContext.SAMLMessage=com.tivoli.am.fim.saml.protocol.Saml20ResponseImpl@fd99163 [IssueInstant: Thu Mar 22 11:15:14 CET 2018, Version: 2.0, ID: FIMRSP_4d3495e1-0162-1808-bfbc-d20dec102947, Consent: null, Destination: https://privat-pilo.atp.dk/sso/sps/AtpSpecialistPrivatFederation01/saml20/login, RelayState: null]} ] Current Requests = [ ; key = FIMREQ_4d349593-0162-1d87-bb79-d20dec102947 value = com.tivoli.am.fim.saml.protocol.Saml20AuthnRequestImpl@69976609 [IssueInstant: Thu Mar 22 11:15:14 CET 2018, Version: 2.0, ID: FIMREQ_4d349593-0162-1d87-bb79-d20dec102947, Consent: null, Destination: http://pilotam.prod.atp.local/sso/sps/AtpLoginSpecialistFederationIdp/saml20/login, RelayState: null] (forceAuthn: false, isPassive: false, AssertionConsumerServiceIndex: <not set>, AssertionConsumerServiceURL: https://privat-pilo.atp.dk/sso/sps/AtpSpecialistPrivatFederation01/saml20/login, protocolBinding: urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST, AttributeConsumingServiceIndex: <not set>) ] SessionInfos = [ ; key = http://pilotam.prod.atp.local/sso/sps/AtpLoginSpecialistFederationIdp/saml20 value = SAML20PartnerProviderSessionInfo [PartnerProviderId = http://pilotam.prod.atp.local/sso/sps/AtpLoginSpecialistFederationIdp/saml20; SessionIndex = uuid4d3495c2-0162-126c-93ba-d20dec102947 ] ] ]177 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:743 CET] 000850a8 id=         com.tivoli.am.fim.saml20.session.SAML20SessionManager        > setSAMLCookie ENTRY222 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:743 CET] 000850a8 id=         com.tivoli.am.fim.saml20.session.SAML20SessionManager        3 setSAMLCookie Setting Cookie: javax.servlet.http.Cookie@df2ce41b177 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:743 CET] 000850a8 id=         com.tivoli.am.fim.om.ObjectManager                           > get(Class<C>) ENTRY183 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:743 CET] 000850a8 id=         com.tivoli.am.fim.om.ObjectManager$Configuration             > getObject(Class<C>) ENTRY278 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:744 CET] 000850a8 id=         com.tivoli.am.fim.om.ObjectManager$Configuration             3 getObject(Class<C>) Class com.tivoli.am.fim.fedmgr2.msg.BrowserResponseImpl$CustomProperties from version 1521468890800.184 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:744 CET] 000850a8 id=         com.tivoli.am.fim.om.ObjectManager$Configuration             < getObject(Class<C>) RETURN178 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:744 CET] 000850a8 id=         com.tivoli.am.fim.om.ObjectManager                           < get(Class<C>) RETURN178 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:744 CET] 000850a8 id=         com.tivoli.am.fim.saml20.session.SAML20SessionManager        < setSAMLCookie RETURN177 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:744 CET] 000850a8 id=         com.tivoli.am.fim.saml20.session.SAML20SessionManager        < storeSession RETURN176 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:744 CET] 000850a8 id=         com.tivoli.am.fim.management.Utils                           > decodeObject ENTRY194 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:744 CET] 000850a8 id=         com.tivoli.am.fim.management.Utils                           < decodeObject RETURN java.lang.String176 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:744 CET] 000850a8 id=         com.tivoli.am.fim.management.Utils                           > decodeObject ENTRY194 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:744 CET] 000850a8 id=         com.tivoli.am.fim.management.Utils                           < decodeObject RETURN java.lang.String176 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:744 CET] 000850a8 id=         com.tivoli.am.fim.management.Utils                           > decodeObject ENTRY209 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:744 CET] 000850a8 id=         com.tivoli.am.fim.management.Utils                           < decodeObject RETURN Cached object: java.lang.String176 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:744 CET] 000850a8 id=         com.tivoli.am.fim.management.Utils                           > decodeObject ENTRY242 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:745 CET] 000850a8 id=         com.tivoli.am.fim.management.Utils                           < decodeObject RETURN com.tivoli.am.fim.saml20.protocol.config.SAML20ConfigurationImpl181 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:745 CET] 000850a8 id=         protocol.actions.sso.callback.SAML20MultiPhaseSignInCallback < postProcessEvent RETURN176 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:745 CET] 000850a8 id=         com.tivoli.am.fim.management.Utils                           > decodeObject ENTRY263 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:745 CET] 000850a8 id=         com.tivoli.am.fim.management.Utils                           < decodeObject RETURN com.tivoli.am.fim.saml20.protocol.actions.sso.callback.SAML20MultiPhaseSignInCallback179 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:745 CET] 000850a8 id=         i.am.fim.fedmgr2.protocol.WebSealSignOutInfoDelegateProtocol < processRequest RETURN280 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:745 CET] 000850a8 id=         com.tivoli.am.fim.fedmgr2.proper.FederationManager           3 doInitialRequestOnDelegate(DelegateProtocol, FederationManagerContextImpl, ProtocolActionChainImpl) Delegate returned true271 <47>Mar 22 11:15:19 ATP-00DSAM02 rt_trc_intern [3/22/18 11:15:14:745 CE

Every event seems to start with 2 or 3 numbers a space a < 2 to 3 numbers and a >

I have tried with the following props.conf:

[isamlog_trace]
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
TIME_PREFIX = 
category = Custom
pulldown_type = 1
TIME_FORMAT = %b %d %H:%M:%S
disabled = false
BREAK_ONLY_BEFORE = 
LINE_BREAKER = ()\d+\s+<\d+>
EVENT_BREAKER = 

But still I get the same result as above.

Any help will be much appreciated.

Kind regards
las

[mayurr98 · Accepted solution]

try this in props.conf if you are using heavy forwarder or else do it on the indexer.

LINE_BREAKER = (\d+\s+\<\d+\>)

After configuration. Restart the server and check for the real-time events only as this will apply to real-time events i.e. from the time you restarted the server after configuration.

let me know if this helps!

[las]

Just to make double sure, I restarted one more time, and as I'm running thru a UF, I added

EVENT_BREAKER_ENABLE = true
EVENT_BREAKER = ()\d{2,3}\s+<\d{2,3}>

to the UF config, and ended up with

[isamlog_trace]
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
TIME_PREFIX = 
category = Custom
pulldown_type = 1
TIME_FORMAT = %b %d %H:%M:%S
disabled = false
LINE_BREAKER = ()\d{2,3}\s+<\d{2,3}>
EVENT_BREAKER = 
SHOULD_LINEMERGE = false

On the indexer.

This seems to be working, but I'll monitor it over some time now.

[tiagofbmm]

EVENT_BREAKER in the Universal forwarder is useless because it has no breaking capabilities.

If you really need that, you need to put in the Indexer.

Don't forget to upvote comments and accept answers that were useful

[FrankVl]

As of version 6.5 UFs are able to recognize event boundaries using the Event_Breaker config. A very useful new feature to improve the behavior of Splunks AutoLoadBalance functionality 🙂

@Ias: your indexer config has <\d{2,3}> without backslashes for the < and > signs. You need to escape those! So it should be:

LINE_BREAKER = ()\d{2,3}\s+\<\d{2,3}\>

[tiagofbmm]

Hey

I tested this LINE_BREAKER on your data.
Can you check please?

LINE_BREAKER=(\<\d+\>)

[las]

Thanks for the suggestion, but unfortunately I still see the same behavior.

[tiagofbmm]

This should be working because I tested with your exact data.

Where are the props.conf file? In the Universal Forwarder or an Indexer/Heavy Forwarder?

[las]

And I agree, this should work.

[tiagofbmm]

Ok so thinking of the basics.

Have you restarted Splunk after changing the props.conf file for testing the LINE_BREAKER I suggested?

Is your data passing through a Heavy Forwarder before arriving to your Indexer?

[las]

It's on the indexer, in the search app.