from btools prop list run on search head.
The events still break on dates within the events rather than the "---------" so we have a bunch of partial events being indexed.
[sourcetypes]
ANNOTATE_PUNCT = True
AUTO_KV_JSON = true
BREAK_ONLY_BEFORE = ----------
BREAK_ONLY_BEFORE_DATE = True
CHARSET = UTF-8
DATETIME_CONFIG =
HEADER_MODE =
KVMODE =
LEARN_SOURCETYPE = true
LINE_BREAKER_LOOKBEHIND = 100
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
MAX_EVENTS = 10000
MAX_TIMESTAMP_LOOKAHEAD = 128
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
NO_BINARY_CHECK = true
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
SHOULD_LINEMERGE = True
TRANSFORMS =
TRUNCATE = 1000000
category = Custom
detect_trailing_nulls = false
disabled = false
maxDist = 100
priority =
pulldown_type = true
sourcetype =
I would try like this for your sourcetype definition (props.conf on indexer/heavy forwarder)
[sourcetypes]
LINE_BREAKER =([\r\n]+)(----------)
SHOULD_LINEMERGE = false
TIME_PREFIX = Date=
TIME_FORMAT = %Y-%m-%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
TRUNCATE = 1000000
I would try like this for your sourcetype definition (props.conf on indexer/heavy forwarder)
[sourcetypes]
LINE_BREAKER =([\r\n]+)(----------)
SHOULD_LINEMERGE = false
TIME_PREFIX = Date=
TIME_FORMAT = %Y-%m-%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
TRUNCATE = 1000000
These pushed out to the search head and forwarders and still doesn't work
Do you have a working solution (this is clicked Accepted
)?
As mentioned it should be in indexer or Heavy forwarder. A restart of SPlunk is required after you push the change.
Refer to following link. Most of the parameters related to line breaking are required on Indexer and not on forwarder.
https://wiki.splunk.com/Community:HowIndexingWorks
Try setting BREAK_ONLY_BEFORE_DATE = false in your props.conf.
still doesnt work
ANNOTATE_PUNCT = True
AUTO_KV_JSON = true
BREAK_ONLY_BEFORE = ----------
BREAK_ONLY_BEFORE_DATE = false
CHARSET = UTF-8
DATETIME_CONFIG =
HEADER_MODE =
KVMODE =
LEARN_SOURCETYPE = true
LINE_BREAKER_LOOKBEHIND = 100
LOOKUP-dropdowns = dropdownsLookup host OUTPUT unix_category unix_group
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
MAX_EVENTS = 10000
MAX_TIMESTAMP_LOOKAHEAD = 128
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
NO_BINARY_CHECK = true
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
SHOULD_LINEMERGE = True
TRANSFORMS =
TRUNCATE = 1000000
category = Custom
detect_trailing_nulls = false
disabled = false
maxDist = 100
priority =
pulldown_type = true
sourcetype =
Sample events please.
here is a sample. it will break on the context create time and sometimes the log time, sometimes the date=, but never the ------ explicitly declared
----------
Date=2016-06-20 15:54:20Z
LogLevel=INFO
Logger=XXXXXX.ContactCenter.NGAT.Web.Infrastructure.MultiCacheProviderFactory
Thread=161
LogContext=UIGeneralLog
Message=MultiCacheProvider: Operation succeeded.
MachineName=XXXXXXXX
MethodName=MultiCacheProviderOperationCompletedHandler
SourcePath=E:\jenkins\workspace\XXXXX\web\Infrastructure\MultiCacheProviderFactory.cs
SourceLine=218
ContextCreateTime=2016-06-20 15:54:20Z
LogTime=2016-06-20 15:54:20Z
SessionId=
InteractionIdStringId=f809f344-d728-4d12-9301-889ba406d86f
CallSessionIdStringId=8c5109ad-2ed1-4ed3-8e33-de5c34be4625
ActivityIdStringId=9d49138f-e264-4799-9279-b3c616bf9927
AgentUserName=
UserAgent=Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.1;+WOW64;+Trident/7.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+.NET4.0C;+.NET4.0E;+InfoPath.3)
RequestPath=https://XXXXX/
Referrer=https://XXXXXX/?wa=wsignin1.0&wtrealm=https:%2f%XXXXXX%2f&wctx=rm%3d0%26id%3dpassive%26ru%3d%252f&wct=2016-06-20T15:54:20Z
SessionIsNew=
SessionKeys=
Controller=
Action=
MC.RequestId=98052227
MC.Operation=AddOrGetExisting
MC.Result=Success
MC.TotalTimeMillis=15.5998
MC.CacheKey=SessionSecurityToken-/;urn:uuid:cc6794c2-65b8-46f4-a36a-eb015d92a50c;
MC.CacheHit=False