I have a file that has multiple multi line events. Each event is broken up into "INFO: ---" or "ERROR: ---"
ERROR: ---
blahNewsLetter: N
birthdate: 1947-10-25
countryId: 1
createdOn: 2011-05-31 13:40:46
...
INFO: ---
blahNewsLetter: ~
birthdate: 0000-00-00
countryId: ~
createdOn: 2011-05-31 13:40:48
...
My props.conf
[dblog]
SHOULD_LINEMERGE = true
# force splunk to detec multiline events
BREAK_ONLY_BEFORE = (.*)(INFO|ERROR):
My inputs.conf
[monitor:///var/log/dblog.log]
disabled = false
index = blah
sourcetype = dblog
blacklist = (\.(gz|bz2|z|zip)$)
followTail = 1
This does not work. It is splitting the file up into events that have lines similar to the "createdOn: 2011-05-31 13:40:48" lines.
modifiedOn: 2011-05-31 13:40:48
postalCode: 1111
promoCode: ~
requestDetail: |-
<request>
<billToEmail>null</billToEmail>
<billToFirstName>Name</billToFirstName>
<billToLastName>Name</billToLastName>
<billToStreet1>null</billToStreet1>
<billToStreet2>null</billToStreet2>
...
and
createdOn: 2011-05-31 13:40:48
email: myemail@email.com
Any help would be appreciated. And I have also tried using BREAK_ONLY_BEFORE_DATE = false with same result. This is currently on the forwarder.
lcasey001,
I would recommend using LINE_BREAKER.
[dblog]
SHOULD_LINEMERGE = false
# force splunk to detec multiline events
LINE_BREAKER = ([\r\n]+)(?:INFO|ERROR):
Update: These configurations need to be on your indexer. I would also recommend modifying Splunk's date/time properties since there is not a timestamp in first 150 characters of your event.
[dblog]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?:INFO|ERROR):
TIME_PREFIX = createdOn:\s+
TIME_FORMAT = %Y-%m-%d %H:%M:%S
lcasey001,
I would recommend using LINE_BREAKER.
[dblog]
SHOULD_LINEMERGE = false
# force splunk to detec multiline events
LINE_BREAKER = ([\r\n]+)(?:INFO|ERROR):
Update: These configurations need to be on your indexer. I would also recommend modifying Splunk's date/time properties since there is not a timestamp in first 150 characters of your event.
[dblog]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?:INFO|ERROR):
TIME_PREFIX = createdOn:\s+
TIME_FORMAT = %Y-%m-%d %H:%M:%S
Not sure if there is a specific list, but any props related to line breaking, time parsing, or of type TRANSFORMS- are performed @ the indexing/full forwarder layer of your Splunk infrastructure.
Where can one obtain that list for future reference? I did not see anything in the props.conf.spec file.
Thanks !!! That was it. It needed to be done on the indexer and not forwarder. So is there a list of things that needs to be done on the indexer vs forwarder regarding props.conf ?
Line breaking and time parsing are done @ the indexer or Full forwarder, not Universal Forwarder/Light Forwarder. Please move these configurations to your indexer(s).
These changes have been on the Universal Forwarder where the log file is hosted. I have other settings on the forwarder for other log files, mostly setting sourcetypes in the props.conf file. Should this all be done on the indexer?
Made your recommended change, restarted the forwarder, and got the same result.
5/31/11
3:31:17.000 PM
modifiedOn: 2011-05-31 15:31:17
postalCode: 11111
promoCode: ''
requestDetail: |-
<request>
<email>email@email.net</email>
<userName>username</userName>
<firstName>firstname</firstName>
<lastName>lastname</lastName>
<language>en</language>
Show all 18 lines