Help with complex multiline backup log

[holtb]

We're trying to analyze a complex multiline backup log and having some difficulties. I was hoping an expert here could help. The backup logs have two states, one of which is success, looking like below. Otherwise it has lines in it that look like this denoting eror:

ANS1802E Incremental backup of '/mnt/wpimaging/acsm-home' finished with 103 failure
ANS1017E Session rejected: TCP/IP connection failure

What I would really like is to only see the summary and ANS....E lines, ignoring the ANS....I or ANS....W lines. It'd be great if all the fields of the summary were captured as fields so we could generate table of all the successful backups.

So far we just have the following in our props.conf, any suggestions would be appreciated

[TSM_REPORTS]
MAX_EVENTS=2000000
LINE_BREAKER=((?!))
LEARN_MODEL = false
BREAK_ONLY_BEFORE= IBM Tivoli Storage Manager
BREAK_ONLY_BEFORE_DATE = False
MUST_BREAK_AFTER= Elapsed processing time:
TRUNCATE=0

Logfile:

IBM Tivoli Storage Manager
Command Line Backup-Archive Client Interface
  Client Version 6, Release 2, Level 2.3  
  Client date/time: 08/28/2012 15:15:02
(c) Copyright by IBM Corporation and other(s) 1990, 2010. All Rights Reserved.
Normal File-->               348 /mnt/wpimaging/analysis/acsm-collaboration/Glioma/BT142-nonLUC-p4_BO32/081712/BO32_m09.hdr [Sent]      
Normal File-->           491,520 /mnt/wpimaging/analysis/acsm-collaboration/Glioma/BT142-nonLUC-p4_BO32/081712/BO32_m09.img [Sent]      
Normal File-->             9,262 /mnt/wpimaging/analysis/acsm-collaboration/Glioma/BT142-nonLUC-p4_BO32/081712/BO32_m09.imp [Sent]      
Normal File-->             3,411 /mnt/wpimaging/analysis/acsm-collaboration/Glioma/BT142-nonLUC-p4_BO32/081712/BO32_m09.mat [Sent]      
Normal File-->           985,608 /mnt/wpimaging/analysis/acsm-collaboration/Glioma/BT142-nonLUC-p4_BO32/081712/BO32_m09.mnc [Sent]      
Normal File-->           986,136 /mnt/wpimaging/analysis/acsm-collaboration/Glioma/BT142-nonLUC-p4_BO32/081712/BO32_m09_norm.mnc [Sent]      
Normal File-->               348 /mnt/wpimaging/analysis/acsm-collaboration/Glioma/BT142-nonLUC-p4_BO32/081712/BO32_m09_norm_dist5.hdr [Sent]      
Normal File-->           983,040 /mnt/wpimaging/analysis/acsm-collaboration/Glioma/BT142-nonLUC-p4_BO32/081712/BO32_m09_norm_dist5.img [Sent]      
Normal File-->               428 /mnt/wpimaging/analysis/acsm-collaboration/Glioma/BT142-nonLUC-p4_BO32/081712/BO32_m09_norm_dist5.mat [Sent]      
Normal File-->               348 /mnt/wpimaging/analysis/acsm-collaboration/Glioma/BT142-nonLUC-p4_BO32/081712/BO32_m09_nuc.hdr [Sent]      
Normal File-->           983,040 /mnt/wpimaging/analysis/acsm-collaboration/Glioma/BT142-nonLUC-p4_BO32/081712/BO32_m09_nuc.img [Sent]      
Normal File-->               428 /mnt/wpimaging/analysis/acsm-collaboration/Glioma/BT142-nonLUC-p4_BO32/081712/BO32_m09_nuc.mat [Sent]      
Successful incremental backup of '/mnt/wpimaging/analysis'
Total number of objects inspected: 8,260,012
Total number of objects backed up:   18,077
Total number of objects updated:          0
Total number of objects rebound:          0
Total number of objects deleted:          0
Total number of objects expired:          8
Total number of objects failed:           0
Total number of bytes inspected:     26.25 TB
Total number of bytes transferred:   78.61 GB
Data transfer time:                46,329.38 sec
Network data transfer rate:        1,779.22 KB/sec
Aggregate data transfer rate:      1,537.23 KB/sec
Objects compressed by:                    0%
Total data reduction ratio:           99.71%
Elapsed processing time:           14:53:42

[holtb]

It seems to work well with the modification to the LINE_BREAKER. I wrote the following rules for field extraction (from transforms.conf), can anyone comment on their efficiency, is there a better way to do this?

[extract_tsm_success]
REGEX = ^Successful\ incremental\ backup\ of\ \'(.+?)\'\n
FORMAT = Filesystem::$1
[extract_tsm_oinspect]
REGEX = ^Total number\ of\ objects\ inspected:\s+(.+?)\n
FORMAT = ObjectsInspected::$1
[extract_tsm_obackup]
REGEX = ^Total\ number\ of\ objects\ backed\ up:\s+(.+?)\n
FORMAT = ObjectsBackedup::$1
[extract_tsm_oupdate]
REGEX = ^Total\ number\ of\ objects\ updated:\s+(.+?)\n
FORMAT = ObjectsUpdated::$1
[extract_tsm_orebound]
REGEX = ^Total\ number\ of\ objects\ rebound:\s+(.+?)\n
FORMAT = ObjectsRebound::$1
[extract_tsm_odeleted]
REGEX = ^Total\ number\ of\ objects\ deleted:\s+(.+?)\n
FORMAT = ObjectsDeleted::$1
[extract_tsm_oexpired]
REGEX = ^Total\ number\ of\ objects\ expired:\s+(.+?)\n
FORMAT = ObjectsExpired::$1
[extract_tsm_ofailed]
REGEX = ^Total\ number\ of\ objects\ failed:\s+(.+?)\n
FORMAT = ObjectsFailed::$1

[extract_tsm_bytesinspected]
REGEX = ^Total\ number\ of\ bytes\ inspected:\s+(.+?)\n
FORMAT = BytesInspected::$1
[extract_tsm_bytestransferred]
REGEX = ^Total number of bytes transferred:\s+(.+?)\n
FORMAT = BytesTransfered::$1
[extract_tsm_transfertime]
REGEX = ^Data\ transfer\ time:\s+(.+?)\n
FORMAT = TransferTime::$1
[extract_tsm_transferrate]
REGEX = ^Network\ data\ transfer\ rate:\s+(.+?)\n
FORMAT = TransferRate::$1
[extract_tsm_aggtransferrate]
REGEX = ^Aggregate\ data\ transfer\ rate:\s+(.+?)\n
FORMAT = AggregateTransferRate::$1
[extract_tsm_objcompression]
REGEX = ^Objects\ compressed\ by:\s+(.+?)\n
FORMAT = ObjectCompression::$1
[extract_tsm_reductionratio]
REGEX = ^Total\ data\ reduction\ ratio:\s+(.+?)\n
FORMAT = ReductionRatio::$1
[extract_tsm_timeelapsed]
REGEX = ^Elapsed\ processing\ time:\s+(.+?)\n
FORMAT = TimeElapsed::$1

[kristian_kolb]

You should have a look at the docs for props.conf. LINE_BREAKER only works if SHOULD_LINEMERGE = false otherwise it's ignored.

Stuff that goes together:

SHOULD_LINEMERGE = true
BREAK_ONLY ... =
MUST BREAK ... =
MAX_EVENTS =

Other stuff that goes together:

SHOULD_LINEMERGE=false
LINE_BREAKER =

Hope that can work as a start. Also, I don't relly understand your regex for the LINE_BREAKER

Please post a few more events, with success and fails following each other. Linebreaking can be a bit tricky when the formats between the types of event differ.

/K

[holtb]

Awesome, thanks. The LINE_BREAKER was stolen from another splunkbase when trying to take each logfile as a single event.