Getting Data In

bacula events again

sergemueller
Explorer

Hi,
i've read and tried this but somehow it does not work for me.

i've put the recommended settings into:

$SPLUNKHOME$/etc/system/local/props.conf
and also tried

$SPLUNKHOME$/etc/apps/search/local/props.conf

but it never really works.
Some events start in the middle of the indented block(eg at Start Time)
some start at the Date, but i never got the whole idendet Block into one event.
Here's an example of the data:


17-Jun 10:05 backup1.ber2-dir JobId 1948: Start Backup JobId 1948, Job=test.2011-06-17_10.05.00_40
17-Jun 10:05 backup1.ber2-dir JobId 1948: Using Device "FileStorage_test"
17-Jun 10:05 backup1.ber2-sd JobId 1948: Volume "test_new02" previously written, moving to end of data.
17-Jun 10:05 backup1.ber2-sd JobId 1948: Ready to append to end of Volume "test_new02" size=31846981494
17-Jun 10:10 backup1.ber2-sd JobId 1948: Job write elapsed time = 00:05:29, Transfer rate = 6.540 M Bytes/second
17-Jun 10:10 backup1.ber2-dir JobId 1948: Bacula backup1.ber2-dir 5.0.3 (04Aug10): 17-Jun-2011 10:10:39
Build OS: x86_64-unknown-linux-gnu debian 5.0.6
JobId: 1948
Job: test.2011-06-17_10.05.00_40
Backup Level: Incremental, since=2011-06-17 06:05:03
Client: "server11227.example.com-fd" 5.0.2 (28Apr10) x86_64-unknown-linux-gnu,redhat,Enterprise release
FileSet: "test Set" 2011-01-16 11:29:38
Pool: "File_test" (From Job resource)
Catalog: "MyCatalog" (From Client resource)
Storage: "File_test" (From Job resource)
Scheduled time: 17-Jun-2011 10:05:00
Start time: 17-Jun-2011 10:05:03
End time: 17-Jun-2011 10:10:39
Elapsed time: 5 mins 36 secs
Priority: 10
FD Files Written: 5
SD Files Written: 5
FD Bytes Written: 2,151,817,284 (2.151 GB)
SD Bytes Written: 2,151,817,820 (2.151 GB)
Rate: 6404.2 KB/s
Software Compression: None
VSS: no
Encryption: no
Accurate: no
Volume name(s): test_new02
Volume Session Id: 32
Volume Session Time: 1308035975
Last Volume Bytes: 34,000,395,480 (34.00 GB)
Non-fatal FD errors: 0
SD Errors: 0
FD termination status: OK
SD termination status: OK
Termination: Backup OK

17-Jun 10:10 backup1.ber2-dir JobId 1948: Begin pruning Jobs older than 6 months .
17-Jun 10:10 backup1.ber2-dir JobId 1948: No Jobs found to prune.
17-Jun 10:10 backup1.ber2-dir JobId 1948: Begin pruning Jobs.
17-Jun 10:10 backup1.ber2-dir JobId 1948: No Files found to prune.
17-Jun 10:10 backup1.ber2-dir JobId 1948: End auto prune.

Any help would be appreciated

Edit: Ah well i forgot:

Excerp from my props.conf:


[bacula]
BREAK_ONLY_BEFORE=^\d{2}-[A-Za-z]{3}\s+\d{2}:\d{2}\s+
SHOULD_LINEMERGE = true
TIME_FORMAT=%d-%b %H:%M
TIME_PREFIX=^
MAX_TIMESTAMP_LOOKAHEAD=13

Tags (2)
0 Karma

kristian_kolb
Ultra Champion

Hi,

did this message board filter out your backslashes or does your props.conf regex actually look like this?

[bacula]
BREAK_ONLY_BEFORE=^d{2}-[A-Za-z]{3}s+d{2}:d{2}s+

Other than that, I think you should be able to break at the correct places with (just being a little more specific on the numbers used for matching date/time);

[bacula]
LINE_BREAKER=([\r\n]+)[0-3][0-9]-[A-Za-z]{3}\s+[0-2][0-9]:[0-5][0-9]\s+
SHOULD_LINEMERGE = False
TIME_FORMAT=%d-%b %H:%M
TIME_PREFIX=^
MAX_TIMESTAMP_LOOKAHEAD=13

Hope this helps,

Kristian

0 Karma

aquilino1
New Member

You might want to double-check the first line of your bacula log file and make sure that it doesn't have a regular syslog style timestamp. If it does, this may be throwing off proper detection of the timestamp event for your bacula sourcetype on initial import. I had to delete the first line in my bacula log file and then re-index the data in order to correct the problem.

Also, at the risk of someone telling me that I'm doing my field extractions all wrong (I'm still learning), here is my bacula_logs sourcetype entry in props.conf in case it's helpful to anyone. So far it seems to be working well. If anyone has a suggestion for how I might improve upon it though, I'm open to ideas. 😃

[bacula_logs]
LEARN_MODEL=true
TIME_FORMAT = %d-%b %H:%M
EXTRACT-jobstatus = Termination:\s+\W*(?<jobstatus>(Backup OK|Backup Error|Backup Canceled))
EXTRACT-backupclient-1 = Client: \s*\W*(?<backupclient>[a-z0-9\-A-Z]*)
EXTRACT-backupclient-2 = gethostbyname\(\) for host "(?<backupclient>[a-z\.0-9\-A-Z]*)"
EXTRACT-backuplevel = Backup Level: \s*(?<backuplevel>\w*)
EXTRACT-compression = Software Compression: \s*(?<compression>\S+\s*%)
EXTRACT-sourcehost = \d+-\w\w\w \d+:\d+ \s*(?<sourcehost>\S*)
EXTRACT-jobid = JobId [:]*(?<jobid>\d+)
EXTRACT-byteswritten = SD Bytes Written:\s+(?<byteswritten>\S*)
EXTRACT-origination = ^\d\d-\w\w\w \d\d:\d\d (?<origination>\S+)
EXTRACT-fatal-error = Fatal error: (?<fatal_error>.*)
EXTRACT-fd-status = FD termination status:  (?<fd_status>.*?(?=SD term))
EXTRACT-sd-status = SD termination status:  (?<sd_status>.*?(?=Termination:))
EXTRACT-jobname = \s+Job: \s+(?<jobname>\S+)
0 Karma

ziegfried
Influencer

You could try specifying a line breaker instead of using line merging:

[bacula]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?!\s+)

This should break events by vertical whitespace not followed by whitespace characters.

0 Karma

sergemueller
Explorer

it also appears, that the TIME_PREFIX is not recognized, since it evaluates times in the middel of the line (The One after " Start Time:"), but the sourcetype is recognized (i tested it with an extraction)

0 Karma

sergemueller
Explorer

doesn really work,

now it breaks between "end time:" and volume session id

i dont really getit

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...