The LSF daemon (lim, mbatchd, mbschd, sbatchd) logs are pretty simple and can be done via standard file input/monitor. Another option is to configure LSF to log the daemon logs to syslog.
The LSF accounting logs (lsb.acct.*) can be quite tricky since the number of fields in each record is variable depending upon the type of LSF job and the specific LSF job options. For example, LSF jobs which specify -n slots (numAskedHosts >0) will then add n additional fields for each askedHost. The lsb.acct man page all has all details on the specific format of the LSF accounting file. We've developed a perl regular expression to extract the LSF accounting file fields into something meaningful, but ran into some splunk limitation on long complex perl regular expressions (it works fine outside splunk when we run it directly, but for some reason fails to parse when configured in props.conf
I've also written some wrapper scripts to extract the LSF cluster file and LSF hostgroups out of lsb.hosts so splunk can use them as well.
... View more