I'm new to splunk and python, but I've been managing a ROCKS-based HPCC for over 3 years now and I'm looking for an easier way to analyze the LSF scheduler accounting logs. They are plain-text, but relatively voluminous (3-5 GB/year). The management here likes graphs and pictures too.
Anyone else have a need like this? I'd love to take it on myself, but as I stated earlier, I'm still quite green with this stuff.
I was just curious if anyone has an update to this thread. Has anyone been working with LSF parsing and is willing to share their information for an app?
Thanks.
HeyBigBen: Would you be willing to share some of your work with us here on SplunkBase? If you don't feel like posting an App, I'd be happy to review your work and bundle it as an App here (I'm a SE at Splunk). If you're willing to share, it would be a great resource for everyone -- feel free to send me what you've got at jmiller (at) splunk (dot) com.
Cheers,
Jesse
The LSF daemon (lim, mbatchd, mbschd, sbatchd) logs are pretty simple and can be done via standard file input/monitor. Another option is to configure LSF to log the daemon logs to syslog.
The LSF accounting logs (lsb.acct.*) can be quite tricky since the number of fields in each record is variable depending upon the type of LSF job and the specific LSF job options. For example, LSF jobs which specify -n slots (numAskedHosts >0) will then add n additional fields for each askedHost. The lsb.acct man page all has all details on the specific format of the LSF accounting file. We've developed a perl regular expression to extract the LSF accounting file fields into something meaningful, but ran into some splunk limitation on long complex perl regular expressions (it works fine outside splunk when we run it directly, but for some reason fails to parse when configured in props.conf
I've also written some wrapper scripts to extract the LSF cluster file and LSF hostgroups out of lsb.hosts so splunk can use them as well.
Alex - could you let me know who this is working on an app? I'd like to add them to my pipeline.
Thanks,
JM
I recently talked to some folks that are splunking their LSF scheduler logs across their grid with some success and have interest in posting an app to Splunkbase. I will do my best to update this question when I have more information.
Hmm. If you could send me a sample of the log (nick at sideviewapps.com) I could take a look at it. Often it takes only a couple hours to build the interactive skeleton for an app. The difficulty level largely depends on the data though.
Splunk even has a tool to help you anonymize it if you need to. http://www.splunk.com/base/Documentation/latest/Admin/Anonymizedatasamplestosendtosupport