Reporting

How do I go about importing Logwatch reports into splunk?

XenoPhage_1
New Member

For a variety of reasons I'm not able to push all of our syslog data to splunk. I can, however, easily generate daily logwatch reports which can either be placed into a directory, emailed, or whatever is needed to get them over to the splunk server. That part is easy. What I'm not sure of is how to go about getting splunk to eat each report as a single report and be able to generate useful reports on the data.

To put it a bit simpler, I get about 100 or so logwatch reports right now and that number is just increasing. What I'd like to do is use splunk to process these reports and generate a unified report with some basic statistics and a section for "these things are broke/unknown" so I can scan one report instead of 100.

Has anyone done this? Any hints on where to begin to get this implemented?

Thanks!

Tags (2)
0 Karma

jrodman
Splunk Employee
Splunk Employee

If the logwatch output is itself structured as a logfile, then this becomes a simple matter of pushing this data into splunk, possibly arranging to use a sinkhole for disk management goals.

If the logwatch output is structured as a document, then you have a good deal of work to get something useful. I see http://www.cyberciti.biz/faq/freebsd-unix-log-analyzer-configuration/ suggesting that the logwatch output is actually a series of discrete components each with custom formatting, which are totally heterogeneous.

If that were my report format, and this were my problem, I would write a script that cuts up the reports, and creates a logfile for each category (logwatch_pam_auth.log), manually parsing the Date Range Processed for each report, and inserting the date between each entry.

At that point getting them as a set of events in splunk is easy.

There are nonscript approaches to getting the events cut up in splunk as well, but might require creating a custom datetime.xml and complex rules for event parsing. However, I suspect giving each data category its own sourcetype will greatly aid in sane field extraction.

Mostly though, you're using a tool that's designed to manage a single system, and isn't built for managing systems in aggregate, and struggling with that. Maybe there are better tools?

Personally I recommend Splunk for this, syslog datarates are usually low enough that it's not a real issue. Any sort of application or appliance data is usually vastly larger. However there are specialized tools which process log data to produce aggrate information. For example http://ossec.net

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...