How is categorization of messages (ERROR, WARNING, INFORMATION ETC) normally handled by a Splunk forwarder?

[bareisd]

I am a Splunk user (with no control of data collection) and have set up color coding for errors (red) warning etc in different colors. To do this, I had to categorize the data I could, but I can't believe that is how it should be done.

My understanding is that each type of source (Windows event logs, IIS logs, webshere java logs, etc) will have a forwarder that can be configured.

I have not seen any mention (in Splunk documentation and sites) that there are any STANDARD categories of messages such as "errors", "warnings" etc. This means either that there is none or that its so fundamental, it's assumed everyone knows this...

It would seem to me that the forwarder should be in charge of categorizing messages with event log messages being the easiest as the source is already categorized, but other files are not hard either. For example, an IIS response code of "500" is obviously an error (and 200 = good) as are events in standard error java logs. Others might need more configuration with regular expressions.

So basically, what I should be able to do is use the passed category to color code my messages as well as search for all errors categorized as "errors".

How is this normally handled?

[lguinn2]

The Splunk forwarder does not categorize any data. It simply collects the data and passes it on to the indexer along with basic metadata, which includes the current time & time zone, the host name, the sourcetype and the destination index. Of course the forwarder will be configured differently for different types of servers, as the actual log files and collection mechanisms vary. But no "category" is created or passed from the forwarder, at least not in terms of defining "errors", "warnings", etc. for events.

Events are categorized in Splunk at search time, not on the forwarder; doing it this way offers much more flexibility. Tags and eventtypes are used to categorize data, and field extractions are usually part of the categorization process. These are all called "knowledge objects" in Splunk and they are often defined based on the sourcetype of the data.

You can create your own sourcetypes and knowledge objects, but many are already defined. A few are defined in Splunk itself, but most are defined in apps. You can download apps from SplunkBase at http://splunkbase.splunk.com; over a thousand apps exist and almost all of them are free. Just a few examples: the Splunk Add-on for Cisco ASA and the Palo Alto Networks Add-on

However, every vendor and app is free to set up and define their log files (and knowledge objects) in any way they like. To make it easier to integrate disparate data sources, Splunk has defined the Common Information Model (CIM). You can read the CIM Manual and even download the free CIM app. Many of the apps on SplunkBase follow the CIM.

What is an "error" vs. "warning" certainly depends on both the data source and your environment. I suggest that you learn more about the apps and the CIM as a starting point. I also suggest that you use tags and eventtypes to categorize your data in a customized way.