I am trying to understand which pipeline deals with CIM? We have four sets of pipelines - is it the indexing pipeline which uses CIM or Parsing Pipeline. Please let me know.
Thanks
nandha
This is kind of a big question, but I will try to be clear.
First, the Common Information Model (CIM) is not a specific step or process inside of Splunk. The CIM is basically a standardized set of fields, tags, and eventtypes. Think of it as a schema where the different data sources follow the rules of the schema by mapping their custom fields to the schema defined fields, e.g., aliasing a sourcetype-specific field "UserName" to the CIM-standard field "user".
In the most common cases, mapping sourcetype-specific fields to CIM is done at search-time, not at index-time. So my answer doesn't really meet your expectations, because search-time field extractions comes after the input, parsing, and indexing pipelines.
Now, one can do index-time field extraction to CIM-compliant names (which would be the parsing pipeline), but I would say that is not the norm. Most CIM-compliant mapping is done in the props.conf, transforms.conf, tags.conf, and eventtypes.conf files on a search-head.
More reading: http://docs.splunk.com/Documentation/CIM/4.6.0/User/UsetheCIMtonormalizedataatsearchtime
This is kind of a big question, but I will try to be clear.
First, the Common Information Model (CIM) is not a specific step or process inside of Splunk. The CIM is basically a standardized set of fields, tags, and eventtypes. Think of it as a schema where the different data sources follow the rules of the schema by mapping their custom fields to the schema defined fields, e.g., aliasing a sourcetype-specific field "UserName" to the CIM-standard field "user".
In the most common cases, mapping sourcetype-specific fields to CIM is done at search-time, not at index-time. So my answer doesn't really meet your expectations, because search-time field extractions comes after the input, parsing, and indexing pipelines.
Now, one can do index-time field extraction to CIM-compliant names (which would be the parsing pipeline), but I would say that is not the norm. Most CIM-compliant mapping is done in the props.conf, transforms.conf, tags.conf, and eventtypes.conf files on a search-head.
More reading: http://docs.splunk.com/Documentation/CIM/4.6.0/User/UsetheCIMtonormalizedataatsearchtime
convincing for me.. thanks
Glad to be helpful.
Please accept the answer if it satisfies your question.