Sourcetype based Eventtype doesn't pickup renamed sourcetypes consistently

[vdurepaire]

Hi Guys,

I am currently facing an issue with ES which seems to be originating from renaming custom sourcetype names to Splunk TA’s expected ones. Seeking for some thoughts.

My client is forcing a custom sourcetype naming convention for each sources at inputs time. These sourcetypes names are different from the ones expected by some Splunk, out of the box, TA's add-on. In order to leverage the CIM mappings provided by the Splunk TA's, we have configured sourcetype renaming at search time in props.conf so that all mappings, field extraction, tagging... are applied seamlessly in Enterprise security.

(Let’s take Splunk_TA_oracle as an example going forward)
When searching from enterprise security, the custom sourcetype for oracle audit logs is successfully renamed to "oracle:audit:text" and all database events are coming up with correct field extractions, mapping to CIM (Including eventtypes and tags). However when searching by tag (e.g. tag=authentication), these event that were successfully tagged as “authentication” in the previous result set do not come up. This is a problem since the population of the CIM data model is based on the tags.....

The authentication data model and therefore ES doesn’t pick up these authentication events.

Would you know what the reason is and is there a workaround?

Kind regards,

Vincent

[hunters_splunk]

Hi vdurepaire,

I don't think any search-time transformation can make data inputs CIM-compliant. Data model compliance should be ensured at index-time. If a sourcetype is not correct during the input phase, you can still fine-tune the sourcetype during the parsing phase, but event and field data cannot be changed after indexing. Any search-time transformation does not resolve data model compliance issues.

Here is an example of how you can fine-tune directory monitor sourcetypes:
inputs.conf
[monitor:///var/log/]
props.conf
[source::/var/log/mail.log]
sourcetype=sendmail
[source::/var/log/secure]
sourcetype=secure
...

Hope it helps. Thanks!
Hunter

[vdurepaire]

Thank you Hunters for taking the time to respond, It is very much apreciated.

I would also prefer to assign correct sourcetype at input/parsing/indexing time however I don't have this luxury in my setup.

The client has decided as per design to set right from the input/parsing time the custom sourcetypes. This cannot be altered and was already source of a lot of discussions at the time of design.

Now I am trying to make ES work without duplicating & tweaking every out of the box Splunk add-ons.

Now regarding your answer, can you elaborate your point about the impossibility to "fix CIM compliance issues at search time? Isn't it how most of CIM mappings are performed OOTB with SPLUNK add-ons? Before reading your comment I would have thought it was the best place for all field extractions, eventtyping...

In my situation the search in ES for "sourcetype=NewSourcetype" actually returns all events with proper CIM mappings (Tags, Eventtypes, CIM aligned field extractions). This means that the search type sourcetype renaming works and that all search time props/Transform do apply correctly. for this search, the tag "authentication" is applied correctly for thousands of events.

The issue occurs only when searching by the Tag. e.g. "sourcetype=NewSourcetype tag=authentication" . The search doesn't return any events. It seems that sourcetype renaming at search time only works when searching by sourcetype... this is very odd.

Does it clarify the issue? I can craft some screenshot if needed.

Kind regards

Vincent