Splunk Enterprise Security

map_notable_fields in ES bug ?

agneticdk
Path Finder

Hi guys

I have this search:

| datamodel "Malware" "Malware_Attacks" search | `drop_dm_object_name(Malware_Attacks)` |`map_notable_fields` | search abc

It does NOT filter on the abc content.

If I do this:

| datamodel "Malware" "Malware_Attacks" search | `drop_dm_object_name(Malware_Attacks)` | search abc |`map_notable_fields`

it works.

I can see in the job inspector that the first search adds a "| noop" to the normalized search just before the "|search abc":

| search (tag=attack tag=malware (index=* OR index=_*) (index="estreamer" OR index="pan" OR index="trendmicro" OR index="wineventlog")) DIRECTIVES(READ_SUMMARY(allow_old_summaries="false" dmid="FAC94FC4-23AE-44F4-BA73-E65513A137DB_DM_Splunk_SA_CIM_Malware" name="Malware.Malware_Attacks" predicate="*" summariesonly="false"),REQUIRED_TAGS(intersect="t" tags="pci")) | eval action=if((isnull(action) OR (action == "")),"unknown",action), category=if((isnull(category) OR (category == "")),"unknown",category), date=strftime('_time',"%m-%d-%Y"), signature=if((isnull(signature) OR (signature == "")),"unknown",signature), dest=if((isnull(dest) OR (dest == "")),"unknown",dest), dest_nt_domain=if((isnull(dest_nt_domain) OR (dest_nt_domain == "")),"unknown",dest_nt_domain), user=if((isnull(user) OR (user == "")),"unknown",user), vendor_product=case(isnotnull(vendor_product),vendor_product,(((isnotnull(vendor) AND (vendor != "unknown")) AND isnotnull(product)) AND (product != "unknown")),((vendor . " ") . product),((isnotnull(vendor) AND (vendor != "unknown")) AND (isnull(product) OR (product == "unknown"))),(vendor . " unknown"),(((isnull(vendor) OR (vendor == "unknown")) AND isnotnull(product)) AND (product != "unknown")),("unknown " . product),isnotnull(sourcetype),sourcetype,true(),"unknown"), is_Allowed_Malware=if(searchmatch("action=\"allowed\""),1,0), is_not_Allowed_Malware=(1 - is_Allowed_Malware), is_Blocked_Malware=if(searchmatch("action=\"blocked\""),1,0), is_not_Blocked_Malware=(1 - is_Blocked_Malware), is_Deferred_Malware=if(searchmatch("action=\"deferred\""),1,0), is_not_Deferred_Malware=(1 - is_Deferred_Malware) | rename action as "Malware_Attacks.action", category as "Malware_Attacks.category", date as "Malware_Attacks.date", dest as "Malware_Attacks.dest", dest_bunit as "Malware_Attacks.dest_bunit", dest_category as "Malware_Attacks.dest_category", dest_nt_domain as "Malware_Attacks.dest_nt_domain", dest_priority as "Malware_Attacks.dest_priority", dest_requires_av as "Malware_Attacks.dest_requires_av", file_hash as "Malware_Attacks.file_hash", file_name as "Malware_Attacks.file_name", file_path as "Malware_Attacks.file_path", is_Allowed_Malware as "Malware_Attacks.is_Allowed_Malware", is_Blocked_Malware as "Malware_Attacks.is_Blocked_Malware", is_Deferred_Malware as "Malware_Attacks.is_Deferred_Malware", is_not_Allowed_Malware as "Malware_Attacks.is_not_Allowed_Malware", is_not_Blocked_Malware as "Malware_Attacks.is_not_Blocked_Malware", is_not_Deferred_Malware as "Malware_Attacks.is_not_Deferred_Malware", sender as "Malware_Attacks.sender", signature as "Malware_Attacks.signature", src as "Malware_Attacks.src", src_bunit as "Malware_Attacks.src_bunit", src_category as "Malware_Attacks.src_category", src_priority as "Malware_Attacks.src_priority", tag as "Malware_Attacks.tag", url as "Malware_Attacks.url", user as "Malware_Attacks.user", user_bunit as "Malware_Attacks.user_bunit", user_category as "Malware_Attacks.user_category", user_priority as "Malware_Attacks.user_priority", vendor_product as "Malware_Attacks.vendor_product" | fields + _time, host, source, sourcetype, "Malware_Attacks.dest_bunit", "Malware_Attacks.dest_category", "Malware_Attacks.dest_priority", "Malware_Attacks.dest_requires_av", "Malware_Attacks.file_hash", "Malware_Attacks.file_name", "Malware_Attacks.file_path", "Malware_Attacks.sender", "Malware_Attacks.src", "Malware_Attacks.src_bunit", "Malware_Attacks.src_category", "Malware_Attacks.src_priority", "Malware_Attacks.tag", "Malware_Attacks.url", "Malware_Attacks.user_bunit", "Malware_Attacks.user_category", "Malware_Attacks.user_priority", "Malware_Attacks.action", "Malware_Attacks.category", "Malware_Attacks.date", "Malware_Attacks.signature", "Malware_Attacks.dest", "Malware_Attacks.dest_nt_domain", "Malware_Attacks.user", "Malware_Attacks.vendor_product", "Malware_Attacks.is_Allowed_Malware", "Malware_Attacks.is_not_Allowed_Malware", "Malware_Attacks.is_Blocked_Malware", "Malware_Attacks.is_not_Blocked_Malware", "Malware_Attacks.is_Deferred_Malware", "Malware_Attacks.is_not_Deferred_Malware" | rename "Malware_Attacks.*" as "*" | noop| search abc

Do any of you know why this is the behavior, and why the macro in itself changes Splunks behavior.
Also if I expand the macros, my search works fine, and the noop is not added in the expanded search.
This is also the behavior in the search app, and not just the ES search.

0 Karma
1 Solution

DMohn
Motivator

As far as I know, the map_notable_fields makro is a legacy component, which is no longer neccessary in current versions of Enterprise Security. This is why in the macro definition it is defined as noop, which means 'no operation' - so basically it does nothing. noopcan be used for debug purposes though, you can check https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Noop for reference

You can remove the macro completely, it should not make any difference. Why it breaks your search if you use it before the filter part is something I do not completely understand. Maybe it mingles around with the search optimization (which is one of the noop debug features). Have you turned off the search optimization in your limits.conf? This maybe could explain the different behavior.

View solution in original post

0 Karma

DMohn
Motivator

As far as I know, the map_notable_fields makro is a legacy component, which is no longer neccessary in current versions of Enterprise Security. This is why in the macro definition it is defined as noop, which means 'no operation' - so basically it does nothing. noopcan be used for debug purposes though, you can check https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Noop for reference

You can remove the macro completely, it should not make any difference. Why it breaks your search if you use it before the filter part is something I do not completely understand. Maybe it mingles around with the search optimization (which is one of the noop debug features). Have you turned off the search optimization in your limits.conf? This maybe could explain the different behavior.

0 Karma

agneticdk
Path Finder

Ahh yes - I see it now. Thank you

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...