Is there a way of ensuring every event has a specific field?

[Robbie1194]

Hi guys,

My goal is to use the map command look over all ip addresses and mac addresses that have had the value policy=policy. However, not every event has a mac address field. Is there a way of making ever event have a mac address field even if it's null? The map search has issues when there are some ips that don't have a corresponding mac address value.

I've tried using "| fillnull value="N/A" mac" just before the map command but it didn't work. Does anyone know of anything else I could do to ensure there is always a mac address field?

The search I'm using is:

index=index sourcetype=sourcetype policy="policy1" rule="rule1"
| dedup ip
| search NOT (ip="ip address" OR ip="ip address" OR ip="ip address")
| map search="search index=index sourcetype=sourcetype ip=$ip$ mac=$mac$"
| eval evalfield = if(policy="policy1" AND status="Match", "yes", "no")
| search evalfield ="no
| stats values(status) as status, values(rule) as rule, values(policy) as policy, values(evalfield) as evalfield, count by ip, mac

If anyone needs any further clarification, let me know.

Any help would be appreciated.

Cheers!

[micahkemp]

I think you should find a way to avoid using map. Not fully understanding what you're looking for, but trying to accomplish similar results I expect your search to yield, this may be a step towards what you're looking for:

index=index sourcetype=sourcetype NOT (ip="ip address" OR ip="ip address" OR ip="ip address") NOT (policy="policy1" AND status="Match")
| fillnull value="N/A" mac
| stats values(status) as status, values(rule) as rule, values(policy) as policy, values(evalfield) as evalfield, count by ip, mac
| search policy="policy1" rule="rule1"

[Robbie1194]

I'm not sure if this search will work but I may be getting a bit lost in my own head. The goal of the search is to:

• Search the index for all events matching policy1
• Take the IP and mac from each of these events and then re-search the same index for when this IP and associated mac (not always there thats why I needed to fillnull) have also matched on policy2.
• If they have matched, then yes, else no.
• Search yes or no (depending on what we need).

[somesoni2]

Try like this
Fixed map search, escaped double quotes

index=index sourcetype=sourcetype policy="policy1" rule="rule1" 
| search NOT (ip="ip address" OR ip="ip address" OR ip="ip address")
| dedup ip
| eval mac=coalesce(mac,"N/A")
| map search="search index=index sourcetype=sourcetype ip=$ip$ 
| eval mac=coalesce(mac,\"N/A\") | search mac=$mac$"
| eval evalfield = if(policy="policy1" AND status="Match", "yes", "no")
| search evalfield ="no"
| stats values(status) as status, values(rule) as rule, values(policy) as policy, values(evalfield) as evalfield, count by ip, mac

[Robbie1194]

I tried this but no results unfortunately.

Cheers though.

[somesoni2]

See if updated search works for you.

[Robbie1194]

I still get no results but I get this message in the job drop down:

"Unable to run query 'search index=fsctcenter sourcetype=fsctcenter_json ip="10.xx.x.xx" | eval mac=coalesce(mac,"N/A" | search mac="00:04:xx:xx:xx:xx"

The x's represent valid numbers

[Robbie1194]

I'm not really familiar with the coalesce function and couldn't really find a good description of it online, what does it do?

[micahkemp]

| fillnull value="N/A" mac should do what you're looking for, which you already tested. Can you run the search with that in place, but exclude the map and everything after it, and see what the results for that look like?

index=index sourcetype=sourcetype policy="policy1" rule="rule1" 
| dedup ip
| search NOT (ip="ip address" OR ip="ip address" OR ip="ip address")
| fillnull value="N/A" mac

[Robbie1194]

Yeah I tried that and it filled in the mac-less events giving 25 mac values to 25 ip values but when I run the above map search it shows that there are 9 ip addresses and 9 mac addresses, the event with the N/A value was not passed through.

I was wondering if there would be something better to use to accomplish this than map? But I'm not too sure.

[micahkemp]

| makeresults
| eval alwayspresent="alreadyhere"
| fillnull value="nolongernull" wasnull
| map search="| makeresults | eval stillpresent=\"$alwayspresent$\", wasnullpassedthrough=\"$wasnull$\""

This should show that the fillnull values make it through to the map command, but there is likely a better search to meet your needs.

Your search has index=index sourcetype=sourcetype in both the main search string and the map search string. Do both searches use the same index and sourcetype, or is the map search looking at different data?

[Robbie1194]

I get what you're doing with the makeresults search so I'll investigate my results further to see why they're not passing over.

My searches do look at the same index and sourcetypes. Is there a better way?

[jplumsdaine22]

remember map will only use 10 events by default