Splunk Search

recommendations for pruning away emptystring-valued fields?

sideview
SplunkTrust
SplunkTrust

In a funny way Im looking for the opposite of fillnull.

I have some fields which are sometimes coming through with emptystring values which creates empty columns in my table and I dont see a way to filter them out.

Reading the docs for eval, and testing this out, i can troubleshoot to verify that the conditional logic is working but i still get the empty columns showing up in tables.

<my search> | eval foo=if(len(foo)=0, null(), foo)

for that matter I would expect even this to effectively null out the field but it doesnt either.

<my search> | eval foo=if(len(foo)=0, some_nonexistent_field_name, foo)

Maybe there is no way to filter them out and the best practice is to ensure that they never get created in the first place?

0 Karma

capnjosh
Explorer

" --your search that filters down to only the event types you want-- | fields + _raw | spath input=_raw | table *"

In my experience you have to filter out the event types you don't want right from the beginning. Otherwise, when you do table *, you will get any and all fields that showed up at any point in your piped search.

The key is to filter down to the specific event type as early as possible in your search pipes; and before you do any spath or field extractions.

For example, if I have a bunch of JSON events, and in order to filter them I have to do "* | spath input=myjsonfield | search Data.eventtype = 33 | table *" in order to show only events of eventtype=33, then that "table *" command will return all fields for all eventtypes. I think that's what is happening in the OP's case.

Note, if your sourcetype has a kv_mode=json in the props.conf, it will always return all the fields if you pipe to "table *".

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Can't you just use ... | fields - foo? Or is it the case that you don't know, but you just want to get rid of the column only if every value of it is null, in which case, I'm not sure how it came into existence in the first place?

sideview
SplunkTrust
SplunkTrust

Showing the column in tables doesn't bother me at all. it's the presence of null values for the keys that causes problems. I really do need some way in the search language of killing the keys entirely if they are null-valued. Any help appreciated.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Well, okay, but if the value is non-null in any row, then you can't prevent the column from showing up in the table. But the value is still null/nonexistent. And it's been my (disappointed) experience that Splunk doesn't create empty fields with named regex field, but leaves them null.

0 Karma

sideview
SplunkTrust
SplunkTrust

They're coming from named capturing groups that are sometimes there and sometimes not. It seems splunk likes to create fields with emptystring values in this case. Hope that makes sense. I cant just whack them away with fields cause on some rows they will be defined.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...