recommendations for pruning away emptystring-valued fields?

[sideview]

In a funny way Im looking for the opposite of fillnull.

I have some fields which are sometimes coming through with emptystring values which creates empty columns in my table and I dont see a way to filter them out.

Reading the docs for eval, and testing this out, i can troubleshoot to verify that the conditional logic is working but i still get the empty columns showing up in tables.

<my search> | eval foo=if(len(foo)=0, null(), foo)

for that matter I would expect even this to effectively null out the field but it doesnt either.

<my search> | eval foo=if(len(foo)=0, some_nonexistent_field_name, foo)

Maybe there is no way to filter them out and the best practice is to ensure that they never get created in the first place?

[capnjosh]

" --your search that filters down to only the event types you want-- | fields + _raw | spath input=_raw | table *"

In my experience you have to filter out the event types you don't want right from the beginning. Otherwise, when you do table *, you will get any and all fields that showed up at any point in your piped search.

The key is to filter down to the specific event type as early as possible in your search pipes; and before you do any spath or field extractions.

For example, if I have a bunch of JSON events, and in order to filter them I have to do "* | spath input=myjsonfield | search Data.eventtype = 33 | table *" in order to show only events of eventtype=33, then that "table *" command will return all fields for all eventtypes. I think that's what is happening in the OP's case.

Note, if your sourcetype has a kv_mode=json in the props.conf, it will always return all the fields if you pipe to "table *".

[gkanapathy]

Can't you just use ... | fields - foo? Or is it the case that you don't know, but you just want to get rid of the column only if every value of it is null, in which case, I'm not sure how it came into existence in the first place?

[sideview]

Showing the column in tables doesn't bother me at all. it's the presence of null values for the keys that causes problems. I really do need some way in the search language of killing the keys entirely if they are null-valued. Any help appreciated.

[gkanapathy]

Well, okay, but if the value is non-null in any row, then you can't prevent the column from showing up in the table. But the value is still null/nonexistent. And it's been my (disappointed) experience that Splunk doesn't create empty fields with named regex field, but leaves them null.

[sideview]

They're coming from named capturing groups that are sometimes there and sometimes not. It seems splunk likes to create fields with emptystring values in this case. Hope that makes sense. I cant just whack them away with fields cause on some rows they will be defined.