Why does filtering IPFIX data at the indexer level result in null events?

[jackhamm25]

I followed some other examples I found on Answers, but I'm having a little trouble getting filtering at the indexer to work. I have a device that sends SSL certificate IPFIX/Netflow data to my Splunk indexer (Splunk Add-on for IPFIX installed) and I want to trim out the data where those fields are null. Here's an example null event:

In splunk/etc/system/local I made these edits:

inputs.conf

[ipfix]
disabled = 0

props.conf

[source::ipfix]
TRANSFORMS-nullQ= nullFilter

transforms.conf

[nullFilter]
REGEX = sslCertificateIssuer=$
DEST_KEY=queue
FORMAT = nullQueue

The notion in my regex being that I want to exclude sslCertificateIssuer="" where there is nothing after the =.

I'm still seeing the null events after restart. I must be missing something about how this works, thoughts?

[DMohn · Accepted solution]

Your general settings are correct, but depending on the format of yout raw events the regex might be wrong. How are the events formatted?

You could try the following setting for transforms.conf

[nullFilter]
REGEX = (?m)sslCertificateIssuer=$
DEST_KEY=queue
FORMAT = nullQueue

This will make the regex multiline sensitive, and match the $for the end of each line, not the whole event.

[jackhamm25]

You got me thinking when you asked about the raw format. It turns out, ipfix comes in as hex and isn't converted until it passes the Splunk_TA_ipfix. From there, filtering on sslCertificateIssuer="" was actually sufficient! Thanks for the brain kick.

[hemendralodhi]

Hello,

I have Netflow data coming into device to my Splunk HF where it is converted and sent to Splunk indexer. I need to filter some fields which may be null or not. How can I do that?

E.g I need to completely filters fields below from indexing:

tcp_flag,fwd_status,src_tos

Will below configuration works? and do I have to perform the change at HF or Indexer level?

[netflow]
TRANSFORMS-nullQ= nullFilter

transforms

[nullFilter]
REGEX = tcp_flag|fwd_status|src_tos
DEST_KEY=queue
FORMAT = nullQueue

Thanks
Hemendra

[jackhamm25]

In case you didn't get an answer: I collect the Netflow directly on the indexer; so, the changes are made there. However, if you're detecting the netflow at an HF; you'd need to drop there as I understand it since that's going to be your pre-index point. Your regex might need the leading delimeter also (e.g. (?m) in case there are chars ahead of it (in my case a 😉