Solved: filter issues

a212830 · ‎12-02-2013

Hi,

I recently added a feed, which was working fine. It read a logfile and parsed out entries that I wanted based upon a field value. So far, so good. I now configured the feed to send additional data, in the same format, but it is not showing up in Splunk. I can't determine why - the data is in the logfile.

Here's some data that does appear:

1385989200000|533831|NormalizedCPUInfo|Utilization|43|CPU|ips-fil-dal1|CPU
1385989200000|533842|NormalizedCPUInfo|Utilization|0|CPU|ips-fil-hkprod001|CPU
1385989200000|133091|NormalizedCPUInfo|Utilization|0|CPU|FILDAL02.abc.com|CPU 2
1385989200000|132974|NormalizedCPUInfo|Utilization|0|CPU|FILMMK02.xyz.com|CPU 2
1385989200000|416841|NormalizedCPUInfo|Utilization|20|CPU|FILDLN01.abc.com|CPU 3
1385989200000|416842|NormalizedCPUInfo|Utilization|4|CPU|FILDLN01.xyz.com|CPU 4

Now, here's the new data, which isn't appearing:

Here's my transforms:

[setnull] 
REGEX = .
DEST_KEY = queue 
FORMAT = nullQueue

[setparsing] 
REGEX = ^(\w+\|){6}(FIL|[Ww][Cc]|[Ii][Pp][Ss]|[Ii][Vv][Ee])
DEST_KEY = queue
FORMAT = indexQueue

Any idea on why the new data isn't appearing?

Ayn · ‎12-02-2013

That's because your regex doesn't match the new events. I put your sample events and your regex and put them into a regex checker (in this case regexpal.com but any tool of your choice would do, including Splunk's own regex command). You need to rewrite your regex. What the final regex would look like is left as an exercise to the reader 😃 Because I don't know the full extent of your data so it would be dangerous to guess which events should and should not match.

EDIT: For what it's worth, the main problem I see with your regex with respect to your new data is that the new data contains a space character in the 6th field ("CBQoS ClassMap"). \w will not match this so you might want to consider changing "(\w+\|)" to "([\w\s]+\|)" in your regex.

View solution in original post

Ayn · ‎12-02-2013

That's because your regex doesn't match the new events. I put your sample events and your regex and put them into a regex checker (in this case regexpal.com but any tool of your choice would do, including Splunk's own regex command). You need to rewrite your regex. What the final regex would look like is left as an exercise to the reader 😃 Because I don't know the full extent of your data so it would be dangerous to guess which events should and should not match.

EDIT: For what it's worth, the main problem I see with your regex with respect to your new data is that the new data contains a space character in the 6th field ("CBQoS ClassMap"). \w will not match this so you might want to consider changing "(\w+\|)" to "([\w\s]+\|)" in your regex.

a212830 · ‎12-03-2013

Actually, never mind, it worked. Typo in the config. Thanks!

a212830 · ‎12-03-2013

Thanks. Unfortunately, that didn't do it. All the events listed above are data that I want to include. So, anything in field 7 that begins with FIL, IVE, IPS... should be sent to the indexer. I changed the regex that you have listed, and that opened the flood gates - including a whole bunch of stuff that I'm not interested in.

lguinn2 · ‎12-02-2013

Wow - that particular regex made my eyes cross. I couldn't figure out where it wasn't matching...

rsennett_splunk · ‎12-02-2013

Do you have that first "w" escaped like this: ^(\w+|) because it isn't showing up here as escaped and if so, in that first bit... you've got "match the lower case letter w one or more times and then a pipe, six times" and I think you really mean "match a word and a pipe six times and then start matching new stuff" right?

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!

lguinn2 · ‎12-02-2013

What's in props.conf that invokes these transforms?

filter issues

Introducing the 2024 SplunkTrust!

Introducing the 2024 Splunk MVPs!

Splunk Custom Visualizations App End of Life