Hi,
I recently added a feed, which was working fine. It read a logfile and parsed out entries that I wanted based upon a field value. So far, so good. I now configured the feed to send additional data, in the same format, but it is not showing up in Splunk. I can't determine why - the data is in the logfile.
Here's some data that does appear:
1385989200000|533831|NormalizedCPUInfo|Utilization|43|CPU|ips-fil-dal1|CPU
1385989200000|533842|NormalizedCPUInfo|Utilization|0|CPU|ips-fil-hkprod001|CPU
1385989200000|133091|NormalizedCPUInfo|Utilization|0|CPU|FILDAL02.abc.com|CPU 2
1385989200000|132974|NormalizedCPUInfo|Utilization|0|CPU|FILMMK02.xyz.com|CPU 2
1385989200000|416841|NormalizedCPUInfo|Utilization|20|CPU|FILDLN01.abc.com|CPU 3
1385989200000|416842|NormalizedCPUInfo|Utilization|4|CPU|FILDLN01.xyz.com|CPU 4
Now, here's the new data, which isn't appearing:
1385989500000|245596|NormalizedQosClassMapInfo|QoSMechanismDiscardedBits|0|CBQoS ClassMap|FILHIL02.abc.com|cbQoS Gi0/0/1-LAN_IN-SIG-IN
1385989500000|245596|NormalizedQosClassMapInfo|PrePolicyPackets|29319|CBQoS ClassMap|FILHIL02.abc.com|cbQoS Gi0/0/1-LAN_IN-SIG-IN
1385989500000|245596|NormalizedQosClassMapInfo|PctDiscard|0|CBQoS ClassMap|FILHIL02.abc.com|cbQoS Gi0/0/1-LAN_IN-SIG-IN
1385989500000|245596|NormalizedQosClassMapInfo|PrePolicyBytes|16389712|CBQoS ClassMap|FILHIL02.abc.com|cbQoS Gi0/0/1-LAN_IN-SIG-IN
1385989500000|245596|NormalizedQosClassMapInfo|PostPolicyBits|131117696|CBQoS ClassMap|FILHIL02.abc.com|cbQoS Gi0/0/1-LAN_IN-SIG-IN
1385989500000|245596|NormalizedQosClassMapInfo|PctQoSMechanismDiscard|0|CBQoS ClassMap|FILHIL02.abc.com|cbQoS Gi0/0/1-LAN_IN-SIG-IN
1385989500000|245596|NormalizedQosClassMapInfo|CongestionDiscardedPackets|0|CBQoS ClassMap|FILHIL02.abc.com|cbQoS Gi0/0/1-LAN_IN-SIG-IN
1385989500000|245596|NormalizedQosClassMapInfo|QoSMechanismDiscardedBytes|0|CBQoS ClassMap|FILHIL02.abc.com|cbQoS Gi0/0/1-LAN_IN-SIG-IN
Here's my transforms:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = ^(\w+\|){6}(FIL|[Ww][Cc]|[Ii][Pp][Ss]|[Ii][Vv][Ee])
DEST_KEY = queue
FORMAT = indexQueue
Any idea on why the new data isn't appearing?
That's because your regex doesn't match the new events. I put your sample events and your regex and put them into a regex checker (in this case regexpal.com but any tool of your choice would do, including Splunk's own regex command). You need to rewrite your regex. What the final regex would look like is left as an exercise to the reader 😃 Because I don't know the full extent of your data so it would be dangerous to guess which events should and should not match.
EDIT: For what it's worth, the main problem I see with your regex with respect to your new data is that the new data contains a space character in the 6th field ("CBQoS ClassMap"). \w
will not match this so you might want to consider changing "(\w+\|)
" to "([\w\s]+\|)
" in your regex.
That's because your regex doesn't match the new events. I put your sample events and your regex and put them into a regex checker (in this case regexpal.com but any tool of your choice would do, including Splunk's own regex command). You need to rewrite your regex. What the final regex would look like is left as an exercise to the reader 😃 Because I don't know the full extent of your data so it would be dangerous to guess which events should and should not match.
EDIT: For what it's worth, the main problem I see with your regex with respect to your new data is that the new data contains a space character in the 6th field ("CBQoS ClassMap"). \w
will not match this so you might want to consider changing "(\w+\|)
" to "([\w\s]+\|)
" in your regex.
Actually, never mind, it worked. Typo in the config. Thanks!
Thanks. Unfortunately, that didn't do it. All the events listed above are data that I want to include. So, anything in field 7 that begins with FIL, IVE, IPS... should be sent to the indexer. I changed the regex that you have listed, and that opened the flood gates - including a whole bunch of stuff that I'm not interested in.
Wow - that particular regex made my eyes cross. I couldn't figure out where it wasn't matching...
Do you have that first "w" escaped like this: ^(\w+|) because it isn't showing up here as escaped and if so, in that first bit... you've got "match the lower case letter w one or more times and then a pipe, six times" and I think you really mean "match a word and a pipe six times and then start matching new stuff" right?
What's in props.conf
that invokes these transforms?