Hi,
I have added following configuration, but logs are not parsing properly. Do I need to change sourcetype?
[udp://X.X.X.X:514]
index = network
sourcetype= barracuda:waf
Yes, you need to change the source type on the UDP input to sourcetype = barracuda:log
per the readme.
What they've done is assume that all of the WAF data is being collected and aggregated into one log file/source.
Based upon that assumption, the Add-on splits that data back out into unique source types based upon regular expression matches.
Sample from the props.conf
file in the Add-on:
[barracuda:log] TRANSFORMS-force_sourcetype_barracuda =force_sourcetype_barracuda_web,force_sourcetype_barracuda_waf,force_sourcetype_barracuda_system,
force_sourcetype_barracuda_firewall,force_sourcetype_barracuda_audit,force_sourcetype_barracuda_system_bis, force_sourcetype_barracuda_system_mongotosqlite,force_sourcetype_barracuda_system_servermon
I'm also facing similar issues with this add-on. After changing the sourcetype to "barracuda:log" I'm not able to find any logs on "barracuda:*" sourcetype.
When I search the internal index, it shows the license_usage and metrics logs for firewall/web sourcetypes.
The only thing I suspect is negative avg_age and max_age in metrics logs.
Can you please help me to identify what I'm doing wrong here.
Event samples,
Sep 10 18:22:16 xxx.xxx.xxx.xxx Sep 11 01:22:16 ABCDXYZ 2020-09-10 22:22:16.494 +0000 ABCDXYZ TR xxx.xxx.xxx.xxx 443 xxx.xxx.xxx.xxx 2202 "-" "-" GET TLSv1.2 www.abcd.com HTTP/1.1 200 58111 943 SERVER DEFAULT PASSIVE VALID /ar/Home-standard/Kitchens-Secret xxx.xxx.xxx.xxx 2202 271
Sep 10 18:22:14 xxx.xxx.xxx.xxx Sep 11 01:22:14 ABCDXYZ 2020-09-10 22:22:14.029 +0000 ABCDXYZ WF ALER HEADER_VALUE_LENGTH_EXCEEDED xxx.xxx.xxx.xxx 63998 xxx.xxx.xxx.xxx 443 LOG NONE [length\="2691 bytes" header\="Referer" value\=" https\\://abcd.com/Serving/aasd.bs?cn\=brd&PluID\=0&Pos\=2342342344&EyeblasterID\=1080963447&clk\=1&sct\=1&dg\=1075209614&dgo\=1075209614&pc\=&sessionid\=19283719823&"] GET www.abbcd.com/ TLSv1.2 xxx.xxx.xxx.xxx 63998
[udp://<ip>:514]
connection_host = ip
sourcetype = barracuda:log
index = barracuda
At first glance:
Begin by checking what you can see in the barracuda index. The way you've defined the UDP input, you should find data tagged with the correct host=(ip address) and a source=udp::514, even if sourcetype=barracuda* is failing.
If you do an 'All time, Real time' search on the barracuda index and see the logs, it's likely the date being parsed is skewed outside of your typical search reporting window (most people use last 24 hours.)
If there's log data but the date and time is skewed, the REGEX extractions for the Add-on aren't a match for your log events. Possibly, the date/time on the Barracuda device itself is an issue. There's notes in the post about using https://regex101.com and a data sample to see what's matching. If the timezone of the barracuda device itself seems to be an issue, you can define a TZ setting in props.conf using source::udp514 as noted in the docs here: Specify time zones in props.conf.
Yes, you need to change the source type on the UDP input to sourcetype = barracuda:log
per the readme.
What they've done is assume that all of the WAF data is being collected and aggregated into one log file/source.
Based upon that assumption, the Add-on splits that data back out into unique source types based upon regular expression matches.
Sample from the props.conf
file in the Add-on:
[barracuda:log] TRANSFORMS-force_sourcetype_barracuda =force_sourcetype_barracuda_web,force_sourcetype_barracuda_waf,force_sourcetype_barracuda_system,
force_sourcetype_barracuda_firewall,force_sourcetype_barracuda_audit,force_sourcetype_barracuda_system_bis, force_sourcetype_barracuda_system_mongotosqlite,force_sourcetype_barracuda_system_servermon
Hi ekost,
Thanks for the information. I have added input sourcetype = barracuda:log and getting three different log source barracuda:system,barracuda:web and barracuda:waf but logs are not parsing properly like souce,destination and domain etc. fields are not populating.
If I understand you, the data is being ingested and is going to the index (network?) you've defined in the input stanza. The data is also being source typed: the index contains source types: barracuda:system, barracuda:web, and barracuda:waf. Is that correct?
If the above is correct, it means that the regex matches are successful at a base level.
Looking a bit deeper at the Add-on's props.conf
for the "barracuda:waf" source type, there are a number of additional fields:
[barracuda_waf_extractions_1]
REGEX = (WF)\s([A-Z]{4})\s(.+?)\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s(\d+)\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s(\d+)\s(DENY|LOG|WARNING)\s(NONE|LOCKED)\s\[.+?\]\s([A-Z]+)\s(.+?)\s(.+?)\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s(\d+)
FORMAT = log_category::$1 vendor_severity::$2 signature::$3 src_ip::$4 src_port::$5 dest_ip::$6 dest_port::$7 vendor_action::$8 vendor_follow_up_action::$9 http_method::$10 url::$11 vendor_protocol::$12 proxy_ip::$13 proxy_port::$14
They were also kind enough to provide a data sample as a reference. You can find the example in: /Splunk_TA_barracuda_waf_adc/samples/barracuda_waf.sample
When I've seen mismatched Add-on parsing for a given data source in the past, it is usually down to product version differences. The Add-on might have been written for an earlier or later version of the barracuda product than the one you're using. As the Add-on doesn't contain any version info, you can always contact the writer for clarification as well. The last line of the readme file has an address to contact.
If those fields are extracting properly, then there might not be any problem. You'd have to find a field that should've extracted from the source data that failed to, and review the regexes designed to capture that field. Grab an event from your index, clean it up a bit, grab the regex strings from the Add-ons, find an on-line regex tester, and do some quick checks! Note: I did this using the regex above with the sample data on regex101, and the resulting groupings were a match for the listed field values.
Hello Faisal,
I have developed this add-on. As pointed out by ekost, it might not be suitable for every product/version.
But feel free to post a sample event not correctly parsed, we'll check if there is an issue or a slight change to make.
Best regards,