Field extraction for FWSM and proper normalization...

[bjozsa]

I'm not an advanced splunk user...just a little regex here and there, but I use it for searching my log data for my lab testing devices. I recently downloaded the Cisco Security Suite and Cisco Firewalls apps a couple of days ago (v.2.0). For some reason they are not parsing out the logs correctly, but I may need to edit the regex and file extraction for my uses. My logs that I want to normalize into a common log format look like the following:

host.domain.com Feb 09 2013 21:33:39 HOSTNAME01 : %FWSM-5-106100: access-list VLAN20_IB permitted tcp VLAN20/X.X.X.X(39876) -> VLAN40/Y.Y.Y.Y(25) hit-cnt 1 (first hit) [0x16fc583b, 0x1b2ed17e]
host.domain.com Feb 09 2013 21:35:39 HOSTNAME01 : %FWSM-5-106100: access-list VLAN90_IB denied tcp VLAN90/X.X.X.X(39876) -> VLAN50/Y.Y.Y.Y(80) hit-cnt 1 (first hit) [0x16fc583b, 0x1b2ed17e]

I'm somewhat new to this (so please go easy). I've read countless things now, so much so, that my head is spinning. Does anyone have any pointers on how I can get these normalized into searchable fields. It would be nice to have fields that go like this:

fwsm_fqdn=host.domain.com, fwsm_timestamp=Feb 09 2013 21:33:39, fwsm_hostname=HOSTNAME01, fwsm_severity=5, fwsm_event-id=106100, fwsm_acl=VLAN20_IB, fwsm_action=permitted, fwsm_protocol=tcp, fwsm_src_int=VLAN20, fwsm_src=X.X.X.X, fwsm_src-port=39876, fwsm_dst_int=VLAN40, fwsm_dst=Y.Y.Y.Y, fwsm_dst-port=80

Is this possible, or does anyone know how I can go about doing this? I may be completely missing things here, but I think this can somehow be done...?

Thanks for any help!!

[bjozsa · Accepted solution]

Perfect! Got it to work now for ACL event ID 106100:

1. Create custom app directory:
2. $SPLUNK/etc/apps//
3. add the following to your ../local or ../default props.conf file (add following text)

   [fwsm]
   EXTRACT-fqdn_hostname = (?i)^(?:[^ ]* ){3}(?P[^ ]+)
   EXTRACT-fwsm_month = (?i)^(?:[^ ]* ){4}(?P[^ ]+)
   EXTRACT-fwsm_date = (?i)^(?:[^ ]* ){5}(?P[^ ]+)
   EXTRACT-fwsm_year = (?i)^(?:[^ ]* ){6}(?P[^ ]+)
   EXTRACT-fwsm_time = (?i)^(?:[^ ]* ){7}(?P[^ ]+)
   EXTRACT-fwsm_hostname = (?i)^(?:[^ ]* ){8}(?P[^ ]+)
   EXTRACT-fw_type = (?i)^[^%]%(?P[^-]+)
   EXTRACT-fwsm_sev = (?i)%FWSM-(?P[^-]+)
   EXTRACT-fwsm_event = (?i)^(?:[^-]-){2}(?P[^:]+)
   EXTRACT-syslog_type = (?i)^(?:[^:]:){6}\s+(?P[^ ]+)
   EXTRACT-fwsm_acl_name = (?i)-list (?P[^ ]+)
   EXTRACT-fwsm_action = (?i)IB (?P[^ ]+)
   EXTRACT-fwsm_protocol = (?i)ed (?P[^ ]+)
   EXTRACT-fwsm_src_int = (?i)-.? (?P\w+)\/
   EXTRACT-fwsm_src_ip = (?i)^[^/]/(?P[^(]+)
   EXTRACT-fwsm_src_port = (?i)-.?((?P\d+)(?=))
   EXTRACT-fwsm_dst_int = (?i)-> (?P[^/]+)
   EXTRACT-fwsm_dst_ip = (?i)^(?:[^/]/){2}(?P[^(]+)
   EXTRACT-fwsm_dst_port = (?i)^(?:[^(](){2}(?P[^)]+)
   EXTRACT-fwsm_hit_cnt = (?i)hit-cnt (?P.+?)\s+(
   EXTRACT-fwsm_src_pkt = (?i) .?) [(?P\d+\w+)(?=,)
   EXTRACT-fwsm_dst_pkt = (?i).?, (?P\d+\w+)(?=])
   

Then add the following to your ../local or ../default transform.conf file:

[force_sourcetype_for_cisco_fwsm]

DEST_KEY = MetaData:Sourcetype

REGEX = %FWSM-\d+-\d+

FORMAT = sourcetype::fwsm

...and there ya have it! Thanks for pointing me in the right direction! Now, I can easily search Splunk and create reports with the fields listed above. This is the first time I've seen anyone detail out an FWSM syslog event 106100, so I wanted to contribute back to the Spunk community. Happy Splunking.

[bjozsa]

ok, so that definitely helped point me in the right direction! i feel like i'm starting to get the hang of it.

that said, when i paste my regex values below into the search app, nothing comes up. do i still them in the props.conf like this, or does it need implemented in a long single line(?):

[custom_fwsm]
EXTRACT-fwsm106100
(?[^ ]+(.org|.net|.com|.cc|.tv|.biz))
(?(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\s[0-3][0-9]\s(19|20)\d\d)
((?([01]?\d|2[0-3])([0-5]?\d):([0-5]?\d):([0-5]?\d))\s(?[^ ]+))
((?FWSM)(-)(?[1-6])(-)(?\d+))
(?access-list)
(?(\bVLAN\B[1-9][0-9]_IB))
(?(\bpermitted\b|\bdenied\b))
(?(\btcp\b|\budp\b\bicmp\b))
(((?(\bVLAN\B[1-9][0-9]))/)(?(\b(?:\d{1,3}\.){3}\d{1,3}\b))(?(\((.*?)\)))(?(\s->\s\bVLAN\B[1-9][0-9]))/(?(\b(?:\d{1,3}\.){3}\d{1,3}\b))(?(\((.*?)\))))

(sorry for all of the questions....)

[martin_mueller]

Two things can go wrong there - either the expression isn't applied to your sourcetype, or the expression doesn't match your events. For the former, make sure the configured expression appears in the manager UI under the correct sourcetype. For the latter, either look for errors or keep simplifying the expression until you get at least some match, then add back complicatedness step by step.

[bjozsa]

yeah, i caught that too before adding to props.conf. i got errors with them being on separate lines, so i ended up combining them. didn't get errors with that method, but for some reason i'm not able to search for the items like i'm expecting either. it's like they don't exist.

[martin_mueller]

It looks like you missed the equals sign between your key (EXTRACT-foo) and the value (regex).

[martin_mueller]

Sounds like all you need is one big expression, along these lines:

^(?<fwsm_host>[^ ]+) (?<fwsm_timestamp>.{20}) (?<fwsm_hostname>[^ ]+) and so on

[martin_mueller]

This belong in props.conf, yes - http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

[bjozsa]

so do i then put that into the props.conf file? my confusion is, once the regex is extracting the information, what configuration file does the information go into, in order to create searchable fileds within the interactive search option (like fwsm_event in the search window would then bring back all of those events).