sourcetype from regex ignored.

[cdherbigny]

Hi guys,

I'm having trouble configuring my splunk.

Indeed, i try to set sourcetype based on regex but, nothing works for me.
The only sourcetype i get is "csv".

My data comes from a file (csv.gz) and contains various type of log sources from various king of devices. I'd like to have for each type of event, a different sourcetype.

In order to do so, i edited the props.conf and transform.conf regarding to the sample found on etc/system/default.

For example: i try to have a sourcetype "Arkoon" for log coming from Arkoon Firewalls. Here's my transform and props configuration :

props.conf

[Arkoon]

EXTRACT-arkoon-IP = fw\=(?P<fwname>[^\s]+) aktype\=(?P<arkoon_type>[^\s]+) ip_log_type\=(?P<ip_log_type>[^\s]+) src\=(?P<src>[^\s]+) dst\=(?P<dst>[^\s]+) proto\=\"\"(?P<proto>[^\"]+)

EXTRACT-fwname-arkoon_type-alert_type-user-alert_level-alert_desc = fw\=(?P<fwname>[^\s]+) aktype\=(?P<arkoon_type>[^\s]+) alert_type\=\"\"(?P<alert_type>[^\"]+)\"\" user\=\"\"(?P<user>[^\"]*)\"\" alert_level\=\"\"(?P<alert_level>[^\"]+)\"\" alert_desc\=\"\"(?P<alert_desc>.+)\"\"\"$

transform.conf

[Arkoon-type]
REGEX = \sAKLOG\s
DEST_KEY = MetaData:Sourcetype
FORMAT = sourcetype::Arkoon

Here is a log sample that should match the regex:

02/14/14 00:00:00,x.x.x.x_General,x.x.x.x,16,6,"<134>Feb 13 22:59:59 x.x.x.x IP-Logs: AKLOG - id=firewall time=""2014-02-13 22:59:59"" gmtime=1392332399 fw=firewall-arkoon aktype=IP ip_log_type=NEWCONN src=x.x.x.x dst=x.x.x.x proto=""https"" protocol=6 port_src=56863 port_dest=443 intf_in=eth0:vr1 intf_out=eth2-6:vr12 pkt_len=52 nat=NO snat_addr=0 snat_port=0 dnat_addr=0 dnat_port=0 tcp_seq=1861073959 tcp_ack=0 tcp_flags=""SYN"" user="""" vpn-src="""" pri=6 rule=""default_rule"" action=ACCEPT"

Do i miss a file to edit ?

Any help would be appreciated !
Thanks

[kristian_kolb]

Ok, there are several issues here:

1) don't edit files in 'default' folders, edit in 'local' instead.

2) the file is called transforms.conf not transform.conf (but maybe that's just a typo)

3) in order to activate a transform, it has to be invoked from props.conf;

props.conf

[your host, source or sourcetype]
TRANSFORMS-blah = setArkoon

transforms.conf

[setArkoon]
your REGEX, FORMAT, and DEST_KEY here (which look ok, by the way)

Also, out of habit I always recommend that you use underscores instead of hyphens in names. Splunk is picky sometimes.

4) depending on how you get your data in, it might be possible to set the sourcetype in inputs.conf. You should look that up.

5) As you've noticed, you base your field extractions on sourcetype (which is A Good Thing), so basically, the format of the events should dictate what calls for a new sourcetype. Don't overdo it, e.g. create sourcetypes like arkoon_fw1, arkoon_fw2 etc, if the actual log format is the same.

Hope this helps,

/K