Why is the Splunk Add-on for F5 BIG-IP not separating sourcetypes as expected?

[jonesnadiam]

Has anyone had issues with the Splunk Add-on for F5 BIG-IP setting/separating the sourcetypes?
According to the documentation, if the sourcetype is set to f5:bigip:syslog, the data should be separated into its specific sourcetypes (f5:bigip:apm:syslog, f5:bigip:asm:syslog, f5:bigip:icontrol, etc), but all of our sourcetypes are still coming in as f5:bigip:syslog.

Is there anything specific that I need to change in the configuration files so that these sourcetypes are automatically updated?

Thanks.

[esmat777]

---

@jonesnadiam wrote:

Has anyone had issues with the Splunk Add-on for F5 BIG-IP setting/separating the sourcetypes?
According to the documentation, if the sourcetype is set to f5:bigip:syslog, the data should be separated into its specific sourcetypes (f5:bigip:apm:syslog, f5:bigip:asm:syslog, f5:bigip:icontrol, etc), but all of our sourcetypes are still coming in as f5:bigip:syslog.

Is there anything specific that I need to change in the configuration files so that these sourcetypes are automatically updated?

Thanks.

---

yes , i have fixed it by asking System administrator to change the 1st part of raw data from F5-Logging profile
to the one which match with "f5_asm" format

and it work
=====> but found another issue with Addon (( Addone -F5 BigIp) didnot tag ASM-logs

which will not be presented on Datamodels Or dashboards 

so i make a new files at local folder
props.conf 

Spoiler

### ASM ###
[f5:bigip:asm:syslog]
EVAL-attack_type = if(isnull(attack_type) or attack_type="" or attack_type="N/A" or attack_type="-", null, attack_type)
EVAL-category = if(isnull(attack_type) or attack_type="" or attack_type="N/A" or attack_type="-", null, attack_type)

and tags.conf

Spoiler

### ASM ###
[eventtype=f5_bigip_asm_syslog_attack]
web = enabled
communicate = enabled
network = enabled
attack = enabled
ids = enabled

[eventtype=f5_bigip_asm_syslog]
web = enabled
communicate = enabled
network = enabled

AND eventtypes.conf

Spoiler

[f5_bigip_asm_syslog]
search = sourcetype="f5:bigip:asm:syslog" (attack_type="N/A" OR NOT attack_type=*)

[f5_bigip_asm_syslog_attack]
search = sourcetype="f5:bigip:asm:syslog" (attack_type!="N/A" AND attack_type=*)

 and now everything is working fine and data are tagged

this should be added to addons in the next release 

[esmat777]

the same issue with F5-addons too

when i change logging profile for F5 v15.1 as below options

1. F5 Logging Profile (Syslog ) ==> addon F5-big IP not working as log come in below format not like
F5-addons format at props/transform files.

130>Sep 30 10:39:44 F5-01.*.com ASM:unit_hostname="F5-01.*.com"

and F5-add on match only below format

<131>Sep 12 23:53:50 F5-01.*.com ASM:f5_asm=Splunk-F5-AS

when i change logging profile from F5 v15.1 to pre-define template format call "Splunk"
2.  F5 Logging Profile (Splunk) ==> logs come in duplicated event parameters

when i change logging profile for F5 v15.1 to custom template at Splunk-Docs 

Configure F5 Logging Profiles for ASM

https://docs.splunk.com/Documentation/AddOns/released/F5BIGIP/Setup

3. 2.  F5 Logging Profile (Custom template) ==> logs come in duplicated event parameters too

[youngsuh]

We're having the same issue with the f5 add-on.  The problem is the add-on because if you don't use the add-on the event braking works with 8.2 just fine.  The sub sorucetype [f5:bigip:apm:syslog] works fine.  The root sourcetype [f5:bigip:syslog] isn't parsing correctly for TCP stream or Syslog format.  

BSD_format.txt
Event 1:
May 17 23:33:58 %masked_host% apmd[13742]: 01490115:5: /Common/fs.training.np2.navy.mil_modern:Common:cbd11dc1: Following rule 'CAC' from item 'Logon Authentication Type Switch' to terminalout 'CAC'

Event 2:
May 17 23:34:01 %masked_host% /Common/fs.training.np2.navy.mil_modern: Common:cbd11dc1: iRule access_policy_default CLIENTSSL_CLIENTCERT | DEBUG | Got 2 certs ||

We had our F5 SME provide two different format that configurable with the F5 appliance. 
Here is the line brake by the TA.
[f5:bigip:syslog]LINE_BREAKER = ([\r\n]+)(.*)(f5_irule|[^"]f5_asm|:\s(?:\d{4})[0-9A-Fa-f]{4}:\d+:\s+(?:[^:\s]{1,100}:[^:\s]{1,100}:\s*)?[0-9a-f]{8}:|( debug | info | warning | err | notice | alert | crit | emerg ))

Fails to parse the above two events.

Splunk_format.txt
Fails to line brake with for all the events.
[f5:bigip:syslog]LINE_BREAKER = ([\r\n]+)(.*)(f5_irule|[^"]f5_asm|:\s(?:\d{4})[0-9A-Fa-f]{4}:\d+:\s+(?:[^:\s]{1,100}:[^:\s]{1,100}:\s*)?[0-9a-f]{8}:|( debug | info | warning | err | notice | alert | crit | emerg ))

Were you able to make progress? 

[esmat777]

yes , i have fixed it by asking System administrator to change the 1st part of raw data from F5-Logging profile
to the one which match with "f5_asm" format

and it work
=====> but found another issue with Addon (( Addone -F5 BigIp) didnot tag ASM-logs

which will not be presented on Datamodels Or dashboards 

so i make a new files at local folder
props.conf

Spoiler

### ASM ###
[f5:bigip:asm:syslog]
EVAL-attack_type = if(isnull(attack_type) or attack_type="" or attack_type="N/A" or attack_type="-", null, attack_type)
EVAL-category = if(isnull(attack_type) or attack_type="" or attack_type="N/A" or attack_type="-", null, attack_type)

and tags.conf

Spoiler

### ASM ###
[eventtype=f5_bigip_asm_syslog_attack]
web = enabled
communicate = enabled
network = enabled
attack = enabled
ids = enabled

[eventtype=f5_bigip_asm_syslog]
web = enabled
communicate = enabled
network = enabled

AND eventtypes.conf

Spoiler

[f5_bigip_asm_syslog]
search = sourcetype="f5:bigip:asm:syslog" (attack_type="N/A" OR NOT attack_type=*)

[f5_bigip_asm_syslog_attack]
search = sourcetype="f5:bigip:asm:syslog" (attack_type!="N/A" AND attack_type=*)

 and now every thing is working fine and data are tagged

this should be added to addons in next release 

[youngsuh]

@esmat777 Could you explain in more which sourcetype worked after fix the f5_bigip_asm_syslog?  I don't see any difference between your prop.conf, eventtpe.conf and tags.  Are you saying you had the Splunk system admin to change the regex on f5:bigip:asm:syslog?  If yes, how are you are ingestion the F5 logs?  TCP or syslog?  Are you able to see the following sourcetypes: TRANSFORMS-sourcetype=f5_bigip-irule-default, f5_bigip-irule-http, f5_bigip-irule-dns-request, f5_bigip-irule-dns-response, f5_bigip-irule-lb-failed, f5_bigip-syslog-asm, f5-bigip-apm-syslog, f5_bigip-irule-exclude-audit, f5_bigip-secure, f5_bigip-ltm-ssl-error, f5_bigip-ltm-tcl-error, f5_bigip-ltm-traffic, f5_bigip-ltm-log-error.

Can you create idea of your fix so, that it could be included in the next release of the add-on?

[ianbow_concur]

You need to make sure the TA is installed on the indexer or heavy forwarder if you are using them. Due to the line breaks etc a Universal forwarder will not expand the events in the transforms to give you the expanded source types.

[youngsuh]

Yes.  This isn't the issue.  It's different beast entirely. 

[esmat777]

this is not an issue where to install the addon

it is related to a raw data format which is not compatible with addon

[esmat777]

yes , i have fixed it by asking System administrator to change the 1st part of raw data from F5-Logging profile
to the one which match with "f5_asm" format

and it work
=====> but found another issue with Addon (( Addone -F5 BigIp) didnot tag ASM-logs

which will not be presented on Datamodels Or dashboards 

so i make a new files at local folder
props.conf 

Spoiler

### ASM ###
[f5:bigip:asm:syslog]
EVAL-attack_type = if(isnull(attack_type) or attack_type="" or attack_type="N/A" or attack_type="-", null, attack_type)
EVAL-category = if(isnull(attack_type) or attack_type="" or attack_type="N/A" or attack_type="-", null, attack_type)

and tags.conf

Spoiler

### ASM ###
[eventtype=f5_bigip_asm_syslog_attack]
web = enabled
communicate = enabled
network = enabled
attack = enabled
ids = enabled

[eventtype=f5_bigip_asm_syslog]
web = enabled
communicate = enabled
network = enabled

AND eventtypes.conf

Spoiler

[f5_bigip_asm_syslog]
search = sourcetype="f5:bigip:asm:syslog" (attack_type="N/A" OR NOT attack_type=*)

[f5_bigip_asm_syslog_attack]
search = sourcetype="f5:bigip:asm:syslog" (attack_type!="N/A" AND attack_type=*)

 and now everything is working fine and data are tagged

this should be added to addons in the next release @jonesnadiam 

[wojtek_emca]

This could happened if F5 is not configured well (sending logs format).

Should be like this:
http://docs.splunk.com/Documentation/AddOns/released/F5BIGIP/Setup

Regards,
Wojtek

[hungpham]

Hmm i got same problem, splunk not separating sourcetypes.
I try follow http://docs.splunk.com/Documentation/AddOns/released/F5BIGIP/Setup but the splunk only receive syslog audit.

[sirajnp]

Hi,

Got same problem here. Add-on cannot extract fields for ASM audit events.

Sourcetype: f5:bigip:syslog
Port: udp:9514

Have you guys get through this issue?

[pvuong]

Hello,
Did you get any answer about this question ? I have the same problem for all my F5:big IP logs

[esmat777]

yes , i have fixed it by asking System administrator to change the 1st part of raw data from F5-Logging profile
to the one which match with "f5_asm" format

and it work
=====> but found another issue with Addon (( Addone -F5 BigIp) didnot tag ASM-logs

which will not be presented on Datamodels Or dashboards 

so i make a new files at local folder
props.conf 

Spoiler

### ASM ###
[f5:bigip:asm:syslog]
EVAL-attack_type = if(isnull(attack_type) or attack_type="" or attack_type="N/A" or attack_type="-", null, attack_type)
EVAL-category = if(isnull(attack_type) or attack_type="" or attack_type="N/A" or attack_type="-", null, attack_type)

and tags.conf

Spoiler

### ASM ###
[eventtype=f5_bigip_asm_syslog_attack]
web = enabled
communicate = enabled
network = enabled
attack = enabled
ids = enabled

[eventtype=f5_bigip_asm_syslog]
web = enabled
communicate = enabled
network = enabled

AND eventtypes.conf

Spoiler

[f5_bigip_asm_syslog]
search = sourcetype="f5:bigip:asm:syslog" (attack_type="N/A" OR NOT attack_type=*)

[f5_bigip_asm_syslog_attack]
search = sourcetype="f5:bigip:asm:syslog" (attack_type!="N/A" AND attack_type=*)

 and now everything is working fine and data are tagged

this should be added to addons in the next release  @pvuong  @jonesnadiam  @ianbow_concur  @sirajnp