Cisco eStreamer eNcore Add-on for Splunk: App is grouping events -- Is this normal behavior?

[plarsenDST]

Cisco eStreamer eNcore is grouping events as seen in the indexer when searched. The old eStreamer client did not do this.

It this normal behavior for certain events grouped together.

Any help would be appreciated. Splunk V6.5.2 Cisco devices are V6

[tomasmoser]

Hi,

I have exactly same problem.

Data preview for a snippet of FMC logs (directly from files in /data directory) looks OK with following props.conf configuration. However, with the same configuration on indexer and real data it does not work correctly for 100%. Some data is grouped as stated before.

I even tried to use LINE_BREAKER as an other way to fix this. Didn't work at all.

Would you know where is the problem?

[cisco:firepower:estreamer:data]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = rec_type=\d+
TRUNCATE = 0
TIME_PREFIX = event_sec=
DATETIME_CONFIG =
MAX_TIMESTAMP_LOOKAHEAD = 10
NO_BINARY_CHECK = true
TIME_FORMAT = %s
pulldown_type = true
...

Tomas

[douglashurd]

We have only seen this when using the CLI as opposed to within our TA.

You’ll need a configuration change.

Required line is:

Sourcetype

[cisco:estreamer:data]
SHOULD_LINEMERGE = false

[plarsenDST]

Can you expand on this statement.

We have only seen this when using the CLI as opposed to within our TA.

On a forwarder I start the client with this.
/apps/splunk/etc/apps/TA-eStreamer/bin>nohup ./splencore.sh start &

TOP portion of props.conf, added the LINEMERGE line, it did not work. It still merges the line when a "burst" of attacks come from the same source IP, DEST IP and country. ie china, but different PORTS.

[cisco:estreamer:log]
EXTRACT-encore_log_fields = ^(?P\d+-\d+-\d+\s+\d+:\d+:\d+,\d+)\s+(?P[^ ]+)\s+(?P\w+)\s+(?P.+)
SHOULD_LINEMERGE = false

[cisco:estreamer]

[cisco:estreamer:data]

SHOULD_LINEMERGE = false
TRUNCATE = 0
TIME_PREFIX = event_sec=

[plarsenDST]

It is combining several log entries even with CR/LF at the end of the line into one event in Splunk. I believe it might be the Old Splunk eStreamer TA installed on the index server that is combining log lines and is not compatible with the new eStreamer client or eNcore as it is named in Splunkbase. I plan to remove the TA as a test.

A Splunk entry looks like this. Where it should break at every rec_type=

rec_type=400 app_proto=HTTP blocked=Yes class=web-application-attack class_desc="Web" client_app="Web browser" connection_id=52698 connection_sec=1497178529 dest_ip=00.00.00.00 dest_ip_country=0 dest_port=5002 event_id=197224 event_sec=1497178529 event_usec=449418 fw_policy=ZZ2 fw_rule=268xx7566 gid=1 http_response=0 ids_policy="Policy" iface_egress=7b91d684-4e7c-11e7-aa06-9b504fd054df iface_ingress=outside impact=0 impact_bits=0 impact_desc="Gray (unknown impact)" instance_id=4 ip_proto=TCP mpls_label=0 msg="SERVER-WEBAPP" num_ioc=0 priority=low rec_type_desc="Event" rec_type_simple="IPS EVENT" rev=1 sec_zone_egress=0000000-WebS sec_zone_ingress=0000000-Outside-Vlan777 security_context=00000000000000000000000000000000 sensor=00002-002 sid=69767 src_ip=10.##.##.7 src_ip_country=unknown src_port=61179 ssl_actual_action=Unknown ssl_flow_status=Unknown user="No Auth Required" vlan_id=2925 web_app=Unknown
rec_type=400 app_proto=HTTP blocked=No class=web-application-attack class_desc="Web" client_app="Web browser" connection_id=52698 connection_sec=1497178529 dest_ip=00.00.00.00 dest_ip_country=0 dest_port=5002 event_id=197225 event_sec=1497178529 event_usec=622822 fw_policy=ZZ2 fw_rule=2682xx7566 gid=1 http_response=200 ids_policy="Policy" iface_egress=7b91d684-4e7c-11e7-aa06-9b504fd054df iface_ingress=outside impact=0 impact_bits=0 impact_desc="Gray (unknown impact)" instance_id=4 ip_proto=TCP mpls_label=0 msg="SERVER-APACHE" num_ioc=0 priority=low rec_type_desc="Event" rec_type_simple="IPS EVENT" rev=5 sec_zone_egress=0000000-WebSrvr sec_zone_ingress=0000000-Outside-Vlan777 security_context=00000000000000000000000000000000 sensor=aafp2-cc2 sid=24648 src_ip=10.11.11.1 src_ip_country=unknown src_port=61179 ssl_actual_action=Unknown ssl_flow_status=Unknown user="No Auth Required" vlan_id=2925 web_app=Unknown

app =   HTTP    
eventtype = cisco_sourcefire    communicate network eventtype = cisco_sourcefire_ids_ips_event  attack ids
product =   Sourcefire IDS  
user =  No Auth Required    

[douglashurd]

Can you please better describe what you mean by grouping? Do you have a screenshot? Anything?