Line breaking - regex and capturing group

[armonsal]

Hello,
Need some help on regex here, am sure i maybe making mistake here but..
I don't undesrtand the problem in splunkd.log said:

(I had this line >11,000)

03-14-2014 17:11:49.108 -0300 ERROR LineBreakingProcessor - Line breaking regex has no capturing groups: \n - data_source="eStreamer", data_host="splunk.infocorp.cl", data_sourcetype="estreamer"

sample output of my events (This is a "estreamer" from Sourcefire) :

I capture this log whit app "estreamer" and i need to use this with ESS Splunk app

rec_type=112 rec_type_simple=POLICY event_sec=1394827933 policy_sensor="Defense Center" policy_event_id=122385 corr_policy="SNMP Y SYSLOG" corr_rule="DUMP TO SYSLOG" priority=0 description="" event_type="Intrusion Event" sensor=192.168.9.42 msg="" sid=26583 gid=1 tv_sec=1394827933 tv_usec=447702 event_id=67506 defined_mask=34059 impact=1 impact_bits=64 ip_proto=UDP net_protocol=0 src_ip=10.150.1.40 src_host_type=Host src_vlan_id=0 src_os_name=Unknown src_os_vendor=Unknown src_os_ver=Unknown src_criticality=None src_user=Unknown src_port=53005 src_app_proto=Unknown dest_ip=192.42.93.30 dest_host_type=Host dest_vlan_id=0 dest_os_name=Unknown dest_os_vendor=Unknown dest_os_ver=Unknown dest_criticality=None dest_user=Unknown dest_port=53 dest_app_proto=Unknown blocked=Yes iface_ingress=s1p5 iface_egress=s1p6 sec_zone_ingress="AS - Lan_Int" sec_zone_egress="AS - Lan_Ext"

rec_type=112 rec_type_simple=POLICY event_sec=1394827933 policy_sensor="Defense Center" policy_event_id=122386 corr_policy="SNMP Y SYSLOG" corr_rule="DUMP TO SYSLOG" priority=0 description="" event_type="Intrusion Event" sensor=192.168.9.42 msg="" sid=26583 gid=1 tv_sec=1394827932 tv_usec=974524 event_id=67505 defined_mask=34059 impact=1 impact_bits=65 ip_proto=UDP net_protocol=0 src_ip=192.168.9.190 src_host_type=Host src_vlan_id=0 src_os_name=Unknown src_os_vendor=Unknown src_os_ver=Unknown src_criticality=None src_user=Unknown src_port=62165 src_app_proto=Unknown dest_ip=192.58.128.30 dest_host_type=Host dest_vlan_id=0 dest_os_name=Unknown dest_os_vendor=Unknown dest_os_ver=Unknown dest_criticality=None dest_user=Unknown dest_port=53 dest_app_proto=Unknown blocked=Yes iface_ingress=s1p5 iface_egress=s1p6 sec_zone_ingress="AS - Lan_Int" sec_zone_egress="AS - Lan_Ext"

rec_type=400 rec_type_simple="IPS EVENT" event_sec=1394827935 event_usec=447631 sensor=192.168.9.42 event_id=59190 msg="BLACKLIST DNS request for known malware domain msnsolution.nicaze.net - Genome Trojan" sid=26583 gid=1 rev=1 class_desc="A Network Trojan was Detected" class=trojan-activity priority=high src_ip=10.150.1.40 dest_ip=192.35.51.30 src_port=58993 dest_port=53 ip_proto=UDP impact_bits=64 impact=1 blocked=Yes mpls_label=0 vlan_id=0 ids_policy=Agrosuper user=Unknown web_app=Unknown client_app=Unknown app_proto=Unknown fw_rule="Default Action" fw_policy=Agrosuper iface_ingress=s1p5 iface_egress=s1p6 sec_zone_ingress="AS - Lan_Int" sec_zone_egress="AS - Lan_Ext" connection_second=1394827935 connection_instance_id=1 connection_counter=36231 src_ip_country=unknown dest_ip_country="united states"

rec_type=2 rec_type_simple=PACKET event_sec=1394827935 sensor=192.168.9.42 event_id=59190 packet_sec=1394827935 packet_usec=447631 link_type=1 packet_len=82 packet=9f6223538fd4060052000000520000000010dbff207000181965a1bf0800450000444b6d00007a11f63c0a960128c023331ee6710035003035bb88dc000000010000000000000b6d736e736f6c7574696f6e066e6963617a65036e65740000010001

rec_type=112 rec_type_simple=POLICY event_sec=1394827935 policy_sensor="Defense Center" policy_event_id=122387 corr_policy="SNMP Y SYSLOG" corr_rule="DUMP TO SYSLOG" priority=0 description="" event_type="Intrusion Event" sensor=192.168.9.42 msg="" sid=26583 gid=1 tv_sec=1394827935 tv_usec=447631 event_id=59190 defined_mask=34059 impact=1 impact_bits=64 ip_proto=UDP net_protocol=0 src_ip=10.150.1.40 src_host_type=Host src_vlan_id=0 src_os_name=Unknown src_os_vendor=Unknown src_os_ver=Unknown src_criticality=None src_user=Unknown src_port=58993 src_app_proto=Unknown dest_ip=192.35.51.30 dest_host_type=Host dest_vlan_id=0 dest_os_name=Unknown dest_os_vendor=Unknown dest_os_ver=Unknown dest_criticality=None dest_user=Unknown dest_port=53 dest_app_proto=Unknown blocked=Yes iface_ingress=s1p5 iface_egress=s1p6 sec_zone_ingress="AS - Lan_Int" sec_zone_egress="AS - Lan_Ext"

rec_type=400 rec_type_simple="IPS EVENT" event_sec=1394827936 event_usec=38701 sensor=192.168.9.42 event_id=185263 msg=HI_CLIENT_DOUBLE_DECODE sid=2 gid=119 rev=1 class_desc="Not Suspicious Traffic" class=not-suspicious priority=low src_ip=192.168.1.229 dest_ip=200.143.16.5 src_port=58387 dest_port=80 ip_proto=TCP impact_bits=0 impact=0 blocked=No mpls_label=0 vlan_id=0 ids_policy=Agrosuper user=Unknown web_app=Unknown client_app=Unknown app_proto=Unknown fw_rule="Default Action" fw_policy=Agrosuper iface_ingress=s1p5 iface_egress=N/A sec_zone_ingress="AS - Lan_Int" sec_zone_egress=N/A connection_second=1394827930 connection_instance_id=3 connection_counter=1946 src_ip_country=unknown dest_ip_country=brazil

rec_type=2 rec_type_simple=PACKET event_sec=1394827936 sensor=192.168.9.42 event_id=185263 packet_sec=1394827936 packet_usec=38701 link_type=1 packet_len=986 packet=a06223532d970000da030000da0300000010dbff207000181965a1bf0800450003cc425440007f061ab6c0a801e5c88f1005e4130050ce86300e351b5db950184029e4040000474554202f7363726970742f3f64633d353535303030303231353b6f72643d3831363036393334393b737263747970653d6966726d3b636c69636b323d687474703a2f2f6164636c69636b2e672e646f75626c65636c69636b2e6e65742f61636c6b2532353346736125323533444c253235323661692532353344424b4574506e32496a55387a4a4d4d48666c6766517859434141746a5f2d714145414141414541456741446741574d6a6b6d7553614157435a38632d42684157434152646a59533177645749744e4445314e5451354e7a4d784e4455334e5441354d37494244486433647935305a584a795953356a624c6f424357646d634639706257466e5a63674243646f4248476830644841364c7939336433637564475679636d457559327776634739796447466b59532d5941764369424d414341754143414f6f43477a45794d5445765932777564475679636d4575614739745a5842685a325576614739745a666743676449656b414f63424a674434414f6f417748674241476742685925323532366e756d25323533443025323532367369672532353344414f4436345f334c2d586d427453724b347a497a36554539372d57646e4d705061672532353236636c69656e74253235334463612d7075622d343135353439373331343537353039332532353236616475726c253235334420485454502f312e310d0a4163636570743a20746578742f68746d6c2c206170706c69636174696f6e2f7868746d6c2b786d6c2c202a2f2a0d0a526566657265723a20687474703a2f2f7777772e74657272612e636c2f676c6f62616c5354415449432f61746d2f332f636f72652f5f74706c2f61646d616e616765722e68746d6c0d0a4163636570742d4c616e67756167653a2065732d434c0d0a557365722d4167656e743a204d6f7a696c6c612f352e302028636f6d70617469626c653b204d5349452031302e303b2057696e646f7773204e5420362e313b2054726964656e742f362e30290d0a4163636570742d456e636f64696e673a20677a69702c206465666c6174650d0a486f73743a20742e64796e61642e6e65740d0a444e543a20310d0a436f6e6e656374696f6e3a204b6565702d416c6976650d0a436f6f6b69653a207569643d313339323033343730353137362e64667474636b627230322e313339313533363437373233363b206463723d31322d353535303030303231352d343030303030313234342e302d353030303030303330322d312d313339333530323939303536360d0a0d0a

rec_type=112 rec_type_simple=POLICY event_sec=1394827936 policy_sensor="Defense Center" policy_event_id=122388 corr_policy="SNMP Y SYSLOG" corr_rule="DUMP TO SYSLOG" priority=0 description="" event_type="Intrusion Event" sensor=192.168.9.42 msg="" sid=2 gid=119 tv_sec=1394827936 tv_usec=38701 event_id=185263 defined_mask=34059 impact=0 impact_bits=0 ip_proto=TCP net_protocol=0 src_ip=192.168.1.229 src_host_type=Host src_vlan_id=0 src_os_name=Unknown src_os_vendor=Unknown src_os_ver=Unknown src_criticality=None src_user=Unknown src_port=58387 src_app_proto=Unknown dest_ip=200.143.16.5 dest_host_type=Host dest_vlan_id=0 dest_os_name=Unknown dest_os_vendor=Unknown dest_os_ver=Unknown dest_criticality=None dest_user=Unknown dest_port=80 dest_app_proto=Unknown blocked=No iface_ingress=s1p5 iface_egress=N/A sec_zone_ingress="AS - Lan_Int" sec_zone_egress=N/A

rec_type=400 rec_type_simple="IPS EVENT" event_sec=1394827939 event_usec=436017 sensor=192.168.9.42 event_id=59192 msg="BLACKLIST DNS request for known malware domain msnsolution.nicaze.net - Genome Trojan" sid=26583 gid=1 rev=1 class_desc="A Network Trojan was Detected" class=trojan-activity priority=high src_ip=192.168.9.190 dest_ip=192.5.5.241 src_port=60461 dest_port=53 ip_proto=UDP impact_bits=65 impact=1 blocked=Yes mpls_label=0 vlan_id=0 ids_policy=Agrosuper user=Unknown web_app=Unknown client_app=Unknown app_proto=Unknown fw_rule="Default Action" fw_policy=Agrosuper iface_ingress=s1p5 iface_egress=s1p6 sec_zone_ingress="AS - Lan_Int" sec_zone_egress="AS - Lan_Ext" connection_second=1394827939 connection_instance_id=1 connection_counter=36579 src_ip_country=unknown dest_ip_country="united states"

rec_type=2 rec_type_simple=PACKET event_sec=1394827939 sensor=192.168.9.42 event_id=59192 packet_sec=1394827939 packet_usec=436017 link_type=1 packet_len=82 packet=a362235331a7060052000000520000000010dbff207000181965a1bf08004500004413f000007f11975cc0a809bec00505f1ec2d003500301f050879000000010000000000000b6d736e736f6c7574696f6e066e6963617a65036e65740000010001

my props from "estreamer" app:

[source::eStreamer]
SHOULD_LINEMERGE = false
LINE_BREAKER = \n
TRUNCATE = 0
TIME_PREFIX = event_sec=

THANKS YOU!!!!

[lguinn2]

Splunk is complaining because of this line in your props.conf

LINE_BREAKER = \n

LINE_BREAKER must have a capture group as defined in the documentation "Index multi-line events" and props.conf.spec.

I suggest either

LINE_BREAKER=(\n+)

or

LINE_BREAKER=([\r\n]+)  # this is the default

Actually, if your data should be parsed as "one line per event", you can leave out the LINE_BREAKER in your props.conf. Then Splunk will use the default.