Why are my udp syslog input events getting merged?

[Chris_R_]

Odd behaviour with some udp syslog input from a Panorama device (palo alto management device) and ArcSight connector using the same udp input port. Input settings as follows

[udp://515]
disabled = false
sourcetype = threat_events
index = myindex

props.conf
[sourcetype::threat_events]
TZ = UTC
SHOULD_LINEMERGE = False

Linemerge settings were set to false for the sourcetype as seen above, however events merged at together sometimes. Is this an example of getting the right LINE_BREAKER= correct for the device?

[jrodman · Accepted solution]

Use a stanza name:

[threat_events]

not

[sourcetype::threat_events]

We don't have any support for that syntax, and it shouldn't be in the docs anywhere (I can see how you might think there would be, of course).

Thus you should have:

[threat_events]
TZ = UTC
SHOULD_LINEMERGE = False 

[Chris_R_]

They were getting merged events where a arcsight event would come through and multiple panorama event (starting with CEF) would get merged one after another ( client info removed )

Mar 4 18:47:25 somehost.com Mar 4 18:48:06 1,2010/03/048:48:06,0004A100609,THREAT,url,46,2010/03/04 18:48:05,10.170.133.122,82.195.186.201,0.0.0.0,0.0.0.0,ProxyAccess-A2,,,web-browsing,xtxs1,Int-FW,Int-FW-Proxy blah....de....blah....informational,0 (<---first event should end here)
CEF:0|Palo Alto|Panorama|||THREAT|Unknown| eventId=1428238 proto=UDP art=1267728483290 rt=1267728481000 shost=somehost.com src=10.97.3.55 sourceZoneURI=/zzz Zones/System Zones/Private Address Space dst=22.11.22.33 blah ....de...blah...dtz=Asia/xyz deviceFacility=IPS (<--- Second event should end here)
CEF:0|Palo Alto|Panorama (<---they continued to get multiple CEF Panorama device events all merged with the above)

[gkanapathy]

look to me then that SHOULD_LINEMERGE = false wasn't taking effect. (And therefore default BREAK_ONLY_BEFORE_DATE = true was in effect.) I always use lower-case false instead of upper-case False. I have no idea if that makes a difference.

[gkanapathy]

Probably. LINE_BREAKER by default is ([\r\n]+), that is, any sequence of newlines and carriage returns. In addition, the end of a UDP packet will also end an event. It's not clear to my why you would have merged events in this case. Was there any pattern or commonality to the merged events?

[Chris_R_]

One workaround was to use separate network ports for the different devices, curious to hear the answer though.