How to split multiline event on output

[paulquinonero]

Hi,

I have windows XML logs in input of my Heavy Forwarder (via the universal forwarder with the TA_windows).

When I send this event through Syslog I can see that some events split due to the carriage return.

Exemple:

<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Microsoft-Windows-Security-Auditing' Guid='{54849625-5478-4994-A5BA-3E3B0328C30D}'/><EventID>4672</EventID><Version>0</Version><Level>0</Level><Task>12548</Task><Opcode>0</Opcode><Keywords>0x8020000000000000</Keywords><TimeCreated SystemTime='2020-03-10T16:15:47.000184600Z'/><EventRecordID>157445</EventRecordID><Correlation/><Execution ProcessID='516' ThreadID='2448'/><Channel>Security</Channel><Computer>MININT-5B0409J.test.lan</Computer><Security/></System><EventData><Data Name='SubjectUserSid'>NT AUTHORITY\SYSTEM</Data><Data Name='SubjectUserName'>MININT-5B0409J$</Data><Data Name='SubjectDomainName'>TEST</Data><Data Name='SubjectLogonId'>0x1bda93a</Data><Data Name='PrivilegeList'>SeSecurityPrivilege
                SeBackupPrivilege
                SeRestorePrivilege
                SeTakeOwnershipPrivilege
                SeSystemEnvironmentPrivilege
                SeLoadDriverPrivilege
                SeImpersonatePrivilege
                SeDelegateSessionUserImpersonatePrivilege
                SeDebugPrivilege
                SeEnableDelegationPrivilege</Data></EventData></Event>

My syslog message looks like this:

<13> MININT-5B0409J <Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Microsoft-Windows-Security-Auditing' Guid='{54849625-5478-4994-A5BA-3E3B0328C30D}'/><EventID>4672</EventID><Version>0</Version><Level>0</Level><Task>12548</Task><Opcode>0</Opcode><Keywords>0x8020000000000000</Keywords><TimeCreated SystemTime='2020-03-10T16:15:47.000184600Z'/><EventRecordID>157445</EventRecordID><Correlation/><Execution ProcessID='516' ThreadID='2448'/><Channel>Security</Channel><Computer>MININT-5B0409J.test.lan</Computer><Security/></System><EventData><Data Name='SubjectUserSid'>NT AUTHORITY\SYSTEM</Data><Data Name='SubjectUserName'>MININT-5B0409J$</Data><Data Name='SubjectDomainName'>TEST</Data><Data Name='SubjectLogonId'>0x1bda93a</Data><Data Name='PrivilegeList'>SeSecurityPrivilege

Here is the conf I use:

props.conf

[host::MININT*]
TRANSFORMS-orano = windows-compagny
MAX_TIMESTAMP_LOOKAHEAD = 16
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = <Event xmlns
LINE_BREAKER = ([\r\n]+)(?=<Event xmlns)

transforms.conf

[windows-compagny]
REGEX = .
#REGEX = <Event ((\S|\s)*?)<\/Event>
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslog_data

outputs.conf

[syslog:syslog_data]
maxEventSize = 9999999
server = 192.168.1.10:515
type = tcp

If someone got an idea.

Thanks in advance.

[FrankVl]

So, you are using a UF to collect windows logs locally on a server and forward them using normal splunktcp to a HF and from there you want to forward them as syslog?

How are you determining what the syslog events look like? Are you checking that on the receiving server, or using tcpdump or so on the HF to see what it is actually sending?

My experience is that Splunk will nicely send 1 event as 1 syslog message (as in: only prepend the syslog header 1x per event, not for each line). The problem is that syslog is not really designed for multiline logs, so likely the receiving syslog server will split it line by line. So you'll likely need to look at tuning the receiving side to handle these multiline logs better. But whether your syslog server is capable of doing that is quite a question.

One alternative would be to configure a clone sourcetype TRANSFORMS on the HF (assuming you also want this data to flow to splunk without modification), and on the cloned sourcetype apply a SEDCMD that strips out the newlines (or replaces them with some specific character) and then send that data to syslog output.

[paulquinonero]

Yes, between my UF and my HF I use splunkTcp.
I use a network tap to get the data between my HF and my syslog relay. I check all the doc on outputs.conf and I think I realy need to use intermediate forwarder to proper send data to my indexer.

[FrankVl]

Can you show what multiple consequtive syslog messages looks like when there are multiline events involved?

Also: what do you mean by "I think I realy need to use intermediate forwarder to proper send data to my indexer."? How is that relevant to understanding what happens in syslog forwarding?

[paulquinonero]

I think my problem with syslog generated message is due to the RFC itself. it tell that the syslog will generate a new message when a carriage return is find.

If I use a splunktcp transmission between my HF and my indexer I fix the problem.

[FrankVl]

Well yes, using splunktcp between HF and Indexer would indeed be much better. Not sure why you would ever want to use syslog for a splunk-to-splunk connection.

[mydog8it]

If I understand your question, this is a syslog-ng configuration question. "Why is syslog-ng splitting my events that I forward from Splunk?"
Am I understanding the issue correctly?

[paulquinonero]

No, I use a universal forwarder to send data from windows to my Heavy Forwarder. but when I send these events through syslog (in the options.conf) the message is splited on the carriage return, and i don't know if the event is already splited in splunk on only when it's change in a syslog message.