Getting Data In

How to edit my configuration to collect Windows event logs with a universal forwarder to send to a syslog collector?

oliverj
Communicator

Yes, this question has been asked a hundred times. I have looked at all of the examples, but my grasp of the different conf files and their interactions is lacking.

First:
I have a Windows device.
It has the Universal forwarder installed. (Version 6.3)
My destination device is a syslog server (TIBCO Loglogic, accepts standard syslog)

My config files are as follows: (This is the entire config, not snippits)
inputs.conf

[default]
host = $decideOnStartup
connection_host = "ip"

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 1

[WinEventLog://Security]
index = winevt
disabled = 0
current_only = 0

transforms.conf

[send_to_syslog]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = my_syslog_group

props.conf

[host::10*]
TRANSFORMS-mine = send_to_syslog

[source:*]
SEDCMD-rmlines=s/[\n\r\t]/ /g

outputs.conf

[tcpout:group1]
server=172.17.1.12:514
sendCookedData = false

[syslog:my_syslog_group]
server = 172.17.1.12:514
type = tcp
timestampformat = %b %e %H:%M:%S

The problems I am having:

  1. I was hoping for something much more simple. Just something in the outputs.conf: Winevent in, syslog out.
  2. I get a lot of junk information (it looks like splunk application info) with "INFO" or "WARN" that has nothing to do with Windows events.
  3. Most importantly: My Windows logs are broken into newlines! A single winevent takes 15 or so lines. My transforms.conf seems to do nothing, nor any of the other examples I have seen.

So yes, I am getting Windows logs as syslog, but the data is not usable to the end user due to the newlines.

Any help would be greatly appreciated!

0 Karma
1 Solution

oliverj
Communicator

Final update:
Use a heavy forwarder, not the universal forwarder.
UDP works out of the box, but I got TCP working with a props.conf edit.
You may need to expand the default message size limit for this.
output.conf

[tcpout]
 defaultGroup = default-autolb-group
 indexAndForward = 0
 useACK = true

 [tcpout:default-autolb-group]
 server = indexer.myindexer.com:9989

 [tcpout-server://indexer.myindexer.com:9989]

 [syslog]
 defaultGroup = syslogGroup

 [syslog:syslogGroup]
 server = indexer.myindexer.com:514
 type = udp
 maxEventSize = 8000

props.conf

[default]
SEDCMD-rmlines=s/[\n\r\t]/ /g
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_DATE = True
MAX_EVENTS = 256

I now have windows event logs coming from universal forwarder -> heavy forwarder -> syslog server as 1 entry per log, using TCP.

Thank you for your help, javiergn

View solution in original post

oliverj
Communicator

Final update:
Use a heavy forwarder, not the universal forwarder.
UDP works out of the box, but I got TCP working with a props.conf edit.
You may need to expand the default message size limit for this.
output.conf

[tcpout]
 defaultGroup = default-autolb-group
 indexAndForward = 0
 useACK = true

 [tcpout:default-autolb-group]
 server = indexer.myindexer.com:9989

 [tcpout-server://indexer.myindexer.com:9989]

 [syslog]
 defaultGroup = syslogGroup

 [syslog:syslogGroup]
 server = indexer.myindexer.com:514
 type = udp
 maxEventSize = 8000

props.conf

[default]
SEDCMD-rmlines=s/[\n\r\t]/ /g
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_DATE = True
MAX_EVENTS = 256

I now have windows event logs coming from universal forwarder -> heavy forwarder -> syslog server as 1 entry per log, using TCP.

Thank you for your help, javiergn

javiergn
SplunkTrust
SplunkTrust

Why don't you use the Windows Snare agent instead of the universal forwarder? It'll read your event logs and send them over in Syslog format to your Syslog server. There's a free version by the way.

See the following links:

https://www.intersectalliance.com/our-product/snare-agent/

https://answers.splunk.com/search.html?f=&redirect=search%2Fsearch&sort=relevance&q=Snare&type=quest...

Hope that helps.

Thanks,
J

oliverj
Communicator

Found it! Mostly.

First: Although Universal Forwarder will output to Syslog, it is breaking up my windows logs into individual lines.
I tried the heavy forwarder instead, with this output.conf file (All others are default, nothing special in props or transforms):

[tcpout]
defaultGroup = default-autolb-group
indexAndForward = 0
useACK = true

[tcpout:default-autolb-group]
server = indexer.myindexer.com:9989

[tcpout-server://indexer.myindexer.com:9989]

[syslog]
defaultGroup = syslogGroup

[syslog:syslogGroup]
server = indexer.myindexer.com:514
type = udp
maxEventSize = 8000

When I do this, all events sent to my heavy forwarder are now forwarded out to my syslog server, and they are 1 event per line. The maxEventSize was manually set because windows logs are over the default of (1000?). Careful here, because UDP packets are limited by MTU size. If your MTU isn't large enough to fit the message, it will most likely be truncated.

So, I hate UDP. But when I switch to type = TCP, everything switched back to one event per line!
Can anyone offer insight on that? Leaving everything else the same, just switching type = tcp causes this.

0 Karma

javiergn
SplunkTrust
SplunkTrust
0 Karma

oliverj
Communicator

That is an excellent plan, but in this environment getting new software approved takes months.
We are migrating off of LogLogic and on to SPLUNK -- the Windows EVT -> Syslog is only an interim solution, while we set up the SPLUNK servers and start migrating over.
Keeping the same forwarding software will make things much easier for all parties involved. Something like Snare would do the trick, but would be adding a 3rd piece of software into the mix, when I am trying to eliminate new applications and simplify deployments for the end users.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...