Getting Data In

How do I collect Windows events with syslog-NG and then send them to Splunk?

Log_wrangler
Builder

I have read that syslog-ng is a good way to aggregate syslog data prior to sending to Splunk, but does anyone care to share experiences using a universal forwarder (on a Windows machine) to send to syslog-ng and then send to Splunk?

Is that a good idea or bad?

Thank you

Tags (2)
0 Karma
1 Solution

FrankVl
Ultra Champion

Syslog-NG is a good tool to aggregate syslog data before sending it to Splunk. As also outlined in @adonio his comment.

There is (highly esoteric use cases excluded) no good reason to send data collected by a UF on a windows system to a syslog daemon first, before sending it to Splunk.

View solution in original post

hendrick
New Member

You might consider syslog-ng collecting Windows event logs agentless, then sending them directly to splunk with the splunk_hec() destination. Handles fail-over and load-balancing to multiple HEC natively. I have heard of an organization doing ~10TB per day through a pair of log servers balancing across ~120 HECs.

Splunk is cool and all, but this provides a log "layer" to handle collection, filtering, parsing, rewriting of your logs with the flexibility to send to lots of destinations including splunk in whatever format works best for that destination. And fewer endpoint agents is not a bad thing either.

my opinion

0 Karma

hendrick
New Member

Have you looked at the Windows Event Collector from syslog-ng? Collects without installing agents on your windows end.
Hook that up with the new splunk_hec() destination and you might be happily surprised.

0 Karma

FrankVl
Ultra Champion

Syslog-NG is a good tool to aggregate syslog data before sending it to Splunk. As also outlined in @adonio his comment.

There is (highly esoteric use cases excluded) no good reason to send data collected by a UF on a windows system to a syslog daemon first, before sending it to Splunk.

Log_wrangler
Builder

Thank you, for the second opinion and confirmation to Adonio.

I agree sending winevents to a syslog server would create unnecessary complication.

Perhaps you have a solution that would work with my use case provided above. If only the UF could send to s3...

Thank you.

0 Karma

adonio
Ultra Champion

what is your use case? any need to send to syslog and not directly to Splunk?
most of the times, the syslog layer is very useful for data sources you cant install forwarders on, like appliances, firewalls, routers, switches etc. most of these devices also support the syslog message format. another benefit will be that a single forwarder can read many files from many data sources that now are hierarchically organized by the syslog server and apply the relevant sourcetypes. it can also now auto load balance to many indexers compare to point a data source via port directly to a single indexer.
i cant recall using a syslog receiver for windows data, but maybe you have use case that requires that.

0 Karma

hendrick
New Member

Updating this for end 2019 - I agree if you want to install a UF on all Windows systems. I wanted to call attention to the syslog-ng Windows Event Forwarder service (wef) that establishes a publisher/subscriber relationship with windows servers so that it collects logs with no agent installed on the Windows side. This has been out about a year and I have tested it through a load balancer to provide fail-over and scalability (although updates now make even this optional as most of those features are built in).
https://www.syslog-ng.com/community/b/blog/posts/how-to-collect-windows-event-logs-with-syslog-ng-wi...

From the syslog-ng system, you then can use the HTTP Event Collector (HEC) destination to natively send to splunk. I like both tools, but I see them fitting in different spaces. syslog-ng for collection, parsing and log routing and splunk as an excellent search tool.
https://www.splunk.com/en_us/blog/tips-and-tricks/syslog-ng-and-hec-scalable-aggregated-data-collect...
https://www.syslog-ng.com/community/b/blog/posts/splunk-hec-sending-logs-using-program-destination-s...
Best,
Jim

0 Karma

Log_wrangler
Builder

Thank you for the repy Adonio.

I was thinking of using syslogNG for "windows" as we are planning to use it for a number of syslog-data sources well. But I agree syslogNG might not be the best solution for a windows events scenario, i.e. security, application, system.

The problem I am facing is that I need to send winevents from UFs (on hosts) to the indexers cooked and to a 3rd party application uncooked/raw.

I have successfully tested sending to both from a HF and it works for a while, but then the HF starts dropping events to the indexers.

11-09-2018 13:06:27.418 +0000 WARN TcpOutputProc - Queue for group INDEXERS_cooked has begun dropping events 
11-09-2018 13:07:03.547 +0000 INFO TcpOutputProc - Queue for group INDEXERS_cooked has stopped dropping events 

Per Splunk Support, I have tried changing dropClonedEventsOnQueue/dropClonedEventsOnQueueFull, but no luck... I still see dropping events. Ultimately, support says that "cloned events" is not a supported feature.

When I comment out the stanzas to send uncooked to the 3rd party, I notice that there are a number of blocked events to the indexers.

11-12-2018 14:12:20.034 +0000 WARN TcpOutputProc - Forwarding to indexer group lb blocked for 90 seconds. 
11-12-2018 14:12:30.054 +0000 WARN TcpOutputProc - Forwarding to indexer group lb blocked for 100 seconds.

It appears when the windows machines (approx 45) send high volume spikes, the HF can handle it but the indexers cannot.

So the primary use case is to get a collection server to aggregate winevents for: 1) to create a buffer for the indexers and 2) has the ability to send raw data to a 3rd party application and or s3....

we plan to ingest >10,000 of endpoint logs...

Any suggestions in this scenario?

Thank you

0 Karma

FrankVl
Ultra Champion

May I suggest that you post this as a new question and perhaps have this particular question closed? As this is a completely different discussion than "can I use syslog-ng for windows logs".

To already give a short response to your question: This sort of use case is typically what HFs are used for. I can think of several things to alleviate the throughput issues you are running into:
- make sure you spread the load over all your indexers (this may require adding additional HFs and making sure the data sources in front of those HFs balance nicely across them). Perhaps add some extra indexers.
- increase the output queue size on the HFs, to better cope with spikes (not ideal as it causes delays, but if you have particularly bursty traffic it may be the only solution apart from sizing your entire setup based on the biggest spikes).

0 Karma

Log_wrangler
Builder

@ FrankVI
Agreed the latest response from Gergely should be addressed as a separate topic.
As far as your recommendations: we are load balancing indexers and we are going to add another HF to the mix (but we feel that is just a temporary fix as we will probably overwhelm multiple HFs). Currently we are only sending winevents to 1 HF and then to 3 IDXs... Per Splunk support and reviewing diags, the dropping issue is based on the cloning of events, so that is why we are considering syslogNG or a WEC or some other winevent collection server... that we could use to send to two places.

I will accept your answer as I do agree with what you are saying...

Thanks

0 Karma

gergely_bodnar
New Member

You have mentioned syslogNG for "windows" which is part of the syslog-ng commercial offering (syslog-ng PE)
With syslog-ng PE there are two options for collecting windows logs,
- the Agent for Windows can gather locally then forward to remote syslog-ng server
- syslog-ng PE is capable to collect Windows events remotely utilising the Windows Event Collector framework.
With both solution you can feed splunk directly with syslog-ng without need any UF on the syslog side. You can use the HTTP destination to feed Splunk. Even more a dedicated Splunk destination will arrive in syslog-ng in this year supporting log batching and load balancing.

0 Karma

Log_wrangler
Builder

@ gregely

any issues with having syslog-ng PE send to splunk and another application? I would not image so, but if you could share more experience it would be great.

Thanks

0 Karma

Log_wrangler
Builder

@ gergely_bodnar

Thank you for the reply. I am going to make a new post called

Using syslog-ng PE for windows event collection...

If you would share your experiences with it that would be great.

Thank you.

0 Karma

jawadak
New Member

Hi, I am also trying to install a syslog-ng and then send logs to Splunk. Could you please help me on how to make it CIM compatible?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...