Deployment Architecture

Forwarder won't stop forwarding

mmoermans
Path Finder

A strange issue is happening, a few of our forwarders are sending a massive amount of data (wineventlog:security) to Splunk. I've tried to remove the Windows_TA_Splunk from the forwarders but they keep sending their eventlogs regardless.

I've tried restarting them several times but they don't stop sending data even though they don't have any received apps. Anyone know how to make them stop sending the wineventlog?

0 Karma
1 Solution

mmoermans
Path Finder

Solved by making a copy of Splunk_TA_Windows called Splunk_TA_Vindows (Using the letter V since it's earlier in alphabet and would take precedent) and disabling wineventlog:security in it, pushing that to the forwarder.

Even though I had removed the client from the Serverclass for windows eventlog it hadn't removed the app for some reason and only stopped after pushing a new app (Splunk_TA_Vindows) above it which disabled the winsecurity logging.

Seems to be a bug for forwarders on 6.3.1., best fix would be to update the forwarder but wasn't possible in this case.

View solution in original post

0 Karma

mmoermans
Path Finder

Solved by making a copy of Splunk_TA_Windows called Splunk_TA_Vindows (Using the letter V since it's earlier in alphabet and would take precedent) and disabling wineventlog:security in it, pushing that to the forwarder.

Even though I had removed the client from the Serverclass for windows eventlog it hadn't removed the app for some reason and only stopped after pushing a new app (Splunk_TA_Vindows) above it which disabled the winsecurity logging.

Seems to be a bug for forwarders on 6.3.1., best fix would be to update the forwarder but wasn't possible in this case.

0 Karma

horsefez
Motivator

Could you go into a bit more detail on how you managed it? Thanks!

0 Karma

ddrillic
Ultra Champion

I had a similar experience recently at Where does the forwarder enqueue files?

My experience was -

-- It is the universal forwarder reading the files. I think it's TailReader versus BatchReader. What I see is that TailReader is real-time versus BatchReader which is not and also we don't seem to have control of the pending batches.

So, when the forwarder was in this BatchReader mode, the only way to stop it was to uninstall and reinstall the forwarder.

Maybe you are in a similar situation...

0 Karma

jgbricker
Contributor

Are they deployments clients? If so they would keep getting updated with the enabled inputs settings even though you are manually changing at the source. You may need to make a new copy of the TA and and setup a separate server class excluding those from the main server class and including in the non security input server class.

somesoni2
Revered Legend

Try to run btool command to see if that eventlog monitoring is enabled elsewhere.

bin/splunk cmd btool inputs list wineventlog:security --debug

If it shows enabled at path other than TA apps, disabled from that place as well (and restart)

Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...