Getting Data In

I have Windows Universal Forwarder that is sending thousands of blank Windows events and duplicating events

andrewcg
Path Finder

On the Windows client server (splunkforwarder-6.2.1-245427-x64-release.msi) the inputs.conf file contains:

[WinEventLog://Application]
current_only = 0
disabled = 0
start_from = newest
index = win_eventlog

[WinEventLog://System]
current_only = 0
disabled = 0
start_from = newest
index = win_eventlog

I cleared the Windows event logs on the client server and deleted all of the events for the server from Splunk. Today, 16 hours after doing that, I have over 2 millions events in Splunk for that Host. The client server shows around 800 new event for the those 16 hours, yet Splunk now has 60K events. They are many "blank" ones like this:

02/27/2015 08:04:38 AM
LogName=System
SourceName=
EventCode=1111
EventType=2
Type=
ComputerName=servername
TaskCategory=
OpCode=
RecordNumber=150188
Keywords=
Message=

The actual event that this should be is:

Log Name: System
Source: Microsoft-Windows-TerminalServices-Printers
Date: 2/27/2015 8:04:38 AM
Event ID: 1111
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: servername
Description:
Driver RICOH Aficio MP 6001 PCL 5e required for printer St C - 4th - RICOH Aficio MP 6001 PCL 5e is unknown. Contact the administrator to install the driver before you log in again.

Searching based on the recordnumber finds 9 records. I also found the entry put in correctly 9 times too. Searching again, now they are both in there ten times. Looking at the recordnumber stats, the older the event the more copies, with events from 2012 (which should not have even been loaded) having 1000 or more copies. All of these events are counting against our license, so the duplicating is occurring before the data is indexed.

I have uninstalled the Splunk forwarder and reinstalled, and this issue is still occurring. It seems the forwarder is just periodically resending all of the events.

0 Karma
1 Solution

lguinn2
Legend

I am not sure why this is happening, but I think that your settings may be in conflict. I would set your stanzas to

disabled = 0
index = win_eventlog

Also, make sure that you don't have multiple inputs.conf files, which might also be indexing the same or similar data. You can search for files named inputs.conf in the Splunk etc subdirectory. You can also use Splunk's btool command from the Windows command line, although it may give you more detail than you really need:

splunk btool inputs list --debug | more

View solution in original post

milesbrennan
Path Finder

We have over 250 domain controllers and spent 2 weeks getting our license smashed with Windows Event duplications - some events were duplicated over 420 times...

....the issue was "inputs.conf"

start_from = newest

Check your Windows Security Event duplications with:

index=wineventlog sourcetype=WinEventLog:Security | stats count by RecordNumber, _time, host | where count > 1

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf says:

start_from = <string>
* Specifies how Splunk should chronologically read the event log channels.
* Setting this attribute to 'oldest' tells Splunk to start reading Windows event logs from oldest to newest.
* Setting this attribute to 'newest' tells Splunk to start reading Windows event logs in reverse, from newest to oldest.  Once the input consumes the backlog of events, it will stop.
* 'newest' is not supported in combination with current_only = 1 (This combination does not make much sense.)
* Defaults to oldest.

current_only = [0|1]
* If set to 1, the input will only acquire events that arrive while Splunk is running and the input is enabled.  Data which was stored in the Windows Event Log while splunk was not running will not be read. This means that there will be gaps in data if splunk is restarted, or experiences downtime.
* current_only = 1 is not supported with start_from = 'newest'. (It would not really make sense.)

What the documentation doesn't say, is "current_only = 1" is the default setting. So if you enable "start_from = newest", you MUST set "current_only = 0", or you will review huge amounts of duplicate events.

We have a support ticket in on this and have asked for the documentation to be updated.

RDAVISS
Path Finder

That certainly makes sense, thanks for posting!

0 Karma

skalliger
SplunkTrust
SplunkTrust

Hello there,

this is an older thread but we ran into a similar issue. But we had a very interesting additional problem here:
Not only the Windows Events were duplicated but also the _internal events were duplicated.

We still don't understand how the Windows Security stanza could have affected the internal events. But after setting "start_from = newest" with "current_only = 0", we have no more duplicated internal events (and, of course, no more duplicated RecordNumbers).

Skalli

0 Karma

RDAVISS
Path Finder

I know this old but I had the same problem. I had configured the Windows TA inputs file vs the local inputs.conf and one host was eating up 15GB's of our license. I knew it was duplicating events because the same RecordNumber was showing hundreds of times. I wanted to comment on this so the solution is easier to find (the final comment was truncated on this thread.)

To correct the problem I removed the "start_from = newest" parameter AND configured the local inputs.conf. Not sure which, or maybe both, caused the problem.

Hope this saves someone else some time.

lguinn2
Legend

I am not sure why this is happening, but I think that your settings may be in conflict. I would set your stanzas to

disabled = 0
index = win_eventlog

Also, make sure that you don't have multiple inputs.conf files, which might also be indexing the same or similar data. You can search for files named inputs.conf in the Splunk etc subdirectory. You can also use Splunk's btool command from the Windows command line, although it may give you more detail than you really need:

splunk btool inputs list --debug | more

andrewcg
Path Finder

I will give that a try.

  • current_only defaults to 0
  • start_from defaults to oldest

So the start_from value will change if I leave them blank. It will take a couple of days to see if this resolves the issue.

0 Karma

andrewcg
Path Finder

The Splunk Universal forwarder now comes with a Windows TA. I am not sure what version that started. The TA alos has settings for the Event Logs. Once I got rid of the TA, and used the default Windows Eventlog settings the duplication issue went away. This TA, Splunk_TA_Windows, is only added in brand new installs. None of my upgrades had this added. Thanks for the help Iguinn

0 Karma

andrewcg
Path Finder

No luck, it is already duplicating again. I installed the exact same installer on another Windows 2008 server. I applied the exact same inputs.conf file (hosts are different) and it is not having the issue. There is something specific to the first server that is causing this issue. It is like it in not keeping track of the events it is loading and just repeatedly loading all of the available events.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...