Removing duplicate WinEventLog data

[JovanMilosevic]

Hi,

Sorry, this is one of those "How do I remove duplicates" questions.

We have a customer who is currently using wmi to collect event logs, but wants to change over to using a UF. At the point of changeover, the UF will by default index everything in the event log at the time it starts. This will result in duplicate events in the index. However, they will have 2 different sourcetypes - WMI:WinEventLog:Security and WinEventLog:Security. What I'd like to do is find the duplicates, and then delete them from one or other of the sourcetypes.

Any ideas on how this could be automated - they have quite a few servers to move over.

Thanks.

[sdwilkerson · Accepted solution]

JovanMilosevic,

Splunk has a way of handling this within the inputs.con with current_only.

Here is what you can do:
- Install the new UF with disabled input (as described below)
- Disable that target host from the wmi polling
- Simultaneous with above enable the input below
- You could have a small gap in data, but it will be very small and will avoid the duplication and dedup mess.

If losing any events is not an option you can do the above, but when you initially install the UF you can leave the inputs as enabled. As soon as you verify events are coming in, then disable the WMI poll for that host.

Sample for inputs.conf:

[WinEventLog:Security]
disabled = 0
index=windows
followTail = 1
current_only = 1

Do dedup events after migration, you can use searches to find the duplicated events then use the delete special search command to remove the extra data.

Best,
Sean

[sdwilkerson]

JovanMilosevic,
I updated my previous answer to address your issue. Good luck.

[JovanMilosevic]

Thanks for taking the time to answer. Unfortunately, this isn't quite what I'm after. I know it is a practical solution, but it will either lose or duplicate data, still. I know it's not much, but once they are getting the information, losing some becomes politically unacceptable. The fact that they did without for years escapes them.