Getting Data In

Universal Forwarder resends entire Security Event log after upgrade.

JeremyHagan
Communicator

I have recently started upgrading Windows universal forwarders from 6.0.3 to 6.2.6. After I upgrade them they seem to be resending the entire Windows Security log (2GB) instead of continuing where they left off. I can see the evidence of this by viewing the index data amount from the host staring after they are upgraded and by doing a report on Windows Security Events and seeing that there are multiple events with the same RecordNumber field.

Now I could modify my install script to drop the Security log, upgrade the software and avoid the licensing issues this is causing, but I'd prefer to get to the root cause.

Has anyone seen this?

0 Karma

JeremyHagan
Communicator

I did some more work on this and incrementally upgraded a forwarder from 6.0.3 to 6.0.4 then 6.0.5 then 6.0.7 then 6.1.1

It was the jump from 6.0.X to 6.1.X that failed. During the installation there was some sort of fatal error and after a couple of retries I rebooted and then found that the UF wasn't installed at all. So when the install happens it is like a fresh install and so it does the whole log, as you would expect.

I've modified my install script to clear the Security log before installing. I'm not sure I can stand the pain of dealing with support.

0 Karma

JeremyHagan
Communicator

A couple more points. I upgraded a very old forwarder from 4.3.3 to 6.2.6 and found no issues. Then I found one on 6.1.1 and upgraded it with no issues. Then I upgraded another 6.0.3 forwarder and reproduced the issue.

0 Karma

jeffland
SplunkTrust
SplunkTrust

Have you had a look at inputs.conf or this documentation, specifically the current_only setting? Set to 1, it should prevent your forwarders from re-reading the entire log of windows events.

0 Karma

JeremyHagan
Communicator

I thought of doing that for the period of the upgrades, but I was hoping to have the product work as advertised.

0 Karma

jeffland
SplunkTrust
SplunkTrust

Yes, this doesn't seem to work as intended. If the settings in your inputs.conf are identical and just the version of the forwarder is different (between the 4.3.3 or the 6.1.1 and the 6.0.3 ones), I would suggest you file a support ticket.

0 Karma

JeremyHagan
Communicator

I should mention my stanza in inputs.conf is very simple:

[WinEventLog:Security]
disabled = 0
index = WindowSecurity
evt_resolve_ad_obj = 1

From what I have read, it should be taking a checkpoint every 5 seconds bye default.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...