Getting Data In

Why are universal forwarder internal logs not getting rotated due to permission issues (Access is denied) in Windows?

koshyk
Super Champion

We have deployed universal forwarders on Windows and are running as "local system" (admin). This is installed in C:\Program Files\SplunkUniversalForwarder. When we checked into the splunkd.log details, none of the logs are getting rotated due to permission issues:

WARN Logger - Error unlinking "C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log.1":  Access is denied
WARN Logger - Error renaming "C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log"  to "C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log.1":  Access is denied

As an admin, I can read/write into the same folder. Splunkd can write the log files Ok as the data and size is growing in each of the files. Any reason why access is denied when it tries to rename/unlink?

0 Karma

gavsdavs_GR
Path Finder

This one is confusing, it's happening on a number of machines here.

The sequence is (or should be):
metrics.log.5 gets deleted
metrics.log.4 is renamed to metrics.log.5
metrics.log.3 is renamed to metrics.log.4
metrics.log.2 is renamed to metrics.log.3
metrics.log.1 is renamed to metrics.log.2
metrics.log is renamed to metrics.log.1
a new metrics.log is created.

We are seeing all permissions removed on metrics.log.5
(i.e. an adminstrator has no permissions on the file to even inspect permissions)

This prevents the above sequence from occurring and our metrics.log files are getting larger and larger.

We do not understand what might be interfering with the permissions of the metrics.log.5 file, since all the other files are acciessible, manageable.

I am pretty sure we don't have people looking at metrics.log.5 with a notepad. It's also happening on a number of machines.

Can a splunk person comment on the sequence of actions taken by the UF when rolling out metrics.log.5 ?

We can't tell if something we have in place is occasionally interfering with the removal of it.

0 Karma

javiergn
SplunkTrust
SplunkTrust

That usually happens when you have a lock on those files somehow.
I've seen it when using tail or notepad.
Make sure nothing is reading from your metrics.log as that's the one that can't be renamed.

koshyk
Super Champion

nothing is reading the file other than Splunk UniversalForwader itself trying to send to Indexer

0 Karma

javiergn
SplunkTrust
SplunkTrust

Can you try using Procexp to double check that?
http://stackoverflow.com/questions/320128/releasing-windows-file-share-locks
http://www.howtogeek.com/128680/how-to-delete-move-or-rename-locked-files-in-windows/

If nothing is locking it according to Procexp, try restarting Splunk and it that works then it probably means Splunk was locking those files and that's not great. I would raise a support request but I guess you might be asked to replicate the problem and that might not be easy.

0 Karma

koshyk
Super Champion

thank you for your assistance. I will hopefully raise a support request

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...