Deployment Architecture

metrics.log filling up filesystem on LWF

Branden
Builder

We have a LWF on Linux that is forwarding to our indexer. We're a little tight on space, but in my experience the LWFs don't use up too much space.

On this particular LWF, the metrics.log file is growing at a much higher rate than our AIX-based LWFs. Now we're at risk of filling up the file system. Is there a easy/safe way to clean up these metrics.log files? Is there a reason that file would be growing much faster on Linux than on AIX?

Thanks!

Tags (2)
1 Solution

southeringtonp
Motivator

The logs should automatically rotate and delete copies older than metrics.log.5. You can tune the file size for rotation and the number of copies retained in $SPLUNK_HOME/etc/log.cfg

By default, metrics.log should be rotating after 25 MB, and keep 5 backup copies, so it would cap at 150 MB.

Splunk indexes those files, so it should be safe to delete all but the most recent metrics.log. You can always go back to the _internal index to get the data. (At least for standalone -- I think this is still true for the LWF).

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

Every 30 seconds, metrics outputs a line for each of:

  • the top ten (by default) sources
  • the top ten (by default) sourcetypes

These items are the ones that are most likely to be of variable number across different lightweight forwarders. There are many others, but e.g., there is usually only one host processed on a lightweight forwarder, the same number of queues and pipelines on each, no network inputs on any of them, the same number of outbound connections, etc.

If one forwarder has only a single source or sourcetype, while the other has ten or more, the latter's log file will grow noticeably more quickly. The other factors could affect this, but it would be much less common for them to be different.

southeringtonp
Motivator

The logs should automatically rotate and delete copies older than metrics.log.5. You can tune the file size for rotation and the number of copies retained in $SPLUNK_HOME/etc/log.cfg

By default, metrics.log should be rotating after 25 MB, and keep 5 backup copies, so it would cap at 150 MB.

Splunk indexes those files, so it should be safe to delete all but the most recent metrics.log. You can always go back to the _internal index to get the data. (At least for standalone -- I think this is still true for the LWF).

Branden
Builder

Thank you very much for the info!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...