Monitoring Splunk

Not enough free disk space on the search head, and not enough free swap space on the indexer

lkeli_spl
Engager

Hi all,
We enlarged RAM and CPU capabilites on the indexer and search head. After that we had a problem on these servers: the free disk space decreases dramatically on the search head (in a few minutes to 0%), and the free swap space decreases dramatically on the indexer.
After servers reboot the problem temporarily disappears. The problem repeats in a few days.
There was no such problem before enlarging RAM and CPU capabilites.
In the logs splunk did not find anytheng.
Can this problem be related to the enlarging RAM and CPU capabilites? Where else to look for the cause of this problem?
Guys could you comment on this?

We have 1 indexer and several search head:
Splunk Enterprise Server 6.5.2
Linux, 47.1 GB Physical Memory, 12 CPU Cores

Labels (1)

woodcock
Esteemed Legend

What exactly do you mean by We enlarged RAM and CPU capabilites?

0 Karma

codebuilder
SplunkTrust
SplunkTrust

Be sure to configure logrotate for the Splunk internal logs, they can grow to around 25MB each pretty quickly.
Another option is to symlink /opt/splunk/var/log/splunk over to /var/log/

Here is how I configure logrotate on my Splunk nodes:

/etc/logrotate.d/splunk
/var/log/splunk/*.log {
    missingok
    notifempty
    compress
    daily
    rotate 7
    create 0644 splunk splunk
}
----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

highsplunker
Contributor

Sorry, but i think, this is not gonna help. @lkeli_spl please comment.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...