Splunk Dev

Official way to clean indexed data from index cluster

napomokoetle
Communicator

Hi,

I have an indexer cluster with two indexers, a separate Cluster Master Node and one separate Search Head.

Both Indexers have 500 GB space and the disk space is now fully used and we're now dropping events.

Can I delete data from the _internal index on the clustered indexers to free some space? If so how do I safely do so?
Can I delete data from some non-interal indexers on the clustered indexers to free some space? If so how do I execute the cleaning with minimal risk of breaking the environment?

I have used the "./splunk clean eventdata -index myIndex -f" command on STANDALONE environments without a problem, but I am not sure if that would be SAFE on a clustered indexer environment due to replication/synchronization/etc.

Look forward to hear from you. Thanks advance for your help.

Kind regards,
J. Napo Mokoetle

0 Karma
1 Solution

gfuente
Motivator

Hello

The best way (i found) to do this, is to set a smaller retention policy in the indexes.conf (through the master apps) to the indexes you want to delete events, apply the changes, wait some minutes until the indexers delete the events, and afterwards, change back the retention policy to the final value

You need to include this setting in your indexes.conf

frozenTimePeriodInSecs =

For example, you could set it to 86400, to delete everything older than 1 day

regards

View solution in original post

gfuente
Motivator

Hello

The best way (i found) to do this, is to set a smaller retention policy in the indexes.conf (through the master apps) to the indexes you want to delete events, apply the changes, wait some minutes until the indexers delete the events, and afterwards, change back the retention policy to the final value

You need to include this setting in your indexes.conf

frozenTimePeriodInSecs =

For example, you could set it to 86400, to delete everything older than 1 day

regards

napomokoetle
Communicator

Thanks for your swift response gfuente.

Just to make sure I'm with you.

The indexes.conf to edit on the master is the one in the path

/opt/splunk/etc/master-apps/_cluster/local/indexes.conf

right?

0 Karma

gfuente
Motivator

Yeah, could be that, or any other in any app under master apps, :

/opt/splunk/etc/master-apps//local/indexes.conf

Basically, wherever you already are configuring your existing indexes

Regards

0 Karma

napomokoetle
Communicator

Hi gfuente,

Your method worked like a charm. Thanks a great deal!

gfuente
Motivator

Good to know it

0 Karma

napomokoetle
Communicator

I am running Splunk 6.3.2 on Linux Centos.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...