Deployment Architecture

What happens to a bucket if all of the data within it is deleted?

caseyra
Explorer

Just what the title says. If I delete a bunch of data from an index and some of the buckets are now effectively empty (E.g. all of them have been marked as deleted), what happens to the bucket? Is it removed or will it stick around until it is frozen?

0 Karma
1 Solution

s2_splunk
Splunk Employee
Splunk Employee

It will stick around until the bucket is frozen. The bucket mover uses the epoch times in the bucket (directory) name and doesn't really inspect the contents of the bucket to determine whether it can be frozen or not.

View solution in original post

s2_splunk
Splunk Employee
Splunk Employee

It will stick around until the bucket is frozen. The bucket mover uses the epoch times in the bucket (directory) name and doesn't really inspect the contents of the bucket to determine whether it can be frozen or not.

caseyra
Explorer

O.K. So, is there a way to determine if a bucket has only deleted entries? And, if so, can we force the bucket to be rolled to frozen? Or, is the file basically stuck there until it is frozen?

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

The short answer is no, buckets will honor the configured retention policy settings and there is no way to force a freeze unless you are (temporarily) reducing the index configuration. I am not aware of any way to safely determine whether a given bucket contains only deleted events.
The only way currently to physically delete events (buckets, really), is to run clean eventdata, which will wipe everything for the given index, so you'll have to come up with a plan to export/collect any events you do not want to delete.

What's your use case? Compliance? New EU regulations?

0 Karma

caseyra
Explorer

Follow-up question: I did a little digging with dbinspect and noticed that some buckets can have an eventCount of zero, but the rawSize and/or sizeOnDiskMB is non-zero. This seems to indicate that these buckets have had all their records deleted, but are still taking up space on the system. Is that correct?

0 Karma

caseyra
Explorer

It's more of a data cleanup issue. I was trying to avoid having to re-index a large chunk of data.

Thanks.

0 Karma

ChrisG
Splunk Employee
Splunk Employee

It depends what you did to delete the data: did you use the delete command (which does not actually remove anything from disk), or clean, or remove index?

If you have not already done so, I highly encourage you to read Remove indexes and indexed data in the Splunk Enterprise documentation.

0 Karma

caseyra
Explorer

Just delete. I'm not trying to remove the entire index. I just wanted to know what would happen from a disk space perspective.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...