Monitoring Splunk

What are the performance impact or other ramifications of deleting events?

the_wolverine
Champion

We are trying out a method of deleting events from the index. It looks like a metadb rebuild takes place every time we do this so would like to know what the performance implications are, if any:

10-09-2012 16:01:13.287 +0000 INFO  databasePartitionPolicy - Detected a deletion, rebuilding metadata in directory '/var/lib/splunkdb/main' 10-09-2012 16:01:38.986 +0000 INFO  databasePartitionPolicy - rebuildMetadata called: full=true path=/var/lib/splunkdb/main reason= delete_fixup 10-09-2012 16:02:05.529
+0000 INFO  databasePartitionPolicy - Detected a deletion, rebuilding metadata in directory '/var/lib/splunkdb/main' 10-09-2012 16:02:30.585 +0000 INFO  databasePartitionPolicy - rebuildMetadata called: full=true path=/var/lib/splunkdb/main reason= delete_fixup 10-09-2012 16:02:56.595
+0000 INFO  databasePartitionPolicy - Detected a deletion, rebuilding metadata in directory '/var/lib/splunkdb/main' 10-09-2012 16:03:23.114 +0000 INFO  databasePartitionPolicy - rebuildMetadata called: full=true path=/var/lib/splunkdb/main reason= delete_fixup 10-09-2012 16:03:48.600
+0000 INFO  databasePartitionPolicy - Detected a deletion, rebuilding metadata in directory '/var/lib/splunkdb/main' 10-09-2012 16:04:14.583 +0000 INFO  databasePartitionPolicy - rebuildMetadata called: full=true path=/var/lib/splunkdb/main reason= delete_fixup 10-09-2012 16:04:40.996
+0000 INFO  databasePartitionPolicy - Detected a deletion, rebuilding metadata in directory '/var/lib/splunkdb/main' 10-09-2012 16:05:06.330 +0000 INFO  databasePartitionPolicy - rebuildMetadata called: full=true path=/var/lib/splunkdb/main reason= delete_fixup 10-09-2012 16:05:32.331
+0000 INFO  databasePartitionPolicy - Detected a deletion, rebuilding metadata in directory '/var/lib/splunkdb/main'
Tags (2)
0 Karma
1 Solution

the_wolverine
Champion

From experience, it appears that DELETE is relatively expensive. When performing DELETE against many events (more than 10k), if done repeatedly in succession, for example, in the case of scheduled search that finds and "DELETE"s, bad things can happen if your system is undersized so watch it carefully. No idea what happens in a properly sized implementation but maybe Splunk can answer.

View solution in original post

0 Karma

the_wolverine
Champion

From experience, it appears that DELETE is relatively expensive. When performing DELETE against many events (more than 10k), if done repeatedly in succession, for example, in the case of scheduled search that finds and "DELETE"s, bad things can happen if your system is undersized so watch it carefully. No idea what happens in a properly sized implementation but maybe Splunk can answer.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...