How can I really delete an event

[w199284]

I have read about the "delete" command and used it. However, my security people want certain events gone without the possibility of recovery. I've looked a little at CLI Search with -output table that looks promising. The idea would be to export the index, remove the offending data and re import/index the result. The original source is long gone. Has anybody had to attempt anything similar?

[w199284]

I found a way that worked for me.

• suspend the offending data feed
• run a search that returns the offending data and pipe to |delete
• Take the index offline
• backup the index
• run a shell script (I'm not much of a script jocky) that returns buckets that have "deletes" folders under rawdata
o pass that bucket name to splunk's exporttool and output as a -csv workfile
o pass the csv workfile output to splunk's importtool and create/output a new bucket with the original name "bucketname.new"
o remove the old bucket and rename bucketname.new to bucketname
• put the index back online/test
• resume the data feed

It is a lot of steps, in my case it took 10 hours to complete (mainly waiting on the export/import to finish) and I had to process warm and cold buckets on 12 index peers. I ran these as background tasks. One for warm buckets. One for cold buckets. I performed a lot of tests before turning this loose. In the end, all the data that had been |delete(d) was gone. Since there was about 5 years of history in play, worth the effort.

[tpeveler_splunk]

One option would be to use the dump command along with the clean command. After which you would re-index the events.

You would essentially execute a search that identifies the good events and dump them to local disk in raw format. See Splunk dump command in Search Reference.

You would then clean the index of all events via the splunk clean ... CLI command. See Remove data from one or all indexes.

Finally, you would re-index the events that were dumped to disk.

[frechetta93]

@tpeveler_splunk How does one re-index the dumped events?

[ChrisG]

The delete search command only marks data for exclusion in subsequent searches.

If you want to remove specific data, you should use the clean CLI command.

If you want to remove an index entirely, use the remove index CLI command.

See Remove indexes and indexed data in the Managing Indexers and Clusters of Indexers manual.