Getting Data In

How to delete data / index (reset start from scratch)

mldeschenes
Explorer

From UI it seems easy to add data but I don't see an option to delete existing data from index. I need the quick an dirty steps to remove existing data/index so we can experiment different types of logs to import/indexed. Trial/Error pilot...

Tags (1)

daymauler
Explorer

This is how I took care of a similar issue:

1. I ran a search on the SH:

    index=my_index sourcetype=my_sourcetype | delete

2. After confirming that the records have been delete, I went to the HF

- stopped Splunk

-  /opt/splunk/bin/splunk cmd btprobe -d /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db/ --file /my_file_path/filename.fileextension --reset

- restarted Splunk

and was able to see new data in SH

0 Karma

Aleksey_18
New Member

Prompt, how to clean an index when indexes in a cluster ?
The architecture of the cluster consists of two server indexers - the server indexer-1 and the server indexer-2, are managed all through the server cluster-master.
How in this case are the indexes cleaned correctly ?

In the documentation, I found 2 ways to clean the index:
1 Through the GUI https://docs.splunk.com/Documentation/Splunk/7.1.1/SearchReference/
2 And through the clean eventdata command http://docs.splunk.com/Documentation/Splunk/7.1.1/Indexer/RemovedatafromSplunk

The first option does not delete the physical data from the index, but makes them unavailable for searching
The second option cleans the index but completely, but in the documentation it is written that the command (clean eventdata) does NOT WORK FOR INDEXER IN THE CLUSTER
How will the PHYSICALLY index in the cluster be cleared ?

0 Karma

goelli
Communicator

If you just want to wipe all data out of the index, perhaps you can use this:

Edit indexes.conf parameter frozenTimePeriodInSecs to 60 seconds
Apply cluster-bundle or restart indexer

Revert changes to indexes.conf and apply cluster-bundle or restart indexer again.

Best regards,
Thomas

isoutamo
SplunkTrust
SplunkTrust

And before that stop UFs and other inputs. I prefer even shorter period than 60 seconds. Wait until you see that this index is empty on all indexers, then update retention back.

Another option is remove index definition from indexes.conf apply cluster-bundle then go to indexers and remove individual index directories an files. Of course you must wait that rolling restart has finished before that.

Ismo

0 Karma

lguinn2
Legend

Great idea!

There isn't any great way to do this from the UI. However, there is a really simple way to do this from the command line:

splunk stop
splunk clean eventdata -index yourindex
splunk start

It cleans very fast, since it is just deleting all the files. If you want to clean ALL indexes, just leave off the

-index yourindex

AnilPujar
Path Finder

What is the steps for indexers in cluster?

0 Karma

lguinn2
Legend

If you still see previous data, then something didn't work. I use the clean command every week and it works, I assure you.

That said - if you reset the index, then IF you have an input defined in inputs.conf, the data will be re-indexed. If you don't want the data re-indexed, then disable the stanza in inputs.conf!

0 Karma

mldeschenes
Explorer

So there is no way to reset the index , I still see previous log/data imported into index. I only had 1 log imported and basically want to start from scratch for testing purposes (test import options). I guess I will have to re-install the application every time it seems?

0 Karma

miteshvohra
Contributor

It will be better to clean the _thefishbucket also to reindex the same data again.

lguinn2
Legend

If you clean an index, the fishbucket on the indexer will be appropriately updated so that the data will be reindexed - if that data came from the indexer.

However, if the data came from a forwarder, then you need to clean the _fishbucket on the forwarder to force the forwarder to resend data that it already sent once. (Because the forwarder doesn't know that you cleaned the indexer...)

So wherever the inputs.conf lives, that's where you need to reset the fishbucket to get the data re-indexed. If you don't want to reset the entire fishbucket, lookup btprobe which will let you reindex a single source.

Good point Mitesh

0 Karma

lguinn2
Legend

If you did this on the indexer, then all the indexes were cleaned. If you have multiple indexers, you will need to do this on each indexer.

splunklogger is used by Splunk for internal info - it will never "clean" and that's okay.

0 Karma

mldeschenes
Explorer

Thank you for quick response, however did what you recommended without -index...
I still see Events Index data from UI?

splunk stop
splunk clean eventdata
This action will permanently erase all events from ALL indexes; it cannot be und
one.
Are you sure you want to continue [y/n]? y
....
Disabled database 'splunklogger': will not clean.
splunk start
All preliminary checks passed.
Done

0 Karma

linu1988
Champion

moreover |delete to hide the existing events and continue testing your data with different source config.

0 Karma

rakesh_498115
Motivator

hi lguinn,

a quick one,

Can we clean the index without the flag , is there any option ??

thanks.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...