Knowledge Management

Move specific data from one index to another index

Navern
New Member

Hello,

I want to move specific data from one index to another index. I don't want to make a full copy of previous index i just need to copy some of sourcetypes to another index. Then i need to delete copied data in old index.

Could someone please provide the best approach to do this?
I've read about collect command after search query but i believe it didn't actually copy your data on hard drive.

And i i've read about splunk cmd exporttool but not so much. Could someone please clarify on this point? Version of splunk is 6.0.1

Tags (2)
0 Karma

yannK
Splunk Employee
Splunk Employee

It's difficult to export indexed events. While it is very easy to move/duplicate buckets.
If you cannot reindex your data, then my recommendation is to copy the buckets and hide the events.

  • identify the buckets with the data you need.

You can check the time range of each bucket endEpoch and startEpoch and the command :

|dbinspect index=myindex | convert ctime(startEpoch) | convert ctime(endEpoch) 

and use this trick to find in which buckets are your data if you want specific events

index=myindex sourcetype=BBBBBB | eval bkt=_bkt |  stats count by bkt

The only issue is that your buckets will not shrink when the events are deleted (until the buckets roll out to frozen), so this will initially use more space than before.

splunkreal
Motivator

Hi @yannK

what about this query?

index=INDEXNAME | eval bkt=_bkt 
|  table index,source,sourcetype,host,bkt
| join bkt 
    [| dbinspect index=INDEXNAME | convert ctime(startEpoch) | convert ctime(endEpoch) 
    |  rename bucketId as bkt]
| table index,source,sourcetype,host,bkt,state,sizeOnDiskMB,startEpoch,endEpoch,splunk_server

Thanks.

* If this helps, please upvote or accept solution 🙂 *
0 Karma

Navern
New Member

I've found one of your previous answers to similiar question: http://answers.splunk.com/answers/25174/how-to-exportimport-events-from-indexes.

I see that i will have problem with exporting specific data with this approach.

It's a pity that there is no script for this even with REST API present.

Do you know how extensively RAM on server will be used during export? I have a very bad server running Splunk.

0 Karma

yannK
Splunk Employee
Splunk Employee

The delete just hide results, and does not reclaim disk space.
What I was saying is that the space will be retrieved once the whole buckets reach the timerentention limit. (default is 6 years or 500GB, but can be tweaked)

0 Karma

lukejadamec
Super Champion

No. You can't modify data once it has been indexed.

0 Karma

Navern
New Member

Thanks for the answer,

I haven't found the way to delete specific data from local disk completely. As far as i know "delete" command just removes this data from search and not from local disk. Command splunk remove index="index" will delete entire index completely.

Can i remove from local disk data which was deleted via splunk search command?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...