Splunk Search

Searching _fishbucket

pembleton
Path Finder

I'm trying to figure out how to analyze and manage specific records in the _fishbucket index.

I have big directories with many files splunk is monitoring, and our only method for reindexing right now is either cleaning the _fishbucket or adding crcSalts, both not very good for most use cases.

I want to be able to handle specific records in the _fishbucket - looking at this old blog post (what-is-this-fishbucket-thing) - it starts off by saying

 To see what’s there, try searching for “index=_thefishbucket”

This simply doesn't work. And I want to also be able to delete specific records.

What's up with this? something changed since this blog post?

Tags (2)
1 Solution

yannK
Splunk Employee
Splunk Employee

There is a tool to selectively forgot a single file from the fishbucket

./splunk cmd btprobe -d $SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db --file $FILE --reset

see options for btprobe

`

There are 2 possible ways to invoke this tool:

 1: btprobe [-h or --help] -d <btree directory> [-k <hex key OR ALL> | --file <filename>] [--salt <salt>] [--validate] [--reset] [--bytes <bytes>]
 Queries the specified BTree for the given key or file.

    -d           Directory that contains the btree index. (Required)

    -k           Hex crc key or ALL to get all the keys.
    --file       File to compute the crc from.
      (One of -k and --file must be specified.

    --validate   Validate the btree to look for errors.
    --salt       Salt the crc if --file param is specified.
    --reset      Reset the fishbucket for the given key or file in the btree.
    --bytes      Number of bytes to read when calculating CRC (default 256).

 2: btprobe [-h or --help] --compute-crc <filename> [--salt <salt>] [--bytes <bytes>]
 Computes a crc from the specified file (using the given salt if any).

 Examples:
 btprobe -d /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db  -k 0xe8d117ddba85e714 --validate
 btprobe -d /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db --file /var/log/inputfile --salt SOME_SALT
 btprobe --compute-crc /var/log/inputfile --salt SOME_SALT

`

View solution in original post

yannK
Splunk Employee
Splunk Employee

There is a tool to selectively forgot a single file from the fishbucket

./splunk cmd btprobe -d $SPLUNK_HOME/var/lib/splunk/fishbucket/splunk_private_db --file $FILE --reset

see options for btprobe

`

There are 2 possible ways to invoke this tool:

 1: btprobe [-h or --help] -d <btree directory> [-k <hex key OR ALL> | --file <filename>] [--salt <salt>] [--validate] [--reset] [--bytes <bytes>]
 Queries the specified BTree for the given key or file.

    -d           Directory that contains the btree index. (Required)

    -k           Hex crc key or ALL to get all the keys.
    --file       File to compute the crc from.
      (One of -k and --file must be specified.

    --validate   Validate the btree to look for errors.
    --salt       Salt the crc if --file param is specified.
    --reset      Reset the fishbucket for the given key or file in the btree.
    --bytes      Number of bytes to read when calculating CRC (default 256).

 2: btprobe [-h or --help] --compute-crc <filename> [--salt <salt>] [--bytes <bytes>]
 Computes a crc from the specified file (using the given salt if any).

 Examples:
 btprobe -d /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db  -k 0xe8d117ddba85e714 --validate
 btprobe -d /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db --file /var/log/inputfile --salt SOME_SALT
 btprobe --compute-crc /var/log/inputfile --salt SOME_SALT

`

sowings
Splunk Employee
Splunk Employee

Are you wanting to delete certain records so that Splunk will (re-)index the files?

Note that you can issue splunk add oneshot from the command line to prompt Splunk to index any file without regard for the fishbucket.

http://docs.splunk.com/Documentation/Splunk/5.0.1/Data/MonitorfilesanddirectoriesusingtheCLI

lguinn2
Legend

Yes, something changed since 2008! In fact, Andrea added a footnote to the end of the article in 2010: "Note that this old post only applies to 3.x versions"

Splunk no longer lets you look at the fishbucket index. You cannot manage the specific records. The format is not published and the files are kept in binary.


Thanks for the update. I guess I am a bit behind on the cool tools! Although it is still true that you can't just go editing or viewing the fishbucket!

Here is a link to the relevant section in the Troubleshooting Manual.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...