Getting Data In

Please route me i am at INDEX deadend.

Rocky31
Path Finder

I appreciate your time and effort. below are questions

1) I want to find out where is the index.conf for my index XYZ (read bunch of docs not sure where to check in SH or Deployment server or something else)

2) with this search i got bunch of index (search “|tstats count where source=/opt/splunk/syslog-ng/* by index | table index”) where is index.conf configs for those indexes?

3) And where to check those indexes over 50% of max size?

Tags (1)
0 Karma

woodcock
Esteemed Legend

Interestingly enough, I just wrote this code last week:

#! /bin/bash

SPLUNK_DB=$(/opt/splunk/bin/splunk envvars | sed 's/.*SPLUNK_DB=\(\S\+\).*$/\1/')
if [[ $(echo ${SPLUNK_DB} | awk -F '/' '{print NF}') -eq 0 ]]; then
   # No '/' characters so splunk is not running here
   TEST_MODE="YES"
   echo "TEST MODE INITIATED"
   SPLUNK_DB="/opt/splunk/var/lib/splunk"
   echo "SPLUNK_DB=${SPLUNK_DB}"
   LOGFILE="/dev/null"
   echo "LOGFILE=${LOGFILE}"
   indexes_file=./indexes.conf
   # assume that the index detail file already exists locally
else # Splunk is running here; assume that it really is an indexer
   LOGFILE="/var/log/$0.$(date +%Y%m%d_%H%M%S)"
   indexes_file=/tmp/indexes.conf
   # generate the index detail file
   /opt/splunk/bin/splunk btool indexes list | egrep '^\[|^homePath\s*=|^path\s*=' > ${indexes_file}
fi

while read -r line; do
   if [ "$(echo ${line} | cut -c 1)" != '[' ] || [ "$(echo ${line} | cut -c1-8)" = '[volume:' ] || [ "$(echo ${line} | cut -c1-17)" = '[provider-family:' ]; then
      continue
   fi
   # capture next index value
   index=$(echo ${line} | awk '{print substr($0, 2, length($0) - 2)}')
   index_lower=$(echo ${index} | tr "[:upper:]" "[:lower:]")
   echo "   index=${index}, index_lower=${index_lower}"
   if [[ $index == "default" ]] || [[ $index == "splunklogger" ]] || [[ $index == "history" ]]; then
      echo "Skipping this one..."
      continue # skip this one
   fi
   # Identifying warm buckets path
   home_path=$(grep -A1 -w "^\[${index}" ${indexes_file} | grep -E "^homePath\s*=" | awk -F '=' '{print $2}'| xargs echo)
   echo "home_path=${home_path}"
   if [[ $home_path == volume:* ]]; then # dereference volume
      warm_volume_name=${home_path%%/*}
      # Identifying Warm Volume Path
      warm_volume_path=$(grep -A1 "^\[${warm_volume_name}" ${indexes_file} | grep path | awk -F '=' '{print $2 }' | xargs echo)
      echo "warm_volume_path=${warm_volume_path}"
   elif [[ $home_path == \$SPLUNK_DB* ]]; then # dereference "$SPLUNK_DB"
      warm_volume_path=${SPLUNK_DB}
   else # just a normal path that needs no modification at all
      warm_volume_path=''
   fi
   home_path=${home_path#*/}
   comp_path=$(echo ${warm_volume_path}/${home_path} | tr "[:upper:]" "[:lower:]")
   # replace "$_index_name" if present
   comp_path=$(echo ${comp_path} | sed "s%/\$_index_name/%/${index_lower}/%")
   echo "comp_path=${comp_path}"

   if [[ ${TEST_MODE} == "YES" ]]; then
      # Not running on an actual indexer (splunk is not running here)
      continue
   fi

   # Do other stuff here
done < ${indexes_file}

Rocky31
Path Finder

Hi Splunk superstar,
this is really a out of box thinking, i need to work on it to understand further. good one.

0 Karma

woodcock
Esteemed Legend

So how did it work out @Rocky31?

0 Karma

woodcock
Esteemed Legend

Be sure to come back after you test and click Accept to close the question (or to point out bugs/fixes)!

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi Rocky31,

1 & 2 can be resolved by using this command on an indexer instances:

$SPLUNK_HOME/bin/splunk btool indexes list --debug 

This will show you the output of all indexes.conf merged and their location. For just one index run it like this:

$SPLUNK_HOME/bin/splunk btool indexes list indexNameHere --debug 

Regarding 3. if you have the monitoring console configured https://docs.splunk.com/Documentation/Splunk/latest/DMC/DMCoverview you will get a lot of pre-built dashboards around index sizing, age and more.

Hope this helps ...

cheers, MuS

Rocky31
Path Finder

thanks for your answer, can you also tell me how to check each index size occupied in indexers in percentages

0 Karma

woodcock
Esteemed Legend

This answer show which configuration files contain the definitions for every index, but it does not fully resolve the path, which is what I believe the OP desires. Please see my answer for how to do that.

0 Karma

Rocky31
Path Finder

Sure, thanks

0 Karma

MuS
SplunkTrust
SplunkTrust

You must be using a different version of btool 😉
I get a nice output of the path when using the posted command $SPLUNK_HOME/bin/splunk btool indexes list --debug

cheers, MuS

0 Karma

woodcock
Esteemed Legend

You get a value for the path but it will not resolve any volume definitions, nor $SPLUNK_DB, which is what my code does.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...