Getting Data In

How to calculate and troubleshoot indexing volume by _internal index.

thezero
Path Finder

Hi ,

How to calculate indexing volume/disk space usage for _internal index /internal DB per day In GB? Any specific query to find out top host, source type which is sending most of the logs to _internal index. Although _internal index log volume do not calculated against license its consuming high disk space and needs to identify the hosts sending high volume of internal logs. Any specific search query/tstat/db inspect command for same . Please advise.

0 Karma
1 Solution

ngatchasandra
Builder

Hi thezero,

Try this :

index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart span=1d sum(GB) by host useother=false|untable _time host sum(GB)|top limit=1 host|join -time[search  index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart span=1d sum(GB) by sourcetype useother=false |untable _time sourcteype sum(GB)]

Also see the following:

http://www.splunk.com/wiki/Community:TroubleshootingIndexedDataVolume

View solution in original post

0 Karma

ngatchasandra
Builder

Hi thezero,

Try this :

index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart span=1d sum(GB) by host useother=false|untable _time host sum(GB)|top limit=1 host|join -time[search  index=_internal source=*license_usage.log type=Usage | eval GB=b/1024/1024/1024 | timechart span=1d sum(GB) by sourcetype useother=false |untable _time sourcteype sum(GB)]

Also see the following:

http://www.splunk.com/wiki/Community:TroubleshootingIndexedDataVolume

0 Karma

fdi01
Motivator

if you are looking to estimate your daily indexing throughput (whether the data counted against your license quota or not), I would recommend to leverage the events of group=thruput name=index_thruput in metrics.log, like so:

index=_internal group=thruput name=index_thruput | timechart span=1d sum(kb) AS daily_GB|eval daily_GB =daily_GB/(1024*1024)."GB"

Do not attempt to use events of group=per_*_thruput to accurately determine license usage or indexing thruput as those represent a sampled measurement.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...