Splunk Search

Splunk search command for daily indexed volume?

remy06
Contributor

Hi,

I am currently testing out two searches to report and alert on the daily indexed volume.

The first search is as follows.:
index=_internal sourcetype=splunkd LicenseManager-Audit todaysBytesIndexed licenseSize | eval todayMb=(todaysBytesIndexed/1024/1024) | eval percentUsed=round(100*todayMb/licenseSize,2)

The second search will have:
index=_internal source=*metrics.log splunk_server="*" group="per_host_thruput" | eval MB=kb/1024 | chart sum(MB) by series | sort sum(MB) | addcoltotals col=t | fillnull value="[ Total Indexed Volume ] last 24 hours" Host(s)

Both are scheduled to run at midnight daily and should report on the indexed volume for that day.

However,I've noticed the reports for today,these 2 searches show different indexed volume which are misleading.

Which command is more accurate?Any suggestion or advice?

1 Solution

sideview
SplunkTrust
SplunkTrust

The LicenseManager search will not count things like index=_internal and index=_audit data, because that volume doesnt count against your license. And the per_host search does.

However you can use the per_index_thruput numbers and then filter out the indexes that have leading underscores.

index=_internal source=*metrics.log splunk_server="*" group="per_index_thruput" | eval MB=kb/1024 | stats sum(MB) by series | rename series as index | search index!=_* | sort sum(MB) | addcoltotals | fillnull value="[ Total Indexed Volume ] last 24 hours" index

If I run this search against the data from yesterday and compare it to the licenseManager's search from today (necessary because the licenseManager runs just after midnight and its talking about yesterday), then the numbers seem very close to eachother but oddly they are not equal. Im not sure why.

View solution in original post

yannK
Splunk Employee
Splunk Employee

Since Splunk 4.2, a new log record the license usage on the license-master.

you can use new searches for the detail per : sourcetype/host/source per pool
see http://wiki.splunk.com/Community:TroubleshootingIndexedDataVolume

sideview
SplunkTrust
SplunkTrust

The LicenseManager search will not count things like index=_internal and index=_audit data, because that volume doesnt count against your license. And the per_host search does.

However you can use the per_index_thruput numbers and then filter out the indexes that have leading underscores.

index=_internal source=*metrics.log splunk_server="*" group="per_index_thruput" | eval MB=kb/1024 | stats sum(MB) by series | rename series as index | search index!=_* | sort sum(MB) | addcoltotals | fillnull value="[ Total Indexed Volume ] last 24 hours" index

If I run this search against the data from yesterday and compare it to the licenseManager's search from today (necessary because the licenseManager runs just after midnight and its talking about yesterday), then the numbers seem very close to eachother but oddly they are not equal. Im not sure why.

jlaigo2
Path Finder

This is what I have been looking for thanks for your answer:)

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...