All Apps and Add-ons

license manager results differ from metrics.log indexed volume

att35
Builder

Hi,

This morning I saw message regarding daily indexing volume being exceeded. From the license manager, it seems that one of the indexers processed around 8 GB(We have 4 Indexers, with master/slave configuration).

I ran the following search to find out which source indexed the most, to figure out the sudden spike.

index="_internal" source="*metrics.log" group="per_sourcetype_thruput" | chart sum(kb) by series | sort - sum(kb)

But when I add the individual volume, it comes out around 4 GB, half of what license manager reports. Am I looking at a incorrect source(metrics.log) for this data? This search was executed directly on the indexer which show high index vol, whereas license manager is on the search head(which also acts as license master).

If we assume license manager to be the correct result, how can I further drill down to find the source or host that sent most data.

Many Thanks,

~ Abhi

0 Karma
1 Solution

masonmorales
Influencer

There's a pretty good app for diagnosing license utilization available on Splunkbase: Splunk Utilization Monitor (SUM)

View solution in original post

masonmorales
Influencer

There's a pretty good app for diagnosing license utilization available on Splunkbase: Splunk Utilization Monitor (SUM)

att35
Builder

Hi Mason,

First of all, Thank you writing a nice all-in-one app for licensing.. 🙂

In a master-slave setup, where should we install this? I believe it should be on license master.. correct?

I installed it on both, the license master as well as one of the search peers(license slave), but it is unable to populate any data. The drop down for Splunk server & pool display "Search produced no results".. whereas the panels are either "no results found" or "NA".

Is there any manual configuration that I might be missing?

Thanks,

~Abhi

0 Karma

masonmorales
Influencer

Hi Abhi,

That's correct, you should install the app on your license master. The drop-downs are populated using the splunk_server and pool fields from:

index=_internal source=*license_usage.log

Please ensure that you are able to search index=_internal on the user account that you are using the App with. If you cannot, you will need to login with a user that has higher privileges (e.g. admin), or go to Settings -> Access controls -> Roles -> (Your Role) and ensure that the _internal index is listed under the "selected search indexes"

att35
Builder

Hi Mason,

That was the issue. The role used to view the license usage did not have permissions on the internal indexers. It's working perfectly now.

Also, I did notice that there's some difference in the indexed volume reported by SUM as compared with the default "License Usage" app.. In the default app, today's usage is 2.54 GB, but within SUM, the first panel for "License Pool Utilization (GB) (Today)" reports 4.41 GB.

I'll do some more checking on my side to see if there's something in the query.

Thanks again for the help,

~Abhi

0 Karma

yannK
Splunk Employee
Splunk Employee

you should ask the app's author -> https://splunkbase.splunk.com/app/2678/

or open the searches and look at them, maybe the host is hardcoded in it.

0 Karma

yannK
Splunk Employee
Splunk Employee

Remember that the metrics.log contains a sample of top 10 only
http://docs.splunk.com/Documentation/Splunk/6.2.2/Troubleshooting/Aboutmetricslog#Thruput_messages

for any license usage, You should trust the searches based on license_usage.log

masonmorales
Influencer

Thanks for pointing this out. I've updated two of my panels that were using metrics.log in the SUM app.

0 Karma

stephane_cyrill
Builder

Hi ,
1- The sum of the indexed volume may have been above the limit of your licenses.

2- assuming that the license manager result is correct, to find the soure or host that sent most data you :
find the max(kb) by host or by source over that period of time.

3- for example:

index="_internal" source="*metrics.log" group="per_sourcetype_thruput" | chart max(kb) by host

0 Karma

nivedita_viswan
Path Finder

I have the same issue!

We have a 20GB license.
The sum of the indexed volume for the last day is just 6GB! I still see a violation for yesterday. Here is the search query:

index="_internal" group="per_index_thruput" | search series!="_audit" | search series!="_internal"| search series!="_introspection"| eval gb=kb/1024/1024 | timechart span="1d" sum(gb) by series

Furthermore, this was the 5th violation so my search has been suspended though the indexed volume was just 6GB the last day.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...