Installation

Why does metrics.log search on a clustered environment display double the license usage value from the license master?

DavidHourani
Super Champion

Hello Splunkers,

I am trying to understand the results of the following command on a Search Head (SH) in a clustered environment (Search Head cluster + indexer cluster):

index="_internal" source="*metrics.log" per_index_thruput  | eval GB=kb/(1024*1024) | timechart span=1d sum(GB) | convert ctime(_time) as timestamp

Using it on a standalone instance this gives me the same values as my license usage. When I use it on a SH in a clustered environment, it's giving me double the license usage value from the license master.

Any idea why this could be the case? Does replication have anything to do with it?

Regards,
David

Labels (3)
0 Karma

twinspop
Influencer

Metrics logs are produced by all Splunk instances. If you have forwarders sending logs to an indexer, for example, you will potentially have 2 sets of metrics logs for any one category/series. You can get around this problem in 1 of 2 ways:

1) Isolate your search to your indexer(s) using host=your_indexer. The drawback here is that metrics are samples. By default, splunk only tracks the top 10 in each category. You can adjust this with maxseries under the metrics stanza in limits.conf.

2) Isolate your search to only the forwarders. One way to do this would be to EXCLUDE the indexers from your search: host!=your_indexer. Splunk still only tracks the top 10, but each forwarder has their own top 10, and you're more likely to get a complete(r) picture.

0 Karma

twinspop
Influencer
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...