_internal index replication causes inaccurate license reports

[jordanking1992]

All,

We noticed something very strange with our reporting. We have recently transitioned to an indexer cluster. We have always had a report that is sent out with yesterdays license usage totals. However, a day after transitioning our reports now show double the license consumption when running searches like : index=_internal source="*license_usage.log" sourcetype=splunkd type="Usage" | stats sum(b) as bytes| eval GB=round(bytes/1024/1024/1024) |fields GB| rename poolszg as "Daily License Quota" GB as "Daily License Quota Used"

Our repfactor and search factor is both set at 2 which correlates with the doubling we see per index.

What seemed to correct our reports was disabling repFactor = auto and setting it to repFactor=0 for the _internal index found under the _cluster app. Since disabling this, our license consumption now matches the trend we were seeing prior to the cluster move.

Has anyone else had this issue with inaccurate license totals when having repFactor=auto set on _internal index?

[ololdach · Accepted solution]

Hi jordanking1992,

this is how splunk calculates the license usage from _internal:

index=_internal [`set_local_host`] source=*license_usage.log* type="RolloverSummary" earliest=-30d@d   | eval _time=_time - 43200 | bin _time span=1d | stats latest(b) AS b by slave, pool, _time | timechart span=1d sum(b) AS "volume" fixedrange=false | join type=outer _time [search index=_internal [`set_local_host`] source=*license_usage.log* type="RolloverSummary" earliest=-30d@d | eval _time=_time - 43200 | bin _time span=1d | stats latest(stacksz) AS "stack size" by _time] | fields - _timediff  | foreach * [eval <<FIELD>>=round('<<FIELD>>'/1024/1024/1024, 3)]

Note the macro set_local_host that limits the results to those from one machine. I assume what you see is not a bug, it's a feature of splunk to store license usage in multiple places for redundancy. Maybe you could adapt the search from the management dashboard for your report in order to be consistent with the number splunk uses internally.

Hope it helps.
Oliver

[gcusello]

Hi jordanking1992
if you don't set repFactor=auto you don't use cluster!
Using the License consuption report [Settings -- Licensing -- License Usage Reporting] what's your result?
I see differences with your search.

Bye.
Giuseppe

[jordanking1992]

okay its displaying half of what my search is reporting...aka what is should be. So if we enabled replication of _internal, why is the search above sayings is double what i see in t [Settings -- Licensing -- License Usage Reporting]