Installation

How to fix license usage not being calculated correctly after applying a Linux yum update?

leeraym
Path Finder

I’m running Splunk 6.4.2 on RedHat Enterprise Linux 6.8. I have a 10GB license and usually use up around 4GB per day. Certain days will go up to around 7GB, but I haven’t exceeded my license in a long time (see my 30-day license usage screenshot).

alt text

I ran a 'yum update' to apply a kernel patch and some others to all my Splunk indexers and search heads on Nov. 2 and rebooted the servers. However, now Splunk thinks all my indexed data applies to just one day (the current day) and it says I’ve indexed 32.332GB for today. As you can see from the 30-day usage report above, the usage data stops at Nov. 2, which is the day I rebooted. In the 9 days since then, I would’ve indexed around 30+GB, so that backs up my assumption that Splunk is globbing all my indexing since Nov. 2 into one day’s license allowance. Here's "today's" license usage report:

alt text

Has anyone ever run into this problem? The Splunk licensing alert says I need to correct this by midnight to avoid violation.

The date on my servers is correct, and I definitely have not been saving up over a week's worth of logs to index all on today.

Thanks,
Ray

Labels (2)
0 Karma
1 Solution

leeraym
Path Finder

Don't know what the root cause was, but perhaps restarting Splunk fixed it. If anyone knows if there's a log file that shows the process that runs at midnight to generate the "type=RolloverSummary" events in $SPLUNK_HOME/var/log/splunk/license_usage.log, please let me know so that I can check for errors. So far, I have not been able to find such a log.

View solution in original post

0 Karma

leeraym
Path Finder

Don't know what the root cause was, but perhaps restarting Splunk fixed it. If anyone knows if there's a log file that shows the process that runs at midnight to generate the "type=RolloverSummary" events in $SPLUNK_HOME/var/log/splunk/license_usage.log, please let me know so that I can check for errors. So far, I have not been able to find such a log.

0 Karma

leeraym
Path Finder

Here's the output of a search I did on the license_usage.log files from Nov. 2 - 11. As you can see, I haven't exceeded my 10GB license and there was data indexed every day in that period....not just the 30+GB today that the License Usage Report shows.

alt text

0 Karma

leeraym
Path Finder

I looked at one of the $SPLUNK_HOME/var/log/splunk/license_usage.log files on my license master which had events that straddled 2 days (e.g., 11-10-2016 23:59 - 11-11-2016 00:00). I did not find any instances where type=RolloverSummary. Everything was type=Usage. It is my understanding that the License Usage Report searches using type=RolloverSummary. So now the question is, why did my license master stop summarizing the previous day's license usage and put it into a type=RolloverSummary in license_usage.log?

0 Karma

sk314
Builder

Are there any blocked queues on the license master? index=_internal sourcetype=splunkd blocked=true

0 Karma

leeraym
Path Finder

There are occasional blocked queues, mostly from Universal Forwarders on chatty Windows servers, but none from the License Master.

This problem seems to have cleared up over the weekend though. I have "type=RolloverSummary" events in my license_usage.log from the past 3 days and the license usage report looks sane again.

Still unsure what the root cause of this was, but I will set up an alert to look for "type=RolloverSummary" in license_usage.log and email me if it doesn't find any.

0 Karma

sk314
Builder

running out of options. Is there any time difference between the License Master and the other servers? If you have the option, contact Splunk support with your problem and request for a reset warning license.

0 Karma

leeraym
Path Finder

No time difference between any of my Splunk servers...they're all synced to our NTP server. Yeah, I guess I might need to request a reset if I don't get this ironed out. Thanks for your suggestions. Your comment about _lastRolloverTime and lastRolloverDay made me look into the type=RolloverSummary that is missing from my license_usage.log files.

0 Karma

sk314
Builder

Do you have a License master? Is it separate from the Indexers? What does the yellow exclamation mark say?

0 Karma

leeraym
Path Finder

Yes, my search head is the license master and is separate from the indexer.

The yellow exclamations say "Unable to distribute to peer named ... because peer has status = "Down". " This is fine because I've purposely shut that peer down.

Nothing else has changed since Nov. 2 besides the RedHat patching.

0 Karma

sk314
Builder

Have you disabled indexing on the license master and are instead forwarding internal logs to the peers? In which case, can you confirm that the outputs.conf on the license master points to the new peers and not the disabled peer? My guess is that your license master is unable to get the _lastRolloverTime and lastRolloverDay because its trying to send its internal logs to the peer you disabled.

leeraym
Path Finder

Actually, the only outputs.conf I have on the search head (license master) are all the defaults. The license master is set to monitor its own $SPLUNK_HOME/var/log/splunk, and that hasn't been disabled.

Here's the output of a search I did on the license_usage.log files from Nov. 2 - 11. As you can see, I haven't exceeded my 10GB license and there was data indexed every day in that period....not just the 30+GB today that the License Usage Report shows.

alt text

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...