License Expired

[bharrell]

I was using SplunkLight in trial mode and the license expired.

I'm only collecting syslog from a few small devices well under <100MB/day. So we're talking a very small environment here.

I received a message upon logon that the license expired. I clicked a message to convert to free mode, and now I seem to be locked out of any searches because of license violations.

I don't understand the license violation because the free version has 500MB/day - and I'm a fraction of that.

Here's a screenshot of the licensing page. How do I fix this? Thanks in advance.

[bharrell · Accepted solution]

Thank you for the responses -

The root problem here was that Splunk Light Trial does not automatically turn into Splunk Free.

So when the trial on Splunk Light ends, it basically goes into non-compliance and locks itself out. So it is an overage - but the limit is 0 - so after any data comes in - lockdown mode.

The only way to fix this is to wait 30 days (I think it's that long) - or contact support and get a reset-license-violation license key. They provided this to me.

Then I had to run this at the command line to get Splunk into free mode:

(for Windows I had to first CD to C:\Program Files\Splunk\bin)

splunk edit licenser-groups Lite_Free -is_active 1

Not a straightforward process - but they were nice enough to help me get the free version running. You would think there would a a button on the license page to simply fix this. But since it's free I won't complain! I appreciate having this powerful tool for free.

[grijhwani]

You should still be able to perform searches on Splunk's own internal data sources, even if general indexes/sources are not available.

Try this to look at where your data sources are:

index=_internal (source=*/metrics.log* OR source=*\\metrics.log*) group=per_sourcetype_thruput
| timechart span=1d sum(kb) by series
| addcoltotals
| addtotals

The resulting table should show you on a daily basis what is causing your excess.

Also you need to be aware that access to the general data will remain blocked as long as 3 or more days of excess remain in a window of the last 30 days. With the Enterprise licence this 5 excess days, which could explain why things broke when you switched to the Free licence - if the licence was expired by 4 days you would have exceeded a cap of zero by 4 days, but the interface would still work. On switching to free you only have a leeway of 2 over-capacity days in 30, which would cause you to be excluded from general data. The only solution to that is to address the excessive data, and then wait until the window is no longer exceeded - worst case scenario, 28 days.

[bharrell]

I need to keep this thread alive! I really want to get Splunk working again. If anyone has any suggestions I would greatly appreciate it!!

[jterry]

also, could you check for ERROR's in the logs?...

cd $SPLUNK_HOME/var/log/splunk ; grep ERROR *

[bharrell]

How do I check for errors in the Windows version?

[jterry]

ugh, well there's no "grep" in windows but i'm pretty sure $SPLUNK_HOME/var/log/splunk is still there. i'm not much of a windows user but i think the explorer has some "find in file" functionality. Also, if you're not running the latest version of SL then i'd recommend upgrading.

[bharrell]

Thanks.

My searches show a handful of errors in a few different log files. However, they are all before this licensing problem, and seem unrelated to licensing.

So - I don't see any errors in any log files that seem related to licensing. Is there a particular log file that would be helpful to analyze?

[jterry]

splunkd.log would be the likely place imo. You could also try this search: "index=_internal ERROR"

[bharrell]

Thanks. Narrowing it down to 1 log file vs. a directory full of log files is helpful.

There are very few ERROR matches - none with anything close to "internal" nearby.

Here are all the entries from the splunkd.log file today:

06-02-2016 00:00:00.287 -0500 INFO  LMStackMgr - should rollover=true because _lastRolloverTime=1464757200 lastRolloverDay=1464757200 snappedNow=1464843600
06-02-2016 00:00:00.287 -0500 INFO  LMStackMgr - finished rollover, new lastRolloverTime=1464843600
06-02-2016 00:00:28.287 -0500 INFO  LMSlaveInfo - Detected that masterTimeFromSlave(Wed Jun 01 23:59:28 2016) < lastRolloverTime(Thu Jun 02 00:00:00 2016), meaning that the master has already rolled over. Ignore slave persisted usage.
06-02-2016 00:16:07.602 -0500 INFO  WatchedFile - Will begin reading at offset=0 for file='C:\Program Files\Splunk\var\log\splunk\audit.log'.
06-02-2016 03:15:58.350 -0500 INFO  BucketMover - will attempt to freeze: candidate='C:\Program Files\Splunk\var\lib\splunk\_introspection\db\db_1463645744_1463042039_3' because frozenTimePeriodInSecs=1209600 is exceeded by the difference between now=1464855358 and latest=1463645744
06-02-2016 03:15:58.366 -0500 INFO  DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='C:\Program Files\Splunk\var\lib\splunk\_introspection\db', pendingBucketUpdates=0 .  Reason='Removing bucket, bid=_introspection~3~7DC151F2-F7FF-4C9F-9D41-FF45B6DA353D'
06-02-2016 03:15:58.491 -0500 INFO  BucketMover - AsyncFreezer freeze succeeded for bkt='C:\Program Files\Splunk\var\lib\splunk\_introspection\db\db_1463645744_1463042039_3'
06-02-2016 03:15:59.288 -0500 INFO  DatabaseDirectoryManager - idx=_introspection Writing a bucket manifest in hotWarmPath='C:\Program Files\Splunk\var\lib\splunk\_introspection\db', pendingBucketUpdates=0 .  Reason=' frozen_buckets'
06-02-2016 06:59:42.116 -0500 INFO  WatchedFile - Checksum for seekptr didn't match, will re-read entire file='C:\Program Files\Splunk\var\log\splunk\metrics.log'.
06-02-2016 06:59:42.116 -0500 INFO  WatchedFile - Will begin reading at offset=0 for file='C:\Program Files\Splunk\var\log\splunk\metrics.log'.
06-02-2016 06:59:42.163 -0500 INFO  WatchedFile - Will begin reading at offset=24991320 for file='C:\Program Files\Splunk\var\log\splunk\metrics.log.1'.

[jterry]

really only looking for "ERROR" log entries, not "INFO". Also note that the search: "index=_internal ERROR" is a directive to show all "ERROR" events from the _internal index.

What about that upgrade option?

[bharrell]

I have screenshots to share- but I'm apparently only allowed 2 posts per day. This is ridiculous.

What upgrade option? You mean not free? I'm still considering it when my trial ran out. I was under the impression that Splunk was free for a very small environment - like mine. It's a bit concerning that right out of the gate it flat out doesn't work.

[jterry]

version upgrade, though license upgrade is always an option. If you're not using the latest version then i recommend upgrading the version. i sympathize w/you but i assure you it does work "out of the gate".

[bharrell]

I did download and install the latest version yesterday after the license didn't work. It made no difference.

All I know is that In my environment, I followed the instructions, did a straightforward install of SplunkLight, used it successfully, the trial expired, and attempted to convert to free mode... and it does not work because of a license error. Seems like a license error should be pretty easy to diagnose/repair.

[jterry]

hm, ok. can you post the results of the search ("index=_internal ERROR") & we'll go from there.

[bharrell]

It won't let me post screenshots here - I'm out of Karma for the day... and I can't paste the text output because it's too long. Is there not a better place we can chat back and forth?

[dkoshe_splunk]

When you switched to Light Free mode, did you restart Splunk Light?
Can you put the output of following commands (assuming Linux):

$SPLUNK_HOME/bin/splunk list licenser-groups

and

$SPLUNK_HOME/bin/splunk list licenser-pools

[bharrell]

Thanks for the responses.

I did restart Splunk after switching to free mode. I also rebooted the server for good measure (twice!)

I am running on Windows. But here are the outputs of the commands from the CLI:

splunk list licenser-groups

    Enterprise
            is_active:0
            stack_ids:

    Forwarder
            is_active:0
            stack_ids:
                    forwarder

    Free
            is_active:0
            stack_ids:
                    free

    Lite
            is_active:0
            stack_ids:
                    lite

    Lite_Free
            is_active:1
            stack_ids:
                    lite_free

splunk list licenser-pools

 auto_generated_pool_forwarder
         description:auto_generated_pool_forwarder
         effective_quota:1048576
         quota:MAX
         slaves:
         stack_id:forwarder
         used_bytes:0

 auto_generated_pool_free
         description:auto_generated_pool_free
         effective_quota:524288000
         quota:MAX
         slaves:
         stack_id:free
         used_bytes:0

 auto_generated_pool_lite
         description:auto_generated_pool_lite
         effective_quota:0
         quota:MAX
         slaves:
         stack_id:lite
         used_bytes:0

 auto_generated_pool_lite_free
         description:auto_generated_pool_lite_free
         effective_quota:511705088
         quota:MAX
         slaves:
         stack_id:lite_free
         used_bytes:49634610

In regards to the error log, what is the best way to review this in Windows?