Why is index=_internal source=*license_usage.log* returning no data?

[joesrepsol]

Running this search from a search head (also tried the indexer) and attempting to breakdown the daily license usage for a particular index over a 1 week period. Seemed easy enough. But getting (0) records returned for even a basic query of usage data. Appears that I'm not tracking licensing usage? Whaaaaat?

Primary Search (using "oracle" index):
index=_internal source=license_usage.log type="Usage" splunk_server=* idx=oracle earliest=-1w@d
| eval Date=strftime(_time, "%Y/%m/%d")
| eventstats sum(b) as volume by st, Date
| eval MB=round(volume/1024/1024,5)
| timechart first(MB) AS volume by st

Tested for license usage data (got zero records returned as well???):
index=_internal source=*license_usage.log

I'm the splunk admin, checked that the admin role has access to all internal indexes & all non-internal indexes. Super strange. Helpful ideas???

Thanks in advance!

Joe

[joesrepsol]

It only shows one active forwards. It's one of my indexers. There is another listed as "configured but inactive forwards"... and it's the other indexer (part of a 2 Indexer cluster).

So how would I configure this to forward and whats the impact? Does it just forward the licensing_usage log? or all of the _internal? Reading up on this feature.

We have (3) splunk environments... DEV, PROD, and a PCI space. All 3 report up to the same license master and we separate the pools there. Wondering if that's an optimal configuration?

Thanks for the help!

Joe

[micahkemp]

Ahhh, it sounds like your license master isn't forwarding its logs to your indexers. This would definitely result in what you are seeing.

I highly recommend putting that forwarding in place. If you run this on your license master you'll likely see no forwarders defined:

splunk list forward-server

[DBattisto]

Can confirm. Had a similar issue when I replaced an indexer.

Created an outputs.conf file that pointed to my indexer and restarted Splunk. All works well now.

[joesrepsol]

I do have the star license_usage.log star in my query... for some reason the web GUI marks that as italics... and it didn't show up on the webpage.

It does work on my license master though... so when using a license master, is all the licensing information only available on that master?

[mayurr98]

hey @joesrepsol

Is your license master forwarding internal logs to indexers?
If you are not then have a look at this doc!
https://docs.splunk.com/Documentation/Splunk/7.0.1/Indexer/Forwardmasterdata

[micahkemp]

Your source isn't license_usage.log, but it can be matched with *license_usage.log:

index=_internal source=*license_usage.log type="Usage" splunk_server=* idx=oracle earliest=-1w@d 
| eval Date=strftime(_time, "%Y/%m/%d") 
| eventstats sum(b) as volume by st, Date 
| eval MB=round(volume/1024/1024,5)
| timechart first(MB) AS volume by st

This worked for me, removing the splunk_server, idx filters:

index=_internal source=*license_usage.log type="Usage"
| eval Date=strftime(_time, "%Y/%m/%d") 
| eventstats sum(b) as volume by st, Date 
| eval MB=round(volume/1024/1024,5)
| timechart first(MB) AS volume by st