All Apps and Add-ons

IIS logs and splunk license usage

bjmennen
New Member

Hi All,

Running Splunk 6 and using the Universal Forwarder (Version 6.0.182611) to forward IIS to splunk. Indexing is working correctly however we have had license breaches in the last 2 days since adding the IIS source where I believe we should have had spare capacity.

Question:

The size of the log files on the server (~120mb yesterday) doesn't seem to match the indexing size even closely. Running the search for yesterday (Only 1 IIS server currently so only 1 sourcetype=iis):

sourcetype=iis | eval size=len(_raw) | stats sum(size)

This search shows it at around around 700mb. Is there a trick to IIS and log usage? How would a 120mb log file consume so much more that its actual size?

This question seems similar to http://answers.splunk.com/answers/129381/iis-log-over-my-licensing which no one has responded.

Any tips, clues, links etc....

Brad

0 Karma

gavin_staplesau
Explorer

crcSalt was not enabled for the input.
It ended up being a bug in an older version (6.0.1). Upgrading splunk and the universal forwarders to 6.0.6 fixed this issue.

jonathansaenz
Explorer

I look forward to updating my forwarders from 6.0 to see if this alleviates our problem. This has been plaguing my production instance of splunk for months.

0 Karma

jonathansaenz
Explorer

I can now confirm that this fixed my issue as well. Thanks Gavin!

0 Karma

gavin_staplesau
Explorer

thanks bmacias84,

the query shows that it was indeed the new IIS logs that were breaking the license.

sourcetype=iis  | eval raw=_raw | convert ctime(_indextime) AS idxtime | stats count AS event_count dc(idxtime) as idxtimes_count, values(source), values(idxtime) by raw  | where event_count > 1

is showing that every event is being indexed multiple times. I am still working with support to solve the problem, but I will post any resolution here in case it helps anyone else.

0 Karma

bmacias84
Champion

I seen that occur when enabling ** crcSalt** on the inputs.conf. If a file role splunk will believe its a new file to index.

0 Karma

bmacias84
Champion

If you want to find out the usage the best way is to use the _internal index.

The flowing search will break license usage by sourcetype and index.
This should get you started


index=_internal source=*license_usage.log type=Usage earliest=-2d@d latest=-1d@d | rename st as sourcetype, idx as index, b as bytes | fields sourcetype index host bytes | stats sum(eval((bytes/1024)/1024)) as MB by sourcetype index

Cheers,

0 Karma

gavin_staplesau
Explorer

same here! splunk reporting 22GB of IIS logs for a day, when only 22Mb of IIS logs were found on the server

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...