Getting Data In

Adding more forwarders when licenses are almost maxed out

skoelpin
SplunkTrust
SplunkTrust

If I have a 100GB licenses and my current indexing rate is around 90GB/day and I turned a forwarder on which had 15GB of log files total but only produced around 1GB/daily. Would this cause me to go over my current 100GB licenses since it will retroactively index all the logs or will it only index up to my license amount of 100GB then grab the rest later when space is available?

0 Karma
1 Solution

javiergn
SplunkTrust
SplunkTrust

It all depends on how you configure your inputs.
If you want your forwarder to backfill all the existing logs and not just real-time ones, then it is likely going to blow your license that day based on what you are saying, but because you can violate the license up to 5 times within a 30 day period before it stops searching, I wouldn't worry too much.

Worst case contact your account manager and request a license violation reset explaining that you are backfilling old logs. It shouldn't be a problem.

View solution in original post

ddrillic
Ultra Champion

-- .. or will it only index up to my license amount of 100GB then grab the rest later when space is available

It doesn't seem that such logic exists in the product.

As a workaround you can control it via the ignoreOlderThan parameter in the inputs.conf file -

[default]
index=xxxx
ignoreOlderThan = 4d

You can set it up for 4 days, for example, and increase it on a daily basis and go further back in time while monitoring the license usage.

javiergn
SplunkTrust
SplunkTrust

It all depends on how you configure your inputs.
If you want your forwarder to backfill all the existing logs and not just real-time ones, then it is likely going to blow your license that day based on what you are saying, but because you can violate the license up to 5 times within a 30 day period before it stops searching, I wouldn't worry too much.

Worst case contact your account manager and request a license violation reset explaining that you are backfilling old logs. It shouldn't be a problem.

skoelpin
SplunkTrust
SplunkTrust

By default it will backfill retroactively right? What .conf file would I modify to prevent this?

0 Karma

javiergn
SplunkTrust
SplunkTrust

If you have 15GB of log files but only want to read 1GB/daily then configure your inputs.conf to read just today's file.

For example, assuming there's 15GB of *.log files under /var/log that you want to read:

[monitor:///var/log/*.log] -> this will tell Splunk to read them all

[monitor:///var/log/*.log] -> this will tell Splunk to ignore files older than 1 day
ignoreOlderThan = 1d

If you still need those 15GB of logs then go for it as you will only violate the license one day out of the 5 you have on any 30 day period.

Or maybe you want to configure Splunk to read the 15GB logs on a weekend when the license usage is lower.

There are plenty of options and none of them is wrong

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...