Getting Data In

How to limit the maximum daily indexing volume

nrjsh1988
New Member

I had installed a Enterprise trial license which was going well for me with searching and reporting.But after installing "Splunk on Splunk" app and "Splunk App for Windows Infrastructure",
I got a warning message that "Daily indexing volume limit exceeded today" after which I was not able to search. I know Splunk does not stop indexing your data, it only blocks search while you exceed your license.

Is it possible that I can configure a threshold so that Splunk can stop indexing after a certain limit ?

Tags (2)
0 Karma
1 Solution

MuS
SplunkTrust
SplunkTrust

Hi nrjsh1988,

No, this is not possible. But you can setup filtering and routing to the null queue to get rid of unwanted events, take a look at the docs about Filter event data and send to queues

Update: to be precise, it is somehow possible to limit the daily indexing amount with Splunk ... BUT ... not in a way you expect! You can limit the daily amount of indexed data if you only use Universal Forwarders and limit their throughput by using the limits.conf and set the according [thruput] so the sum of all UF's transmitted data will not exceed your daily license volume.

hope this helps ...
cheers, MuS

View solution in original post

tweaktubbie
Communicator

Pretty weird it isn't still included in some config file, an option to 'hold indexing' when e.g. 99% of the daily license is used. Limits.conf (what's in a name) can't limit the license usage...

For multiple test environments with their own splunk environments where unexpected things occur daily, you don't want the hassle of having to monitor the license during the day. Or when something in a long weekend is bouncing and dumping, you don't want this annoying license warning coming at ya.

Can't be so difficult to have this option. Yes, you can with a larger environment do some remote searches and make a splunk-stop script for the heavy forwarder. But on a single server it'll bring the whole webinterface down as well. On production environments I understand it's not desired usually to implement.

0 Karma

davidpaper
Contributor

nrjsh1988,

I disagree with MuS here.

limits.conf: [thruput] maxKBps= will do what you want, if you set it on the indexer itself. If you have multiple indexers, each indexer gets a fraction of the total.

Be aware there are caveats to this solution, which I will leave an an exercise to the reader.

0 Karma

jpass
Contributor

This should be a feature available per input imo. for now, gfuente's solution should be easy enough to implement.

-write a perl or python script to be used as a scripted input -the script starts by using the RESTapi to check for current values like indexed t -if the license has room, the script checks for new events by reading them from a file,folder,database or whatever -last, print the events so splunk can index them -don't forget to account for the amount of space remaining on the daily useage allowed vs. the amount your new events that will be added -also if you have multiple input scripts doing this you would need to check if any of they are running and wait until they're done before executing

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi nrjsh1988,

No, this is not possible. But you can setup filtering and routing to the null queue to get rid of unwanted events, take a look at the docs about Filter event data and send to queues

Update: to be precise, it is somehow possible to limit the daily indexing amount with Splunk ... BUT ... not in a way you expect! You can limit the daily amount of indexed data if you only use Universal Forwarders and limit their throughput by using the limits.conf and set the according [thruput] so the sum of all UF's transmitted data will not exceed your daily license volume.

hope this helps ...
cheers, MuS

nrjsh1988
New Member

But currently I am not using any universal forwarders, it is just a test machine in which i had previously installed some apps due to which my license violated.

0 Karma

gfuente
Motivator

Hello

Perhaps you could write an script, that once that the license reach certain point, you turn off receiving (using a rest endpoint) so Splunk stop indexing anything new (at least from forwarders), you could do the same for any other tcp or udp, scripted inputs...

Regards

0 Karma

MuS
SplunkTrust
SplunkTrust

Sure you can set an alert. Regarding the limit, see my update

0 Karma

nrjsh1988
New Member

Thanks MuS, apart from it if it is not possible, I think I can also set an alert on say 400 MB exceeded, so that I can take preventive measures before my license violates.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...