Can indexer be taken off line to prevent license violation due to getting a backlog of data?

[RVDowning]

We have had a number of forwarders down for a considerable period and now that they are forwarding their data we are exceeding our daily limit. Today will be the third day over limit on an enterprise license.

Any way to keep the indexer from indexing while still allowing searches? Then if I see the limit is being neared, I could stop the indexer from indexing for the day. Not sure how many days this will continue, but probably a few thereby putting us in violation of our license.

I guess the drastic step would just be to stop splunk if the limit was being neared and continue doing that until the backlog has been cleared.

[kristian_kolb]

It sure sounds as if there is a lot of backlog, so to speak. Quite often, the solution is to try to squeeze in as much log as possible during a day, since there is no difference between violating the license by 1 byte or 100 GB. A violation is a violation.

Anyway, I would say that it's probably better to stop the forwarder than the indexer, but this depends a little on how your landscape looks. For this I am assuming that your ordinary forwarders are sending considerably less than the recently restarted forwarders. Thus you'd have to make a little guesswork on when to stop those forwarders, so that you'll still stay below the license limit with the traffic from your regular forwarders.

Shutting down the indexer will not be as good, as you'll start getting behind for all your forwarders.

Also, depending the load on your network and the hosts involved, you might want to increase the speed at which the forwarder sends data. The default limit is at 256 Kbps, but can be configured in limits.conf on the forwarder, thus shortening the time needed to transfer the logs to the indexer.

Another question is whether all that old data is still needed. If you don't need all of it, perhaps you can ignore parts of that data with an ignoreOlderThan setting in inputs.conf on the forwarder side.

Should you be locked out, you can always request a reset-key, which will remove the violations. Just open a case with Splunk Support, and describe the circumstances. Normally this will work fine, but don't do this too often... 🙂

/K

[kristian_kolb]

Well, the idea behind that reasoning is that you'll get more data through faster. So while you have violations, they'll still be fewer than 5 (hopefully).

If this does not suit you, you now know of setting that can lower the total throughput. Don't go too low, though, or you'll never catch up.

[RVDowning]

Well, we have 196 forwarders and I'm not sure which ones have the big backlogs to index. Increasing the speed the forwarder sends data to the indexer is probably the opposite of what I would need. The faster data is transferred, the faster I exceed my daily allocation.

[linu1988]

Easy,
create a new pool which has a limit less than the license limit. It will stop indexing for the day and start from the next day. You may loose the data if not put in persistent queue.

Thanks