How will forwarders behave if I add more GBs to the indexer after having low disk space?

[josefa]

Hi everyone,

I have a Splunk indexer server, which receives data from 3 forwarders and also through UDP. I got the alert of running out of space on my indexer a couple of days ago. I will add more GBs to the disk, but I'm concerned about violating the daily amount of data that my license permits, because I'm not quite sure of what the behaviour of the forwarders will be. Will they start sending the data the moment the Splunk indexer "allows" them to do so, because it has already enough space?

If so, I will need to come out with a way of sending the paused data, little by little. (for which, I also appreciate some ideas)

Thank you all in advance for your help.

[lguinn2]

Actually, the forwarders queue as much data as they can, but then they start to drop data. I would look at the Splunk logs on the forwarders to see if data has been dropped. Ditto on the indexer itself. Hopefully your alert was "running low on space" and not "indexing has paused."

Regardless of whether this has happened, it is possible that your forwarders will have to "catch up" and you will have a day of license violations. Remember that nothing "locks up" until you hit violation #5 within 30 days (or #3 for a free license). So you will probably be fine even if you get a violation. But if you think it will be a problem, you could contact Splunk Support for help. They can get you through the catch-up period.

[chanfoli]

Yes I neglected to mention the possibility of forwarders dropping data if queues are exceeded, so josefa will want to check on that as you suggested.

[josefa]

Right, I'll check my logs, hopefully forwarders will still be queuing. Thanks both!

[josefa]

Hello again,

Not sure if I should post this here because it's like a follow up of the first question, or should I create a new question.

So, I got more space on my indexer, forwarders started sending data again (I checked logs before this, apparently no blocked queues, or dropped data) but I do not seem to see any data from the period my indexer was out of disk space. I thought I understood that forwarders will send data from the day the indexer failed. If not, how can I send the missing information?

Thanks

[chanfoli · Accepted solution]

If your "catch-up" phase happens in one day and you currently don't have license warnings, you should not see any impact to search (or indexing for that matter).

Here is the documentation on license violations, warnings and how services might be impacted.
http://docs.splunk.com/Documentation/Splunk/6.2.1/Admin/Aboutlicenseviolations

[josefa]

Thank you for your answer, and yeah, sorry, I was thinking in the indexing part especifically.

As I daily use 8 of a 10GB license, I was thinking that, as the way forwarders work (that is, if something happens and it stops/stop sending data, when it resumes, it starts sending data from the point it last sent it), the flow of data to the indexer won't be the usual per-day amount, but two or three days amount for each forwarder. So on my indexer, will I see that forwarders are sending more data than usual and thus exceed my license?

PS: not a native english speaker, hope I made myself clear, so you can help me.

[chanfoli]

Yeah you will probably exceed your licensed volume for the day but splunk will only raise a warning on your indexer for each 24 hour period in which you do this, you get 4 "passes" in the past 30 days before splunk raises a violation flag (on the 5th warning). At this point search is disabled until the warnings are cleared.

If you exceed your volume once to catch up your indexes, it does not matter by how much, it is still a single warning per 24 hour period so you should not have anything to worry about.