Powering Down a Splunk Indexer for Maintenance

[dalgibbard]

Hi all,
Basically for example's sake; lets say i have 45 web server clients logging to a Splunk Indexer and it is the one and only box. Let's then say that i need to power down the Indexer for 45minutes.

A) What would happen to the logs? Would they still be indexed on return on the Indexer?
B) I presume that the logs will remain on the client until the indexer returns?

Thanks in advance.

[gkanapathy]

You should also be certain not to power down forwarders unless an indexer is running and is available to receive any events that get flushed from the forwarder queue during shutdown. Otherwise, those events have nowhere to go and are lost. (If you just leave the forwarders running, it will be fine, as the queue will just block and wait.)

[DrewO]

Dalgibbard for your boxes that use syslog instead of a Splunk light forwarder, you have a few options. Following bojanz's suggestion you could stand up syslogNG on your indexer, except that wouldn't help since you are asking to power down your Splunk indexer for 45 mins which means nothing would be running on that indexer. Not knowing too much about your setup the easiest way might be for you to install light forwarders on all your systems and make sure their buffer settings are set high enough to allow you to power off for 45 mins. Another option would be to stand up a separate system running syslogNG and have that collect the logs from your non-light forwarder servers, and have your indexer use its monitor function to monitor the directories on the syslogNG server. The monitor function is resilient and will remember where it was in the files on the syslogNG server when it went down and will pick back up where it left off.

[bojanz]

Syslog logs would be lost, yes. If you want to keep those, I suggest that you install syslog-ng (or equivalent) on your indexer and configure syslog-ng to log received data into text files. Then configure Splunk to index those text files (which you can safely delete after they have been indexed). Now you can restart Splunk whenever you want without losing syslog data since they will be stored in text file and picked up by Splunk when you restart it.

[dalgibbard]

It might be worth adding to this also; as some of our boxes log using Syslog rather than the Splunk daemon; therefore the UDP traffic would be lost, and these entries would not be logged...

Correct me if i'm wrong of course!

[Brian_Osburn · Accepted solution]

They should just queue up on the indexer to a point.

Check out http://answers.splunk.com/questions/1114/what-happens-to-my-events-at-splunk-light-forwarder-when-th...