Managing through a Power Outage
On Friday night (13th) there was a localised powercut at about 11pm and we got to see if all the APC kit worked as intended. In our main server room we have a number of systems including Hyper-V Hosts and NAS devices.
After first finding a candle I walked into the office heard that distinctive buzz of APC batteries off the mains. We had clusters of lights flashing in parts of the room as the workstations sat silently. After checking outside to see that the power was off in about 3 blocks all round us I went back in and wondered how the servers would cope.
A few minutes later it was obvious that this was not just a quick blip so it was time for the network to go down. At about 5 minutes the NAS boxes powered down – leaving about a half charge in the units supporting them. This is important as the devices are not active units with detailed battery status so I would rather keep them charged and under load than flat for any period of time. If power cuts were more common (which thankfully they are not) then this would have a long term effect on the life of the battery units. A full discharge seems to take a few % of the total capacity each time.
Now the servers started to busy themselves with activity. The noise levels rose as the APC units triggered the boxes to shutdown. Now if they had been all physical boxes I would have put them all into hibernate mode earlier, but you cannot run Hyper-V and hibernation on Server 2008 R2. All the hosted servers were quickly suspended and then the systems did a full managed shutdown.
With silence all around it was a good oportunity to see what re-cabling could be done, although with no mains lighting any work was not a good idea.
About an hour later the power came back on and all the servers started to roar!! Remembering that a virtual box has to startup twice I watched as everything settled down, modems, routers,l access points, NAS boxes, and even a workstation. About 10 minutes later my phone buzzed at me to tell me the email was back and flowing – now for bed.
The next day most of my servers were happy and unconcerned about the previous nights “issues”. Only one little server in the office had no APC unit (well it had but there was no battery in it!) and it complained about the power state. So with a little bit of planning I can trust the network to handle a power outage with no interaction and no real disruption.