The phone rang at 4AM and I knew that it wasn’t going to be good news. No one ever rings at 4AM with good news, it is always something that has happened or gone wrong. It was a call about a server upgrade that had gone wrong and a customer needed to get their data back as quickly as possible as their employees would start coming in to work at 6AM.
The customer’s server had a couple of issues, it was running out of space and the backup wasn’t working. The customer thought the lack of space was the biggest issue and he planned to upgrade the hard drives after hours. We had talked through it and strongly suggested that maybe the backup should be fixed first, especially given that the disk upgrade was a little tricky.
Suffice to say that the disk upgrade turned out to be very tricky. It had gone pear-shaped and they had tried a couple of things to sort it out, but without any luck.
I was onsite by 4:30 and started going through how we could get them back online. There were a couple of issues. They had hit a minor issue with the hard drive upgrade and tried a range of solutions to fix it, but each fix had only made things worse. By the time they called me, it was too late. Their data was gone, never to return. We shipped the drives off to a specialist recovery place, but as we thought, the information was lost forever.
The next step was to restore from the last good backup. It was 4 days old. Tragically, that was $100,000 of turnover. All of those transactions, bookings, customer updates and planning were gone, so not only was $100,000 lost, but a lot of customer service issues were going to pop up over the next few months.
After realising that 4 days’ worth of data was gone, I started looking at the transaction flow in the business and how jobs were processed and given in paper format to drivers. So we got their team looking for all of the invoices. Some were in bins, some were in drivers cabs and some had been filed away. All in all, we managed to find $80,000 worth of transactions, which wasn’t too bad a result. It could have been a lot worse. The next week the customer placed an order for a backup system that met their needs of their business. These sort of things can and do happen and they happen fairly frequently. Human error is the most common cause.
What were the lessons learnt from this?
Check your backup daily to ensure it works. When it doesn’t work, get it fixed.
Conduct a trial restore to ensure you can restore when you need to.
Work out how long you can be down without your server and factor that into your planning.
Work out how many transactions you can afford to lose – 1 hour, 1 day, 1 week – and factor that into your planning as well.
Ensure there is a documented procedure on how to restore. Restoring data is stressful and good documentation is critical in ensuring that the restoration is quick, complete and even possible.
Don’t rely on luck to keep your companies data safe. If you have any questions, please call James and the Team at Extreme Networks on 03) 9785 7162. We are the only company on the Mornington Peninsula with staff accredited in Disaster Recovery practices, and our solutions keep Hospitals, Accountants, Lawyers and hundreds of other businesses running each and every day. Our software is able to automatically check your backup and take the stress out of doing backups.
