- Date published:
- Author:Kevin Johnson
Lessons Learned from Backup Failures
The old adage goes that you learn more from your failures than your success. But it’s a painful education.
Kevin Johnson, NFINIT VP of Product Strategy, has a treasure trove of lessons learned from others’ backup disasters. Small-and-medium sized businesses highlighted here implemented backup plans that for the most part, were sound, until something went disastrously wrong.
On-Premise Backup Plan is All Wet
The CEO of a regional retail chain with over 25 stores called late one night in a near panic. The organization was meticulous about maintaining systems with one exception – backup servers were in a secure room at the corporate headquarters, rather than off-site.
A sprinkler head sprung a leak directly above the server stack. Fortunately it was detected early before major damage was done. As their team was physically moving servers, they were coordinating with our team to deploy off-prem cloud backup.
Human Error Causes Tax Time Stress
An accounting office had completed the April 15th rush and the IT team was moving on to maintenance items. One of the items they were addressing was updating virus protection for their servers and PC’s. The IT staff installed the latest version of their chosen virus protection software and life was good. Except for one little detail that they discovered months later.
They had failed to exclude .tax files from the “bad” list. The virus scan did its job and automatically cleansed .tax files from the system.
Fast forward to tax time almost a year later and the accountants found that previous year’s files weren’t there. This was a HUGE issue for their business.
Fortunately, they were backed-up in our cloud and we were able to help them restore the files. The IT team changed the virus scan to stop cleaning the tax files backup, and the customer was back in business.
“Forklift Upgrade” Gone Bad
An employee in a manufacturing company decided to enjoy a liquid lunch one day. It was about 3:00 in the afternoon when this employee, driving the forklift, took a left turn instead of a right and plowed through the warehouse wall into the bathroom. It took out all the bathroom piping and caused a major flood.
Unfortunately for this employee and the company, the bathroom was next to the server room. None of the equipment survived the flood, bringing the facility and entire company to a standstill. They couldn’t process or fill orders.
They were lucky though (actually smart). Fortunately, they had taken the time to think through their mission critical applications, determine which data was essential to running the business and were replicating them to our cloud backup and disaster recovery services. Within minutes they had their critical infrastructure up and running in the cloud and were back in business.
We never heard what happened to forklift driver.
Car Break-in Disrupts Backup Plan
A sizable family-run retailer was using old-school media for backup. For years, a very reliable employee transported daily backups to an offsite storage location without incident. Occasionally, she detoured to the grocery store or to meet a friend before delivering backups to the offsite location.
Then that all changed. The employee stopped for dinner with friends and upon leaving, found that her car had been broken into. All of the backup tapes and drives were gone! As was the sensitive customer and company financial information.
For the next several weeks the company worried that this information would leak publicly. They spent months and months working with compliance folks and notifying customers of the potential risks involved with the compromised data.
The company thought that they were doing the backups right. 20/20 hindsight showed that the 3rd offsite copy of data needs to be in the CLOUD and must be encrypted.
The Best Location for Backup is 32110
In this case, 32110 isn’t a ZIP Code but a foolproof strategy.
3 – Different copies of date
2 – Different media
1 – Of which is offsite
1 – Of which is immutable or air-gapped
0 – Errors after recoverability verification
Don’t be an April Fool!
NFINIT is encouraging all customers to review backup practices during the month of March in preparation for World Backup Day, March 31. Reach out to us if you’d like help reviewing your backup strategy and protocols.
Kevin Johnson is NFINIT VP of Product Strategy.