Daryl C. W. O’Shea: Lazy Network Engineer

With non-secure backups the best backup is the one nobody else knows about

February 1st, 2007

Twice this month I was reminded why when you’re relying on someone else to backup data to any sort of removal media (or any media at all really, but removal media seems to be the worst) you should always if at all possible keep as many duplicate copies of the data being backed up on the same network (or if necessary, same machine) as the source data.

Why?  Without fail, the most technologically clueless person in the orginization will blindly destroy the backups — all of the backups — that they know about whenever there’s a problem accessing the source data on the system.  Always.  The big red letters on the backup media itself saying not to do a backup if they’re having problems and to call whoever manages the system will never stop them.  Nor will all the messages, saying the same thing, that the backup process spits out.

I’ve managed a number of point of sale stations and networks for a number of small (sales under 5 million or so a year) independent retailers for about 15 years.  A lot of the systems are quite out dated — lots of Racal InterLan token ring gear, 10base5, Novell 2, NetWare Lite, LanMan, 386s, some Samsung 286s, and a whole lot of dBase.  Along with the dBase comes a number of programs that can’t resist corrupting any dBase database they touch, the worst being early 90s era VendorWare.  With dBase and VendorWare you’d better have rolling backups coming out of your ass.

Anyway… both times this month it was a case of corrupted dBase databases and VendorWare.  As long as you’ve got a recent backup of the database it’s easy to fix after a VendorWare program has trashed the database since 99% of the time it trashes a database that rarely changes or doesn’t have any data you can’t easily get back in sync.  Well, what do you know, before contacting me both of the organizations decided (like usual) that they’d try to make it work themselves.  The first step in their troubleshooting process… try doing a backup to each set of backup media (overwriting the backups of the working database) and checking to see if things magically started to work.

Here’s the great thing about point of sale systems (POS) these days; they’ve almost always got a tonne of free disk space.  This applies to both current POS software and ancient POS software.  The ancient stuff was designed to run on systems with 40 or 80MB of space… most of those systems now have at least 512 MB partitioned (usually on a 10 or 40+ GB disk with an overlay on it).  With all of that free space you’ve got lots of room to keep a week, or a month, or even a couple months of daily (or more often) copies of their databases right on the same disk.  It’s useless if the disk goes up in flames/whatever — that’s why you’ve got the removal media — but it’s priceless for when there’s just data corruption caused by the POS software; you’ve got all the copies you’ll need to get the database back in a functional state.

Of course, if the users didn’t blindly sabatoge any chance of you using the removal media, the on disk copies wouldn’t be necessary (although the on disk copies are so much faster to use anyway), but if there’s one constant I’ve found with these type of installations (or any installation really) is that no amount of training, warnings, documentation, or whatever, is going to stop them from trashing your only backups — if they know about them.

Bottom line, keep backup copies on the same system as the source data and don’t tell them about it.  If you tell them about it they’ll surely delete it, somehow.  There shouldn’t be any sort of liability issue with it… if they’re purposely trying to destroy data (say to avoid pesky tax auditors, or whatever reason you can imagine someone would want to destroy their data for), anyone competent enough to figure out where the POS software is storing its data should notice the backups on the disk.  You can usually get away with making it quite obvious to them too… store the backups in directories called backup.1, backup.2…, in the main volume… the clueless staff won’t notice them (the malicious staff is a different story — but that requires a whole different backup strategy).

Entry Filed under: Technology