Backup Software for Hard Disk Media

For hard disk media? What does that mean? Of course, we mean to say that external, portable hard disks are used as a target medium instead of a tape.

Motivation or: Why use external hard disks as a backup medium?

Disadvantages of tape media

Today, the amount of data on file servers, database servers and e-mail servers that is caused by running projects and has to be archived by us on a regular basis is approx. 400 GBytes. In addition, another 1000 GBytes of data for completed projects are stored on our servers. 

All these data must be stored on external media on a regular basis - in the case of running projects every day. Until recently, we have used DDS2 DAT tapes (for incremental backups, Monday through Thursday) and SLR100 tapes (for the weekly backup on Friday) for this. The used backup software was Arcserve™ by Computer Associates and later BackupExec™ by Veritas.

This solution did not cause substantial trouble, and due to rotating backup sets it was possible to reconstruct any data status of the last three months at any time. By the way, until today we never had the situation that data loss was caused by a hardware failure. Whenever our backup concept had to proof its operativeness, the data loss had been caused by users who deleted files by mistake.

On the other hand, the expenses and the effort for maintaining our backup concept during the last years were considerable:

• Expenses for backup media: approx. 120 DAT tapes, approx. 40 SLR 100 tapes (1 SLR 100 tape costs approx. 55 EUR today).
• A total of four repairs (two of them after the guarantee had expired) for two HP DDS2 streamers and a Tandberg SLR100 streamer. Yes, these components seem to break regularly if they are forced to do a lot of work.
• Approx. 2000 Euro for buying the backup software and approx. 4000 Euro for the hardware.
• High effort for changing the tapes during a backup run when using SLR100 media at the weekend.
• A lot of time required for double-checking if the backups can be restored properly.

For these reasons, we have always been looking around for alternatives to our existing backup concept. In particular, we had turned our attention to the use of external hard disks as backup media already one or two years ago. 

When both of our HP streamers started to write more and more garbage onto the tapes within several weeks, we finally decided to put the project "Use of external hard disks as backup media" into action.

Advantages of archiving to hard disks

The advantages of external hard disks with respect to tape media are obvious:

• Today, IDE hard disks are far superior to tapes with regard to capacity and purchase cost. One 200 GByte IDE disk with FireWire and USB interface in an external casing is much cheaper than four SLR100 media with 50 GByte native capacity each (140 EUR vs. 240 EUR).
• No additional hardware is required at all. An external hard disk can be connected to any modern PC via USB or FireWire. The stored data are available immediately. There are no expenses for streamers, cleaning tapes and repair work.
• There is no need anymore to change tapes during a backup run. A 200 GByte IDE disk contains four times the capacity of an SLR100 tape. In addition, we assume that the capacities of IDE disks will keep pace with our ever-growing amount of data to be stored regularly. We cannot expect a comparable trend for tape media. For these reasons, we decided to save the money that we had already set aside for buying an automatic tape changer.
• In the case of a "real disaster recovery", the archived data will be available much faster because there is no need for new acquisition and installation of streamers and streamer software.
• If the hard disks are formatted with Windows NTFS and the data are archived via Filecopy, the result of the backup process is transparent and checkable also for users who are not administrators. The data can be restored quickly and easily on any PC.

Possible problems when archiving to external hard disks

After we decided to put the project into action, we first had to remove some stumbling blocks along the path to an operative backup concept:

First of all, we had to take care of the fact that hard disks are more prone to error than tapes. In order to handle this problem, we simply decided to archive the data simultaneously to another internal hard disk.

It also became evident soon that commercially available backup software does not meet our special requirements. Professional software of well-known manufacturers is optimized for tape media. In addition, searching and restoring of archived data is cumbersome. There is only rudimentary support for archiving "via Filecopy".

Windows tools also had to be ruled out for archiving our file servers. XCOPY is not designed for large amounts of data and a deep directory hierarchy. ROBOCOPY offers more features and runs stable, but the required transparency and the convenience of professional backup solutions cannot be achieved via scripting. "Windows Backup" archives into a single file only and lacks convenient operation if data are to be restored. These and similar solutions also do not offer methods to react to special events during the archiving process (e.g. if the hard disk is full). In addition, it is not possible to use a database in order to find archived data files.

Our Backup Software

In short: We have developed "CPTec Backup" - software that will do exactly what is important for us: 

• Simple command line tool that is controlled via a config file. Scheduling is possible via Windows Scheduler.
• Configuration file for backup control. Windows or web dialogs are not required.
• Incremental backups and full backups can be written to different hard disks. Each file can be written to any number of media. Each job may define several targets. This means that data can be mirrored without the need for special software or hardware.
• Creation of log entries into the Windows event log. Creation of backup logs which list every archived file. Creation of warning logs which list files that could not be archived. Creation of summary backup reports.
• Sending e-mails of log files and backup reports.
• Logging of the saved files and warnings in a database (MS SQL Server or MS Jet (MS Access)).
• Automatic deletion of full hard disk media if the need arises. This is a counterpart to the "Overwrite tape" option, but much smarter because only the oldest backup will be deleted if necessary and because different targets can also be written to alternatingly.

A detailed list of all features can hopefully :-) soon be found under "Backup Software Performance Features". A demo version is also available. You may, of course, also pay for it: You can buy our backup tool for 39 EUR (Job License) or 116 EUR (Source Code License). The software is for personal use only.

And now we tell you something about the required backup hardware.