Backup strategy: the 3-2-1 rule and testing your restores

Servers fail, disks die, people delete the wrong thing, and ransomware exists. A good backup is the only thing that turns these from disasters into bad afternoons.

The 3-2-1 rule

• 3 copies of your data (the live one plus two backups).
• 2 different types of storage or systems.
• 1 copy offsite, in another place or provider.

A backup on the same disk, or the same server, does not protect you from the failures that matter. Some teams add a copy that is immutable or offline, so ransomware or a compromised account cannot delete it.

Decide two numbers first

• RPO (recovery point objective): how much data you can afford to lose. If it is one hour, you need backups at least hourly.
• RTO (recovery time objective): how long you can be down. This decides how you restore and how much to automate.

What to back up

• Databases (the most important).
• User uploads and media.
• Configuration and secrets (stored safely): service files, web server config, environment variable names and how to rebuild them.
• Code is in Git, which is also a backup. Document how to deploy it.
• Anything that is hard to recreate. Skip what is easy to rebuild, like caches and node_modules.

Backing up databases correctly

Copying the files of a running database can give you a broken backup. Use the proper tool:

• PostgreSQL: pg_dump for logical backups; add WAL archiving or a managed service's point-in-time recovery for finer recovery.
• MySQL: mysqldump or a consistent snapshot.
• SQLite: the .backup command or VACUUM INTO, not a plain copy while it is being written.
• Many managed databases offer automatic backups and point-in-time restore. Check the retention and test them anyway.

Full, incremental and snapshots

• Full: everything each time. Simple, large.
• Incremental: only changes since the last backup. Small and fast to make, slightly more complex to restore.
• Snapshots (disk or filesystem): fast, useful, but keep them somewhere other than the same disk.
• Tools like restic and borg do encrypted, deduplicated, incremental backups to many destinations.

Protect the backups themselves

• Encrypt them and keep the key somewhere separate. A backup with every secret in plain text is a second breach waiting to happen.
• Use credentials that can write backups but not delete old ones where possible.
• Restrict who can read them. Treat them like the production data they contain.

Retention and automation

• Keep several generations, for example daily for a week, weekly for a month, monthly for a year.
• Automate with a timer or scheduler and alert when a backup fails or is missing. A silent failure is the common way people discover they have no backup.
• Watch the size, so backups do not fill a disk.

Test your restores

This is the step people skip. On a schedule:

1. Restore the latest backup to a separate machine or database.
2. Start the app against it and check the data looks right.
3. Time it, and compare with your RTO.
4. Write the steps down in a runbook.

Also practice the worst case: the server is gone and you must rebuild from the backup and your notes alone.

Takeaway: keep three copies on two kinds of storage with one offsite, back up databases with the proper tool, encrypt and protect the copies, alert on failures, and regularly restore to prove it works.

---

Linux & DevOps series, Part 2 (delivery and operations): 4 of 5. Previous: Monitoring basics: metrics, Prometheus, Grafana and alerts.