Skip to main content
WindowFlow
All articles
Infrastructure·3 min read

The backup you never restored is a hope, not a plan

Most companies discover their backup strategy doesn’t work at the worst moment. What a recovery plan that survives contact with reality looks like.

Almost every company has backups. Very few have recently restored one. That gap between having backups and having proven recovery is where businesses get hurt, and it stays invisible right up until the day it’s the only thing that matters.

Ransomware, a bad migration, a deleted production database, a datacenter incident at your hosting provider: the trigger varies, but the failure pattern is remarkably consistent. The backup job had been silently failing for months. The snapshots existed but lived in the same account the attacker controlled. The restore worked, but took four days nobody had budgeted for. The data came back, but the sixteen services around it had no documented order to come back in.

Start from the questions the business can answer

Recovery planning goes wrong when it starts with tooling. It should start with two numbers your leadership team signs off on (not your engineers): how much data you can afford to lose (recovery point objective), and how long you can afford to be down (recovery time objective). An e-commerce business doing most of its revenue in four evening hours has very different answers than a B2B platform whose customers work weekdays.

Those two numbers drive everything else: snapshot frequency, replication strategy, how much you invest in automation, and whether you need a warm standby environment or can rebuild from scratch. Without them, every technical decision is a guess with a price tag.

The four properties of a backup that will actually save you

  • Isolated: at least one copy lives outside your primary cloud account and credentials, so the incident that takes production down can’t take the backups with it.
  • Encrypted: at rest and in transit, with keys that don’t sit next to the data they protect.
  • Monitored: a backup job that fails should page someone - silence is the most common failure mode we find.
  • Rehearsed: restores are drilled on a schedule, timed, and documented, so the first real restore is never the first restore.

Restore drills are the whole point

A restore drill answers questions no architecture diagram can: Does the runbook still match reality after six months of infrastructure changes? Does the restored database actually boot with the current application version? Who has the credentials, and are they reachable on a Sunday? How long does it truly take - not in theory, but with today’s data volume?

Scheduled restore drills are a standard part of how we design managed infrastructure, and the first drill typically finds something: an expired credential, a snapshot region nobody remembers choosing, a dependency that got added without making it into the recovery order. Finding those on a calm Tuesday is the entire value of the exercise.

A backup strategy is measured by one thing: how confidently your team can say, out loud, how long a full recovery takes, because they’ve done it.

Where to start this week

If you do one thing after reading this: pick your most important system and restore it somewhere, today, with a stopwatch running. The result, smooth or humbling, tells you more about your resilience than any audit document. If you’d like a second pair of eyes on what you find, our infrastructure team runs recovery assessments as a standard engagement.

Written by the WindowFlow engineering team.
Questions about anything here? [email protected]

Talk it through with the people who wrote it

Every article here comes from running production systems. If it resonates, the conversation is free.

Prefer email? Write to [email protected]