Two questions for the team that determine the infrastructure budget

Conversations about fault tolerance usually dive into technical details within the first two minutes: replication, failover, timeouts, quorum. The executive is left to assess the proposal based on the speaker’s confidence — which is a poor way to allocate the budget.

Two parameters help bring the discussion back to business. They don’t require technical knowledge, are set by the business, and determine the architecture as a whole.

                         OUTAGE
                            │
   ─────────────────────────┼─────────────────────────►
        last point of       │        recovery /
        saved data          │       return to work
                            │
   ◄──────── RPO ──────────►│◄──────── RTO ──────────►
   how much data we'll lose│   how long until we're back up

RTO: how long until we recover

RTO (recovery time objective) — the maximum allowable time from the moment of failure to the restoration of operation.

In business terms: after how many minutes or hours customers will be able to buy again.

What actually goes into this time

This is where the main planning mistake hides. RTO is almost always estimated as the duration of the technical restoration work, whereas it includes the entire chain:

detection → notifying the right person → diagnosis → decision → restoration → verification

The restoration itself often turns out to be the shortest part. If monitoring detects a failure after twenty minutes, the on-call wakes up fifteen minutes later, and diagnosis takes forty, then a one-hour RTO is unachievable, even if the procedure itself takes five minutes.

The practical conclusion: it’s almost always cheaper to reduce RTO at the beginning of the chain. Setting up alerts on specific business metrics and establishing a clear on-call process costs incomparably less than automatic database failover, and often saves more time.

How RTO determines architecture

A day. One server and daily backups are enough. An engineer brings up a new machine and deploys the system without hurry.

Several hours. You need a pre-prepared standby site, a database replica, and a written failover procedure. A human makes the decision, so the time depends on how quickly they’re available.

Minutes. Automation is required: orchestration, automatic database failover. Note that even automatic failover doesn’t happen instantly — typically it takes tens of seconds, because the system needs to be sure the primary is really unavailable and not just hanging. An overly impatient configuration leads to spurious failovers, which is worse than the outage itself.

Seconds. Achievable for individual components, but not for the whole system. A requirement of “seconds” for the entire business in practice means “we didn’t do the math”.

RPO: how much data we’re willing to lose

RPO (recovery point objective) is the maximum amount of data—measured in time—that may be lost irretrievably.

In business terms: what time period of orders, payments, and requests will vanish and need to be re-entered by customers.

How RPO determines architecture

A day. Nightly backups are enough. A failure at 6pm means losing all data for the workday. Acceptable for a content site; usually not for an online store — but this should be verified by the business, not the engineer.

Minutes. Continuous archiving (shipping) of the database transaction log to a separate storage is required. On failure you restore the last full copy and “replay” the log up to the last saved record. Loss is limited to the interval between log shipments.

Near zero. Log shipping alone is not enough — this is a common misconception. Guaranteeing zero data loss requires that the primary acknowledge a write to the client only after a replica has accepted it, i.e. synchronous replication. The cost is that each write operation becomes slower by the time of exchange with the replica; within a single site this is fractions of a millisecond, between regions it’s tens of milliseconds per write. That’s exactly why zero RPO across geographic regions is expensive not only in money, but also in product performance.

Both parameters are set per function, not per system

Demanding uniform RTO and RPO for the whole infrastructure is a sure way to overpay. Payment processing and monthly report exports cannot have the same requirements: the former’s RPO tends toward zero, the latter can tolerate a day’s loss.

Therefore the requirements table is made per business function — the same used in component criticality assessment. Usually it turns out that strict requirements are needed for two or three functions out of twenty, and that reduces the estimate dramatically.

The conversation worth having

A typical scene: the business formulates the requirement as “we need nothing to be lost and everything to always work.” Engineers take this literally, bring a quote for a geo-distributed architecture with synchronous replication, and the discussion ends nowhere — both sides walk away accusing each other of not understanding.

The discussion is more productive if you start with three questions for each key function:

1. How long can it be down before we suffer irreparable damage? Not “how long would we like”, but after what period do consequences become irreversible.
2. For what period can we restore data manually? Sometimes the answer is reassuring: orders from the last ten minutes are in email and CRM and can be re-entered. Then strict RPO is unnecessary.
3. What does it cost? Every tightening of requirements should be accompanied by a price. The difference between “four hours” and “fifteen minutes” is usually a multiple.

Practice shows that after the third question requirements soften by themselves. For most mid-market systems a reasonable answer is recovery within a few hours and data loss measured in minutes. This is achievable with proven and inexpensive means.

What to record

The result of the discussion is a one-page table: function, RTO, RPO, how it’s provided, when it was last tested.

The last column is more important than the others. A stated RTO is an assumption; one validated by drills is a fact. How to separate them is discussed separately: a backup that nobody has restored.