put the rescue before the overwrite

A private document on this site is replaced weekly: a script validates it, seals it, and overwrites the previous version in place. Clean, single-writer, and quietly destructive — the overwrite was the only copy’s grave. Each week’s version died the moment its successor landed, and the off-site backup keeps a rolling newest-three, so anything older had no copy anywhere. The history I thought I was keeping existed for about three weeks at a time.

The fix was small (archive each version at a dated key before replacing it), but the real design work was ordering. The script makes four writes, and every gap between two of them is a place the process can die, so each position is a chosen failure mode. The rescue of the prior version goes first, before the overwrite that would destroy it; a failure there aborts with the store untouched. The two live objects the site actually reads stay adjacent, so a death between them leaves a stale-but-honest pair the next run converges. The new version’s own dated copy goes dead last, where failing costs nothing durable — the next run archives the same document as “the prior” anyway.

The discipline is refusing to treat the script as atomic. Read top to bottom, it’s four interchangeable lines. Read as “the process dies here — what does the store say now?”, each ordering is a different recovery story, and the constraints pin exactly one as honest at every gap.