save the work, then mark it done

The tone translator has an agent that mines real usage for failing translations and proposes new eval cases. One run did all its work (“Reviewed 238, proposed 38 new cases”), then crashed on the very last step: a dropped socket on a query that counts the remaining rows, there purely for the summary line. The 38 proposals were gone. Worse, the agent had already advanced its “seen” watermark, so a re-run would skip those 238 rows forever. A blip on a cosmetic call threw away a batch of paid judge work.

Two faults compounded. A best-effort call was allowed to be fatal, when a failed row count should just report “unknown”. And the real one: the code marked the inputs consumed before it saved the output, and the crash landed in the gap.

The watermark had a sibling bug. A single monotonic timestamp can only say “everything up to here is done”; it can’t say “all of these except the one that errored”. So when one row’s judge call failed and the loop moved on, advancing the mark silently dropped that row from ever being mined again. The fix is to freeze the mark at the first failure, even when later rows succeeded.

Same rule both times: record that you consumed an input only after the work behind it is durably saved, and accept recoverable duplicates over silent loss. A frozen watermark re-processes a few clean rows next run, which is strictly better than a gap you can’t see.