We Quarantined Our Flaky Tests Instead of Fixing Them

Note: This is demonstration content. The team, tooling, and all figures below are fictional and created to illustrate the publishing setup.

Our CI had a rerun button, and everyone knew to press it. A red build meant "try again," not "something is broken." That habit is expensive in a way that doesn't show up on any dashboard: it teaches an entire team to ignore the one signal that's supposed to stop a bad change from shipping.

Automatic retries hide the problem twice

We had configured the runner to retry failed tests twice before reporting failure. The intent was reasonable — stop bothering people about known-noisy tests. The effect was that genuinely broken code could pass, because a test that fails intermittently for a bad reason looks exactly like a test that fails intermittently for a boring reason.

Worse, retries erased the evidence. By the time someone looked, the run was green and the failure was gone from the summary.

The first thing we did was stop hiding it. Retries stayed on, but every retry got recorded:

afterEach((ctx) => {
  if (ctx.attempt > 1) {
    recordFlake({
      test: ctx.fullName,
      attempt: ctx.attempt,
      passedOnRetry: ctx.state === "passed",
      durationMs: ctx.duration,
    });
  }
});

A week of that gave us a ranked list. Most of the pain came from a small number of suites — the long tail barely mattered.

Quarantine, don't delete

The tempting move is to delete a flaky test. It's also the worst one: a flaky test is usually testing something real, badly. Deleting it trades a noisy signal for no signal.

Instead we moved the worst offenders into a quarantined suite that runs on every commit but does not block merges. That gave us three things at once: the main suite became trustworthy again, the quarantined tests kept producing data, and there was a visible list that made the debt hard to forget.

What was actually wrong

Working through the quarantine list, the causes were less exotic than expected:

• Shared state between tests. Roughly half. Tests passed alone and failed when a sibling ran first, usually via a module-level cache nobody reset.
• Real time. Tests that waited a fixed number of milliseconds for something async, then asserted. Fine on a fast laptop, not on a loaded CI runner.
• Ordering assumptions. Assertions on the sequence of results from a query with no ORDER BY, which is unspecified and happens to be stable until it isn't.
• Genuine race conditions in product code. A small number, but these were the valuable ones. The test was right and the code was wrong.

That last category is the argument for quarantining rather than deleting. Those bugs were real, would have reached users, and the only thing pointing at them was a test people had been trained to rerun.

Rules we kept

• A test that fails twice in a week gets quarantined automatically. No debate.
• Quarantine has an owner and a date. An untended quarantine is just a slower delete.
• The main suite never retries. If it's in the main suite, a failure means something.

The honest summary is that we didn't make our tests better so much as we made them tell the truth about which ones we trusted. That turned out to be the prerequisite for fixing anything.