AI Prompt to Fix a Flaky CI Test

A flaky CI test fails on one run and passes on the next with zero code changes in between, and that single fact is why most developers waste hours on it: the usual debugging instinct of "reproduce it, then fix it" doesn't work when the failure won't reproduce on command. An AI prompt for flaky test diagnosis works differently than normal AI debugging, because the model isn't looking for a bug in logic, it's looking for a timing assumption your test made that isn't actually guaranteed.

I've fixed a lot of these on client CI pipelines, and the pattern is almost always the same: a test assumes something happens in order, or within a fixed amount of time, and that assumption holds 95% of the time and quietly breaks the other 5%. Here's the actual workflow I use to get an AI model to find that assumption fast, instead of guessing at retries and timeouts like most "fix flaky tests" advice tells you to.

Why retries and longer timeouts aren't a real fix

The default advice for a flaky test is almost always "add a retry" or "bump the timeout." Both of those treat the symptom, not the cause, and both make the real problem harder to find later because now the test fails even less often, which means fewer data points the next time someone has to actually debug it.

A retry can genuinely be the right call for a test that's flaky because of real external-system variance, like a third-party API that's occasionally slow. It's the wrong call when the flakiness comes from your own code having a race condition, because now that race condition is quietly running in production too, it's just not failing a test about it anymore.

Feeding an AI model what it actually needs to diagnose this

A plain "why is this test flaky" prompt gets you a list of generic causes: race conditions, shared state, network calls, time-dependent logic. True, and not useful on its own. What actually works is giving the model the same four things you'd want if you were pairing with a senior engineer on this:

• The test code itself, not a paraphrase of what it does
• The code under test, specifically the parts that involve async operations, shared state, or ordering
• A log or output diff from a passing run versus a failing run, if you have one
• How often it fails — "1 in 50 runs" points somewhere very different than "1 in 3 runs"

Here's a real example. This test passes locally almost every time and fails maybe 1 in 20 times in CI:

test('processes queued orders in order', async () => {
  queueOrder({ id: 1, priority: 'high' });
  queueOrder({ id: 2, priority: 'low' });

  await processQueue();

  const results = getProcessedOrders();
  expect(results[0].id).toBe(1); // fails intermittently
});

And the relevant part of processQueue:

async function processQueue() {
  const orders = getQueuedOrders();
  await Promise.all(orders.map(order => processOrder(order)));
}

That's the whole bug, and it's a one-line fix once you see it: Promise.all runs every order concurrently with no ordering guarantee at all, so "processes in order" was never actually true, it just usually finished in submission order because the operations were fast and consistent in a local environment. CI runners under load expose the race that was always there.

The prompt structure that surfaces this

This test is flaky - it fails roughly 1 in 20 runs in CI, passes
almost always locally.

Test:
[paste the test]

Code under test:
[paste the relevant function(s), including anything async or
involving shared state]

The assertion that fails: [paste it]

Don't just suggest a retry or a longer timeout. Walk through what
ordering or timing assumption this test is making, whether that
assumption is actually guaranteed by the code under test, and
specifically what would make it break under load but not locally.

That last instruction, forbidding retries and timeouts as an answer, is the part most people leave out, and it's the difference between getting a real diagnosis and getting the same generic advice you'd find in any flaky-test blog post. Run that prompt against the example above and a decent model will flag that Promise.all guarantees nothing about completion order, point at that exact line, and suggest either processing sequentially with a for...of loop or explicitly sorting results by a sequence number set when each order was queued.

Where I'd push back on the common advice

Most flaky-test guides treat "quarantine the test" as an acceptable short-term move — tag it, skip it in the main suite, deal with it later. I think that's close to always the wrong call for anything touching ordering or shared state, because "later" rarely comes, and the underlying race condition doesn't go away just because the test stopped reporting it. I'd rather spend the twenty minutes running this prompt against the real failure the day it shows up than carry a quarantined test and a live race condition in production for months.

The one place I do think a retry is legitimately fine, not a band-aid: tests that call a real third-party service with no local stub, where the flakiness genuinely is network variance outside your code. Even there, I'd still run the diagnosis prompt once first, specifically to confirm the failure really is external and not your own code assuming the third-party call always responds within some window it doesn't guarantee.

Making this part of your actual CI workflow

A couple of things that make this repeatable instead of a one-off rescue when something's on fire:

• Capture the failing run's full log output before it scrolls off your CI dashboard — most CI tools purge old run logs eventually, and the diagnosis prompt is much weaker without an actual failure log to compare against a passing one
• Note the rough failure rate ("1 in 20," "1 in 200") the first time you notice a test is flaky, even before you have time to dig in, because that number alone changes what kind of cause is plausible
• If the same file keeps producing flaky tests, treat that as a signal the file itself has a shared-state or ordering problem worth fixing once at the source, rather than diagnosing each symptom test separately

Frequently Asked Questions

What if I can't reproduce the failure even in CI logs, just a pass/fail history? Give the model the test and the code under test anyway, and be explicit that you have no failure log, just a failure rate. It can still reason about which assumptions in the code are statistically likely to break under concurrent load, it just can't point at a specific line of output as proof.

Does this approach work for flaky end-to-end/UI tests too, or just unit tests? The same structure works, but the "code under test" section needs to include whatever's actually variable, usually network request timing or animation/transition timing, not just your assertion code. E2E flakiness is almost always a wait condition that's implicit instead of explicit.

My test suite has dozens of flaky tests already. Do I run this prompt on all of them individually? Group them by symptom first. A handful of flaky tests that all touch the same queue, cache, or shared fixture are usually one root cause wearing several disguises, and diagnosing the worst offender first often fixes three or four "unrelated" flaky tests at once.

Key Takeaway

Stop asking an AI model "why is this flaky" with nothing but the test file. Give it the code under test, a real failure rate, and an explicit instruction to rule out retries and timeouts as the answer, and ask it to name the specific ordering or timing assumption your test is making that the code doesn't actually guarantee — that's the question that gets you the real race condition instead of generic flaky-test advice.