What Happens When Devin Can't Reproduce the Bug a Customer Reported?
It ships a guess. Devin has no documented step where a session pauses because it couldn't actually trigger the failure a ticket describes, and nothing in Cognition's own guidance says otherwise. What Devin does instead is scope the ticket text, whatever's there, produce a plan and a confidence estimate from that text, and start working. If the repro steps are missing or wrong, the confidence estimate doesn't know that. It's reading the ticket, not the bug.
That gap matters because a ticket filed by a customer through support almost never carries the same detail an engineer would write for themselves. A customer says "the report looks wrong sometimes." An engineer would say "the export undercounts by exactly the DST-transition week, in workspaces with more than one currency." Devin can execute the second sentence. Handed the first one, it doesn't ask which week or which workspace. It picks something plausible and runs with it.
“we need SSO before rollout”
“any update on single sign-on?”
“SSO came up twice on this call”
Linear issue, quotes attached
insight report in Notion
DevinThe scoping comment is reading the ticket, not the bug
Devin's Jira integration has a scoping-only mode built for exactly this uncertainty: turn it on, and instead of opening a full session, "Devin only analyzes the ticket and posts a scoping comment with a summary, implementation plan, and confidence estimate," and a person decides from there whether to let it proceed. Teams that use it well catch a lot of bad handoffs before any code gets written, but the one thing it can't do is tell you the ticket is unreproducible. Confidence, as far as Cognition documents it, comes from how legible the ticket text is: concrete repro steps, a bounded description, checkable acceptance criteria. A ticket that reads clean but is wrong, because the customer misremembered which button they clicked, or the bug only fires under a data shape nobody described, scores exactly like a ticket that reads clean and is right. The estimate is a read on the writing. It has no independent way to go verify the underlying claim.
Devin doesn't surface a dead end early. It works through it.
The most direct evidence for what actually happens sits in a documented case from Cognition's own team, reported in Answer.AI's review of Devin. Asked to deploy multiple applications to a single Railway deployment, something Railway's platform doesn't support, Devin didn't identify the limitation and stop. It "spent over a day attempting various approaches and hallucinating features that didn't exist," according to the writeup, working through a task that was never going to succeed as scoped. Nobody got an early flag that the premise was broken. The session just kept generating attempts until a person noticed and pulled the plug.
Reproduction failure is the same shape of problem at smaller scale. If a customer's bug can't be triggered with the information in the ticket, that's a dead end exactly like the unsupported Railway config was. Devin's failure mode isn't refusing the ticket. It's not recognizing the dead end and proceeding anyway, on an assumption about what the "obviously" broken behavior must be.
A repro gap that made it past scoping
A ticket comes in from a customer's controller, relayed by support into Jira:
Support (relaying the customer's report): Our weekly revenue rollup is showing a number that doesn't match the CSV export for the same range. Started sometime this month, we think. It's off by a meaningful amount, not a rounding thing.
It goes to Devin through the Jira integration, because on the surface it reads like a well-scoped bug: one dashboard, one number, off by more than rounding. The scoping comment comes back with a specific hypothesis, a timezone conversion bug in the rollup query, and a confidence estimate high enough that the fix looks routine.
Devin (scoping comment): Summary: weekly rollup total diverges from the CSV export for the same date range. Implementation plan: the rollup query buckets by UTC day while the export buckets by account-local day; align the bucketing and add a regression test. Confidence: High.
The PR that follows touches the timezone handling and passes every existing test. It misses the actual cause. The customer's workspace has two currencies attached, and the rollup discrepancy only appears on weeks that cross a currency-conversion rate update, a detail nothing in the ticket named. Devin never had the exchange-rate data or the specific workspace to check against. It had a plausible story about timezones that happened to compile and pass tests, and a confidence estimate that scored the story, not the ground truth.
Reviewer (in the PR): This is a real timezone edge case and worth fixing separately, but it's not the bug here. The workspace has two currencies. Can we get the actual rollup query output for that account before we guess again?
The second attempt, once someone pulls the workspace ID and the specific week's exchange-rate log, finds the real cause in under an hour. The first attempt wasn't a wasted PR exactly, but it was a full session and a review cycle spent on a hypothesis nobody had checked was reproducible.
Build the check before the ticket reaches Devin
The fix is procedural, and it's the same one Devin's own effective-instructing guidance points at without naming reproduction directly: give Devin more than a symptom. Concretely, that means someone confirms three things before a bug ticket gets routed:
- The failure was actually triggered, by a human, with the specific inputs in the ticket, not paraphrased from what the customer described.
- The environment is named, meaning the account, workspace, plan tier, or data condition the bug depends on, if it depends on one.
- A negative case exists, some workspace or account where the same steps don't fail, so the ticket points at what's different rather than just what's broken.
That check takes ten to twenty minutes on a bug like the rollup discrepancy above. It's also the exact step that gets skipped under load, because a support engineer routing forty tickets a week doesn't have twenty spare minutes forty times, and the ticket that skips the check looks identical to the one that didn't until a PR comes back wrong.
The check holds until volume breaks it
At low volume, one disciplined person catching this before routing is genuinely sufficient. The predictable reason it stops holding up is that the repro detail a triager needs, things like the workspace ID, the currency setting, or the exact week, usually lives in the original customer conversation and nowhere else. Someone has to go back to Intercom, or the call transcript, or the Slack thread, read it again, and manually carry the detail into the ticket. That step is easy to skip once, and it compounds, because every skipped check produces a Devin session that looks confident and is wrong in a way nobody catches until review.
This is the point where we think an aggregation layer earns its place, and it's the category Modem works in. Modem keeps the customer's own repro language, the account, and any environment detail attached to the topic from the moment the conversation happens, so the person routing a bug to Devin can check whether reproduction is even plausible before a session starts, instead of reconstructing it from memory after a wrong PR comes back. The brief Devin receives carries the actual quote and account, not a paraphrase written under time pressure. One note before that pitch goes further: Modem is our product, and the fair test is whether it holds up against your own ticket queue, not against how we describe it here. Below a few dozen bug tickets a month, a disciplined twenty-minute habit like the one above is the right amount of process, and it's worth doing whether or not a tool like Modem is involved.
The related question of what a well-formed bug ticket needs beyond reproduction, scope, reference code, a measurable definition of done, is covered in what context Devin actually needs to fix a customer-reported bug. For the broader test of whether a report is ready to hand off at all, see how to tell if a bug report is too vague to hand to Devin.
Try this on the next bug ticket
Run the three-question check above out loud, before the ticket gets routed, not after. If any answer comes up empty, the ticket needs another pass with the customer's original message open beside it, not a session with Devin.
