How Do You Tell If a Bug Report Is Too Vague to Hand to Devin?
Run the report through four questions before it becomes a Devin task. Does it describe one bug or several bundled together, does it point at one code path or could it be any of three, can someone reproduce it on demand, and is there a concrete way to check a fix actually worked. If the answer to any of those is no, the report needs a human to look at it first, not an agent. Devin's own guidance backs this up directly. Its guide to when Devin is a good fit says tasks need "a clear start and end, plus explicit success criteria" and that "the more context, the better," which is another way of saying an agent given an unbounded, underspecified problem doesn't fail loudly. It fills the gaps with something plausible and hands you back a PR that reviews clean.
That's the trap worth designing around. A vague report doesn't make Devin error out. It makes Devin confident about the wrong thing, and the four questions below are a fast way to catch that before a session starts rather than after a diff needs untangling.
Cognition's own docs draw the line between a task Devin can do and one it can't in terms that map directly onto a bug report:
- One bounded symptom, not several. "The export is slow, and sometimes it also drops rows" is two bugs wearing one ticket. Devin will pick one to chase, and there's no guarantee it's the one that matters.
- A named path, not a guess. If the report can't say whether the bug lives in the frontend, the export worker, or a third-party library, Devin has to guess too, and its guess becomes the scope of the whole task.
- Steps that reproduce it on demand. Not a paraphrase of what a customer said. A sequence someone actually ran and watched fail.
- A way to check the fix. A specific input that should produce a specific output, or a test that fails now and should pass after. Without this, Devin's confidence estimate is reading the ticket, not the bug.
None of these are exotic asks. They're the same four things an engineer writing their own bug report would include without thinking about it, because they already know the answers. A customer-reported bug arrives without any of them by default, and someone has to add them, or flag that they can't be added yet.
The ticket that failed all four
Ingrid Vasseur runs the six-engineer platform team at Solum, which schedules maintenance visits for the HVAC, elevator, and fire-suppression contractors that service commercial buildings. For the past few months her team has routed well-scoped bugs straight to Devin through its GitHub integration. A ticket came in from a support rep, relayed from a call:
Support (relayed from a call): Customer says work orders sometimes show up assigned to the wrong technician. They noticed it twice this week, not sure how often it's actually happening.
Ingrid ran it against the checklist before it went anywhere:
- One bounded symptom? Not really. "Sometimes" and "not sure how often" mean nobody yet knows if this is one bug or three.
- A named path? No. Assignment could be wrong at creation, at a later reassignment, or in how the technician's app displays it.
- Reproducible? No. Nobody had a sequence that made it happen on demand.
- A way to check the fix? No, because nobody could yet state what "fixed" would look like beyond the customer not complaining again.
Four out of four failed. Ingrid pulled the ticket back before it reached Devin and spent twenty minutes in Solum's audit logs instead, filtering work orders by reassignment events for the customer's account over the past month. Every mis-assignment, it turned out, had happened within about ninety seconds of two dispatchers editing the same job at once, a last-write-wins race in the assignment save path that had nothing to do with the technician-matching logic the original ticket implied. Nobody could have written a testable success criterion for "fix the wrong-technician bug" because the bug, as described, didn't point at a real mechanism yet.
With the race condition identified, the rewritten ticket passed all four checks: one symptom (concurrent edits overwrite the technician field), one path (assignments/save.ts), a repro (two browser tabs editing the same work order within two seconds), and a success criterion (the second edit should merge or reject, not silently overwrite). That version went to Devin and came back as a PR that added an optimistic-lock check on save. The first version, sent as written, would have had Devin guessing at technician-matching logic that was never actually broken.
What "failing the checklist" looks like when nobody's checking
Ingrid's twenty minutes worked because she happened to have the audit logs and the habit of checking before routing. Absent that habit, a vague report sent straight to an agent doesn't fail visibly. Answer.AI's month-long test of Devin against 20 real engineering tasks logged 14 outright failures, and reading through them, several look less like Devin refusing and more like Devin committing hard to a wrong approach. Asked to pull recent papers from Google Scholar, Devin, in the team's words, "went into a rabbit hole of trying to parse HTML that it seems like it couldn't get out of. It got stuck and went to sleep." The team's own conclusion was blunt: "tasks that seemed similar to our early successes would fail in unexpected ways," with no reliable signal upfront for which one you'd gotten.
A vague bug report produces the same shape of failure on a smaller scale. Devin doesn't stop to ask whether "work orders sometimes show up wrong" is a display bug, a save-path race, or a training issue with the dispatchers. It picks whichever reading the surrounding code makes most legible, tests only that reading, and ships a plausible PR either way.
The checklist's blind spots
Ingrid running four questions in her head against every incoming ticket works at Solum's current size. It stops working for reasons that show up in a predictable order as volume grows:
- The same vague symptom arrives more than once, and nobody notices. A second, third, and fourth version of "wrong technician" report, each filed by a different rep in a different wording, each independently fails the checklist and gets set aside, without anyone connecting the four reports into the pattern that would make the "reproducible" box checkable.
- The checklist only works if the checker knows the codebase. Ingrid could rule out technician-matching logic because she'd built it. A support rep running the same four questions has no way to guess which path is plausible, so the checklist becomes a formality rather than a filter.
- Nothing tracks which reports are sitting in limbo. A report that fails the checklist needs someone to come back and re-check it once more evidence exists. Without a place that holds "failed, pending investigation" reports, they just don't get re-examined until a customer escalates again.
The first and third gaps are where an aggregation layer earns its place. Modem reads Slack, support tools, and sales call notes as they arrive, and clusters reports describing the same underlying symptom into one topic instead of leaving four vaguely-worded tickets to fail the checklist independently. A topic that's accumulated three reports of "work orders assigned wrong" carries that count and every account that hit it, which is itself evidence toward naming a path and a repro, the two hardest boxes on the checklist to check from a single report alone. When a topic clears the bar, the Modem agent composes the brief and hands it to Devin directly, carrying the accumulated detail instead of restating one customer's original wording as if it were the whole picture.
“we need SSO before rollout”
“any update on single sign-on?”
“SSO came up twice on this call”
Linear issue, quotes attached
insight report in Notion
DevinA note on where this is coming from: Modem is what we sell. Check the specifics above against your own setup rather than take our framing of them at face value. A broader comparison of how teams make this handoff is in the six best tools for handing customer-reported bugs to Devin.
The checklist itself doesn't change. What changes is whether a report that fails it the first time gets forgotten or gets a second look once the second and third reports arrive. For the fuller pipeline this connects to, from a report's first mention through a merged PR a customer gets told about, see from user report to merged fix.
Run the four questions on the next report
Before the next bug reaches Devin: does it describe one symptom or several, does it name one code path, does someone have steps that reproduce it right now, and could you write down what a passing test looks like. A yes on all four means it's ready. A no on any of them means somebody needs about twenty minutes with the logs before an agent gets involved, the same twenty minutes Ingrid spent that turned an unfixable ticket into a one-line optimistic-lock check.
