Caselet · SWE lead trims review debt without hiding TLX

Context

The SWE lead at a fintech firm faced growing review debt. PRs averaged 19 comments and 2.7 handoffs. Leadership wanted “AI in the loop,” but the lead refused to roll out another copilot dashboard without evidence.

They picked a single task category—high-risk payout diff—and ran the SWE Analyzer pack twice per engineer:

1. Manual review with the existing rubric.
2. AI-assisted review using a locked prompt + checklist.

Every run captured self-rating, reviewer score, Δ, reviewer minutes, and TLX (mental demand + frustration).

Findings

• Manual baseline. Δ averaged +3 (self-ratings 8 vs. reviewer 5). Reviewer minutes per diff: 22. TLX: 74/58—engineers were cooked.
• AI-assisted. Δ tightened to +1 (self 7, reviewer 6). Reviewer minutes dropped to 17. TLX averaged 56/41 because the AI suggestions bundled lint/contact surfaces.
• Defect leakage. Because reviewers logged their rework minutes in the Analyzer, leadership saw that the AI suggestions reduced rework instead of hiding it.

Each retro now starts with three tiles:

| Metric | Manual | AI-assisted | | --- | --- | --- | | Δ (self vs. reviewer) | +3 | +1 | | Reviewer minutes | 22 | 17 | | TLX (mental / frustration) | 74 / 58 | 56 / 41 |

How they shared it

• Engineers paste the TLX chart and Δ comparison into the retro doc with a link to /help/interpretation so PMs know what “56/41” means.
• Reviewer minutes are plotted alongside defect hotspots so Legal sees the guardrails.
• Exec memos include a link to /methodology plus the exact prompt scaffold. No black boxes.

Apply this pattern

• Run the SWE quickstart once a week. Copy the built-in Task Frontier + System Shift diagrams into your stand-up deck so people see when AI should review code.
• When TLX sneaks above 60 again, the team pauses and rereads the Interpretation guide before green-lighting more AI suggestions.
• Use /resources/ai-code-review-best-practices for the reviewer checklist and /resources/ai-ethics-into-workflows to keep the paper trail intact.

Proof beat enthusiasm. The lead now answers “is AI hurting review quality?” with tiles, not promises.