Skip to content

Instantly share code, notes, and snippets.

@altras
Last active April 8, 2026 12:47
Show Gist options
  • Select an option

  • Save altras/30ee6a104039b85ec2d237e8e256820f to your computer and use it in GitHub Desktop.

Select an option

Save altras/30ee6a104039b85ec2d237e8e256820f to your computer and use it in GitHub Desktop.
Cognitive Rebase — Reddit Problem Validation Experiment (Pretotyping XYZ + MEH)

Cognitive Rebase: Reddit Problem Validation Experiment

Experiment Type: Pretotype (Fake Door / Discussion Seed)

Owner: Tsvetan Tsvetanov Date: 2026-04-08 Product: cognitive-rebase.camplight.net (AI Engineering Training by Camplight) Stage: Problem Validation (pre-solution)


XYZ Hypothesis

At least X% of Y will Z.

At least 30% of Reddit posts (3 out of 10) will generate 20+ substantive comments from engineering professionals confirming that AI code review bottlenecks and testing gaps are a real, unresolved problem on their teams.

Variable Value
X 30% (3 of 10 posts)
Y Discussion posts on engineering subreddits (r/ExperiencedDevs, r/programming, r/softwareengineering, r/CodingWithAI, r/cscareerquestions, r/devops, r/CTO)
Z Generate 20+ substantive comments confirming the review bottleneck / testing gap problem

MEH (Minimum Experiment Hypothesis)

The minimum success criteria to justify moving forward.

Metric MEH Threshold Stretch Kill
Posts with 20+ comments 3 of 10 5 of 10 < 2 of 10
"Me too" / shared experience replies 5+ per qualifying post 10+ < 3 across all posts
Upvote ratio (on qualifying posts) > 70% > 85% < 50%
DMs or follow-up requests 1+ 3+ 0
Pushback ratio ("skill issue" / dismissive) < 40% of comments < 20% > 50%

If MEH is met: Proceed to solution validation (landing page + waitlist or workshop pilot offer via Reddit). If Kill threshold hit: Reframe the problem or pivot the channel. Reddit may not be the audience, or the framing needs work.


CustDev Foundation

This experiment is grounded in 1 structured interview + 27 workshop attendees across 2 course editions.

Interview #1: Todor Petrov (Team Lead, tmg-solutions)

Pain Point Severity Detail
Review bottleneck 92% 13 tasks waiting for review in one week
Testing gaps 80% E2e + FE component tests lagging behind AI code output
TDD ambiguity 75% "I'm just not sure if we do or do not do or how to do TDD"
Cost pressure 15% Not a trigger, removed from campaign framing

Key quote:

"The problem is not output by the AI, but reviewing and testing everything that the AI does."

Validated Pattern (Cross-Team)

AI output velocity has outpaced the team's ability to review and test it. The bottleneck shifted from writing code to validating it.

3 Validated Buyer Personas

Persona Description Evidence
A: Scaling CTO CTO/VP Eng, 100-500 employees, buys seats for team Mihail Tsvyatkov (20 seats), Emil Petkov (both courses)
B: Intuitive Builder Team Lead shipping fast with AI but without conscious methodology Todor Petrov (interviewed)
C: Self-Investing Dev Senior dev paying with personal/L&D budget Fanka (Commerzbank), Yoanna (OfficeRnD), Darina (HedgeServ)

Reddit targets Persona B and C: individual contributors and team leads who experience the problem daily.


Experiment Design: 10 Reddit Posts

Rules of Engagement

  • Zero links to Cognitive Rebase or Camplight
  • Zero selling, pure problem validation
  • Post from a personal dev account, not a brand account
  • Engage genuinely in comments, the replies are the real signal
  • All posts framed as questions or shared observations from a practitioner

Post Plan

# Subreddit Title CustDev Signal Persona
1 r/ExperiencedDevs "AI shifted our bottleneck from writing code to reviewing it. Anyone else?" 13 PRs waiting for review (92% severity) B
2 r/programming "The real cost of AI coding tools isn't hallucinations, it's the review backlog" AI output velocity > review capacity B, C
3 r/softwareengineering "Does anyone have a structured review process specifically for AI-generated code?" No formal AI SDLC, ad-hoc scripts B
4 r/webdev "Our e2e and frontend tests can't keep up with how fast AI generates code. How are you handling this?" Testing gaps at 80% severity B, C
5 r/CodingWithAI "TDD in an AI-first workflow: does anyone actually do this, or are we all winging it?" "I'm not sure if we do TDD or not" B, C
6 r/ArtificialIntelligence "AI adoption on my team is uneven. One senior refuses to use it. How do you handle resistance?" QA member resistant, can't force adoption B
7 r/cscareerquestions "Team leads: how much of your week is now spent reviewing AI-generated PRs?" Review bottleneck as management problem B
8 r/devops "AI-generated code is hitting production faster than our test coverage can keep up. What's your safety net?" E2e/FE testing lagging behind AI pace C
9 r/CTO "We have AI tools across the team but zero structured methodology around them. Is anyone doing this well?" No formal AI SDLC A
10 r/agile "What would a 'definition of done' look like for AI-assisted development?" "Otherwise it's just another info tool I'll read and forget" B, C

Post Bodies (Full Drafts)

Post 1: r/ExperiencedDevs

Title: AI shifted our bottleneck from writing code to reviewing it. Anyone else?

We've been using Cursor + Claude heavily for about 6 months now. The team has genuinely gotten faster at producing code. But there's something nobody warned us about.

The bottleneck moved. It used to be "we need to write this faster." Now it's "we have 13 PRs open and nobody has reviewed them."

AI generates code faster than the team can meaningfully review it. Our e2e tests and frontend component tests haven't kept up with the pace either. We're basically running on trust plus a custom smoke agent we hacked together.

And the weird part is, it's not that AI writes bad code. It's that we don't have a process for validating AI output at the speed it's produced. There's no "AI SDLC." We're just winging it.

A few specific things I'm struggling with:

  • Do you do TDD when working with AI? We do something TDD-like but honestly I'm not sure it counts
  • How do you review AI PRs differently from human PRs? Or do you even bother?

Curious if other teams are hitting this same wall or if it's just us.


Post 2: r/programming

Title: The real cost of AI coding tools isn't hallucinations, it's the review backlog

Everyone talks about AI hallucinations and bad code quality. Honestly, that's not our main problem anymore. The models have gotten good enough.

Our actual problem is that AI produces code faster than humans can review it. We went from "I wish the team could ship faster" to "there are 10+ PRs sitting in the queue because nobody can review them fast enough."

The testing layer hasn't caught up either. E2e tests, frontend component tests, they all lag behind the AI-generated code. So we're shipping more but with less confidence.

Has anyone actually solved this at the process level? Not "use better prompts," I mean a real workflow change that handles the throughput mismatch between AI-generated output and human review capacity.


Post 3: r/softwareengineering

Title: Does anyone have a structured review process specifically for AI-generated code?

Genuine question. We use AI coding tools heavily (Cursor, Claude) and the output quality is fine. But we have zero process for reviewing it differently from human-written code.

Right now our "process" is:

  • CodeRabbit for automated PR review
  • A smoke agent that catches obvious issues
  • Human review when someone has time (spoiler: there's a backlog)

The thing is, even with CodeRabbit, someone still has to review the review. You get a wall of AI-generated suggestions and you're back to the same problem: a human needs to read it, understand the context, and decide what actually matters. It moved the bottleneck one step over but didn't remove it.

There's no formal AI SDLC. No structured way to verify AI output at the speed it's produced. We're winging it, and it works until it doesn't.

Is anyone doing something more intentional here? What does your AI code review workflow actually look like day to day?


Post 4: r/webdev

Title: Our e2e and frontend tests can't keep up with how fast AI generates code. How are you handling this?

We've hit a testing bottleneck I didn't see coming. AI tools have genuinely sped up code generation. Features that took days now take hours. Great.

But our test suite didn't magically speed up with it. E2e tests are still slow to write and maintain. Frontend component tests are still manual. The gap between "code generated" and "code tested" keeps growing every sprint.

The result is we're shipping faster but our test coverage is actually going down as a percentage. Not because we're writing fewer tests, but because the denominator grew 3x.

Anyone found a good rhythm for keeping tests in sync with AI-accelerated development? Or is everyone just quietly accepting lower coverage?


Post 5: r/CodingWithAI

Title: TDD in an AI-first workflow: does anyone actually do this, or are we all winging it?

Honest question. Our team uses AI for most code generation now. We do something that looks like TDD. Write some tests, then let AI generate the implementation. But I'm genuinely not sure if what we're doing counts as TDD or if it's just "write tests sometimes."

The tricky part is that AI doesn't naturally work in the red-green-refactor loop. It wants to generate the whole solution at once. So we end up with tests that were written after the fact to cover AI output, not tests that actually drove the design.

Does anyone have a disciplined TDD workflow that actually integrates with AI tools? Or has TDD just quietly died in AI-first teams and nobody wants to admit it?


Post 6: r/ArtificialIntelligence

Title: AI adoption on my team is uneven. One senior refuses to use it. How do you handle resistance?

We've gone all-in on AI coding tools. Most of the team loves it. But one of our senior QA engineers won't touch it. Not hostile about it, just ignores it completely and works the old way.

The thing is I can't force adoption without risking losing a good team member. And mandating tools feels wrong. But the gap is growing. The AI-using devs are producing 3x the output, and the review/QA process is becoming a bottleneck partly because of this asymmetry.

Anyone navigated this? I'm specifically wondering:

  • Do you mandate AI tool adoption or leave it optional?
  • How do you handle the productivity gap between AI-adopters and non-adopters on the same team?
  • Is there a way to get resistant team members on board without it feeling coercive?

Post 7: r/cscareerquestions

Title: Team leads: how much of your week is now spent reviewing AI-generated PRs?

Serious question. My team adopted AI coding tools about 6 months ago. Output has increased a lot. But my calendar hasn't gotten any emptier.

I'm now spending what feels like 40% of my time reviewing PRs that were generated with AI assistance. The volume is up, and each one still requires careful review because AI-generated code looks correct at first glance but can have subtle issues you only catch if you really read it.

Before AI tools, I'd review maybe 5-8 PRs a week. Now it's 15+. The cognitive load hasn't gone down. It just shifted from "wait for code" to "validate code."

Other team leads / engineering managers: are you seeing the same thing? Has AI actually reduced your workload, or just changed what you're busy with?


Post 8: r/devops

Title: AI-generated code is hitting production faster than our test coverage can keep up. What's your safety net?

From an ops perspective, I'm seeing a pattern that worries me. Teams using AI tools are shipping code to production significantly faster, but the testing and observability layers haven't scaled with them.

Specific things I'm noticing:

  • PR velocity up 2-3x, but CI/CD pipeline coverage hasn't changed
  • E2e tests are the same ones from 6 months ago, now covering a smaller percentage of the codebase
  • AI-generated code tends to be harder to debug in production because the original developer didn't fully write or understand it
  • Incident retrospectives are starting to include "this was AI-generated and not thoroughly reviewed" as a contributing factor

What's your safety net these days? Feature flags? Canary deploys? Heavier monitoring? Or are teams just accepting more risk and hoping for the best?


Post 9: r/CTO

Title: We have AI tools across the team but zero structured methodology around them. Is anyone doing this well?

We've invested in AI coding tools. Cursor, Claude, the usual stack. Adoption is high. But I've realized we have no actual methodology around how the team uses AI. Everyone has their own approach, their own prompts, their own review habits.

The result is inconsistent:

  • Some developers are genuinely more productive
  • Some are producing volume without quality
  • Review is now the bottleneck (it used to be development speed)
  • Testing hasn't kept up with the output increase
  • We have no way to measure if AI tools are actually providing ROI or just creating a different kind of debt

I'm not looking for tool recommendations. We have the tools. I'm looking for process and methodology. Has anyone established a structured AI development workflow that actually works at the team level? Something more than "use AI and hope for the best"?


Post 10: r/agile

Title: What would a "definition of done" look like for AI-assisted development?

Traditional DoD: code written, tests pass, PR reviewed, deployed. Pretty clear.

But with AI-assisted development, I think the DoD needs to change. Some questions I'm wrestling with:

  • Does "code reviewed" mean the same thing when AI wrote the code? Do you need to review more carefully, or less?
  • Should "developer understands the code" be an explicit requirement now? It was always implicit before.
  • How do you verify test quality when both the code AND the tests were AI-generated?
  • At what point is AI-generated code "the developer's responsibility" vs "the AI's output that happened to work"?

I feel like most teams, mine included, haven't updated their definition of done for the AI era. We're applying old standards to a new workflow, and the gaps are showing up in production.

What does your team's DoD look like since you started using AI tools?


Posting Schedule

Batch Posts When Subreddits
1 #1, #3, #5, #7, #9 Day 1 ExperiencedDevs, softwareengineering, CodingWithAI, cscareerquestions, CTO
2 #2, #4, #6, #8, #10 Day 2 programming, webdev, ArtificialIntelligence, devops, agile

Split into 2 batches to avoid Reddit spam detection. Engage in comments on Batch 1 before posting Batch 2.


Measurement Plan

Data Collection (Per Post)

Metric How to Measure
Upvotes Reddit score at 24h and 72h
Comments Total count + substantive count (exclude bots/one-word)
"Me too" replies Count of comments sharing similar experience
Pushback replies Count of dismissive / "skill issue" comments
DMs received Manual count
Cross-posts / shares Manual tracking

Analysis Template (Fill After 72h Per Post)

Post #: [number]
Subreddit: [sub]
Upvotes (72h): [count]
Comments (72h): [total] / [substantive]
"Me too" count: [count]
Pushback count: [count]
Top insight from comments: [quote or summary]
Surprising finding: [anything unexpected]

Decision Criteria (After All 10 Posts)

Outcome Criteria Next Step
STRONG VALIDATE MEH met + DMs received Build Reddit-specific landing page, plan solution validation posts
VALIDATE MEH met, no DMs Refine framing based on top-performing posts, run solution validation
INCONCLUSIVE Mixed results, 2 posts hit threshold Double down on the resonant framing, test 5 more posts
INVALIDATE Kill threshold hit Problem may be real but Reddit isn't the channel, or framing needs fundamental rework

Cost & Effort

Item Cost
Time to post ~2 hours (posting + initial engagement)
Time to monitor (7 days) ~30 min/day
Financial cost $0
Risk Low, no brand exposure, personal account

Appendix: Why This Framing (Not Others)

Framing We Rejected Why
"AI generates bad code" Too easily dismissed as skill issue. Models have improved.
"AI is expensive / wasteful tokens" Cost pressure validated at only 15% severity. Not a trigger.
"AI will replace developers" Clickbait. Attracts wrong audience.
"Teams need AI training" Solution-forward. We're validating the problem, not the solution.
Framing We Chose Why
"Review is the new bottleneck" 92% severity in interview. Structural problem, not skill problem.
"Testing can't keep up" 80% severity. Universal across teams.
"No AI SDLC exists" Novel framing. Teams know it's true but haven't named it.
"TDD ambiguity with AI" 75% severity. Relatable confusion, not ignorance.

Experiment designed 2026-04-08. Review results by 2026-04-15.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment