Workflow Evaluation: Dev's Playbook vs Skill Bar

workflow-eval.md

Workflow Evaluation: Dev's Playbook (A) vs Skill Bar (B)

Revision 2 — Updated to reflect Skill Bar's actual multi-agent pipeline: Scott (Principal Architect) → Mod (PM skill, Linear tickets) → Donny (Dev skill, Codex/Opus loops to level 9) → PR → Human approval → CI

Weighted scores: A = 5.1/10 | B = 8.9/10

Security, enforcement, and auditability weighted 1.5x.

Scores by Dimension

Dimension	Dev's Playbook (A)	Skill Bar (B)	Weight
Clarity of Intent	8	9	1.0x
Scope Control	9	9	1.0x
Session Resilience	8	7	1.0x
Verification Rigor	7	9	1.0x
Security Posture	3	9	1.5x
Multi-Model Orchestration	2	10	1.0x
Scalability	4	9	1.0x
Incident Learning	5	8	1.0x
Formality & Auditability	3	8	1.5x
Developer Experience	8	7	1.0x
Enforcement vs Guidance	4	8	1.5x
Ecosystem Awareness	3	8	1.0x

Why the Revision?

The original evaluation treated Skill Bar as a single-agent system. In practice, it's a multi-agent pipeline with role separation:

• Scott (Principal Architect) — designs architecture, directs work
• Mod (Product Manager skill) — fleshes out Linear tickets with full context, acceptance criteria, and architecture decisions
• Donny (Software Developer skill) — builds using Codex, Opus reviews to level 9 before PR creation

This changes several scores:

• Scope Control (6→9): Agent role boundaries are scope control. Donny doesn't decide what to build — he gets a fully specified ticket. That's at least as strong as "confirm task # before acting."
• Multi-Model Orchestration (8→10): Three distinct agents with purpose-built skills, plus Codex/Opus scoring loops within the dev skill. Not comparable to single-model workflows.
• Verification Rigor (8→9): Opus scoring to level 9 gates quality before human review and CI even begin.
• Clarity of Intent (7→9): A Linear ticket fleshed out by a dedicated PM skill is a richer contract than a TASKS.md line item.
• Developer Experience (5→7): Each agent only sees what's relevant to their role. Donny doesn't need 20 ADRs — he gets a ticket and follows his skill.

Key Strengths Unique to Each

Dev's Playbook

• Session resilience. CONTEXT.md as an explicit handoff artifact is clean and practical.
• Near-zero onboarding. One dev, one file, go. But that's a feature of being small, not being good.
• Bug fix protocol. Root cause before code change prevents symptom-patching.

Skill Bar

• Security is structural, not aspirational. Dual-layer scanning (gitleaks + TruffleHog), branch protection, sandbox tiers — enforced, not suggested.
• Multi-agent orchestration is real. Architect → PM → Dev pipeline with distinct skills and models. Quality compounds at each handoff.
• Opus scoring to level 9. Code is reviewed and iterated before it becomes a PR. The dev's "run verify when you feel like it" can't compete.
• Institutional memory is formalized. 20 ADRs + 11 anti-patterns + reassessment triggers.
• Built for scale. Microkernel + plugins + org-wide CI + multi-agent roles.

In A but NOT in B

• CONTEXT.md as a named session bridge artifact
• Investigation mode (read-only exploration before editing)
• Micro-commit discipline (one task = one commit)
• Bug fixing protocol (root cause → approval → fix)

In B but NOT in A

• Multi-agent pipeline with role separation (Architect → PM → Dev)
• Multi-model review pipeline (Codex builds, Opus reviews to level 9)
• Structural security enforcement (gitleaks, TruffleHog, branch protection, CODEOWNERS)
• Formal ADR discipline with reassessment triggers
• Anti-pattern library with 11 named failures
• Sandbox enforcement / tiered security model
• Completion checklist with verifiable evidence
• SOC 2 alignment
• Plugin/skill architecture for composability
• Org-wide reusable CI
• Security review program ("Skill Bar Verified")
• Linear integration for ticket management

What Skill Bar Could Still Adopt

1. CONTEXT.md session bridge — Human-readable "here's where we are" artifact, auto-generated from state JSON.
2. Investigation mode — Formal read-only exploration phase before code changes.
3. Micro-commit discipline — One logical change = one commit within PRs.
4. Bug fix protocol — Root cause analysis → human approval → fix as a named pattern.

What Dev's Playbook Is Missing

1. Any security at all — No secret scanning, no branch protection, no sandbox.
2. Structural enforcement — Every rule is a suggestion. Nothing prevents the AI from going rogue.
3. Multi-model/multi-agent orchestration — Single model throughout. "Senior engineer persona" is a prompt trick, not real separation.
4. Formal decision records — No ADRs means the "why" evaporates.
5. Anti-pattern library — Without named failures, every contributor rediscovers the same mistakes.
6. Composability — No answer for "how do 10 teams use this?"
7. Compliance readiness — Nothing to show an auditor beyond git log.

---

Appendix: GitHub Authentication Patterns

Two approaches to giving AI agents GitHub access:

Pattern A: Personal Access Tokens + MCP (Dev's Playbook)

Create a GitHub PAT, feed it to Claude via MCP (Model Context Protocol). Claude interacts with GitHub natively through structured API calls inside the conversation.

Pros:

• Claude can interact with GitHub natively — no shell needed
• Works in cloud/hosted environments with no local CLI
• MCP provides structured tool access (create PR, read file, search issues)

Cons:

• PATs are static secrets requiring manual rotation
• Easy to over-scope (most people create overly broad tokens)
• Token gets sent to the AI provider's API as part of MCP config
• If the token leaks (logs, config, session context), it's valid until manually revoked
• One more secret to manage and secure

Pattern B: OAuth + CLI (Skill Bar)

Authenticate gh CLI via OAuth locally. AI agents run gh commands through shell access — no tokens to manage.

Pros:

• No static tokens — OAuth handles auth with auto-refresh
• Token never leaves the machine or hits an external API's context window
• Revocation is instant via gh auth logout
• Same auth works for everything (clone, PR, issues, gists, releases)
• Nothing to configure — gh is already authenticated

Cons:

• Requires local shell access (won't work in pure cloud/API setups)
• Less "native" than structured MCP calls
• No granular scoping — agent has whatever gh has

Recommendation: For local agent setups, OAuth + CLI is more secure. No static secrets, no tokens in API context windows. PAT + MCP makes sense for hosted environments without shell access, but for persistent local agents it's a downgrade.