A repeatable, opinionated execution framework for delivering web apps, APIs, backend-heavy systems, and AI/ML products in a mid-size company with small teams (2–5 engineers).
Owner: Engineering Lead · Review cadence: Quarterly · Status: Living document
- Mandatory items (marked ✅) are non-negotiable gates. A phase does not end until they pass.
- Recommended items (marked ◻︎) are strong defaults. Deviations require a 2-line justification in the ADR or PR description.
- Measurable targets are stated as numbers so teams can self-audit.
- Tailor ceremonies to team size, but never skip gates. Discipline > process theater.
- Clarity before code. Ambiguity is the most expensive bug.
- Small batches, fast feedback. Optimize for lead time, not LOC/day.
- Reversible > perfect. Prefer decisions you can undo cheaply.
- Boring tech by default. Choose excitement only where it creates leverage.
- Security, observability, and cost are features — not afterthoughts.
- You build it, you run it. Authors own their code in production.
Guiding rule: If a choice increases operational complexity without a clear, measurable ROI — reject it.
1. Intake & Discovery
2. Technical Design & Planning
3. Setup & Foundations
4. Development Execution
5. Quality Control (PR + Testing)
6. Release & Deployment
7. Post-Release, Operations & Iteration
Each phase has: required skills · mandatory gates · deliverables · common failure modes.
| Role | Primary Responsibility | Notes |
|---|---|---|
| Tech Lead | Architecture, ADRs, final technical call, unblocking | Reviews every non-trivial design |
| Engineers (Mid–Sr.) | End-to-end feature ownership (design → prod → on-call) | Write ADRs for their own domain |
| DevOps / Platform | CI/CD, infra, release engineering, IaC, secrets | Often shared or fractional |
| Security Champion | Threat modeling, dep scans, review for auth/PII PRs | Rotating role if no dedicated hire |
| Reviewer Pool | PR quality enforcement per CODEOWNERS | SLA: first response < 1 business day |
| On-Call Engineer | Production triage, incident response | Weekly rotation, handoff doc required |
In small teams, one person often wears multiple hats — document who holds which hat this sprint in the team README.
Goal: Understand what to build and under what constraints.
Required skills: requirement analysis, stakeholder communication, systems thinking, cost/ROI framing.
Mandatory gates (✅):
- Problem statement in 1 sentence (who, pain, outcome).
- Success metrics defined (e.g., "p95 < 300 ms", "conversion +2%", "< $X/mo infra").
- Scope boundaries: in / out / deferred lists.
- Non-functional requirements captured:
- Performance & scale (RPS, p95/p99, data volume)
- Availability / SLA target
- Security & compliance (PII, GDPR, SOC2, HIPAA as applicable)
- Cost envelope (cloud spend ceiling)
- Constraints documented (time, budget, team skill gaps).
Deliverables:
- Lightweight PRD (1–2 pages). Template: Problem · Users · Success metrics · Scope · Non-goals · Risks.
- Initial architecture sketch (one diagram is enough).
Common failure modes:
- Jumping to code before success metrics are defined.
- Confusing features with outcomes.
- Ignoring non-functional requirements until launch week.
Goal: Make the correct early decisions — this is the single biggest risk reducer.
Required skills: system design, trade-off analysis, DevOps awareness, threat modeling, data modeling.
Mandatory decisions (✅) — captured in ADRs:
| Area | Options to choose between |
|---|---|
| Architecture | Monolith / Modular monolith / Microservices |
| API style | REST / GraphQL / gRPC / Events |
| Data layer | SQL / NoSQL / Hybrid; primary + cache + search |
| Async processing | Queue (SQS/Redis/Kafka) / Cron / Workflow engine |
| Caching | Client / CDN / Edge / Application / DB |
| AuthN / AuthZ | OAuth2/OIDC / JWT / Session; RBAC / ABAC |
| Secrets | Vault / Cloud KMS / SOPS; rotation policy |
| Observability | Logs / Metrics / Traces stack; SLO targets |
| Deployment | Blue-green / Canary / Rolling; rollback strategy |
| AI/ML (if any) | Hosted API vs self-hosted; eval strategy; cost guardrails |
Security — mandatory (✅):
- Lightweight threat model (STRIDE or "what could go wrong?") documented.
- Data classification: Public / Internal / Confidential / Restricted.
- PII flow diagram if user data is handled.
- Dependency & container scanning plan.
DevOps — mandatory (✅):
- Environments defined: local · dev · staging · prod (minimum: local + staging + prod).
- Branching model chosen (default: trunk-based with short-lived branches).
- Release cadence target (e.g., daily to staging, weekly to prod).
- Rollback strategy per component.
Gates:
- ADRs written for every mandatory decision (template below).
- Architecture reviewed by Tech Lead + one peer.
- API contracts drafted (OpenAPI / GraphQL schema).
- DB schema + indexing plan drafted.
- Capacity estimate: expected RPS, storage, egress, cost/month.
- Threat model reviewed by Security Champion.
Deliverables:
- ADR set in
/docs/adr/NNNN-title.md - Architecture doc with diagram (C4 level 1–2 is enough)
- API spec (OpenAPI 3.x preferred)
- DB schema + migrations plan
Common failure modes:
- Overengineering (microservices, event sourcing, k8s, etc., with no justification).
- Under-designing (no capacity or failure mode analysis).
- Skipping threat modeling because "we'll add auth later."
ADR template (keep it short):
# ADR-NNNN: <Decision>
Status: Proposed | Accepted | Superseded by ADR-XXXX
Date: YYYY-MM-DD
Context: <1 paragraph>
Options considered: <A, B, C with 1-line trade-offs>
Decision: <what and why>
Consequences: <good + bad>
Goal: Build a production-shaped foundation on day one.
Required skills: DevOps basics, repo hygiene, IaC, tooling.
Mandatory setup (✅):
- Repository
- Monorepo or polyrepo chosen intentionally.
README.mdwith: purpose, local setup, test, run, contact.CODEOWNERSfile.CONTRIBUTING.mdor equivalent (PR rules, commit style).- Conventional commits (recommended) or stated alternative.
.editorconfig,.gitignore,.dockerignore, license.
- CI/CD pipeline runs on every PR:
- Lint ✅
- Type check ✅
- Unit tests ✅
- Build (and Docker build if applicable) ✅
- Dependency + container vulnerability scan ✅
- Secret scan (e.g., gitleaks) ✅
- Target: PR pipeline < 10 minutes.
- Containerization — Dockerfile with pinned base image and non-root user.
- Environment configuration — 12-factor; never commit secrets.
- Secrets management — vault/KMS; rotation policy documented.
- Observability baseline
- Structured JSON logs with request/correlation IDs.
- Health endpoint (
/healthz) + readiness (/ready). - Metrics endpoint or exporter.
- Error reporting (Sentry/equivalent) wired.
- Quality guardrails
- Formatter + linter + type checker enforced in CI.
- Pre-commit hooks configured.
- Test framework wired with example passing test.
- Dependency hygiene
- Renovate / Dependabot enabled.
- Lockfiles committed.
- License policy documented.
Gates:
- CI green on
mainfor a trivial change (smoke PR merged). - Local setup reproducible from README in < 15 minutes by a new engineer.
- Staging environment deployable from CI.
- Secrets provisioned and accessible only via vault.
- On-call runbook skeleton exists (even if empty sections).
Common failure modes:
- Treating foundations as optional — chaos compounds quarterly.
- "We'll add tests later." (You won't.)
Goal: Deliver value incrementally with quality held constant.
Branching & flow (default: trunk-based):
- Short-lived branches (< 2 days ideal, < 5 days max).
- PRs merge to
main;mainis always releasable. - Large features behind feature flags.
PR discipline (measurable):
- Size: ≤ 400 lines changed ideally; hard stop at 800 unless justified.
- One logical change per PR.
- PR description template: What · Why · How · Tests · Risks · Rollback.
- Link to issue/ticket.
Definition of Done (DoD):
- Code + tests merged.
- Docs updated (README / API / ADR as needed).
- Feature flagged if risky; default off in prod.
- Metrics / logs / alerts added for the new path.
- Deployed to staging and validated.
- Rollback path verified.
Testing expectations (the pyramid):
| Layer | Target | Notes |
|---|---|---|
| Unit | 70%+ | Pure logic, fast, deterministic |
| Integration / API | 20% | Real DB/queues via containers |
| E2E | ~10% | Happy path + critical flows only |
| Contract (if APIs) | As needed | Consumer-driven where cross-team |
- Line coverage target: ≥ 70% on changed code (ratchet up over time).
- Flaky tests are P1 bugs — quarantine + fix, do not ignore.
Backend standards:
- Clear layering (Handler → Service → Repository).
- Errors are typed and logged with context (never swallowed).
- Idempotency for all non-GET operations that may be retried.
- Timeouts, retries with jitter, and circuit breakers on outbound calls.
- Database migrations are forward-only and backward-compatible (add-column, then backfill, then cutover).
Frontend standards:
- Component-driven; clear boundary between presentational and container logic.
- API access through a single typed client layer.
- State management chosen intentionally (local > context > store).
- Accessibility: WCAG 2.1 AA for public surfaces; keyboard + screen reader tested.
- Performance budget: LCP < 2.5s, INP < 200ms, bundle < agreed ceiling.
- i18n-ready strings; no hardcoded user-facing text in components.
AI/ML standards (if applicable):
- Eval suite before shipping: golden dataset, metrics (accuracy, grounding, toxicity, latency, cost/req).
- Prompt / model versioning — prompts live in code, reviewed like code.
- Input/output validation & guardrails (schema, PII redaction, jailbreak filters).
- Cost & latency budgets per request; circuit breaker on upstream model APIs.
- Monitoring: quality drift, input distribution drift, error rate, token spend.
- Data governance: training data provenance, retention, opt-out honored.
- Human-in-the-loop fallback for high-risk flows.
Goal: Keep main healthy. Reviews are a learning and gating function.
Review rules (enforced via branch protection):
- Minimum reviewers:
- Default: 1 reviewer for teams of 2–3; 2 reviewers for teams of 4+.
- Mandatory 2 reviewers for: auth, payments, PII, infra/IaC, schema migrations, public API changes, ML model/prompt changes.
- At least one reviewer from CODEOWNERS for touched paths.
- Author may not self-approve.
- Review SLA: first response within 1 business day; block-merge if older than 3 business days without triage.
Reviewer checklist (mandatory):
- Meets the Definition of Done above.
- Matches the ADR / agreed design.
- No hardcoded secrets, tokens, or PII in logs.
- Inputs validated; outputs sanitized.
- Error handling covers failure, partial failure, and cancellation.
- Sensible logs + metrics for the new code path.
- Performance considered (N+1, unbounded loops, large payloads).
- Security considered (authZ checked, injection, SSRF, deserialization).
- Tests exist, are meaningful, and run fast.
- Backward-compatible or migration plan stated.
- Docs updated.
Hard blockers (do not merge):
- Failing CI (tests, lint, types, scans).
- Missing required approvals / CODEOWNERS.
- Unaddressed security or data-handling concerns.
- Undocumented breaking changes.
Healthy review culture:
- Comments are specific, kind, and actionable. Prefer "suggest" over "demand."
- Distinguish blocking · non-blocking · nit in comments.
- Pair on complex reviews; don't solo-approve things you don't understand.
- "LGTM" without evidence of reading is a review anti-pattern.
Common failure mode: rubber-stamp reviews — the single largest long-term risk to code health.
Goal: Ship safely, predictably, and reversibly.
Deployment strategy (pick one per service, document it):
- Rolling — default for stateless services.
- Blue/green — when instant rollback matters.
- Canary — for high-risk or high-traffic changes (route 1–5% → 25% → 100%).
- Feature flag — to decouple deploy from release.
Pre-release gates (✅):
- Deployed to staging; smoke + critical-path E2E tests green.
- DB migrations are backward-compatible; rollback tested.
- Rollback plan written in the PR / release notes (target: < 5 min to rollback).
- Monitoring & alerts reviewed — new signals exist for new code.
- Feature flags default to off in prod unless explicitly launched.
- Load / performance check done for risky changes.
- Security: no debug flags, no verbose stack traces, no open CORS in prod.
Release hygiene:
- Tag + changelog generated (Conventional Commits → auto-changelog).
- Release notes (1 paragraph) posted to the team channel.
- Release owner on-call for the next 24h for the change.
Goal: Keep the system healthy; learn from reality.
Observability baseline (every service):
- Logs: structured, correlated, sampled where high-volume.
- Metrics: RED (Rate, Errors, Duration) for services; USE (Utilization, Saturation, Errors) for resources.
- Traces: end-to-end request tracing for multi-service paths.
- SLOs defined with error budgets (e.g., 99.9% monthly availability, p95 < 300 ms).
- Alerts: alert on symptoms (user-visible SLO burn), not causes. No pager without a runbook.
On-call & incidents:
- Severity levels defined (S1–S4) with response-time targets.
- Runbooks for every alert (what does it mean, how to mitigate, who to escalate to).
- Incident response: declare → mitigate → communicate → resolve → postmortem.
- Blameless postmortem for every S1/S2 within 5 business days. Action items tracked.
DORA metrics (track quarterly):
- Deployment frequency
- Lead time for changes
- Change failure rate (target < 15%)
- Mean time to restore (target < 1 hour for S1)
Operational hygiene:
- Dependency updates reviewed weekly; security patches within SLA (crit < 7d, high < 30d).
- Cost review monthly; unexpected spikes investigated within 1 business day.
- Backups tested quarterly (restore drills, not just "the backup job ran").
- Access reviewed quarterly (least privilege).
- Data retention honored; PII purges automated.
Mandatory gates (✅):
- Error monitoring + alerting live.
- SLOs + dashboards in place.
- Runbook exists for every alert.
- On-call rotation defined.
- Post-release review held for material features.
- Start with a modular monolith. Split only when a boundary earns its keep (team autonomy, scale, or blast-radius reasons).
- Design for change, not prediction. Keep seams where future split is plausible.
- Every architecturally significant decision → ADR.
- Layered: Handler → Service → Repository; no cross-layer shortcuts.
- Idempotent writes where retries are possible.
- Timeouts, retries (exponential + jitter), and circuit breakers on all external I/O.
- Migrations: expand → migrate → contract.
- Feature flags for risky rollouts.
- Single typed API client; no scattered
fetchcalls. - Accessibility (WCAG 2.1 AA) is a requirement, not a nice-to-have.
- Core Web Vitals budgets enforced in CI for critical pages.
- Avoid tight backend coupling; treat backend as a versioned contract.
- Schemas are reviewed like code; migrations have rollback notes.
- Indexes justified; query plans checked for hot queries.
- PII classified and access-controlled; encryption at rest and in transit.
- Retention + deletion policy honored end-to-end (including backups, caches, logs).
- Everything containerized; images pinned and scanned.
- Infra as code (Terraform / Pulumi / equivalent). No click-ops in prod.
- CI/CD is the only path to prod. No manual deploys.
- Environments are as similar as possible; differences documented.
- Threat model at design time; revisit on major changes.
- No secrets in code, env files committed, or logs.
- Dependency & container scans blocking on critical/high.
- SAST/DAST as available; at minimum, linters with security rules.
- SBOM generated for releases (supply chain visibility).
- Principle of least privilege for humans and services.
- MFA enforced; short-lived credentials preferred.
- Treat prompts, eval sets, and model configs as versioned code.
- No model ships without an eval suite and a red-team pass.
- Guardrails on input (PII, injection) and output (toxicity, PII leak, hallucination checks where feasible).
- Cost + latency budgets enforced; graceful degradation path.
- Human-in-the-loop for high-stakes decisions.
- Performance budgets per surface (API p95, frontend LCP/INP, job duration).
- Load test before launch for anything user-critical.
- Cost tags on every resource; monthly FinOps review.
When making non-trivial choices, rank on:
- Complexity vs team size — can 2 people operate this at 3 a.m.?
- Real (not imagined) scale — today's RPS, not next year's fantasy.
- Time-to-delivery and learning velocity.
- Operational overhead (who babysits this?).
- Reversibility — how expensive is "we were wrong"?
- Total cost of ownership (build + run + retire).
Default: simplest thing that credibly solves today's problem + next quarter's. Document deviations in an ADR.
- Overengineering early (microservices, event sourcing, k8s without a reason).
- Skipping design; "we'll figure it out in PRs."
- Rubber-stamp reviews.
- Postponing DevOps/observability to "phase 2" — there is no phase 2.
- Tight coupling between frontend and backend release cycles.
- Shared mutable state across services.
- No feature flags → every deploy is a release.
- No on-call rotation; heroics culture.
- Silent retries that hide systemic failures.
- Monitoring dashboards without SLOs; alerts without runbooks.
- Untested rollbacks.
- "Temporary" workarounds that outlive their authors.
- Committing secrets (even "just in dev").
- Ignoring flaky tests.
- Letting
maingo red for > 1 hour.
- PRD signed off.
- Success metrics measurable.
- ADRs written for key decisions.
- API contract drafted.
- Threat model reviewed.
- Capacity & cost estimated.
- ≤ 400 lines, one logical change.
- Tests added/updated; CI green locally.
- Docs/ADR updated.
- PR description: What · Why · How · Tests · Risks · Rollback.
- Feature-flagged if risky.
- Required approvals (per CODEOWNERS).
- CI green (tests, lint, types, scans).
- No new security or secret findings.
- Observability for new paths.
- Migration plan (if schema change).
- Validated in staging.
- Rollback plan written and tested.
- Monitoring + alerts in place.
- Release owner on the hook for 24h.
- Stakeholders notified.
- Timeline written.
- Root cause(s) identified (5 Whys or equivalent).
- Blameless postmortem held.
- Action items tracked with owners + dates.
- Runbook updated.
Day 1–2
- Access to repos, cloud, secrets vault, monitoring, paging, chat, ticketing.
- Read: this SOP, team README, top 5 ADRs.
- Local env up; trivial PR merged.
Week 1
- Shadow a code review and an on-call handoff.
- Ship a small, real change end-to-end (design → merge → deploy).
- Meet with Tech Lead, Security Champion, DevOps.
Week 2
- Own a feature slice.
- Secondary on-call (shadow).
- Write 1 ADR or doc improvement.
| Metric | Target (default) |
|---|---|
| PR size (p75) | ≤ 400 LOC changed |
| PR time-to-first-review | < 1 business day |
| PR lead time (open → merge) | < 2 business days |
| CI pipeline duration | < 10 min |
| Main branch green ratio | ≥ 95% |
| Test coverage (changed code) | ≥ 70% |
| Flaky test rate | < 1% |
| Deployment frequency | ≥ daily to staging |
| Change failure rate | < 15% |
| MTTR (S1) | < 1 hour |
| SLO attainment | per service definition |
| Critical vuln patch time | < 7 days |
Track on a dashboard; review quarterly; adjust thresholds pragmatically.
Small teams rarely fail from lack of talent. They fail from inconsistent decisions and lack of discipline.
This SOP enforces:
- Clarity before coding.
- Quality before merging.
- Operability before launching.
- Stability before scaling.
- Learning after every release.
Follow it. Improve it. Challenge it — in writing.