monoconversion.md

Monorepo Conversion Architecture Guide

This guide explains how to turn multiple GitHub repositories into a single monorepo with a clean developer experience, predictable releases, and environment-safe operations. It is written for an intermediate full-stack developer and is based on patterns used in this repository.

1. What You Are Building

Target state:

• One repository with multiple services and shared packages.
• One consistent local environment via Dev Containers.
• One task orchestrator for local workflows using VS Code tasks.
• One dynamic environment switcher (dev/test/canary/stable) backed by 1Password.
• One release channel model with controlled promotion.
• Reusable CI/CD workflows that understand channels.

Anti-goals:

• Do not migrate everything in one giant PR.
• Do not keep per-service custom local setup scripts forever.
• Do not let release branches receive direct random commits.

2. Recommended Monorepo Layout

Use top-level folders for products/services and shared runtime utilities.

Suggested shape:

.
├── .devcontainer/
├── .github/
│   └── workflows/
├── .vscode/
├── api/
├── web/
├── marketing/
├── runner/
├── image-worker/
├── sdk/
├── shared/
├── scripts/
├── envs/
└── tests/

Design notes:

• Keep each deployable app in its own folder with its own package manifest.
• Keep cross-service logic in shared/ (env resolution, constants, helpers).
• Keep operational scripts in scripts/.
• Keep environment templates in envs/.

3. Migration Plan from Many Repos

Phase A: Inventory and Boundaries

• List all current repositories, services, and deploy targets.
• Mark ownership and runtime dependencies between services.
• Identify shared code duplicated in multiple repos.

Phase B: Create Monorepo Skeleton

• Create top-level folders for each service and shared code.
• Initialize root-level tooling (lint/test conventions, scripts, tasks).
• Add branch/channel policy docs immediately.

Phase C: Import Existing Repositories

Two practical options:

1. Use git subtree for each repo to preserve commit history under a subfolder.
2. Use git filter-repo to rewrite each repo into a subdirectory, then merge.

Use one approach consistently.

Phase D: Stabilize Runtime and Tests

• Make all services runnable from the monorepo root via tasks.
• Add smoke tests that validate core user paths.
• Cut over CI to the monorepo before deleting old pipelines.

Phase E: Decommission Old Repositories

• Set old repos to archived/read-only.
• Link all contributors to the monorepo contribution path.
• Keep rollback tags and migration notes for at least one release cycle.

4. Dev Container Architecture

Use Dev Containers as the source of truth for local setup.

Core files and responsibilities:

• .devcontainer/devcontainer.json: editor/runtime config, ports, extensions.
• .devcontainer/docker-compose.yml: app container plus dependencies (for example MongoDB, Redis).
• .devcontainer/post-create.sh: one-time bootstrap of CLI tools and dependencies.
• .devcontainer/run-service.sh: wrapper for long-running services so env refresh/restart is controlled.

Why this matters:

• Every developer gets the same runtime and tools.
• Onboarding time drops sharply.
• Environment drift is minimized.

Practical rules:

• Keep stateful services (DB/queue/cache) in compose.
• Keep app processes started by tasks, not by compose entrypoints.
• Keep post-create script idempotent.

5. VS Code Task Orchestration

Treat .vscode/tasks.json as your local command plane.

Task taxonomy:

• Service tasks: API, web, worker, docs, runner.
• Infra tasks: tunnels, webhooks, remote helpers.
• Test tasks: service-specific and full-suite runners.
• Admin tasks: environment switching and secret sync.
• Composite tasks: startup/shutdown bundles.

Patterns to copy:

• Use dependsOn and dependsOrder for deterministic startup.
• Build one top-level startup task for day-to-day development.
• Keep long-running tasks in background mode.
• Add explicit labels that map to team language.

Recommended startup model:

1. Set environment first.
2. Start core services in parallel.
3. Start optional helpers (tunnels/webhooks) after core if needed.

6. Dynamic Environment Model with 1Password

Goal: switch environments safely without manual copy/paste of secrets.

Reference pattern:

• .devcontainer/set-env.sh selects dev, test, canary, or stable.
• Script reads secrets from 1Password Developer Environments.
• Script writes/updates .env and preserves machine-specific values when needed.
• Script restarts relevant processes so environment changes take effect.

Add plan-aware overrides:

• Store per-plan test fixtures in JSON.
• Select API keys or account context by plan (free, pro, enterprise) at switch time.

Security/operations guidance:

• Prefer service account tokens in containers.
• Never commit resolved .env with secrets.
• Separate credential values from infrastructure overrides.

7. Release Channels and Branch Policy

Use channel branches that represent rollout stages:

• test: integration/staging default branch.
• canary: beta production.
• stable: production.

Promotion flow:

1. Merge feature work to test.
2. Promote test to canary via controlled workflow.
3. Promote canary to stable via controlled workflow.

Key policy rules:

• No direct commits to canary and stable.
• Promotions happen through automation only.
• Use branch sync status checks to reveal drift.

Versioning approach:

• Prerelease identifiers for non-stable channels (-test.x, -canary.x).
• Stable releases publish normal semver.
• Use one script to synchronize versions across all packages.

8. Workflow Architecture (CI/CD)

Design workflows as reusable building blocks.

Suggested model:

• Reusable release workflow using workflow_call.
• Reusable test workflow parameterized by environment/channel.
• Reusable deploy workflow parameterized by target channel.
• Promotion workflows that call reusable release and deploy workflows.

Typical workflow set:

• release.yaml: versioning + package publishing + GitHub release metadata.
• promote-test-to-canary.yaml: branch promotion + release + deploy.
• promote-canary-to-stable.yaml: branch promotion + release + deploy.
• test workflows for Linux/Windows or channel matrices.

Implementation tips:

• Keep environment-specific values in env files and workflow inputs, not hardcoded shell blocks.
• Use workflow outputs for version numbers and artifact handoff.
• Keep deployment workflows callable by other workflows and manual triggers.

9. Secrets and Environment Sync Strategy

For hosted services (for example Fly.io, trigger.dev), use sync scripts that translate channel to target resources.

Pattern:

• Read base secrets from 1Password.
• Apply channel-specific infrastructure overrides.
• Push to target platform using a deterministic script.

Benefits:

• One source of secret truth.
• Lower chance of manual drift.
• Easy repeatability in local and CI contexts.

10. Common Pitfalls and Safeguards

Pitfall: Environment switch succeeds but services still use old vars

Safeguard:

• Restart long-running processes from the switch script.
• Use service wrappers with PID tracking.

Pitfall: Promotion branches diverge due to direct commits

Safeguard:

• Protect canary and stable.
• Allow updates only from promotion workflows.

Pitfall: Secret token missing in containerized setup

Safeguard:

• Validate required auth variables before env switch.
• Fail fast with clear error messages.

Pitfall: CI and local env precedence differ

Safeguard:

• Document precedence order clearly.
• Use a shared env loader utility across services.

Pitfall: Too much complexity on day one

Safeguard:

• Start with dev/test channels first.
• Add canary/stable promotion after local and CI stability is proven.

11. First-Week Adoption Blueprint

Day 1:

• Create monorepo skeleton and import one core service.
• Add devcontainer and one startup task.

Day 2:

• Import remaining services and shared code.
• Add environment switch script with two channels.

Day 3:

• Add reusable test/release workflows.
• Add branch protection and promotion workflow draft.

Day 4:

• Add secrets sync scripts and channel overrides.
• Run full end-to-end release simulation on non-production channels.

Day 5:

• Freeze old repos to read-only.
• Enable canonical monorepo contribution path.

12. Minimal Starter Checklist

• Devcontainer with app + data dependencies.
• Shared env loader and channel metadata.
• Environment switch script with restart behavior.
• VS Code task graph with one composite startup task.
• Reusable release/test/deploy workflows.
• Protected promotion branches.
• Version sync script across packages.
• Secret sync scripts for external platforms.
• Migration and rollback notes.

13. What to Simplify If Your Friend Is Solo

If the project is small, start with this subset:

1. Two channels only (dev, stable).
2. One promotion workflow.
3. No tunnels initially; direct local ports.
4. One composite startup task.
5. One reusable test workflow.

Then scale to test/canary/stable once release cadence increases.

14. Quick File Mapping from This Repository

Use these files as concrete examples while building your friend version:

• .devcontainer/devcontainer.json
• .devcontainer/docker-compose.yml
• .devcontainer/post-create.sh
• .devcontainer/set-env.sh
• .devcontainer/run-service.sh
• .vscode/tasks.json
• scripts/sync-fly-secrets.sh
• scripts/sync-trigger-secrets.sh
• scripts/branch-status.sh
• scripts/sync-versions.mjs
• shared/environments.json
• shared/load-env.js
• .github/workflows/release.yaml
• .github/workflows/promote-test-to-canary.yaml
• .github/workflows/promote-canary-to-stable.yaml

These are implementation references, not strict templates. Keep the architecture and rename details for your friend domain.