swamp extension quality checklist

EXTENSION_QUALITY_ANALYSIS_UNIFIED.md

Swamp Extension Quality Analysis

A Practical Guide to Building Production-Ready AI Automation Extensions

This analysis synthesizes real-world failure patterns from PR #66 (AWS Adoption extension) with systeminit/swamp's skills framework and established best practices.

Always use the swamp provided skills when authoring extensions and performaing quality checks. This list provides a specific analysis of a particular failure mode with an example PR reference.

---

Quick Start

If you're publishing an extension, follow this 5-minute checklist:

1. Template Syntax: All interpolations use ${variable} (double-brace syntax ${{ }} is Jinja, not supported)
2. Data Contracts: Model discovery output names MUST match workflow input parameter names exactly
3. Error Handling: Never use allowFailure: true as a substitute for validation
4. Naming Conventions: Use kebab-case for extensions, snake_case for internal references
5. Testing: Validate models with swamp model validate before publishing

---

Critical Failure Patterns (Real Examples from PR #66)

Pattern 1: Template Syntax Mismatch

The Problem: Python/Jinja2 workflows use ${{ variable }} syntax, but swamp extensions use ${variable}. The PR #66 extension used Python-style syntax throughout, causing silent failures during interpolation.

Real Example from PR #66:

# ❌ BROKEN: Python/Jinja2 style
workflow:
  - step: ProcessAWS
    input:
      region: ${{ aws_region }}  # Won't interpolate
      tags: ${{ tags | join(",") }}  # Python slicing unsupported

Correct Pattern:

# ✅ CORRECT: swamp extension style
workflow:
  - step: ProcessAWS
    input:
      region: ${aws_region}
      tags: ${tags}  # No Python transformations; use workflow steps instead

Why It Failed: The workflow parser treated ${{ aws_region }} as a literal string, not a variable. Parameters arrived as the string "${{ aws_region }}" instead of the actual value. This cascaded into failures in downstream steps that expected proper data types.

Prevention:

• Use ${variable} everywhere in extension definitions
• For data transformations (filtering, joining, slicing), add explicit workflow steps
• Test interpolation by running swamp workflow validate <extension-name>

---

Pattern 2: Naming Contract Violations

The Problem: Model discovery outputs names that don't match workflow input parameter names. This causes silent failures because swamp can't bind the discovered data to workflow inputs.

Real Example from PR #66:

# Model definition
model:
  name: aws-inventory
  outputs:
    discovered_instances:  # Output name
      type: array

# Workflow definition
workflow:
  - step: Process
    input:
      instances: ${aws_instances}  # Input expects "aws_instances", not "discovered_instances"

The model produces discovered_instances, but the workflow tries to consume aws_instances. The binding fails silently, and workflow receives null.

Correct Pattern:

# Model definition
model:
  name: aws-inventory
  outputs:
    instances:  # Matches workflow input name
      type: array

# Workflow definition
workflow:
  - step: Process
    input:
      instances: ${instances}  # Matches model output

Prevention:

• Define a Data Contract document that maps:
  • Model discovery outputs → Workflow inputs
  • Data types (array, object, string)
  • Required vs optional fields
• Run swamp model validate and swamp workflow validate as integration tests
• Test end-to-end: swamp workflow run <extension> --dry-run

---

Pattern 3: Silent Failures with allowFailure

The Problem: allowFailure: true hides errors instead of handling them. When combined with data contract violations, failures become invisible until production.

Real Example from PR #66:

workflow:
  - step: FetchAWSData
    allowFailure: true  # ❌ This masks the data contract violation
    input:
      region: ${aws_region}  # Wrong variable name

The step fails silently. Downstream steps receive null for AWS data. The workflow continues but produces incomplete results.

When allowFailure Is Appropriate:

workflow:
  - step: FetchOptionalEnhancement
    allowFailure: true  # OK: Enhancement data is truly optional
    input:
      cache_ttl: 3600

  - step: ProcessWithFallback
    input:
      primary_data: ${main_data}
      fallback_data: ${enhancement_data}  # Will be null if previous step failed
      # Subsequent logic explicitly handles null fallback

Prevention:

• Never use allowFailure: true to hide mistakes
• Use it only when a step is genuinely optional and downstream logic handles null/missing data
• Add explicit validation:

  - step: ValidateRequiredData
    input:
      data: ${required_field}
    assertion: "data != null"  # Swamp validates this before proceeding

• Log failures with detailed error context, never suppress them

---

Integration with Swamp Skills Framework

Swamp provides dedicated skills for extension development. Reference these in your workflow:

swamp-extension Skill

When to use: Creating models, vaults, drivers, datastores, reports from scratch.

NOT for running existing models — that's swamp model run or workflow integration.

Covers:

• Model input/output schemas
• Vault setup (sensitive data storage)
• Driver creation (tool integration points)
• Datastore definitions (persistent state)
• Report templates (structured output)

Example:

# Use the skill to design a model
swamp-extension: "Design a model that discovers AWS instances
  with outputs: instance_id (string), instance_type (string), region (string)"

# Result: Skill generates model definition with proper schema validation

swamp-extension-publish Skill

State-machine checklist for publishing extensions:

1. Repository Setup: Git init, AGENTS.md, CLAUDE.md, extensions/ directory
2. Authentication: gh auth login, verify GitHub CLI is configured
3. Manifest: Create extension.manifest.json with name, description, version
4. Collective Ownership: Add team members to extension ownership in manifest
5. Version Bump: Increment version, test against previous versions
6. Formatting: Run deno fmt, ensure no linting errors
7. Dry-Run: gh gist create --private --dry-run (or equivalent)
8. Publish: gh gist create --private (no dry-run, final publication)

Key insight: Publishing is a state machine. Each step builds on the previous one. Skipping validation steps is how PR #66 ended up broken.

swamp-model Skill

For model development:

• Define input schemas with validation rules
• Specify output formats and types
• Test models with swamp model validate
• Profile models for performance

swamp-workflow Skill

For workflow composition:

• Step orchestration (sequential, parallel, conditional)
• Data binding between steps
• Error handling and retry strategies
• Dry-run testing before production

issue-lifecycle Skill (Adversarial Review Patterns)

Patterns for stress-testing extensions:

1. Challenge the Plan: "Does this extension handle AWS region failures? What if AWS API is down?"
2. Evaluate Architecture: "Are data contracts defined? Do input names match output names?"
3. Validate Behavior: "Run the extension with edge cases: empty arrays, null values, missing fields"
4. Stress-Test Assumptions: "If allowFailure: true, does downstream logic handle null data?"

Applied to PR #66:

• Challenge: "How does the extension handle template syntax errors?" → Found ${{ }} syntax issues
• Architecture: "Are output names documented?" → Found missing data contract
• Behavior: "What happens if a model returns null?" → Found silent failures with allowFailure: true
• Stress-Test: "Run with minimal AWS permissions, missing regions, rate-limited APIs" → All failed silently

---

Pre-Publication Checklist

Phase 1: Design & Architecture (Skill: swamp-extension)

• Extension name follows kebab-case (e.g., aws-adoption, not AwsAdoption)
• Models defined with clear input/output schemas
• Data contracts documented (model outputs → workflow inputs)
• Error handling strategy defined (what's required vs optional?)
• Naming conventions consistent (snake_case for variables)

Phase 2: Development & Testing

• Models created and tested: swamp model validate <model-name>
• Models tested end-to-end: swamp model run <model-name> --dry-run
• Workflows created with proper template syntax (${var}, not ${{ var }})
• Workflow validated: swamp workflow validate <workflow-name>
• All data bindings verified (output names match input names)
• No allowFailure: true without explicit null-handling logic downstream
• Unit tests written for custom logic
• Integration tests written for end-to-end flows

Phase 3: Code Review (Skill: issue-lifecycle - Adversarial Review)

• Challenge the plan: Can extension handle service outages, rate limits, missing data?
• Evaluate architecture: Are data contracts documented? Are naming conventions consistent?
• Validate behavior: Test with edge cases (empty arrays, null values, malformed input)
• Stress-test assumptions: Run with minimal permissions, missing fields, timeout scenarios
• Security review: Does extension sanitize user input? Are secrets properly vaulted?

Phase 4: Documentation & Publishing (Skill: swamp-extension-publish)

• README.md written with examples, troubleshooting
• extension.manifest.json created with version, description, dependencies
• CHANGELOG.md maintained
• All code formatted: deno fmt
• GitHub CLI authenticated: gh auth login
• Dry-run successful: gh gist create --private (preview)
• Final publication: gh gist create --private (final)

Phase 5: Post-Publication

• Monitor logs: journalctl -u swamp-extension-<name> -f (if deployed)
• Set up alerts for silent failures
• Track usage metrics on swamp.club/leaderboard
• Respond to community issues (GitHub Issues, swamp-club Discord)
• Plan maintenance & updates

---

Code Review Checklist for Extensions

Template & Interpolation

• All variables use ${variable} syntax
• No Python/Jinja2 syntax (${{ }}, | filter, slicing)
• Nested objects interpolate correctly: ${object.field}
• Arrays handled without transformation: use workflow steps instead

Data Contracts

• Model outputs documented in YAML comments
• Model output names listed: outputs: [name1, name2, name3]
• Workflow inputs documented with types
• Each workflow input has a corresponding model output with matching name
• Data types consistent (if model outputs array, workflow expects array)

Error Handling

• allowFailure: true only used for genuinely optional steps
• When allowFailure: true, downstream logic handles null/missing data
• Critical steps do NOT have allowFailure: true
• Error messages are descriptive (no generic "failed" messages)
• Retry logic appropriate for each step (API calls: yes, local operations: usually no)

Naming & Conventions

• Extension name: kebab-case (aws-adoption, not awsAdoption or AwsAdoption)
• Variables: snake_case (aws_region, instance_count)
• Model outputs: snake_case (discovered_instances, but matches workflow inputs)
• Workflow steps: PascalCase (ProcessInstances, ValidateRegion)
• No magic strings or numbers

Testing & Validation

• Model validation passes: swamp model validate <name>
• Workflow validation passes: swamp workflow validate <name>
• Dry-run successful: swamp workflow run <name> --dry-run
• Edge cases tested (empty arrays, null values, missing fields)
• Error cases tested (API failures, timeouts, permission errors)

Documentation

• README.md includes: What it does, how to use, troubleshooting, examples
• Data contract documented: Model outputs and their types
• Error handling documented: Which errors are expected, how they're handled
• Dependencies listed: Required models, external services, permissions

---

Practical Examples

Example 1: Correct Data Contract

Model Definition:

model:
  name: inventory-collector
  description: "Discovers cloud resources"
  inputs:
    cloud_provider:
      type: string
      required: true
  outputs:
    resources:           # Output name
      type: array
      description: "List of discovered resources"
    resource_count:      # Output name
      type: integer

Workflow Definition:

workflow:
  name: process-cloud-inventory
  steps:
    - name: discover
      model: inventory-collector
      input:
        cloud_provider: aws

    - name: analyze
      input:
        discovered_resources: ${resources}        # Matches model output name
        total_count: ${resource_count}          # Matches model output name
      assertion: "discovered_resources != null"  # Explicit validation

Why This Works:

• Model outputs: resources, resource_count
• Workflow consumes: ${resources}, ${resource_count} (names match)
• Data types consistent (array/integer)
• No template syntax mismatches
• Explicit validation before processing

Example 2: Handling Optional Data Correctly

❌ WRONG:

step: EnhanceWithMetadata
allowFailure: true  # Silently hides errors
input:
  data: ${resources}

✅ CORRECT:

steps:
  - name: fetch-optional-metadata
    # No allowFailure; let it fail loudly if there's a real problem
    input:
      cache_ttl: 3600
    
  - name: process-with-optional-metadata
    input:
      resources: ${resources}                    # Required
      metadata: ${metadata}                      # May be null from previous step
    logic: |
      if (metadata == null) {
        // Explicit handling of missing data
        use_default_metadata();
      } else {
        merge_metadata(resources, metadata);
      }

Why This Works:

• Optional data is explicit in logic, not hidden by allowFailure
• Error handling is intentional, not defensive
• Downstream code clearly handles both success and failure cases

Example 3: Proper Error Message Design

❌ WRONG:

error_message: "Failed to process"

✅ CORRECT:

error_message: "Failed to fetch AWS instances from region '${aws_region}': ${error.message}. 
Ensure AWS credentials are configured and region is valid."

Why This Works:

• Includes context (which region? what was being done?)
• Includes the actual error message
• Includes remediation (how to fix it)
• Enables debugging without contacting support

---

Adversarial Review Framework (from swamp issue-lifecycle skill)

When reviewing extensions, ask these questions systematically:

1. Challenge the Plan

Question: "Does the extension handle failure scenarios?"

Apply to PR #66:

• ❌ "What if AWS API is unavailable?" → Extension has no retry logic
• ❌ "What if region is invalid?" → No region validation, silent failure with allowFailure: true
• ❌ "What if user lacks AWS permissions?" → Permission errors treated as data, not errors

Result: Design flawed from the start; failures invisible.

2. Evaluate Architecture

Question: "Are data contracts defined and enforced?"

Apply to PR #66:

• ❌ "What are the model outputs?" → Outputs not documented in discovery output names
• ❌ "What do workflows expect as input?" → Workflow input names don't match model outputs
• ❌ "How are they connected?" → No documented binding between them

Result: Data bindings are ambiguous; failures in binding silently fail.

3. Validate Behavior

Question: "What happens with edge cases?"

Apply to PR #66:

• ❌ "Run with empty AWS account" → Extension fails silently, no error surfaced
• ❌ "Run with 10,000 instances" → No pagination logic, truncates silently
• ❌ "Run with special characters in tags" → No escaping, template syntax breaks

Result: Extension fails on real-world data; no warnings.

4. Stress-Test Assumptions

Question: "What breaks under load or adverse conditions?"

Apply to PR #66:

• ❌ "Run with 1-second timeout" → No timeout handling, hangs indefinitely
• ❌ "Run with rate-limited API" → No exponential backoff, hits rate limits
• ❌ "Run with minimal AWS permissions" → No permission checking, cryptic failures

Result: Unusable in production under realistic conditions.

---

Model Validation Best Practices

1. Schema Validation

# Validate model schema
swamp model validate aws-inventory

# Expected output:
# ✓ Model name follows conventions
# ✓ Inputs properly typed
# ✓ Outputs properly typed
# ✓ No missing required fields

2. Runtime Testing

# Dry-run to catch runtime errors without side effects
swamp model run aws-inventory --dry-run \
  --input '{"cloud_provider": "aws"}'

# Expected output:
# ✓ Model initializes
# ✓ All steps execute (or fail with clear errors)
# ✓ Outputs match schema

3. Data Type Validation

# Verify output matches declared types
swamp model run aws-inventory --json | \
  jq '.[].outputs | keys' # Check output names

swamp model run aws-inventory --json | \
  jq '.[].outputs.resources | type' # Verify array type

4. Integration Testing

# Test workflow bindings
swamp workflow validate process-cloud-inventory

# Expected output:
# ✓ All inputs bound to outputs
# ✓ No undefined variables
# ✓ Data types consistent

---

Summary: Why PR #66 Failed

Category	Issue	Impact	Prevention
Template Syntax	${{ }} instead of ${}	Interpolation failed silently	Use swamp workflow validate
Data Contracts	Output names ≠ input names	Data bindings broke silently	Document contracts, validate integration
Error Handling	allowFailure: true hides mistakes	Failures invisible until production	Use only for genuinely optional steps
Naming	Inconsistent conventions	Hard to track data flow	Follow kebab-case, snake_case patterns
Testing	No validation before publishing	No feedback before publication	Use swamp model/workflow validate

Lessons Learned:

1. Syntax errors are silent — Always validate templates
2. Data contracts are critical — Document and enforce input/output bindings
3. allowFailure masks mistakes — Use only for truly optional steps
4. Testing is non-negotiable — Run validation before publication
5. Adversarial review catches hidden assumptions — Challenge every design decision

---

Resources

• swamp-extension skill: Model, vault, driver, datastore, report creation
• swamp-extension-publish skill: Publishing state machine and checklist
• swamp-workflow skill: Workflow composition, orchestration, testing
• swamp-model skill: Model development, validation, profiling
• issue-lifecycle skill: Adversarial review patterns for architecture
• GitHub: https://github.com/webframp/swamp-extensions
• swamp.club: https://swamp.club/leaderboard (track usage metrics)

---

Last updated: 2025-05-21 Analysis based on systeminit/swamp (commit 694acbd4) and PR webframp/swamp-extensions#66