DDEV Add-on Ecosystem Improvements - Container Runtime Proposal (php)

ddev-addon-improvements-proposal.md

DDEV Add-on Ecosystem Improvements

Problem Statement

DDEV's current add-on ecosystem has two key weaknesses:

1. Complex install.yaml syntax - Advanced add-ons like ddev-platformsh result in unreadable and unmaintainable bash scripts mixed with Go templating
2. Host dependency issues - install.yaml relies on locally installed tools (primarily bash), but bash capabilities vary across environments, leading to inconsistent success rates

Current Architecture Analysis

Key Components

• cmd/ddev/cmd/addon*.go - CLI commands for add-on management
• pkg/ddevapp/addons.go - Core add-on processing logic
• ProcessAddonAction() - Executes bash scripts via exec.RunHostCommand(bashPath, "-c", action)

Current Limitations

• Actions execute on host with exec.RunHostCommand(bashPath, "-c", action) in pkg/ddevapp/addons.go:136
• Complex bash/Go template mixing in install.yaml creates debugging challenges
• Host bash variations cause environment-specific failures
• Limited error handling and recovery options
• Go templating is difficult to use and debug

Phase 1 Solution: Container Runtime with Backward Compatibility

Design Principles

1. Zero Breaking Changes - All existing add-ons work unchanged
2. Opt-in Container Execution - New behavior only when explicitly requested
3. Graceful Fallback - Falls back to host execution if container unavailable
4. Progressive Enhancement - New add-ons can use containers while old ones use bash

Enhanced Schema (Backward Compatible)

// pkg/ddevapp/addons.go - extend existing InstallDesc struct
type InstallDesc struct {
    // Existing fields remain unchanged
    Name                  string            `yaml:"name"`
    ProjectFiles          []string          `yaml:"project_files"`
    GlobalFiles           []string          `yaml:"global_files,omitempty"`
    DdevVersionConstraint string            `yaml:"ddev_version_constraint,omitempty"`
    Dependencies          []string          `yaml:"dependencies,omitempty"`
    PreInstallActions     []string          `yaml:"pre_install_actions,omitempty"`
    PostInstallActions    []string          `yaml:"post_install_actions,omitempty"`
    RemovalActions        []string          `yaml:"removal_actions,omitempty"`
    YamlReadFiles         map[string]string `yaml:"yaml_read_files"`
    
    // NEW: Optional container execution settings
    ContainerRuntime      *ContainerConfig  `yaml:"container_runtime,omitempty"`
}

type ContainerConfig struct {
    Enabled         bool              `yaml:"enabled,omitempty"`           // Default: false
    Image           string            `yaml:"image,omitempty"`             // Default: "ddev/addon-runner:latest"
    Timeout         int               `yaml:"timeout_seconds,omitempty"`   // Default: 300
    Environment     map[string]string `yaml:"environment,omitempty"`       // Additional env vars
}

API Changes

// NEW: Enhanced function with container support
func ProcessAddonActionWithConfig(action string, dict map[string]interface{}, bashPath string, verbose bool, containerConfig *ContainerConfig, app *DdevApp) error {
    // Check for debug environment variables
    if os.Getenv("DDEV_ADDON_DEBUG") == "true" || os.Getenv("DDEV_ADDON_DEBUG") == "host" {
        util.Warning("DDEV_ADDON_DEBUG set - executing on host for debugging")
        return ProcessAddonAction(action, dict, bashPath, verbose)
    }
    
    if os.Getenv("DDEV_ADDON_DEBUG") == "interactive" {
        util.Warning("DDEV_ADDON_DEBUG=interactive - starting debug container")
        return executeInDebugContainer(app, action, containerConfig)
    }
    
    // Container execution path
    if containerConfig \!= nil && containerConfig.Enabled {
        return processAddonActionInContainer(action, dict, verbose, containerConfig, app)
    }
    
    // Fallback to existing implementation - NO CHANGES to current logic
    return ProcessAddonAction(action, dict, bashPath, verbose)
}

// Keep existing function unchanged for backward compatibility
func ProcessAddonAction(action string, dict map[string]interface{}, bashPath string, verbose bool) error {
    // Current implementation remains EXACTLY the same - no changes
}

Container Execution Strategy

func executeInAddonContainer(app *DdevApp, action string, containerConfig *ContainerConfig, addonSourceDir string) error {
    // Comprehensive mount strategy
    mounts := []string{
        // 1. DDEV project configuration directory (where files get installed)
        fmt.Sprintf("%s:/mnt/ddev-config", app.AppConfDir()),
        
        // 2. Project root (for accessing project files like package.json, composer.json)
        fmt.Sprintf("%s:/mnt/project-root:ro", app.AppRoot),
        
        // 3. Add-on source directory (templates, configs, scripts)
        fmt.Sprintf("%s:/mnt/addon-source:ro", addonSourceDir),
        
        // 4. Generated script file
        fmt.Sprintf("%s:/tmp/addon-script:ro", scriptFile),
        
        // 5. Global DDEV config (for accessing global settings)
        fmt.Sprintf("%s:/mnt/global-ddev:ro", globalconfig.GetGlobalDdevDir()),
        
        // 6. Configuration data as JSON (alternative to Go templating)
        fmt.Sprintf("%s:/tmp/ddev-config.json:ro", configJsonFile),
    }
    
    // Container execution with proper context and fallback mechanisms
    // ... implementation details
}

Handling YAML Imports and Moving Away from Go Templating

Current yaml_read_files Challenge

The existing yaml_read_files system creates configuration data that's processed through Go templating on the host:

// Current flow in cmd/ddev/cmd/addon-get.go:162-189
yamlMap := make(map[string]interface{})
for name, f := range s.YamlReadFiles {
    yamlMap[name], err = util.YamlFileToMap(fullpath)
}
yamlMap["DdevGlobalConfig"], err = util.YamlFileToMap(globalconfig.GetGlobalConfigPath())
yamlMap["DdevProjectConfig"] = projectConfigMap

dict, err := util.YamlToDict(yamlMap)
// dict is used for Go templating: {{ .DdevProjectConfig.name }}

Phase 1 Solution: Hybrid Approach

Maintain Go templating for backward compatibility while providing cleaner alternatives:

func processAddonActionInContainer(action string, dict map[string]interface{}, verbose bool, containerConfig *ContainerConfig, app *DdevApp) error {
    // OPTION 1: Pre-process Go templating on host (backward compatibility)
    action = "set -eu -o pipefail\n" + action
    t, err := template.New("ProcessAddonAction").Funcs(getTemplateFuncMap()).Parse(action)
    if err \!= nil {
        return fmt.Errorf("could not parse action '%s': %v", action, err)
    }

    var doc bytes.Buffer
    err = t.Execute(&doc, dict)  // Go templating happens on HOST
    if err \!= nil {
        return fmt.Errorf("could not parse/execute action '%s': %v", action, err)
    }
    processedAction := doc.String()  // Fully-processed script with values substituted
    
    // OPTION 2: Also export config as JSON for modern usage
    configJsonFile, err := createConfigJsonFile(dict)
    if err \!= nil {
        return err
    }
    defer os.Remove(configJsonFile)
    
    // Send processed script + config file to container
    return executeInAddonContainer(app, processedAction, containerConfig, addonSourceDir, configJsonFile)
}

func createConfigJsonFile(dict map[string]interface{}) (string, error) {
    configData, err := json.MarshalIndent(dict, "", "  ")
    if err \!= nil {
        return "", err
    }
    
    tmpFile, err := os.CreateTemp("", "ddev-addon-config-*.json")
    if err \!= nil {
        return "", err
    }
    
    _, err = tmpFile.Write(configData)
    tmpFile.Close()
    return tmpFile.Name(), err
}

Migration Examples

Legacy add-on (Go templating - still works):

name: legacy-addon
yaml_read_files:
  myconfig: "config.yaml"
pre_install_actions:
  - echo "Project name is {{ .DdevProjectConfig.name }}"
  - echo "Custom value is {{ .myconfig.some_value }}"

Modern PHP add-on (JSON config):

name: php-addon
yaml_read_files:
  myconfig: "config.yaml"  # Still processed, available in JSON
container_runtime:
  enabled: true
pre_install_actions:
  - |
    <?php
    // Read all config data from standard JSON file
    $config = json_decode(file_get_contents('/tmp/ddev-config.json'), true);
    
    $projectName = $config['DdevProjectConfig']['name'];
    $customValue = $config['myconfig']['some_value'];
    
    echo "Setting up PHP project: $projectName\n";
    echo "Custom configuration: $customValue\n";
    
    // Access project files directly
    $composerJson = json_decode(file_get_contents('/mnt/project-root/composer.json'), true);
    if (isset($composerJson['require']['drupal/core'])) {
        exec('composer require drush/drush');
    }
    ?>

Modern Node.js add-on:

name: node-addon  
yaml_read_files:
  buildconfig: "build.yaml"
container_runtime:
  enabled: true
pre_install_actions:
  - |
    #\!/usr/bin/env node
    const fs = require('fs');
    
    // Read configuration
    const config = JSON.parse(fs.readFileSync('/tmp/ddev-config.json'));
    const projectName = config.DdevProjectConfig.name;
    const buildConfig = config.buildconfig;
    
    console.log(`Setting up Node.js project: ${projectName}`);
    
    // Generate webpack config based on DDEV + custom settings
    const webpackConfig = {
      entry: './src/index.js',
      output: {
        path: '/mnt/ddev-config/web/dist',
        publicPath: `https://${projectName}.ddev.site/dist/`
      },
      // ... use buildConfig values
    };
    
    fs.writeFileSync('.ddev/webpack.config.js', 
      `module.exports = ${JSON.stringify(webpackConfig, null, 2)}`);

Configuration Data Structure

The JSON config file provides the same data that was available via Go templating:

{
  "DdevProjectConfig": {
    "name": "my-project",
    "type": "drupal10", 
    "database": "mysql:8.0",
    "webserver_type": "nginx-fpm",
    "docroot": "web"
  },
  "DdevGlobalConfig": {
    "web_environment": {
      "CUSTOM_VAR": "value"
    }
  },
  "myconfig": {
    "some_value": "from yaml_read_files"
  },
  "buildconfig": {
    "webpack_mode": "development"
  }
}

Debugging Strategy

Container execution introduces significant debugging challenges, but we provide multiple solutions:

Environment-based Debug Control

# Normal installation (container execution if enabled)
ddev add-on get myaddon

# Force host execution for full debugging capabilities  
DDEV_ADDON_DEBUG=host ddev add-on get myaddon

# Interactive container debugging
DDEV_ADDON_DEBUG=interactive ddev add-on get myaddon

# Verbose container execution with detailed logging
DDEV_ADDON_DEBUG=verbose ddev add-on get myaddon

Debug Mode Implementation

func ProcessAddonActionWithConfig(...) error {
    // Check debug environment variables
    debugMode := os.Getenv("DDEV_ADDON_DEBUG")
    
    switch debugMode {
    case "true", "host":
        util.Warning("DDEV_ADDON_DEBUG=%s - executing on host for debugging", debugMode)
        return ProcessAddonAction(action, dict, bashPath, verbose)
        
    case "interactive":
        util.Warning("DDEV_ADDON_DEBUG=interactive - starting debug container")
        return executeInDebugContainer(app, action, containerConfig)
        
    case "verbose":
        verbose = true // Enable detailed container logging
    }
    
    // Normal execution path...
}

func executeInDebugContainer(app *DdevApp, action string, containerConfig *ContainerConfig) error {
    // Create persistent debug container instead of ephemeral one
    containerName := fmt.Sprintf("ddev-addon-debug-%s", app.Name)
    
    dockerCmd := []string{
        "run", "-it",  // Interactive terminal
        "--name", containerName,
        // ... same mounts as production
        "-v", fmt.Sprintf("%s:/tmp/debug-script.sh", scriptFile),
        "-v", fmt.Sprintf("%s:/tmp/ddev-config.json:ro", configJsonFile),
        image,
        "bash", // Drop into shell for manual debugging
    }
    
    util.Warning("Debug container started. To execute your script:")
    util.Warning("  bash /tmp/debug-script.sh")
    util.Warning("")
    util.Warning("Available files:")
    util.Warning("  /tmp/debug-script.sh - Your processed addon script")
    util.Warning("  /tmp/ddev-config.json - Configuration data")
    util.Warning("  /mnt/ddev-config/ - Project .ddev directory")
    util.Warning("  /mnt/project-root/ - Project root directory")
    util.Warning("  /mnt/addon-source/ - Add-on source files")
    util.Warning("")
    util.Warning("Container name: %s", containerName)
    util.Warning("Use 'docker exec -it %s bash' for additional shells", containerName)
    
    return exec.RunInteractiveCommand("docker", dockerCmd...)
}

Debugging Workflow Examples

PHP Add-on Debugging:

# Start interactive debug session
DDEV_ADDON_DEBUG=interactive ddev add-on get my-php-addon

# Inside container:
bash-5.1# php -l /tmp/debug-script.sh  # Check syntax
bash-5.1# cat /tmp/ddev-config.json | jq .  # Inspect config
bash-5.1# php /tmp/debug-script.sh  # Execute with full error output

Step-by-step Debugging:

# Host debugging for step-debugging with IDE
DDEV_ADDON_DEBUG=host ddev add-on get my-addon

# Or extract and debug manually:
ddev add-on get my-addon --extract-only /tmp/addon-debug
cd /tmp/addon-debug
# Edit and test install.yaml actions directly

Enhanced Logging

func processAddonActionInContainer(...) error {
    if verbose || os.Getenv("DDEV_ADDON_DEBUG") == "verbose" {
        // Log complete container setup
        util.Warning("Container execution details:")
        util.Warning("  Image: %s", image)
        util.Warning("  Working directory: %s", workingDir)
        util.Warning("  Timeout: %d seconds", timeout)
        util.Warning("  Mounts:")
        for _, mount := range mounts {
            util.Warning("    %s", mount)
        }
        util.Warning("  Environment variables:")
        for k, v := range containerConfig.Environment {
            util.Warning("    %s=%s", k, v)
        }
        
        // Save reproducible docker command
        cmdStr := strings.Join(dockerCmd, " ")
        util.Warning("  Reproduction command:")
        util.Warning("    docker %s", cmdStr)
        util.Warning("")
    }
    
    // ... execute and capture output
    
    if verbose || os.Getenv("DDEV_ADDON_DEBUG") == "verbose" {
        util.Warning("Container output:")
        util.Warning(output)
        if err \!= nil {
            util.Warning("Container error: %v", err)
        }
    }
}

Debug-friendly Development Workflow

1. Start with host debugging: DDEV_ADDON_DEBUG=host for full IDE integration
2. Move to verbose container: DDEV_ADDON_DEBUG=verbose to see container setup
3. Use interactive debugging: DDEV_ADDON_DEBUG=interactive for container environment testing
4. Production testing: Normal execution to verify final behavior

This approach maintains debugging capabilities while providing the benefits of containerized execution.

Multi-Language Support

Container execution is language-agnostic - add-on authors specify their interpreter:

# PHP execution
name: drupal-addon
container_runtime:
  enabled: true
pre_install_actions:
  - |
    <?php
    // Read configuration from standard JSON file
    $config = json_decode(file_get_contents('/tmp/ddev-config.json'), true);
    
    $projectName = $config['DdevProjectConfig']['name'];
    echo "Setting up Drupal environment for $projectName\n";
    
    // Generate Drupal-specific configuration
    $drushConfig = [
        'drush' => [
            'paths' => ['/var/www/html/vendor/bin/drush']
        ]
    ];
    file_put_contents('.ddev/config.drush.yaml', yaml_emit($drushConfig));
    ?>

# Node.js execution  
name: frontend-addon
container_runtime:
  enabled: true
pre_install_actions:
  - |
    #\!/usr/bin/env node
    const fs = require('fs');
    
    // Read configuration
    const config = JSON.parse(fs.readFileSync('/tmp/ddev-config.json'));
    console.log(`Setting up frontend tools for ${config.DdevProjectConfig.name}`);

# Bash (backward compatible)
name: traditional-addon
container_runtime:
  enabled: true
pre_install_actions:
  - |
    #\!/bin/bash
    echo "Still works with bash"
    composer install

Addon-Runner Container

# containers/ddev-addon-runner/Dockerfile
FROM debian:bookworm-slim

# Install standardized toolset
RUN apt-get update && apt-get install -y \
    bash curl wget git jq unzip ca-certificates \
    php-cli php-yaml composer nodejs npm \
    && rm -rf /var/lib/apt/lists/*

# Create standard directories accessible from all mount points
RUN mkdir -p /mnt/ddev-config /mnt/project-root /mnt/addon-source /mnt/global-ddev

# Set consistent environment
ENV DEBIAN_FRONTEND=noninteractive
ENV PATH="/usr/local/bin:$PATH"

WORKDIR /mnt/ddev-config
CMD ["bash"]

Backward Compatibility Guarantees

• Zero Breaking Changes: All existing install.yaml files work unchanged
• API Compatibility: ProcessAddonAction function signature unchanged
• Fallback Safety: Container failures fall back to host execution
• Progressive Enhancement: New features only activate when explicitly enabled
• Go Templating Preserved: Existing yaml_read_files and Go templating continue to work
• Debugging Maintained: Host execution mode preserves full debugging capabilities

Benefits

1. Eliminates Host Dependencies - Standardized container environment
2. Reduces Complexity - Cleaner separation of concerns and simpler config access
3. Improves Reliability - Consistent execution environment
4. Maintains Compatibility - Zero impact on existing add-ons
5. Enables Innovation - Foundation for future declarative improvements
6. Better Debugging - Multiple debugging modes for different development needs
7. Language Flexibility - Support for PHP, Python, Node.js, bash, and others