gistfile1.txt

RFC: Implementing Circuit Breaker Functionality in Backstage Backend Using Opossum
Metadata
	•	RFC ID: [Internal - Circuit Breaker Integration]
	•	Authors: Grok (based on user query)
	•	Status: Draft Proposal
	•	Date: September 29, 2025
	•	Version: 1.0
Abstract
This RFC proposes the integration of circuit breaker patterns into the Backstage backend to enhance resilience against failures in API endpoints. We evaluate two options: a global circuit breaker applied to the entire backend and a per-plugin circuit breaker that can be optionally added to individual plugins. We recommend the per-plugin approach for better isolation and extensibility, avoiding modifications to core Backstage components. As an example, we demonstrate how to apply this to the Catalog API plugin.
Introduction
Backstage’s backend uses a modular plugin system where each plugin exposes API endpoints under /api/. To prevent cascading failures (e.g., due to downstream service outages or internal errors), we propose using the Opossum library for circuit breakers. Opossum monitors request outcomes and opens the circuit on repeated failures, rejecting subsequent requests with a 503 (Service Unavailable) until recovery.
Key goals:
	•	Protect against failures without impacting unrelated plugins.
	•	Minimize changes to existing code; support automatic or opt-in configuration.
	•	Configurable via app-config.yaml (e.g., thresholds for error percentage, timeout).
Assumptions:
	•	Using the new Backstage backend system (@backstage/backend-app-api).
	•	Failures defined as response status >= 500 (configurable).
	•	Opossum installed via yarn add opossum.
Options
Option 1: Global Circuit Breaker
Apply a single circuit breaker to the root HTTP router, wrapping all incoming requests to the backend.
Implementation Outline:
	•	In packages/backend/src/index.ts, access the root HTTP router via coreServices.rootHttpRouter.
	•	Create a middleware that integrates Opossum to fire on every request, observing response status via res.on('finish').
	•	If the circuit opens, reject all requests with 503.
	•	Example code snippet:
import { createBackend } from '@backstage/backend-defaults';
	•	import CircuitBreaker from 'opossum';
	•	import { coreServices } from '@backstage/backend-plugin-api';
	•	
	•	const backend = createBackend();
	•	
	•	// Custom root router wrapper
	•	backend.add(createServiceFactory({
	•	  service: coreServices.rootHttpRouter,
	•	  deps: { config: coreServices.rootConfig },
	•	  async factory({ config }) {
	•	    const router = PromiseRouter(); // Use express-promise-router
	•	    const circuit = new CircuitBreaker(async (req, res, next) => {
	•	      // Similar observation logic as per-plugin
	•	      return new Promise((resolve, reject) => {
	•	        // ... (observe res.finish, resolve/reject based on status)
	•	      });
	•	    }, { /* options from config */ });
	•	
	•	    router.use(async (req, res, next) => {
	•	      try {
	•	        await circuit.fire(req, res, next);
	•	      } catch (err) {
	•	        res.status(503).send('Service Unavailable');
	•	      }
	•	    });
	•	
	•	    return {
	•	      use(handler) { router.use(handler); },
	•	      register(...args) { router.register(...args); },
	•	    };
	•	  },
	•	}));
	•	
	•	backend.start();
	•	
Pros:
	•	Simple to implement; protects the entire backend with minimal code.
	•	Centralized configuration.
Cons:
	•	Lack of isolation: A failure in one plugin (e.g., high error rate in Catalog) could open the circuit, blocking requests to all plugins (e.g., TechDocs, Scaffolder).
	•	Requires forking or overriding the out-of-the-box root HTTP router service, which is not recommended as it modifies core behavior and could break future Backstage updates or make the codebase less maintainable.
	•	No per-plugin customization (e.g., different thresholds for critical vs. non-critical plugins).
Option 2: Per-Plugin Circuit Breaker (Opt-In)
Provide a utility to wrap individual plugin routers with their own circuit breaker instances. Plugins can opt-in by applying this wrapper during setup.
Implementation Outline:
	•	Create a shared utility function (e.g., in packages/backend/src/lib/circuitBreaker.ts) that returns a middleware powered by Opossum.
	•	Plugins import and apply this middleware to their router.
	•	Each plugin gets an independent circuit, ensuring failures are isolated.
	•	No override of core services; fully extensible.
Pros:
	•	Isolation: Failures in one plugin don’t affect others.
	•	Opt-in flexibility: Only apply to plugins that need it (e.g., those with external dependencies).
	•	No forking of the out-of-the-box HTTP router; uses standard plugin router APIs.
	•	Supports per-plugin configuration (e.g., via plugin-specific config sections).
Cons:
	•	Requires minor changes in each plugin that opts in (but this is minimal and backward-compatible).
	•	Slightly more decentralized management.
Recommendation
We recommend Option 2: Per-Plugin Circuit Breaker (Opt-In). A global approach violates the principle of isolation in a modular system like Backstage, where plugins should fail independently. Moreover, implementing a global circuit breaker necessitates forking or overriding the out-of-the-box root HTTP router service, which introduces maintenance overhead, potential conflicts with Backstage updates, and deviates from the plugin-centric architecture. The per-plugin approach aligns with Backstage’s extensibility model, allowing gradual adoption without core modifications.
Implementation Details for Per-Plugin Approach
Shared Utility
Create a utility in packages/backend/src/lib/circuitBreaker.ts:
import CircuitBreaker from 'opossum';
import { Handler, Request, Response, NextFunction } from 'express';

export function createCircuitBreakerMiddleware(options: any = {}): Handler {
  const defaultOptions = {
    timeout: 3000,
    errorThresholdPercentage: 50,
    resetTimeout: 30000,
    volumeThreshold: 10,
    ...options,
  };

  const circuit = new CircuitBreaker(async (req: Request, res: Response, next: NextFunction) => {
    return new Promise((resolve, reject) => {
      const cleanup = () => {
        res.removeListener('finish', onFinish);
        res.removeListener('error', onError);
      };
      const onFinish = () => {
        cleanup();
        if (res.statusCode >= 500) {
          reject(new Error(`Failure: ${res.statusCode}`));
        } else {
          resolve();
        }
      };
      const onError = (err: Error) => {
        cleanup();
        reject(err);
      };
      res.on('finish', onFinish);
      res.on('error', onError);
      next();
    });
  }, defaultOptions);

  // Event logging (optional)
  circuit.on('open', () => console.log('Circuit opened'));
  circuit.on('close', () => console.log('Circuit closed'));
  circuit.on('halfOpen', () => console.log('Circuit half-open'));

  return async (req: Request, res: Response, next: NextFunction) => {
    try {
      await circuit.fire(req, res, next);
    } catch (err) {
      if (circuit.opened) {
        res.status(503).send('Service Unavailable');
      } else {
        next(err);
      }
    }
  };
}
This middleware can be configured per-plugin (e.g., pass options from config).
Example: Adding to Catalog API Plugin
For the Catalog backend (@backstage/plugin-catalog-backend), we can create a custom module that extends the default with the circuit breaker. In packages/backend/src/plugins/catalog.ts (or similar):
import { CatalogBuilder } from '@backstage/plugin-catalog-backend';
import { createRouter } from '@backstage/plugin-catalog-backend/alpha';
import { PluginEnvironment } from '../types'; // Assume this has logger, config, etc.
import { createCircuitBreakerMiddleware } from '../lib/circuitBreaker';

export default async function createPlugin(env: PluginEnvironment) {
  const builder = await CatalogBuilder.create(env);
  // Add processors, etc., as usual
  const { processingEngine, router: defaultRouter } = await builder.build();
  await processingEngine.start();

  // Create wrapped router with circuit breaker
  const circuitMiddleware = createCircuitBreakerMiddleware({
    // Optional: Pull from config, e.g., env.config.getConfig('catalog.circuitBreaker')
  });

  // Apply middleware to the default router
  defaultRouter.use(circuitMiddleware);

  return defaultRouter; // Or use createRouter if overriding fully
}
In packages/backend/src/index.ts, add the plugin as usual:
import { createBackend } from '@backstage/backend-defaults';
import catalogPlugin from './plugins/catalog';

const backend = createBackend();
backend.add(catalogPlugin); // Now includes circuit breaker
backend.start();
This adds the circuit breaker only to /api/catalog endpoints. Other plugins remain unaffected unless they similarly opt in.
Next Steps
	•	Prototype and test with simulated failures.
	•	Add config schema for thresholds.
	•	Consider metrics integration (e.g., via @backstage/backend-plugin-api telemetry).