bootstrapper-refactoring-plan.md

Bootstrapper Class Refactoring Plan

Executive Summary

The current Bootstrapper class in src/fromager/bootstrapper.py violates the Single Responsibility Principle by combining 7 distinct responsibilities into a single 1500+ line class. This document outlines a decomposition strategy to improve maintainability, testability, and code organization.

Current Problems

1. Monolithic Design: Single class handling version resolution, caching, builds, dependencies, error handling, and state tracking
2. Testing Complexity: Difficult to unit test individual concerns in isolation
3. Tight Coupling: Changes in one area (e.g., caching) can affect unrelated areas (e.g., version resolution)
4. Poor Reusability: Individual components cannot be reused independently
5. Maintenance Burden: Large class makes it difficult to understand and modify behavior

Identified Responsibilities

1. Version Resolution (Lines 169-196, 1065-1351)

• Resolving package versions from PyPI, git URLs, and cached graphs
• Handling git repository cloning and version extraction
• Managing resolution caching and constraint satisfaction

2. Dependency Management (Lines 531-601, 657-706)

• Extracting build dependencies (build-system, build-backend, build-sdist)
• Extracting install dependencies from wheels and sdists
• Managing dependency installation in build environments

3. Source Management (Lines 707-736, 797-820)

• Downloading source archives and git repositories
• Unpacking and preparing source code for building
• Managing working directories and source preparation

4. Build Orchestration (Lines 749-776, 821-852)

• Coordinating wheel and sdist builds
• Managing build environments and build processes
• Handling cached wheel usage and build fallbacks

5. Cache Management (Lines 618-656, 943-1027)

• Finding cached wheels across multiple locations
• Managing local, download, and remote cache hierarchies
• Validating build tags and extracting metadata

6. Error Handling & Test Mode (Lines 440-472, 871-941)

• Recording test mode failures with categorization
• Managing fallback strategies for failed builds
• Generating comprehensive failure reports

7. State Tracking (Lines 1432-1499)

• Tracking seen requirements and build orders
• Managing dependency graph updates
• Maintaining progress and completion state

Proposed Decomposition

1. PackageVersionResolver

Responsibility: Resolve package versions from all sources

Public Interface:

class PackageVersionResolver:
    def resolve_version(self, req: Requirement, req_type: RequirementType) -> tuple[str, Version]
    def clear_resolution_cache(self) -> None
    def get_cached_resolution(self, req: Requirement) -> tuple[str, Version] | None

Key Methods:

• resolve_source_version() - Handle PyPI and git URL resolution
• resolve_prebuilt_version() - Handle wheel resolution from servers
• resolve_version_from_git_url() - Git cloning and version extraction
• get_version_from_package_metadata() - Extract version from package metadata
• resolve_from_previous_graph() - Use cached resolutions from dependency graph
• resolve_from_version_source() - Resolve using provided version candidates

Dependencies: WorkContext, DependencyGraph, sources module, wheels module, resolver module

2. PackageDependencyExtractor

Responsibility: Extract and manage all types of dependencies

Public Interface:

class PackageDependencyExtractor:
    def get_build_dependencies(self, req: Requirement, sdist_root_dir: Path, build_env: BuildEnvironment) -> set[Requirement]
    def get_install_dependencies(self, req: Requirement, resolved_version: Version, wheel_filename: Path | None, sdist_filename: Path | None, sdist_root_dir: Path | None, build_env: BuildEnvironment | None, unpack_dir: Path | None) -> list[Requirement]

Key Methods:

• get_build_system_dependencies() - Extract build system requirements
• get_build_backend_dependencies() - Extract build backend requirements
• get_build_sdist_dependencies() - Extract build sdist requirements
• extract_from_wheel() - Extract install dependencies from wheels
• extract_from_sdist() - Extract install dependencies from source

Dependencies: WorkContext, dependencies module, BuildEnvironment

3. PackageSourceManager

Responsibility: Handle source code operations

Public Interface:

class PackageSourceManager:
    def download_source(self, req: Requirement, resolved_version: Version, source_url: str) -> Path
    def prepare_source(self, req: Requirement, source_filename: Path, version: Version) -> Path
    def download_git_source(self, req: Requirement, url_to_clone: str, destination_dir: Path, ref: str | None) -> None

Key Methods:

• download_source_archive() - Download source archives from URLs
• prepare_source_directory() - Unpack and prepare source for building
• clone_git_repository() - Clone git repositories with specific refs
• get_source_type() - Determine source type (git, archive, etc.)
• create_unpack_directory() - Create working directories for source

Dependencies: WorkContext, sources module

4. PackageBuildOrchestrator

Responsibility: Coordinate builds and manage build environments

Public Interface:

class PackageBuildOrchestrator:
    def build_package(self, req: Requirement, resolved_version: Version, sdist_root_dir: Path, build_sdist_only: bool, cached_wheel_filename: Path | None) -> tuple[Path | None, Path | None]
    def create_build_environment(self, req: Requirement, resolved_version: Version, parent_dir: Path) -> BuildEnvironment

Key Methods:

• build_wheel() - Build wheels from prepared source
• build_sdist() - Build source distributions
• use_cached_wheel() - Use existing cached wheels
• update_wheel_mirror() - Update local wheel mirrors
• clean_build_directories() - Clean up build artifacts

Dependencies: WorkContext, build_environment module, wheels module, server module

5. PackageCacheManager

Responsibility: Handle all caching operations

Public Interface:

class PackageCacheManager:
    def find_cached_wheel(self, req: Requirement, resolved_version: Version) -> tuple[Path | None, Path | None]
    def download_prebuilt_wheel(self, req: Requirement, req_type: RequirementType, resolved_version: Version, wheel_url: str) -> tuple[Path, Path]

Key Methods:

• look_for_existing_wheel() - Search local directories for cached wheels
• download_wheel_from_cache_server() - Download from remote cache servers
• validate_build_tag() - Ensure build tags match expectations
• unpack_metadata_from_wheel() - Extract metadata and requirements
• extract_build_requirements_from_wheel() - Extract fromager build requirements

Dependencies: WorkContext, wheels module, finders module, server module

6. TestModeErrorHandler

Responsibility: Handle test mode failures and fallback strategies

Public Interface:

class TestModeErrorHandler:
    def record_failure(self, req: Requirement, version: str | None, err: Exception, failure_type: FailureType, log_level: Literal["error", "warning"] = "error") -> None
    def handle_build_failure(self, req: Requirement, resolved_version: Version, req_type: RequirementType, build_error: Exception) -> SourceBuildResult | None
    def generate_failure_report(self) -> dict[str, list[FailureRecord]]
    def write_failure_report_to_file(self, work_dir: Path) -> Path

Key Methods:

• attempt_prebuilt_fallback() - Try prebuilt wheels when source builds fail
• log_failure_message() - Log formatted failure messages with context
• get_failed_packages() - Get list of all recorded failures
• clear_failures() - Reset failure tracking state
• has_failures() - Check if any failures have been recorded

Dependencies: PackageVersionResolver, PackageCacheManager (for fallback)

7. BootstrapStateTracker

Responsibility: Track bootstrap state and progress

Public Interface:

class BootstrapStateTracker:
    def mark_as_seen(self, req: Requirement, version: Version, sdist_only: bool = False) -> None
    def has_been_seen(self, req: Requirement, version: Version, sdist_only: bool = False) -> bool
    def add_to_build_order(self, req: Requirement, version: Version, source_url: str, source_type: SourceType, prebuilt: bool = False, constraint: Requirement | None = None) -> None
    def get_build_order(self) -> list[dict[str, Any]]
    def write_build_order_to_file(self) -> None

Key Methods:

• create_resolved_key() - Generate unique keys for requirement tracking
• is_already_in_build_order() - Check if package already in build queue
• add_to_dependency_graph() - Update dependency graph with new edges
• sort_requirements() - Sort requirements for deterministic processing

Dependencies: WorkContext, DependencyGraph

8. PackageBootstrapper (Main Orchestrator)

Responsibility: Orchestrate the bootstrap process using service classes

Public Interface:

class PackageBootstrapper:
    def resolve_and_add_top_level(self, req: Requirement) -> tuple[str, Version] | None
    def bootstrap(self, req: Requirement, req_type: RequirementType) -> None
    def finalize(self) -> int

Key Methods:

• bootstrap_implementation() - Core bootstrap logic using services
• handle_build_requirements() - Process build dependencies recursively
• track_dependency_chain() - Context manager for dependency tracking
• processing_build_requirement() - Determine if processing build deps

Dependencies: All service classes listed above

Implementation Strategy

Phase 1: Extract Service Classes (Low Risk)

Risk Level: Low - These classes encapsulate well-defined functionality

1. PackageVersionResolver - Extract version resolution logic

   • Move methods: resolve_version(), _resolve_source_with_history(), etc.
   • Minimal external dependencies
   • Easy to unit test in isolation

2. PackageCacheManager - Extract caching operations

   • Move methods: _find_cached_wheel(), _look_for_existing_wheel(), etc.
   • Clear separation from other concerns
   • Straightforward to test with mock file systems

3. PackageSourceManager - Extract source management

   • Move methods: _download_source(), _prepare_source(), etc.
   • Well-defined input/output contracts
   • Can be tested with temporary directories

4. TestModeErrorHandler - Extract error handling/reporting

   • Move methods: _record_test_mode_failure(), _handle_test_mode_failure()
   • Pure data management with clear interfaces
   • Easy to test error scenarios

Phase 2: Extract Core Logic (Medium Risk)

Risk Level: Medium - These classes have more complex interactions

5. PackageDependencyExtractor - Extract dependency management

   • Move methods: _prepare_build_dependencies(), _get_install_dependencies()
   • Requires coordination with build environments
   • Needs careful testing of dependency resolution

6. PackageBuildOrchestrator - Extract build coordination

   • Move methods: _do_build(), _build_wheel(), _build_sdist()
   • Complex interactions with build environments
   • Requires integration testing with actual builds

7. BootstrapStateTracker - Extract state management

   • Move methods: _mark_as_seen(), _add_to_build_order(), etc.
   • Manages critical bootstrap state
   • Needs careful testing of state consistency

Phase 3: Refactor Main Class (Higher Risk)

Risk Level: Higher - Changes main orchestration logic

8. PackageBootstrapper Orchestrator - Refactor main class

   • Rewrite bootstrap() and _bootstrap_impl() to use services
   • Update dependency injection and service coordination
   • Comprehensive integration testing required

9. Update Imports and Callers - System-wide changes

   • Update all imports of Bootstrapper class
   • Ensure backward compatibility during transition
   • Update command-line interface and entry points

10. Add Integration Tests - Verify end-to-end behavior

    • Test complete bootstrap workflows
    • Verify test mode functionality
    • Performance regression testing

Risk Mitigation Strategy

Testing Strategy

1. Unit Tests for Each Service: Write comprehensive unit tests for each extracted class before integration
2. Mock Dependencies: Use dependency injection and mocking to isolate components
3. File-Scoped Testing: Run hatch run test:test tests/test_<module>.py after each extraction
4. Integration Testing: Maintain end-to-end tests to verify overall behavior

Backward Compatibility

1. Gradual Migration: Keep original Bootstrapper class during transition
2. Facade Pattern: Provide compatibility wrapper if needed
3. Feature Flags: Use configuration to enable new implementation gradually
4. Rollback Plan: Maintain ability to revert to original implementation

Quality Assurance

1. Code Reviews: Each extracted class requires thorough review
2. Type Checking: Run hatch run mypy:check on all new classes
3. Lint Checks: Ensure hatch run lint:fix passes on all new code
4. Performance Testing: Verify no performance regressions

Expected Benefits

Maintainability

• Single Responsibility: Each class has one clear purpose
• Easier Debugging: Problems isolated to specific components
• Simpler Code Reviews: Smaller, focused changes

Testability

• Unit Testing: Each component can be tested independently
• Mock Dependencies: Clear interfaces enable effective mocking
• Faster Test Cycles: Test only relevant components during development

Reusability

• Component Extraction: Services can be reused in other contexts
• Dependency Injection: Easy to substitute implementations
• Plugin Architecture: Services can be extended or replaced

Code Quality

• Reduced Complexity: Smaller classes are easier to understand
• Better Encapsulation: Clear boundaries between concerns
• Improved Documentation: Each class has focused documentation

Migration Timeline

Week 1-2: Infrastructure

• Create service class skeletons
• Set up dependency injection framework
• Write initial unit tests

Week 3-4: Low-Risk Extractions

• Extract PackageVersionResolver
• Extract PackageCacheManager
• Extract PackageSourceManager
• Extract TestModeErrorHandler

Week 5-6: Medium-Risk Extractions

• Extract PackageDependencyExtractor
• Extract PackageBuildOrchestrator
• Extract BootstrapStateTracker

Week 7-8: High-Risk Refactoring

• Refactor PackageBootstrapper orchestrator
• Update all imports and callers
• Comprehensive integration testing

Week 9: Validation & Cleanup

• Performance testing and optimization
• Documentation updates
• Remove deprecated code

Success Criteria

1. Functional Equivalence: All existing functionality preserved
2. Test Coverage: Maintain or improve test coverage
3. Performance: No significant performance degradation
4. Code Quality: Improved maintainability metrics
5. Documentation: Clear documentation for each service class

Conclusion

This refactoring will transform a monolithic 1500+ line class into 8 focused, testable, and maintainable components. The gradual migration strategy minimizes risk while delivering immediate benefits in code organization and testability.

The investment in proper decomposition will pay dividends in reduced maintenance burden, faster feature development, and improved code quality for the Fromager project.