Skip to content

Instantly share code, notes, and snippets.

@Elbehery

Elbehery/https-dep_issue.md

Created November 4, 2025 00:50
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save Elbehery/d42ca15b7f767fe3265fdfdee44459c6 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save Elbehery/d42ca15b7f767fe3265fdfdee44459c6 to your computer and use it in GitHub Desktop.
Download ZIP
httpx-dep-issue
Raw
https-dep_issue.md

RHAIENG-1790: httpx Upload Failure - Problem Summary

Problem

The CI pipeline fails when uploading httpx-0.26.0.tar.gz to GitLab PyPI registry with the following error:

ERROR    InvalidDistribution: Invalid distribution metadata: license-expression
         introduced in metadata version 2.4, not 2.1

Root Cause

  1. httpx 0.26.0 uses Metadata-Version 2.4 - The upstream PyPI package for httpx 0.26.0 includes the license-expression field, which was introduced in PEP 639 and requires Metadata-Version 2.4
  2. GitLab PyPI registry doesn't support Metadata-Version 2.4 - GitLab's package registry currently only supports up to Metadata-Version 2.1
  3. Newer httpx versions (0.27.x+) also use 2.4 - So we cannot upgrade httpx without hitting the same issue

Why We Need httpx 0.26.0

  • litellm<1.60.0 requires httpx<0.28.0
  • httpx 0.26.0 is compatible with h11>=0.16.0 (required by llama-stack)
  • httpx 0.27.x+ requires metadata 2.4, which GitLab doesn't support
  • We've constrained litellm>=1.53.0,<1.60.0 in constraints.txt

Implemented Fix

1. Added httpx to skip-packages.txt (commit a1b333a - RHAIENG-1791)

File: collections/llama-stack/cpu-ubi9/skip-packages.txt

# httpx 0.26.0 from PyPI uses Metadata-Version 2.4 (license-expression field)
# GitLab PyPI registry doesn't support metadata 2.4
# Fixes: https://issues.redhat.com/browse/RHAIENG-1790
# See also: constraints.txt for httpx version pinning
httpx

Intent: Packages listed in skip-packages.txt should NOT be uploaded to GitLab PyPI registry. Instead, they will be fetched directly from PyPI during installation.

2. Pinned httpx version in constraints.txt

File: collections/llama-stack/cpu-ubi9/constraints.txt

# httpx conflict resolution
# Fixes: https://issues.redhat.com/browse/RHAIENG-1762, RHAIENG-1775, RHAIENG-1787, RHAIENG-1788, RHAIENG-1790
# httpx 0.26.0 is compatible with h11>=0.16.0 (required by llama-stack)
# httpx 0.26.0 uses httpcore==1.* which supports h11>=0.16
# httpx 0.27.x uses Metadata-Version 2.4 which GitLab PyPI registry doesn't support
# httpx 0.26.0 from PyPI also uses Metadata-Version 2.4 (license-expression field)
# httpx is listed in skip-packages.txt to prevent upload to GitLab registry
# It will be fetched directly from PyPI during installation instead
httpx==0.26.0
httpcore==1.*

3. Constrained setuptools in build.yaml

File: distribution/build.yaml

build_system:
  requires:
    - wheel>=0.45.1
    # setuptools < 70.1.0 to avoid Metadata-Version 2.4 generation
    # setuptools 70.1.0+ generates metadata 2.4 for packages with license-expression
    # GitLab PyPI registry doesn't support metadata 2.4, causing upload failures
    # See: https://issues.redhat.com/browse/RHAIENG-1762, RHAIENG-1775
    - setuptools>=69.0.0,<70.1.0

Note: This prevents our packages from generating metadata 2.4, but doesn't affect upstream packages like httpx.

Current Status

⚠️ The fix is not working yet - The CI pipeline is still attempting to upload httpx-0.26.0.tar.gz

Possible Reasons

  1. Builder version may not support skip-packages.txt

    • Current builder version: v24.0.2 (from builder-image-version.yml)
    • The skip-packages.txt feature may not be implemented or has a bug in this version

  2. CI cache issue

    • Pipeline might be using cached source distributions built before skip-packages.txt was added
    • Cache invalidation may be needed

  3. File format issue

    • skip-packages.txt might need a specific format (e.g., version specifiers, glob patterns)
    • Need to verify the expected format from builder documentation

Next Steps

Option 1: Verify Builder Support (Recommended)

  • Check if redhat/rhel-ai/wheels/builder:v24.0.2 supports skip-packages.txt
  • Review builder documentation/code for correct skip-packages.txt format
  • Consider upgrading to a newer builder version if available

Option 2: Alternative Workarounds

  • Modify upload step: Filter out httpx before twine upload
  • Rebuild source distribution: Patch httpx metadata during build to downgrade to 2.1
  • Use --exclude flag: Check if builder pipeline has an exclude option

Option 3: Cache Invalidation

  • Clear GitLab CI cache
  • Force rebuild of all source distributions
  • Ensure skip-packages.txt changes are picked up

Related Issues

  • RHAIENG-1762: Initial httpx metadata issues
  • RHAIENG-1775: mcp package removal due to httpx>=0.27.1 requirement
  • RHAIENG-1787: Related httpx conflicts
  • RHAIENG-1788: Related httpx conflicts
  • RHAIENG-1790: This issue - httpx upload failure
  • RHAIENG-1791: Added skip-packages.txt (fix attempt)

Files Modified

  • collections/llama-stack/cpu-ubi9/skip-packages.txt - Added httpx to skip list
  • collections/llama-stack/cpu-ubi9/constraints.txt - Extensive httpx pinning and documentation
  • distribution/build.yaml - Setuptools constraint, removed mcp provider
  • distribution/build.py - Filters to exclude google-cloud-aiplatform, google-genai, and mcp

Impact

  • Runtime: httpx will be fetched from PyPI during installation (not from GitLab registry)
  • Dependencies: All dependent packages (litellm, etc.) will continue to work
  • Build time: No impact once the fix is working correctly
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment