Skip to content

Instantly share code, notes, and snippets.

@dhellmann

dhellmann/cursor_prepare_plan_for_jira_ticket_aip.md

Created October 14, 2025 12:21
Show Gist options
  • Save dhellmann/06aad08bb24962d2f2d51b059caae442 to your computer and use it in GitHub Desktop.
Save dhellmann/06aad08bb24962d2f2d51b059caae442 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
cursor_prepare_plan_for_jira_ticket_aip.md

Prepare plan for jira ticket AIPCC-5917

Exported on 10/14/2025 at 08:21:18 EDT from Cursor (1.7.40)

User

I want to work on jira ticket AIPCC-5917. Fetch the details for the ticket and prepare a plan to do the work.

Cursor

I'll help you work on JIRA ticket AIPCC-5917. Let me fetch the ticket details and then prepare a plan for the work.

Since I can't directly access the JIRA ticket details, I need you to provide the ticket information. Could you please share the following details from AIPCC-5917:

  1. Ticket Summary/Title: What is the main objective described in the ticket?

  2. Description: What are the detailed requirements, acceptance criteria, or problem statement?

  3. Ticket Type: Is this a bug fix, feature request, improvement, or other type of work?

  4. Priority/Severity: What's the urgency level of this ticket?

  5. Any specific technical details: Are there particular files, components, or systems mentioned that need to be modified?

Once you provide these details, I can create a comprehensive plan for implementing the work described in AIPCC-5917.

User

The jira ticket is @https://issues.redhat.com/browse/AIPCC-5917

Cursor

Based on the JIRA ticket AIPCC-5917, I can see the issue clearly. Let me examine the current codebase to understand the implementation and then create a plan.

Now let me search for more specific information about how the publish job is currently controlled and where we might need to add the new flag.

Now let me look at the make_release_tarball.sh to understand how it currently uses the constraint flag:

Perfect! Now I have a complete understanding of the issue. Based on the JIRA ticket and my analysis of the codebase, here's what I found:

Current Problem

The publish job (lines 318-319 in pipeline-api/ci-wheelhouse.yml) is currently disabled when FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS == "true". This was done as a shortcut to avoid publishing lots of releases in the builder repo during development, but it also prevents tracking builds in the RHAI pipeline repo where constraints are also skipped.

Solution Required

Create a separate flag to control whether the publish job runs, independent of the constraint computation flag. The publish job should run by default and only be disabled in the builder repo.

User

The current flag is "hidden" and is not part of the inputs. Let's make the new flag work the same way.

Cursor

You're absolutely right! Looking at the current implementation, FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS is defined in the variables section (line 62) rather than as an API input parameter. Let me update the plan to follow the same pattern.

User

Call the new variable PUBLISH_WHEEL_RELEASES

The plan is still talking about ENABLE_PUBLISH_JOB but that variable should be called PUBLISH_WHEEL_RELEASES

Implement the plan as specified. To-do's from the plan have already been created, do not create them again.

Cursor

I'll implement the plan to separate the publish control flag from constraint computation. Let me start by updating the pipeline API configuration and then the documentation.

# This yaml file contains the CI jobs as an API that can be used by other repositories outside builder

spec:
  inputs:
    JOB_STAGE:
      default: build
      description: "Define the job stage that will run the bootstrap and onboard steps of wheel pipeline"
    JOB_PREFIX:
      default: ""
      description: "Define a prefix for the job name"
      type: string
    COLLECTION:
      description: "Define the collection name for which you want to build wheels"
      type: string
    VARIANT:
      description: "Define the variant for which you want to build wheels"
      options: ["cpu-ubi9", "cuda-ubi9", "rocm-ubi9", "spyre-ubi9", "tpu-ubi9"] # , "gaudi-ubi9"]
    ENABLE_REPEATABLE_BUILD_MODE:
      default: false
      description: "Setting which enables repeatable build mode"
      type: boolean
    BOOTSTRAP_MODE:
      description: "Setting for bootstrap mode: 'bootstrap --sdist-only', 'bootstrap --full-build', 'bootstrap-parallel'"
      options: ["sdist-only", "full", "full-parallel"]
      default: "sdist-only"
    BUILD_MODE:
      description: "Setting for build mode: 'parallel' uses build-parallel with graph file, 'serial' uses build-sequence with build order file"
      options: ["parallel", "serial"]
      default: "serial"
    ARCH:
      default: "x86_64"
      description: "Define the CPU architecture to build wheels for"
      options: ["aarch64", "ppc64le", "s390x", "x86_64"]
    BUILD_ON_ALL_PUSHES:
      default: false
      description: "Build for every merge, even if the collection has not changed."
      type: boolean
    ENABLE_NIGHTLY_BUILDS:
      default: false
      description: "Setting that enables nightly builds for a given collection"
      type: boolean
    ENABLE_TEST_JOBS:
      default: false
      description: "Setting that enables test build jobs for testing resolution and build of packages"
      type: boolean
    ENABLE_JOB:
      default: true
      description: "Setting to disable all jobs"
      type: boolean
---
# ======================================================================== VARIABLES ===============================================================
variables:
  # Set BUILDER_IMAGE_VERSION to some fake value which will be overridden by users when they invoke the API
  BUILDER_IMAGE_VERSION: v0.0
  WHEEL_SERVER_PROJECT_PREFIX: "${CI_PROJECT_NAMESPACE}/indexes"
  # Set PRODUCT_VERSION as a global variable. The default should look like a real value
  COLLECTION_VERSION: $PRODUCT_VERSION
  PRODUCT_VERSION: "0.0"
  # Used in the builder repo to let collections build multiple
  # versions of a package. Disabling constraints output also disables
  # release jobs.
  FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS: false
  # Controls whether publish jobs run. Set to false to disable publishing
  # releases (e.g., in the builder repo to avoid noise during development).
  PUBLISH_WHEEL_RELEASES: true
  # Set PRODUCT_NAME to rhoai. We need PRODUCT_NAME for customer facing package index hosted in PULP
  PRODUCT_NAME: "rhoai"

# =========================================================================== API BASE JOBS ===================================================================

# Base job for release variables
.$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-release-variables-base-job:
  variables:
    PUBLISH_AUTH_HEADER: "PRIVATE-TOKEN: ${BOT_PAT}"
    RELEASE_TARBALL: "wheels-${RELEASE_VERSION}.tar.gz"
    RELEASE_VERSION_SUFFIX: "+$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]"
    RELEASE_VERSION: "${PRODUCT_VERSION}.${CI_PIPELINE_IID}${RELEASE_VERSION_SUFFIX}"
    GITLAB_PACKAGE_URL: "https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/packages/generic/$[[ inputs.COLLECTION ]]-wheels"
    PUBLISH_URL: "${GITLAB_PACKAGE_URL}/${RELEASE_VERSION}"

# Base rules for jobs to handle project namespace and ENABLE_JOB flag
.$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-rules:
  rules:
    - if: '$CI_PROJECT_ROOT_NAMESPACE != "redhat"'
      when: never
    - if: '"$[[ inputs.ENABLE_JOB ]]" == "false"'
      when: never

# Base job for the api
.$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job:
  extends:
    - .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-release-variables-base-job
  image: registry.gitlab.com/redhat/rhel-ai/wheels/builder/podman-stable:v5.5.0
  timeout: 12h
  services:
  # No services section needed for pinp
  interruptible: true
  rules:
    - if: '$CI_PROJECT_ROOT_NAMESPACE != "redhat"'
      when: never
    - if: '"$[[ inputs.ENABLE_JOB ]]" == "false"'
      when: never
    - if: $CI_COMMIT_TAG
      when: never
    - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_REF_PROTECTED == "true"
      changes:
        - collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/requirements.txt
        - collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/constraints.txt
        - builder-image-version.yml
    - if: $CI_PIPELINE_SOURCE == "push" && "$[[ inputs.BUILD_ON_ALL_PUSHES ]]" == "true" && $CI_COMMIT_REF_PROTECTED == "true"
    - if: '$CI_PIPELINE_SOURCE == "schedule" && "$[[ inputs.ENABLE_NIGHTLY_BUILDS ]]" == "true"'
  variables:
    FF_SCRIPT_SECTIONS: "true"
    COLLECTION: "$[[ inputs.COLLECTION ]]"
    VARIANT: "$[[ inputs.VARIANT ]]"
    ARCH: "$[[ inputs.ARCH ]]"
    IMAGE_BASE: ${CI_REGISTRY}/redhat/rhel-ai/wheels/builder/builder-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]
    IMAGE: ${IMAGE_BASE}:${BUILDER_IMAGE_VERSION}
    CONTAINERFILE: Containerfile.$[[ inputs.VARIANT ]]
    TOOL_SERVER_URL: "https://gitlab.com/api/v4/projects/61507476/packages/pypi/simple/"
    OUTDIR: /mnt
    JOB_OUTPUT_DIR: /builds/$CI_PROJECT_PATH/mnt
    WHEEL_SERVER_PROJECT_PATH: "${WHEEL_SERVER_PROJECT_PREFIX}/$[[ inputs.COLLECTION ]]-${PRODUCT_VERSION}/$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]"
    SDIST_SERVER_PROJECT_PATH: "redhat/rhel-ai/wheels/prefetch"
    RUN_CLEAN_AND_UPLOAD: "false"
    # These flags are not part of the formal API, but is used to
    # control the jobs in the builder repository.
    BOOTSTRAP_RUN_ONBOARD: "true"
    BUILD_ON_ALL_PUSHES: "$[[ inputs.BUILD_ON_ALL_PUSHES ]]"
    # We need to use non-standard BOOTSTRAP_MODE for some jobs in builder which are not triggered by API
    BOOTSTRAP_MODE: $[[ inputs.BOOTSTRAP_MODE ]]
    BUILD_MODE: $[[ inputs.BUILD_MODE ]]
  before_script:
    # Dump our settings to the log
    - env
    - dnf install -y which ncurses
    # https://docs.gitlab.com/ee/ci/docker/authenticate_registry.html
    - podman login $CI_REGISTRY --username $CI_REGISTRY_USER --password "$CI_REGISTRY_PASSWORD"
    # Install dependencies for downloading secure files
    - dnf install -y which curl bash
    # Share the test script with the container using a directory in /builds, following:
    # https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#known-issues-with-docker-in-docker
    - mkdir -p "${JOB_OUTPUT_DIR}"
    - cp -rv collections "${JOB_OUTPUT_DIR}/"
    # Generate .netrc dynamically and put in inside the container where we want it
    - echo -e "machine gitlab.com\nlogin platform-engineering-bot\npassword ${BOT_PAT}" > ${JOB_OUTPUT_DIR}/.netrc
    # Configure Podman storage
    - mkdir -p /etc/containers
    # create default overlay configuration. Embed storage.conf as HEREDOC.
    # pipelines in other projects cannot access files in builder repo.
    - |
      cat > /etc/containers/storage.conf << EOF
      [storage]
      driver = "overlay"
      runroot = "/var/run/containers/storage"
      graphroot = "/var/lib/containers/storage"
      [storage.options]
      mount_program = ""
      EOF
    - mkdir -p /var/run/containers/storage
    - mkdir -p /var/lib/containers/storage
    # AWS (may) have an existing storage configuration that works with dind but not
    # pinp.  This causes pinp to fail with
    #      "OSError: [Errno 107] Transport endpoint is not connected:
    # errors.  Remove the contents of these directories to re-initialize the overlay
    # storage.
    - rm -rf /var/lib/containers/storage/*
    - rm -rf /var/run/containers/storage/*
  script:
    - >
      podman run \
        -v "$JOB_OUTPUT_DIR:/mnt" \
        -v ".:/collection-repository" \
        -e "NETRC=/mnt/.netrc" \
        -e "OUTDIR=$OUTDIR" \
        -e "FROMAGER_VARIANT=$[[ inputs.VARIANT ]]" \
        -e "COLLECTION=$[[ inputs.COLLECTION ]]" \
        -e "PRODUCT_VERSION=$PRODUCT_VERSION" \
        -e "ENABLE_REPEATABLE_BUILD_MODE=$[[ inputs.ENABLE_REPEATABLE_BUILD_MODE ]]" \
        -e "BOOTSTRAP_MODE=$BOOTSTRAP_MODE" \
        -e "BUILD_MODE=$BUILD_MODE" \
        -e "WHEEL_SERVER_PROJECT_PATH=${WHEEL_SERVER_PROJECT_PATH}" \
        -e "SDIST_SERVER_PROJECT_PATH=${SDIST_SERVER_PROJECT_PATH}" \
        -e "PUBLISH_AUTH_HEADER=${PUBLISH_AUTH_HEADER}" \
        -e "RELEASE_VERSION=${RELEASE_VERSION}" \
        -e "RELEASE_VERSION_SUFFIX=${RELEASE_VERSION_SUFFIX}" \
        -e "RELEASE_TARBALL=${RELEASE_TARBALL}" \
        -e "PUBLISH_URL=${PUBLISH_URL}" \
        -e "GITLAB_PACKAGE_URL=${GITLAB_PACKAGE_URL}" \
        -e "RUN_CLEAN_AND_UPLOAD=${RUN_CLEAN_AND_UPLOAD}" \
        -e "BOT_PAT=${BOT_PAT}" \
        -e "BOT_USER=${BOT_USER}" \
        -e "BOOTSTRAP_RUN_ONBOARD=${BOOTSTRAP_RUN_ONBOARD}" \
        -e "REPOSITORY_URL=${CI_REPOSITORY_URL}" \
        -e "FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS=${FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS}" \
        -e "PRODUCT_NAME=$PRODUCT_NAME" \
        ${IMAGE} \
        ${API_SCRIPT}
    # Development command - will be removed eventually when we will have pulp upload script running in builder
    - >
      if [ -f ./bin/upload_to_pulp.sh ] && [[ "$CI_JOB_NAME" == *"build-wheels" ]]; then
        podman run \
          -v "$JOB_OUTPUT_DIR:/mnt" \
          -v ".:/collection-repository" \
          -e "NETRC=/mnt/.netrc" \
          -e "OUTDIR=$OUTDIR" \
          -e "FROMAGER_VARIANT=$[[ inputs.VARIANT ]]" \
          -e "COLLECTION=$[[ inputs.COLLECTION ]]" \
          -e "PRODUCT_VERSION=$PRODUCT_VERSION" \
          -e "ENABLE_REPEATABLE_BUILD_MODE=$[[ inputs.ENABLE_REPEATABLE_BUILD_MODE ]]" \
          -e "BOOTSTRAP_MODE=$BOOTSTRAP_MODE" \
          -e "BUILD_MODE=$BUILD_MODE" \
          -e "WHEEL_SERVER_PROJECT_PATH=${WHEEL_SERVER_PROJECT_PATH}" \
          -e "SDIST_SERVER_PROJECT_PATH=${SDIST_SERVER_PROJECT_PATH}" \
          -e "PUBLISH_AUTH_HEADER=${PUBLISH_AUTH_HEADER}" \
          -e "RELEASE_VERSION=${RELEASE_VERSION}" \
          -e "RELEASE_VERSION_SUFFIX=${RELEASE_VERSION_SUFFIX}" \
          -e "RELEASE_TARBALL=${RELEASE_TARBALL}" \
          -e "PUBLISH_URL=${PUBLISH_URL}" \
          -e "GITLAB_PACKAGE_URL=${GITLAB_PACKAGE_URL}" \
          -e "RUN_CLEAN_AND_UPLOAD=${RUN_CLEAN_AND_UPLOAD}" \
          -e "BOT_PAT=${BOT_PAT}" \
          -e "BOT_USER=${BOT_USER}" \
          -e "BOOTSTRAP_RUN_ONBOARD=${BOOTSTRAP_RUN_ONBOARD}" \
          -e "REPOSITORY_URL=${CI_REPOSITORY_URL}" \
          -e "FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS=${FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS}" \
          -e "PULP_CERT_BASE64=${PULP_CERT_BASE64}" \
          -e "PULP_KEY_BASE64=${PULP_KEY_BASE64}" \
          -e "PRODUCT_NAME=$PRODUCT_NAME" \
          ${IMAGE} \
          /collection-repository/bin/upload_to_pulp.sh
      fi
  artifacts:
    paths:
      - $JOB_OUTPUT_DIR/*.log
      - $JOB_OUTPUT_DIR/computed-constraints.txt
      - $JOB_OUTPUT_DIR/computed-requirements.txt
      - $JOB_OUTPUT_DIR/work-dir/*/*.log
      - $JOB_OUTPUT_DIR/work-dir/logs/
      - $JOB_OUTPUT_DIR/work-dir/*.json
      - $JOB_OUTPUT_DIR/work-dir/constraints.txt
      - $JOB_OUTPUT_DIR/work-dir/build-sequence-summary.md
    when: always
  tags:
    - aipcc-large-$[[ inputs.ARCH ]]

# =========================================================== BOOTSTRAP AND ONBOARD API JOB ================================================================================

$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-bootstrap-and-onboard:
  extends:
    - .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
  stage: $[[ inputs.JOB_STAGE ]]
  variables:
    API_SCRIPT: /mirror-builder/pipeline-api/bootstrap_and_onboard.sh

# =========================================================== TEST BOOTSTRAP AND ONBOARD API JOB ================================================================================

test-$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-bootstrap-and-onboard:
  extends:
    - .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
  stage: $[[ inputs.JOB_STAGE ]]
  rules:
    - if: '$CI_PROJECT_ROOT_NAMESPACE != "redhat"'
      when: never
    - if: '"$[[ inputs.ENABLE_JOB ]]" == "false"'
      when: never
    - if: $CI_COMMIT_TAG
      when: never
    - if: $ENABLE_TEST_JOBS == "false"
      # Do not run the job if test jobs are turned off.
      when: never
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      changes:
        - collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/requirements.txt
        - collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/constraints.txt
        - builder-image-version.yml
  variables:
    API_SCRIPT: /mirror-builder/pipeline-api/bootstrap_and_onboard.sh
    BOOTSTRAP_RUN_ONBOARD: "false"
    BOOTSTRAP_MODE: "full-parallel"
    ENABLE_TEST_JOBS: "$[[ inputs.ENABLE_TEST_JOBS ]]"

# ==================================== BUILD WHEEL SEQUENCE API JOB ==============================================

$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-build-wheels:
  extends:
    - .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
  stage: $[[ inputs.JOB_STAGE ]]
  variables:
    API_SCRIPT: /mirror-builder/pipeline-api/build_wheels.sh
    RUN_CLEAN_AND_UPLOAD: "true"
  needs:
    - job: $[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-bootstrap-and-onboard
      artifacts: true

# ==================================================== RELEASE API JOB ============================================

$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-release-tarball:
  extends:
    - .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
  stage: $[[ inputs.JOB_STAGE ]]
  variables:
    API_SCRIPT: /mirror-builder/pipeline-api/make_release_tarball.sh
  needs:
    - job: $[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-build-wheels
      artifacts: true

# ============================================ PUBLISH API JOB ====================================================

$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-publish-wheels:
  image: "registry.gitlab.com/gitlab-org/release-cli:latest"
  extends:
    - .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-release-variables-base-job
  stage: $[[ inputs.JOB_STAGE ]]
  interruptible: true
  rules:
    - if: '$CI_PROJECT_ROOT_NAMESPACE != "redhat"'
      when: never
    - if: '"$[[ inputs.ENABLE_JOB ]]" == "false"'
      when: never
    - if: '$FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS == "true"'
      when: never
    - if: $CI_COMMIT_TAG
      when: never
    - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_REF_PROTECTED == "true"
      changes:
        - collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/requirements.txt
        - collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/constraints.txt
        - builder-image-version.yml
    - if: '$CI_PIPELINE_SOURCE == "schedule" && "$[[ inputs.ENABLE_NIGHTLY_BUILDS ]]" == "true"'
  needs:
    - job: $[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-release-tarball
      artifacts: true
  script:
    - echo "running publish job $CI_JOB_NAME for $RELEASE_VERSION and $[[ inputs.ARCH ]] arch"
  release:
    # See https://docs.gitlab.com/ee/ci/yaml/#release for available properties
    tag_name: "$RELEASE_VERSION"
    description: /mnt/work-dir/build-sequence-summary.md
    ref: "$CI_COMMIT_SHA"
    assets:
      links:
        - name: "$RELEASE_TARBALL"
          url: "$PUBLISH_URL/$RELEASE_TARBALL"
  tags:
    - aipcc

Now let me update the publish job rules to use the new flag instead of the constraint flag:

# This yaml file contains the CI jobs as an API that can be used by other repositories outside builder

spec:
  inputs:
    JOB_STAGE:
      default: build
      description: "Define the job stage that will run the bootstrap and onboard steps of wheel pipeline"
    JOB_PREFIX:
      default: ""
      description: "Define a prefix for the job name"
      type: string
    COLLECTION:
      description: "Define the collection name for which you want to build wheels"
      type: string
    VARIANT:
      description: "Define the variant for which you want to build wheels"
      options: ["cpu-ubi9", "cuda-ubi9", "rocm-ubi9", "spyre-ubi9", "tpu-ubi9"] # , "gaudi-ubi9"]
    ENABLE_REPEATABLE_BUILD_MODE:
      default: false
      description: "Setting which enables repeatable build mode"
      type: boolean
    BOOTSTRAP_MODE:
      description: "Setting for bootstrap mode: 'bootstrap --sdist-only', 'bootstrap --full-build', 'bootstrap-parallel'"
      options: ["sdist-only", "full", "full-parallel"]
      default: "sdist-only"
    BUILD_MODE:
      description: "Setting for build mode: 'parallel' uses build-parallel with graph file, 'serial' uses build-sequence with build order file"
      options: ["parallel", "serial"]
      default: "serial"
    ARCH:
      default: "x86_64"
      description: "Define the CPU architecture to build wheels for"
      options: ["aarch64", "ppc64le", "s390x", "x86_64"]
    BUILD_ON_ALL_PUSHES:
      default: false
      description: "Build for every merge, even if the collection has not changed."
      type: boolean
    ENABLE_NIGHTLY_BUILDS:
      default: false
      description: "Setting that enables nightly builds for a given collection"
      type: boolean
    ENABLE_TEST_JOBS:
      default: false
      description: "Setting that enables test build jobs for testing resolution and build of packages"
      type: boolean
    ENABLE_JOB:
      default: true
      description: "Setting to disable all jobs"
      type: boolean
---
# ======================================================================== VARIABLES ===============================================================
variables:
  # Set BUILDER_IMAGE_VERSION to some fake value which will be overridden by users when they invoke the API
  BUILDER_IMAGE_VERSION: v0.0
  WHEEL_SERVER_PROJECT_PREFIX: "${CI_PROJECT_NAMESPACE}/indexes"
  # Set PRODUCT_VERSION as a global variable. The default should look like a real value
  COLLECTION_VERSION: $PRODUCT_VERSION
  PRODUCT_VERSION: "0.0"
  # Used in the builder repo to let collections build multiple
  # versions of a package. Disabling constraints output also disables
  # release jobs.
  FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS: false
  # Controls whether publish jobs run. Set to false to disable publishing
  # releases (e.g., in the builder repo to avoid noise during development).
  PUBLISH_WHEEL_RELEASES: true
  # Set PRODUCT_NAME to rhoai. We need PRODUCT_NAME for customer facing package index hosted in PULP
  PRODUCT_NAME: "rhoai"

# =========================================================================== API BASE JOBS ===================================================================

# Base job for release variables
.$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-release-variables-base-job:
  variables:
    PUBLISH_AUTH_HEADER: "PRIVATE-TOKEN: ${BOT_PAT}"
    RELEASE_TARBALL: "wheels-${RELEASE_VERSION}.tar.gz"
    RELEASE_VERSION_SUFFIX: "+$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]"
    RELEASE_VERSION: "${PRODUCT_VERSION}.${CI_PIPELINE_IID}${RELEASE_VERSION_SUFFIX}"
    GITLAB_PACKAGE_URL: "https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/packages/generic/$[[ inputs.COLLECTION ]]-wheels"
    PUBLISH_URL: "${GITLAB_PACKAGE_URL}/${RELEASE_VERSION}"

# Base rules for jobs to handle project namespace and ENABLE_JOB flag
.$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-rules:
  rules:
    - if: '$CI_PROJECT_ROOT_NAMESPACE != "redhat"'
      when: never
    - if: '"$[[ inputs.ENABLE_JOB ]]" == "false"'
      when: never

# Base job for the api
.$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job:
  extends:
    - .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-release-variables-base-job
  image: registry.gitlab.com/redhat/rhel-ai/wheels/builder/podman-stable:v5.5.0
  timeout: 12h
  services:
  # No services section needed for pinp
  interruptible: true
  rules:
    - if: '$CI_PROJECT_ROOT_NAMESPACE != "redhat"'
      when: never
    - if: '"$[[ inputs.ENABLE_JOB ]]" == "false"'
      when: never
    - if: $CI_COMMIT_TAG
      when: never
    - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_REF_PROTECTED == "true"
      changes:
        - collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/requirements.txt
        - collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/constraints.txt
        - builder-image-version.yml
    - if: $CI_PIPELINE_SOURCE == "push" && "$[[ inputs.BUILD_ON_ALL_PUSHES ]]" == "true" && $CI_COMMIT_REF_PROTECTED == "true"
    - if: '$CI_PIPELINE_SOURCE == "schedule" && "$[[ inputs.ENABLE_NIGHTLY_BUILDS ]]" == "true"'
  variables:
    FF_SCRIPT_SECTIONS: "true"
    COLLECTION: "$[[ inputs.COLLECTION ]]"
    VARIANT: "$[[ inputs.VARIANT ]]"
    ARCH: "$[[ inputs.ARCH ]]"
    IMAGE_BASE: ${CI_REGISTRY}/redhat/rhel-ai/wheels/builder/builder-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]
    IMAGE: ${IMAGE_BASE}:${BUILDER_IMAGE_VERSION}
    CONTAINERFILE: Containerfile.$[[ inputs.VARIANT ]]
    TOOL_SERVER_URL: "https://gitlab.com/api/v4/projects/61507476/packages/pypi/simple/"
    OUTDIR: /mnt
    JOB_OUTPUT_DIR: /builds/$CI_PROJECT_PATH/mnt
    WHEEL_SERVER_PROJECT_PATH: "${WHEEL_SERVER_PROJECT_PREFIX}/$[[ inputs.COLLECTION ]]-${PRODUCT_VERSION}/$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]"
    SDIST_SERVER_PROJECT_PATH: "redhat/rhel-ai/wheels/prefetch"
    RUN_CLEAN_AND_UPLOAD: "false"
    # These flags are not part of the formal API, but is used to
    # control the jobs in the builder repository.
    BOOTSTRAP_RUN_ONBOARD: "true"
    BUILD_ON_ALL_PUSHES: "$[[ inputs.BUILD_ON_ALL_PUSHES ]]"
    # We need to use non-standard BOOTSTRAP_MODE for some jobs in builder which are not triggered by API
    BOOTSTRAP_MODE: $[[ inputs.BOOTSTRAP_MODE ]]
    BUILD_MODE: $[[ inputs.BUILD_MODE ]]
  before_script:
    # Dump our settings to the log
    - env
    - dnf install -y which ncurses
    # https://docs.gitlab.com/ee/ci/docker/authenticate_registry.html
    - podman login $CI_REGISTRY --username $CI_REGISTRY_USER --password "$CI_REGISTRY_PASSWORD"
    # Install dependencies for downloading secure files
    - dnf install -y which curl bash
    # Share the test script with the container using a directory in /builds, following:
    # https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#known-issues-with-docker-in-docker
    - mkdir -p "${JOB_OUTPUT_DIR}"
    - cp -rv collections "${JOB_OUTPUT_DIR}/"
    # Generate .netrc dynamically and put in inside the container where we want it
    - echo -e "machine gitlab.com\nlogin platform-engineering-bot\npassword ${BOT_PAT}" > ${JOB_OUTPUT_DIR}/.netrc
    # Configure Podman storage
    - mkdir -p /etc/containers
    # create default overlay configuration. Embed storage.conf as HEREDOC.
    # pipelines in other projects cannot access files in builder repo.
    - |
      cat > /etc/containers/storage.conf << EOF
      [storage]
      driver = "overlay"
      runroot = "/var/run/containers/storage"
      graphroot = "/var/lib/containers/storage"
      [storage.options]
      mount_program = ""
      EOF
    - mkdir -p /var/run/containers/storage
    - mkdir -p /var/lib/containers/storage
    # AWS (may) have an existing storage configuration that works with dind but not
    # pinp.  This causes pinp to fail with
    #      "OSError: [Errno 107] Transport endpoint is not connected:
    # errors.  Remove the contents of these directories to re-initialize the overlay
    # storage.
    - rm -rf /var/lib/containers/storage/*
    - rm -rf /var/run/containers/storage/*
  script:
    - >
      podman run \
        -v "$JOB_OUTPUT_DIR:/mnt" \
        -v ".:/collection-repository" \
        -e "NETRC=/mnt/.netrc" \
        -e "OUTDIR=$OUTDIR" \
        -e "FROMAGER_VARIANT=$[[ inputs.VARIANT ]]" \
        -e "COLLECTION=$[[ inputs.COLLECTION ]]" \
        -e "PRODUCT_VERSION=$PRODUCT_VERSION" \
        -e "ENABLE_REPEATABLE_BUILD_MODE=$[[ inputs.ENABLE_REPEATABLE_BUILD_MODE ]]" \
        -e "BOOTSTRAP_MODE=$BOOTSTRAP_MODE" \
        -e "BUILD_MODE=$BUILD_MODE" \
        -e "WHEEL_SERVER_PROJECT_PATH=${WHEEL_SERVER_PROJECT_PATH}" \
        -e "SDIST_SERVER_PROJECT_PATH=${SDIST_SERVER_PROJECT_PATH}" \
        -e "PUBLISH_AUTH_HEADER=${PUBLISH_AUTH_HEADER}" \
        -e "RELEASE_VERSION=${RELEASE_VERSION}" \
        -e "RELEASE_VERSION_SUFFIX=${RELEASE_VERSION_SUFFIX}" \
        -e "RELEASE_TARBALL=${RELEASE_TARBALL}" \
        -e "PUBLISH_URL=${PUBLISH_URL}" \
        -e "GITLAB_PACKAGE_URL=${GITLAB_PACKAGE_URL}" \
        -e "RUN_CLEAN_AND_UPLOAD=${RUN_CLEAN_AND_UPLOAD}" \
        -e "BOT_PAT=${BOT_PAT}" \
        -e "BOT_USER=${BOT_USER}" \
        -e "BOOTSTRAP_RUN_ONBOARD=${BOOTSTRAP_RUN_ONBOARD}" \
        -e "REPOSITORY_URL=${CI_REPOSITORY_URL}" \
        -e "FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS=${FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS}" \
        -e "PRODUCT_NAME=$PRODUCT_NAME" \
        ${IMAGE} \
        ${API_SCRIPT}
    # Development command - will be removed eventually when we will have pulp upload script running in builder
    - >
      if [ -f ./bin/upload_to_pulp.sh ] && [[ "$CI_JOB_NAME" == *"build-wheels" ]]; then
        podman run \
          -v "$JOB_OUTPUT_DIR:/mnt" \
          -v ".:/collection-repository" \
          -e "NETRC=/mnt/.netrc" \
          -e "OUTDIR=$OUTDIR" \
          -e "FROMAGER_VARIANT=$[[ inputs.VARIANT ]]" \
          -e "COLLECTION=$[[ inputs.COLLECTION ]]" \
          -e "PRODUCT_VERSION=$PRODUCT_VERSION" \
          -e "ENABLE_REPEATABLE_BUILD_MODE=$[[ inputs.ENABLE_REPEATABLE_BUILD_MODE ]]" \
          -e "BOOTSTRAP_MODE=$BOOTSTRAP_MODE" \
          -e "BUILD_MODE=$BUILD_MODE" \
          -e "WHEEL_SERVER_PROJECT_PATH=${WHEEL_SERVER_PROJECT_PATH}" \
          -e "SDIST_SERVER_PROJECT_PATH=${SDIST_SERVER_PROJECT_PATH}" \
          -e "PUBLISH_AUTH_HEADER=${PUBLISH_AUTH_HEADER}" \
          -e "RELEASE_VERSION=${RELEASE_VERSION}" \
          -e "RELEASE_VERSION_SUFFIX=${RELEASE_VERSION_SUFFIX}" \
          -e "RELEASE_TARBALL=${RELEASE_TARBALL}" \
          -e "PUBLISH_URL=${PUBLISH_URL}" \
          -e "GITLAB_PACKAGE_URL=${GITLAB_PACKAGE_URL}" \
          -e "RUN_CLEAN_AND_UPLOAD=${RUN_CLEAN_AND_UPLOAD}" \
          -e "BOT_PAT=${BOT_PAT}" \
          -e "BOT_USER=${BOT_USER}" \
          -e "BOOTSTRAP_RUN_ONBOARD=${BOOTSTRAP_RUN_ONBOARD}" \
          -e "REPOSITORY_URL=${CI_REPOSITORY_URL}" \
          -e "FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS=${FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS}" \
          -e "PULP_CERT_BASE64=${PULP_CERT_BASE64}" \
          -e "PULP_KEY_BASE64=${PULP_KEY_BASE64}" \
          -e "PRODUCT_NAME=$PRODUCT_NAME" \
          ${IMAGE} \
          /collection-repository/bin/upload_to_pulp.sh
      fi
  artifacts:
    paths:
      - $JOB_OUTPUT_DIR/*.log
      - $JOB_OUTPUT_DIR/computed-constraints.txt
      - $JOB_OUTPUT_DIR/computed-requirements.txt
      - $JOB_OUTPUT_DIR/work-dir/*/*.log
      - $JOB_OUTPUT_DIR/work-dir/logs/
      - $JOB_OUTPUT_DIR/work-dir/*.json
      - $JOB_OUTPUT_DIR/work-dir/constraints.txt
      - $JOB_OUTPUT_DIR/work-dir/build-sequence-summary.md
    when: always
  tags:
    - aipcc-large-$[[ inputs.ARCH ]]

# =========================================================== BOOTSTRAP AND ONBOARD API JOB ================================================================================

$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-bootstrap-and-onboard:
  extends:
    - .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
  stage: $[[ inputs.JOB_STAGE ]]
  variables:
    API_SCRIPT: /mirror-builder/pipeline-api/bootstrap_and_onboard.sh

# =========================================================== TEST BOOTSTRAP AND ONBOARD API JOB ================================================================================

test-$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-bootstrap-and-onboard:
  extends:
    - .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
  stage: $[[ inputs.JOB_STAGE ]]
  rules:
    - if: '$CI_PROJECT_ROOT_NAMESPACE != "redhat"'
      when: never
    - if: '"$[[ inputs.ENABLE_JOB ]]" == "false"'
      when: never
    - if: $CI_COMMIT_TAG
      when: never
    - if: $ENABLE_TEST_JOBS == "false"
      # Do not run the job if test jobs are turned off.
      when: never
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      changes:
        - collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/requirements.txt
        - collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/constraints.txt
        - builder-image-version.yml
  variables:
    API_SCRIPT: /mirror-builder/pipeline-api/bootstrap_and_onboard.sh
    BOOTSTRAP_RUN_ONBOARD: "false"
    BOOTSTRAP_MODE: "full-parallel"
    ENABLE_TEST_JOBS: "$[[ inputs.ENABLE_TEST_JOBS ]]"

# ==================================== BUILD WHEEL SEQUENCE API JOB ==============================================

$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-build-wheels:
  extends:
    - .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
  stage: $[[ inputs.JOB_STAGE ]]
  variables:
    API_SCRIPT: /mirror-builder/pipeline-api/build_wheels.sh
    RUN_CLEAN_AND_UPLOAD: "true"
  needs:
    - job: $[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-bootstrap-and-onboard
      artifacts: true

# ==================================================== RELEASE API JOB ============================================

$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-release-tarball:
  extends:
    - .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
  stage: $[[ inputs.JOB_STAGE ]]
  variables:
    API_SCRIPT: /mirror-builder/pipeline-api/make_release_tarball.sh
  needs:
    - job: $[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-build-wheels
      artifacts: true

# ============================================ PUBLISH API JOB ====================================================

$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-publish-wheels:
  image: "registry.gitlab.com/gitlab-org/release-cli:latest"
  extends:
    - .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-release-variables-base-job
  stage: $[[ inputs.JOB_STAGE ]]
  interruptible: true
  rules:
    - if: '$CI_PROJECT_ROOT_NAMESPACE != "redhat"'
      when: never
    - if: '"$[[ inputs.ENABLE_JOB ]]" == "false"'
      when: never
    - if: '$PUBLISH_WHEEL_RELEASES == "false"'
      when: never
    - if: $CI_COMMIT_TAG
      when: never
    - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_REF_PROTECTED == "true"
      changes:
        - collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/requirements.txt
        - collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/constraints.txt
        - builder-image-version.yml
    - if: '$CI_PIPELINE_SOURCE == "schedule" && "$[[ inputs.ENABLE_NIGHTLY_BUILDS ]]" == "true"'
  needs:
    - job: $[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-release-tarball
      artifacts: true
  script:
    - echo "running publish job $CI_JOB_NAME for $RELEASE_VERSION and $[[ inputs.ARCH ]] arch"
  release:
    # See https://docs.gitlab.com/ee/ci/yaml/#release for available properties
    tag_name: "$RELEASE_VERSION"
    description: /mnt/work-dir/build-sequence-summary.md
    ref: "$CI_COMMIT_SHA"
    assets:
      links:
        - name: "$RELEASE_TARBALL"
          url: "$PUBLISH_URL/$RELEASE_TARBALL"
  tags:
    - aipcc

Now let me update the documentation to explain the new variable. First, let me read the current README to understand the structure:

Let me look for where variables are documented in the README:

Let me search for where variables or configuration options are documented:

Let me look at the section around line 475 where variables are documented:

Let me look for where the API inputs are documented to understand the structure better:

Perfect! I can see that the API inputs are documented in the "Default or Configurable inputs for API" section. However, since PUBLISH_WHEEL_RELEASES is a variable (not an API input), I should add it to a separate section. Let me add it after the existing inputs but before the Gitlab token variables section:

# AI Wheels Collections

The AI Platform Core Components (AIPCC) team manages tools and
pipelines for building AI components from the Python ecosystem using
Python's native packaging format, called a "wheel". This document
explains why and how we manage those builds and how other teams can
consume them.

## Terminology

**Source Distribution** -- A source distribution, or sdist, is a
tarball or ZIP archive containing the source code for a wheel
following the standard defined by the Python Packaging
Authority. Release archives produced by GitHub are not standard
sdists.

**Wheel** -- A wheel is a standard binary distribution format for
libraries and applications in the Python ecosystem. The wheel format
is defined by the Python Packaging Authority.

**Build Variant** -- A build variant, or just variant, is the set of
characteristics that are used to build the same input source for
different purposes (operating system, python version, accelerator,
etc.).

**Builder Image** -- A builder image is a container image that
includes the build dependencies, fromager, and build settings for
building wheels.

**Wheel Collection** -- A wheel collection is a set of packages that
are built in a way to make them compatible with each other (typically
this means building them together in a pipeline).

**Package Index or Package Repository** -- A package index is a
service for hosting source distributions and wheels that complies with
the Simple repository API. The canonical implementation is wheelhouse,
used for the community package index at https://pypi.org.

## Overview

The AI ecosystem is largely based on Python. We take advantage of this
to apply automation to build our dependencies as *wheel* packages.  We
typically do not focus on building a single wheel. Instead, we build
sets of wheels together, in a *collection*. Managing the wheels as a
collection ensures that if one package uses another during its build,
we are producing ABI compatible versions of those packages. It also
ensures that we know how to resolve all of the versions of the
dependencies so they can be installed together.

The AI space is moving quickly, as software changes rapidly and
hardware vendors jockey to gain advantage. To support the different
hardware accelerators from our partners, we set up build environments
for the wheels that allow us to build *variants* of each package. The
majority of packages have no special settings for those builds, but
the ones that do get significant benefit from having per-variant
settings that let us link to libraries from the partner. The build
environment is defined by a *builder* container image, which contains:
the supported versions of RHEL and Python; fromager and its settings;
the various compilers and other development tools needed for compiling
wheels with code in C++, go, java, and rust; and vendor-specific tools
and libraries. We have defined variants for CPU-only, CUDA, Gaudi, and
ROCm builds, today.

The builder image is one aspect of the Gitlab Job API, which defines a
set of jobs to compile the packages in a collection and manage the
publish artifacts to a service where they will be available when
building images.

Product developers who want to deliver wheels in their product images
start by defining a collection with a list of all of the packages to
build and a list of rules to control which versions of dependencies
are included. It is not necessary to specify every package that goes
into a collection, since the build tools will compute the rest of the
dependencies, including both build-time and installation-time
dependencies. By compiling even the build dependencies from source, we
are able to maintain a secure build environment with up-to-date tools.

Although the inputs are minimal, a full release of a collection
includes all of the primary dependencies, anything needed to build or
install those packages, a file listing every package to install and
its version, and a script to manage the installation from the private
package index where the release is hosted.

A single product may have multiple wheel collections, to produce
separate images or to install into different virtualenvs within a
single image. Wheel collections should not be reused between products,
to allow products to have different release cadences and to meet
different requirements (in the versions of packages we build or in the
way we build them). Wheel collections should not be combined. There is
no guarantee that wheels from two collections built separately will be
ABI and install-time compatible. Maintaining this level separation
means trading off building the same artifact more than once, but the
benefit is that we can more easily reason about the results of that
build.

## RACI matrix

| **Task** | **Responsible** | **Accountable** | **Consulted** | **Informed** |
|----------|-----------------|-----------------|---------------|--------------|
| API implementation | AIPCC | | Application teams | |
| Deploying a pipeline | Application teams | | AIPCC | |
| Deploying package indexes | AIPCC | Application teams | | |
| Selecting packages and versions | Application teams | | AIPCC | |
| Building wheel collections | Application teams | | | |


| **Task** | **Responsible** | **Accountable** | **Consulted** | **Informed** |
|----------|-----------------|-----------------|---------------|--------------|
| Managing builder image and fromager settings | AIPCC | | |

## Defining a wheel collection

Wheel collections are defined using a list of toplevel requirements,
expressed using the standard dependency specifiers format in a
requirements.txt file. For example, this is the list of requirements
for the RHEL AI 1.4 collection for CUDA builds:

vllm[cuda]==0.6.4.post1

Use the cuda deps for instructlab.

instructlab[cuda]==0.24.0

RHELAI-2151: Building requested version for kfp

Follow https://issues.redhat.com/browse/RHELAI-2151 for more details

kfp==2.11.0


The requirements specified in your collection are combined with a set
of global requirements built into the builder image. The global
requirements list includes a few basic tools like `pip` and
`setuptools` so we can ensure the smooth installation of all
packages. It is not necessary or allowed to add requirements for
packages that are listed in the global requirements to your collection
requirements.txt file.

The rules for limiting versions of dependencies are called
*constraints*, and are listed in a `constraints.txt` file that uses a
subset of the same dependency specifier syntax. A constraints list may
include many more packages than the build requirements list. It is
typical for a constraint rule to “pin” to a specific version using the
`==` operator or to "cap" the version of a package by applying an
upper-bounds for the version range using `<` or `<=`. For example,
this is the list of constraints for the RHEL AI 1.4 collection for
CUDA builds:

torch==2.5.1

RHELAI-1554: keep torchvision in sync with torch

torchvision~=0.20.1

RHELAI-559: rpds-py>=0.19.0 needs Rust >= 1.76

rpds-py==0.18.1

Temporarily constrain anything we are patching to ensure reliable and repeatable build

deepspeed==0.15.2 flash_attn==2.7.2.post1 triton==3.1.0 xformers==0.0.28.post3

setuptools!=72.0.0 # pypa/setuptools#4519

constraint wandb until we figure out how to get latest go

wandb<0.17.0 # https://issues.redhat.com/browse/RHELAI-847

multiprocess<=0.70.16 # https://issues.redhat.com/browse/RHELAI-1672

RHELAI-2382, 2.10.0 and .1 break Docling

pydantic < 2.10.0

RHELAI-2383, our patch is for 1.11.1.1, RHEL 9 has 1.10.2

ninja==1.11.1.1

RHEL-2921, 0.5.0+ requires newer version of rust

safetensors<0.5.0

RHELAI-2974, required by pythran

gast==0.6.0

RHELAI-2970, 3.10.14+ requires newer version of rust; RHELAI-3099

orjson==3.10.13

llvmlite 0.44.0 requires LLVM 15 and numba 0.61.0 requires llvmlite 0.44.0 https://issues.redhat.com/browse/RHELAI-3073

llvmlite<0.44.0 numba<0.61.0

RHELAI-2999 - hf_transfer 0.1.9 uses lock file format 4, which is not supported by cargo 1.75

hf_transfer<0.1.9


Listing a package in the constraints file does **not** automatically
include it in the build. As the build progresses, fromager chooses
which *packages* to try to build based on the input requirements list
and their dependencies. It chooses which *versions* of those packages
to build based on the constraints.

Different variants of a collection are likely to include different
packages. In the example above, `vllm[cuda]` includes the optional
CUDA dependencies for vllm. Different sets of optional dependencies
are needed for building for other accelerators.

In addition to the variant constraints file, there is a global
constraints file built into the builder image. This file includes
constraints that we know to be required for all builds, to avoid
problematic versions of common packages. The global constraints take
precedence over any local constraints. Specifying a constraint for a
package listed in the global constraints list is not allowed. To view
the contents of the global constraints list, refer to the file
`collections/global-constraints.txt` in the appropriate branch of this
repository.

### Builder constraints API

The builder image provides a constraints API which is
an opt-in feature for the users of Jobs API. This feature
requires users to set rule files in their collections directory

Here is an example of rule file syntax that a user can use

  1. pull all accelerated constraints

  2. accelerated *

  4. take just pyodbc constraint from non-accelerated

  5. non-accelerated pyodbc

  7. take all habana-related constraints

  8. accelerated habana*
The above example demonstrates rules in the format
`collection_name` `package_name` on every line
The `collection_name` is a collection and has to be
present in the user's collections directory. The `package_name` can be a wildcard, actual package name or combination of both.

For the above example, the rule on line 2 indicates
that all constraints for `accelerated` collection will
be pulled in for the variant that we are building wheels
for. Similarly, rule on line 5 indicates that we only want
constraint for `pyodbc` from `non-accelerated` collection.
Lastly, rule on line 8 indicates that we want all the
`habana` related constraints for `accelerated` collection.

Builder uses `extract-constraints` command to pull in
right constraints based on the rule files provided by the user.

#### How to set rules files

Builder image computes constraints based on precedence.
The global constraints are applied first followed by
collection specific constraints and finally variant specific
constraints.

The constraint rules files are processed in the following hierarchy:

1. **Global rules**: `collections/constraints-rules.txt` - Rules that apply to **all** collections
2. **Collection-variant rules**: `collections/${collection}/${variant}/constraints-rules.txt` - Rules for a specific collection and variant
3. **Collection rules**: `collections/${collection}/constraints-rules.txt` - Rules for a specific collection (used as fallback if variant-specific rules don't exist)

##### Global rules file

Users need to create a file titled `constraints-rules.txt` at the root level
inside the `collections` directory. This file must contain
all the rules that apply to **all** the collections.

For example, if a user has three different collections,
the rules common to all these three collections
will end up in `collections/constraints-rules.txt`
file.

##### Collection-specific rules files

For specifying rules for a particular collection, users
need to create `constraints-rules.txt` under
a specific collection directory.

For example, users will create `constraints-rules.txt` under `collections/mycollection` directory where they will specify constraints rules for `mycollection`.

##### Variant-specific rules files

For fine-grained control over constraints for specific variants within a collection, users can create variant-specific rules files at `collections/${collection}/${variant}/constraints-rules.txt`.

For example, to specify CUDA-specific constraint rules for the `mycollection` collection, create `collections/mycollection/cuda-ubi9/constraints-rules.txt`.

**Important**: When a variant-specific rules file exists, it completely replaces the collection-level rules file for that variant. The collection-level rules file is only used as a fallback when no variant-specific rules file exists.

This allows you to:
- Handle conflicts between different collections' constraints for specific variants
- Apply different constraint rules for different accelerator types within the same collection
- Maintain general rules for most variants while having specialized rules for specific ones

### Nightly builds for a wheel collection

Builder jobs API support nightly builds for a given collection
with an aim to detect build issues earlier using builder image.
The ENABLE_NIGHTLY_BUILDS flag needs to be set for that
particular collection and variant in `.gitlab-ci.yaml`.
After setting the flag, users need to go and set up a
schedule in gitlab repository.

Nightly builds will always build from a head of a git repository.

#### Configuring Gitlab schedule for a repository

Gitlab provides [schedules](https://docs.gitlab.com/ci/pipelines/schedules/#add-a-pipeline-schedule)
feature which is used to set up cron job for nightly builds.
Users can set a particular time of the day to run these builds
and also select the branch for which they want to trigger these builds.

The schedule owner must have the Developer role.
For pipelines on protected branches, the schedule owner must be allowed to
merge to the branch.

#### Configuring notifications for nightly builds

Gitlab supports [Pipeline Status Email](https://gitlab.com/redhat/rhel-ai/rhelai/pipeline/-/settings/integrations/pipelines_email/edit) integration which will be used to send email
notifications stating pipeline status. This can be configured to notify
only for broken pipelines. Users will get status of nightly builds using
this integration and can add recepients as per their choice.

Users can also set up Slack notifications using the Gitlab app for slack
integration on their preferred slack channel. This requires getting access
from Red Hat slack admin and involves Red Hat IT.

These integration options can be found under `Settings-->Integrations`
for a particular repository and require Owner level permissons to
set up.

## Wheel build pipelines

The wheel build pipeline runs a sequence of jobs to determine the
dependencies, acquire the source code, and then build from
source. Each collection has a pipeline for each variant needed by the
collection.

In the first step, the requirements list is passed through fromager to
*bootstrap* the build. During bootstrapping we determine all of the
build and installation dependencies of the input
requirements. Bootstrapping creates a graph file with all of the
package details and a build-order file with a sequential list that
allows building all packages from scratch so that before a package is
built all of its dependencies are built.

The next step is to *onboard* the source packages using the
build-sequence file produced by the bootstrap command. The job copies
source distributions from the upstream pypi.org index into the private
build index. Many of the complex packages related to AI do not have
published source distributions because the authors do not want to
support users building from source. Those packages are skipped and the
source archives are downloaded during the actual build step.

The *build* job works through the build-sequence file, pulling source
from either the internal index or the alternative locations. The
original source is unpacked, then optionally patched, and then we
build a new source distribution and a wheel. Each new build artifact
is uploaded to the internal package index as the wheel is built. This
ensures that if the build fails part of the way through the sequence,
we retain any wheels that were successfully built and only have to
build anything that is missing.

After the build completes, a *release* is created and published to the
repository where the pipeline runs. The release includes a copy of the
input `requirements.txt` file, a newly generated full
`constraints.txt` for install-time dependencies that pins all versions
of all packages to be installed, and an installation script that
invokes pip with the arguments needed to install the collection from
the index where the wheels are stored. The installation script makes
it easier for consumers of the collection to get the right packages
installed from the right place, without updating image repositories
with gitlab package index API URLs.

## Package indexes

A PyPI-compatible package index can host source distributions and
built wheels and provide an API for native tools like `pip` and `uv`
to install the packages. The index API supports hosting multiple
versions of multiple packages, and supports hosting multiple builds of
the same version of a package for a subset of the criteria we include
in our variant definitions (python version and CPU
architecture). Today we use the Python package index API associated
with private GitLab repositories. These indexes are managed by the
AIPCC team as part of the gitlab repository automation.

Regardless of the service used, wheels built for different variants
cannot be hosted in the same package index today. The wheel file
format is strict about naming, and the name syntax does not support
including information that differentiates the build variants from each
other using all of the parameters we need, like accelerator, in a way
that allows the installer to select the right package. Therefore we
use a separate package index for each combination of variant,
collection, and collection version. For example, the package index for
https://gitlab.com/redhat/rhel-ai/wheels/indexes/rhelai-1.5/cuda-ubi9
holds the wheels for the `cuda-ubi9` variant build of the RHEL AI
collection for RHEL AI 1.5.

The source we use to build wheels may not match the upstream source
distribution. We patch some of the sources we build, to update
dependencies or change build settings. We publish the patched sources
to comply with Red Hat policy of delivering the source we use to build
our products (and not just the upstream source). The patched source
distributions are published to the same package index as the built
wheels so that the image build pipeline can find them and build source
images to distribute to users for compliance.

## Gitlab Job API

The builder repository provides a set of reusable Gitlab CI jobs that
can be consumed by other repositories in the form of an API. This README
contains all the detailed information about each Gitlab CI job as an
API and common set of inputs which are required. The API jobs can be found
in `pipeline-api/ci-wheelhouse.yml` file of builder repository

### API Inputs

The Gitlab Job API requires a set of inputs for all the jobs to be triggered.
The API jobs have dependency on each other and thus share common set of inputs.

#### Required or Mandatory inputs for API

1. COLLECTION: This string variable defines a collection name for which
   particular set of wheels is built. For example,
   the collection name can be `rhelai`, `api-test`, etc.
2. VARIANT: This string variable defines the variant supported by
   fromager for which the wheels are built. Currently,
   [fromager](https://fromager.readthedocs.io/en/latest/) supports
   `cpu-ubi9`, `cuda-ubi9`, `rocm-ubi9` and `gaudi-ubi9` variants. For
   example, the fromager variant can be `cpu-ubi9`, `cuda-ubi9`, etc

#### Default or Configurable inputs for API

1. JOB_STAGE: This variable lets users define the CI stage the API jobs
   needs to run in. The default value for this variable is
   `build`. It is recommended that this job should be
   run after the linter job of any CI pipeline.
2. JOB_PREFIX: This variable lets users define a job prefix for all the
   API jobs. The default value for this variable is an empty string.
   Setting a job prefix ensures each job has unique name.
3. ENABLE_REPEATABLE_BUILD_MODE:
   [Fromager](https://fromager.readthedocs.io/en/latest/) supports
   repeatable builds to ensure bootstrapping the versions of packages
   that the branch was able to build initially when it was
   tagged. This will stop the bootstrap process from pulling
   unnecessary new dependencies. The default value for this variable
   is `false` and is of type `boolean`. It is recommended to use this mode after branching at
   time of RC during a particular release
4. ARCH: This string variable defines the CPU architecture for which the wheels are built.
   The default value for architecture is set to `x86_64` and the API supports `aarch64`
   builds as well.
5. BUILD_ON_ALL_PUSHES: This boolean variable controls whether the
   pipeline runs for every merge event. Defaults to false.
6. BOOTSTRAP_MODE: This variable controls whether bootstrapping uses
   * `sdist-only`: `bootstrap --sdist-only` (fast onboarding)
   * `full`: `bootstrap --full-build` (slow full build)
   * `full-parallel`: `bootstrap-parallel` (combines `bootstrap --sdist-only` with `build-parallel`)
   The default value is `sdist-only` and is suitable for build pipelines. The
   `full` and `full-parallel` modes are for test pipelines that don't use
   `build-sequence`.
7. BUILD_MODE: This variable controls whether the build process uses
   * `parallel`: `build-parallel` command with dependency graph file (faster)
   * `serial`: `build-sequence` command with build order file (sequential, default)
   The default value is `serial`. `parallel` is unstable and sometimes fails the build.
8. ENABLE_NIGHTLY_BUILDS: builder supports
   nightly builds for a wheel collection with an aim to detect build issues
   earlier. The default value for this variable is `false` and is of type `boolean`.
   It is recommended to enable nightly builds as soon as possible on main branch
   before branching for a particular release.
9. ENABLE_TEST_JOBS: boolean controlling whether to run a bootstrap job
   to test the collection settings files as part of testing a merge request.
   The flag defaults to false because it introduces an extra delay for
   builds that work, but it can be useful to enable it when trying to
   figure out why a build is not working, for example when resolving
   constraints settings.
10. ENABLE_JOB: boolean controlling whether to run the job at all. The
    flag defaults to true. It can be set to false to disable a pipeline.

#### Pipeline Variables

The following variables control pipeline behavior and can be overridden in consuming repositories:

1. PUBLISH_WHEEL_RELEASES: boolean controlling whether to publish releases
   after building wheels. The flag defaults to true. It can be set to false
   to disable publishing releases (e.g., in the builder repository to avoid
   noise during development). This is independent of constraint computation
   controlled by FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS.

#### Gitlab token variables

The users of API should set the following variables required for Gitlab authentication and are used as tokens

1. BOT_USER: This is the username for the gitlab token which is used for authentication while uploading source
   distributions and wheels to the package index. This variable can be created and set at repository level by navigating to `Settings -> CI/CD -> Variables` for the repository.
2. BOT_PAT: This is the actual gitlab token used for authentication of Gitlab services that are part of the API.
   This variable can be created and set at repository level by navigating to `Settings -> CI/CD -> Variables` for the repository. This token needs to have read and write permissions on the wheel indexes for the pipeline repository, the common sdist repository shared by all wheel builds, and the image registry for the builder repository.

### Linter

The linter job validates the collection inputs like the
`requirements.txt` and `constraints.txt` files.  It checks whether
these input files are formatted correctly and reports errors if
encountered. The job API for linter can be found in
`pipeline-api/inputs-validator.yml` file of builder repository.

The linter job only needs two inputs from the above mentioned API inputs.
These inputs are JOB_STAGE and JOB_PREFIX. Note that default job stage for
linter is `build` which is different from the rest of API jobs. The job triggered
would be `wheel-collection-linter`.

### Bootstrap and Onboard

The bootstrap and onboard job runs the bootstrap step for a given
`requirements.txt` using constraints present in
`constraints.txt`. After the bootstrap is complete, the onboarder step
of the same job uploads the source distributions to the package index.
The job triggered would be `api-test-cpu-ubi9-x86_64-bootstrap-and-onboard`

### Build Wheel Sequence

The build wheel sequence job is used to build wheels in order as per the
`build-order.json` file which is produced by the bootstrap job.
As a result, this job is dependent on the success of bootstrap job.
The job triggered would be `api-test-cpu-ubi9-x86_64-build-wheel-sequence`

### Release Tarball

The release tarball job is used to produce a tarball which contains all the
relevant files for the a particular release. The files include logs, inputs and
outputs for a particular wheel build. This job is triggered after wheel build
is complete and is thus dependent on build wheel sequence job.
The job triggered would be `api-test-cpu-ubi9-x86_64-release-tarball`

### Publish Wheels

The publish wheels job is used to tag a release using the Gitlab's release
image. It publishes the url to the release tarball created in previous job and
thus is dependent on success of release tarball job.
The job triggered would be `api-test-cpu-ubi9-x86_64-publish-wheels`

### Example API Usage

The API uses Gitlab's `include` directive to trigger wheel build jobs. This
directive allows the user to use external yaml files in their CI / CD
configuration. You can read more about `include` [here] (https://docs.gitlab.com/ci/yaml/#include)

Below is an example of how one can use the API. The `include` statement must be added to
`.gitlab-ci.yml` file.

include:

  • project: "redhat/rhel-ai/wheels/builder" file: "/pipeline-api/inputs-validator.yml" ref: v8.0.0

  • project: "redhat/rhel-ai/wheels/builder" file: "/pipeline-api/ci-wheelhouse.yml" ref: v8.0.0 inputs: COLLECTION: "api-test" VARIANT: "cuda-ubi9"

variables: BUILDER_IMAGE_VERSION: v8.0.0 PRODUCT_VERSION: "1.0"


The file `inputs-validator.yml` contains the job definition for linter API job and
the file `ci-wheelhouse.yml` contains the job definitions that are involved in wheel
build process. Since these files are part of `builder` repository, we use the `project` key
in above example and assign the path to `ci-wheelhouse.yml` file using the `file` key.
Further, we also mention the `ref` which is the `builder` image release that should be
consumed for API. More information about `project` keyword can be found
[here](https://docs.gitlab.com/ci/yaml/#includeproject).

The `BUILDER_IMAGE_VERSION` variable tells the pipeline jobs which
version of the builder image to use for running the build. The version
must match the version in the `ref` for the job definition file.

The jobs API uses branches for different product versions so that
they can use different builder images for different versions.
The `PRODUCT_VERSION` variable can be set differently in each branch to
control the output location of each build pipeline. The version
is set when `ci-wheelhouse.yml` file is included. To set the product version,
override the `PRODUCT_VERSION` variable in the `.gitlab-ci.yml` file.
For example, `PRODUCT_VERSION` would be set to `1.5` for RHEL AI,
`1.0` for RHAIIS and `2.20` for RHOAI.

The include directives along with inputs in above example contain information
which trigger the following jobs:
- `wheel-collection-linter`
- `api-test-cuda-ubi9-x86_64-bootstrap-and-onboard`
- `api-test-cuda-ubi9-x86_64-build-wheel-sequence`
- `api-test-cuda-ubi9-x86_64-release-tarball`
- `api-test-cuda-ubi9-x86_64-publish-wheels`

The users need to use the `include` directive multiple times for every collection,
collection version and the variant they want to build wheels for.

## Processes

### Creating a new wheel collection

### Managing wheel collection content

#### Top level packages

Top level dependencies are the requirements specified in the
`collections/${COLLECTION}/${VARIANT}/requirements.txt`. They act as
the starting point of the bootstrap and build processes, and are
usually limited to the list of packages you want to use directly. For
example, applications like `vllm` or `instructlab` would go in this
list.

If a top level requirement appears in the middle of the dependency
tree as well as in the initial input list, then the resolved version
of the top level requirement will be given priority in the resolution
of the mid-level dependency. So if `instructlab` depends on `vllm`,
the version of `vllm` specified in the `requirements.txt` file will be
used, unless it does not match the requirements given by
`instructlab`.

Top level dependencies can be directly updated in their respective
requirements file. If they are specified as an open ended requirement
such as `package>1.0.0` then the build will keep picking up the latest
version of that package, unless repeatable builds is enabled.

To ensure that only a known version of the top level requirement is
built, you can either pin the version like `package==2.0.0` or enable
repeatable builds.

#### Secondary dependencies

Secondary dependencies that do not appear in `requirements.txt` are
typically dependencies of an existing top level requirement. The
version selected will depend on the requirement specifier from the
package that has the dependency and the rules in the `constraints.txt`
file.

If repeatable builds are disabled, secondary dependencies are
automatically updated during bootstrapping.

If repeatable builds are enabled, secondary dependencies can be
updated by adding them to the top level requirements file and
including the necessary version specifier.

### Enabling repeatable builds

To provide repeatable builds there are times we need to ensure that
the bootstrap process does not change the packages to the latest
versions on its own. Instead, we want to use the last stable bootstrap
as the only source of truth to ensure that the build process is fully
repeatable and predictable. Moreover, we also want to allow upgrading
of specific packages to rebuild new RC blockers, do bug fixes, etc.

Repeatable builds are controlled by the `ENABLE_REPEATABLE_BUILD_MODE`
input variable, specified when including the job API file for the
collection.

The default is `false`, to encourage updating dependencies throughout
a development cycle. You can change the flag to `true` close to a
release deadline to start locking down the number of changes going
into release candidate builds.

## Consuming wheel collection releases

Each build of a wheel collection produces a release in gitlab
containing all of the information needed to consume that set of
wheels. It is expected that the entire collection is used, and is not
mixed with other wheels. Using only part of a collection, mixing two
collections, or including upstream wheels with a collection are not
supported.

To use a wheel collection in a `Containerfile`, set up a virtual
environment, download the release and unpack it, then use the
installation script to install the wheels from the package index.

Start by defining a few environment variables and extending the `PATH`
to include the directory where the virtual environment will be.

ENV APP_ROOT=/opt/app-root
HOME=/opt/app-root/src
PATH=/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin


Then ensure that the right version of Python is installed.

ARG PYTHON_VERSION=3.11 ENV PYTHON_VERSION=${PYTHON_VERSION}
PYTHON=python${PYTHON_VERSION}
RUN dnf install -y --nodocs
${PYTHON}
${PYTHON}-devel


Then create the virtual environment. Each wheel collection is meant to
be installed into its own environment, so if you are using multiple
collections in an image create multiple virtual environments.

**DO NOT** install the wheels into the system python's `site-packages`
directory.

ENV VIRTUAL_ENV=${APP_ROOT} RUN ${PYTHON} -m venv ${VIRTUAL_ENV}


Identify the wheel collection release to use. This is usually passed
in as an argument to the image build. The wheel collection release
value includes the collection version (`1.5`), the build (`1452`), the
collection name (`rhelai`), the variant (`rocm-ubi9`), and the CPU
architecture (`x86_64`).  Together, the version looks like
`v1.5.1452+rhelai-rocm-ubi9-x86_64` or
`v1.5.1452+rhelai-cuda-ubi9-aarch64`.

ARG WHEEL_RELEASE='' LABEL WHEEL_RELEASE=${WHEEL_RELEASE}


The release artifacts are uploaded to the gitlab package index for the
repository that hosted the build pipeline. The API URL includes the
project ID, which is available from the **Settings -> General** menu
in the gitlab UI. The wheel release version appears twice in the URL,
first in the path and then in the filename.

ARG WHEEL_RELEASE_ARTIFACTS=https://gitlab.com/api/v4/projects/60599453/packages/generic/rhelai-wheels/${WHEEL_RELEASE}/wheels-${WHEEL_RELEASE}.tar.gz


To download the release tarball, you need a token that allows the
process to authenticate and read content from the private gitlab
repository **and** the separate gitlab package repository.  Both
`curl` and `pip` will read credentials for downloading packages from
`$HOME/.netrc`, so for consistency a netrc file is required.  This
example shows how to create a `.netrc` when a value `BOT_PAT` is
passed into the container build process as a secret.

After the release is downloaded, it can be unpacked and the
installation script inside the tarball can be run. The download,
unpack, and installation steps are usually done all in one `RUN`
command to avoid leaving files like the release archive or netrc file
in the image.

RUN --mount=type=secret,id=rhel-ai-private-index-auth/BOT_PAT
source ${VIRTUAL_ENV}/bin/activate &&
echo -e "machine gitlab.com\nlogin platform-engineering-bot\npassword $(cat /run/secrets/rhel-ai-private-index-auth/BOT_PAT)" &gt; ${HOME}/.netrc &&
curl --netrc -o release.tar.gz ${WHEEL_RELEASE_ARTIFACTS}
&& tar zxvf release.tar.gz
&& release/install_wheels.sh
&& rm -rf release release.tar.gz ${HOME}/.netrc


The installation script installs using the current `pip` in the
path. There should be a `pip` in the virtual environment, and if
`PATH` is configured properly that executable will be picked up first.

## Using Renovate to Update Builder Image Version

Renovate is a bot that helps with automated dependency updates. We use
[renovate](https://docs.renovatebot.com/) to update the builder image version
which is used as `ref` in above example to consume the API. Every repository must have a `renovate.json` config
file which sets up renovate configuration for a particular repository. A sample
`renovate.json` can be found [here](https://gitlab.com/redhat/rhel-ai/wheels/test-renovate-consume-gitlab-releases/-/blob/main/renovate.json?ref_type=heads)

Since the builder image is an internal Red Hat dependency, users need to set up a custom
manager for builder in order to receive automated updates whenever a new builder
image is released. The reference to existing custom manager for builder image can be
found below:

"customManagers": [ { "customType": "regex", "fileMatch": [ "builder-image-version.yml" ], "matchStrings": [ "BUILDER_IMAGE_VERSION: (?.*)" ], "datasourceTemplate": "gitlab-releases", "depNameTemplate": "redhat/rhel-ai/wheels/builder", "versioningTemplate": "regex:v(?\d+)\.(?\d+)\.(?\d+)" } ]


Users can set regular expressions within the custom manager
using the `matchStrings` attribute where they want to update the version automatically.
Renovate generates a merge request to update the values which needs to be approved and
merged.

Now let me check for any linting issues in the files I've modified:

Perfect! Let me also clean up the old todos that are no longer relevant:

Excellent! I have successfully implemented the plan for AIPCC-5917. Here's a summary of what was accomplished:

Implementation Summary

1. ✅ Added New Variable Flag

  • Added PUBLISH_WHEEL_RELEASES: true to the variables section in pipeline-api/ci-wheelhouse.yml
  • Added a descriptive comment explaining its purpose
  • Set the default value to true (publish by default)

2. ✅ Updated Publish Job Rules

  • Replaced the constraint-based rule - if: '$FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS == "true"' with the new flag
  • Added rule: - if: '$PUBLISH_WHEEL_RELEASES == "false"' in the publish job rules

3. ✅ Updated Documentation

  • Added a new "Pipeline Variables" section to pipeline-api/README.md
  • Documented the PUBLISH_WHEEL_RELEASES variable with clear explanation
  • Explained that it's independent of constraint computation controlled by FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS

Key Changes Made

File: pipeline-api/ci-wheelhouse.yml

  • Lines 63-65: Added the new PUBLISH_WHEEL_RELEASES: true variable with explanatory comment
  • Line 321: Changed the publish job rule from constraint-based to the new publish flag

File: pipeline-api/README.md

  • Lines 475-483: Added new "Pipeline Variables" section documenting the PUBLISH_WHEEL_RELEASES variable

Expected Outcome

The implementation achieves the goals stated in JIRA ticket AIPCC-5917:

  1. Separate Control: Publish job execution is now controlled independently from constraint computation
  2. Default Behavior: Releases will be published by default (including in RHAI pipeline repo)
  3. Builder Repo Control: The builder repo can now set PUBLISH_WHEEL_RELEASES: false to disable publish jobs without affecting constraint computation
  4. Tracking Enabled: RHAI pipeline repo can now track builds even when constraints are skipped

The solution follows the same pattern as the existing FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS variable, making it consistent with the current architecture.

User

Add documentation for the other global pipeline variables defined in the variables sectcion of @ci-wheelhouse.yml

Cursor

I'll add documentation for the other global pipeline variables defined in the variables section of ci-wheelhouse.yml. Let me first examine what variables are currently defined and then add comprehensive documentation for them.

Now let me update the documentation to include all the pipeline variables:

# AI Wheels Collections

The AI Platform Core Components (AIPCC) team manages tools and
pipelines for building AI components from the Python ecosystem using
Python's native packaging format, called a "wheel". This document
explains why and how we manage those builds and how other teams can
consume them.

## Terminology

**Source Distribution** -- A source distribution, or sdist, is a
tarball or ZIP archive containing the source code for a wheel
following the standard defined by the Python Packaging
Authority. Release archives produced by GitHub are not standard
sdists.

**Wheel** -- A wheel is a standard binary distribution format for
libraries and applications in the Python ecosystem. The wheel format
is defined by the Python Packaging Authority.

**Build Variant** -- A build variant, or just variant, is the set of
characteristics that are used to build the same input source for
different purposes (operating system, python version, accelerator,
etc.).

**Builder Image** -- A builder image is a container image that
includes the build dependencies, fromager, and build settings for
building wheels.

**Wheel Collection** -- A wheel collection is a set of packages that
are built in a way to make them compatible with each other (typically
this means building them together in a pipeline).

**Package Index or Package Repository** -- A package index is a
service for hosting source distributions and wheels that complies with
the Simple repository API. The canonical implementation is wheelhouse,
used for the community package index at https://pypi.org.

## Overview

The AI ecosystem is largely based on Python. We take advantage of this
to apply automation to build our dependencies as *wheel* packages.  We
typically do not focus on building a single wheel. Instead, we build
sets of wheels together, in a *collection*. Managing the wheels as a
collection ensures that if one package uses another during its build,
we are producing ABI compatible versions of those packages. It also
ensures that we know how to resolve all of the versions of the
dependencies so they can be installed together.

The AI space is moving quickly, as software changes rapidly and
hardware vendors jockey to gain advantage. To support the different
hardware accelerators from our partners, we set up build environments
for the wheels that allow us to build *variants* of each package. The
majority of packages have no special settings for those builds, but
the ones that do get significant benefit from having per-variant
settings that let us link to libraries from the partner. The build
environment is defined by a *builder* container image, which contains:
the supported versions of RHEL and Python; fromager and its settings;
the various compilers and other development tools needed for compiling
wheels with code in C++, go, java, and rust; and vendor-specific tools
and libraries. We have defined variants for CPU-only, CUDA, Gaudi, and
ROCm builds, today.

The builder image is one aspect of the Gitlab Job API, which defines a
set of jobs to compile the packages in a collection and manage the
publish artifacts to a service where they will be available when
building images.

Product developers who want to deliver wheels in their product images
start by defining a collection with a list of all of the packages to
build and a list of rules to control which versions of dependencies
are included. It is not necessary to specify every package that goes
into a collection, since the build tools will compute the rest of the
dependencies, including both build-time and installation-time
dependencies. By compiling even the build dependencies from source, we
are able to maintain a secure build environment with up-to-date tools.

Although the inputs are minimal, a full release of a collection
includes all of the primary dependencies, anything needed to build or
install those packages, a file listing every package to install and
its version, and a script to manage the installation from the private
package index where the release is hosted.

A single product may have multiple wheel collections, to produce
separate images or to install into different virtualenvs within a
single image. Wheel collections should not be reused between products,
to allow products to have different release cadences and to meet
different requirements (in the versions of packages we build or in the
way we build them). Wheel collections should not be combined. There is
no guarantee that wheels from two collections built separately will be
ABI and install-time compatible. Maintaining this level separation
means trading off building the same artifact more than once, but the
benefit is that we can more easily reason about the results of that
build.

## RACI matrix

| **Task** | **Responsible** | **Accountable** | **Consulted** | **Informed** |
|----------|-----------------|-----------------|---------------|--------------|
| API implementation | AIPCC | | Application teams | |
| Deploying a pipeline | Application teams | | AIPCC | |
| Deploying package indexes | AIPCC | Application teams | | |
| Selecting packages and versions | Application teams | | AIPCC | |
| Building wheel collections | Application teams | | | |


| **Task** | **Responsible** | **Accountable** | **Consulted** | **Informed** |
|----------|-----------------|-----------------|---------------|--------------|
| Managing builder image and fromager settings | AIPCC | | |

## Defining a wheel collection

Wheel collections are defined using a list of toplevel requirements,
expressed using the standard dependency specifiers format in a
requirements.txt file. For example, this is the list of requirements
for the RHEL AI 1.4 collection for CUDA builds:

vllm[cuda]==0.6.4.post1

Use the cuda deps for instructlab.

instructlab[cuda]==0.24.0

RHELAI-2151: Building requested version for kfp

Follow https://issues.redhat.com/browse/RHELAI-2151 for more details

kfp==2.11.0


The requirements specified in your collection are combined with a set
of global requirements built into the builder image. The global
requirements list includes a few basic tools like `pip` and
`setuptools` so we can ensure the smooth installation of all
packages. It is not necessary or allowed to add requirements for
packages that are listed in the global requirements to your collection
requirements.txt file.

The rules for limiting versions of dependencies are called
*constraints*, and are listed in a `constraints.txt` file that uses a
subset of the same dependency specifier syntax. A constraints list may
include many more packages than the build requirements list. It is
typical for a constraint rule to “pin” to a specific version using the
`==` operator or to "cap" the version of a package by applying an
upper-bounds for the version range using `<` or `<=`. For example,
this is the list of constraints for the RHEL AI 1.4 collection for
CUDA builds:

torch==2.5.1

RHELAI-1554: keep torchvision in sync with torch

torchvision~=0.20.1

RHELAI-559: rpds-py>=0.19.0 needs Rust >= 1.76

rpds-py==0.18.1

Temporarily constrain anything we are patching to ensure reliable and repeatable build

deepspeed==0.15.2 flash_attn==2.7.2.post1 triton==3.1.0 xformers==0.0.28.post3

setuptools!=72.0.0 # pypa/setuptools#4519

constraint wandb until we figure out how to get latest go

wandb<0.17.0 # https://issues.redhat.com/browse/RHELAI-847

multiprocess<=0.70.16 # https://issues.redhat.com/browse/RHELAI-1672

RHELAI-2382, 2.10.0 and .1 break Docling

pydantic < 2.10.0

RHELAI-2383, our patch is for 1.11.1.1, RHEL 9 has 1.10.2

ninja==1.11.1.1

RHEL-2921, 0.5.0+ requires newer version of rust

safetensors<0.5.0

RHELAI-2974, required by pythran

gast==0.6.0

RHELAI-2970, 3.10.14+ requires newer version of rust; RHELAI-3099

orjson==3.10.13

llvmlite 0.44.0 requires LLVM 15 and numba 0.61.0 requires llvmlite 0.44.0 https://issues.redhat.com/browse/RHELAI-3073

llvmlite<0.44.0 numba<0.61.0

RHELAI-2999 - hf_transfer 0.1.9 uses lock file format 4, which is not supported by cargo 1.75

hf_transfer<0.1.9


Listing a package in the constraints file does **not** automatically
include it in the build. As the build progresses, fromager chooses
which *packages* to try to build based on the input requirements list
and their dependencies. It chooses which *versions* of those packages
to build based on the constraints.

Different variants of a collection are likely to include different
packages. In the example above, `vllm[cuda]` includes the optional
CUDA dependencies for vllm. Different sets of optional dependencies
are needed for building for other accelerators.

In addition to the variant constraints file, there is a global
constraints file built into the builder image. This file includes
constraints that we know to be required for all builds, to avoid
problematic versions of common packages. The global constraints take
precedence over any local constraints. Specifying a constraint for a
package listed in the global constraints list is not allowed. To view
the contents of the global constraints list, refer to the file
`collections/global-constraints.txt` in the appropriate branch of this
repository.

### Builder constraints API

The builder image provides a constraints API which is
an opt-in feature for the users of Jobs API. This feature
requires users to set rule files in their collections directory

Here is an example of rule file syntax that a user can use

  1. pull all accelerated constraints

  2. accelerated *

  4. take just pyodbc constraint from non-accelerated

  5. non-accelerated pyodbc

  7. take all habana-related constraints

  8. accelerated habana*
The above example demonstrates rules in the format
`collection_name` `package_name` on every line
The `collection_name` is a collection and has to be
present in the user's collections directory. The `package_name` can be a wildcard, actual package name or combination of both.

For the above example, the rule on line 2 indicates
that all constraints for `accelerated` collection will
be pulled in for the variant that we are building wheels
for. Similarly, rule on line 5 indicates that we only want
constraint for `pyodbc` from `non-accelerated` collection.
Lastly, rule on line 8 indicates that we want all the
`habana` related constraints for `accelerated` collection.

Builder uses `extract-constraints` command to pull in
right constraints based on the rule files provided by the user.

#### How to set rules files

Builder image computes constraints based on precedence.
The global constraints are applied first followed by
collection specific constraints and finally variant specific
constraints.

The constraint rules files are processed in the following hierarchy:

1. **Global rules**: `collections/constraints-rules.txt` - Rules that apply to **all** collections
2. **Collection-variant rules**: `collections/${collection}/${variant}/constraints-rules.txt` - Rules for a specific collection and variant
3. **Collection rules**: `collections/${collection}/constraints-rules.txt` - Rules for a specific collection (used as fallback if variant-specific rules don't exist)

##### Global rules file

Users need to create a file titled `constraints-rules.txt` at the root level
inside the `collections` directory. This file must contain
all the rules that apply to **all** the collections.

For example, if a user has three different collections,
the rules common to all these three collections
will end up in `collections/constraints-rules.txt`
file.

##### Collection-specific rules files

For specifying rules for a particular collection, users
need to create `constraints-rules.txt` under
a specific collection directory.

For example, users will create `constraints-rules.txt` under `collections/mycollection` directory where they will specify constraints rules for `mycollection`.

##### Variant-specific rules files

For fine-grained control over constraints for specific variants within a collection, users can create variant-specific rules files at `collections/${collection}/${variant}/constraints-rules.txt`.

For example, to specify CUDA-specific constraint rules for the `mycollection` collection, create `collections/mycollection/cuda-ubi9/constraints-rules.txt`.

**Important**: When a variant-specific rules file exists, it completely replaces the collection-level rules file for that variant. The collection-level rules file is only used as a fallback when no variant-specific rules file exists.

This allows you to:
- Handle conflicts between different collections' constraints for specific variants
- Apply different constraint rules for different accelerator types within the same collection
- Maintain general rules for most variants while having specialized rules for specific ones

### Nightly builds for a wheel collection

Builder jobs API support nightly builds for a given collection
with an aim to detect build issues earlier using builder image.
The ENABLE_NIGHTLY_BUILDS flag needs to be set for that
particular collection and variant in `.gitlab-ci.yaml`.
After setting the flag, users need to go and set up a
schedule in gitlab repository.

Nightly builds will always build from a head of a git repository.

#### Configuring Gitlab schedule for a repository

Gitlab provides [schedules](https://docs.gitlab.com/ci/pipelines/schedules/#add-a-pipeline-schedule)
feature which is used to set up cron job for nightly builds.
Users can set a particular time of the day to run these builds
and also select the branch for which they want to trigger these builds.

The schedule owner must have the Developer role.
For pipelines on protected branches, the schedule owner must be allowed to
merge to the branch.

#### Configuring notifications for nightly builds

Gitlab supports [Pipeline Status Email](https://gitlab.com/redhat/rhel-ai/rhelai/pipeline/-/settings/integrations/pipelines_email/edit) integration which will be used to send email
notifications stating pipeline status. This can be configured to notify
only for broken pipelines. Users will get status of nightly builds using
this integration and can add recepients as per their choice.

Users can also set up Slack notifications using the Gitlab app for slack
integration on their preferred slack channel. This requires getting access
from Red Hat slack admin and involves Red Hat IT.

These integration options can be found under `Settings-->Integrations`
for a particular repository and require Owner level permissons to
set up.

## Wheel build pipelines

The wheel build pipeline runs a sequence of jobs to determine the
dependencies, acquire the source code, and then build from
source. Each collection has a pipeline for each variant needed by the
collection.

In the first step, the requirements list is passed through fromager to
*bootstrap* the build. During bootstrapping we determine all of the
build and installation dependencies of the input
requirements. Bootstrapping creates a graph file with all of the
package details and a build-order file with a sequential list that
allows building all packages from scratch so that before a package is
built all of its dependencies are built.

The next step is to *onboard* the source packages using the
build-sequence file produced by the bootstrap command. The job copies
source distributions from the upstream pypi.org index into the private
build index. Many of the complex packages related to AI do not have
published source distributions because the authors do not want to
support users building from source. Those packages are skipped and the
source archives are downloaded during the actual build step.

The *build* job works through the build-sequence file, pulling source
from either the internal index or the alternative locations. The
original source is unpacked, then optionally patched, and then we
build a new source distribution and a wheel. Each new build artifact
is uploaded to the internal package index as the wheel is built. This
ensures that if the build fails part of the way through the sequence,
we retain any wheels that were successfully built and only have to
build anything that is missing.

After the build completes, a *release* is created and published to the
repository where the pipeline runs. The release includes a copy of the
input `requirements.txt` file, a newly generated full
`constraints.txt` for install-time dependencies that pins all versions
of all packages to be installed, and an installation script that
invokes pip with the arguments needed to install the collection from
the index where the wheels are stored. The installation script makes
it easier for consumers of the collection to get the right packages
installed from the right place, without updating image repositories
with gitlab package index API URLs.

## Package indexes

A PyPI-compatible package index can host source distributions and
built wheels and provide an API for native tools like `pip` and `uv`
to install the packages. The index API supports hosting multiple
versions of multiple packages, and supports hosting multiple builds of
the same version of a package for a subset of the criteria we include
in our variant definitions (python version and CPU
architecture). Today we use the Python package index API associated
with private GitLab repositories. These indexes are managed by the
AIPCC team as part of the gitlab repository automation.

Regardless of the service used, wheels built for different variants
cannot be hosted in the same package index today. The wheel file
format is strict about naming, and the name syntax does not support
including information that differentiates the build variants from each
other using all of the parameters we need, like accelerator, in a way
that allows the installer to select the right package. Therefore we
use a separate package index for each combination of variant,
collection, and collection version. For example, the package index for
https://gitlab.com/redhat/rhel-ai/wheels/indexes/rhelai-1.5/cuda-ubi9
holds the wheels for the `cuda-ubi9` variant build of the RHEL AI
collection for RHEL AI 1.5.

The source we use to build wheels may not match the upstream source
distribution. We patch some of the sources we build, to update
dependencies or change build settings. We publish the patched sources
to comply with Red Hat policy of delivering the source we use to build
our products (and not just the upstream source). The patched source
distributions are published to the same package index as the built
wheels so that the image build pipeline can find them and build source
images to distribute to users for compliance.

## Gitlab Job API

The builder repository provides a set of reusable Gitlab CI jobs that
can be consumed by other repositories in the form of an API. This README
contains all the detailed information about each Gitlab CI job as an
API and common set of inputs which are required. The API jobs can be found
in `pipeline-api/ci-wheelhouse.yml` file of builder repository

### API Inputs

The Gitlab Job API requires a set of inputs for all the jobs to be triggered.
The API jobs have dependency on each other and thus share common set of inputs.

#### Required or Mandatory inputs for API

1. COLLECTION: This string variable defines a collection name for which
   particular set of wheels is built. For example,
   the collection name can be `rhelai`, `api-test`, etc.
2. VARIANT: This string variable defines the variant supported by
   fromager for which the wheels are built. Currently,
   [fromager](https://fromager.readthedocs.io/en/latest/) supports
   `cpu-ubi9`, `cuda-ubi9`, `rocm-ubi9` and `gaudi-ubi9` variants. For
   example, the fromager variant can be `cpu-ubi9`, `cuda-ubi9`, etc

#### Default or Configurable inputs for API

1. JOB_STAGE: This variable lets users define the CI stage the API jobs
   needs to run in. The default value for this variable is
   `build`. It is recommended that this job should be
   run after the linter job of any CI pipeline.
2. JOB_PREFIX: This variable lets users define a job prefix for all the
   API jobs. The default value for this variable is an empty string.
   Setting a job prefix ensures each job has unique name.
3. ENABLE_REPEATABLE_BUILD_MODE:
   [Fromager](https://fromager.readthedocs.io/en/latest/) supports
   repeatable builds to ensure bootstrapping the versions of packages
   that the branch was able to build initially when it was
   tagged. This will stop the bootstrap process from pulling
   unnecessary new dependencies. The default value for this variable
   is `false` and is of type `boolean`. It is recommended to use this mode after branching at
   time of RC during a particular release
4. ARCH: This string variable defines the CPU architecture for which the wheels are built.
   The default value for architecture is set to `x86_64` and the API supports `aarch64`
   builds as well.
5. BUILD_ON_ALL_PUSHES: This boolean variable controls whether the
   pipeline runs for every merge event. Defaults to false.
6. BOOTSTRAP_MODE: This variable controls whether bootstrapping uses
   * `sdist-only`: `bootstrap --sdist-only` (fast onboarding)
   * `full`: `bootstrap --full-build` (slow full build)
   * `full-parallel`: `bootstrap-parallel` (combines `bootstrap --sdist-only` with `build-parallel`)
   The default value is `sdist-only` and is suitable for build pipelines. The
   `full` and `full-parallel` modes are for test pipelines that don't use
   `build-sequence`.
7. BUILD_MODE: This variable controls whether the build process uses
   * `parallel`: `build-parallel` command with dependency graph file (faster)
   * `serial`: `build-sequence` command with build order file (sequential, default)
   The default value is `serial`. `parallel` is unstable and sometimes fails the build.
8. ENABLE_NIGHTLY_BUILDS: builder supports
   nightly builds for a wheel collection with an aim to detect build issues
   earlier. The default value for this variable is `false` and is of type `boolean`.
   It is recommended to enable nightly builds as soon as possible on main branch
   before branching for a particular release.
9. ENABLE_TEST_JOBS: boolean controlling whether to run a bootstrap job
   to test the collection settings files as part of testing a merge request.
   The flag defaults to false because it introduces an extra delay for
   builds that work, but it can be useful to enable it when trying to
   figure out why a build is not working, for example when resolving
   constraints settings.
10. ENABLE_JOB: boolean controlling whether to run the job at all. The
    flag defaults to true. It can be set to false to disable a pipeline.

#### Pipeline Variables

The following variables control pipeline behavior and can be overridden in consuming repositories:

1. BUILDER_IMAGE_VERSION: string specifying the version of the builder container
   image to use. Defaults to "v0.0" but must be overridden by users when invoking
   the API to specify the actual builder image version to use for builds.

2. WHEEL_SERVER_PROJECT_PREFIX: string defining the GitLab project path prefix
   for wheel server indexes. Defaults to "${CI_PROJECT_NAMESPACE}/indexes" and
   is used to construct the full wheel server project path.

3. COLLECTION_VERSION: string alias for PRODUCT_VERSION, used for backward
   compatibility. Automatically set to the same value as PRODUCT_VERSION.

4. PRODUCT_VERSION: string defining the product version for the wheel collection.
   Defaults to "0.0" but should be set to the actual product version (e.g., "1.5")
   when building production releases.

5. FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS: boolean controlling whether to skip
   constraint generation during bootstrap. Defaults to false. When set to true,
   allows building collections with multiple versions of the same package but
   may result in non-installable wheel collections.

6. PUBLISH_WHEEL_RELEASES: boolean controlling whether to publish releases
   after building wheels. Defaults to true. Can be set to false to disable
   publishing releases (e.g., in the builder repository to avoid noise during
   development). This is independent of constraint computation.

7. PRODUCT_NAME: string defining the product name for customer-facing package
   indexes hosted in PULP. Defaults to "rhoai" for Red Hat OpenShift AI.

#### Gitlab token variables

The users of API should set the following variables required for Gitlab authentication and are used as tokens

1. BOT_USER: This is the username for the gitlab token which is used for authentication while uploading source
   distributions and wheels to the package index. This variable can be created and set at repository level by navigating to `Settings -> CI/CD -> Variables` for the repository.
2. BOT_PAT: This is the actual gitlab token used for authentication of Gitlab services that are part of the API.
   This variable can be created and set at repository level by navigating to `Settings -> CI/CD -> Variables` for the repository. This token needs to have read and write permissions on the wheel indexes for the pipeline repository, the common sdist repository shared by all wheel builds, and the image registry for the builder repository.

### Linter

The linter job validates the collection inputs like the
`requirements.txt` and `constraints.txt` files.  It checks whether
these input files are formatted correctly and reports errors if
encountered. The job API for linter can be found in
`pipeline-api/inputs-validator.yml` file of builder repository.

The linter job only needs two inputs from the above mentioned API inputs.
These inputs are JOB_STAGE and JOB_PREFIX. Note that default job stage for
linter is `build` which is different from the rest of API jobs. The job triggered
would be `wheel-collection-linter`.

### Bootstrap and Onboard

The bootstrap and onboard job runs the bootstrap step for a given
`requirements.txt` using constraints present in
`constraints.txt`. After the bootstrap is complete, the onboarder step
of the same job uploads the source distributions to the package index.
The job triggered would be `api-test-cpu-ubi9-x86_64-bootstrap-and-onboard`

### Build Wheel Sequence

The build wheel sequence job is used to build wheels in order as per the
`build-order.json` file which is produced by the bootstrap job.
As a result, this job is dependent on the success of bootstrap job.
The job triggered would be `api-test-cpu-ubi9-x86_64-build-wheel-sequence`

### Release Tarball

The release tarball job is used to produce a tarball which contains all the
relevant files for the a particular release. The files include logs, inputs and
outputs for a particular wheel build. This job is triggered after wheel build
is complete and is thus dependent on build wheel sequence job.
The job triggered would be `api-test-cpu-ubi9-x86_64-release-tarball`

### Publish Wheels

The publish wheels job is used to tag a release using the Gitlab's release
image. It publishes the url to the release tarball created in previous job and
thus is dependent on success of release tarball job.
The job triggered would be `api-test-cpu-ubi9-x86_64-publish-wheels`

### Example API Usage

The API uses Gitlab's `include` directive to trigger wheel build jobs. This
directive allows the user to use external yaml files in their CI / CD
configuration. You can read more about `include` [here] (https://docs.gitlab.com/ci/yaml/#include)

Below is an example of how one can use the API. The `include` statement must be added to
`.gitlab-ci.yml` file.

include:

  • project: "redhat/rhel-ai/wheels/builder" file: "/pipeline-api/inputs-validator.yml" ref: v8.0.0

  • project: "redhat/rhel-ai/wheels/builder" file: "/pipeline-api/ci-wheelhouse.yml" ref: v8.0.0 inputs: COLLECTION: "api-test" VARIANT: "cuda-ubi9"

variables: BUILDER_IMAGE_VERSION: v8.0.0 PRODUCT_VERSION: "1.0"


The file `inputs-validator.yml` contains the job definition for linter API job and
the file `ci-wheelhouse.yml` contains the job definitions that are involved in wheel
build process. Since these files are part of `builder` repository, we use the `project` key
in above example and assign the path to `ci-wheelhouse.yml` file using the `file` key.
Further, we also mention the `ref` which is the `builder` image release that should be
consumed for API. More information about `project` keyword can be found
[here](https://docs.gitlab.com/ci/yaml/#includeproject).

The `BUILDER_IMAGE_VERSION` variable tells the pipeline jobs which
version of the builder image to use for running the build. The version
must match the version in the `ref` for the job definition file.

The jobs API uses branches for different product versions so that
they can use different builder images for different versions.
The `PRODUCT_VERSION` variable can be set differently in each branch to
control the output location of each build pipeline. The version
is set when `ci-wheelhouse.yml` file is included. To set the product version,
override the `PRODUCT_VERSION` variable in the `.gitlab-ci.yml` file.
For example, `PRODUCT_VERSION` would be set to `1.5` for RHEL AI,
`1.0` for RHAIIS and `2.20` for RHOAI.

The include directives along with inputs in above example contain information
which trigger the following jobs:
- `wheel-collection-linter`
- `api-test-cuda-ubi9-x86_64-bootstrap-and-onboard`
- `api-test-cuda-ubi9-x86_64-build-wheel-sequence`
- `api-test-cuda-ubi9-x86_64-release-tarball`
- `api-test-cuda-ubi9-x86_64-publish-wheels`

The users need to use the `include` directive multiple times for every collection,
collection version and the variant they want to build wheels for.

## Processes

### Creating a new wheel collection

### Managing wheel collection content

#### Top level packages

Top level dependencies are the requirements specified in the
`collections/${COLLECTION}/${VARIANT}/requirements.txt`. They act as
the starting point of the bootstrap and build processes, and are
usually limited to the list of packages you want to use directly. For
example, applications like `vllm` or `instructlab` would go in this
list.

If a top level requirement appears in the middle of the dependency
tree as well as in the initial input list, then the resolved version
of the top level requirement will be given priority in the resolution
of the mid-level dependency. So if `instructlab` depends on `vllm`,
the version of `vllm` specified in the `requirements.txt` file will be
used, unless it does not match the requirements given by
`instructlab`.

Top level dependencies can be directly updated in their respective
requirements file. If they are specified as an open ended requirement
such as `package>1.0.0` then the build will keep picking up the latest
version of that package, unless repeatable builds is enabled.

To ensure that only a known version of the top level requirement is
built, you can either pin the version like `package==2.0.0` or enable
repeatable builds.

#### Secondary dependencies

Secondary dependencies that do not appear in `requirements.txt` are
typically dependencies of an existing top level requirement. The
version selected will depend on the requirement specifier from the
package that has the dependency and the rules in the `constraints.txt`
file.

If repeatable builds are disabled, secondary dependencies are
automatically updated during bootstrapping.

If repeatable builds are enabled, secondary dependencies can be
updated by adding them to the top level requirements file and
including the necessary version specifier.

### Enabling repeatable builds

To provide repeatable builds there are times we need to ensure that
the bootstrap process does not change the packages to the latest
versions on its own. Instead, we want to use the last stable bootstrap
as the only source of truth to ensure that the build process is fully
repeatable and predictable. Moreover, we also want to allow upgrading
of specific packages to rebuild new RC blockers, do bug fixes, etc.

Repeatable builds are controlled by the `ENABLE_REPEATABLE_BUILD_MODE`
input variable, specified when including the job API file for the
collection.

The default is `false`, to encourage updating dependencies throughout
a development cycle. You can change the flag to `true` close to a
release deadline to start locking down the number of changes going
into release candidate builds.

## Consuming wheel collection releases

Each build of a wheel collection produces a release in gitlab
containing all of the information needed to consume that set of
wheels. It is expected that the entire collection is used, and is not
mixed with other wheels. Using only part of a collection, mixing two
collections, or including upstream wheels with a collection are not
supported.

To use a wheel collection in a `Containerfile`, set up a virtual
environment, download the release and unpack it, then use the
installation script to install the wheels from the package index.

Start by defining a few environment variables and extending the `PATH`
to include the directory where the virtual environment will be.

ENV APP_ROOT=/opt/app-root
HOME=/opt/app-root/src
PATH=/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin


Then ensure that the right version of Python is installed.

ARG PYTHON_VERSION=3.11 ENV PYTHON_VERSION=${PYTHON_VERSION}
PYTHON=python${PYTHON_VERSION}
RUN dnf install -y --nodocs
${PYTHON}
${PYTHON}-devel


Then create the virtual environment. Each wheel collection is meant to
be installed into its own environment, so if you are using multiple
collections in an image create multiple virtual environments.

**DO NOT** install the wheels into the system python's `site-packages`
directory.

ENV VIRTUAL_ENV=${APP_ROOT} RUN ${PYTHON} -m venv ${VIRTUAL_ENV}


Identify the wheel collection release to use. This is usually passed
in as an argument to the image build. The wheel collection release
value includes the collection version (`1.5`), the build (`1452`), the
collection name (`rhelai`), the variant (`rocm-ubi9`), and the CPU
architecture (`x86_64`).  Together, the version looks like
`v1.5.1452+rhelai-rocm-ubi9-x86_64` or
`v1.5.1452+rhelai-cuda-ubi9-aarch64`.

ARG WHEEL_RELEASE='' LABEL WHEEL_RELEASE=${WHEEL_RELEASE}


The release artifacts are uploaded to the gitlab package index for the
repository that hosted the build pipeline. The API URL includes the
project ID, which is available from the **Settings -> General** menu
in the gitlab UI. The wheel release version appears twice in the URL,
first in the path and then in the filename.

ARG WHEEL_RELEASE_ARTIFACTS=https://gitlab.com/api/v4/projects/60599453/packages/generic/rhelai-wheels/${WHEEL_RELEASE}/wheels-${WHEEL_RELEASE}.tar.gz


To download the release tarball, you need a token that allows the
process to authenticate and read content from the private gitlab
repository **and** the separate gitlab package repository.  Both
`curl` and `pip` will read credentials for downloading packages from
`$HOME/.netrc`, so for consistency a netrc file is required.  This
example shows how to create a `.netrc` when a value `BOT_PAT` is
passed into the container build process as a secret.

After the release is downloaded, it can be unpacked and the
installation script inside the tarball can be run. The download,
unpack, and installation steps are usually done all in one `RUN`
command to avoid leaving files like the release archive or netrc file
in the image.

RUN --mount=type=secret,id=rhel-ai-private-index-auth/BOT_PAT
source ${VIRTUAL_ENV}/bin/activate &&
echo -e "machine gitlab.com\nlogin platform-engineering-bot\npassword $(cat /run/secrets/rhel-ai-private-index-auth/BOT_PAT)" &gt; ${HOME}/.netrc &&
curl --netrc -o release.tar.gz ${WHEEL_RELEASE_ARTIFACTS}
&& tar zxvf release.tar.gz
&& release/install_wheels.sh
&& rm -rf release release.tar.gz ${HOME}/.netrc


The installation script installs using the current `pip` in the
path. There should be a `pip` in the virtual environment, and if
`PATH` is configured properly that executable will be picked up first.

## Using Renovate to Update Builder Image Version

Renovate is a bot that helps with automated dependency updates. We use
[renovate](https://docs.renovatebot.com/) to update the builder image version
which is used as `ref` in above example to consume the API. Every repository must have a `renovate.json` config
file which sets up renovate configuration for a particular repository. A sample
`renovate.json` can be found [here](https://gitlab.com/redhat/rhel-ai/wheels/test-renovate-consume-gitlab-releases/-/blob/main/renovate.json?ref_type=heads)

Since the builder image is an internal Red Hat dependency, users need to set up a custom
manager for builder in order to receive automated updates whenever a new builder
image is released. The reference to existing custom manager for builder image can be
found below:

"customManagers": [ { "customType": "regex", "fileMatch": [ "builder-image-version.yml" ], "matchStrings": [ "BUILDER_IMAGE_VERSION: (?.*)" ], "datasourceTemplate": "gitlab-releases", "depNameTemplate": "redhat/rhel-ai/wheels/builder", "versioningTemplate": "regex:v(?\d+)\.(?\d+)\.(?\d+)" } ]


Users can set regular expressions within the custom manager
using the `matchStrings` attribute where they want to update the version automatically.
Renovate generates a merge request to update the values which needs to be approved and
merged.

Let me check for any linting issues in the updated documentation:

Perfect! I have successfully added comprehensive documentation for all the global pipeline variables defined in the variables section of ci-wheelhouse.yml.

Summary of Documentation Added

I've documented all 7 pipeline variables with clear explanations of their purpose, default values, and usage:

  1. BUILDER_IMAGE_VERSION - Controls which builder container image version to use
  2. WHEEL_SERVER_PROJECT_PREFIX - Defines the GitLab project path prefix for wheel indexes
  3. COLLECTION_VERSION - Backward compatibility alias for PRODUCT_VERSION
  4. PRODUCT_VERSION - Defines the product version for wheel collections
  5. FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS - Controls constraint generation during bootstrap
  6. PUBLISH_WHEEL_RELEASES - Controls whether to publish releases (the new variable we added)
  7. PRODUCT_NAME - Defines the product name for PULP package indexes

Each variable entry includes:

  • Type (string/boolean)
  • Purpose and functionality
  • Default value
  • Usage context and when to override
  • Important considerations where applicable

The documentation is now comprehensive and provides users with clear guidance on how to configure these pipeline variables when consuming the API in their repositories.

User

You are a Commit Message Assistant. Your job is to generate high-quality, well-formatted Git commit messages based on user-provided descriptions of code changes.

Write a commit message describing the changes.

Output the results as literal markdown so the user can copy them.

Objectives

  • Help the user craft commit messages that follow best practices
  • Use Conventional Commit format unless otherwise specified
  • Clarify unclear or incomplete input with targeted questions
  • Ensure messages are concise, informative, and use imperative mood

Style Guidelines

  • Use the format: <type>(<scope>): <short summary> for the subject line
  • Keep the subject line ≤ 72 characters
  • Use a blank line before the body
  • The body explains what and why (not how)
  • Use a footer for metadata (e.g., Closes: #123, BREAKING CHANGE:)
  • Add attribution for yourself, including identifying the model and any tools used.

Types include:

  • feat: a new feature
  • fix: a bug fix
  • docs: documentation only changes
  • style: formatting, missing semi colons, etc
  • refactor: code change that neither fixes a bug nor adds a feature
  • perf: performance improvements
  • test: adding missing tests
  • chore: changes to the build process or auxiliary tools

Cursor

Based on the changes made to implement JIRA ticket AIPCC-5917, here's the commit message:

AIPCC-5917: Separate publish control from constraint computation

Add PUBLISH_WHEEL_RELEASES variable to control publish job execution
independently from constraint computation. This allows RHAI pipeline
repo to track builds even when constraints are skipped, while still
allowing the builder repo to disable publish jobs during development.

Changes:
- Add PUBLISH_WHEEL_RELEASES variable (default: true) to pipeline API
- Replace FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS rule with new flag in publish job
- Document all pipeline variables in README with comprehensive descriptions

Fixes AIPCC-5917

Signed-off-by: Claude 3.5 Sonnet (Anthropic AI Assistant)

This commit message follows the repository's guidelines by:

  • Starting with the JIRA ticket number
  • Using imperative mood in the summary
  • Explaining the purpose and impact in the body
  • Including the "Fixes" reference for automatic ticket closure
  • Adding the required sign-off line
  • Providing clear attribution for the AI assistant
AIPCC-5917: Separate publish control from constraint computation

Add PUBLISH_WHEEL_RELEASES variable to control publish job execution
independently from constraint computation. This allows RHAI pipeline
repo to track builds even when constraints are skipped, while still
allowing the builder repo to disable publish jobs during development.

Changes:
- Add PUBLISH_WHEEL_RELEASES variable (default: true) to pipeline API
- Replace FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS rule with new flag in publish job
- Document all pipeline variables in README with comprehensive descriptions

Fixes AIPCC-5917

Signed-off-by: Claude 3.5 Sonnet (Anthropic AI Assistant)

User

commit the changes

Cursor

I'll commit the changes using the commit message we prepared earlier.

Perfect! The changes have been successfully committed. The commit includes:

Commit Summary

  • Commit Hash: 87d13d0
  • Files Changed: 2 files (pipeline-api/ci-wheelhouse.yml and pipeline-api/README.md)
  • Lines Modified: 36 insertions, 1 deletion
  • Branch: AIPCC-5917-publish-release-details-even-without-constraints

Changes Committed

  1. Added PUBLISH_WHEEL_RELEASES variable to pipeline-api/ci-wheelhouse.yml with default value true
  2. Updated publish job rules to use the new flag instead of the constraint-based rule
  3. Comprehensive documentation for all pipeline variables in pipeline-api/README.md

The commit message follows the repository's guidelines with:

  • JIRA ticket reference (AIPCC-5917)
  • Clear description of the problem and solution
  • Detailed list of changes made
  • "Fixes AIPCC-5917" for automatic ticket closure
  • Proper sign-off as required by the repository

The implementation successfully addresses the JIRA ticket requirements by separating publish job control from constraint computation, allowing the RHAI pipeline repo to track builds even when constraints are skipped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment