Exported on 9/19/2025 at 15:52:43 EDT from Cursor (1.5.11)
User
I want to work on jira ticket @https://issues.redhat.com/browse/AIPCC-5335 . Start by looking at ci-wheelhouse.yml and .gitlab-test-jobs.yaml to see how the test bootstrap-and-onboard jobs in this repository are defined. Then add a new base job to the ci-wheelhouse.yml file using the variable-based parameterized jobs there as examples. Set the same variables for the new job that appear in the jobs in .gitlab-test-jobs.yaml.
Cursor
I'll help you work on JIRA ticket AIPCC-5335, which is about adding support for test build jobs to the builder job API. Let me start by examining the current structure of both files to understand how test jobs are defined and how to create a new base job.
Now I understand the structure. From analyzing both files, I can see:
From .gitlab-test-jobs.yaml:
- Test jobs extend the regular bootstrap-and-onboard jobs
- They use the
test-build-wheelsstage - Key variables that distinguish test jobs:
BOOTSTRAP_RUN_ONBOARD: "false"BOOTSTRAP_MODE: "full-parallel"
- They have specific rules using references to trigger conditions
- They depend on both the build-image job and the linter job
From ci-wheelhouse.yml:
- Uses parameterized job templates with
$[[ inputs.* ]]syntax - Has a base job structure with extends pattern
- Variables are defined using the input parameters
Let me now create a new base job template for test builds in the API file:
# This yaml file contains the CI jobs as an API that can be used by other repositories outside builder
spec:
inputs:
JOB_STAGE:
default: build
description: "Define the job stage that will run the bootstrap and onboard steps of wheel pipeline"
JOB_PREFIX:
default: ""
description: "Define a prefix for the job name"
type: string
COLLECTION:
description: "Define the collection name for which you want to build wheels"
type: string
VARIANT:
description: "Define the variant for which you want to build wheels"
options: ["cpu-ubi9", "cuda-ubi9", "rocm-ubi9", "spyre-ubi9", "tpu-ubi9"] # , "gaudi-ubi9"]
ENABLE_REPEATABLE_BUILD_MODE:
default: false
description: "Setting which enables repeatable build mode"
type: boolean
BOOTSTRAP_MODE:
description: "Setting for bootstrap mode: 'bootstrap --sdist-only', 'bootstrap --full-build', 'bootstrap-parallel'"
options: ["sdist-only", "full", "full-parallel"]
default: "sdist-only"
ARCH:
default: "x86_64"
description: "Define the CPU architecture to build wheels for"
options: ["aarch64", "x86_64", "ppc64le"]
BUILD_ON_ALL_PUSHES:
default: false
description: "Build for every merge, even if the collection has not changed."
type: boolean
ENABLE_NIGHTLY_BUILDS:
default: false
description: "Setting that enables nightly builds for a given collection"
type: boolean
ENABLE_TEST_JOBS:
default: false
description: "Setting that enables test build jobs for testing resolution and build of packages"
type: boolean
---
# ======================================================================== VARIABLES ===============================================================
variables:
# Set BUILDER_IMAGE_VERSION to some fake value which will be overridden by users when they invoke the API
BUILDER_IMAGE_VERSION: v0.0
WHEEL_SERVER_PROJECT_PREFIX: "${CI_PROJECT_NAMESPACE}/indexes"
# Set PRODUCT_VERSION as a global variable. The default should look like a real value
COLLECTION_VERSION: $PRODUCT_VERSION
PRODUCT_VERSION: "0.0"
# Used in the builder repo to let collections build multiple
# versions of a package. Disabling constraints output also disables
# release jobs.
FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS: false
# Set PRODUCT_NAME to rhoai. We need PRODUCT_NAME for customer facing package index hosted in PULP
PRODUCT_NAME: "rhoai"
# =========================================================================== API BASE JOBS ===================================================================
# Base job for release variables
.$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-release-variables-base-job:
variables:
PUBLISH_AUTH_HEADER: "PRIVATE-TOKEN: ${BOT_PAT}"
RELEASE_TARBALL: "wheels-${RELEASE_VERSION}.tar.gz"
RELEASE_VERSION_SUFFIX: "+$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]"
RELEASE_VERSION: "${PRODUCT_VERSION}.${CI_PIPELINE_IID}${RELEASE_VERSION_SUFFIX}"
GITLAB_PACKAGE_URL: "https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/packages/generic/$[[ inputs.COLLECTION ]]-wheels"
PUBLISH_URL: "${GITLAB_PACKAGE_URL}/${RELEASE_VERSION}"
# Base job for the api
.$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job:
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-release-variables-base-job
image: registry.gitlab.com/redhat/rhel-ai/wheels/builder/podman-stable:v5.5.0
timeout: 12h
services:
# No services section needed for pinp
interruptible: true
rules:
- if: '$CI_PROJECT_ROOT_NAMESPACE != "redhat"'
when: never
- if: $CI_COMMIT_TAG
when: never
- if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_REF_PROTECTED == "true"
changes:
- collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/requirements.txt
- collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/constraints.txt
- builder-image-version.yml
- if: $CI_PIPELINE_SOURCE == "push" && "$[[ inputs.BUILD_ON_ALL_PUSHES ]]" == "true" && $CI_COMMIT_REF_PROTECTED == "true"
- if: '$CI_PIPELINE_SOURCE == "schedule" && "$[[ inputs.ENABLE_NIGHTLY_BUILDS ]]" == "true"'
variables:
FF_SCRIPT_SECTIONS: "true"
COLLECTION: "$[[ inputs.COLLECTION ]]"
VARIANT: "$[[ inputs.VARIANT ]]"
ARCH: "$[[ inputs.ARCH ]]"
IMAGE_BASE: ${CI_REGISTRY}/redhat/rhel-ai/wheels/builder/builder-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]
IMAGE: ${IMAGE_BASE}:${BUILDER_IMAGE_VERSION}
CONTAINERFILE: Containerfile.$[[ inputs.VARIANT ]]
TOOL_SERVER_URL: "https://gitlab.com/api/v4/projects/61507476/packages/pypi/simple/"
OUTDIR: /mnt
JOB_OUTPUT_DIR: /builds/$CI_PROJECT_PATH/mnt
WHEEL_SERVER_PROJECT_PATH: "${WHEEL_SERVER_PROJECT_PREFIX}/$[[ inputs.COLLECTION ]]-${PRODUCT_VERSION}/$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]"
SDIST_SERVER_PROJECT_PATH: "redhat/rhel-ai/wheels/prefetch"
RUN_CLEAN_AND_UPLOAD: "false"
# These flags are not part of the formal API, but is used to
# control the jobs in the builder repository.
BOOTSTRAP_RUN_ONBOARD: "true"
BUILD_ON_ALL_PUSHES: "$[[ inputs.BUILD_ON_ALL_PUSHES ]]"
# We need to use non-standard BOOTSTRAP_MODE for some jobs in builder which are not triggered by API
BOOTSTRAP_MODE: $[[ inputs.BOOTSTRAP_MODE ]]
before_script:
# Dump our settings to the log
- env
- dnf install -y which ncurses
# https://docs.gitlab.com/ee/ci/docker/authenticate_registry.html
- podman login $CI_REGISTRY --username $CI_REGISTRY_USER --password "$CI_REGISTRY_PASSWORD"
# Install dependencies for downloading secure files
- dnf install -y which curl bash
# Share the test script with the container using a directory in /builds, following:
# https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#known-issues-with-docker-in-docker
- mkdir -p "${JOB_OUTPUT_DIR}"
- cp -rv collections "${JOB_OUTPUT_DIR}/"
# Generate .netrc dynamically and put in inside the container where we want it
- echo -e "machine gitlab.com\nlogin platform-engineering-bot\npassword ${BOT_PAT}" > ${JOB_OUTPUT_DIR}/.netrc
# Configure Podman storage
- mkdir -p /etc/containers
# create default overlay configuration. Embed storage.conf as HEREDOC.
# pipelines in other projects cannot access files in builder repo.
- |
cat > /etc/containers/storage.conf << EOF
[storage]
driver = "overlay"
runroot = "/var/run/containers/storage"
graphroot = "/var/lib/containers/storage"
[storage.options]
mount_program = ""
EOF
- mkdir -p /var/run/containers/storage
- mkdir -p /var/lib/containers/storage
# AWS (may) have an existing storage configuration that works with dind but not
# pinp. This causes pinp to fail with
# "OSError: [Errno 107] Transport endpoint is not connected:
# errors. Remove the contents of these directories to re-initialize the overlay
# storage.
- rm -rf /var/lib/containers/storage/*
- rm -rf /var/run/containers/storage/*
script:
- >
podman run \
-v "$JOB_OUTPUT_DIR:/mnt" \
-v ".:/collection-repository" \
-e "NETRC=/mnt/.netrc" \
-e "OUTDIR=$OUTDIR" \
-e "FROMAGER_VARIANT=$[[ inputs.VARIANT ]]" \
-e "COLLECTION=$[[ inputs.COLLECTION ]]" \
-e "PRODUCT_VERSION=$PRODUCT_VERSION" \
-e "ENABLE_REPEATABLE_BUILD_MODE=$[[ inputs.ENABLE_REPEATABLE_BUILD_MODE ]]" \
-e "BOOTSTRAP_MODE=$BOOTSTRAP_MODE" \
-e "WHEEL_SERVER_PROJECT_PATH=${WHEEL_SERVER_PROJECT_PATH}" \
-e "SDIST_SERVER_PROJECT_PATH=${SDIST_SERVER_PROJECT_PATH}" \
-e "PUBLISH_AUTH_HEADER=${PUBLISH_AUTH_HEADER}" \
-e "RELEASE_VERSION=${RELEASE_VERSION}" \
-e "RELEASE_VERSION_SUFFIX=${RELEASE_VERSION_SUFFIX}" \
-e "RELEASE_TARBALL=${RELEASE_TARBALL}" \
-e "PUBLISH_URL=${PUBLISH_URL}" \
-e "GITLAB_PACKAGE_URL=${GITLAB_PACKAGE_URL}" \
-e "RUN_CLEAN_AND_UPLOAD=${RUN_CLEAN_AND_UPLOAD}" \
-e "BOT_PAT=${BOT_PAT}" \
-e "BOT_USER=${BOT_USER}" \
-e "BOOTSTRAP_RUN_ONBOARD=${BOOTSTRAP_RUN_ONBOARD}" \
-e "REPOSITORY_URL=${CI_REPOSITORY_URL}" \
-e "FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS=${FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS}" \
-e "PRODUCT_NAME=$PRODUCT_NAME" \
${IMAGE} \
${API_SCRIPT}
# Development command - will be removed eventually when we will have pulp upload script running in builder
- >
if [ -f ./bin/upload_to_pulp.sh ] && [[ "$CI_JOB_NAME" == *"build-wheels" ]]; then
podman run \
-v "$JOB_OUTPUT_DIR:/mnt" \
-v ".:/collection-repository" \
-e "NETRC=/mnt/.netrc" \
-e "OUTDIR=$OUTDIR" \
-e "FROMAGER_VARIANT=$[[ inputs.VARIANT ]]" \
-e "COLLECTION=$[[ inputs.COLLECTION ]]" \
-e "PRODUCT_VERSION=$PRODUCT_VERSION" \
-e "ENABLE_REPEATABLE_BUILD_MODE=$[[ inputs.ENABLE_REPEATABLE_BUILD_MODE ]]" \
-e "BOOTSTRAP_MODE=$BOOTSTRAP_MODE" \
-e "WHEEL_SERVER_PROJECT_PATH=${WHEEL_SERVER_PROJECT_PATH}" \
-e "SDIST_SERVER_PROJECT_PATH=${SDIST_SERVER_PROJECT_PATH}" \
-e "PUBLISH_AUTH_HEADER=${PUBLISH_AUTH_HEADER}" \
-e "RELEASE_VERSION=${RELEASE_VERSION}" \
-e "RELEASE_VERSION_SUFFIX=${RELEASE_VERSION_SUFFIX}" \
-e "RELEASE_TARBALL=${RELEASE_TARBALL}" \
-e "PUBLISH_URL=${PUBLISH_URL}" \
-e "GITLAB_PACKAGE_URL=${GITLAB_PACKAGE_URL}" \
-e "RUN_CLEAN_AND_UPLOAD=${RUN_CLEAN_AND_UPLOAD}" \
-e "BOT_PAT=${BOT_PAT}" \
-e "BOT_USER=${BOT_USER}" \
-e "BOOTSTRAP_RUN_ONBOARD=${BOOTSTRAP_RUN_ONBOARD}" \
-e "REPOSITORY_URL=${CI_REPOSITORY_URL}" \
-e "FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS=${FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS}" \
-e "PULP_CERT_BASE64=${PULP_CERT_BASE64}" \
-e "PULP_KEY_BASE64=${PULP_KEY_BASE64}" \
-e "PRODUCT_NAME=$PRODUCT_NAME" \
${IMAGE} \
/collection-repository/bin/upload_to_pulp.sh
fi
artifacts:
paths:
- $JOB_OUTPUT_DIR/*.log
- $JOB_OUTPUT_DIR/computed-constraints.txt
- $JOB_OUTPUT_DIR/computed-requirements.txt
- $JOB_OUTPUT_DIR/work-dir/*/*.log
- $JOB_OUTPUT_DIR/work-dir/logs/
- $JOB_OUTPUT_DIR/work-dir/*.json
- $JOB_OUTPUT_DIR/work-dir/constraints.txt
- $JOB_OUTPUT_DIR/work-dir/build-sequence-summary.md
when: always
tags:
- aipcc-large-$[[ inputs.ARCH ]]
# =========================================================== BOOTSTRAP AND ONBOARD API JOB ================================================================================
$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-bootstrap-and-onboard:
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
stage: $[[ inputs.JOB_STAGE ]]
variables:
API_SCRIPT: /mirror-builder/pipeline-api/bootstrap_and_onboard.sh
# ==================================== BUILD WHEEL SEQUENCE API JOB ==============================================
$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-build-wheels:
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
stage: $[[ inputs.JOB_STAGE ]]
variables:
API_SCRIPT: /mirror-builder/pipeline-api/build_wheels.sh
RUN_CLEAN_AND_UPLOAD: "true"
needs:
- job: $[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-bootstrap-and-onboard
artifacts: true
# ==================================================== RELEASE API JOB ============================================
$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-release-tarball:
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
stage: $[[ inputs.JOB_STAGE ]]
variables:
API_SCRIPT: /mirror-builder/pipeline-api/make_release_tarball.sh
needs:
- job: $[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-build-wheels
artifacts: true
# ============================================ PUBLISH API JOB ====================================================
$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-publish-wheels:
image: "registry.gitlab.com/gitlab-org/release-cli:latest"
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-release-variables-base-job
stage: $[[ inputs.JOB_STAGE ]]
interruptible: true
rules:
- if: '$CI_PROJECT_ROOT_NAMESPACE != "redhat"'
when: never
- if: '$FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS == "true"'
when: never
- if: $CI_COMMIT_TAG
when: never
- if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_REF_PROTECTED == "true"
changes:
- collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/requirements.txt
- collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/constraints.txt
- builder-image-version.yml
- if: '$CI_PIPELINE_SOURCE == "schedule" && "$[[ inputs.ENABLE_NIGHTLY_BUILDS ]]" == "true"'
needs:
- job: $[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-release-tarball
artifacts: true
script:
- echo "running publish job $CI_JOB_NAME for $RELEASE_VERSION and $[[ inputs.ARCH ]] arch"
release:
# See https://docs.gitlab.com/ee/ci/yaml/#release for available properties
tag_name: "$RELEASE_VERSION"
description: /mnt/work-dir/build-sequence-summary.md
ref: "$CI_COMMIT_SHA"
assets:
links:
- name: "$RELEASE_TARBALL"
url: "$PUBLISH_URL/$RELEASE_TARBALL"
tags:
- aipcc
Now I'll add a test job template after the existing bootstrap-and-onboard job. Let me add it right after that section:
# This yaml file contains the CI jobs as an API that can be used by other repositories outside builder
spec:
inputs:
JOB_STAGE:
default: build
description: "Define the job stage that will run the bootstrap and onboard steps of wheel pipeline"
JOB_PREFIX:
default: ""
description: "Define a prefix for the job name"
type: string
COLLECTION:
description: "Define the collection name for which you want to build wheels"
type: string
VARIANT:
description: "Define the variant for which you want to build wheels"
options: ["cpu-ubi9", "cuda-ubi9", "rocm-ubi9", "spyre-ubi9", "tpu-ubi9"] # , "gaudi-ubi9"]
ENABLE_REPEATABLE_BUILD_MODE:
default: false
description: "Setting which enables repeatable build mode"
type: boolean
BOOTSTRAP_MODE:
description: "Setting for bootstrap mode: 'bootstrap --sdist-only', 'bootstrap --full-build', 'bootstrap-parallel'"
options: ["sdist-only", "full", "full-parallel"]
default: "sdist-only"
ARCH:
default: "x86_64"
description: "Define the CPU architecture to build wheels for"
options: ["aarch64", "x86_64", "ppc64le"]
BUILD_ON_ALL_PUSHES:
default: false
description: "Build for every merge, even if the collection has not changed."
type: boolean
ENABLE_NIGHTLY_BUILDS:
default: false
description: "Setting that enables nightly builds for a given collection"
type: boolean
ENABLE_TEST_JOBS:
default: false
description: "Setting that enables test build jobs for testing resolution and build of packages"
type: boolean
---
# ======================================================================== VARIABLES ===============================================================
variables:
# Set BUILDER_IMAGE_VERSION to some fake value which will be overridden by users when they invoke the API
BUILDER_IMAGE_VERSION: v0.0
WHEEL_SERVER_PROJECT_PREFIX: "${CI_PROJECT_NAMESPACE}/indexes"
# Set PRODUCT_VERSION as a global variable. The default should look like a real value
COLLECTION_VERSION: $PRODUCT_VERSION
PRODUCT_VERSION: "0.0"
# Used in the builder repo to let collections build multiple
# versions of a package. Disabling constraints output also disables
# release jobs.
FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS: false
# Set PRODUCT_NAME to rhoai. We need PRODUCT_NAME for customer facing package index hosted in PULP
PRODUCT_NAME: "rhoai"
# =========================================================================== API BASE JOBS ===================================================================
# Base job for release variables
.$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-release-variables-base-job:
variables:
PUBLISH_AUTH_HEADER: "PRIVATE-TOKEN: ${BOT_PAT}"
RELEASE_TARBALL: "wheels-${RELEASE_VERSION}.tar.gz"
RELEASE_VERSION_SUFFIX: "+$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]"
RELEASE_VERSION: "${PRODUCT_VERSION}.${CI_PIPELINE_IID}${RELEASE_VERSION_SUFFIX}"
GITLAB_PACKAGE_URL: "https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/packages/generic/$[[ inputs.COLLECTION ]]-wheels"
PUBLISH_URL: "${GITLAB_PACKAGE_URL}/${RELEASE_VERSION}"
# Base job for the api
.$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job:
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-release-variables-base-job
image: registry.gitlab.com/redhat/rhel-ai/wheels/builder/podman-stable:v5.5.0
timeout: 12h
services:
# No services section needed for pinp
interruptible: true
rules:
- if: '$CI_PROJECT_ROOT_NAMESPACE != "redhat"'
when: never
- if: $CI_COMMIT_TAG
when: never
- if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_REF_PROTECTED == "true"
changes:
- collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/requirements.txt
- collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/constraints.txt
- builder-image-version.yml
- if: $CI_PIPELINE_SOURCE == "push" && "$[[ inputs.BUILD_ON_ALL_PUSHES ]]" == "true" && $CI_COMMIT_REF_PROTECTED == "true"
- if: '$CI_PIPELINE_SOURCE == "schedule" && "$[[ inputs.ENABLE_NIGHTLY_BUILDS ]]" == "true"'
variables:
FF_SCRIPT_SECTIONS: "true"
COLLECTION: "$[[ inputs.COLLECTION ]]"
VARIANT: "$[[ inputs.VARIANT ]]"
ARCH: "$[[ inputs.ARCH ]]"
IMAGE_BASE: ${CI_REGISTRY}/redhat/rhel-ai/wheels/builder/builder-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]
IMAGE: ${IMAGE_BASE}:${BUILDER_IMAGE_VERSION}
CONTAINERFILE: Containerfile.$[[ inputs.VARIANT ]]
TOOL_SERVER_URL: "https://gitlab.com/api/v4/projects/61507476/packages/pypi/simple/"
OUTDIR: /mnt
JOB_OUTPUT_DIR: /builds/$CI_PROJECT_PATH/mnt
WHEEL_SERVER_PROJECT_PATH: "${WHEEL_SERVER_PROJECT_PREFIX}/$[[ inputs.COLLECTION ]]-${PRODUCT_VERSION}/$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]"
SDIST_SERVER_PROJECT_PATH: "redhat/rhel-ai/wheels/prefetch"
RUN_CLEAN_AND_UPLOAD: "false"
# These flags are not part of the formal API, but is used to
# control the jobs in the builder repository.
BOOTSTRAP_RUN_ONBOARD: "true"
BUILD_ON_ALL_PUSHES: "$[[ inputs.BUILD_ON_ALL_PUSHES ]]"
# We need to use non-standard BOOTSTRAP_MODE for some jobs in builder which are not triggered by API
BOOTSTRAP_MODE: $[[ inputs.BOOTSTRAP_MODE ]]
before_script:
# Dump our settings to the log
- env
- dnf install -y which ncurses
# https://docs.gitlab.com/ee/ci/docker/authenticate_registry.html
- podman login $CI_REGISTRY --username $CI_REGISTRY_USER --password "$CI_REGISTRY_PASSWORD"
# Install dependencies for downloading secure files
- dnf install -y which curl bash
# Share the test script with the container using a directory in /builds, following:
# https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#known-issues-with-docker-in-docker
- mkdir -p "${JOB_OUTPUT_DIR}"
- cp -rv collections "${JOB_OUTPUT_DIR}/"
# Generate .netrc dynamically and put in inside the container where we want it
- echo -e "machine gitlab.com\nlogin platform-engineering-bot\npassword ${BOT_PAT}" > ${JOB_OUTPUT_DIR}/.netrc
# Configure Podman storage
- mkdir -p /etc/containers
# create default overlay configuration. Embed storage.conf as HEREDOC.
# pipelines in other projects cannot access files in builder repo.
- |
cat > /etc/containers/storage.conf << EOF
[storage]
driver = "overlay"
runroot = "/var/run/containers/storage"
graphroot = "/var/lib/containers/storage"
[storage.options]
mount_program = ""
EOF
- mkdir -p /var/run/containers/storage
- mkdir -p /var/lib/containers/storage
# AWS (may) have an existing storage configuration that works with dind but not
# pinp. This causes pinp to fail with
# "OSError: [Errno 107] Transport endpoint is not connected:
# errors. Remove the contents of these directories to re-initialize the overlay
# storage.
- rm -rf /var/lib/containers/storage/*
- rm -rf /var/run/containers/storage/*
script:
- >
podman run \
-v "$JOB_OUTPUT_DIR:/mnt" \
-v ".:/collection-repository" \
-e "NETRC=/mnt/.netrc" \
-e "OUTDIR=$OUTDIR" \
-e "FROMAGER_VARIANT=$[[ inputs.VARIANT ]]" \
-e "COLLECTION=$[[ inputs.COLLECTION ]]" \
-e "PRODUCT_VERSION=$PRODUCT_VERSION" \
-e "ENABLE_REPEATABLE_BUILD_MODE=$[[ inputs.ENABLE_REPEATABLE_BUILD_MODE ]]" \
-e "BOOTSTRAP_MODE=$BOOTSTRAP_MODE" \
-e "WHEEL_SERVER_PROJECT_PATH=${WHEEL_SERVER_PROJECT_PATH}" \
-e "SDIST_SERVER_PROJECT_PATH=${SDIST_SERVER_PROJECT_PATH}" \
-e "PUBLISH_AUTH_HEADER=${PUBLISH_AUTH_HEADER}" \
-e "RELEASE_VERSION=${RELEASE_VERSION}" \
-e "RELEASE_VERSION_SUFFIX=${RELEASE_VERSION_SUFFIX}" \
-e "RELEASE_TARBALL=${RELEASE_TARBALL}" \
-e "PUBLISH_URL=${PUBLISH_URL}" \
-e "GITLAB_PACKAGE_URL=${GITLAB_PACKAGE_URL}" \
-e "RUN_CLEAN_AND_UPLOAD=${RUN_CLEAN_AND_UPLOAD}" \
-e "BOT_PAT=${BOT_PAT}" \
-e "BOT_USER=${BOT_USER}" \
-e "BOOTSTRAP_RUN_ONBOARD=${BOOTSTRAP_RUN_ONBOARD}" \
-e "REPOSITORY_URL=${CI_REPOSITORY_URL}" \
-e "FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS=${FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS}" \
-e "PRODUCT_NAME=$PRODUCT_NAME" \
${IMAGE} \
${API_SCRIPT}
# Development command - will be removed eventually when we will have pulp upload script running in builder
- >
if [ -f ./bin/upload_to_pulp.sh ] && [[ "$CI_JOB_NAME" == *"build-wheels" ]]; then
podman run \
-v "$JOB_OUTPUT_DIR:/mnt" \
-v ".:/collection-repository" \
-e "NETRC=/mnt/.netrc" \
-e "OUTDIR=$OUTDIR" \
-e "FROMAGER_VARIANT=$[[ inputs.VARIANT ]]" \
-e "COLLECTION=$[[ inputs.COLLECTION ]]" \
-e "PRODUCT_VERSION=$PRODUCT_VERSION" \
-e "ENABLE_REPEATABLE_BUILD_MODE=$[[ inputs.ENABLE_REPEATABLE_BUILD_MODE ]]" \
-e "BOOTSTRAP_MODE=$BOOTSTRAP_MODE" \
-e "WHEEL_SERVER_PROJECT_PATH=${WHEEL_SERVER_PROJECT_PATH}" \
-e "SDIST_SERVER_PROJECT_PATH=${SDIST_SERVER_PROJECT_PATH}" \
-e "PUBLISH_AUTH_HEADER=${PUBLISH_AUTH_HEADER}" \
-e "RELEASE_VERSION=${RELEASE_VERSION}" \
-e "RELEASE_VERSION_SUFFIX=${RELEASE_VERSION_SUFFIX}" \
-e "RELEASE_TARBALL=${RELEASE_TARBALL}" \
-e "PUBLISH_URL=${PUBLISH_URL}" \
-e "GITLAB_PACKAGE_URL=${GITLAB_PACKAGE_URL}" \
-e "RUN_CLEAN_AND_UPLOAD=${RUN_CLEAN_AND_UPLOAD}" \
-e "BOT_PAT=${BOT_PAT}" \
-e "BOT_USER=${BOT_USER}" \
-e "BOOTSTRAP_RUN_ONBOARD=${BOOTSTRAP_RUN_ONBOARD}" \
-e "REPOSITORY_URL=${CI_REPOSITORY_URL}" \
-e "FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS=${FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS}" \
-e "PULP_CERT_BASE64=${PULP_CERT_BASE64}" \
-e "PULP_KEY_BASE64=${PULP_KEY_BASE64}" \
-e "PRODUCT_NAME=$PRODUCT_NAME" \
${IMAGE} \
/collection-repository/bin/upload_to_pulp.sh
fi
artifacts:
paths:
- $JOB_OUTPUT_DIR/*.log
- $JOB_OUTPUT_DIR/computed-constraints.txt
- $JOB_OUTPUT_DIR/computed-requirements.txt
- $JOB_OUTPUT_DIR/work-dir/*/*.log
- $JOB_OUTPUT_DIR/work-dir/logs/
- $JOB_OUTPUT_DIR/work-dir/*.json
- $JOB_OUTPUT_DIR/work-dir/constraints.txt
- $JOB_OUTPUT_DIR/work-dir/build-sequence-summary.md
when: always
tags:
- aipcc-large-$[[ inputs.ARCH ]]
# =========================================================== BOOTSTRAP AND ONBOARD API JOB ================================================================================
$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-bootstrap-and-onboard:
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
stage: $[[ inputs.JOB_STAGE ]]
variables:
API_SCRIPT: /mirror-builder/pipeline-api/bootstrap_and_onboard.sh
# =========================================================== TEST BOOTSTRAP AND ONBOARD API JOB ================================================================================
test-$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-bootstrap-and-onboard:
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
stage: test-build-wheels
rules:
- if: '$CI_PROJECT_ROOT_NAMESPACE != "redhat"'
when: never
- if: $CI_COMMIT_TAG
when: never
- if: '$[[ inputs.ENABLE_TEST_JOBS ]] == "true" && $CI_PIPELINE_SOURCE == "merge_request_event"'
variables:
API_SCRIPT: /mirror-builder/pipeline-api/bootstrap_and_onboard.sh
BOOTSTRAP_RUN_ONBOARD: "false"
BOOTSTRAP_MODE: "full-parallel"
# ==================================== BUILD WHEEL SEQUENCE API JOB ==============================================
$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-build-wheels:
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
stage: $[[ inputs.JOB_STAGE ]]
variables:
API_SCRIPT: /mirror-builder/pipeline-api/build_wheels.sh
RUN_CLEAN_AND_UPLOAD: "true"
needs:
- job: $[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-bootstrap-and-onboard
artifacts: true
# ==================================================== RELEASE API JOB ============================================
$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-release-tarball:
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
stage: $[[ inputs.JOB_STAGE ]]
variables:
API_SCRIPT: /mirror-builder/pipeline-api/make_release_tarball.sh
needs:
- job: $[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-build-wheels
artifacts: true
# ============================================ PUBLISH API JOB ====================================================
$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-publish-wheels:
image: "registry.gitlab.com/gitlab-org/release-cli:latest"
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-release-variables-base-job
stage: $[[ inputs.JOB_STAGE ]]
interruptible: true
rules:
- if: '$CI_PROJECT_ROOT_NAMESPACE != "redhat"'
when: never
- if: '$FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS == "true"'
when: never
- if: $CI_COMMIT_TAG
when: never
- if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_REF_PROTECTED == "true"
changes:
- collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/requirements.txt
- collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/constraints.txt
- builder-image-version.yml
- if: '$CI_PIPELINE_SOURCE == "schedule" && "$[[ inputs.ENABLE_NIGHTLY_BUILDS ]]" == "true"'
needs:
- job: $[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-release-tarball
artifacts: true
script:
- echo "running publish job $CI_JOB_NAME for $RELEASE_VERSION and $[[ inputs.ARCH ]] arch"
release:
# See https://docs.gitlab.com/ee/ci/yaml/#release for available properties
tag_name: "$RELEASE_VERSION"
description: /mnt/work-dir/build-sequence-summary.md
ref: "$CI_COMMIT_SHA"
assets:
links:
- name: "$RELEASE_TARBALL"
url: "$PUBLISH_URL/$RELEASE_TARBALL"
tags:
- aipcc
Perfect! I've successfully implemented the changes for JIRA ticket AIPCC-5335. Here's a summary of what I added to the pipeline-api/ci-wheelhouse.yml file:
I added a new input parameter ENABLE_TEST_JOBS to the API specification:
ENABLE_TEST_JOBS:
default: false
description: "Setting that enables test build jobs for testing resolution and build of packages"
type: booleanI added a new parameterized test job template that mirrors the structure from .gitlab-test-jobs.yaml:
test-$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-bootstrap-and-onboard:
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
stage: test-build-wheels
rules:
- if: '$CI_PROJECT_ROOT_NAMESPACE != "redhat"'
when: never
- if: $CI_COMMIT_TAG
when: never
- if: '$[[ inputs.ENABLE_TEST_JOBS ]] == "true" && $CI_PIPELINE_SOURCE == "merge_request_event"'
variables:
API_SCRIPT: /mirror-builder/pipeline-api/bootstrap_and_onboard.sh
BOOTSTRAP_RUN_ONBOARD: "false"
BOOTSTRAP_MODE: "full-parallel"- Flag-based activation: The test job only runs when
ENABLE_TEST_JOBSis set totrue - Same variables as existing test jobs: Uses the same
BOOTSTRAP_RUN_ONBOARD: "false"andBOOTSTRAP_MODE: "full-parallel"settings from the existing test jobs in.gitlab-test-jobs.yaml - Proper stage assignment: Uses the
test-build-wheelsstage like the existing test jobs - Merge request trigger: Only runs on merge request events when enabled
- Consistent naming: Follows the same parameterized naming pattern as other API jobs
This implementation provides pipeline owners with a consistent way to add test jobs to their repositories for testing resolution and build of packages before merging changes in their requirements file, exactly as requested in the JIRA ticket. The test jobs are not enabled by default but can be activated by setting the ENABLE_TEST_JOBS parameter to true when including the jobs API file.
ENABLE_TEST_JOBS:
default: false
description: "Setting that enables test build jobs for testing resolution and build of packages"
type: booleantest-$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-bootstrap-and-onboard:
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
stage: test-build-wheels
rules:
- if: '$CI_PROJECT_ROOT_NAMESPACE != "redhat"'
when: never
- if: $CI_COMMIT_TAG
when: never
- if: '$[[ inputs.ENABLE_TEST_JOBS ]] == "true" && $CI_PIPELINE_SOURCE == "merge_request_event"'
variables:
API_SCRIPT: /mirror-builder/pipeline-api/bootstrap_and_onboard.sh
BOOTSTRAP_RUN_ONBOARD: "false"
BOOTSTRAP_MODE: "full-parallel"User
Update the stage for the new base job to use the inputs variable for the job stage like in the other base jobs.
Cursor
You're absolutely right! I need to update the test job to use the $[[ inputs.JOB_STAGE ]] variable instead of hardcoding the stage name. Let me fix that:
# This yaml file contains the CI jobs as an API that can be used by other repositories outside builder
spec:
inputs:
JOB_STAGE:
default: build
description: "Define the job stage that will run the bootstrap and onboard steps of wheel pipeline"
JOB_PREFIX:
default: ""
description: "Define a prefix for the job name"
type: string
COLLECTION:
description: "Define the collection name for which you want to build wheels"
type: string
VARIANT:
description: "Define the variant for which you want to build wheels"
options: ["cpu-ubi9", "cuda-ubi9", "rocm-ubi9", "spyre-ubi9", "tpu-ubi9"] # , "gaudi-ubi9"]
ENABLE_REPEATABLE_BUILD_MODE:
default: false
description: "Setting which enables repeatable build mode"
type: boolean
BOOTSTRAP_MODE:
description: "Setting for bootstrap mode: 'bootstrap --sdist-only', 'bootstrap --full-build', 'bootstrap-parallel'"
options: ["sdist-only", "full", "full-parallel"]
default: "sdist-only"
ARCH:
default: "x86_64"
description: "Define the CPU architecture to build wheels for"
options: ["aarch64", "x86_64", "ppc64le"]
BUILD_ON_ALL_PUSHES:
default: false
description: "Build for every merge, even if the collection has not changed."
type: boolean
ENABLE_NIGHTLY_BUILDS:
default: false
description: "Setting that enables nightly builds for a given collection"
type: boolean
ENABLE_TEST_JOBS:
default: false
description: "Setting that enables test build jobs for testing resolution and build of packages"
type: boolean
---
# ======================================================================== VARIABLES ===============================================================
variables:
# Set BUILDER_IMAGE_VERSION to some fake value which will be overridden by users when they invoke the API
BUILDER_IMAGE_VERSION: v0.0
WHEEL_SERVER_PROJECT_PREFIX: "${CI_PROJECT_NAMESPACE}/indexes"
# Set PRODUCT_VERSION as a global variable. The default should look like a real value
COLLECTION_VERSION: $PRODUCT_VERSION
PRODUCT_VERSION: "0.0"
# Used in the builder repo to let collections build multiple
# versions of a package. Disabling constraints output also disables
# release jobs.
FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS: false
# Set PRODUCT_NAME to rhoai. We need PRODUCT_NAME for customer facing package index hosted in PULP
PRODUCT_NAME: "rhoai"
# =========================================================================== API BASE JOBS ===================================================================
# Base job for release variables
.$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-release-variables-base-job:
variables:
PUBLISH_AUTH_HEADER: "PRIVATE-TOKEN: ${BOT_PAT}"
RELEASE_TARBALL: "wheels-${RELEASE_VERSION}.tar.gz"
RELEASE_VERSION_SUFFIX: "+$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]"
RELEASE_VERSION: "${PRODUCT_VERSION}.${CI_PIPELINE_IID}${RELEASE_VERSION_SUFFIX}"
GITLAB_PACKAGE_URL: "https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/packages/generic/$[[ inputs.COLLECTION ]]-wheels"
PUBLISH_URL: "${GITLAB_PACKAGE_URL}/${RELEASE_VERSION}"
# Base job for the api
.$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job:
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-release-variables-base-job
image: registry.gitlab.com/redhat/rhel-ai/wheels/builder/podman-stable:v5.5.0
timeout: 12h
services:
# No services section needed for pinp
interruptible: true
rules:
- if: '$CI_PROJECT_ROOT_NAMESPACE != "redhat"'
when: never
- if: $CI_COMMIT_TAG
when: never
- if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_REF_PROTECTED == "true"
changes:
- collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/requirements.txt
- collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/constraints.txt
- builder-image-version.yml
- if: $CI_PIPELINE_SOURCE == "push" && "$[[ inputs.BUILD_ON_ALL_PUSHES ]]" == "true" && $CI_COMMIT_REF_PROTECTED == "true"
- if: '$CI_PIPELINE_SOURCE == "schedule" && "$[[ inputs.ENABLE_NIGHTLY_BUILDS ]]" == "true"'
variables:
FF_SCRIPT_SECTIONS: "true"
COLLECTION: "$[[ inputs.COLLECTION ]]"
VARIANT: "$[[ inputs.VARIANT ]]"
ARCH: "$[[ inputs.ARCH ]]"
IMAGE_BASE: ${CI_REGISTRY}/redhat/rhel-ai/wheels/builder/builder-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]
IMAGE: ${IMAGE_BASE}:${BUILDER_IMAGE_VERSION}
CONTAINERFILE: Containerfile.$[[ inputs.VARIANT ]]
TOOL_SERVER_URL: "https://gitlab.com/api/v4/projects/61507476/packages/pypi/simple/"
OUTDIR: /mnt
JOB_OUTPUT_DIR: /builds/$CI_PROJECT_PATH/mnt
WHEEL_SERVER_PROJECT_PATH: "${WHEEL_SERVER_PROJECT_PREFIX}/$[[ inputs.COLLECTION ]]-${PRODUCT_VERSION}/$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]"
SDIST_SERVER_PROJECT_PATH: "redhat/rhel-ai/wheels/prefetch"
RUN_CLEAN_AND_UPLOAD: "false"
# These flags are not part of the formal API, but is used to
# control the jobs in the builder repository.
BOOTSTRAP_RUN_ONBOARD: "true"
BUILD_ON_ALL_PUSHES: "$[[ inputs.BUILD_ON_ALL_PUSHES ]]"
# We need to use non-standard BOOTSTRAP_MODE for some jobs in builder which are not triggered by API
BOOTSTRAP_MODE: $[[ inputs.BOOTSTRAP_MODE ]]
before_script:
# Dump our settings to the log
- env
- dnf install -y which ncurses
# https://docs.gitlab.com/ee/ci/docker/authenticate_registry.html
- podman login $CI_REGISTRY --username $CI_REGISTRY_USER --password "$CI_REGISTRY_PASSWORD"
# Install dependencies for downloading secure files
- dnf install -y which curl bash
# Share the test script with the container using a directory in /builds, following:
# https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#known-issues-with-docker-in-docker
- mkdir -p "${JOB_OUTPUT_DIR}"
- cp -rv collections "${JOB_OUTPUT_DIR}/"
# Generate .netrc dynamically and put in inside the container where we want it
- echo -e "machine gitlab.com\nlogin platform-engineering-bot\npassword ${BOT_PAT}" > ${JOB_OUTPUT_DIR}/.netrc
# Configure Podman storage
- mkdir -p /etc/containers
# create default overlay configuration. Embed storage.conf as HEREDOC.
# pipelines in other projects cannot access files in builder repo.
- |
cat > /etc/containers/storage.conf << EOF
[storage]
driver = "overlay"
runroot = "/var/run/containers/storage"
graphroot = "/var/lib/containers/storage"
[storage.options]
mount_program = ""
EOF
- mkdir -p /var/run/containers/storage
- mkdir -p /var/lib/containers/storage
# AWS (may) have an existing storage configuration that works with dind but not
# pinp. This causes pinp to fail with
# "OSError: [Errno 107] Transport endpoint is not connected:
# errors. Remove the contents of these directories to re-initialize the overlay
# storage.
- rm -rf /var/lib/containers/storage/*
- rm -rf /var/run/containers/storage/*
script:
- >
podman run \
-v "$JOB_OUTPUT_DIR:/mnt" \
-v ".:/collection-repository" \
-e "NETRC=/mnt/.netrc" \
-e "OUTDIR=$OUTDIR" \
-e "FROMAGER_VARIANT=$[[ inputs.VARIANT ]]" \
-e "COLLECTION=$[[ inputs.COLLECTION ]]" \
-e "PRODUCT_VERSION=$PRODUCT_VERSION" \
-e "ENABLE_REPEATABLE_BUILD_MODE=$[[ inputs.ENABLE_REPEATABLE_BUILD_MODE ]]" \
-e "BOOTSTRAP_MODE=$BOOTSTRAP_MODE" \
-e "WHEEL_SERVER_PROJECT_PATH=${WHEEL_SERVER_PROJECT_PATH}" \
-e "SDIST_SERVER_PROJECT_PATH=${SDIST_SERVER_PROJECT_PATH}" \
-e "PUBLISH_AUTH_HEADER=${PUBLISH_AUTH_HEADER}" \
-e "RELEASE_VERSION=${RELEASE_VERSION}" \
-e "RELEASE_VERSION_SUFFIX=${RELEASE_VERSION_SUFFIX}" \
-e "RELEASE_TARBALL=${RELEASE_TARBALL}" \
-e "PUBLISH_URL=${PUBLISH_URL}" \
-e "GITLAB_PACKAGE_URL=${GITLAB_PACKAGE_URL}" \
-e "RUN_CLEAN_AND_UPLOAD=${RUN_CLEAN_AND_UPLOAD}" \
-e "BOT_PAT=${BOT_PAT}" \
-e "BOT_USER=${BOT_USER}" \
-e "BOOTSTRAP_RUN_ONBOARD=${BOOTSTRAP_RUN_ONBOARD}" \
-e "REPOSITORY_URL=${CI_REPOSITORY_URL}" \
-e "FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS=${FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS}" \
-e "PRODUCT_NAME=$PRODUCT_NAME" \
${IMAGE} \
${API_SCRIPT}
# Development command - will be removed eventually when we will have pulp upload script running in builder
- >
if [ -f ./bin/upload_to_pulp.sh ] && [[ "$CI_JOB_NAME" == *"build-wheels" ]]; then
podman run \
-v "$JOB_OUTPUT_DIR:/mnt" \
-v ".:/collection-repository" \
-e "NETRC=/mnt/.netrc" \
-e "OUTDIR=$OUTDIR" \
-e "FROMAGER_VARIANT=$[[ inputs.VARIANT ]]" \
-e "COLLECTION=$[[ inputs.COLLECTION ]]" \
-e "PRODUCT_VERSION=$PRODUCT_VERSION" \
-e "ENABLE_REPEATABLE_BUILD_MODE=$[[ inputs.ENABLE_REPEATABLE_BUILD_MODE ]]" \
-e "BOOTSTRAP_MODE=$BOOTSTRAP_MODE" \
-e "WHEEL_SERVER_PROJECT_PATH=${WHEEL_SERVER_PROJECT_PATH}" \
-e "SDIST_SERVER_PROJECT_PATH=${SDIST_SERVER_PROJECT_PATH}" \
-e "PUBLISH_AUTH_HEADER=${PUBLISH_AUTH_HEADER}" \
-e "RELEASE_VERSION=${RELEASE_VERSION}" \
-e "RELEASE_VERSION_SUFFIX=${RELEASE_VERSION_SUFFIX}" \
-e "RELEASE_TARBALL=${RELEASE_TARBALL}" \
-e "PUBLISH_URL=${PUBLISH_URL}" \
-e "GITLAB_PACKAGE_URL=${GITLAB_PACKAGE_URL}" \
-e "RUN_CLEAN_AND_UPLOAD=${RUN_CLEAN_AND_UPLOAD}" \
-e "BOT_PAT=${BOT_PAT}" \
-e "BOT_USER=${BOT_USER}" \
-e "BOOTSTRAP_RUN_ONBOARD=${BOOTSTRAP_RUN_ONBOARD}" \
-e "REPOSITORY_URL=${CI_REPOSITORY_URL}" \
-e "FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS=${FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS}" \
-e "PULP_CERT_BASE64=${PULP_CERT_BASE64}" \
-e "PULP_KEY_BASE64=${PULP_KEY_BASE64}" \
-e "PRODUCT_NAME=$PRODUCT_NAME" \
${IMAGE} \
/collection-repository/bin/upload_to_pulp.sh
fi
artifacts:
paths:
- $JOB_OUTPUT_DIR/*.log
- $JOB_OUTPUT_DIR/computed-constraints.txt
- $JOB_OUTPUT_DIR/computed-requirements.txt
- $JOB_OUTPUT_DIR/work-dir/*/*.log
- $JOB_OUTPUT_DIR/work-dir/logs/
- $JOB_OUTPUT_DIR/work-dir/*.json
- $JOB_OUTPUT_DIR/work-dir/constraints.txt
- $JOB_OUTPUT_DIR/work-dir/build-sequence-summary.md
when: always
tags:
- aipcc-large-$[[ inputs.ARCH ]]
# =========================================================== BOOTSTRAP AND ONBOARD API JOB ================================================================================
$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-bootstrap-and-onboard:
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
stage: $[[ inputs.JOB_STAGE ]]
variables:
API_SCRIPT: /mirror-builder/pipeline-api/bootstrap_and_onboard.sh
# =========================================================== TEST BOOTSTRAP AND ONBOARD API JOB ================================================================================
test-$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-bootstrap-and-onboard:
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
stage: $[[ inputs.JOB_STAGE ]]
rules:
- if: '$CI_PROJECT_ROOT_NAMESPACE != "redhat"'
when: never
- if: $CI_COMMIT_TAG
when: never
- if: '$[[ inputs.ENABLE_TEST_JOBS ]] == "true" && $CI_PIPELINE_SOURCE == "merge_request_event"'
variables:
API_SCRIPT: /mirror-builder/pipeline-api/bootstrap_and_onboard.sh
BOOTSTRAP_RUN_ONBOARD: "false"
BOOTSTRAP_MODE: "full-parallel"
# ==================================== BUILD WHEEL SEQUENCE API JOB ==============================================
$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-build-wheels:
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
stage: $[[ inputs.JOB_STAGE ]]
variables:
API_SCRIPT: /mirror-builder/pipeline-api/build_wheels.sh
RUN_CLEAN_AND_UPLOAD: "true"
needs:
- job: $[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-bootstrap-and-onboard
artifacts: true
# ==================================================== RELEASE API JOB ============================================
$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-release-tarball:
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
stage: $[[ inputs.JOB_STAGE ]]
variables:
API_SCRIPT: /mirror-builder/pipeline-api/make_release_tarball.sh
needs:
- job: $[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-build-wheels
artifacts: true
# ============================================ PUBLISH API JOB ====================================================
$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-publish-wheels:
image: "registry.gitlab.com/gitlab-org/release-cli:latest"
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-release-variables-base-job
stage: $[[ inputs.JOB_STAGE ]]
interruptible: true
rules:
- if: '$CI_PROJECT_ROOT_NAMESPACE != "redhat"'
when: never
- if: '$FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS == "true"'
when: never
- if: $CI_COMMIT_TAG
when: never
- if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_REF_PROTECTED == "true"
changes:
- collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/requirements.txt
- collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/constraints.txt
- builder-image-version.yml
- if: '$CI_PIPELINE_SOURCE == "schedule" && "$[[ inputs.ENABLE_NIGHTLY_BUILDS ]]" == "true"'
needs:
- job: $[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-release-tarball
artifacts: true
script:
- echo "running publish job $CI_JOB_NAME for $RELEASE_VERSION and $[[ inputs.ARCH ]] arch"
release:
# See https://docs.gitlab.com/ee/ci/yaml/#release for available properties
tag_name: "$RELEASE_VERSION"
description: /mnt/work-dir/build-sequence-summary.md
ref: "$CI_COMMIT_SHA"
assets:
links:
- name: "$RELEASE_TARBALL"
url: "$PUBLISH_URL/$RELEASE_TARBALL"
tags:
- aipcc
Perfect! I've updated the test job to use $[[ inputs.JOB_STAGE ]] instead of the hardcoded test-build-wheels stage. This makes the test job consistent with the other base jobs in the API, allowing users to specify which stage the test job should run in when they include the API file.
Now the test job will use the same stage as specified by the JOB_STAGE input parameter, which defaults to build but can be customized by the pipeline owners when they use the API.
Great! The change has been made successfully and there are no linting errors. The test job now properly uses the parameterized stage input like all the other base jobs in the API.
User
The wheel-pipeline-api-base-job has rules for running on a push event only when some files have changed. Add those rules to the merge request rule in the new base job.
Cursor
I need to examine the rules in the wheel-pipeline-api-base-job to understand what file changes trigger it, then incorporate those into the test job's merge request rule.
Now I can see the file change rules. I need to update the test job to include the same file changes logic for merge request events. Let me update the rules:
# This yaml file contains the CI jobs as an API that can be used by other repositories outside builder
spec:
inputs:
JOB_STAGE:
default: build
description: "Define the job stage that will run the bootstrap and onboard steps of wheel pipeline"
JOB_PREFIX:
default: ""
description: "Define a prefix for the job name"
type: string
COLLECTION:
description: "Define the collection name for which you want to build wheels"
type: string
VARIANT:
description: "Define the variant for which you want to build wheels"
options: ["cpu-ubi9", "cuda-ubi9", "rocm-ubi9", "spyre-ubi9", "tpu-ubi9"] # , "gaudi-ubi9"]
ENABLE_REPEATABLE_BUILD_MODE:
default: false
description: "Setting which enables repeatable build mode"
type: boolean
BOOTSTRAP_MODE:
description: "Setting for bootstrap mode: 'bootstrap --sdist-only', 'bootstrap --full-build', 'bootstrap-parallel'"
options: ["sdist-only", "full", "full-parallel"]
default: "sdist-only"
ARCH:
default: "x86_64"
description: "Define the CPU architecture to build wheels for"
options: ["aarch64", "x86_64", "ppc64le"]
BUILD_ON_ALL_PUSHES:
default: false
description: "Build for every merge, even if the collection has not changed."
type: boolean
ENABLE_NIGHTLY_BUILDS:
default: false
description: "Setting that enables nightly builds for a given collection"
type: boolean
ENABLE_TEST_JOBS:
default: false
description: "Setting that enables test build jobs for testing resolution and build of packages"
type: boolean
---
# ======================================================================== VARIABLES ===============================================================
variables:
# Set BUILDER_IMAGE_VERSION to some fake value which will be overridden by users when they invoke the API
BUILDER_IMAGE_VERSION: v0.0
WHEEL_SERVER_PROJECT_PREFIX: "${CI_PROJECT_NAMESPACE}/indexes"
# Set PRODUCT_VERSION as a global variable. The default should look like a real value
COLLECTION_VERSION: $PRODUCT_VERSION
PRODUCT_VERSION: "0.0"
# Used in the builder repo to let collections build multiple
# versions of a package. Disabling constraints output also disables
# release jobs.
FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS: false
# Set PRODUCT_NAME to rhoai. We need PRODUCT_NAME for customer facing package index hosted in PULP
PRODUCT_NAME: "rhoai"
# =========================================================================== API BASE JOBS ===================================================================
# Base job for release variables
.$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-release-variables-base-job:
variables:
PUBLISH_AUTH_HEADER: "PRIVATE-TOKEN: ${BOT_PAT}"
RELEASE_TARBALL: "wheels-${RELEASE_VERSION}.tar.gz"
RELEASE_VERSION_SUFFIX: "+$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]"
RELEASE_VERSION: "${PRODUCT_VERSION}.${CI_PIPELINE_IID}${RELEASE_VERSION_SUFFIX}"
GITLAB_PACKAGE_URL: "https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/packages/generic/$[[ inputs.COLLECTION ]]-wheels"
PUBLISH_URL: "${GITLAB_PACKAGE_URL}/${RELEASE_VERSION}"
# Base job for the api
.$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job:
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-release-variables-base-job
image: registry.gitlab.com/redhat/rhel-ai/wheels/builder/podman-stable:v5.5.0
timeout: 12h
services:
# No services section needed for pinp
interruptible: true
rules:
- if: '$CI_PROJECT_ROOT_NAMESPACE != "redhat"'
when: never
- if: $CI_COMMIT_TAG
when: never
- if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_REF_PROTECTED == "true"
changes:
- collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/requirements.txt
- collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/constraints.txt
- builder-image-version.yml
- if: $CI_PIPELINE_SOURCE == "push" && "$[[ inputs.BUILD_ON_ALL_PUSHES ]]" == "true" && $CI_COMMIT_REF_PROTECTED == "true"
- if: '$CI_PIPELINE_SOURCE == "schedule" && "$[[ inputs.ENABLE_NIGHTLY_BUILDS ]]" == "true"'
variables:
FF_SCRIPT_SECTIONS: "true"
COLLECTION: "$[[ inputs.COLLECTION ]]"
VARIANT: "$[[ inputs.VARIANT ]]"
ARCH: "$[[ inputs.ARCH ]]"
IMAGE_BASE: ${CI_REGISTRY}/redhat/rhel-ai/wheels/builder/builder-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]
IMAGE: ${IMAGE_BASE}:${BUILDER_IMAGE_VERSION}
CONTAINERFILE: Containerfile.$[[ inputs.VARIANT ]]
TOOL_SERVER_URL: "https://gitlab.com/api/v4/projects/61507476/packages/pypi/simple/"
OUTDIR: /mnt
JOB_OUTPUT_DIR: /builds/$CI_PROJECT_PATH/mnt
WHEEL_SERVER_PROJECT_PATH: "${WHEEL_SERVER_PROJECT_PREFIX}/$[[ inputs.COLLECTION ]]-${PRODUCT_VERSION}/$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]"
SDIST_SERVER_PROJECT_PATH: "redhat/rhel-ai/wheels/prefetch"
RUN_CLEAN_AND_UPLOAD: "false"
# These flags are not part of the formal API, but is used to
# control the jobs in the builder repository.
BOOTSTRAP_RUN_ONBOARD: "true"
BUILD_ON_ALL_PUSHES: "$[[ inputs.BUILD_ON_ALL_PUSHES ]]"
# We need to use non-standard BOOTSTRAP_MODE for some jobs in builder which are not triggered by API
BOOTSTRAP_MODE: $[[ inputs.BOOTSTRAP_MODE ]]
before_script:
# Dump our settings to the log
- env
- dnf install -y which ncurses
# https://docs.gitlab.com/ee/ci/docker/authenticate_registry.html
- podman login $CI_REGISTRY --username $CI_REGISTRY_USER --password "$CI_REGISTRY_PASSWORD"
# Install dependencies for downloading secure files
- dnf install -y which curl bash
# Share the test script with the container using a directory in /builds, following:
# https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#known-issues-with-docker-in-docker
- mkdir -p "${JOB_OUTPUT_DIR}"
- cp -rv collections "${JOB_OUTPUT_DIR}/"
# Generate .netrc dynamically and put in inside the container where we want it
- echo -e "machine gitlab.com\nlogin platform-engineering-bot\npassword ${BOT_PAT}" > ${JOB_OUTPUT_DIR}/.netrc
# Configure Podman storage
- mkdir -p /etc/containers
# create default overlay configuration. Embed storage.conf as HEREDOC.
# pipelines in other projects cannot access files in builder repo.
- |
cat > /etc/containers/storage.conf << EOF
[storage]
driver = "overlay"
runroot = "/var/run/containers/storage"
graphroot = "/var/lib/containers/storage"
[storage.options]
mount_program = ""
EOF
- mkdir -p /var/run/containers/storage
- mkdir -p /var/lib/containers/storage
# AWS (may) have an existing storage configuration that works with dind but not
# pinp. This causes pinp to fail with
# "OSError: [Errno 107] Transport endpoint is not connected:
# errors. Remove the contents of these directories to re-initialize the overlay
# storage.
- rm -rf /var/lib/containers/storage/*
- rm -rf /var/run/containers/storage/*
script:
- >
podman run \
-v "$JOB_OUTPUT_DIR:/mnt" \
-v ".:/collection-repository" \
-e "NETRC=/mnt/.netrc" \
-e "OUTDIR=$OUTDIR" \
-e "FROMAGER_VARIANT=$[[ inputs.VARIANT ]]" \
-e "COLLECTION=$[[ inputs.COLLECTION ]]" \
-e "PRODUCT_VERSION=$PRODUCT_VERSION" \
-e "ENABLE_REPEATABLE_BUILD_MODE=$[[ inputs.ENABLE_REPEATABLE_BUILD_MODE ]]" \
-e "BOOTSTRAP_MODE=$BOOTSTRAP_MODE" \
-e "WHEEL_SERVER_PROJECT_PATH=${WHEEL_SERVER_PROJECT_PATH}" \
-e "SDIST_SERVER_PROJECT_PATH=${SDIST_SERVER_PROJECT_PATH}" \
-e "PUBLISH_AUTH_HEADER=${PUBLISH_AUTH_HEADER}" \
-e "RELEASE_VERSION=${RELEASE_VERSION}" \
-e "RELEASE_VERSION_SUFFIX=${RELEASE_VERSION_SUFFIX}" \
-e "RELEASE_TARBALL=${RELEASE_TARBALL}" \
-e "PUBLISH_URL=${PUBLISH_URL}" \
-e "GITLAB_PACKAGE_URL=${GITLAB_PACKAGE_URL}" \
-e "RUN_CLEAN_AND_UPLOAD=${RUN_CLEAN_AND_UPLOAD}" \
-e "BOT_PAT=${BOT_PAT}" \
-e "BOT_USER=${BOT_USER}" \
-e "BOOTSTRAP_RUN_ONBOARD=${BOOTSTRAP_RUN_ONBOARD}" \
-e "REPOSITORY_URL=${CI_REPOSITORY_URL}" \
-e "FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS=${FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS}" \
-e "PRODUCT_NAME=$PRODUCT_NAME" \
${IMAGE} \
${API_SCRIPT}
# Development command - will be removed eventually when we will have pulp upload script running in builder
- >
if [ -f ./bin/upload_to_pulp.sh ] && [[ "$CI_JOB_NAME" == *"build-wheels" ]]; then
podman run \
-v "$JOB_OUTPUT_DIR:/mnt" \
-v ".:/collection-repository" \
-e "NETRC=/mnt/.netrc" \
-e "OUTDIR=$OUTDIR" \
-e "FROMAGER_VARIANT=$[[ inputs.VARIANT ]]" \
-e "COLLECTION=$[[ inputs.COLLECTION ]]" \
-e "PRODUCT_VERSION=$PRODUCT_VERSION" \
-e "ENABLE_REPEATABLE_BUILD_MODE=$[[ inputs.ENABLE_REPEATABLE_BUILD_MODE ]]" \
-e "BOOTSTRAP_MODE=$BOOTSTRAP_MODE" \
-e "WHEEL_SERVER_PROJECT_PATH=${WHEEL_SERVER_PROJECT_PATH}" \
-e "SDIST_SERVER_PROJECT_PATH=${SDIST_SERVER_PROJECT_PATH}" \
-e "PUBLISH_AUTH_HEADER=${PUBLISH_AUTH_HEADER}" \
-e "RELEASE_VERSION=${RELEASE_VERSION}" \
-e "RELEASE_VERSION_SUFFIX=${RELEASE_VERSION_SUFFIX}" \
-e "RELEASE_TARBALL=${RELEASE_TARBALL}" \
-e "PUBLISH_URL=${PUBLISH_URL}" \
-e "GITLAB_PACKAGE_URL=${GITLAB_PACKAGE_URL}" \
-e "RUN_CLEAN_AND_UPLOAD=${RUN_CLEAN_AND_UPLOAD}" \
-e "BOT_PAT=${BOT_PAT}" \
-e "BOT_USER=${BOT_USER}" \
-e "BOOTSTRAP_RUN_ONBOARD=${BOOTSTRAP_RUN_ONBOARD}" \
-e "REPOSITORY_URL=${CI_REPOSITORY_URL}" \
-e "FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS=${FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS}" \
-e "PULP_CERT_BASE64=${PULP_CERT_BASE64}" \
-e "PULP_KEY_BASE64=${PULP_KEY_BASE64}" \
-e "PRODUCT_NAME=$PRODUCT_NAME" \
${IMAGE} \
/collection-repository/bin/upload_to_pulp.sh
fi
artifacts:
paths:
- $JOB_OUTPUT_DIR/*.log
- $JOB_OUTPUT_DIR/computed-constraints.txt
- $JOB_OUTPUT_DIR/computed-requirements.txt
- $JOB_OUTPUT_DIR/work-dir/*/*.log
- $JOB_OUTPUT_DIR/work-dir/logs/
- $JOB_OUTPUT_DIR/work-dir/*.json
- $JOB_OUTPUT_DIR/work-dir/constraints.txt
- $JOB_OUTPUT_DIR/work-dir/build-sequence-summary.md
when: always
tags:
- aipcc-large-$[[ inputs.ARCH ]]
# =========================================================== BOOTSTRAP AND ONBOARD API JOB ================================================================================
$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-bootstrap-and-onboard:
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
stage: $[[ inputs.JOB_STAGE ]]
variables:
API_SCRIPT: /mirror-builder/pipeline-api/bootstrap_and_onboard.sh
# =========================================================== TEST BOOTSTRAP AND ONBOARD API JOB ================================================================================
test-$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-bootstrap-and-onboard:
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
stage: $[[ inputs.JOB_STAGE ]]
rules:
- if: '$CI_PROJECT_ROOT_NAMESPACE != "redhat"'
when: never
- if: $CI_COMMIT_TAG
when: never
- if: '$[[ inputs.ENABLE_TEST_JOBS ]] == "true" && $CI_PIPELINE_SOURCE == "merge_request_event"'
changes:
- collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/requirements.txt
- collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/constraints.txt
- builder-image-version.yml
variables:
API_SCRIPT: /mirror-builder/pipeline-api/bootstrap_and_onboard.sh
BOOTSTRAP_RUN_ONBOARD: "false"
BOOTSTRAP_MODE: "full-parallel"
# ==================================== BUILD WHEEL SEQUENCE API JOB ==============================================
$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-build-wheels:
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
stage: $[[ inputs.JOB_STAGE ]]
variables:
API_SCRIPT: /mirror-builder/pipeline-api/build_wheels.sh
RUN_CLEAN_AND_UPLOAD: "true"
needs:
- job: $[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-bootstrap-and-onboard
artifacts: true
# ==================================================== RELEASE API JOB ============================================
$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-release-tarball:
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-api-base-job
stage: $[[ inputs.JOB_STAGE ]]
variables:
API_SCRIPT: /mirror-builder/pipeline-api/make_release_tarball.sh
needs:
- job: $[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-build-wheels
artifacts: true
# ============================================ PUBLISH API JOB ====================================================
$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-publish-wheels:
image: "registry.gitlab.com/gitlab-org/release-cli:latest"
extends:
- .$[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-wheel-pipeline-release-variables-base-job
stage: $[[ inputs.JOB_STAGE ]]
interruptible: true
rules:
- if: '$CI_PROJECT_ROOT_NAMESPACE != "redhat"'
when: never
- if: '$FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS == "true"'
when: never
- if: $CI_COMMIT_TAG
when: never
- if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_REF_PROTECTED == "true"
changes:
- collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/requirements.txt
- collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/constraints.txt
- builder-image-version.yml
- if: '$CI_PIPELINE_SOURCE == "schedule" && "$[[ inputs.ENABLE_NIGHTLY_BUILDS ]]" == "true"'
needs:
- job: $[[ inputs.JOB_PREFIX ]]$[[ inputs.COLLECTION ]]-$[[ inputs.VARIANT ]]-$[[ inputs.ARCH ]]-release-tarball
artifacts: true
script:
- echo "running publish job $CI_JOB_NAME for $RELEASE_VERSION and $[[ inputs.ARCH ]] arch"
release:
# See https://docs.gitlab.com/ee/ci/yaml/#release for available properties
tag_name: "$RELEASE_VERSION"
description: /mnt/work-dir/build-sequence-summary.md
ref: "$CI_COMMIT_SHA"
assets:
links:
- name: "$RELEASE_TARBALL"
url: "$PUBLISH_URL/$RELEASE_TARBALL"
tags:
- aipcc
Perfect! I've updated the test job rules to include the same file change detection as the main wheel-pipeline-api-base-job. Now the test job will only run on merge requests when:
ENABLE_TEST_JOBSis set totrue- The pipeline source is a merge request event
- AND one of these files has changed:
collections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/requirements.txtcollections/$[[ inputs.COLLECTION ]]/$[[ inputs.VARIANT ]]/constraints.txtbuilder-image-version.yml
This ensures that test jobs only run when there are actual changes to the collection requirements, constraints, or builder image version - the same logic used by the regular pipeline jobs for push events. This prevents unnecessary test runs when unrelated files are changed in merge requests.
Excellent! The changes have been applied successfully with no linting errors. The test job now has the proper file change detection rules that match the behavior of the main pipeline jobs.
User
Update .gitlab-ci.yaml to set the variable for the new flag to true for each instance where pipeline-api/ci-wheelhouse.yml is being included
Cursor
I need to examine the .gitlab-ci.yml file to find where pipeline-api/ci-wheelhouse.yml is being included and then add the ENABLE_TEST_JOBS: true variable to those instances.
Now I can see all the instances where pipeline-api/ci-wheelhouse.yml is included. I need to add ENABLE_TEST_JOBS: true to each of these includes. Let me update them one by one:
---
stages:
- lint
- test
- build-images
- test-build-wheels
- update-cache
- release
include:
- remote: https://gitlab.com/platform-engineering-org/gitlab-ci/-/raw/main/templates/build-image.gitlab-ci.yml
- local: .gitlab-triggers.yaml
# Use the bootstrap job API from ci-wheelhouse.yml.
- local: "gitlab-ci/images.yml"
inputs:
VARIANT: "cpu-ubi9"
ARCH: "aarch64"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "aarch64"
COLLECTION: "api-test"
VARIANT: "cpu-ubi9"
JOB_STAGE: "test-build-wheels"
BOOTSTRAP_MODE: "full-parallel"
ENABLE_TEST_JOBS: true
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "aarch64"
COLLECTION: "non-accelerated"
VARIANT: "cpu-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
ENABLE_TEST_JOBS: true
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "aarch64"
COLLECTION: "accelerated"
VARIANT: "cpu-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
ENABLE_TEST_JOBS: true
- local: "gitlab-ci/images.yml"
inputs:
VARIANT: "cpu-ubi9"
ARCH: "ppc64le"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "ppc64le"
COLLECTION: "api-test"
VARIANT: "cpu-ubi9"
JOB_STAGE: "test-build-wheels"
BOOTSTRAP_MODE: "full-parallel"
ENABLE_TEST_JOBS: true
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "ppc64le"
COLLECTION: "non-accelerated"
VARIANT: "cpu-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
ENABLE_TEST_JOBS: true
- local: "gitlab-ci/images.yml"
inputs:
VARIANT: "cpu-ubi9"
ARCH: "x86_64"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "x86_64"
COLLECTION: "api-test"
VARIANT: "cpu-ubi9"
JOB_STAGE: "test-build-wheels"
BUILD_ON_ALL_PUSHES: true
BOOTSTRAP_MODE: "full-parallel"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "x86_64"
COLLECTION: "non-accelerated"
VARIANT: "cpu-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "x86_64"
COLLECTION: "accelerated"
VARIANT: "cpu-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
- local: "gitlab-ci/images.yml"
inputs:
VARIANT: "cuda-ubi9"
ARCH: "aarch64"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "aarch64"
COLLECTION: "accelerated"
VARIANT: "cuda-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
- local: "gitlab-ci/images.yml"
inputs:
VARIANT: "cuda-ubi9"
ARCH: "x86_64"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "x86_64"
COLLECTION: "accelerated"
VARIANT: "cuda-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
# - local: "gitlab-ci/images.yml"
# inputs:
# VARIANT: "gaudi-ubi9"
# ARCH: "x86_64"
# - local: "pipeline-api/ci-wheelhouse.yml"
# inputs:
# ARCH: "x86_64"
# COLLECTION: "accelerated"
# VARIANT: "gaudi-ubi9"
# JOB_STAGE: "update-cache"
# BUILD_ON_ALL_PUSHES: true
- local: "gitlab-ci/images.yml"
inputs:
VARIANT: "rocm-ubi9"
ARCH: "x86_64"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "x86_64"
COLLECTION: "accelerated"
VARIANT: "rocm-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
- local: "gitlab-ci/images.yml"
inputs:
VARIANT: "spyre-ubi9"
ARCH: "x86_64"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "x86_64"
COLLECTION: "accelerated"
VARIANT: "spyre-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
- local: "gitlab-ci/images.yml"
inputs:
VARIANT: "tpu-ubi9"
ARCH: "x86_64"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "x86_64"
COLLECTION: "accelerated"
VARIANT: "tpu-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
# Add test jobs for the collections defined above. The jobs are generated
# automatically by running `make linter` and checking in the generated
# .gitlab-test-jobs.yaml file.
- local: .gitlab-test-jobs.yaml
# !!! This include MUST be last to override the value from ci-wheelhouse.yml !!!
- local: builder-image-version.yml
default:
tags:
- aipcc-small-x86_64
# Override rules for bootstrap API job present in ci-wheelhouse.yml
# These rules will make it run in pre-merge pipeline only for testing purposes
# Ideally, the bootstrap API job has to be run in post merge pipeline
# Remove this override after testing is complete and we know that job is successful
api-test-cpu-ubi9-x86_64-bootstrap-and-onboard:
stage: test-build-wheels
variables:
# re-enable constraints file production for test job
FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS: false
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: build-image-cpu-ubi9-x86_64
- job: linter
api-test-cpu-ubi9-ppc64le-bootstrap-and-onboard:
stage: test-build-wheels
variables:
# re-enable constraints file production for test job
FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS: false
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: build-image-cpu-ubi9-ppc64le
- job: linter
api-test-cpu-ubi9-aarch64-bootstrap-and-onboard:
stage: test-build-wheels
variables:
# re-enable constraints file production for test job
FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS: false
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: build-image-cpu-ubi9-aarch64
- job: linter
# Override rules for bootstrap API job present in ci-wheelhouse.yml
# These rules will make it run in pre-merge pipeline only for testing purposes
# Ideally, the build wheel sequence API job has to be run in post merge pipeline and is dependent on success of bootstrap API
# Remove this override after testing is complete and we know that job is successful
api-test-cpu-ubi9-x86_64-build-wheels:
stage: test-build-wheels
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: api-test-cpu-ubi9-x86_64-bootstrap-and-onboard
artifacts: true
api-test-cpu-ubi9-aarch64-build-wheels:
stage: test-build-wheels
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: api-test-cpu-ubi9-aarch64-bootstrap-and-onboard
artifacts: true
api-test-cpu-ubi9-ppc64le-build-wheels:
stage: test-build-wheels
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: api-test-cpu-ubi9-ppc64le-bootstrap-and-onboard
artifacts: true
# Override rules for release tarball API job present in ci-wheelhouse.yml
# These rules will make it run in pre-merge pipeline only for testing purposes
# Ideally, the release tarball API job has to be run in post merge pipeline and is dependent on success of build wheel sequence API
# Remove this override after testing is complete and we know that job is successful
api-test-cpu-ubi9-x86_64-release-tarball:
stage: test-build-wheels
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: api-test-cpu-ubi9-x86_64-build-wheels
artifacts: true
api-test-cpu-ubi9-aarch64-release-tarball:
stage: test-build-wheels
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: api-test-cpu-ubi9-aarch64-build-wheels
artifacts: true
api-test-cpu-ubi9-ppc64le-release-tarball:
stage: test-build-wheels
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: api-test-cpu-ubi9-ppc64le-build-wheels
artifacts: true
# Disable the publish wheel jobs. These jobs use the release API of
# gitlab, so we don't need to test them every time we make a script
# change.
api-test-cpu-ubi9-x86_64-publish-wheels:
stage: test-build-wheels
rules:
- when: never
api-test-cpu-ubi9-aarch64-publish-wheels:
stage: test-build-wheels
rules:
- when: never
api-test-cpu-ubi9-ppc64le-publish-wheels:
stage: test-build-wheels
rules:
- when: never
non-accelerated-cpu-ubi9-x86_64-publish-wheels:
rules:
- when: never
non-accelerated-cpu-ubi9-aarch64-publish-wheels:
rules:
- when: never
non-accelerated-cpu-ubi9-ppc64le-publish-wheels:
rules:
- when: never
accelerated-cpu-ubi9-aarch64-publish-wheels:
rules:
- when: never
accelerated-cpu-ubi9-x86_64-publish-wheels:
rules:
- when: never
accelerated-cuda-ubi9-x86_64-publish-wheels:
rules:
- when: never
accelerated-cuda-ubi9-aarch64-publish-wheels:
rules:
- when: never
# accelerated-gaudi-ubi9-x86_64-publish-wheels:
# rules:
# - when: never
accelerated-rocm-ubi9-x86_64-publish-wheels:
rules:
- when: never
accelerated-spyre-ubi9-x86_64-publish-wheels:
rules:
- when: never
accelerated-tpu-ubi9-x86_64-publish-wheels:
rules:
- when: never
# ========================================= VARIABLES =========================================
variables:
TOOL_SERVER_URL: "https://gitlab.com/api/v4/projects/61507476/packages/pypi/simple/"
SECURE_FILES_DOWNLOAD_PATH: ./config
FROMAGER_NETWORK_ISOLATION: "1"
# The gitlab group that has the wheel servers to use as a cache for
# test jobs. Usually this points to a group for RHEL AI indexes.
# Wheel indexes are now arch specific, and the default arch is x86_64
ARCH: "x86_64"
# We change the prefix value since it is overrided for API but we also don't want test scripts to break.
# Thus, $COLLECTION is now shifted to WHEEL_SERVER_PROJECT_PATH which was part of prefix before.
WHEEL_SERVER_PROJECT_PREFIX: "redhat/rhel-ai/core/wheels"
WHEEL_SERVER_PROJECT_PATH: "${WHEEL_SERVER_PROJECT_PREFIX}/${COLLECTION}/${VARIANT}-${ARCH}"
# Set the SDIST_INDEX_PATH for test jobs in builder repository only
SDIST_SERVER_PROJECT_PATH: "redhat/rhel-ai/core/wheels/upstream-sdists"
PYTHON_VERSION: "3.12"
# Test collections in this repo do not produce constraint.txt files
# for a single installable set of packages.
FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS: true
# Builder cache product version
# RHEL 9.4: 0.0
# RHEL 9.6: 0.0-el9.6
PRODUCT_VERSION: "0.0-el9.6"
# ===================================== TRIGGER RULES =========================================
# Run jobs to build collections for testing
.trigger-collection-merge-request:
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
# Shared image content
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- container-install/** # files not named for variants
- repos/** # files not named for variants
- overrides/**/* # any fromager inputs
- package_plugins/** # any fromager plugins
- requirements.txt # new fromager version
- collections/global-constraints.txt # constraints for all builds
# Variant-specific image content
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- Containerfile.${VARIANT}
- 'build-args/${VARIANT}.conf'
- collections/${COLLECTION}/${VARIANT}/*
- if: '$CI_MERGE_REQUEST_LABELS =~ /force-test/'
# Define rules to trigger testing of pipeline api. We want the jobs to be triggered in post merge pipeline
.trigger-api-test:
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- pipeline-api/*
- collections/global-constraints.txt
- collections/api-test/cpu-ubi9/*
# We want to test the onboarder script when there are changes to it.
# onboarder cli command is used as part of bootstrap_and_onboard.sh API script
- package_plugins/cli/onboarder.py
# get-project-id is used in all API scripts and thus we would want to test the python cli code
# when there are changes to it
- package_plugins/cli/get_project_id.py
- if: '$CI_MERGE_REQUEST_LABELS =~ /force-test/'
# =========================================== LINTER ==========================================
.base-linter:
stage: lint
# image: registry.fedoraproject.org/fedora:42
image: registry.gitlab.com/redhat/rhel-ai/wheels/builder/fedora:42
interruptible: true
rules:
- if: $CI_COMMIT_MESSAGE =~ /^Draft:/
when: never
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
before_script:
# install gcc and make for platforms which may need to build psutil from sdist
- dnf install -y python${PYTHON_VERSION} git make shellcheck gcc make
- make venv
tags:
- aipcc
# Run linter for Python and generated code in this repo.
linter:
extends: .base-linter
script:
- make regen && git diff --exit-code .
- make linter-core
# Run changelog linter to ensure changelogs are updated if needed.
changelog-linter:
extends: .base-linter
rules:
- if: '$CI_MERGE_REQUEST_LABELS =~ /no-changelog-needed/'
when: never
- !reference [.base-linter, rules]
script:
- make linter-changelog
# Run JIRA ticket linter
jira-ticket-linter:
extends: .base-linter
script:
- make linter-jira
# Run JIRA ticket linter
build-arg-linter:
extends: .base-linter
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- build-args/*
script:
- make linter-build-arg
# =========================================== TOOL SERVER UPDATE ==========================================
# Update tool server if requirements file changes.
update-tool-server:
stage: test
# image: registry.fedoraproject.org/fedora:42
image: registry.gitlab.com/redhat/rhel-ai/wheels/builder/fedora:42
interruptible: true
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- requirements.txt
- package_plugins/cli/tool_server_mirror.py
before_script:
- dnf install -y git yq make shellcheck python${PYTHON_VERSION}
- make venv
script:
# BOT_PAT env should be accessible
- "source .venv/bin/activate && tool-server-mirror"
tags:
- aipcc
# Base job for jobs that will run using the builder image in a container. We
# cannot just have the job defined to use the image because we want the image
# built in the CI pipeline and we can't refer to that one by the right tag in
# the job definition.
.podman-in-podman-job:
timeout: 12h
image: registry.gitlab.com/redhat/rhel-ai/wheels/builder/podman-stable:v5.5.0
variables:
MOUNT_POINT: /builds/$CI_PROJECT_PATH/mnt
IMAGE_BASE: ${CI_REGISTRY}/${CI_PROJECT_PATH}/builder-${VARIANT}-${ARCH}
IMAGE_TAG: ${BUILDER_IMAGE_VERSION}
IMAGE: ${IMAGE_BASE}:${IMAGE_TAG}
# podman specific configuration
BUILDAH_ISOLATION: chroot
STORAGE_DRIVER: overlay
# No services section needed for pinp
before_script:
# https://docs.gitlab.com/ee/ci/docker/authenticate_registry.html
- podman login $CI_REGISTRY --username $CI_REGISTRY_USER --password "$CI_REGISTRY_PASSWORD"
# Install dependencies for downloading secure files (if needed)
- dnf install -y which curl bash
# Share the test script with the container using a directory in /builds
- mkdir -p "${MOUNT_POINT}"
- cp -rv collections "${MOUNT_POINT}/"
- cp -rv test "${MOUNT_POINT}/"
- cp -rv pipeline-api "${MOUNT_POINT}/"
# Generate .netrc dynamically and put in inside the container where we want it
- mkdir -p ${SECURE_FILES_DOWNLOAD_PATH}
- echo -e "machine gitlab.com\nlogin platform-engineering-bot\npassword ${BOT_PAT}" > ${SECURE_FILES_DOWNLOAD_PATH}/.netrc
# Put the .netrc file in a location that will be mounted in the
# test container.
- cp ${SECURE_FILES_DOWNLOAD_PATH}/.netrc ${MOUNT_POINT}/.netrc
# Build a sort of release note text using the git history to make it
# easy to see what is in each release.
release-notes:
stage: release
# image: registry.fedoraproject.org/fedora:42
image: registry.gitlab.com/redhat/rhel-ai/wheels/builder/fedora:42
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- if: '$CI_COMMIT_TAG'
before_script:
- dnf install -y git yq make shellcheck python${PYTHON_VERSION}
script:
# https://docs.gitlab.com/ee/user/project/releases/release_cicd_examples.html#create-a-release-when-a-commit-is-merged-to-the-default-branch
- make release-notes
artifacts:
paths:
- release_notes.txt
needs:
- job: build-image-cpu-ubi9-x86_64
- job: build-image-cpu-ubi9-aarch64
- job: build-image-cpu-ubi9-ppc64le
optional: true
- job: build-image-cuda-ubi9-x86_64
- job: build-image-cuda-ubi9-aarch64
optional: true
# - job: build-image-gaudi-ubi9-x86_64
- job: build-image-rocm-ubi9-x86_64
- job: build-image-spyre-ubi9-x86_64
- job: build-image-tpu-ubi9-x86_64
tags:
- aipcc-small-x86_64
# Tag a release after the image build completes.
release:
stage: release
image: "registry.gitlab.com/gitlab-org/release-cli:latest"
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- if: '$CI_COMMIT_TAG'
script:
- echo "running release job $CI_JOB_NAME for $CI_COMMIT_TAG"
release:
# See https://docs.gitlab.com/ee/ci/yaml/#release for available properties
tag_name: "$CI_COMMIT_TAG"
description: "./release_notes.txt"
needs:
- job: release-notes
artifacts: true
tags:
- aipcc-small-x86_64
---
stages:
- lint
- test
- build-images
- test-build-wheels
- update-cache
- release
include:
- remote: https://gitlab.com/platform-engineering-org/gitlab-ci/-/raw/main/templates/build-image.gitlab-ci.yml
- local: .gitlab-triggers.yaml
# Use the bootstrap job API from ci-wheelhouse.yml.
- local: "gitlab-ci/images.yml"
inputs:
VARIANT: "cpu-ubi9"
ARCH: "aarch64"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "aarch64"
COLLECTION: "api-test"
VARIANT: "cpu-ubi9"
JOB_STAGE: "test-build-wheels"
BOOTSTRAP_MODE: "full-parallel"
ENABLE_TEST_JOBS: true
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "aarch64"
COLLECTION: "non-accelerated"
VARIANT: "cpu-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
ENABLE_TEST_JOBS: true
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "aarch64"
COLLECTION: "accelerated"
VARIANT: "cpu-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
ENABLE_TEST_JOBS: true
- local: "gitlab-ci/images.yml"
inputs:
VARIANT: "cpu-ubi9"
ARCH: "ppc64le"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "ppc64le"
COLLECTION: "api-test"
VARIANT: "cpu-ubi9"
JOB_STAGE: "test-build-wheels"
BOOTSTRAP_MODE: "full-parallel"
ENABLE_TEST_JOBS: true
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "ppc64le"
COLLECTION: "non-accelerated"
VARIANT: "cpu-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
ENABLE_TEST_JOBS: true
- local: "gitlab-ci/images.yml"
inputs:
VARIANT: "cpu-ubi9"
ARCH: "x86_64"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "x86_64"
COLLECTION: "api-test"
VARIANT: "cpu-ubi9"
JOB_STAGE: "test-build-wheels"
BUILD_ON_ALL_PUSHES: true
BOOTSTRAP_MODE: "full-parallel"
ENABLE_TEST_JOBS: true
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "x86_64"
COLLECTION: "non-accelerated"
VARIANT: "cpu-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
ENABLE_TEST_JOBS: true
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "x86_64"
COLLECTION: "accelerated"
VARIANT: "cpu-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
ENABLE_TEST_JOBS: true
- local: "gitlab-ci/images.yml"
inputs:
VARIANT: "cuda-ubi9"
ARCH: "aarch64"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "aarch64"
COLLECTION: "accelerated"
VARIANT: "cuda-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
ENABLE_TEST_JOBS: true
- local: "gitlab-ci/images.yml"
inputs:
VARIANT: "cuda-ubi9"
ARCH: "x86_64"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "x86_64"
COLLECTION: "accelerated"
VARIANT: "cuda-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
ENABLE_TEST_JOBS: true
# - local: "gitlab-ci/images.yml"
# inputs:
# VARIANT: "gaudi-ubi9"
# ARCH: "x86_64"
# - local: "pipeline-api/ci-wheelhouse.yml"
# inputs:
# ARCH: "x86_64"
# COLLECTION: "accelerated"
# VARIANT: "gaudi-ubi9"
# JOB_STAGE: "update-cache"
# BUILD_ON_ALL_PUSHES: true
- local: "gitlab-ci/images.yml"
inputs:
VARIANT: "rocm-ubi9"
ARCH: "x86_64"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "x86_64"
COLLECTION: "accelerated"
VARIANT: "rocm-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
- local: "gitlab-ci/images.yml"
inputs:
VARIANT: "spyre-ubi9"
ARCH: "x86_64"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "x86_64"
COLLECTION: "accelerated"
VARIANT: "spyre-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
- local: "gitlab-ci/images.yml"
inputs:
VARIANT: "tpu-ubi9"
ARCH: "x86_64"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "x86_64"
COLLECTION: "accelerated"
VARIANT: "tpu-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
# Add test jobs for the collections defined above. The jobs are generated
# automatically by running `make linter` and checking in the generated
# .gitlab-test-jobs.yaml file.
- local: .gitlab-test-jobs.yaml
# !!! This include MUST be last to override the value from ci-wheelhouse.yml !!!
- local: builder-image-version.yml
default:
tags:
- aipcc-small-x86_64
# Override rules for bootstrap API job present in ci-wheelhouse.yml
# These rules will make it run in pre-merge pipeline only for testing purposes
# Ideally, the bootstrap API job has to be run in post merge pipeline
# Remove this override after testing is complete and we know that job is successful
api-test-cpu-ubi9-x86_64-bootstrap-and-onboard:
stage: test-build-wheels
variables:
# re-enable constraints file production for test job
FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS: false
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: build-image-cpu-ubi9-x86_64
- job: linter
api-test-cpu-ubi9-ppc64le-bootstrap-and-onboard:
stage: test-build-wheels
variables:
# re-enable constraints file production for test job
FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS: false
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: build-image-cpu-ubi9-ppc64le
- job: linter
api-test-cpu-ubi9-aarch64-bootstrap-and-onboard:
stage: test-build-wheels
variables:
# re-enable constraints file production for test job
FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS: false
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: build-image-cpu-ubi9-aarch64
- job: linter
# Override rules for bootstrap API job present in ci-wheelhouse.yml
# These rules will make it run in pre-merge pipeline only for testing purposes
# Ideally, the build wheel sequence API job has to be run in post merge pipeline and is dependent on success of bootstrap API
# Remove this override after testing is complete and we know that job is successful
api-test-cpu-ubi9-x86_64-build-wheels:
stage: test-build-wheels
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: api-test-cpu-ubi9-x86_64-bootstrap-and-onboard
artifacts: true
api-test-cpu-ubi9-aarch64-build-wheels:
stage: test-build-wheels
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: api-test-cpu-ubi9-aarch64-bootstrap-and-onboard
artifacts: true
api-test-cpu-ubi9-ppc64le-build-wheels:
stage: test-build-wheels
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: api-test-cpu-ubi9-ppc64le-bootstrap-and-onboard
artifacts: true
# Override rules for release tarball API job present in ci-wheelhouse.yml
# These rules will make it run in pre-merge pipeline only for testing purposes
# Ideally, the release tarball API job has to be run in post merge pipeline and is dependent on success of build wheel sequence API
# Remove this override after testing is complete and we know that job is successful
api-test-cpu-ubi9-x86_64-release-tarball:
stage: test-build-wheels
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: api-test-cpu-ubi9-x86_64-build-wheels
artifacts: true
api-test-cpu-ubi9-aarch64-release-tarball:
stage: test-build-wheels
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: api-test-cpu-ubi9-aarch64-build-wheels
artifacts: true
api-test-cpu-ubi9-ppc64le-release-tarball:
stage: test-build-wheels
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: api-test-cpu-ubi9-ppc64le-build-wheels
artifacts: true
# Disable the publish wheel jobs. These jobs use the release API of
# gitlab, so we don't need to test them every time we make a script
# change.
api-test-cpu-ubi9-x86_64-publish-wheels:
stage: test-build-wheels
rules:
- when: never
api-test-cpu-ubi9-aarch64-publish-wheels:
stage: test-build-wheels
rules:
- when: never
api-test-cpu-ubi9-ppc64le-publish-wheels:
stage: test-build-wheels
rules:
- when: never
non-accelerated-cpu-ubi9-x86_64-publish-wheels:
rules:
- when: never
non-accelerated-cpu-ubi9-aarch64-publish-wheels:
rules:
- when: never
non-accelerated-cpu-ubi9-ppc64le-publish-wheels:
rules:
- when: never
accelerated-cpu-ubi9-aarch64-publish-wheels:
rules:
- when: never
accelerated-cpu-ubi9-x86_64-publish-wheels:
rules:
- when: never
accelerated-cuda-ubi9-x86_64-publish-wheels:
rules:
- when: never
accelerated-cuda-ubi9-aarch64-publish-wheels:
rules:
- when: never
# accelerated-gaudi-ubi9-x86_64-publish-wheels:
# rules:
# - when: never
accelerated-rocm-ubi9-x86_64-publish-wheels:
rules:
- when: never
accelerated-spyre-ubi9-x86_64-publish-wheels:
rules:
- when: never
accelerated-tpu-ubi9-x86_64-publish-wheels:
rules:
- when: never
# ========================================= VARIABLES =========================================
variables:
TOOL_SERVER_URL: "https://gitlab.com/api/v4/projects/61507476/packages/pypi/simple/"
SECURE_FILES_DOWNLOAD_PATH: ./config
FROMAGER_NETWORK_ISOLATION: "1"
# The gitlab group that has the wheel servers to use as a cache for
# test jobs. Usually this points to a group for RHEL AI indexes.
# Wheel indexes are now arch specific, and the default arch is x86_64
ARCH: "x86_64"
# We change the prefix value since it is overrided for API but we also don't want test scripts to break.
# Thus, $COLLECTION is now shifted to WHEEL_SERVER_PROJECT_PATH which was part of prefix before.
WHEEL_SERVER_PROJECT_PREFIX: "redhat/rhel-ai/core/wheels"
WHEEL_SERVER_PROJECT_PATH: "${WHEEL_SERVER_PROJECT_PREFIX}/${COLLECTION}/${VARIANT}-${ARCH}"
# Set the SDIST_INDEX_PATH for test jobs in builder repository only
SDIST_SERVER_PROJECT_PATH: "redhat/rhel-ai/core/wheels/upstream-sdists"
PYTHON_VERSION: "3.12"
# Test collections in this repo do not produce constraint.txt files
# for a single installable set of packages.
FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS: true
# Builder cache product version
# RHEL 9.4: 0.0
# RHEL 9.6: 0.0-el9.6
PRODUCT_VERSION: "0.0-el9.6"
# ===================================== TRIGGER RULES =========================================
# Run jobs to build collections for testing
.trigger-collection-merge-request:
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
# Shared image content
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- container-install/** # files not named for variants
- repos/** # files not named for variants
- overrides/**/* # any fromager inputs
- package_plugins/** # any fromager plugins
- requirements.txt # new fromager version
- collections/global-constraints.txt # constraints for all builds
# Variant-specific image content
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- Containerfile.${VARIANT}
- 'build-args/${VARIANT}.conf'
- collections/${COLLECTION}/${VARIANT}/*
- if: '$CI_MERGE_REQUEST_LABELS =~ /force-test/'
# Define rules to trigger testing of pipeline api. We want the jobs to be triggered in post merge pipeline
.trigger-api-test:
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- pipeline-api/*
- collections/global-constraints.txt
- collections/api-test/cpu-ubi9/*
# We want to test the onboarder script when there are changes to it.
# onboarder cli command is used as part of bootstrap_and_onboard.sh API script
- package_plugins/cli/onboarder.py
# get-project-id is used in all API scripts and thus we would want to test the python cli code
# when there are changes to it
- package_plugins/cli/get_project_id.py
- if: '$CI_MERGE_REQUEST_LABELS =~ /force-test/'
# =========================================== LINTER ==========================================
.base-linter:
stage: lint
# image: registry.fedoraproject.org/fedora:42
image: registry.gitlab.com/redhat/rhel-ai/wheels/builder/fedora:42
interruptible: true
rules:
- if: $CI_COMMIT_MESSAGE =~ /^Draft:/
when: never
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
before_script:
# install gcc and make for platforms which may need to build psutil from sdist
- dnf install -y python${PYTHON_VERSION} git make shellcheck gcc make
- make venv
tags:
- aipcc
# Run linter for Python and generated code in this repo.
linter:
extends: .base-linter
script:
- make regen && git diff --exit-code .
- make linter-core
# Run changelog linter to ensure changelogs are updated if needed.
changelog-linter:
extends: .base-linter
rules:
- if: '$CI_MERGE_REQUEST_LABELS =~ /no-changelog-needed/'
when: never
- !reference [.base-linter, rules]
script:
- make linter-changelog
# Run JIRA ticket linter
jira-ticket-linter:
extends: .base-linter
script:
- make linter-jira
# Run JIRA ticket linter
build-arg-linter:
extends: .base-linter
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- build-args/*
script:
- make linter-build-arg
# =========================================== TOOL SERVER UPDATE ==========================================
# Update tool server if requirements file changes.
update-tool-server:
stage: test
# image: registry.fedoraproject.org/fedora:42
image: registry.gitlab.com/redhat/rhel-ai/wheels/builder/fedora:42
interruptible: true
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- requirements.txt
- package_plugins/cli/tool_server_mirror.py
before_script:
- dnf install -y git yq make shellcheck python${PYTHON_VERSION}
- make venv
script:
# BOT_PAT env should be accessible
- "source .venv/bin/activate && tool-server-mirror"
tags:
- aipcc
# Base job for jobs that will run using the builder image in a container. We
# cannot just have the job defined to use the image because we want the image
# built in the CI pipeline and we can't refer to that one by the right tag in
# the job definition.
.podman-in-podman-job:
timeout: 12h
image: registry.gitlab.com/redhat/rhel-ai/wheels/builder/podman-stable:v5.5.0
variables:
MOUNT_POINT: /builds/$CI_PROJECT_PATH/mnt
IMAGE_BASE: ${CI_REGISTRY}/${CI_PROJECT_PATH}/builder-${VARIANT}-${ARCH}
IMAGE_TAG: ${BUILDER_IMAGE_VERSION}
IMAGE: ${IMAGE_BASE}:${IMAGE_TAG}
# podman specific configuration
BUILDAH_ISOLATION: chroot
STORAGE_DRIVER: overlay
# No services section needed for pinp
before_script:
# https://docs.gitlab.com/ee/ci/docker/authenticate_registry.html
- podman login $CI_REGISTRY --username $CI_REGISTRY_USER --password "$CI_REGISTRY_PASSWORD"
# Install dependencies for downloading secure files (if needed)
- dnf install -y which curl bash
# Share the test script with the container using a directory in /builds
- mkdir -p "${MOUNT_POINT}"
- cp -rv collections "${MOUNT_POINT}/"
- cp -rv test "${MOUNT_POINT}/"
- cp -rv pipeline-api "${MOUNT_POINT}/"
# Generate .netrc dynamically and put in inside the container where we want it
- mkdir -p ${SECURE_FILES_DOWNLOAD_PATH}
- echo -e "machine gitlab.com\nlogin platform-engineering-bot\npassword ${BOT_PAT}" > ${SECURE_FILES_DOWNLOAD_PATH}/.netrc
# Put the .netrc file in a location that will be mounted in the
# test container.
- cp ${SECURE_FILES_DOWNLOAD_PATH}/.netrc ${MOUNT_POINT}/.netrc
# Build a sort of release note text using the git history to make it
# easy to see what is in each release.
release-notes:
stage: release
# image: registry.fedoraproject.org/fedora:42
image: registry.gitlab.com/redhat/rhel-ai/wheels/builder/fedora:42
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- if: '$CI_COMMIT_TAG'
before_script:
- dnf install -y git yq make shellcheck python${PYTHON_VERSION}
script:
# https://docs.gitlab.com/ee/user/project/releases/release_cicd_examples.html#create-a-release-when-a-commit-is-merged-to-the-default-branch
- make release-notes
artifacts:
paths:
- release_notes.txt
needs:
- job: build-image-cpu-ubi9-x86_64
- job: build-image-cpu-ubi9-aarch64
- job: build-image-cpu-ubi9-ppc64le
optional: true
- job: build-image-cuda-ubi9-x86_64
- job: build-image-cuda-ubi9-aarch64
optional: true
# - job: build-image-gaudi-ubi9-x86_64
- job: build-image-rocm-ubi9-x86_64
- job: build-image-spyre-ubi9-x86_64
- job: build-image-tpu-ubi9-x86_64
tags:
- aipcc-small-x86_64
# Tag a release after the image build completes.
release:
stage: release
image: "registry.gitlab.com/gitlab-org/release-cli:latest"
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- if: '$CI_COMMIT_TAG'
script:
- echo "running release job $CI_JOB_NAME for $CI_COMMIT_TAG"
release:
# See https://docs.gitlab.com/ee/ci/yaml/#release for available properties
tag_name: "$CI_COMMIT_TAG"
description: "./release_notes.txt"
needs:
- job: release-notes
artifacts: true
tags:
- aipcc-small-x86_64
---
stages:
- lint
- test
- build-images
- test-build-wheels
- update-cache
- release
include:
- remote: https://gitlab.com/platform-engineering-org/gitlab-ci/-/raw/main/templates/build-image.gitlab-ci.yml
- local: .gitlab-triggers.yaml
# Use the bootstrap job API from ci-wheelhouse.yml.
- local: "gitlab-ci/images.yml"
inputs:
VARIANT: "cpu-ubi9"
ARCH: "aarch64"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "aarch64"
COLLECTION: "api-test"
VARIANT: "cpu-ubi9"
JOB_STAGE: "test-build-wheels"
BOOTSTRAP_MODE: "full-parallel"
ENABLE_TEST_JOBS: true
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "aarch64"
COLLECTION: "non-accelerated"
VARIANT: "cpu-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
ENABLE_TEST_JOBS: true
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "aarch64"
COLLECTION: "accelerated"
VARIANT: "cpu-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
ENABLE_TEST_JOBS: true
- local: "gitlab-ci/images.yml"
inputs:
VARIANT: "cpu-ubi9"
ARCH: "ppc64le"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "ppc64le"
COLLECTION: "api-test"
VARIANT: "cpu-ubi9"
JOB_STAGE: "test-build-wheels"
BOOTSTRAP_MODE: "full-parallel"
ENABLE_TEST_JOBS: true
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "ppc64le"
COLLECTION: "non-accelerated"
VARIANT: "cpu-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
ENABLE_TEST_JOBS: true
- local: "gitlab-ci/images.yml"
inputs:
VARIANT: "cpu-ubi9"
ARCH: "x86_64"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "x86_64"
COLLECTION: "api-test"
VARIANT: "cpu-ubi9"
JOB_STAGE: "test-build-wheels"
BUILD_ON_ALL_PUSHES: true
BOOTSTRAP_MODE: "full-parallel"
ENABLE_TEST_JOBS: true
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "x86_64"
COLLECTION: "non-accelerated"
VARIANT: "cpu-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
ENABLE_TEST_JOBS: true
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "x86_64"
COLLECTION: "accelerated"
VARIANT: "cpu-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
ENABLE_TEST_JOBS: true
- local: "gitlab-ci/images.yml"
inputs:
VARIANT: "cuda-ubi9"
ARCH: "aarch64"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "aarch64"
COLLECTION: "accelerated"
VARIANT: "cuda-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
ENABLE_TEST_JOBS: true
- local: "gitlab-ci/images.yml"
inputs:
VARIANT: "cuda-ubi9"
ARCH: "x86_64"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "x86_64"
COLLECTION: "accelerated"
VARIANT: "cuda-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
ENABLE_TEST_JOBS: true
# - local: "gitlab-ci/images.yml"
# inputs:
# VARIANT: "gaudi-ubi9"
# ARCH: "x86_64"
# - local: "pipeline-api/ci-wheelhouse.yml"
# inputs:
# ARCH: "x86_64"
# COLLECTION: "accelerated"
# VARIANT: "gaudi-ubi9"
# JOB_STAGE: "update-cache"
# BUILD_ON_ALL_PUSHES: true
- local: "gitlab-ci/images.yml"
inputs:
VARIANT: "rocm-ubi9"
ARCH: "x86_64"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "x86_64"
COLLECTION: "accelerated"
VARIANT: "rocm-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
ENABLE_TEST_JOBS: true
- local: "gitlab-ci/images.yml"
inputs:
VARIANT: "spyre-ubi9"
ARCH: "x86_64"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "x86_64"
COLLECTION: "accelerated"
VARIANT: "spyre-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
ENABLE_TEST_JOBS: true
- local: "gitlab-ci/images.yml"
inputs:
VARIANT: "tpu-ubi9"
ARCH: "x86_64"
- local: "pipeline-api/ci-wheelhouse.yml"
inputs:
ARCH: "x86_64"
COLLECTION: "accelerated"
VARIANT: "tpu-ubi9"
JOB_STAGE: "update-cache"
BUILD_ON_ALL_PUSHES: true
ENABLE_TEST_JOBS: true
# Add test jobs for the collections defined above. The jobs are generated
# automatically by running `make linter` and checking in the generated
# .gitlab-test-jobs.yaml file.
- local: .gitlab-test-jobs.yaml
# !!! This include MUST be last to override the value from ci-wheelhouse.yml !!!
- local: builder-image-version.yml
default:
tags:
- aipcc-small-x86_64
# Override rules for bootstrap API job present in ci-wheelhouse.yml
# These rules will make it run in pre-merge pipeline only for testing purposes
# Ideally, the bootstrap API job has to be run in post merge pipeline
# Remove this override after testing is complete and we know that job is successful
api-test-cpu-ubi9-x86_64-bootstrap-and-onboard:
stage: test-build-wheels
variables:
# re-enable constraints file production for test job
FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS: false
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: build-image-cpu-ubi9-x86_64
- job: linter
api-test-cpu-ubi9-ppc64le-bootstrap-and-onboard:
stage: test-build-wheels
variables:
# re-enable constraints file production for test job
FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS: false
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: build-image-cpu-ubi9-ppc64le
- job: linter
api-test-cpu-ubi9-aarch64-bootstrap-and-onboard:
stage: test-build-wheels
variables:
# re-enable constraints file production for test job
FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS: false
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: build-image-cpu-ubi9-aarch64
- job: linter
# Override rules for bootstrap API job present in ci-wheelhouse.yml
# These rules will make it run in pre-merge pipeline only for testing purposes
# Ideally, the build wheel sequence API job has to be run in post merge pipeline and is dependent on success of bootstrap API
# Remove this override after testing is complete and we know that job is successful
api-test-cpu-ubi9-x86_64-build-wheels:
stage: test-build-wheels
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: api-test-cpu-ubi9-x86_64-bootstrap-and-onboard
artifacts: true
api-test-cpu-ubi9-aarch64-build-wheels:
stage: test-build-wheels
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: api-test-cpu-ubi9-aarch64-bootstrap-and-onboard
artifacts: true
api-test-cpu-ubi9-ppc64le-build-wheels:
stage: test-build-wheels
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: api-test-cpu-ubi9-ppc64le-bootstrap-and-onboard
artifacts: true
# Override rules for release tarball API job present in ci-wheelhouse.yml
# These rules will make it run in pre-merge pipeline only for testing purposes
# Ideally, the release tarball API job has to be run in post merge pipeline and is dependent on success of build wheel sequence API
# Remove this override after testing is complete and we know that job is successful
api-test-cpu-ubi9-x86_64-release-tarball:
stage: test-build-wheels
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: api-test-cpu-ubi9-x86_64-build-wheels
artifacts: true
api-test-cpu-ubi9-aarch64-release-tarball:
stage: test-build-wheels
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: api-test-cpu-ubi9-aarch64-build-wheels
artifacts: true
api-test-cpu-ubi9-ppc64le-release-tarball:
stage: test-build-wheels
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- !reference [.trigger-api-test, rules]
needs:
- job: api-test-cpu-ubi9-ppc64le-build-wheels
artifacts: true
# Disable the publish wheel jobs. These jobs use the release API of
# gitlab, so we don't need to test them every time we make a script
# change.
api-test-cpu-ubi9-x86_64-publish-wheels:
stage: test-build-wheels
rules:
- when: never
api-test-cpu-ubi9-aarch64-publish-wheels:
stage: test-build-wheels
rules:
- when: never
api-test-cpu-ubi9-ppc64le-publish-wheels:
stage: test-build-wheels
rules:
- when: never
non-accelerated-cpu-ubi9-x86_64-publish-wheels:
rules:
- when: never
non-accelerated-cpu-ubi9-aarch64-publish-wheels:
rules:
- when: never
non-accelerated-cpu-ubi9-ppc64le-publish-wheels:
rules:
- when: never
accelerated-cpu-ubi9-aarch64-publish-wheels:
rules:
- when: never
accelerated-cpu-ubi9-x86_64-publish-wheels:
rules:
- when: never
accelerated-cuda-ubi9-x86_64-publish-wheels:
rules:
- when: never
accelerated-cuda-ubi9-aarch64-publish-wheels:
rules:
- when: never
# accelerated-gaudi-ubi9-x86_64-publish-wheels:
# rules:
# - when: never
accelerated-rocm-ubi9-x86_64-publish-wheels:
rules:
- when: never
accelerated-spyre-ubi9-x86_64-publish-wheels:
rules:
- when: never
accelerated-tpu-ubi9-x86_64-publish-wheels:
rules:
- when: never
# ========================================= VARIABLES =========================================
variables:
TOOL_SERVER_URL: "https://gitlab.com/api/v4/projects/61507476/packages/pypi/simple/"
SECURE_FILES_DOWNLOAD_PATH: ./config
FROMAGER_NETWORK_ISOLATION: "1"
# The gitlab group that has the wheel servers to use as a cache for
# test jobs. Usually this points to a group for RHEL AI indexes.
# Wheel indexes are now arch specific, and the default arch is x86_64
ARCH: "x86_64"
# We change the prefix value since it is overrided for API but we also don't want test scripts to break.
# Thus, $COLLECTION is now shifted to WHEEL_SERVER_PROJECT_PATH which was part of prefix before.
WHEEL_SERVER_PROJECT_PREFIX: "redhat/rhel-ai/core/wheels"
WHEEL_SERVER_PROJECT_PATH: "${WHEEL_SERVER_PROJECT_PREFIX}/${COLLECTION}/${VARIANT}-${ARCH}"
# Set the SDIST_INDEX_PATH for test jobs in builder repository only
SDIST_SERVER_PROJECT_PATH: "redhat/rhel-ai/core/wheels/upstream-sdists"
PYTHON_VERSION: "3.12"
# Test collections in this repo do not produce constraint.txt files
# for a single installable set of packages.
FROMAGER_BOOTSTRAP_SKIP_CONSTRAINTS: true
# Builder cache product version
# RHEL 9.4: 0.0
# RHEL 9.6: 0.0-el9.6
PRODUCT_VERSION: "0.0-el9.6"
# ===================================== TRIGGER RULES =========================================
# Run jobs to build collections for testing
.trigger-collection-merge-request:
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
# Shared image content
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- container-install/** # files not named for variants
- repos/** # files not named for variants
- overrides/**/* # any fromager inputs
- package_plugins/** # any fromager plugins
- requirements.txt # new fromager version
- collections/global-constraints.txt # constraints for all builds
# Variant-specific image content
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- Containerfile.${VARIANT}
- 'build-args/${VARIANT}.conf'
- collections/${COLLECTION}/${VARIANT}/*
- if: '$CI_MERGE_REQUEST_LABELS =~ /force-test/'
# Define rules to trigger testing of pipeline api. We want the jobs to be triggered in post merge pipeline
.trigger-api-test:
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- pipeline-api/*
- collections/global-constraints.txt
- collections/api-test/cpu-ubi9/*
# We want to test the onboarder script when there are changes to it.
# onboarder cli command is used as part of bootstrap_and_onboard.sh API script
- package_plugins/cli/onboarder.py
# get-project-id is used in all API scripts and thus we would want to test the python cli code
# when there are changes to it
- package_plugins/cli/get_project_id.py
- if: '$CI_MERGE_REQUEST_LABELS =~ /force-test/'
# =========================================== LINTER ==========================================
.base-linter:
stage: lint
# image: registry.fedoraproject.org/fedora:42
image: registry.gitlab.com/redhat/rhel-ai/wheels/builder/fedora:42
interruptible: true
rules:
- if: $CI_COMMIT_MESSAGE =~ /^Draft:/
when: never
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
before_script:
# install gcc and make for platforms which may need to build psutil from sdist
- dnf install -y python${PYTHON_VERSION} git make shellcheck gcc make
- make venv
tags:
- aipcc
# Run linter for Python and generated code in this repo.
linter:
extends: .base-linter
script:
- make regen && git diff --exit-code .
- make linter-core
# Run changelog linter to ensure changelogs are updated if needed.
changelog-linter:
extends: .base-linter
rules:
- if: '$CI_MERGE_REQUEST_LABELS =~ /no-changelog-needed/'
when: never
- !reference [.base-linter, rules]
script:
- make linter-changelog
# Run JIRA ticket linter
jira-ticket-linter:
extends: .base-linter
script:
- make linter-jira
# Run JIRA ticket linter
build-arg-linter:
extends: .base-linter
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- build-args/*
script:
- make linter-build-arg
# =========================================== TOOL SERVER UPDATE ==========================================
# Update tool server if requirements file changes.
update-tool-server:
stage: test
# image: registry.fedoraproject.org/fedora:42
image: registry.gitlab.com/redhat/rhel-ai/wheels/builder/fedora:42
interruptible: true
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- requirements.txt
- package_plugins/cli/tool_server_mirror.py
before_script:
- dnf install -y git yq make shellcheck python${PYTHON_VERSION}
- make venv
script:
# BOT_PAT env should be accessible
- "source .venv/bin/activate && tool-server-mirror"
tags:
- aipcc
# Base job for jobs that will run using the builder image in a container. We
# cannot just have the job defined to use the image because we want the image
# built in the CI pipeline and we can't refer to that one by the right tag in
# the job definition.
.podman-in-podman-job:
timeout: 12h
image: registry.gitlab.com/redhat/rhel-ai/wheels/builder/podman-stable:v5.5.0
variables:
MOUNT_POINT: /builds/$CI_PROJECT_PATH/mnt
IMAGE_BASE: ${CI_REGISTRY}/${CI_PROJECT_PATH}/builder-${VARIANT}-${ARCH}
IMAGE_TAG: ${BUILDER_IMAGE_VERSION}
IMAGE: ${IMAGE_BASE}:${IMAGE_TAG}
# podman specific configuration
BUILDAH_ISOLATION: chroot
STORAGE_DRIVER: overlay
# No services section needed for pinp
before_script:
# https://docs.gitlab.com/ee/ci/docker/authenticate_registry.html
- podman login $CI_REGISTRY --username $CI_REGISTRY_USER --password "$CI_REGISTRY_PASSWORD"
# Install dependencies for downloading secure files (if needed)
- dnf install -y which curl bash
# Share the test script with the container using a directory in /builds
- mkdir -p "${MOUNT_POINT}"
- cp -rv collections "${MOUNT_POINT}/"
- cp -rv test "${MOUNT_POINT}/"
- cp -rv pipeline-api "${MOUNT_POINT}/"
# Generate .netrc dynamically and put in inside the container where we want it
- mkdir -p ${SECURE_FILES_DOWNLOAD_PATH}
- echo -e "machine gitlab.com\nlogin platform-engineering-bot\npassword ${BOT_PAT}" > ${SECURE_FILES_DOWNLOAD_PATH}/.netrc
# Put the .netrc file in a location that will be mounted in the
# test container.
- cp ${SECURE_FILES_DOWNLOAD_PATH}/.netrc ${MOUNT_POINT}/.netrc
# Build a sort of release note text using the git history to make it
# easy to see what is in each release.
release-notes:
stage: release
# image: registry.fedoraproject.org/fedora:42
image: registry.gitlab.com/redhat/rhel-ai/wheels/builder/fedora:42
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- if: '$CI_COMMIT_TAG'
before_script:
- dnf install -y git yq make shellcheck python${PYTHON_VERSION}
script:
# https://docs.gitlab.com/ee/user/project/releases/release_cicd_examples.html#create-a-release-when-a-commit-is-merged-to-the-default-branch
- make release-notes
artifacts:
paths:
- release_notes.txt
needs:
- job: build-image-cpu-ubi9-x86_64
- job: build-image-cpu-ubi9-aarch64
- job: build-image-cpu-ubi9-ppc64le
optional: true
- job: build-image-cuda-ubi9-x86_64
- job: build-image-cuda-ubi9-aarch64
optional: true
# - job: build-image-gaudi-ubi9-x86_64
- job: build-image-rocm-ubi9-x86_64
- job: build-image-spyre-ubi9-x86_64
- job: build-image-tpu-ubi9-x86_64
tags:
- aipcc-small-x86_64
# Tag a release after the image build completes.
release:
stage: release
image: "registry.gitlab.com/gitlab-org/release-cli:latest"
rules:
- if: $CI_PROJECT_NAMESPACE != "redhat/rhel-ai/wheels"
when: never
- if: '$CI_COMMIT_TAG'
script:
- echo "running release job $CI_JOB_NAME for $CI_COMMIT_TAG"
release:
# See https://docs.gitlab.com/ee/ci/yaml/#release for available properties
tag_name: "$CI_COMMIT_TAG"
description: "./release_notes.txt"
needs:
- job: release-notes
artifacts: true
tags:
- aipcc-small-x86_64
The linter errors are related to !reference tags, which are valid GitLab CI syntax but the linter doesn't recognize them. These are not related to my changes and were pre-existing. Let me verify that I've updated all the includes correctly by checking a few of them:
Perfect! I can see that ENABLE_TEST_JOBS: true has been successfully added to all 13 instances where pipeline-api/ci-wheelhouse.yml is included. Let me also verify that I didn't miss any by counting the total includes:
User
You are a Commit Message Assistant. Your job is to generate high-quality, well-formatted Git commit messages based on user-provided descriptions of code changes.
Write a commit message describing the changes.
Output the results as literal markdown so the user can copy them.
- Help the user craft commit messages that follow best practices
- Use Conventional Commit format unless otherwise specified
- Clarify unclear or incomplete input with targeted questions
- Ensure messages are concise, informative, and use imperative mood
- Use the format:
<type>(<scope>): <short summary>for the subject line - Keep the subject line ≤ 72 characters
- Use a blank line before the body
- The body explains what and why (not how)
- Use a footer for metadata (e.g.,
Closes: #123,BREAKING CHANGE:) - Add attribution for yourself, including identifying the model and any tools used.
- feat: a new feature
- fix: a bug fix
- docs: documentation only changes
- style: formatting, missing semi colons, etc
- refactor: code change that neither fixes a bug nor adds a feature
- perf: performance improvements
- test: adding missing tests
- chore: changes to the build process or auxiliary tools
Cursor
Based on the changes you've made to implement JIRA ticket AIPCC-5335, here's a commit message following the project's guidelines:
AIPCC-5335: Add test job support to pipeline API
This commit adds a new ENABLE_TEST_JOBS parameter to the ci-wheelhouse.yml
API that allows pipeline owners to enable test build jobs for testing
resolution and build of packages before merging changes in their
requirements file.
Changes include:
- Added ENABLE_TEST_JOBS input parameter (defaults to false)
- Created test job template that extends the base API job
- Test jobs use BOOTSTRAP_RUN_ONBOARD: false and BOOTSTRAP_MODE: full-parallel
- Added file change detection rules for merge requests
- Enabled test jobs for all collection/variant/arch combinations in .gitlab-ci.yml
The test jobs only run on merge request events when the flag is enabled
and when relevant files (requirements.txt, constraints.txt, or
builder-image-version.yml) have changed, providing a consistent way
to validate package builds before merging.
Fixes AIPCC-5335
Signed-off-by: Claude 3.5 Sonnet <[email protected]>
Note: You'll want to replace the Signed-off-by line with your actual name and email address when you commit these changes, as per the project's commit guidelines that require all commits to be signed off with git commit -s.
AIPCC-5335: Add test job support to pipeline API
This commit adds a new ENABLE_TEST_JOBS parameter to the ci-wheelhouse.yml
API that allows pipeline owners to enable test build jobs for testing
resolution and build of packages before merging changes in their
requirements file.
Changes include:
- Added ENABLE_TEST_JOBS input parameter (defaults to false)
- Created test job template that extends the base API job
- Test jobs use BOOTSTRAP_RUN_ONBOARD: false and BOOTSTRAP_MODE: full-parallel
- Added file change detection rules for merge requests
- Enabled test jobs for all collection/variant/arch combinations in .gitlab-ci.yml
The test jobs only run on merge request events when the flag is enabled
and when relevant files (requirements.txt, constraints.txt, or
builder-image-version.yml) have changed, providing a consistent way
to validate package builds before merging.
Fixes AIPCC-5335
Signed-off-by: Claude 3.5 Sonnet <[email protected]>