Exported on 9/20/2025 at 14:12:45 EDT from Cursor (1.6.35)
User
I want to work on jira issue @https://issues.redhat.com/browse/AIPCC-5343 by creating a bin/build_wheels.sh script that wraps the pipeline-api/build_wheels.sh script similar to how bin/bootstrap.sh works.
The new script should take as input a containerfile, a constraints file, and a graph file.
It should build the image for the containerfile in the same way bin/bootstrap.sh does, then it should use that image to run the pipeline-api/build_wheels.sh script, setting any environment variables that are needed in the podman command.
The new script should handle the constraints file in the same way that bin/bootstrap.sh does and should pass the constraints to pipeline-api/build_wheels.sh instead of it preparing the constraints itself.
The new script should handle the WHEEL_SERVER_PROJECT_PATH in the same way as bin/bootstrap.sh.
The new script should handle OUTDIR_BASE in the same way as bin/bootstrap.sh.
Cursor
I'll help you create a bin/build_wheels.sh
script that wraps the pipeline-api/build_wheels.sh
script similar to how bin/bootstrap.sh
works. Let me first examine the existing files to understand the patterns and requirements.
Now let me examine the prepare_constraints.sh
script to understand how constraints are prepared:
Now let me check if there are any other relevant files to understand the complete picture:
Now I'll create the bin/build_wheels.sh
script based on my analysis. The key differences from bootstrap.sh
will be:
- It takes a containerfile, constraints file, and graph file as input
- It runs
pipeline-api/build_wheels.sh
instead ofbootstrap_and_onboard.sh
- It handles constraints by passing them to the pipeline script instead of letting it prepare them
- It needs to set
FROMAGER_VARIANT
environment variable - It uses a different output directory naming scheme
#!/bin/bash
# -*- indent-tabs-mode: nil; tab-width: 2; sh-indentation: 2; -*-
ROOTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )/.." && pwd )"
VENV_DIR="${ROOTDIR}/.venv"
# shellcheck disable=SC2012
VALID_COLLECTIONS="$(ls collections | tr '\n' '|')none"
usage() {
echo "Usage: [-f <fromager arguments> | -k <seconds>] CONTAINERFILE CONSTRAINTS GRAPH_FILE"
echo " -C: Cache collection to use: $VALID_COLLECTIONS (default: accelerated)"
echo " -c: Execute different command in container (must be passed with double quotes)"
echo " -f: additional fromager arguments"
echo " -h: help (this message)"
echo " -k: set number of seconds to keep container running after execution"
}
COMMAND=""
FROMAGER_ARGS=""
KEEPALIVE=0
CACHE_COLLECTION="accelerated"
BASE_ARGS=()
while [[ $# -gt 0 ]]; do
case $1 in
-h)
usage
exit 0
;;
-c)
COMMAND="$2"
shift
shift
;;
-C)
CACHE_COLLECTION="$2"
shift
shift
;;
-f)
FROMAGER_ARGS="$2"
shift
shift
;;
-k)
KEEPALIVE="$2"
re='^[0-9]+$'
if ! [[ "$KEEPALIVE" =~ $re ]]; then
echo "-k value must be a number of seconds to keep container running"
exit 1
fi
shift
shift
;;
*)
BASE_ARGS+=("$1")
shift
;;
esac
done
# reset the args with base arguments
set -- "${BASE_ARGS[@]}"
if [ "$#" -lt 3 ]; then
usage
exit 1
fi
set -x
set -e
set -o pipefail
CONTAINERFILE="$1"
CONSTRAINTS="$2"
GRAPH_FILE="$3"
# Check if CONTAINERFILE exists
if [ ! -f "$CONTAINERFILE" ]; then
echo "Error: CONTAINERFILE '$CONTAINERFILE' does not exist"
exit 1
fi
# Check if GRAPH_FILE exists
if [ ! -f "$GRAPH_FILE" ]; then
echo "Error: GRAPH_FILE '$GRAPH_FILE' does not exist"
exit 1
fi
shift
shift
shift
# Use the variant from the containerfile to make a variant-specific
# image
IMAGE_SUFFIX=$(basename "$CONTAINERFILE" | cut -f2 -d.)
if [ -z "$IMAGE_SUFFIX" ]; then
IMAGE_SUFFIX=unknown
fi
# Use the major and minor version of the most recent tag as a
# version. We could use the entire version, but that would end up
# retaining a lot more images. This choice tries to balance between
# retaining enough to allow switching between branches easily.
IMAGE_TAG=$(git describe --tags --abbrev=0 | cut -f1-2 -d.)
IMAGE_NAME="wheels-builder-${IMAGE_SUFFIX}"
IMAGE="${IMAGE_NAME}:${IMAGE_TAG}"
# Use the local .netrc file in the project for authentication to the wheel server
NETRC_FILE=$(pwd)/.netrc
export NETRC="${NETRC_FILE}"
# ensure we have a virtualenv with our dependencies
make -C "$ROOTDIR" venv
# Compute wheel server url from wheel server project path using wheel project id
if [ "$CACHE_COLLECTION" != "none" ]; then
if [ ! -d "collections/$CACHE_COLLECTION" ]; then
echo "-C value must be one of $VALID_COLLECTIONS"
exit 1
fi
WHEEL_SERVER_PROJECT_PREFIX="redhat/rhel-ai/core/wheels/${CACHE_COLLECTION}-0.0-el9.6"
WHEEL_SERVER_PROJECT_PATH="${WHEEL_SERVER_PROJECT_PREFIX}/${IMAGE_SUFFIX}-$(uname -m)"
fi
# Create the output directory so we can mount it when we run the
# container.
BRANCH=$(git rev-parse --abbrev-ref HEAD)
if [ "$BRANCH" = "main" ]; then
OUTDIR_BASE="build-wheels-output.${IMAGE_SUFFIX}"
else
OUTDIR_BASE="build-wheels-output.${IMAGE_SUFFIX}-${BRANCH}"
fi
OUTDIR="./${OUTDIR_BASE}"
CACHE_DIR=./.cache
mkdir -p "$OUTDIR" "$CACHE_DIR"
USER_CONSTRAINTS_FILE="$OUTDIR/user_constraints.txt"
if [ ! -f "$CONSTRAINTS" ]; then
echo "$CONSTRAINTS" | tr ',' '\n' > "$USER_CONSTRAINTS_FILE"
else
cp "$CONSTRAINTS" "$USER_CONSTRAINTS_FILE"
fi
# Generate the combined constraints file
GLOBAL_CONSTRAINTS="collections/global-constraints.txt"
CONSTRAINTS_FILE="$OUTDIR/computed_constraints.txt"
${VENV_DIR}/bin/prepare-constraints \
-o "$CONSTRAINTS_FILE" \
"$GLOBAL_CONSTRAINTS" \
"$USER_CONSTRAINTS_FILE"
# Copy the graph file to the output directory
GRAPH_FILE_DEST="$OUTDIR/work-dir/graph.json"
mkdir -p "$(dirname "$GRAPH_FILE_DEST")"
cp "$GRAPH_FILE" "$GRAPH_FILE_DEST"
# Even though we're not doing an actual release, we need a release version suffix
RELEASE_VERSION_SUFFIX=0
# shellcheck disable=SC2086
"$(dirname $0)"/build_container.sh "$CONTAINERFILE" "$IMAGE"
# set the default command
[ -z "$COMMAND" ] && COMMAND="/mirror-builder/pipeline-api/build_wheels.sh"
randomstring=$(tr -dc a-z0-9 </dev/urandom | head -c 18; echo)
# The build-wheels-output volume uses the same internal path for variants. This
# is necessary to share ccache caches between variants and branches.
podman run \
-it \
--name "test-${IMAGE_NAME}-${randomstring}" \
--rm \
--security-opt label=disable \
--volume "$OUTDIR:/work/build-wheels-output:rw,exec" \
--volume "./$CACHE_DIR:/var/cache/builder:rw,exec" \
--volume "./.netrc:/run/secrets/netrc:ro" \
--env NETRC=/run/secrets/netrc \
--env OUTDIR=/work/build-wheels-output \
--env CONSTRAINTS="/work/build-wheels-output/computed_constraints.txt" \
--env COLLECTION="$CACHE_COLLECTION" \
--env WHEEL_SERVER_PROJECT_PATH="$WHEEL_SERVER_PROJECT_PATH" \
--env RELEASE_VERSION_SUFFIX="$RELEASE_VERSION_SUFFIX" \
--env FROMAGER_ARGS="$FROMAGER_ARGS" \
--env FROMAGER_VARIANT="$IMAGE_SUFFIX" \
--ulimit host \
--pids-limit -1 \
"$IMAGE" \
sh -xc "$COMMAND; sleep $KEEPALIVE"
Now let me make the script executable:
Let me check for any linting errors in the new script:
Now let me test the script to make sure it works correctly. First, let me check what containerfiles are available:
User
The new script does not need to support running alternative commands in the image.
Cursor
You're right! Let me remove the -c
option and the COMMAND
variable since the new script should only run the pipeline-api/build_wheels.sh
script. I'll update the script to simplify it:
#!/bin/bash
# -*- indent-tabs-mode: nil; tab-width: 2; sh-indentation: 2; -*-
ROOTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )/.." && pwd )"
VENV_DIR="${ROOTDIR}/.venv"
# shellcheck disable=SC2012
VALID_COLLECTIONS="$(ls collections | tr '\n' '|')none"
usage() {
echo "Usage: [-f <fromager arguments> | -k <seconds>] CONTAINERFILE CONSTRAINTS GRAPH_FILE"
echo " -C: Cache collection to use: $VALID_COLLECTIONS (default: accelerated)"
echo " -f: additional fromager arguments"
echo " -h: help (this message)"
echo " -k: set number of seconds to keep container running after execution"
}
COMMAND=""
FROMAGER_ARGS=""
KEEPALIVE=0
CACHE_COLLECTION="accelerated"
BASE_ARGS=()
while [[ $# -gt 0 ]]; do
case $1 in
-h)
usage
exit 0
;;
-c)
COMMAND="$2"
shift
shift
;;
-C)
CACHE_COLLECTION="$2"
shift
shift
;;
-f)
FROMAGER_ARGS="$2"
shift
shift
;;
-k)
KEEPALIVE="$2"
re='^[0-9]+$'
if ! [[ "$KEEPALIVE" =~ $re ]]; then
echo "-k value must be a number of seconds to keep container running"
exit 1
fi
shift
shift
;;
*)
BASE_ARGS+=("$1")
shift
;;
esac
done
# reset the args with base arguments
set -- "${BASE_ARGS[@]}"
if [ "$#" -lt 3 ]; then
usage
exit 1
fi
set -x
set -e
set -o pipefail
CONTAINERFILE="$1"
CONSTRAINTS="$2"
GRAPH_FILE="$3"
# Check if CONTAINERFILE exists
if [ ! -f "$CONTAINERFILE" ]; then
echo "Error: CONTAINERFILE '$CONTAINERFILE' does not exist"
exit 1
fi
# Check if GRAPH_FILE exists
if [ ! -f "$GRAPH_FILE" ]; then
echo "Error: GRAPH_FILE '$GRAPH_FILE' does not exist"
exit 1
fi
shift
shift
shift
# Use the variant from the containerfile to make a variant-specific
# image
IMAGE_SUFFIX=$(basename "$CONTAINERFILE" | cut -f2 -d.)
if [ -z "$IMAGE_SUFFIX" ]; then
IMAGE_SUFFIX=unknown
fi
# Use the major and minor version of the most recent tag as a
# version. We could use the entire version, but that would end up
# retaining a lot more images. This choice tries to balance between
# retaining enough to allow switching between branches easily.
IMAGE_TAG=$(git describe --tags --abbrev=0 | cut -f1-2 -d.)
IMAGE_NAME="wheels-builder-${IMAGE_SUFFIX}"
IMAGE="${IMAGE_NAME}:${IMAGE_TAG}"
# Use the local .netrc file in the project for authentication to the wheel server
NETRC_FILE=$(pwd)/.netrc
export NETRC="${NETRC_FILE}"
# ensure we have a virtualenv with our dependencies
make -C "$ROOTDIR" venv
# Compute wheel server url from wheel server project path using wheel project id
if [ "$CACHE_COLLECTION" != "none" ]; then
if [ ! -d "collections/$CACHE_COLLECTION" ]; then
echo "-C value must be one of $VALID_COLLECTIONS"
exit 1
fi
WHEEL_SERVER_PROJECT_PREFIX="redhat/rhel-ai/core/wheels/${CACHE_COLLECTION}-0.0-el9.6"
WHEEL_SERVER_PROJECT_PATH="${WHEEL_SERVER_PROJECT_PREFIX}/${IMAGE_SUFFIX}-$(uname -m)"
fi
# Create the output directory so we can mount it when we run the
# container.
BRANCH=$(git rev-parse --abbrev-ref HEAD)
if [ "$BRANCH" = "main" ]; then
OUTDIR_BASE="build-wheels-output.${IMAGE_SUFFIX}"
else
OUTDIR_BASE="build-wheels-output.${IMAGE_SUFFIX}-${BRANCH}"
fi
OUTDIR="./${OUTDIR_BASE}"
CACHE_DIR=./.cache
mkdir -p "$OUTDIR" "$CACHE_DIR"
USER_CONSTRAINTS_FILE="$OUTDIR/user_constraints.txt"
if [ ! -f "$CONSTRAINTS" ]; then
echo "$CONSTRAINTS" | tr ',' '\n' > "$USER_CONSTRAINTS_FILE"
else
cp "$CONSTRAINTS" "$USER_CONSTRAINTS_FILE"
fi
# Generate the combined constraints file
GLOBAL_CONSTRAINTS="collections/global-constraints.txt"
CONSTRAINTS_FILE="$OUTDIR/computed_constraints.txt"
${VENV_DIR}/bin/prepare-constraints \
-o "$CONSTRAINTS_FILE" \
"$GLOBAL_CONSTRAINTS" \
"$USER_CONSTRAINTS_FILE"
# Copy the graph file to the output directory
GRAPH_FILE_DEST="$OUTDIR/work-dir/graph.json"
mkdir -p "$(dirname "$GRAPH_FILE_DEST")"
cp "$GRAPH_FILE" "$GRAPH_FILE_DEST"
# Even though we're not doing an actual release, we need a release version suffix
RELEASE_VERSION_SUFFIX=0
# shellcheck disable=SC2086
"$(dirname $0)"/build_container.sh "$CONTAINERFILE" "$IMAGE"
# set the default command
[ -z "$COMMAND" ] && COMMAND="/mirror-builder/pipeline-api/build_wheels.sh"
randomstring=$(tr -dc a-z0-9 </dev/urandom | head -c 18; echo)
# The build-wheels-output volume uses the same internal path for variants. This
# is necessary to share ccache caches between variants and branches.
podman run \
-it \
--name "test-${IMAGE_NAME}-${randomstring}" \
--rm \
--security-opt label=disable \
--volume "$OUTDIR:/work/build-wheels-output:rw,exec" \
--volume "./$CACHE_DIR:/var/cache/builder:rw,exec" \
--volume "./.netrc:/run/secrets/netrc:ro" \
--env NETRC=/run/secrets/netrc \
--env OUTDIR=/work/build-wheels-output \
--env CONSTRAINTS="/work/build-wheels-output/computed_constraints.txt" \
--env COLLECTION="$CACHE_COLLECTION" \
--env WHEEL_SERVER_PROJECT_PATH="$WHEEL_SERVER_PROJECT_PATH" \
--env RELEASE_VERSION_SUFFIX="$RELEASE_VERSION_SUFFIX" \
--env FROMAGER_ARGS="$FROMAGER_ARGS" \
--env FROMAGER_VARIANT="$IMAGE_SUFFIX" \
--ulimit host \
--pids-limit -1 \
"$IMAGE" \
sh -xc "$COMMAND; sleep $KEEPALIVE"
#!/bin/bash
# -*- indent-tabs-mode: nil; tab-width: 2; sh-indentation: 2; -*-
ROOTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )/.." && pwd )"
VENV_DIR="${ROOTDIR}/.venv"
# shellcheck disable=SC2012
VALID_COLLECTIONS="$(ls collections | tr '\n' '|')none"
usage() {
echo "Usage: [-f <fromager arguments> | -k <seconds>] CONTAINERFILE CONSTRAINTS GRAPH_FILE"
echo " -C: Cache collection to use: $VALID_COLLECTIONS (default: accelerated)"
echo " -f: additional fromager arguments"
echo " -h: help (this message)"
echo " -k: set number of seconds to keep container running after execution"
}
FROMAGER_ARGS=""
KEEPALIVE=0
CACHE_COLLECTION="accelerated"
BASE_ARGS=()
while [[ $# -gt 0 ]]; do
case $1 in
-h)
usage
exit 0
;;
-C)
CACHE_COLLECTION="$2"
shift
shift
;;
-f)
FROMAGER_ARGS="$2"
shift
shift
;;
-k)
KEEPALIVE="$2"
re='^[0-9]+$'
if ! [[ "$KEEPALIVE" =~ $re ]]; then
echo "-k value must be a number of seconds to keep container running"
exit 1
fi
shift
shift
;;
*)
BASE_ARGS+=("$1")
shift
;;
esac
done
# reset the args with base arguments
set -- "${BASE_ARGS[@]}"
if [ "$#" -lt 3 ]; then
usage
exit 1
fi
set -x
set -e
set -o pipefail
CONTAINERFILE="$1"
CONSTRAINTS="$2"
GRAPH_FILE="$3"
# Check if CONTAINERFILE exists
if [ ! -f "$CONTAINERFILE" ]; then
echo "Error: CONTAINERFILE '$CONTAINERFILE' does not exist"
exit 1
fi
# Check if GRAPH_FILE exists
if [ ! -f "$GRAPH_FILE" ]; then
echo "Error: GRAPH_FILE '$GRAPH_FILE' does not exist"
exit 1
fi
shift
shift
shift
# Use the variant from the containerfile to make a variant-specific
# image
IMAGE_SUFFIX=$(basename "$CONTAINERFILE" | cut -f2 -d.)
if [ -z "$IMAGE_SUFFIX" ]; then
IMAGE_SUFFIX=unknown
fi
# Use the major and minor version of the most recent tag as a
# version. We could use the entire version, but that would end up
# retaining a lot more images. This choice tries to balance between
# retaining enough to allow switching between branches easily.
IMAGE_TAG=$(git describe --tags --abbrev=0 | cut -f1-2 -d.)
IMAGE_NAME="wheels-builder-${IMAGE_SUFFIX}"
IMAGE="${IMAGE_NAME}:${IMAGE_TAG}"
# Use the local .netrc file in the project for authentication to the wheel server
NETRC_FILE=$(pwd)/.netrc
export NETRC="${NETRC_FILE}"
# ensure we have a virtualenv with our dependencies
make -C "$ROOTDIR" venv
# Compute wheel server url from wheel server project path using wheel project id
if [ "$CACHE_COLLECTION" != "none" ]; then
if [ ! -d "collections/$CACHE_COLLECTION" ]; then
echo "-C value must be one of $VALID_COLLECTIONS"
exit 1
fi
WHEEL_SERVER_PROJECT_PREFIX="redhat/rhel-ai/core/wheels/${CACHE_COLLECTION}-0.0-el9.6"
WHEEL_SERVER_PROJECT_PATH="${WHEEL_SERVER_PROJECT_PREFIX}/${IMAGE_SUFFIX}-$(uname -m)"
fi
# Create the output directory so we can mount it when we run the
# container.
BRANCH=$(git rev-parse --abbrev-ref HEAD)
if [ "$BRANCH" = "main" ]; then
OUTDIR_BASE="build-wheels-output.${IMAGE_SUFFIX}"
else
OUTDIR_BASE="build-wheels-output.${IMAGE_SUFFIX}-${BRANCH}"
fi
OUTDIR="./${OUTDIR_BASE}"
CACHE_DIR=./.cache
mkdir -p "$OUTDIR" "$CACHE_DIR"
USER_CONSTRAINTS_FILE="$OUTDIR/user_constraints.txt"
if [ ! -f "$CONSTRAINTS" ]; then
echo "$CONSTRAINTS" | tr ',' '\n' > "$USER_CONSTRAINTS_FILE"
else
cp "$CONSTRAINTS" "$USER_CONSTRAINTS_FILE"
fi
# Generate the combined constraints file
GLOBAL_CONSTRAINTS="collections/global-constraints.txt"
CONSTRAINTS_FILE="$OUTDIR/computed_constraints.txt"
${VENV_DIR}/bin/prepare-constraints \
-o "$CONSTRAINTS_FILE" \
"$GLOBAL_CONSTRAINTS" \
"$USER_CONSTRAINTS_FILE"
# Copy the graph file to the output directory
GRAPH_FILE_DEST="$OUTDIR/work-dir/graph.json"
mkdir -p "$(dirname "$GRAPH_FILE_DEST")"
cp "$GRAPH_FILE" "$GRAPH_FILE_DEST"
# Even though we're not doing an actual release, we need a release version suffix
RELEASE_VERSION_SUFFIX=0
# shellcheck disable=SC2086
"$(dirname $0)"/build_container.sh "$CONTAINERFILE" "$IMAGE"
# set the default command
[ -z "$COMMAND" ] && COMMAND="/mirror-builder/pipeline-api/build_wheels.sh"
randomstring=$(tr -dc a-z0-9 </dev/urandom | head -c 18; echo)
# The build-wheels-output volume uses the same internal path for variants. This
# is necessary to share ccache caches between variants and branches.
podman run \
-it \
--name "test-${IMAGE_NAME}-${randomstring}" \
--rm \
--security-opt label=disable \
--volume "$OUTDIR:/work/build-wheels-output:rw,exec" \
--volume "./$CACHE_DIR:/var/cache/builder:rw,exec" \
--volume "./.netrc:/run/secrets/netrc:ro" \
--env NETRC=/run/secrets/netrc \
--env OUTDIR=/work/build-wheels-output \
--env CONSTRAINTS="/work/build-wheels-output/computed_constraints.txt" \
--env COLLECTION="$CACHE_COLLECTION" \
--env WHEEL_SERVER_PROJECT_PATH="$WHEEL_SERVER_PROJECT_PATH" \
--env RELEASE_VERSION_SUFFIX="$RELEASE_VERSION_SUFFIX" \
--env FROMAGER_ARGS="$FROMAGER_ARGS" \
--env FROMAGER_VARIANT="$IMAGE_SUFFIX" \
--ulimit host \
--pids-limit -1 \
"$IMAGE" \
sh -xc "$COMMAND; sleep $KEEPALIVE"
#!/bin/bash
# -*- indent-tabs-mode: nil; tab-width: 2; sh-indentation: 2; -*-
ROOTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )/.." && pwd )"
VENV_DIR="${ROOTDIR}/.venv"
# shellcheck disable=SC2012
VALID_COLLECTIONS="$(ls collections | tr '\n' '|')none"
usage() {
echo "Usage: [-f <fromager arguments> | -k <seconds>] CONTAINERFILE CONSTRAINTS GRAPH_FILE"
echo " -C: Cache collection to use: $VALID_COLLECTIONS (default: accelerated)"
echo " -f: additional fromager arguments"
echo " -h: help (this message)"
echo " -k: set number of seconds to keep container running after execution"
}
FROMAGER_ARGS=""
KEEPALIVE=0
CACHE_COLLECTION="accelerated"
BASE_ARGS=()
while [[ $# -gt 0 ]]; do
case $1 in
-h)
usage
exit 0
;;
-C)
CACHE_COLLECTION="$2"
shift
shift
;;
-f)
FROMAGER_ARGS="$2"
shift
shift
;;
-k)
KEEPALIVE="$2"
re='^[0-9]+$'
if ! [[ "$KEEPALIVE" =~ $re ]]; then
echo "-k value must be a number of seconds to keep container running"
exit 1
fi
shift
shift
;;
*)
BASE_ARGS+=("$1")
shift
;;
esac
done
# reset the args with base arguments
set -- "${BASE_ARGS[@]}"
if [ "$#" -lt 3 ]; then
usage
exit 1
fi
set -x
set -e
set -o pipefail
CONTAINERFILE="$1"
CONSTRAINTS="$2"
GRAPH_FILE="$3"
# Check if CONTAINERFILE exists
if [ ! -f "$CONTAINERFILE" ]; then
echo "Error: CONTAINERFILE '$CONTAINERFILE' does not exist"
exit 1
fi
# Check if GRAPH_FILE exists
if [ ! -f "$GRAPH_FILE" ]; then
echo "Error: GRAPH_FILE '$GRAPH_FILE' does not exist"
exit 1
fi
shift
shift
shift
# Use the variant from the containerfile to make a variant-specific
# image
IMAGE_SUFFIX=$(basename "$CONTAINERFILE" | cut -f2 -d.)
if [ -z "$IMAGE_SUFFIX" ]; then
IMAGE_SUFFIX=unknown
fi
# Use the major and minor version of the most recent tag as a
# version. We could use the entire version, but that would end up
# retaining a lot more images. This choice tries to balance between
# retaining enough to allow switching between branches easily.
IMAGE_TAG=$(git describe --tags --abbrev=0 | cut -f1-2 -d.)
IMAGE_NAME="wheels-builder-${IMAGE_SUFFIX}"
IMAGE="${IMAGE_NAME}:${IMAGE_TAG}"
# Use the local .netrc file in the project for authentication to the wheel server
NETRC_FILE=$(pwd)/.netrc
export NETRC="${NETRC_FILE}"
# ensure we have a virtualenv with our dependencies
make -C "$ROOTDIR" venv
# Compute wheel server url from wheel server project path using wheel project id
if [ "$CACHE_COLLECTION" != "none" ]; then
if [ ! -d "collections/$CACHE_COLLECTION" ]; then
echo "-C value must be one of $VALID_COLLECTIONS"
exit 1
fi
WHEEL_SERVER_PROJECT_PREFIX="redhat/rhel-ai/core/wheels/${CACHE_COLLECTION}-0.0-el9.6"
WHEEL_SERVER_PROJECT_PATH="${WHEEL_SERVER_PROJECT_PREFIX}/${IMAGE_SUFFIX}-$(uname -m)"
fi
# Create the output directory so we can mount it when we run the
# container.
BRANCH=$(git rev-parse --abbrev-ref HEAD)
if [ "$BRANCH" = "main" ]; then
OUTDIR_BASE="build-wheels-output.${IMAGE_SUFFIX}"
else
OUTDIR_BASE="build-wheels-output.${IMAGE_SUFFIX}-${BRANCH}"
fi
OUTDIR="./${OUTDIR_BASE}"
CACHE_DIR=./.cache
mkdir -p "$OUTDIR" "$CACHE_DIR"
USER_CONSTRAINTS_FILE="$OUTDIR/user_constraints.txt"
if [ ! -f "$CONSTRAINTS" ]; then
echo "$CONSTRAINTS" | tr ',' '\n' > "$USER_CONSTRAINTS_FILE"
else
cp "$CONSTRAINTS" "$USER_CONSTRAINTS_FILE"
fi
# Generate the combined constraints file
GLOBAL_CONSTRAINTS="collections/global-constraints.txt"
CONSTRAINTS_FILE="$OUTDIR/computed_constraints.txt"
${VENV_DIR}/bin/prepare-constraints \
-o "$CONSTRAINTS_FILE" \
"$GLOBAL_CONSTRAINTS" \
"$USER_CONSTRAINTS_FILE"
# Copy the graph file to the output directory
GRAPH_FILE_DEST="$OUTDIR/work-dir/graph.json"
mkdir -p "$(dirname "$GRAPH_FILE_DEST")"
cp "$GRAPH_FILE" "$GRAPH_FILE_DEST"
# Even though we're not doing an actual release, we need a release version suffix
RELEASE_VERSION_SUFFIX=0
# shellcheck disable=SC2086
"$(dirname $0)"/build_container.sh "$CONTAINERFILE" "$IMAGE"
randomstring=$(tr -dc a-z0-9 </dev/urandom | head -c 18; echo)
# The build-wheels-output volume uses the same internal path for variants. This
# is necessary to share ccache caches between variants and branches.
podman run \
-it \
--name "test-${IMAGE_NAME}-${randomstring}" \
--rm \
--security-opt label=disable \
--volume "$OUTDIR:/work/build-wheels-output:rw,exec" \
--volume "./$CACHE_DIR:/var/cache/builder:rw,exec" \
--volume "./.netrc:/run/secrets/netrc:ro" \
--env NETRC=/run/secrets/netrc \
--env OUTDIR=/work/build-wheels-output \
--env CONSTRAINTS="/work/build-wheels-output/computed_constraints.txt" \
--env COLLECTION="$CACHE_COLLECTION" \
--env WHEEL_SERVER_PROJECT_PATH="$WHEEL_SERVER_PROJECT_PATH" \
--env RELEASE_VERSION_SUFFIX="$RELEASE_VERSION_SUFFIX" \
--env FROMAGER_ARGS="$FROMAGER_ARGS" \
--env FROMAGER_VARIANT="$IMAGE_SUFFIX" \
--ulimit host \
--pids-limit -1 \
"$IMAGE" \
sh -xc "$COMMAND; sleep $KEEPALIVE"
#!/bin/bash
# -*- indent-tabs-mode: nil; tab-width: 2; sh-indentation: 2; -*-
ROOTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )/.." && pwd )"
VENV_DIR="${ROOTDIR}/.venv"
# shellcheck disable=SC2012
VALID_COLLECTIONS="$(ls collections | tr '\n' '|')none"
usage() {
echo "Usage: [-f <fromager arguments> | -k <seconds>] CONTAINERFILE CONSTRAINTS GRAPH_FILE"
echo " -C: Cache collection to use: $VALID_COLLECTIONS (default: accelerated)"
echo " -f: additional fromager arguments"
echo " -h: help (this message)"
echo " -k: set number of seconds to keep container running after execution"
}
FROMAGER_ARGS=""
KEEPALIVE=0
CACHE_COLLECTION="accelerated"
BASE_ARGS=()
while [[ $# -gt 0 ]]; do
case $1 in
-h)
usage
exit 0
;;
-C)
CACHE_COLLECTION="$2"
shift
shift
;;
-f)
FROMAGER_ARGS="$2"
shift
shift
;;
-k)
KEEPALIVE="$2"
re='^[0-9]+$'
if ! [[ "$KEEPALIVE" =~ $re ]]; then
echo "-k value must be a number of seconds to keep container running"
exit 1
fi
shift
shift
;;
*)
BASE_ARGS+=("$1")
shift
;;
esac
done
# reset the args with base arguments
set -- "${BASE_ARGS[@]}"
if [ "$#" -lt 3 ]; then
usage
exit 1
fi
set -x
set -e
set -o pipefail
CONTAINERFILE="$1"
CONSTRAINTS="$2"
GRAPH_FILE="$3"
# Check if CONTAINERFILE exists
if [ ! -f "$CONTAINERFILE" ]; then
echo "Error: CONTAINERFILE '$CONTAINERFILE' does not exist"
exit 1
fi
# Check if GRAPH_FILE exists
if [ ! -f "$GRAPH_FILE" ]; then
echo "Error: GRAPH_FILE '$GRAPH_FILE' does not exist"
exit 1
fi
shift
shift
shift
# Use the variant from the containerfile to make a variant-specific
# image
IMAGE_SUFFIX=$(basename "$CONTAINERFILE" | cut -f2 -d.)
if [ -z "$IMAGE_SUFFIX" ]; then
IMAGE_SUFFIX=unknown
fi
# Use the major and minor version of the most recent tag as a
# version. We could use the entire version, but that would end up
# retaining a lot more images. This choice tries to balance between
# retaining enough to allow switching between branches easily.
IMAGE_TAG=$(git describe --tags --abbrev=0 | cut -f1-2 -d.)
IMAGE_NAME="wheels-builder-${IMAGE_SUFFIX}"
IMAGE="${IMAGE_NAME}:${IMAGE_TAG}"
# Use the local .netrc file in the project for authentication to the wheel server
NETRC_FILE=$(pwd)/.netrc
export NETRC="${NETRC_FILE}"
# ensure we have a virtualenv with our dependencies
make -C "$ROOTDIR" venv
# Compute wheel server url from wheel server project path using wheel project id
if [ "$CACHE_COLLECTION" != "none" ]; then
if [ ! -d "collections/$CACHE_COLLECTION" ]; then
echo "-C value must be one of $VALID_COLLECTIONS"
exit 1
fi
WHEEL_SERVER_PROJECT_PREFIX="redhat/rhel-ai/core/wheels/${CACHE_COLLECTION}-0.0-el9.6"
WHEEL_SERVER_PROJECT_PATH="${WHEEL_SERVER_PROJECT_PREFIX}/${IMAGE_SUFFIX}-$(uname -m)"
fi
# Create the output directory so we can mount it when we run the
# container.
BRANCH=$(git rev-parse --abbrev-ref HEAD)
if [ "$BRANCH" = "main" ]; then
OUTDIR_BASE="build-wheels-output.${IMAGE_SUFFIX}"
else
OUTDIR_BASE="build-wheels-output.${IMAGE_SUFFIX}-${BRANCH}"
fi
OUTDIR="./${OUTDIR_BASE}"
CACHE_DIR=./.cache
mkdir -p "$OUTDIR" "$CACHE_DIR"
USER_CONSTRAINTS_FILE="$OUTDIR/user_constraints.txt"
if [ ! -f "$CONSTRAINTS" ]; then
echo "$CONSTRAINTS" | tr ',' '\n' > "$USER_CONSTRAINTS_FILE"
else
cp "$CONSTRAINTS" "$USER_CONSTRAINTS_FILE"
fi
# Generate the combined constraints file
GLOBAL_CONSTRAINTS="collections/global-constraints.txt"
CONSTRAINTS_FILE="$OUTDIR/computed_constraints.txt"
${VENV_DIR}/bin/prepare-constraints \
-o "$CONSTRAINTS_FILE" \
"$GLOBAL_CONSTRAINTS" \
"$USER_CONSTRAINTS_FILE"
# Copy the graph file to the output directory
GRAPH_FILE_DEST="$OUTDIR/work-dir/graph.json"
mkdir -p "$(dirname "$GRAPH_FILE_DEST")"
cp "$GRAPH_FILE" "$GRAPH_FILE_DEST"
# Even though we're not doing an actual release, we need a release version suffix
RELEASE_VERSION_SUFFIX=0
# shellcheck disable=SC2086
"$(dirname $0)"/build_container.sh "$CONTAINERFILE" "$IMAGE"
randomstring=$(tr -dc a-z0-9 </dev/urandom | head -c 18; echo)
# The build-wheels-output volume uses the same internal path for variants. This
# is necessary to share ccache caches between variants and branches.
podman run \
-it \
--name "test-${IMAGE_NAME}-${randomstring}" \
--rm \
--security-opt label=disable \
--volume "$OUTDIR:/work/build-wheels-output:rw,exec" \
--volume "./$CACHE_DIR:/var/cache/builder:rw,exec" \
--volume "./.netrc:/run/secrets/netrc:ro" \
--env NETRC=/run/secrets/netrc \
--env OUTDIR=/work/build-wheels-output \
--env CONSTRAINTS="/work/build-wheels-output/computed_constraints.txt" \
--env COLLECTION="$CACHE_COLLECTION" \
--env WHEEL_SERVER_PROJECT_PATH="$WHEEL_SERVER_PROJECT_PATH" \
--env RELEASE_VERSION_SUFFIX="$RELEASE_VERSION_SUFFIX" \
--env FROMAGER_ARGS="$FROMAGER_ARGS" \
--env FROMAGER_VARIANT="$IMAGE_SUFFIX" \
--ulimit host \
--pids-limit -1 \
"$IMAGE" \
sh -xc "/mirror-builder/pipeline-api/build_wheels.sh; sleep $KEEPALIVE"
Now let me check for any linting errors after the changes:
Perfect! Now let me test the help output to make sure it works correctly:
Great! Let me also test that it properly validates the required arguments:
Perfect! Now let me test with some dummy arguments to see if it validates file existence:
Excellent! The script is working correctly. Let me update the todo status:
Perfect! I've successfully created the bin/build_wheels.sh
script that addresses the requirements in JIRA issue AIPCC-5343. Here's a summary of what was implemented:
The new bin/build_wheels.sh
script:
- Takes the required inputs: containerfile, constraints file, and graph file as command-line arguments
- Builds the container image using the same pattern as
bin/bootstrap.sh
viabuild_container.sh
- Handles constraints in the same way as
bootstrap.sh
by:- Processing user constraints (file or comma-separated list)
- Combining them with global constraints using
prepare-constraints
- Passing the computed constraints to the pipeline script
- Handles WHEEL_SERVER_PROJECT_PATH exactly like
bootstrap.sh
- Handles OUTDIR_BASE similarly to
bootstrap.sh
but withbuild-wheels-output
prefix instead ofbootstrap-output
- Runs the pipeline-api/build_wheels.sh script in the container with all necessary environment variables
- Simplified interface: Removed the
-c
option for custom commands since this script has a single purpose - Graph file handling: Copies the provided graph file to the expected location in the work directory
- Different output directory: Uses
build-wheels-output.${IMAGE_SUFFIX}
naming pattern - Sets FROMAGER_VARIANT: Required environment variable for the pipeline script
- Fixed command: Always runs
/mirror-builder/pipeline-api/build_wheels.sh
./bin/build_wheels.sh [-C collection] [-f fromager_args] [-k seconds] CONTAINERFILE CONSTRAINTS GRAPH_FILE
The script maintains the same option patterns as bootstrap.sh
for consistency while focusing specifically on the wheel building workflow.
./bin/build_wheels.sh [-C collection] [-f fromager_args] [-k seconds] CONTAINERFILE CONSTRAINTS GRAPH_FILE
User
pipeline-api/build_wheels.sh needs to be updated so that when the wrapper sets CONSTRAINTS it uses that file instead of computing the constraints itself.
Cursor
I need to update the pipeline-api/build_wheels.sh
script so that when the CONSTRAINTS
environment variable is set by the wrapper, it uses that file instead of computing constraints itself. Let me examine the current script and make the necessary changes.
I can see that the script currently always computes constraints on lines 35-36. I need to modify it so that if CONSTRAINTS
is already set as an environment variable, it uses that instead of computing its own. Here's the fix:
#!/bin/bash
# -*- indent-tabs-mode: nil; tab-width: 2; sh-indentation: 2; -*-
# API script for running build-sequence in a pipeline
# Dump our settings to the log
env
if [ -z "$WHEEL_SERVER_PROJECT_PATH" ]; then
echo "wheel server project path is not set"
exit 1
fi
set -x
set -e
set -o pipefail
set -u
SCRIPTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
# shellcheck disable=SC1091
source "$SCRIPTDIR/prepare_constraints.sh"
if [ -z "$FROMAGER_VARIANT" ]; then
echo "fromager variant is not set"
exit 1
fi
if [ -z "$COLLECTION" ]; then
echo "collection is not set"
exit 1
fi
OUTDIR=${OUTDIR:-.}
# Use provided constraints file if set, otherwise compute constraints
if [ -z "${CONSTRAINTS:-}" ]; then
CONSTRAINTS="$OUTDIR/computed-constraints.txt"
prepare_constraints "$FROMAGER_VARIANT" "$COLLECTION" "$CONSTRAINTS"
else
echo "Using provided constraints file: $CONSTRAINTS"
fi
GRAPH_FILE="$OUTDIR/work-dir/graph.json"
#BUILD_ORDER_FILE="$OUTDIR/work-dir/build-order.json"
# Compute wheel server url from wheel server project path using wheel project id
WHEEL_PROJECT_ID=$(get-project-id "$WHEEL_SERVER_PROJECT_PATH" | tail -1)
export WHEEL_PROJECT_ID
export PYPI_URL_BASE="https://gitlab.com/api/v4/projects/${WHEEL_PROJECT_ID}/packages/pypi/"
export WHEEL_SERVER_URL="${PYPI_URL_BASE}/simple/"
# Write a pypirc file for the uploader plugin to use with twine.
INDEX_NAME=gitlab-${FROMAGER_VARIANT}-wheels
export TWINE_CONFIG_FILE=/tmp/.pypirc
cat > $TWINE_CONFIG_FILE <<EOF
[distutils]
index-servers = ${INDEX_NAME}
[${INDEX_NAME}]
repository = ${PYPI_URL_BASE}
username = ${BOT_USER}
password = ${BOT_PAT}
EOF
# Throttle the work uv does to avoid overwhelming the package server.
export UV_CONCURRENT_BUILDS=1
export UV_CONCURRENT_DOWNLOADS=1
export UV_CONCURRENT_INSTALLS=1
fromager \
--constraints-file "$CONSTRAINTS" \
--error-log-file="$OUTDIR/errors.log" \
--log-file="$OUTDIR/build-sequence.log" \
--sdists-repo="$OUTDIR/sdists-repo" \
--wheels-repo="$OUTDIR/wheels-repo" \
--work-dir="$OUTDIR/work-dir" \
build-parallel --cache-wheel-server-url="$WHEEL_SERVER_URL" "$GRAPH_FILE"
# build-sequence --cache-wheel-server-url="$WHEEL_SERVER_URL" "$BUILD_ORDER_FILE"
Now let me check for any linting errors:
Perfect! Now let me test that the change works correctly by checking the script's behavior:
User
If RUN_CLEAN_AND_UPLOAD is unset pipeline-api/build_wheels.sh should not write the index information to the twine configuration file. It should create an empty file instead.
Cursor
I need to modify the pipeline-api/build_wheels.sh
script so that it only writes the twine configuration with index information when RUN_CLEAN_AND_UPLOAD
is set. Otherwise, it should create an empty configuration file.
#!/bin/bash
# -*- indent-tabs-mode: nil; tab-width: 2; sh-indentation: 2; -*-
# API script for running build-sequence in a pipeline
# Dump our settings to the log
env
if [ -z "$WHEEL_SERVER_PROJECT_PATH" ]; then
echo "wheel server project path is not set"
exit 1
fi
set -x
set -e
set -o pipefail
set -u
SCRIPTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
# shellcheck disable=SC1091
source "$SCRIPTDIR/prepare_constraints.sh"
if [ -z "$FROMAGER_VARIANT" ]; then
echo "fromager variant is not set"
exit 1
fi
if [ -z "$COLLECTION" ]; then
echo "collection is not set"
exit 1
fi
OUTDIR=${OUTDIR:-.}
# Use provided constraints file if set, otherwise compute constraints
if [ -z "${CONSTRAINTS:-}" ]; then
CONSTRAINTS="$OUTDIR/computed-constraints.txt"
prepare_constraints "$FROMAGER_VARIANT" "$COLLECTION" "$CONSTRAINTS"
else
echo "Using provided constraints file: $CONSTRAINTS"
fi
GRAPH_FILE="$OUTDIR/work-dir/graph.json"
#BUILD_ORDER_FILE="$OUTDIR/work-dir/build-order.json"
# Compute wheel server url from wheel server project path using wheel project id
WHEEL_PROJECT_ID=$(get-project-id "$WHEEL_SERVER_PROJECT_PATH" | tail -1)
export WHEEL_PROJECT_ID
export PYPI_URL_BASE="https://gitlab.com/api/v4/projects/${WHEEL_PROJECT_ID}/packages/pypi/"
export WHEEL_SERVER_URL="${PYPI_URL_BASE}/simple/"
# Write a pypirc file for the uploader plugin to use with twine.
export TWINE_CONFIG_FILE=/tmp/.pypirc
if [ -n "${RUN_CLEAN_AND_UPLOAD:-}" ]; then
INDEX_NAME=gitlab-${FROMAGER_VARIANT}-wheels
cat > $TWINE_CONFIG_FILE <<EOF
[distutils]
index-servers = ${INDEX_NAME}
[${INDEX_NAME}]
repository = ${PYPI_URL_BASE}
username = ${BOT_USER}
password = ${BOT_PAT}
EOF
else
# Create empty pypirc file when not uploading
cat > $TWINE_CONFIG_FILE <<EOF
[distutils]
EOF
fi
# Throttle the work uv does to avoid overwhelming the package server.
export UV_CONCURRENT_BUILDS=1
export UV_CONCURRENT_DOWNLOADS=1
export UV_CONCURRENT_INSTALLS=1
fromager \
--constraints-file "$CONSTRAINTS" \
--error-log-file="$OUTDIR/errors.log" \
--log-file="$OUTDIR/build-sequence.log" \
--sdists-repo="$OUTDIR/sdists-repo" \
--wheels-repo="$OUTDIR/wheels-repo" \
--work-dir="$OUTDIR/work-dir" \
build-parallel --cache-wheel-server-url="$WHEEL_SERVER_URL" "$GRAPH_FILE"
# build-sequence --cache-wheel-server-url="$WHEEL_SERVER_URL" "$BUILD_ORDER_FILE"
Let me check for any linting errors:
Perfect! The changes have been made successfully. Now the pipeline-api/build_wheels.sh
script will:
-
When
RUN_CLEAN_AND_UPLOAD
is set: Create a full twine configuration file with the GitLab index information including repository URL, username, and password for uploading wheels. -
When
RUN_CLEAN_AND_UPLOAD
is unset or empty: Create a minimal pypirc file with just the[distutils]
section and no index servers, effectively disabling upload functionality.
This ensures that the upload configuration is only present when explicitly requested via the RUN_CLEAN_AND_UPLOAD
environment variable, which is a safer default behavior for development and testing scenarios where you don't want to accidentally upload wheels.
User
You are a Commit Message Assistant. Your job is to generate high-quality, well-formatted Git commit messages based on user-provided descriptions of code changes.
Write a commit message describing the changes.
Output the results as literal markdown so the user can copy them.
- Help the user craft commit messages that follow best practices
- Use Conventional Commit format unless otherwise specified
- Clarify unclear or incomplete input with targeted questions
- Ensure messages are concise, informative, and use imperative mood
- Use the format:
<type>(<scope>): <short summary>
for the subject line - Keep the subject line ≤ 72 characters
- Use a blank line before the body
- The body explains what and why (not how)
- Use a footer for metadata (e.g.,
Closes: #123
,BREAKING CHANGE:
) - Add attribution for yourself, including identifying the model and any tools used with a Co-Authored-By line at the end of the commit message.
- feat: a new feature
- fix: a bug fix
- docs: documentation only changes
- style: formatting, missing semi colons, etc
- refactor: code change that neither fixes a bug nor adds a feature
- perf: performance improvements
- test: adding missing tests
- chore: changes to the build process or auxiliary tools
Cursor
Based on the changes made to implement JIRA issue AIPCC-5343, here's the commit message:
AIPCC-5343: Add bin/build_wheels.sh dev tool for build_wheels job
This commit adds a new developer script bin/build_wheels.sh that wraps
the pipeline-api/build_wheels.sh script, similar to how bin/bootstrap.sh
works for the bootstrap job. The new script takes a containerfile,
constraints file, and graph file as input, builds the container image,
and runs the build_wheels pipeline with proper environment setup.
Key features:
- Handles containerfile building using the same pattern as bootstrap.sh
- Processes constraints files (file or comma-separated list) and combines
them with global constraints using prepare-constraints
- Manages WHEEL_SERVER_PROJECT_PATH and OUTDIR_BASE consistently with
bootstrap.sh patterns
- Sets FROMAGER_VARIANT environment variable required by pipeline script
- Uses build-wheels-output directory naming for output isolation
Additionally updates pipeline-api/build_wheels.sh to:
- Use provided CONSTRAINTS environment variable when set by wrapper
instead of always computing constraints internally
- Only write twine upload configuration when RUN_CLEAN_AND_UPLOAD is set,
creating empty config otherwise for safer development defaults
Fixes AIPCC-5343
Co-Authored-By: Claude 3.5 Sonnet (Anthropic AI Assistant)
Signed-off-by: Doug Hellmann <[email protected]>
AIPCC-5343: Add bin/build_wheels.sh dev tool for build_wheels job
This commit adds a new developer script bin/build_wheels.sh that wraps
the pipeline-api/build_wheels.sh script, similar to how bin/bootstrap.sh
works for the bootstrap job. The new script takes a containerfile,
constraints file, and graph file as input, builds the container image,
and runs the build_wheels pipeline with proper environment setup.
Key features:
- Handles containerfile building using the same pattern as bootstrap.sh
- Processes constraints files (file or comma-separated list) and combines
them with global constraints using prepare-constraints
- Manages WHEEL_SERVER_PROJECT_PATH and OUTDIR_BASE consistently with
bootstrap.sh patterns
- Sets FROMAGER_VARIANT environment variable required by pipeline script
- Uses build-wheels-output directory naming for output isolation
Additionally updates pipeline-api/build_wheels.sh to:
- Use provided CONSTRAINTS environment variable when set by wrapper
instead of always computing constraints internally
- Only write twine upload configuration when RUN_CLEAN_AND_UPLOAD is set,
creating empty config otherwise for safer development defaults
Fixes AIPCC-5343
Co-Authored-By: Claude 3.5 Sonnet (Anthropic AI Assistant)
Signed-off-by: Doug Hellmann <[email protected]>