Skip to content

Instantly share code, notes, and snippets.

@ddunbar

ddunbar/deployable-artifacts.md

Last active June 27, 2021 07:46
Show Gist options
  • Save ddunbar/401d41acb04bb8c10ee0a5a4f12dcddf to your computer and use it in GitHub Desktop.
Save ddunbar/401d41acb04bb8c10ee0a5a4f12dcddf to your computer and use it in GitHub Desktop.
Download ZIP
Deployable Artifacts thoughts
Raw
deployable-artifacts.md

My hopes for deployable artifacts:

  1. Packages would be able to just describe how the things they build should get laid out in a tarball.
  2. We would implement this as part of the build, it should be super fast (we might want to consider actually generating the tar directly for portability & control purposes).
  3. Packages could have control over the permissions inside the tar file (ideally without having to manipulate the permissions in the filesystem, problematic sometimes if root permission in the tar is desired).
  4. The mechanism would support knowing how to handle RPATH issues.
  5. The mechanism would know how to handle libswift* portability issues.
  6. The mechanism would know how to handle cross-Linux distro issues.
  7. Unlike below, I wouldn't expect the mechanism to handle the GIT REVISION stuff, that belongs in a separate proposal.
  8. The mechanism would support typical Unix-y things, like bin, lib, libexec, share.
  9. The mechanism could support vendored assets like other DSOs or frameworks that can't live in the SwiftPM world for some reason.

As a concrete example of my current life:

$ ls */*/Utilities/mkdist | wc -l
      35

Shame on me for not refactoring these into a single tool, but I knew I wanted to write such a proposal eventually and then let it drag on too long.

Almost all of these scripts look something like this:

#!/bin/bash
#
# Build a redistributable version, using SwiftPM.
#

if [ "$1" = "-h" -o "$1" = "--help" ]; then
    echo "usage: $0 [clean] | [<additional_build_arguments>]"
    echo
    echo "Targets:"
    echo "  clean     Remove distribution dir."
    echo
    echo "Important Environment Variables:"
    echo "  SRCROOT   Sources path (including cached .build/checkouts)."
    echo "  OBJROOT   Intermediate object path (where .build should be placed)."
    echo "  DSTROOT   Destination path (where products should be installed)."
    echo "  SYMROOT   Symbols path (where debug symbols should be placed)."
    exit 1
fi

set -e

# Package contents definitions
PKG_NAME=some-cool-name
PKG_BINARIES=(
    tool-1
    tool-2
)
PKG_LIBEXEC=(
    some-tool-daemon
)

BIN_PATH=bin
# FIXME: libexec should ultimately end up in libexec/$PKG_NAME or some
# such, but for now the tools expect them alongside in bin.
LIBEXEC_PATH=bin

if (uname | grep -qi darwin); then
    platform=darwin
elif (uname | grep -qi linux); then
    platform=linux
else
    echo "error: unknown platform: $(uname)"
    exit 1
fi

if [ -z "${CONFIGURATION}" ]; then
    CONFIGURATION=release
fi

# Externally overrideable paths
SRCROOT=${SRCROOT:-$(pwd)}
OBJROOT=${OBJROOT:-$(pwd)}

# Internally derived paths
BUILD_DIR=${OBJROOT}/.build
PRODUCT_DIR=${BUILD_DIR}/${CONFIGURATION}
STAGING_DIR=${BUILD_DIR}/dist
OUTPUT_PATH=${OBJROOT}/.releases

# FIXME: This actually isn't that common, but is what we do sometimes to
# workaround e.g. lack of parameterized build flags and ad hoc actions.
if [ -n "${GIT_HASH}" ]; then
    PROJECT_REVISION="${GIT_HASH}-rio"
else
    PROJECT_REVISION="$(git rev-parse --short HEAD 2>/dev/null || echo \(development\))"

    # Annotate if the source directory has local modifications
    GIT_IS_CLEAN="$(git status --porcelain 2>/dev/null || true)"
    if [ -n "${GIT_IS_CLEAN}" ]; then
        PROJECT_REVISION="${PROJECT_REVISION}-dirty"
    else
        PROJECT_REVISION="${PROJECT_REVISION}-clean"
    fi
fi

# Set up the build arguments.
SWIFT_BUILD_ARGS=(
    -Xcc "-DPROJECT_VERSION=\"1.1-${PROJECT_REVISION}\""
)

if [ $platform = "darwin" ]; then
    SWIFT_BUILD_ARGS=(
        -Xswiftc -g -Xcc -g
        "${SWIFT_BUILD_ARGS[@]}"
        # -export_dynamic: https://github.com/swift-server/swift-backtrace
        -Xlinker -export_dynamic
    )

    # Add an rpath for swift standard libaries (also packaged with the binaries
    # below).  This is useful for building and deploying to systems that do not
    # include the ABI stable libraries...
    #
    # FIXME: This can probably be removed.
    if [[ -d "${SRCROOT}/lib/swift" || ! -z "${RC_XBS}" ]]; then
        SWIFT_BUILD_ARGS=(
            -Xlinker -rpath -Xlinker "@executable_path/../lib/swift"
            "${SWIFT_BUILD_ARGS[@]}"
    )
    fi

elif [ $platform = "linux" ]; then
    # Since --static-swift-stdlib doesn't yet work on Linux, we include the
    # runtime libraries in the archive.
    SWIFT_BUILD_ARGS=(
        # --export-dynamic: https://github.com/swift-server/swift-backtrace
        -Xlinker --export-dynamic
        -Xlinker -rpath -Xlinker "\$ORIGIN/../lib/swift/linux"
        # FIXME: The following allows task binaries to find the Swift runtime
        # libraries, when they are run on a Linux deployment. This is not
        # kosher, and needs to be fixed eventually...
        -Xlinker -rpath -Xlinker "/app/.swift/lib/swift/linux"
        "${SWIFT_BUILD_ARGS[@]}"
    )
else
    echo "error: unknown platform"
    exit 1
fi

if [ "$1" = "clean" ]; then
    set -e
    rm -rf "${STAGING_DIR}"
    exit 0
fi

echo "Running release script ..."
echo
echo "Building the project from:"
echo "  ${SRCROOT}"
echo

# Build the package
echo "Building the project in:"
echo "  ${BUILD_DIR}"
echo
if [ ! "${SRCROOT}" = "${OBJROOT}" ]; then
    # SwiftPM caches dependency sources in .build. When OBJROOT != SRCROOT,
    # we need to copy those files into the actual build dir.
    mkdir -p "${BUILD_DIR}"
    rsync -amr "${SRCROOT}/.build"/*-state.json "${SRCROOT}/.build/checkouts" "${BUILD_DIR}"
fi;

# Extra passed by the execution environment.
PASSED_BUILD_FLAGS="$@"

# By default build in release configuration.
: ${CONFIGURATION:=release}

while [ "$#" -gt 0 ]; do
    case "$1" in
      --configuration)
        shift
        CONFIGURATION=$1
        shift
        ;;
      --build-path)
        shift
        if [[ "${BUILD_DIR}" != "$1" ]]; then
            echo "WARN: Specified --build-path is different from inferred:"
            echo "SPECIFIED: $1"
            echo "INFERRED:  ${BUILD_DIR}"
        fi
        shift
        ;;
      --disable-sandbox)
        # Ignore this argument, we'll add it ourselves.
        shift
        ;;
      *) SWIFT_BUILD_ARGS+=("$1"); shift ;;
    esac
done
SWIFT_BUILD_ARGS+=(
                --disable-sandbox
                --configuration "${CONFIGURATION}"
                --build-path "${BUILD_DIR}")
set -x
swift build "${SWIFT_BUILD_ARGS[@]}"
set +x

# Create the release file layout
echo "Constructing distribution in:"
echo "  ${STAGING_DIR}"
echo

rm -rf "${STAGING_DIR}"

mkdir -p "${STAGING_DIR}/${PKG_NAME}/${BIN_PATH}"
for FILE in ${PKG_BINARIES[@]}; do
    cp "${PRODUCT_DIR}/${FILE}" "${STAGING_DIR}/${PKG_NAME}/${BIN_PATH}"
    if [ -n "${SYMROOT}" ]
    then
        strip -S "${STAGING_DIR}/${PKG_NAME}/${BIN_PATH}/${FILE}"
    fi
done
mkdir -p "${STAGING_DIR}/${PKG_NAME}/${LIBEXEC_PATH}"
for FILE in ${PKG_LIBEXEC[@]}; do
    cp "${PRODUCT_DIR}/${FILE}" "${STAGING_DIR}/${PKG_NAME}/${LIBEXEC_PATH}"
    if [ -n "${SYMROOT}" ]
    then
        strip -S "${STAGING_DIR}/${PKG_NAME}/${LIBEXEC_PATH}/${FILE}"
    fi
done

# Add in runtime libraries, for Linux.
case "$platform" in
  darwin)
    # FIXME: This is a hack when we end up having to vendor some Frameworks.
    if [ -d  "${SRCROOT}/Frameworks" ]; then
        mkdir -p "${STAGING_DIR}/${PKG_NAME}/lib/Frameworks"
        cp -r "${SRCROOT}/Frameworks/"* "${STAGING_DIR}/${PKG_NAME}/lib/Frameworks"
    fi
    if [ -d  "${SRCROOT}/lib/swift" ]; then
        mkdir -p "${STAGING_DIR}/${PKG_NAME}/lib/swift"
        cp -r "${SRCROOT}/lib/swift/"* "${STAGING_DIR}/${PKG_NAME}/lib/swift"
    fi
    ;;
  linux)
    # The resulting package will still require the following libraries:
    #
    #   apt-get install libatomic1 libcurl3 libicu52 libxml2
    SWIFT_LIB_PATH=/usr/lib/swift/linux
    mkdir -p "${STAGING_DIR}/${PKG_NAME}/lib/swift/linux"
    for LIB in swiftCore.so \
               swiftGlibc.so \
               swiftDispatch.so \
               swiftSwiftOnoneSupport.so \
               Foundation.so \
               FoundationNetworking.so \
               dispatch.so \
               BlocksRuntime.so \
               icui18nswift.so.65 \
               icuucswift.so.65 \
               icudataswift.so.65 ; do
        # FIXME: The if is just for Swift 5/5.2 stuff.
        if [ -f "${SWIFT_LIB_PATH}/lib${LIB}" ]; then
            cp "${SWIFT_LIB_PATH}/lib${LIB}" "${STAGING_DIR}/${PKG_NAME}/lib/swift/linux"
        fi
    done

    # Import the full list of runtime libraries we need, so we can run this on a
    # different Linux distribution.
    mkdir -p "${STAGING_DIR}/${PKG_NAME}/lib/swift/linux/all_libs"
    ldd ${STAGING_DIR}/${PKG_NAME}/${BIN_PATH}/* | grep -o "/[^ ]*" | grep -v ":$" | sort -u | xargs -I% cp % "${STAGING_DIR}/${PKG_NAME}/lib/swift/linux/all_libs/"
    cp /lib/x86_64-linux-gnu/libnss_files*.so* /lib/x86_64-linux-gnu/libnss_dns*.so* "${STAGING_DIR}/${PKG_NAME}/lib/swift/linux/all_libs/"
    cat > "${STAGING_DIR}/${PKG_NAME}/bin/start" << EOF
#!/usr/bin/env bash
MYPATH=\$(dirname "\$0")
export LD_LIBRARY_PATH="\$MYPATH"/../lib/swift/linux/all_libs
LD_LIBRARY_PATH="\$MYPATH"/../lib/swift/linux/all_libs exec "\$MYPATH"/../lib/swift/linux/all_libs/ld-linux-x86-64.so.2 --library-path "\$MYPATH"/../lib/swift/linux/all_libs "\$@"
EOF
    chmod +x "${STAGING_DIR}/${PKG_NAME}/bin/start"
    ;;
esac

# If SYMROOT defined, copy debug symbols
if [ -n "${SYMROOT}" ]
then
    echo "Copying symbols to:"
    echo "  ${SYMROOT}"
    echo
    mkdir -p "${SYMROOT}"
    for FILE in ${PKG_BINARIES[@]} ${PKG_LIBEXEC[@]}; do
        cp -r "${PRODUCT_DIR}/${FILE}.dSYM" "${SYMROOT}/"
    done
fi

# Package the products
echo "Packaging distribution to:"
echo "  ${OUTPUT_PATH}/${PKG_NAME}.tgz"
echo
mkdir -p "${OUTPUT_PATH}"
pushd "${STAGING_DIR}" 2>&1 >/dev/null
tar zcf "${OUTPUT_PATH}/${PKG_NAME}.tgz" "${PKG_NAME}"
popd 2>&1 >/dev/null

# If DSTROOT defined, install files
if [ -n "${DSTROOT}" ]
then
    echo "Installing in:"
    echo "  ${DSTROOT}"
    echo
    mkdir -p "${DSTROOT}"
    rsync -amr "${STAGING_DIR}/${PKG_NAME}/" "${DSTROOT}/"
fi

echo "Successfully built ${PKG_NAME} release."
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment