Consuming packages from a private Azure Pipelines Python artifact feed

README.md

Consuming Azure Pipelines Python artifact feeds in Docker

Recently, I was building out a set of Python packages and services and needed to find a way to pull down packages from an Azure Artifacts feed into a Docker image. It was straightforward to use the tasks to package an artifact, authenticate to the feed, and publish.

I had to do a bit more digging to piece together a flow I was comfortable with for building container images. This post describes some of the challenges involved and how I solved for them.

What's the problem?

The PipAuthenticate task is great - it authenticates with your artifacts feed and per the docs, will store the location of a config file that can be used to connect in the PYPIRC_PATH environment variable.

That said - by design, containers run in an isolated environment. We can't directly access it while building a container image. We need a way to get that config inside the build phase so that our calls to python -m pip install are successful. You are using a virtual environment & python -m pip install to install packages, right?

Challenge 1: No volumes at build time!

Docker doesn't currently support* mounting volumes at build time. So we can't just mount our PYPIRC_PATH file from the Azure Pipelines host into the build.

It would be much easier to pass a string as a --build-arg to Docker and then consume it. Azure Pipelines tasks are open source on GitHub, so I thought I'd take a look to see how the task worked and possibly extend it. It turns out that the PipAuthenticate task has some undocumented behavior bonus features and it already does what I want! It populates the PIP_EXTRA_INDEX_URL environment variable, which is automatically picked up by pip.

*Well, sort of! You can solve this with --mount=type=secret when you enable BuildKit. If this was a personal project, I'd have stopped there and said #shipit! In this case, I was really looking to find something that works for all users and isn't explicitly marked "experimental".

Challenge 2: Keep it secret, keep it safe!

Great! We pass in our build arg, set ENV PIP_EXTRA_INDEX_URL=$PIP_EXTRA_INDEX_URL and call it a day, right! Right...?

Not so fast - we want to have PIP_EXTRA_INDEX_URL available when we pull packages, but we don't want secret environment variables baked into any of the layers of a runtime image. So we'll combine what we've learned so far with a multi-stage build and we're off to the races!

Bonus!

In my real container build, I needed to install gcc, musl-dev, python3-dev and a bunch of other things to pull down my dependencies & build wheels - so a multi-stage build drops my final image size from >1GB down to ~100MB anyway

Wrapping up

I've attached a few sample files that I pulled from my working pipeline to get you started with this approach. I hope this helps and plan for this post to be soon obsolete after I complete a few pull requests into Microsoft docs! :)

azure-pipelines-docker.yml

trigger:
- master

pool:
  vmImage: ubuntu-16.04

variables:
  artifactFeed: myfeed                # the name of an Azure artifacts feed
  azureSubscription: mysubscription   # the name of an Azure Resource Manager Service Connection
  azureContainerRegistry: myregistry  # the name of an Azure Container Registry
  imageName: my-container

steps:
- task: AzureCLI@1
  displayName: Login to ACR
  inputs:
    azureSubscription: $(azureSubscription)
    scriptLocation: inlineScript
    inlineScript: |
      az acr login -n $(azureContainerRegistry)

# This task populates the PIP_EXTRA_INDEX_URL environment variable
# https://github.com/microsoft/azure-pipelines-tasks/blob/7eab2bc96011927a971f2613ce6e85d93ee9b3f1/Tasks/PipAuthenticateV0/pipauthenticatemain.ts#L60
- task: PipAuthenticate@0
  displayName: Authenticate with artifact feed
  inputs:
    artifactFeeds: $(artifactFeed)

# Docker build w/ a build arg to pass PIP_EXTRA_INDEX_URL into the build phase
- bash: |
    docker build \
      --build-arg 'INDEX_URL=$(PIP_EXTRA_INDEX_URL)' \
      -t $(azureContainerRegistry).azurecr.io/$(imageName):$(Build.BuildNumber) \
      -t $(azureContainerRegistry).azurecr.io/$(imageName):latest \
      .

    docker push $(azureContainerRegistry).azurecr.io/$(imageName):$(Build.BuildNumber)
    docker push $(azureContainerRegistry).azurecr.io/$(imageName):latest
  displayName: Build and push container

azure-pipelines-package.yml

trigger:
- master

pool:
  vmImage: ubuntu-16.04

variables:
  artifactFeed: myfeed                # the name of an Azure artifacts feed

steps:
- task: UsePythonVersion@0
  inputs:
    versionSpec: 3.6

- script: python -m pip install -U pip setuptools wheel twine
  displayName: Install build tools

- script: python setup.py bdist_wheel
  displayName: Build package

- task: TwineAuthenticate@0
  displayName: Configure twine authentication
  inputs:
    artifactFeeds: $(artifactFeed)

- script: twine upload -r $(artifactFeed) --config-file $(PYPIRC_PATH) dist/*
  displayName: Publish artifacts

Dockerfile

# We set an environment variable in this phase so it gets picked up by pip, but we don't want to bake secrets into our container image
FROM python:3.6-alpine AS builder

ARG INDEX_URL
ENV PIP_EXTRA_INDEX_URL=$INDEX_URL

COPY requirements.txt .

RUN pip install -U pip \
    && pip install --user -r requirements.txt

# We use a multistage build to start fresh (and drop PIP_EXTRA_INDEX_URL from our published image)
FROM python:3.6-alpine

COPY --from=builder /root/.local /root/.local
ENV PATH=/root/.local/bin:$PATH

WORKDIR /app
COPY app .

CMD ["python", "app.py"]

LICENSE

MIT License

Copyright (c) 2019 Noel Bundick

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Thank you for a really nice complete and conscise write-up :) It was very helpful. I am still amazed that the Azure DevOps docs currently have no information about this consumption pattern. It seems so obviously useful.

For reference, I have used a Docker task. Only thing to note is, contrary to @syonekura, it only worked without single quotes:

- task: Docker@1
          displayName: 'Build Docker Image'
          inputs:
            azureSubscriptionEndpoint: $(azreResourceManagerConnection)
            azureContainerRegistry: $(azureContainerRegistry)
            dockerFile: Dockerfile
            imageName: '$(prefix)/$(name):$(version)'
            arguments: --build-arg INDEX_URL=$(PIP_EXTRA_INDEX_URL)

and from the logs:

docker build -f .../Dockerfile ... --build-arg INDEX_URL=https://build:***@.../_packaging/.../pypi/simple

Hi, how do you get PIP_EXTRA_INDEX_URL manually? I want to build the docker container on my machine before doing builds on pipelines. I can't seem to find where to get the secure index url.

Thanks for the helpful explanation. Note that the newer version PipAuthenticate@1 needs to set an additional parameter onlyAddExtraIndex in order to export the environment variable as also discussed in this thread.

- task: PipAuthenticate@1 
  inputs:
    artifactFeeds: $(artifactFeed)
    onlyAddExtraIndex: True

Hi, how do you get PIP_EXTRA_INDEX_URL manually? I want to build the docker container on my machine before doing builds on pipelines. I can't seem to find where to get the secure index url.

I was wondering the same thing. Is there some AZ command I can run? Maybe I need to create a PAT and keep it on my local .env file?

Also I noticed that the PipAuthenticate@1 task saves the URL in PIP_INDEX_URL

Hi, how do you get PIP_EXTRA_INDEX_URL manually?

At least there are the standard ways specified in the Devops documentation (Option 2), does that not fulfill your needs?

You can form a manual extra index using a PAT, see here

Tutorial looks nice but I'm not able to see any use of ENV PIP_EXTRA_INDEX_URL=$INDEX_URL
I mean how we can install the artifacts from Azure artifact, I did not see any pip install pakg name in the docker file.

Also i tried to install but it does not worked.

Cheers,
Anil

What if I have to do it without pipelines ? I want to have a manual release cycle for my pip package that has been deployed on Azure Artifact. I require to consume it as in install it in my container ~ through Dockerfile / requirements.txt... Can anyone help as the problem comes when I do
pip install -i https://url-to-my-package-on-azure-artifact my-package , the 2-factor Authentication comes up and If I tried it in docker, the it fails due to whatever interruption due to 2-factor Authentication..

Security advice:

As demonstrated in this excellent blog article and as recommended in this Azure white paper, it is strongly advised to use PIP_INDEX_URL instead of PIP_EXTRA_INDEX_URL to mitigate risk of package substitution.

Thanks for the gist, by the way. It was very useful and worked nicely for me.

Don't know if its an azure issue or a docker-compose one, but I adapted this example to run a docker-compose build instead, and it keeps failing because something adds an empty space at the beggining of the PIP_EXTRA_INDEX_URL. The task is defined as:

- task: DockerCompose@0
      displayName: 'Build Docker images'
      inputs:
        azureSubscription: ...
        azureContainerRegistry: ...
        projectName: ...
        dockerComposeCommand: build --build-arg 'INDEX_URL=$(PIP_EXTRA_INDEX_URL)'

But the execution runs (note the space between = and https://build):

[command]/usr/local/bin/docker-compose -f /home/vsts/work/1/s/docker-compose.yml -f /home/vsts/agents/2.153.1/.docker-compose.1560464790310.yml build --build-arg 'INDEX_URL= https://build:***@...'

do you know what could be the issue?

Add this bash script in order to trim de var.

        - task: Bash@3   
          displayName: trim index_url var
          inputs:        
            targetType: 'inline'          
            script: |       
              PIP_INDEX_URL=`echo $(PIP_EXTRA_INDEX_URL) | sed 's/ //g'`              
              echo $PIP_INDEX_URL
              echo "##vso[task.setvariable variable=PIP_INDEX_URL]$PIP_INDEX_URL"

Then just use the PIP_INDEX_URL var