pip downstream private packages

Readme.md

Using Private Packages in Python

With companies moving to microservices, it's becoming common to have a system scattered across multiple repositories. It's frequent to abstract common patterns and code into private repositories that are then included in each service.

But using packages from private repos with Python can be tricky. This guide will guide you through the necessary steps to correctly use private package with pip or pipenv. We have also built a little app that will generate the right incantations for you to copy and paste.

Figuring out the URL

First you need to choose which format you want to use: tarball (ie: .zip) or vcs (most likely git). I prefer just downloading the tarball because it doesn't require any additional vcs software. But if use pipenv, you'll have to use vcs (see below).

Authentication

In order to have access to the private repo or tarball, you'll need an authentication token. GitHub and other providers allows you to create a private access token for your user, but because we are going to hardcode such token in code, I strongly reccomend to create a 'bot' user and use its access token.

• Github: access token at https://github.com/settings/tokens. You can select just the repo scopes.
• Githost: access token at https://<myorg>.githost.io/profile/personal_access_tokens. You can just select api scope.
• Gitlab: access token at https://gitlab.com/profile/personal_access_tokens. You can just select api scope.
• Bitbucket: access token at https://bitbucket.org/account/user/<botusername>/app-passwords. You can just select repositories/read scope.

Versioning

You will laso need to decide which version of your package you want to install. We'll call this 'the ref. You could target a branch, a commit, or a tag / version. I prefer to use version because it gives me a human-readable point of reference.

Building the url

Once you have your access token and your ref, the next step is figuring out a URL at which pip can download or clone the code.

• Github

  • Tarball URL will be https://<access_token>@github.com/<myorg>/<myproject>/archive/<ref>.zip
  • VCS Tarball will be git+https://<access_token>@github.com/<myorg>/<myproject>.git@<ref>

• Githost

  • Tarball URL will be https://<myorg>.githost.io/<myteam>/<myproject>/repository/archive.zip?private_token=<access_token>&ref=<ref>
  • VCS Tarball will be git+https://<botusername>:<access_token@<myorg>.githost.io/<myteam>/<myproject>.git@<ref>

• Gitlab

  • Tarball URL will be https://gitlab.com/<myteam>/<myproject>/repository/archive.zip?private_token=<access_token>&ref=<ref>
  • VCS Tarball will be git+https://<botusername>:<access_token>@gitlab.com/<myorg>/<myproject>.git@<ref>

• Bitbucket

  • Tarball URL will be https://<botusername>:<access_token>@bitbucket.org/<myorg>/<myproject>/get/<ref>.zip
  • VCS Tarball will be git+https://<botusername>:<access_token>@bitbucket.org/<myorg>/<myproject>.git@<ref>

Specifing the dependency

pip requirements file

Once you have the URL, you can just add it to you requirements files, or use with pip install:

$ pip install "<URL>"  # You'll need the quotes to escape the URL

Pipfile

To add the URL to you Pifile, use this syntax:

[packages]
revsys-teams = {file = "<URL>"}

Private dependency of a private package

If you need to specify your private package as a dependency of another package, you'll have to add the URL in your the dependee's setup.py. Note that the exact syntax of the specification is a little tricky:

import os
from setuptools import setup, find_packages

setup(
    name='upstreamprivatepackage',
    install_requires=[
        # The 'version' is required. Doesn't have to be an actual version of the package.
        # but it must match what's in the `egg` fragment of its entry in `dependency_links`.
        "<myproject>==0.0.1",
    ],
    dependency_links=[
        # there must be a version identifier in the `egg=` fragment and it must match what's
        # in `install_requires`
        '<url>#egg=<myproject>-0.0.1',
    ],
)

When installing the upstream package, you'll also have to tell pip to actually parse the entries in dependency_links:

# `pip` has to be run with the `PIP_PROCESS_DEPENDENCY_LINKS` env var or with `--process-dependency-links` option.
# As of v9.0.1, `pip` will print a big red scary deprecation warning, but that's the only way.
PIP_PROCESS_DEPENDENCY_LINKS=1 pip install ...

# or
pip install ... --process-dependency-links

I thought this copy was really good.