python pip caching.md

Two methods:

how-do-i-install-from-a-local-cache-with-pip

First Method

PIP_DOWNLOAD_CACHE has some serious problems. Most importantly, it encodes the hostname of the download into the cache, so using mirrors becomes impossible.

The better way to manage a cache of pip downloads is to separate the "download the package" step from the "install the package" step. The downloaded files are commonly referred to as "sdist files" (source distributions) and I'm going to store them in a directory $SDIST_CACHE.

The two steps end up being:

pip install --no-install --use-mirrors -I --download=$SDIST_CACHE <package name>

Which will download the package and place it in the directory pointed to by $SDIST_CACHE. It will not install the package. And then you run:

pip install --find-links=file://$SDIST_CACHE --no-index --index-url=file:///dev/null <package name> 

To install the package into your virtual environment. Ideally, $SDIST_CACHE would be committed under your source control. When deploying to production, you would run only the second pip command to install the packages without downloading them.

Second Method

In my opinion, pip2pi is a much more elegant and reliable solution for this problem.

From the docs:

pip2pi builds a PyPI-compatible package repository from pip requirements

pip2pi allows you to create your own PyPI index by using two simple commands:

1. To mirror a package and all of its requirements, use pip2tgz:

    $ cd /tmp/; mkdir package/
    $ pip2tgz packages/ httpie==0.2
    ...
    $ ls packages/
    Pygments-1.5.tar.gz
    httpie-0.2.0.tar.gz
    requests-0.14.0.tar.gz
   

2. To build a package index from the previous directory:

    $ ls packages/
    bar-0.8.tar.gz
    baz-0.3.tar.gz
    foo-1.2.tar.gz
    $ dir2pi packages/
    $ find packages/
    /httpie-0.2.0.tar.gz
    /Pygments-1.5.tar.gz
    /requests-0.14.0.tar.gz
    /simple
    /simple/httpie
    /simple/httpie/httpie-0.2.0.tar.gz
    /simple/Pygments
    /simple/Pygments/Pygments-1.5.tar.gz
    /simple/requests
    /simple/requests/requests-0.14.0.tar.gz
   

3. To install from the index you built in step 2., you can simply use:

    pip install --index-url=file:///tmp/packages/simple/ httpie==0.2
   

You can even mirror your own index to a remote host with pip2pi.