Poetry docker file that can support private git repository packages

poetry_dockerfile

# syntax=docker/dockerfile:experimental

FROM python:3.7-slim AS base

# ENV LANG=C.UTF-8  # Sets utf-8 encoding for Python et al
# ENV PYTHONDONTWRITEBYTECODE=1  # Turns off writing .pyc files; superfluous on an ephemeral container.
# ENV PYTHONUNBUFFERED=1  # Seems to speed things up

ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    LANG=C.UTF-8 \
    PIP_NO_CACHE_DIR=off \
    PIP_DISABLE_PIP_VERSION_CHECK=on \
    PIP_DEFAULT_TIMEOUT=100 \
    POETRY_PATH=/opt/poetry \
    VENV_PATH=/opt/venv \
    POETRY_VERSION=1.0.0

# Ensures that the python and pip executables used
# in the image will be those from our virtualenv.
ENV PATH="$POETRY_PATH/bin:$VENV_PATH/bin:$PATH" \
    PYTHONPATH=/app

RUN apt-get -qy update && apt-get install --no-install-recommends -y git libev-dev

FROM base as poetrydev

RUN apt-get update \
    && apt-get install --no-install-recommends -y \
    # deps for installing poetry
    curl \
    # deps for building python deps
    build-essential \
    # deps for using ssh
    openssh-client \
    \
    # install poetry - uses $POETRY_VERSION internally
    && curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python \
    && mv /root/.poetry $POETRY_PATH \
    && poetry --version \
    \
    # configure poetry & make a virtualenv ahead of time since we only need one
    && python -m venv $VENV_PATH \
    && poetry config virtualenvs.create false \
    \
    # cleanup
    && rm -rf /var/lib/apt/lists/*

WORKDIR /venv_build

COPY poetry.lock pyproject.toml ./

RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts
RUN --mount=type=ssh poetry install --no-interaction --no-ansi -vvv


# Dev version of Docker image
FROM base AS dev

WORKDIR /app
EXPOSE 80

# Install OS package dependencies.
# Do all of this in one RUN to limit final image size.
RUN rm -rf /var/cache/apt/* /var/lib/apt/lists/*

COPY --from=poetrydev $VENV_PATH $VENV_PATH
COPY ./app /app

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--debug", "--port", "80", "--reload-dir", "CHANGE_ME_DIR_NAME"]

Hey, thanks for the share. Can I check if this script still works for you?
I am facing an error when trying this. Thanks

I think it works for me whats your error?
Maybe you forgot to add your keys?
This is where I had issues.

eval $(ssh-agent)
ssh-add ~/.ssh/id_ed25519
DOCKER_BUILDKIT=1 docker build  --ssh default ...  

@Dimfred It is failing on this command
executor failed running [/bin/sh -c poetry install --no-interaction --no-ansi -vvv]: exit code: 1
I also checked out yours: ( https://gist.github.com/Dimfred/5d6a9577fa9a971ae905b0a5ec1b0dab) but the same error. Not able to clone the repo

I mean the full error would be good to see. Whats the dependency? Is it a dep which you added over ssh? Maybe you could show the entry in the pyproject.toml. As said for me the problem was solved when I called the ssh-agent and added the --ssh default to my build command.

Hey Dimfred, I followed your guidelines and came right. I had to do a cleanup of packages. But fundamentally adding the keys as suggested did work.
Appreciate the help. Thank you

Great that it worked out, you're welcome.

Hey, thanks for this, but I get the same error.
How exactly did you resolve it?

My dependencies include more private repositories. Might that be the problem?

DOCKER_BUILDKIT=1 docker build -t template-cs --ssh default .

Also not working with explicit path to ~/.ssh/id_rsa.
I checked if it's the right one by doing: GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa" git pull which works...

Any ideas?

returned non-zero exit status 128.
#15 1.593 
#15 1.625 Traceback (most recent call last):
#15 1.625   File "/opt/poetry/lib/poetry/_vendor/py3.8/clikit/console_application.py", line 131, in run
#15 1.625     status_code = command.handle(parsed_args, io)
#15 1.625   File "/opt/poetry/lib/poetry/_vendor/py3.8/clikit/api/command/command.py", line 120, in handle
#15 1.625     status_code = self._do_handle(args, io)
#15 1.625   File "/opt/poetry/lib/poetry/_vendor/py3.8/clikit/api/command/command.py", line 171, in _do_handle
#15 1.625     return getattr(handler, handler_method)(args, io, self)
#15 1.625   File "/opt/poetry/lib/poetry/_vendor/py3.8/cleo/commands/command.py", line 92, in wrap_handle
#15 1.625     return self.handle()
#15 1.625   File "/opt/poetry/lib/poetry/console/commands/install.py", line 63, in handle
#15 1.625     return_code = installer.run()
#15 1.625   File "/opt/poetry/lib/poetry/installation/installer.py", line 74, in run
#15 1.625     self._do_install(local_repo)
#15 1.625   File "/opt/poetry/lib/poetry/installation/installer.py", line 225, in _do_install
#15 1.625     ops = solver.solve(use_latest=whitelist)
#15 1.625   File "/opt/poetry/lib/poetry/puzzle/solver.py", line 36, in solve
#15 1.625     packages, depths = self._solve(use_latest=use_latest)
#15 1.625   File "/opt/poetry/lib/poetry/puzzle/solver.py", line 180, in _solve
#15 1.625     result = resolve_version(
#15 1.625   File "/opt/poetry/lib/poetry/mixology/__init__.py", line 7, in resolve_version
#15 1.625     return solver.solve()
#15 1.625   File "/opt/poetry/lib/poetry/mixology/version_solver.py", line 80, in solve
#15 1.625     next = self._choose_package_version()
#15 1.625   File "/opt/poetry/lib/poetry/mixology/version_solver.py", line 355, in _choose_package_version
#15 1.625     packages = self._provider.search_for(dependency)
#15 1.625   File "/opt/poetry/lib/poetry/puzzle/provider.py", line 130, in search_for
#15 1.625     packages = self.search_for_vcs(dependency)
#15 1.625   File "/opt/poetry/lib/poetry/puzzle/provider.py", line 167, in search_for_vcs
#15 1.625     package = self.get_package_from_vcs(
#15 1.625   File "/opt/poetry/lib/poetry/puzzle/provider.py", line 196, in get_package_from_vcs
#15 1.625     git.clone(url, tmp_dir)
#15 1.625   File "/opt/poetry/lib/poetry/vcs/git.py", line 161, in clone
#15 1.625     return self.run("clone", repository, str(dest))
#15 1.625   File "/opt/poetry/lib/poetry/vcs/git.py", line 243, in run
#15 1.625     subprocess.check_output(["git"] + list(args), stderr=subprocess.STDOUT)
#15 1.625   File "/usr/local/lib/python3.8/subprocess.py", line 415, in check_output
#15 1.625     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
#15 1.625   File "/opt/poetry/lib/poetry/utils/_compat.py", line 205, in run
#15 1.625     raise CalledProcessError(
#15 1.625 
------
executor failed running [/bin/sh -c poetry install --no-interaction --no-ansi -vvv]: exit code: 1

Idk sad thing is that those poetry errors are mostly saying nothing. Are you sure that docker is using GIT_SSH_COMMAND? Really try just adding your key to the ssh agent as suggested.

I'm encountering a similar issue when pulling a dependency from bitbucket. Was the dependency called in the pyproject.toml file as package_name = {git = "ssh://[email protected]/tenant/repo.git", tag="v0.0.0"} or something similar?

Yes.
I have not tried further to solve the problem, as it is not too important for me at the moment.
If I get to it and find a solution, I will post it here.