Python Learnings

General Learnings

__init__.py tells Python that the directory is a package

Standard Project Setup

project/
├── MANIFEST.in
├── README.md
├── setup.py                   important for building distribution out of a package
├── src
│    └── tasks
│        ├── __init__.py
│        ├── api.py
│        ├── cli.py
│        ├── config.py
│        ├── tasksdb_pymongo.py
│        └── tasksdb_tinydb.py
└── tests
    ├── conftest.py
    ├── pytest.ini
    ├── func
    │    ├── __init__.py      empty to make pytest look for pytest.ini one level up
    │    ├── test_add.py
    │    └── ...
    └── unit
        ├── __init__.py       empty to make pytest look for pytest.ini one level up
        ├── test_task.py
        └── ...

Package / Dependency Management

see https://stefanoborini.com/current-status-of-python-packaging/
python packaging is not really about python packages. It’s about python distributions
A distribution of Python is a bundle that contains an implementation of Python along with a bunch of libraries or tools.

"New" way

use Poetry
create a pyproject.toml and specify dependencies with a minimum version

"Old" way

install virtualenv

python3 -m pip install --user virtualenv

create a new virtual environment in venv via
```
python3 -m venv venv
```
activate virtual environment via
```
source venv/bin/activate
```
create a requirements.txt file with your dependencies
```
pytest >= 6.2.4
```
install dependencies into your venv via
```
pip install -r requirements.txt
```

Creating virtual environments

$ pip3 install -U virtualenv
$ python3 -m virtualenv venv
$ source venv/bin/activate
$ pip install pytest

Testing

Pytest setup

see https://docs.pytest.org/en/6.2.x/goodpractices.html

Naming conventions of files

pytest auto-discovers files with the following naming conventions:

Test files test_...py or ..._test.py
Test methods test_<something>
Test classes Test<Something>

Filtering tests

If you want to run only a single test, you can use

$ pytest -v tasks/test_four.py::test_asdict

Organising Tests

Markers

Add a marker to your test file:

import pytest

@pytest.mark.run_these_please
def test_asdict():

Add a pytest.ini to your tests directory

[pytest]
markers =
    run_these_please: mark a test as please to be run

run your tests via
```
$ pytest -v -m run_these_please
```

You can specify the location of the pytest.ini file with the -c flag:

$ pytest -c ../pytest.ini -v -m please_run_me

Resources

tox

Once you are done with your work and want to make sure that your actual package passes all tests you may want to look into tox, the virtualenv test automation tool and its pytest support. tox helps you to setup virtualenv environments with pre-defined dependencies and then executing a pre-configured test command with options. It will run tests against the installed package and not against your source code checkout, helping to detect packaging glitches.

Dockerizing

Building Python Docker Images

FROM python:3.9-slim-bullseye

WORKDIR /usr/src/app

USER root

RUN addgroup --gid 1001 --system mbdrna
RUN adduser --uid 1001 --gid 1001 mbdrna
RUN apt-get update
RUN apt-get install -y gcc libffi-dev musl-dev make g++

# see https://pythonspeed.com/articles/activate-virtualenv-dockerfile/
ENV VIRTUAL_ENV=/opt/venv
RUN python3 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

RUN pip install --upgrade pip
RUN pip install --upgrade azure-cli
RUN pip install --upgrade setuptools

# everything above can be cached, so split pip install here

COPY . .

RUN . venv/bin/activate && pip install --upgrade -r sdp_flask/requirements.txt \
    && chown -R mbdrna:mbdrna /usr/src/app 

WORKDIR /usr/src/app/sdp_flask

USER mbdrna
ENV FLASK_RUN_PORT=1337 
ENV FLASK_ENV=development
CMD [ "../venv/bin/flask", "run", "--host=0.0.0.0" ]

Docker Compose for dev environments

version: '3.5'

networks:
  default:
    name: default
    driver: bridge

services:
  flask:
    image: 'flask-app:local-dev-version'
    depends_on:
      - rabbit
      - redis
    ports:
      - "1337:1337"
    volumes:
      - ./app_flask:/usr/src/app/app_flask
    container_name: flask-app
    networks:
      - default
    environment:
      CELERY_BROKER_URL: amqp://guest:guest@rabbit/
      REDIS_HOST: redis

  # see https://hub.docker.com/_/rabbitmq
  rabbit:
    image: "rabbitmq:3.9.5"
    networks:
      - default
    ports:
      - "5672:5672"
    container_name: rabbit

  # see https://hub.docker.com/_/redis
  redis:
    image: "redis:6.2.5"
    networks:
      - default
    ports:
      - "6379:6379"
    container_name: redis

michaellihs/python-learning.md