Python Development on FreeBSD 13

Python Development on FreeBSD.MD

Pythonic way to setup Python Development Environment on FreeBSD

This gist lists steps needed to setup python development environment on FreeBSD using isolated virtual environments.

Why not use pkg or ports?

• Pythonic way is to create virtual enviroments for projects and add libs to these venvs not system python.
• Binary packages available only for one python version. Currently this is 3.8.
• Although we can build python packages against another version using ports, it's still not the Pythonic way.

Overview

• Install python 3.9 (latest stable) using pkg
• Install pip using get-pip.py
• Install pipx using pip
• Install poetry using pipx
• Create and Manage python virtual environments using poetry

NOTE: Both pipx and poetry provide isolated environments for python projects with different objectives. pipx is used to install python-based CLI/TUI tools in isolated env, while poetry is used to created isolated python environments for coding/development/testing etc.

Here pipx is used to install poetry in an isolated env. and from there on we use poetry to create/manage our virtual enviromentns for coding/development etc. Instead you can simply install poetry directly using pip w/o installing pipx.

Important Locations

• System Python binary - /usr/local/bin/python3.9
• System Python libraries - /usr/local/lib/python3.9
• User Python binaries - $HOME/.local/bin
• User Python libraries - $HOME/.local/lib/python3.9/site-packages
• pipx installed binaries - $HOME/.local/bin
• pipx installed virtual environments - $HOME/.local/pipx/venvs
• poetry virtual environments - $HOME/.cache/pypoetry/virtualenvs

setup-FreeBSD-Python-Dev.sh

#!/usr/bin/env bash

# Step 1: Install python 3.9 plus other reqd packages
sudo pkg install cmake gcc10 openblas blas libzmq4 python39 py39-sqlite3 py39-cython py39-numpy openjpeg libxml2 libxslt curl
python3.9 -VV # Make sure it works

# Step 2: Setup Variables
 # Use $HOME/TMP coz /tmp is mounted with 'noexec' option
mkdir -p $HOME/TMP 
export TMPDIR=$HOME/TMP
# By default /usr/local/include are not in compiler's and linker's search path
export CFLAGS="$CFLAGS -I/usr/local/include/openjpeg-2.4 -I/usr/local/include"
export CXXFLAGS="$CXXFLAGS -I/usr/local/include/openjpeg-2.4 -I/usr/local/include"
export LDFLAGS="$LDFLAGS -Wl,-rpath,/usr/local/lib -L/usr/local/lib -lomp"
export ZMQ_HOME="/usr/local" # Force pyzmq to use system zeromq.

# Step 3: Install pip in user's home directory
curl -sSL https://bootstrap.pypa.io/get-pip.py -o $HOME/get-pip.py
python3.9 $HOME/get-pip.py --user

# Step 4: Install pipx using pip in user's home directory
pip install -U pipx
pipx --version
pipx ensurepath # Add pipx to your $PATH
pipx completions # Setup pipx shell completions if reqd.

# Step 5: Install poetry using pipx in user's home directory
pipx install poetry
poetry about # Make sure it works
poetry completions --help # Setup poetry shell completions if reqd.

# Create a new venv using poetry
cd $HOME
poetry new myvenv
cd myvenv

# Add development dependencies
poetry add -D mypy flake8 black 
poetry add -D python-lsp-server -E all

# Add Jupyter Notebook 
poetry add -D jupyter jupyterlab

# Add Jupyter Interactive Widgets if needed
poetry add -D ipywidgets
poetry run jupyter nbextension enable --py widgetsnbextension --sys-prefix

# Add Contrib Extensions if needed
poetry add -D jupyter-contrib-nbextensions jupyter_nbextensions_configurator
poetry run jupyter contrib nbextension install --sys-prefix
poetry run jupyter nbextensions_configurator enable --sys-prefix
poetry run jupyter nbextension enable execute_time/ExecuteTime --sys-prefix

# Core Data Science
poetry add numpy pandas numexpr bottleneck

# Visualization
poetry add matplotlib seaborn graphviz

# Stats/ML
poetry add scipy scikit-learn statsmodels
poetry add xgboost pygam lightgbm

# Probabilistic Programming (Bayessian)
poetry add pymc3 
#poetry add pystan # Currently doesn't install

# Deep Learning
#poetry add tensorflow keras # Currenly doesn't install
#poetry add pytorch pyro numpyro # Currently doesn't install

poetry check # Final Check

poetry run jupyter-notebook