Setting up a Python Development Environment

This guide explains how to setup a computer for effective Python development, while maintaining organized environments supporting multiple Python versions (if necessary). There are several Python environment managers out there; for this guide we will use Conda. Conda is a general-purpose environment manager that can be used for more than just Python, is cross-platform compatible, and integrates nicely with common terminals.

Install the Conda environment manager

The installation we will use is called "miniconda". It just comes with bare-bones Python without a bunch of unnecessary fluff (as opposed to full Anaconda). For all installs, I will assume you have a 64-bit system since it's not 2001.

Windows

On Windows, you have the option to install natively, which will give you the Anaconda Prompt command line to access python. If you're already familiar with Command Prompt syntax, this will be fine. However, I highly suggest installing the Ubuntu subsystem (found in the Windows App Store) and following the Linux install instructions below. Development will be much smoother in Linux and you will encounter less problems down the road. Plus, working in Linux is just a valuable skill to have.

Okay so you didn't listen to my rant and want to install on Windows natively.

Visit https://docs.conda.io/en/latest/miniconda.html and download the Python 3.x installer for your system (likely 64-bit). Do not download Python 2.x; that language is officially dead.
Run the installer.
- Default destination probably
- I generally check both of the boxes in "Advanced Options", I forget why but something weird happens if you don't.
- Click next a bunch and finish. (Probably want to uncheck the tutorial box and the Learn More box)
After installed, you will find in the start menu a new "Anaconda Prompt" application. You should use this to follow this guide as your "terminal".

Linux

Install on linux can be done from a terminal:

Run the following

curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh | sh

Then follow the prompts.

Do you accept the license? (yes)
Where do you want to install? (probably default, just press enter unless you have strong objections)
Do you want to run conda init? (yes)

Log back out then log back in. If you run:

conda info

You should see the active environment "base".

If (2) didn't work, you may have to try again running:

conda init

MacOS

Install MacOS can also be done via the terminal:

Run and follow the prompts (defaults are good)

curl https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh | sh

Then follow the prompts.

Do you accept the license? (yes)
Where do you want to install? (probably default, just press enter unless you have strong objections)
Do you want to run conda init? (yes)

Log back out then log back in. If you run:

conda info

You should see the active environment "base".

If (2) didn't work, you may have to try again running:

conda init

If (2) and (3) didn't work and you're using the zsh shell, sometimes conda won't recognize it or configure it. Try running (for a default conda install location):

/Users/$USER/miniconda3/bin/conda init zsh

Python Environments

Conda allows you to have many different "environments", each having their own Python, C, and other language libraries installed. This is particularly useful in python because many projects run in different python versions and may require conflicting libraries.

By default you will have be in the "base" library, which will probably have python 3.7 included from your installer. The python command along with the pip command should both point to the specific python installation in the current active environment. Most modern terminals (bash, zsh) will even include a tag in your prompt somewhere listing which environment your in (there may be a "(base)" tag located to the left of your prompt).

Creating new environments

To create a new environment, use syntax like:

conda create -n dev python=3.7

Where

"dev" is the name of your new environment
"3.7" is the version of python you want

You can then switch to any of your environments with:

conda activate dev

To go back to the previous environment:

conda deactivate

If you've forgotten what environments you've installed:

conda env list

If you want to remove an environment and start over:

conda env remove -n dev

I generally don't use the "base" environment; I like to keep that clean. Every once in a while something gets messed up with dependency versions and its nice to just delete your environment and start over. For example, I generally work in an environment called "dev". I then have a line at the end of my .bashrc|.bash_profile|.zshrc like:

conda activate dev

so that I'm in that environment by default on login.

Installing libraries

When in a conda environment, you can install libraries via pip or via conda. There are some advantages/disadvantages to using either:

conda install numpy
- Advantage: Installing through conda can often give more system-specific optimized libraries. For example, if you're running an Intel processor that supports MKL instructions, you'll get a version of numpy compiled with those instructions for better performance, along with the C libs required to do so.
- Advantage: You can install more than just Python libraries. For example, install the latest node binaries with conda install nodejs.
- Disadvantages: Conda doesn't always have the latest versions. If you're looking for a new library that was just released yesterday, many times it won't bet on conda yet.
- Disadvantages: Conda doesn't always have everything. Some lesser-known python libraries may not even be on conda. Small projects generally upload to pypi, but don't have people maintaining conda distributions.
pip install numpy
- Advantage: Everyone in the python world distributes their library on pypi. If theres a public library, it's here, and it's the latest version.
- Disadvantage: May miss out on some hardware optimizations.
- Disadvantage: Can only install Python libraries. I pretty much install python libraries with pip install just because it's more familiar. Only on super critical performance environments like tensorflow with cuda/cudnn dependencies do I worry about conda install.

IDE Integration

All IDEs should be familiar with multiple python environments and offer commands to switch between them.

VSCode

Install the "Pylance" extension authored by Microsoft
Run command (cmd+shift+p or ctrl+shift+p): Python:Select Interpreter
Select from the dropdown which environment you want to use

I also recommend pairing this with a linter such as flake8:

In terminal: pip install flake8
In VSCode: run command (cmd+shift+p or ctrl+shift+p), Python:Select Linter
Select flake8 from the dropdown.

PyCharm

Open PyCharm Preferences
Expand Project: Name-Of-Project
Python Interpreter
Click the gear icon > Add
Base Interpreter: Find the location of your python bin. This should be something like /Users/userName/miniconda3/bin/python. If you're on Linux/MacOS you can find this by typing which python in the environment. If you're on Windows then good for you.
Check "Make available to all projects" so you never have to do this again.
Press OK a few times

Jupyter Setup

If you want to setup use of notebooks, they can be a great tool for experimenting, data analysis, or just debugging. I develop heavily with Jupyter, and often test new code interactively in a notebook just because it's so convenient. If you come from a Matlab or R background, this is python's response to interactive programming (if you accidentally installed Spyder, I highly suggest giving jupyter a try instead).

Setup the Jupyter Server

Jupyter comes in two flavors, Jupyter Notebooks and Jupyter Lab. Lab is the new hotness so we will use it for our examples. They both do about the same thing and installing one will give you the other. Install the server with:

pip install jupyterlab

This will install the server along with a "kernel" for your current environment. Basically a kernel is just what allows Jupyter to run python code via your current environment. You can then launch the server with:

jupyter lab

This should automatically open a browser window and bring you to the Jupyter landing page. If not, there will be printed some URL/token in your terminal window that you can copy and paste in your browser to visit the server. I just keep a small terminal minimized running a Jupyter server in the background 24/7, and localhost:8888 is probably my most visited website.

Multiple Environments in Jupyter

You can install a kernel in any conda environment and get access to your python libs in Jupyter. In doing this, you can have one Jupyter server running and make notebooks in any of your environments. You can also swap an existing notebook between environments by clicking "Kernel" > "Change Kernel..." in the top menu.

Lets say I've made a new conda environment called "ml":

conda create -n ml python=3.7

Then I want to access this environment from Jupyter, I need to install the kernel:

conda activate ml
pip install ipykernel
python -m ipykernel install --user --name ml --display-name "Machine Learning (3.7)"

Where

"ml" is an alias for the kernel. I recommend keeping this the same as the conda environment name, otherwise it gets confusing.
"Machine Learning (3.7)" is the display name that will appear in the Jupyter UI. I like to put the python version in here so I remember.

Then you restart the Jupyter server and you will see the kernel available. Note: in the above section we just installed the Jupyter kernel, not the server. So if you try to run jupyter lab in this new conda environment, it will tell you that's not a real command. You will have to switch to the previous environment (conda deactivate) to run the server. You could, of course, install Jupyter Lab again in the new environment, but that is unnecessary. These jupyter kernels are accessible to any "jupyter lab" server launched from any environment.

You view all your installed kernels with:

jupyter kernelspec list

You can remove a kernel with:

jupyter kernelspec remove ml

Where "ml" is the name you want to remove. Note: if you remove a conda environment, it doesn't automatically remove the kernel. You will need to remember to do that separately.

Other Jupyter Kernels

Like conda, Jupyter is a language-agnostic platform. Kernels have been written for many different programming languages for Jupyter. Some notable ones:

Rust - A high-performance systems programming language, the most loved language several years in a row now according to the Stack Overflow Developer Survey. Instructions to install Google's Rust kernel: https://github.com/google/evcxr/blob/master/evcxr_jupyter/README.md. This assumes you have the rust compiler installed: https://www.rust-lang.org/tools/install
Octave - A open-source version of Matlab. Instructions to install: https://github.com/Calysto/octave_kernel
R - An open statistics programming language. Website: https://irkernel.github.io/

If you have any other desired languages, you can probably google "(language) jupyter" and find at least a partially working kernel for it.

Running in Docker

If you need a portable solution, there are plenty of pre-made Jupyter installations that you can find on Docker: https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html

See instructions on running: https://jupyter-docker-stacks.readthedocs.io/en/latest/using/running.html

kykosic/Python-Development-Setup.md