Linux Setup

linux_setup.md

Dev System Setup for Linux/mac or Windows with WSL

Note: for a Windows setup without WSL see this gist

Install Windows Subsystem for Linux, Ubuntu

See also Windows Subsystem for Linux Installation Guide for Windows 10

1. Open PowerShell as Administrator and run the following. Restart when instructed:

> Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

2. Add Ubuntu 18.04LTS from Windows Store
3. Launch Ubuntu shell from Start Menu
4. Follow prompts to add UNIX username and password

Set up Linux Environment

1. Update Linux packages

$ sudo apt update && sudo apt upgrade

2. Migrate from bash to zsh (because it's better):

$ sudo apt-get update && sudo apt-get -y install zsh

3. Check version to verify installation

$ zsh --version

zsh 5.4.2 (x86_64-ubuntu-linux-gnu)

4. Make Zsh your default shell, and close the terminal

$ chsh -s /bin/zsh
$ exit

5. Open Ubuntu shell again. Choose option 2 in the Z shell configuration function that comes up

6. Install basic software build tools (gcc, etc.)

$ sudo apt-get install build-essential

Install git and Github keys

Git should already be installed, but check:

$ git --version

git version 2.17.1

Add Github security keys. When prompted, set the file location to the default ~/.ssh and be sure to add a passphrase.

$ mkdir -p ~/.ssh
$ ssh-keygen -t rsa -C <[email protected]>

Log in to Github. Go to Settings->SSH and GPG Keys->New SSH key Give the key a name and paste in the contents of ~/.ssh/id_rsa.pub.

Back in Linux run:

$ ssh-agent /bin/zsh
$ ssh-add ~/.ssh/id_rsa

Set up Dot Files

Replace default dotfiles (e.g. .zshrc, .gitconfig) with preferences from some other system you use (e.g. fork thoughtbot/dotfiles and customize as in mine). (If prompted to overwrite e.g. .zshrc, choose yes.)

$ git clone [email protected]:<username>/dotfiles ~/.dotfiles
$ sudo add-apt-repository ppa:martin-frost/thoughtbot-rcm
$ sudo apt-get update
$ sudo apt-get install rcm
$ env RCRC=$HOME/.dotfiles/rcrc rcup
$ rcup -d ~/.dotfiles

Close and relaunch the Linux shell. If you get a warning like:

zsh compinit: insecure directories, run compaudit for list.

Run compaudit and then the following for each directory in the compaudit output. Then close and relaunch the terminal.

For files not under /home/<username>

$ sudo chown -R root:staff <directory>

For files under /home/<username>

$ sudo chown -R <username>:<username> <directory>
$ sudo chmod go-w <directory>

Finally, add symbolic links to your user directory (~) pointing to everything in your ~/.dotfiles/local directory.

If you have pre-requisites in your local config (e.g. Powerline shell, Python scripts, etc., install that stuff first.

For each file and folder in ~/.dotfiles/local do something like this example (note prefixing the link target with a .):

$ ln -s ~/.dotfiles/local/aliases.local ~/.aliases.local

To do this in one command 😎:

$ for f in ~/.dotfiles/local/*; do ln -s $f .`basename $f`; done

Also, create a starting local environment file (this one doesn't get checked in):

$ touch ~/.env.local

If you forgot to install something that a local config file needs, you should be able to just e.g.:

$ rm ~/.<offending-local-config>.local

Windows PATH in Linux

By default (part of) your Windows path will be appended to your Linux path. This can sometimes cause problems (e.g. if you have NPM/Node installed in Windows, it may fail to run in Linux because it finds the Windows npm first).

This can be solved by updating a registry entry in Windows. WSL_DISTRIBUTION_FLAGS for the distribution default to being all set. You need to clear the bit for appending the "NT Path".

The registry entry is HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Lxss\{GUID}\Flags (where GUID varies). The flags are a bit field, so clear bit 1 (i.e. change 7 to 5).

After this, close all Linux windows. Then open a Linux window and check the path:

$ echo $PATH

This should not include any entries like /mnt/c/Users/<username>/AppData.

A Couple Other Things

Note: some of these are just notes for posterity or things to check. Especially any dotfile changes should already be checked-in to your dotfile repo under local/ assuming you are duplicating an existing setup on a new machine.

Get colored listings in Linux

Set up an alias for ls

Edit .aliases.local and add:

alias ls="ls --color=auto"

In order to check this local dotfile in and use it on other machines, you can add cases based on uname. See, for example, this alias.local.

Set up dircolors

Also set up dircolors support and add to .zshrc. You will need a definition file too.

if [ "$(uname 2> /dev/null)" == "Linux" ]; then
  # Set up directory listing coloring
  if [ -f ~/.dircolors ]; then
    eval `dircolors ~/.dircolors`
  fi
fi

Get Configurable Colored Prompt

Install and configure powerline-cli (requires python, see below).

$ pip install powerline-shell

Install silver searcher (ag)

$ sudo apt-get install silversearcher-ag

Create a symlink to your Windows documents folder

$ ln -s /mnt/c/Users/<windows-username>/Documents ~/Documents

Download and install X Server for Windows. Then set the display environment variable on the Linux side.

$ echo "export DISPLAY=localhost:0.0" >> ~/.zshrc

Improved Terminal in Windows

Setup Terminator as an improved terminal that you can run through the X Server instead of the default bash shell that WSL provides. This allows better color control, gitmoji for source control, etc.

sudo apt install dbus-x11
sudo systemd-machine-id-setup
sudo apt-get install terminator

See also this post.

On the Windows side, set up a shortcut to run a simple VBScript. The Target is:

C:\Windows\System32\wscript.exe "<path to>\startTerminator.vbs"

and Start in is %USERPROFILE%. Terminator will start in /mnt/c/Users/<windows username which is probably not what you want (i.e. not Linux $HOME). So add to .zshrc.local:

if [ "$(uname 2> /dev/null)" == "Linux" ]; then
  if [ -t 1 ]; then
    cd $HOME
  fi
fi

The VBScript for the Windows side is simply:

args = "-c" & " -l " & """DISPLAY=:0 terminator"""
WScript.CreateObject("Shell.Application").ShellExecute "bash", args, "", "open", 0

Once launched, you can configure colors, window sizes, etc. which will get stored on the Linux side in ~/.config/terminator/config. Of course, you can link this to your source-controlled dotfiles as well.

To use a particular font in the terminal, that font needs to be install on the Linux side. A good font is and extension of Google's Source Code Pro for use with the Powerline terminal found here.

Copy these files somewhere on the Linux side (e.g. ~/SourceCodePro. Then:

$ sudo cp -r ~/SourceCodePro /usr/local/share/fonts
$ rm -rf ~/SourceCodePro
$ sudo fc-cache -fv

The last line there updates the font cache.

VS Code Setup

Install Visual Studio Code and the WSL Remote Extension. Note that originally (and maybe still) this only worked with the Insiders version of VS Code.

Set bash to run zsh so Visual Studio Code integrated terminal is zsh. Edit ~/.bashrc in Linux and add at the top:

# Launch Zsh
if [ -t 1 ]; then
  exec zsh
fi

There are lots of other extensions to install to VS Code as well depending on the project you're working on. Things for C/C++, Javascript, Python, Markdown, linting, debugging, testing, etc. TODO: aggregate list.

One important one with multiple systems is Settings Sync. This lets you upload/download your VS Code settings stored in a Github Gist.

Current extensions list (2019-12-09):

dan-c-underwood.arm
ms-vscode.azure-account
vsciot-vscode.azure-iot-toolkit
ms-azure-devops.azure-pipelines
ms-vscode.azurecli
msazurermtools.azurerm-vscode-tools
Shan.code-settings-sync
streetsidesoftware.code-spell-checker
marus25.cortex-debug
mine.cpplint
ms-vscode.cpptools
msjsdiag.debugger-for-chrome
sidneys1.gitconfig
andys8.jest-snippets
eriklynd.json-tools
julialang.language-julia
jameselderfield.language-weave
James-Yu.latex-workshop
ritwickdey.LiveServer
zhuangtongfa.Material-theme
bpruitt-goddard.mermaid-markdown-syntax-highlighting
ms-mssql.mssql
toasty-technologies.octave
esbenp.prettier-vscode
ms-python.python
ms-vscode-remote.remote-containers
ms-vscode-remote.remote-ssh
ms-vscode-remote.remote-ssh-edit
ms-vscode-remote.remote-ssh-explorer
ms-vscode-remote.remote-wsl
humao.rest-client
lextudio.restructuredtext
stkb.rewrap
syler.sass-indented
shardulm94.trailing-spaces
vscodevim.vim
ms-azuretools.vscode-apimanagement
ms-azuretools.vscode-azureappservice
ms-azuretools.vscode-azurefunctions
ms-azuretools.vscode-azurestorage
ms-azuretools.vscode-cosmosdb
ms-azuretools.vscode-docker
dbaeumer.vscode-eslint
Orta.vscode-jest
rtbenfield.vscode-jest-test-adapter
mgmcdermott.vscode-language-babel
ms-vscode.vscode-node-azure-pack
littlefoxteam.vscode-python-test-adapter
ms-vscode-remote.vscode-remote-extensionpack
wingrunr21.vscode-ruby
cssho.vscode-svgviewer
hbenl.vscode-test-explorer
pflannery.vscode-versionlens

Python Scientific (and Whatever Else) Programming

Setup Python and Friends

To ensure that the same versions of libraries (and Python itself via pyenv) get used regardless of the development machine, OS, etc., this project uses pipenv to manage package versions and a virtual environment for running code. *If you are familiar with pip and virtualenv, using pipenv means you no longer have to manage these separately.)

Install pyenv

First, install pyenv to manage the Python version in your virtual environment. On macOS, install via Homebrew:

$ brew update
$ brew install pyenv

Or on (Ubuntu) Linux install some necessary packages and then install pyenv through github checkout as described in the README).

On Linux, some additional libraries are needed to proceed with the python installation (see also):

$ sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev \
  libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
  xz-utils tk-dev libffi-dev liblzma-dev python-openssl git

For Ubuntu, it specifies this as well:

$ sudo apt install libedit-dev

Install python Version

Now install an appropriate python version, e.g.:

$ pyenv install 3.7.3

This will also install the compatible version of pip. To make this version the system default run:

$ pyenv global 3.7.3

Install pipenv Globally

To install pipenv (if you don't already have it) run:

$ brew install pipenv

on macOS and:

$ pip install --user pipenv

on Linux (see also these notes on user installation)

Test Python/Scipy

To do a quick test of the installation, install numpy and matplotlib and create a plot. (On Windows/WSL, ensure your XServer is running so the plot window can be displayed.)

$ mkdir plottest
$ cd plottest
$ pipenv install numpy matplotlib

The last command creates new virtual environment and installs the numerical and plotting libraries. Now create a simple test program as test.py using your editor of choice:

$ vi test.py

import numpy as np
import matplotlib.pyplot as plt

t = np.arange(0, 10, 1/100)
plt.plot(t, np.sin(1.7 * 2 * np.pi * t) + np.sin(1.9 * 2 * np.pi * t))
plt.show()

Now run the script. (Again, if you're on Windows/WSL, make sure your XServer is active at this point.)

$ pipenv run python test.py

The graph should appear in a window. Close the plot window to allow the script to finish.

Note that this command runs python test.py from inside the virtual environment that pipenv set up. Alternatively, you can make every command you type run from inside the environment. Try:

$ pipenv shell
plottest-uHQw95fx $ cd plottest     # You might need to change directories again
plottest-uHQw95fx $ python test.py
plottest-uHQw95fx $ exit
$ 

At this point, you can remove the generated virtualenv and test directory. From within plottest/:

$ pipenv --rm
$ cd ..
$ rm -r plottest

Other Useful Scientific Libraries

Some more Python libraries that are very useful for scientific stuff:

• scipy - a collection of libraries including numpy
• pint - library for handling units and dimensions
• mendeleev - database of atomic properties

Documentation Generation

To go along with scientific programming, technical writing, etc., it is nice to have some tools set up to generate documents with nice plots and graphics, templates (maybe with branding), and keep everything in version control (of course). A good solution to this seems to be writing in Markdown and running the documents through pandoc to get HTML or PDF (via xelatex) outputs.

For a while, pweave seemed to be a good solution (especially for use with Python), but eventually it made sense to develop a custom pandoc filter (zzz link when ready) for just this purpose.

Pandoc

Install pandoc and necessary filters in order to generate various document formats from (primarily) Markdown.

$ sudo apt-get install pandoc

In order to get certain plugins, the shortest path is installing via linuxbrew.

$ sudo apt-get install build-essential curl file git
$ sh -c "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh)"

Then install these two filters that provide support for nice cross-referencing and bibliography/reference handling:

$ brew install pandoc-crossref
$ brew install pandoc-citeproc

PDF via Latex

To generate PDF documents, install xelatex.

$  sudo apt-get install texlive-xetex

Then add the Windows fonts folder to the Linux side. Ensure necessary fonts are installed in Windows. Edit /etc/fonts/local.conf and add:

<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
    <dir>/mnt/c/Windows/Fonts</dir>
</fontconfig>

Finally update the font cache on the Linux side:

$ sudo fc-cache -fv

Node.js

For developing in Javascript or using utilities written for node, install NVM and Node.js. (See also this gist)

$ curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.0/install.sh | zsh

Restart your terminal, and then:

$ nvm install stable
$ nvm use stable

Install Gitmoji CLI

To use Gitmoji in git commit messages, install the command line tools and git log integration. (Requires Node.js above).

$ npm i -g gitmoji-cli
$ npm i -g gitmoji-log
$ sudo apt install fonts-emojione

Also install the definition file. If this isn't already checked in with your other dotfiles, download a copy and store it in ~/.gitmoji/:

$ mkdir -p ~/.gitmoji
$ curl https://raw.githubusercontent.com/carloscuesta/gitmoji/master/src/data/gitmojis.json > ~/.gitmoji/gitmojis.json

(You should then check this file in to your dotfiles and generate a symbolic link instead.)