Steps to setup a new OS X laptop with a data science & computational biology development environment.
Last updated: 7 Jan 2017 macOS: Yosemite => Sierra
Sure you could try to do this from dotfiles, but historically something has broken and required manual input anyway. So, inspired by the pragmatism of paul irish bash setup script and a few other gists written in markdown like this, I rolled my own.
- I tried a few password manager, but settled on using macOS keychain
- However, i) needing to use chrome for work and ii) access on mobile meant that I decided to transition to lastpass
- Most of my documents are synced by dropbox or google drive, so i keep minimal stuff on local hard drive only
- Configuration are synced on dropbox in the
~/Dropbox/OSXdotfilesdirectory, with the exception of those API tokens and encrypted info, which better stay off cloud services. when I first did this, I was not aware of mackup which automates all of this - after trying to use anaconda python for a while, I opted to switch back to homebrew's python. I like that i can install other non-scientific packages without breaking
python-packagesdependencies. If I use anaconda, I place the anaconda path at the end, rather than at the beginning of$PATH - I rely on arq 5 for mac to backup most things to the unlimited amazon cloud drive
mkdir ~/migration && cd ~/migrationPut this very important stuff on a USB key. I make a regular copy more than once a day using Arq backup to amazon cloud drive (55$/year unlimited), but using borg to a free Dropbox account works well as a cheaper alternative.
# all the keychains and secret keys
rsync -av ~/Library/Keychains/ migration/Keychains/
rsync -av ~/.ssh migration/.ssh
rsync -av ~/.gnupg migration/.gnupg
rsync -av ~/.borg migration/.borg
rsync -av ~/.airvpn migration/.airvpn
# all local tokens and paths that are not normally in public repo
for f in ~/.*.local; do cp $f migration/; donersync -av ~/.[^.]* migration/dotfilesnote this works in bash but not zsh because of automatic * expansion.
zip -er migration.zip migration/Perfect destination is an external USB drive or key. Drag the resulting encrypted zip on there. Unzip in the new machine.
brew leaves | sed 's/^/install /' > ~/migration/Brewfile # all top-level brew installs
brew cask list > ~/migration/CaskfileA more sophisticated approach is using a Brewfile like the one in this gist. If you have the homebrew bundle command installed you can create a Brewfile by simply using
brew bundle dump
# this creates a Brewfile in the current directory (eg. ~/migration)The brewfile I keep here is the master, but my machine usually gains a few packages and the file becomes out of date. To compare and add differences I do:
mv ~/migration/Brewfile ~/migration/Brewfile.now
comm -23 <(sort ~/migration/Brewfile.now) <(sort thisgist_brewfile)
# outputs a list of those packages on the machine but not included in the gist.
# copy paste manually the ones you actually want or simply cat
cat ~/migration/Brewfile.now thisgist_brewfile | sort | uniq > ~/migration/Brewfile
Attached to this gist is a pip requirements.txt file that list the main packages that should be there.
We can compare that list with what has been installed on the machine we are about to leave.
pip3 freeze | sed 's/==.*//' > old-requirements.txt
# show the packages that are installed on the old machine and are not included in the requirements file
comm -23 <(sort ~/migration/old-requirements.txt) <(sort requirements.txt)Getting a full list of any settings changed is difficult. I have opted for forking the mighty .osx, but this might become out of date with the actual preferences I have selected on the laptop. Hopefully nothing I can't live with.
make sure the zprezto repo is fully commited and pushed to github. I wish there was a oh-my-bash out there.
I keep sublime text preferences synced in Dropbox ~/Dropbox/OSXdotfiles/Subl/. One needs only to save the User directory.
- Chrome tabs via OneTab One can probably go in the onetab extension and export everything, but the local files shoud be in:
cp ~/Library/Application\ Support/Google/Chrome/Default/Local\ Storage/chrome-extension_chphlpgkkbolifaimnlloiipkdnihall_0.localstorage-journal ~/Dropbox/OSXdotfiles/onetab/
cp ~/Library/Application\ Support/Google/Chrome/Default/Local\ Storage/chrome-extension_chphlpgkkbolifaimnlloiipkdnihall_0.localstorage ~/Dropbox/OSXdotfiles/onetab/- iTerm settings. I keep them synced in Dropbox
Dropbox/OSXdotfiles/iTerm - safari tabs / chrome tabs + bookmarks
- software licenses
- Movies
- Pictures
Before wiping, make a copy of the disk in its entirety with SuperDuper or CarbonCopyCloner.
Add passwords for google accounts (pwd + 2fa token) and check what you want synced. At the moment I rely on google as my main base so:
- Turn on google calendar, contacts sync
- Turn off icloud calendar + contacts sync
Log into gmail on safari using google password + 2fa token
Update the software to the latest version. Apple Icon > Software Update...
Fix the sidebar (TODO: automate) in the finder panels
i could do this with brew cask, but it's a lot easier to download and install the client by hand. Also for google drive. I need both to sync before i can continue much of the below.
So login, authenticate and take a nap until directories are all synced. If you can't wait, at least sync:
OSXdotfilesUse selective sync in the dropbox app preference to do that.
- install lastpass
- install pocket
- install todoist (my current choice of todo)
- install simplenote
install the client, put the licence code in, fix amazon cloud drive as the destination
While you wait for Dropbox syncing, download iTerm2 and install it. Dont load your preferences for now, as they rely on the presence of zsh.
Install XCode CLI Tools from the command line.
xcode-select --install &> /dev/nullIf installing Xcode one needs to also agree to the license using sudo xcodebuild -license, but when i installed only the commandline tools, it doesn't let me run the command saying it needs a valid instance of xcode. So far not a limitation though.
There also seems to be a way to install the tools in headless mode without a GUI
Run the osx.sh script in this same gist.
Copy the ssh keys and gnupg keys. NOTE This is really bad SSH practice. SSH keys should only be transferred if you are really not using the old machine anymore. Otherwise you should make new keys for each laptop/machine you plan to ever use. That said, it is much more comfortable not having to re-upload all those public keys to github, servers, etc.
rsync -av migration/.ssh/ ~
rsync -av migration/.gnupg/ ~Add the old keychain to the list of your keychains.
cp migration/Keychains/login.keychain ~/Library/Keychains/login.oldmachine.keychainThen open the keychain in Keychain Access and type in the password for the login.oldmachine.keychain as it was in the old machine.
optional
copy every password that was there to the main login keychain. After you are done delete the login.oldmachine.keychain file in the directory, to avoid any future confusion.
If you have already a lot of keychain items than there are ways to speed this up, but not by much.
I have looked at using the security command from terminal but it's more cumbersome than copying and pasting all items across.
All of this may only be pleasing your inner OCD self rather than being useful.
UPDATE(nov 2015) -- In fact, this can easily break the keychain in the new system!! be warned. I won't be using this trick in the future
**TODO update the script so rathen than rsyncing it with a volume (or folder linke ~/Sync) it actually creates backups. (using for eg. duplicity, tar.gz, bup, burp, borg, arq) **
Set up a cron job on mac to regularly backup the keychain with borg.
Depending if you are taking the script from the gist:
git clone https://gist.github.com/elipapa/81251ec54f08e86a4819 kbak
mv kbak/com.elipapa.keychains-backup.plist ~/Library/LaunchAgents/
mv kbak/keychains_backup.sh ~/dotfiles/
sudo launchctl load ~/Library/LaunchAgents/com.elipapa.keychains-backup.plist
rm -rf kbakor from the dotfiles directory:
ln -s com.elipapa.keychains-backup.plist ~/Library/LaunchAgents/
sudo launchctl load ~/Library/LaunchAgents/com.elipapa.keychains-backup.plistDownload iTerm2 and install it. Dont load your preferences for now, as they rely on the presence of zsh.
Run the osx.sh script.
Also:
- change the caps lock to control in
keyboardsystem control panel
Install XCode CLI Tools from the command line. One needs to also agree to the license
if [ $(xcode-select -p &> /dev/null; printf $?) -ne 0 ]; then
xcode-select --install &> /dev/null
# Wait until the XCode Command Line Tools are installed
while [ $(xcode-select -p &> /dev/null; printf $?) -ne 0 ]; do
sleep 5
done
xcode-select -p &> /dev/null
if [ $? -eq 0 ]; then
# Prompt user to agree to the terms of the Xcode license
# https://github.com/alrra/dotfiles/issues/10
sudo xcodebuild -license
fi
fiInstructions should not change often, but can be double checked on the homepage. We must have set up the github token before starting to use homebrew
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"check everything worked ok
brew doctorinstall the key ones first and foremost
brew install git
brew install zshI forked the original repo to add some changes to paths and the modules I use. I want to keep as close as possible to the original runcoms offered by the repo.
# make sure you have the homebrew zsh installed
which zsh
#launch it
zsh
# (fork and )clone prezto
git clone --recursive https://github.com/sorin-ionescu/prezto.git "${ZDOTDIR:-$HOME}/.zprezto"
# using my repo
git clone --recursive https://github.com/elipapa/prezto.git ~/.zprezto
# Create a new Zsh configuration by copying the Zsh configuration files provided:
setopt EXTENDED_GLOB
for rcfile in "${ZDOTDIR:-$HOME}"/.zprezto/runcoms/^README.md(.N); do
ln -s "$rcfile" "${ZDOTDIR:-$HOME}/.${rcfile:t}"
done
# add the homebrew zsh to /etc/shells
# see http://stackoverflow.com/questions/791227/unable-to-update-my-bash-in-mac-by-macports
sudo -s
echo /usr/local/bin/zsh >> /etc/shells
exit
# now you can set the homebrew Zsh as your default shell:
chsh -s /usr/local/bin/zshInstructions on github on how to keep the fork up to date
One could reinstall iTerm2 using cask, which helps keeping things up to date later on.
tap caskroom/cask
tap caskroom/versions
install caskroom/cask/brew-cask
# remove old iTerm2
cask install iterm2Now that you have zsh installed, load the profiles that live in Dropbox. You can load the ones you have exported by hand or read the file copied from the directory like so:
defaults read ~/migration/com.googlecode.iterm2Also double check that iTerm2 starts with zsh as the command rather than login, as this could cause [massive slowdown on terminal startup].
To fix the powerline fonts
Link working directories into $HOME:
#i keep any dirty code i am working on in here
ln -s ~/Dropbox/dev ~
#i keep comp bio projects here
ln -s ~/Dropbox/bio ~
#i keep my published repos here
ln -s ~/Dropbox/src ~Keep user documents, user desktop and user fonts in cloud storage. Notice how this fonts folder already contains patched powerline fonts
sudo rm -rf ~/Documents
ln -s ~/Dropbox/Documents ~/Documents
sudo rm -rf ~/Desktop
ln -s ~/Google\ Drive/Desktop ~/Desktop
sudo rm -rf ~/Library/Fonts
ln -s ~/Dropbox/Fonts ~/Library/FontsAlso sync all preferences from the dropbox directory ~/Dropbox/OSXdotfiles/install_symlinks.sh.
One can also use Mackup which simply automates the same steps.
Once that is done, we can use the Brewfile created above to install packages and Casks.
brew tap Homebrew/bundle
brew bundle # in the directory where the Brewfile is located
# add the sublime command
ln -s /opt/homebrew-cask/Caskroom/sublime-text3/3047/Sublime\ Text.app/Contents/SharedSupport/bin/subl /usr/local/bin/subl
# old way was..
#ln -sf "/Applications/Sublime Text.app/Contents/SharedSupport/bin/subl" ~/bin/sublSynchronize the packages as they are kept in dropbox
# remove the local copies
rm -rf ~/Library/Application\ Support/Sublime\ Text\ 3/Packages
rm -rf ~/Library/Application\ Support/Sublime\ Text\ 3/Installed\ Packages
# link to the master copies
ln -s ~/Dropbox/OSXdotfiles/Subl/Packages ~/Library/Application\ Support/Sublime\ Text\ 3/Packages
ln -s ~/Dropbox/OSXdotfiles/Subl/Installed\ Packages ~/Library/Application\ Support/Sublime\ Text\ 3/Installed\ PackagesUse pip3 install -r requirements.txt with the attached file to install an inital list of packages.
Nothing needed except the installation of the IRKernel into the jupyter notebooks. In R console type:
install.packages(c('rzmq','repr','IRkernel','IRdisplay'),
repos = c('http://irkernel.github.io/', getOption('repos')))In the Brewfile java is installed using cask, which turns out to be the best way to do it. As suggested in the previous link, if more than one version is necessary, one can use jenv.
brew install jenv
brew cask tap caskroom/versions
brew cask install java7
brew cask install java6
brew cask install java # though this is already in the brew file
# install using zshrc
echo 'export PATH="$HOME/.jenv/bin:$PATH"' >> ~/.zshrc
echo 'eval "$(jenv init -)"' >> ~/.zshrc
# add the installed java versions
jenv add /Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home
brew cask install qlcolorcode qlstephen qlmarkdown quicklook-json qlprettypatch quicklook-csv betterzipql webp-quicklook suspicious-package && qlmanage -r
Install ruby using rbenv
#To use Homebrew's directories rather than ~/.rbenv add to your profile:
export RBENV_ROOT=/usr/local/var/rbenv
#To enable shims and autocompletion add to your profile:
if which rbenv > /dev/null; then eval "$(rbenv init -)"; fi
one should also turn on the ruby module in the .zpreztorc file.
Install pow and anvil
#Create the required host directories:
mkdir -p ~/Library/Application\ Support/Pow/Hosts
ln -s ~/Library/Application\ Support/Pow/Hosts ~/.pow
#Setup port 80 forwarding and launchd agents:
sudo pow --install-system
pow --install-local
#Load launchd agents:
sudo launchctl load -w /Library/LaunchDaemons/cx.pow.firewall.plist
launchctl load -w ~/Library/LaunchAgents/cx.pow.powd.plist
brew cask install anvil
Node and NPM will be installed by the Brewfile. Other than that you can install some key packages globally
npm install -g coffee-script
npm install -g grunt-cli
npm install -g jshint
Binary apps that have to be installed by hand (via the App Store):
- Pages
- Keynote
- Simplenote
- DayOne
Some binary apps benefit from additional work around their installation
brew cask install google-chrome
# # Allow installing user scripts via GitHub Gist or Userscripts.org
defaults write com.google.Chrome ExtensionInstallSources -array "https://gist.githubusercontent.com/" "http://userscripts.org/*"
defaults write com.google.Chrome.canary ExtensionInstallSources -array "https://gist.githubusercontent.com/" "http://userscripts.org/*"
# # Disable the all too sensitive backswipe on trackpads
# defaults write com.google.Chrome AppleEnableSwipeNavigateWithScrolls -bool false
# defaults write com.google.Chrome.canary AppleEnableSwipeNavigateWithScrolls -bool false
# # Disable the all too sensitive backswipe on Magic Mouse
# defaults write com.google.Chrome AppleEnableMouseSwipeNavigateWithScrolls -bool false
# defaults write com.google.Chrome.canary AppleEnableMouseSwipeNavigateWithScrolls -bool false
# Use the system-native print preview dialog
defaults write com.google.Chrome DisablePrintPreview -bool true
defaults write com.google.Chrome.canary DisablePrintPreview -bool true
# Expand the print dialog by default
defaults write com.google.Chrome PMPrintingExpandedStateForPrint2 -bool true
defaults write com.google.Chrome.canary PMPrintingExpandedStateForPrint2 -bool true
brew cask install transmission
# Use `~/Documents/Torrents` to store incomplete downloads
defaults write org.m0k.transmission UseIncompleteDownloadFolder -bool true
defaults write org.m0k.transmission IncompleteDownloadFolder -string "${HOME}/Documents/Torrents"
# Don’t prompt for confirmation before downloading
defaults write org.m0k.transmission DownloadAsk -bool false
# Trash original torrent files
defaults write org.m0k.transmission DeleteOriginalTorrent -bool true
# Hide the donate message
defaults write org.m0k.transmission WarningDonate -bool false
# Hide the legal disclaimer
defaults write org.m0k.transmission WarningLegal -bool false
authenticate with duo to download the installer from mit then you can login with userid+pwd from kerberos.
Find the client in google drive. Find the login in the password manager.
- drop the password manager. keychain icloud sync has improved significantly!
- alias git with hub which provides a slightly more intuitive interface
- brew install gist https://github.com/defunkt/gist#readme
- buy and try out oyster for regex building
- a cloud server https://www.digitalocean.com/
- a backup solution (though i am not convinced i need it with all my files in a cloud storage system) https://www.backblaze.com/