Roll your own iPython Notebook server with Amazon Web Services (EC2) using their Free Tier.
- An active AWS account. First time sign-ups are eligible for the free tier for a year
- One Micro Tier EC2 Instance
- With AWS we will use the stock Ubuntu Server AMI and customize it.
- Anaconda for Python.
- Coffee/Beer/Time
I had been looking for complete tutorials on setting up my own iPython Notebook on EC2 from scratch and I couldn't quite find what I was looking for. I hope someone finds this useful!
I've used the Enthought Free distribution on my own system for a while but it was missing some packages that I loved. For future uses I wanted a distribution that I didn't have to modify out of the box. I looked around and stumbled across Continuum's package. I liked where they were taking their products. Not to mention their free distribution had many, many awesome packages already included. Why not?
- Head over to the AWS Console and log in.
- We're going to Launch an Instance. If this is your first time at the Rodeo you won't have any instances listed.
- We're going to use the Classic Wizard for the Instance Creation
- There are many OS's to choose from. All the eligble OS's eligible for the Free Tier are marked by big, bold stars. We're going to use Ubuntu Server 12.04.1 LTS.
- The free tier is only eligible for the T1 Micro. I will use all defaults here.
- Again in the Advanced Instaces options I leave all options at their default settings.
- You can create tags for machines. I haven't found the need to yet. Defaults!
-
The Wizard is going to ask you to create a public key (or use an existing one if this isn't your first time). Create and download that key and save it in a nifty (read: memorable) spot.
-
The Security Group step is a very important step. This step can only be configured when you launch the instance! Under the Create New Security Group.
- You should have rules for SSH(22): 0.0.0.0/0, HTTPS(443): 0.0.0.0/0, and 8888: 0.0.0.0/0. Where 0.0.0.0/0 means access from ANY outside I.P. address. We will use port 8888 for our iPython Notebook server. Your rules should look like the right hand size of the photo below.
- This is the last page of the Wizard! Review the options you set and if they look good the you're all set.
- I'm going to assume you're using the Terminal in Mac OS X (or something vaguely Unix-y).
- Right-Click on the instance you just created from the Instances panel then click Connect
- The Connect diaglog box will give you some code you can use to log in to your instance. In your terminal Navigate to where you saved your PEM key and run the command from there. WATCH OUT #1: If you haven't changed the permissions on your key then the server is going to reject the connection. CHMOD 400 (your PEM). WATCH OUT #2: Ubuntu wants you to login as ubuntu rather than root. Change that.
- Here's an example of my terminal ouput with both of those "gotcha's" happening.
- If all goes well then you're logged in!
- Now it's time to set up your python environment with Anaconda. Use wget to download the Anaconda installer to your server
- Run the Bash script. I've will leave everything with its default setting. Anaconda will install everthing in an folder in your home directory.
- After a few minutes the install will finish and tell you to put the folder that was just created at the top of your $PATH. Modify your .bashrc.
- Make sure to reload your .bashrc with the source command. You can double check which Python you are using by calling: $ which python
- Say whaaaat? Now you've got a pretty good base to do all your Scientific Computing in Python.
-
We're almost there. I can feel it. We've got a few more steps: Create a iPython Profile for our server; Create a password for our notebook log in; Create a self-signed SSL certificate for HTTPS access; and finally modify our ipython_notebook_configuration.py file. Much of (if not all) of this code follows the iPython Notebook doc very closely.
-
From anywhere type: $ ipython profile create nbserver
-
Next, we're going to create a password for your notebook server. I'm going to do everything from within iPython right now. You can access the shell commands by prepending your commands with "!". Some commands like "cd" and "ls" don't need an "!" in front. It's pretty awesome. See "Magic Functions" in the resources section.
-
The output of passwd() is going to be used in the notebook configuration file later. So save/remember it!
- Up next we're going to create a directory in our home folder called "certificates". In this folder we're going to save a self signed SSL certificate. Complete the questions as best you can.
- We're going to need the name of the certifcate and the absoulte path for the notebook configuration file.
- Last, but not least, it's time to modify our ipython_notebook_config.py file.
- Cruise over to ~/.ipython/profile_nbserver/ (or whatever you decided to call your profile) and open up the ipython_notebook_config.py file.
- You can either go through the config file and uncomment each of these one by one, or just use the iPython HTML Notebook docs and copy the code to the top of the file like I did. You'll need to change "certfile" and "password" to the values you just got from the previous steps. Change the "port" to 8888.
- Save!
- From anywhere run the following:
$ ipython notebook inline --profile=nbserver
- If you're successful you should have output like the above! WATCH OUT: make sure your server is running at "all ip addresses on your system" rather than just 127.0.0.1 (or something like that). You WILL have a bad time.
- In order to log in via the browser we need the Public DNS for our sever. Cruise over to your AWS Console and select your instance from the Instances page. Under description you should find this:
-
Using your public DNS go to your fav browser and type: https://your-Instance's-public-DNS:8888
-
If successfull you'll get a warning about the self signed certificate. It's ok! Click Continue.
- You're in! Enter your password, create a notebook, and start doing...stuff!
A few things:
- Why not associate an Elastic IP to your instance(s)? Now you've got a static ip no matter what kind of instance you start.
- Use your Elastic IP and set up forwarding from a domain that you own. Instead of having a random IP address (albeit static) you can access your notebook via a memorable domain name. E.g. https://theres_always_money_in_the_banana_stand.com/notebook
- Create a Snapshot of your AMI. Now you can use this AMI on ANY instance type you like.
- Create an EBS volume to store all your data/work. Use your newly created Python AMI and run a massive instance for a few hours. Your setup Python setup will be the same but your hardware will be pretty beefy.
- Dive in to the .ipython folder in your home directory (or where you installed it) and modify your ipython_configuration.py. There are lots of options for configuring what packages are imported and what they are called at startup.
@Vikramank If you don't have the right permissions for a certain file or directory, you may need to change its permissions or its owner. To do this on the command line, use
chmodorchownrespectively.Or sometimes you can override permissions. Just precede a command with the word sudo and that'll get things done.