Adding summary, preparing to update screenshots per latest Amazon UI

roll_ipython_in_aws.md

What

Roll your own iPython Notebook server with Amazon Web Services (EC2) using their Free Tier.

What are we using? What do you need?

• An active AWS account. First time sign-ups are eligible for the free tier for a year
• One Micro Tier EC2 Instance
• With AWS we will use the stock Ubuntu Server AMI and customize it.
• Anaconda for Python.
• Coffee/Beer/Time

Why?

I had been looking for complete tutorials on setting up my own iPython Notebook on EC2 from scratch and I couldn't quite find what I was looking for. I hope someone finds this useful!

Why Anaconda over Enthought?

I've used the Enthought Free distribution on my own system for a while but it was missing some packages that I loved. For future uses I wanted a distribution that I didn't have to modify out of the box. I looked around and stumbled across Continuum's package. I liked where they were taking their products. Not to mention their free distribution had many, many awesome packages already included. Why not?

How (Summary)

Amazon Instance Launch

1. Choose AMI
2. Choose Instance Type
3. Configure Instance
4. Add Storage
5. Tag Instance
6. Configure Security Group
7. Review
8. Key

iPython Setup

1. SSH into Instance
2. Get Anaconda
3. ipython profile
4. SSL Certificate
5. ipython config
6. Launch & Login

How (Details)

AWS Console and EC2 Instance Launch.

• Head over to the AWS Console and log in. Choose "EC2" from the list

• Click "Launch Instance"

Choose AMI

• Choose an "Amazon Machine Instance" (AMI). AMI's contain the

1. operating system
2. app server, and
3. applications.

All free AMI's are marked with "Free teer eligible" badges. We're going to use Ubuntu Server 14.04 LTS (HVM), SSD Volume Type.

###Choose Instance Type

• Choose an "Instance Type". Instances are virtual servers with varying combinations of CPU, memory, storage, and networking capacity. The t2.micro type is the only free-tier eligible. It has:

1. 1 vCPU
2. 1 GiB memory
3. EBS (Elastic Block Storage), and
4. "Low to Moderate" Network Performance

###Configure Instance

• Configure Instance Details.

1. Instance: 1
2. Network: vpc-c7ecfea5 [172.31.0.0/16] (default)
3. Subnet: No preference
4. Auto-assign public IP: Use subnet setting (Enable)
5. IAM role: None
6. Shutdown behavior: stop
7. Enable termination protection: unchecked
8. Monitoring: unchecked
9. Tenancy: Shared tenancy (multi-tenant hardware)
10. Advanced Details: N/A

###Add Storage

• Add Storage ; use the defaults.

1. Size: 8 GiB
2. Volume Type: General Purpose
3. iOPS: 24/3000
4. Delete on Termination: Checked
5. Encrypted: Not Encrypted

Tag Instance

• You can create tags for machines. Tags are a series of Name-Value pairs. I haven't found the need to yet. Defaults!

Configure Security Groups

• The Security Group step is a very important step. A security group is a set of firewall rules that control traffic for your instance.
• This step can only be configured when you launch the instance!
• Choose "Create New Security Group" radio button. Pres "Add Rule" until you have these 3 rules:

1. SSH:

• Port: 22
• IP: 0.0.0.0/0

2. HTTPS

• Port: 443
• IP: 0.0.0.0/0

3. Custom TCP Rule

• Port: 8888

• IP: 0.0.0.0/0

• Note: 0.0.0.0/0 means access from ANY outside I.P. address. We will use port 8888 for our iPython Notebook server.

###Review Instance Launch

• Confirm all the details are consistent with previous steps. If the settings are wrong for a given step, there is an "Edit _______" link in the upper right; click to return to that step and fix the settings.

###Key

• "Click Launch to assign a key pair to your instance and complete the launch process."
• In summary (more info here):

1. To log in to your instance, you must create a key pair
2. specify the name of the key pair when you launch the instance
3. and provide the private key when you connect to the instance.

• On creating a key name, (or using an existing key name), your browser will prompt you to download a file with .pem extension -- this is your private key, don't lose it. Amazon uses 1024-bit SSH-2 RSA keys.

##iPython Setup (Detail)

SSH into Instance

• I'm going to assume you're using the Terminal in Mac OS X (or something vaguely Unix-y).
• If you're using Windows, consider downloading Git for Windows and using the Git Bash terminal.
• Alternatively, Amazon gives you instructions for using PuTTY from WIndows
• Right-Click on the instance you just created from the Instances panel then click Connect

• The Connect diaglog box will give you some code you can use to log in to your instance. In your Bash Terminal Navigate (using cd command) to the .pem file and run the command from there.
• WATCH OUT #1: If you haven't changed the permissions on your key then the server is going to reject the connection. CHMOD 400 (your PEM).
• WATCH OUT #2: Ubuntu wants you to login as ubuntu rather than root. Change that.
• Here's an example of my terminal ouput with both of those "gotcha's" happening.

• An example of successful login
• (from Nathaniel's edit; on a Windows; notice
• chmod 400 ipython.pem and
• chmod -r------ ipython.pem (the equivalent)
• will still leave the permissions as -r--r--r--
• but this is acceptable to the SSH client. More info on permissions [here] (http://www.grymoire.com/Unix/Permissions.html) ):
• 

Get Anaconda

• Now it's time to set up your python environment with Anaconda. This consists of three steps:

1. Use the wget command to download the Anaconda installer to your server.
2. Use the bash command to run the installer
3. Use the vim command (or any editor; or accept the automatic prompt after installer is finished) to modify .bashrc to add Anaconda to the run directory.

• Download Anaconda:

• (from Nathaniel's edit; on a Windows)

• Using Anaconda 3, Python 3.4, so he could use lxml and the latest arelle

• Run the Bash script. I've will leave everything with its default setting.

• Anaconda will install everthing in an folder in your home directory.

• (from Nathaniel's edit; on a Windows)

• Run the bash script

• After a few minutes the install will finish and tell you to put the folder that was just created at the top of your $PATH. Modify your .bashrc.

• (from Nathaniel's edit; on a Windows)
• Accept the Anaconda prompt to modify bashrc

• Make sure to reload your .bashrc with the source command.
• You may want to re-open the Git Bash and SSH back into this source.
• You can double check which Python you are using by calling: $ which python

• (from Nathaniel's edit; on a Windows)

• Say whaaaat? Now you've got a pretty good base to do all your Scientific Computing in Python.

Setting up iPython Notebook

• We're almost there. I can feel it. We've got a few more steps:

1. Create a iPython Profile for our server;
2. Create a password for our notebook log in;
3. Create a self-signed SSL certificate for HTTPS access;
4. and finally modify our ipython_notebook_configuration.py file. Much of (if not all) of this code follows the iPython Notebook doc very closely.

###iPython Profile

• From anywhere type: $ ipython profile create nbserver
• Notice it creates three config files. We will be editing the 2nd one in a later step.

1. ipython_config.py
2. ipython_notebook_config.py
3. ipython_nbconvert_config.py

• (from Nathaniel's edit; on a Windows)

####Password

• Next, we're going to create a password for your notebook server. I'm going to do everything from within iPython right now.
• You can access the shell commands by prepending your commands with "!".
• Some commands like "cd" and "ls" don't need an "!" in front. It's pretty awesome. See "Magic Functions" in the resources section.
• The output of passwd() is going to be used in the notebook configuration file later. So save/remember it!

• (from Nathaniel's edit; on a Windows)

###SSL Certificate

• Up next we're going to create a directory in our home folder called "certificates". In this folder we're going to save a self signed SSL certificate. Complete the questions as best you can.

####OpenSSL CNF Issues

• (from Nathaniel's edit; on a Windows)
• This SO answer explained how to locate the config file.

####Key and Cert to same File

• (from Nathaniel's edit; on a Windows)

• Other tutorials write -keyout and -out to different files

• (notice previous screenshot I used mykey.pem and mycert.pem respectively)

• You'll need them written to the same file (named mycert.pem [I used newcert.pem after screwing it up the first time!]) ; so use the command as provided in this tutorial:

• openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem -config _{path_to_your_config}___

• Correct

• We're going to need the name of the certifcate and the absoulte path for the notebook configuration file.

###iPython Config

• Last, but not least, it's time to modify our ipython_notebook_config.py file
• Remember it is inside the directory /home/ubuntu/.ipython/profile_nbserver with 2 other config files.
• Cruise over to ~/.ipython/profile_nbserver/ (or whatever you decided to call your profile) and open up the ipython_notebook_config.py file. (Nate made the mistake of editing ipython_config.py )

• You can either go through the config file and uncomment each of these one by one, or just use the iPython HTML Notebook docs and copy the code to the top of the file like I did. You'll need to change "certfile" and "password" to the values you just got from the previous steps. Change the "port" to 8888.

• Save!

• (from Nathaniel's edit; on a Windows)

###Launch notebook

• From anywhere run the following: $ ipython notebook --profile=nbserver

• If you're successful you should have output like the above!
• WATCH OUT: make sure your server is running at "all ip addresses on your system" rather than just 127.0.0.1 (or something like that). You WILL have a bad time.

Logging in to your Notebook from the Browser

• In order to log in via the browser we need the Public DNS for our sever. Cruise over to your AWS Console and select your instance from the Instances page. Under description you should find this:

• Using your public DNS go to your fav browser and type: https://your-Instance's-public-DNS:8888

• If successfull you'll get a warning about the self signed certificate. It's ok! Click Continue.

• You're in! Enter your password, create a notebook, and start doing...stuff!

What next?

A few things:

• Why not associate an Elastic IP to your instance(s)? Now you've got a static ip no matter what kind of instance you start.
• Use your Elastic IP and set up forwarding from a domain that you own. Instead of having a random IP address (albeit static) you can access your notebook via a memorable domain name. E.g. https://theres_always_money_in_the_banana_stand.com/notebook
• Create a Snapshot of your AMI. Now you can use this AMI on ANY instance type you like.
• Create an EBS volume to store all your data/work. Use your newly created Python AMI and run a massive instance for a few hours. Your setup Python setup will be the same but your hardware will be pretty beefy.
• Dive in to the .ipython folder in your home directory (or where you installed it) and modify your ipython_configuration.py. There are lots of options for configuring what packages are imported and what they are called at startup.

Resources and Further Reading

1. iPython HTML Notebok Documentation
2. iPython on EC2 Screen Cast
3. EC2 Screencast on YouTube
4. Contiuum.io Anaconda Download
5. Anaconda Package Index
6. iPython Tutorial -> Magic Functions

@AvatarGara, Short answer: why are you looking for "/...../profile_nbserver/inline"? No such directory "inline" , where does "inline" come from? Not mentioned in this gist. Long answer: I also get similar error when I execute this:

cd ~/home/ubuntu/.ipython/profile_nbserver/inline

It says:

-bash: cd: /home/ubuntu/home/ubuntu/.ipython/profile_nbserver: No such file or directory

Did you do this step, notice how it creates the directory "/home/ubuntu/.ipython/profile_nbserver", and specifically these 3 files inside there:

• ipython_config.py
• ipython_notebook_config.py
• ipython_nbconvert_config.py

But you look for a directory "inline". I cannot find it. Why you look for that?
I cannot find your information to contact you directly.

for Jupyter AKA ipython 4.0 you need to use
jupyter-notebook --config='~/.ipython/profile_nbserver/ipython_notebook_config.py'

Hi, I'm trying to do this on Windows. Did nate use a ubuntu VM?

Thanks for the revised write-up, it was quite helpful !

/Ram

OMG, for real why do we have to type that awfully long http address by hand to download anaconda?

@gaomrx You can also download it from the links on the official Anaconda download page.

To get Python 3 for 64-bit Linux, run the command wget http://repo.continuum.io/archive/Anaconda3-4.1.1-Linux-x86_64.sh

If you find that the configuration file ipython_notebook_config.py is missing, here's why:

Things have changed in jupyter since "the big switch", where notebook configurations have migrated to the .jupyter folder....

if your notebook config is missing (which was my case when starting ipython to create a profile), you can generate a jupyter config with the following:

jupyter notebook --generate-config

this should create a ~/.jupyter/jupyter_notebook_config.py file with relevant notebook settings ready to be uncommented/modified such as ports and passwords.

Hat tip: http://stackoverflow.com/a/33702208/3345375

Note: This advice is a little different from what townie mentions above. It worked for me, though.

Thank you very much!

Some of the screenshots are missing. Could you mix this? I am nor sure I am following the steps properly.