Hosting a Django application on AWS EC2 (running AMI Linux) - steps involved

Host_Django_on_AWS

As a newbie, I recently went through the process of migrating my Django application that was developed and tested on a Windows environment onto AWS (AMI Linux) to host it using Apache and mod_wsgi. The source code for the application was on GitHub.

My sincere thanks to the people who have contributed the References that I've listed in my notes below. They were immensely helpful for me to get through this process.

I'm posting my notes compiling all the steps involved, in case it helps others.
This is NOT production level hosting.

I Creating AWS account and instances
=====================================
Reference : http://docs.aws.amazon.com/gettingstarted/latest/wah-linux/web-app-hosting-intro.html

- Logged into AWS website with my existing account
- Created IAM credentials
- Logged out of AWS, and signed in with IAM credentials
- Created keypair, security group, AWS VPC, public/private subnets, routes, and redundant EC2 instances (loaded with AMI Linux)
- Note: Did not launch RDS instances, because that is not part of the free tier service.

On Windows laptop:

- Installed PuTTY on Windows laptop, used PuTTYgen to generate pvt key (.ppk file) from downloaded keypair (.pem file)
- Used PuTTY and connected to AWS EC2's public ip-address as ec2-user over SSH with pvt key generated above

II Installing required packages on AWS EC2
===========================================
References:

https://gist.github.com/havencruise/8307140
http://simononsoftware.com/virtualenv-tutorial-part-2/
http://nickpolet.com/blog/deploying-django-on-aws/1/
http://thecodeship.com/deployment/deploy-django-apache-virtualenv-and-mod_wsgi/
https://code.google.com/p/modwsgi/wiki/IntegrationWithDjango

On AWS EC2, Python v2.6.9 is pre-installed, but Django 1.7 (which I used) requires Python 2.7 or higher.
Hence need to create virtualenv to isolate the two Python environments. Even otherwise, it is a good practice to isolate the Python instance used for your project from that used by the host machine for its own applications.

On AWS EC2:

$ sudo yum update  ----> get latest security patches and bugfixes
$ sudo yum install http ---> installed Apache web server ver 2.2.29 and its dependencies
$ sudo yum install httpd-devel  ---> httpd-devel ver 2.2.9, needed to make mod_wsgi as an Apache module
$ sudo yum install mysql mysql-server mysql-devel --> installed MySQL 5.5-1.6, MySQL Server 5.5-1.6

$ sudo service httpd start ---> To start Apache server
Connect via browser to http://<ip address>:80 ---> It will display the test page for Amazon Linux AMI with Apache!

---------------------------------------- Setting up Python environment and modules -------------------------------------------------

$ curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"   ---> install get-pip script
$ sudo python get-pip.py   ---> execute the script and install pip. Henceforth, use PIP to install all Python packages

$ sudo pip install virtualenv --> installed virtualenv-12.0.7, to isolate Python 2.7 used for Django and the Python2.6.9 on AWS EC2
$ sudo pip install virtualenvwrapper ; source /usr/bin/virtualenvwrapper.sh   --> provides wrapper commands to create/activate/deactivate/remove a virtual environment

$ sudo yum install python27 --> installed Python 2.7.8 (as Django 1.7 is incompatible with Python 2.6)
$ which python27 ---> gives path as /usr/bin/python27

$ mkvirtualenv  test_env -p /usr/bin/python27  ---> associates this virtual environment test_env with the Python 2.7 interpreter

[Note: If not using virtualenvwrapper, the commands are:

$ virtualenv virtualenv-27  ---> creates new directory virtualenv-27 and installs python, pip, etc in subdirectories
$ source virtualenv-27/bin/activate --> activates new shell in this virtual environment ]

(test_env)$ pip install Django==1.7  ---> installed Django ver 1.7, compatible with Python 2.7.x

-------------------------- Installing mod_wsgi -----------------------------------------------------------------------------

Django uses mod_wsgi (WSGI = Web Server Gateway Interface) to deploy applications on Apache httpd server.
mod_wsgi exists as a Python module, as well as an Apache module.

On Windows laptop:

     From https://pypi.python.org/pypi/mod_wsgi/4.4.10
     Downloaded mod_wsgi-4.4.10.tar.gz onto Windows
     Installed pscp and ftp'd the file to AWS EC2 instance :
     C:\ pscp -i <keypair.ppk> <source file> ec2-user@<ip address>:/home/ec2-user

On AWS EC2 :

$ tar xvf mod_wsgi-4.4.10.tar.gz  --> unzips and untars contents into mod_wsgi-4.4.10 directory

$ sudo yum install python27-devel --> Python 2.7 development package, needed to make mod_wsgi as a Python module

$ cd mod_wsgi-4.4.10
$ workon test_env ---> to choose Python27 to compile with

This step needs to be repeated every time a new virtualenv is created :
----------------------------------------------------------------------
(test_env)$ python setup.py install  --> makes and installs mod_wsgi-express script under  /home/ec2-user/.virtualenvs/test_env/bin
and /home/ec2-user/.virtualenvs/test_env/lib/python2.7/site-packages/mod_wsgi-4.4.10-py2.7-linux-x86_64.egg

(test_env)$ sudo make install --> makes and installs the Apache module into the standard location for all Apache modules. In this case, /usr/lib64/httpd/modules

---------------------------Stitching together Django, mod_wsgi and Apache for a test_site project -------------------------------------
$mkdir test
$cd test
(test_env)$ django-admin.py startproject test_site --> results in the following directory structure :
/home/ec2-user/test/
                  test_site/
                          manage.py
                          test_site/
                                  settings.py
                                  wsgi.py

Edit Django project files:
----------------------------
In file settings.py, add to
INSTALLED_APPs = (
     .................
     'mod_wsgi.server',
)

Edit wsgi.py to add the lines:

import sys
sys.path.append('home/ec2-user/test/test_site')

before the line (auto-generated by Django):
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "test_site.settings")

Tell Apache server about wsgi:
-------------------------------
$ cd /etc/httpd/conf.d
Create a new file wsgi.conf
$ sudo vim wsgi.com
and add this line in it :
     LoadModule wsgi_module modules/mod_wsgi.so  ---> this will look where all Apache modules are  ie /usr/lib64/httpd/modules

$ cd /etc/httpd/conf
$ sudo vim httpd.conf
and add these lines in it:

WSGIScriptAlias / /home/ec2-user/test/test_site/test_site/wsgi.py

WSGIPythonPath /home/ec2-user/test:/home/ec2-user/.virtualenvs/test_env/lib/python2.7/site-packages

<Directory /home/ec2-user/test/test_site/test_site>
<Files wsgi.py>
Order deny,allow
Allow from all
</Files>
</Directory>

$ sudo service httpd restart ---> To restart Apache server

Ensure that others have execute permissions on /home/ec2-user : chmod o+x /home/ec2-user, 
because Apache will point the root directory / to sub-directories of /home/ec2-user

Pointing the browser to http://<ip address>:80 now shows the default Django page!


III Migrating my own Django project from a Windows environment to AWS EC2
=========================================================================
On Windows laptop:

C:\ cd <path\to\myproject>
C:\ pip freeze > requirements.txt         ----> this generates the list of needed packages for this project. 

FTP this file requirements.txt to AWS EC2 server.
The Django application code resides on GitHub.

On AWS EC2:

Pull my Django project code from GitHub to AWS
----------------------------------------------
References :
https://help.github.com/articles/set-up-git/#platform-linux
http://git-scm.com/book/en/v2/Git-Basics-Getting-a-Git-Repository

Install git on AWS EC2

$ sudo yum install git   ---> installs git on AWS EC2

Configure git with user's name and email address

$  git config --global user.name "MY NAME"
$  git config --global user.email "<email address>"

Enable caching of GitHub password in git for 1 hr so as to clone with HTTPS later
$ git config --global credential.helper cache
$ git config --global credential.helper 'cache  --timeout=3600'

Clone existing repository from GitHub onto AWS EC2

$ git clone -b <branch> --single-branch https://github.com/<username>/<repository>.git  mysite

where the argument:
-b <branch> --> indicates which branch to clone
option single-branch --> tells git to fetch only this branch
<URL> --> is obtained by copying from GitHub repository's sidebar
mysite --> is the local directory to be created on AWS EC2, inside which the .git directory contains the local repository

This fetches all the project files for my project from GitHub.

Resulting directory structure:

/home/ec2-user/mysite/
                  myproject/
                          manage.py
                          myproject/
                                  settings.py
                                  wsgi.py

Create a new virtual environment and install required packages
--------------------------------------------------------------
$ mkvirtualenv  my_env -p /usr/bin/python27  --> creates a new virtualenv my_env and activates it

(my_env)$ cd mod_wsgi-4.4.10

(my_env)$ python setup.py install  --> Installs mod_wsgi python module in my_env site-packages
Installing mod_wsgi-express script to /home/ec2-user/.virtualenvs/my_env/bin
Installed /home/ec2-user/.virtualenvs/my_env/lib/python2.7/site-packages/mod_wsgi-4.4.10-py2.7-linux-x86_64.egg

$ cd mysite/myproject
Copy the requirements.txt that was transferred from Windows into this directory.
$ pip install -r requirements.txt
Installed required packages

Specific edits in myproject directory :
---------------------------------------------------
$ cd mysite/myproject/myproject

Edit settings.py: 
Add to
INSTALLED_APPs = (
     .................
     'mod_wsgi.server',
)
TEMPLATE_DIR ---> set new path on Linux
MEDIA_ROOT ---> set new path on Linux

Edit myproject/wsgi.py to add:
import sys
sys.path.append('home/ec2-user/mysite/myproject')

before the auto-generated line:
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")

Specific edits to point Apache webserver to this project:
-----------------------------------------------------------------
$ cd /etc/httpd/conf
$ sudo vim httpd.conf
and add these lines in it:

WSGIScriptAlias      /      /home/ec2-user/mysite/myproject/myproject/wsgi.py

WSGIPythonPath      /home/ec2-user/mysite:/home/ec2-user/.virtualenvs/my_env/lib/python2.7/site-packages

<Directory /home/ec2-user/mysite/myproject/myproject>
<Files wsgi.py>
Order deny,allow
Allow from all
</Files>
</Directory>

$ sudo service httpd restart ---> To restart Apache server

Pointing the browswer to http://<ip address>:80 now shows the home page of MyProject. 
However, it does not use the project's stylesheets and other static files.

Also, individual app pages cannot be loaded because the MySQL database has not been setup.


IV Setup MySQL database and connect via Django project
========================================================

$ sudo service mysqld start  ---> starts the MySQL service on AWS EC2
$ sudo service mysqld status
mysqld (pid  19959) is running...

$ mysql_secure_installation
Follow the prompts, to set root password and other privileges


$ mysql -u root -p
Enter password: <password>
> create database XYZ;
> create user '<user>' identified by '<password>'
> grant all on XYZ.* to '<user>';
> quit;

$ workon my_env
$ cd /home/ec2-user/mysite/myproject

Edit myproject/settings.py : set user '<user>' and password associated with database XYZ  (case sensitive!)

$ python manage.py syncdb
Prompts to setup admin superuser and password
Connects Django applications of this project to MySQL database staffyoutrust.

Connecting browser to http://<ip address>:80 and navigating to application page now shows that there is no
data loaded in the application.

V Import data into database
===========================
On Windows:

Export data that has been entered into local database into a json file.
C:\<path\to\myproject> python manage.py dumpdata --natural-foreign --natural-primary > database.json

Transfer it to AWS EC2
C:\<path\to\myproject> pscp -i <keypair.ppk> database.json ec2-user@<public ip addr>:/home/ec2-user/myrepo/myproject

On AWS EC2:

$ python manage.py loaddata database.json  ---> imports the data into XYZ database

Connecting via browser and navigating to the application page now shows all the earlier data that was present in my Windows environment.

VI Serving static files
=========================
Refer:
https://docs.djangoproject.com/en/1.7/howto/static-files/
https://docs.djangoproject.com/en/1.7/howto/static-files/deployment/
https://docs.djangoproject.com/en/1.7/howto/deployment/wsgi/modwsgi/#serving-files


On AWS EC2:

There are  specific static files (.css, .js ) and media (images) for the Django admin pages as well as for the project and its apps.

The Django admin static and media files live under django.contrib.admin installed in site-packages for this virtualenv.
Tho project/app based ones live under the project directory structure.

To tell Apache to serve them, add these lines to /etc/httpd/conf/httpd.conf:

Alias /static/admin/      /home/ec2-user/.virtualenvs/my_env/lib/python2.7/site-packages/django/contrib/admin/static/admin/
Alias /media/admin/      /home/ec2-user/.virtualenvs/my_env/lib/python2.7/site-packages/django/contrib/admin/media/media/
Alias      /media/      /home/ec2-user/mysite/myproject/myapp/media/
Alias      /static/      /home/ec2-user/mysite/myproject/myapp/static/

<Directory /home/ec2-user/mysite/myproject/myproject/myapp/media>
Order deny,allow
Allow from all
</Directory>

<Directory /home/ec2-user/mysite/myproject/myproject/myapp/static>
Order deny,allow
Allow from all
</Directory>

<Directory /home/ec2-user/.virtualenvs/my_env/lib/python2.7/site-packages/django/contrib/admin/>
Order deny,allow
Allow from all
</Directory>


The best way to serve static files during deployment is by collecting them all into a single directory from where they are served.

$ cd /home/ec2-user/mysite/myproject
$ vim myproject/settings.py

Set STATIC_ROOT = '/var/www/mysite/static/'  ---> Directory where static files that are collected from each installed app's static sub-directory will be stored.
Set permissions so that collectstatic can write into this directory

$ python manage.py collectstatic --> copies all static files into it /var/www/mysite/static

$ sudo service httpd restart

Pointing the browser to home page as well as admin page shows that corresponding stylesheets are being used.

VII Push back changed files in local repository to GitHub
=========================================================
Refer: http://www.djangobook.com/en/2.0/chapter12.html

Before pushing back the altered settings.py file (as it will not work on a Windows environment now), edit it to specify conditional settings of specific parameters that are different for Windows and on AWS.

if socket.gethostname == 'my-pc':
     <PARAMETER> = <Windows environment setting>
else:
     <PARAMETER> = <AWS Linux environment setting>

Such parameters are DEBUG (True on Windows, False on AWS), INSTALLED_APPS (include mod_wsgi only on Linux), MEDIA_ROOT, STATIC_ROOT, TEMPLATE_DIRS.

Refer:
http://git-scm.com/book/en/v2/Git-Basics-Recording-Changes-to-the-Repository
http://git-scm.com/docs/git-push

$ cd /home/ec2-user/mysite
$ git status
On branch dev1
Your branch is up-to-date with 'origin/dev1'.
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   myproject/myproject/settings.py 
        modified:   myproject/myproject/wsgi.py   

no changes added to commit (use "git add" and/or "git commit -a")

$  git commit -a -m "Comment"   ---> Commits changes to local repository

$ git status
On branch dev1
Your branch is ahead of 'origin/dev1' by 1 commit.
  (use "git push" to publish your local commits)
nothing to commit, working directory clean

$ git push origin   ---> Pushes changes in current branch to corresponding remote branch that 'git pull' uses to update current branch. 
Username for 'https://github.com': <username>
Password for 'https://[email protected]':<password>
Counting objects: x, done.
Compressing objects: 100% (x/x), done.
Writing objects: 100% (x/x), y KiB | 0 bytes/s, done.
Total x (delta x), reused 0 (delta 0)
To https://github.com/username/mysite.git
   bb62814..b9fbfe0  dev1 -> dev1

small edit for those who use Apache 2.4
2.2 configuration:

Order allow,deny
Allow from all

2.4 configuration:
Require all granted