How to set up a new Magento installation on AWS EC2

_magento_on_ec2.md

💖 Magento & AWS EC2 💖

This is a checklist and guideline to get you through the setup and of a magento (1.9.x) install on ec2. This guide is meant to be general, but was made while setting up Magento 1.9.1 on an EC2 instance running Ubuntu 14.04.2.

Setting up security groups

You'll need to make 2 different security groups, one for the EC2 instance and one for the RDS instance. Make sure to name them accordingly so you can remember which to apply to which instance.

EC2 Security Group

• Go to the EC2 dashboard
• Go to security groups and create a new security group
• Inbound rules must include the following:
  • Type: HTTPS; Port Range: 443; Source: 0.0.0.0/0;
  • Type: HTTP; Port Range: 80; Source: 0.0.0.0/0;
  • Add SSH rules for any IPs used to connect directly to the machine
• Outbound rules must include the following:
  • Type: All Traffic; Protocol: All; Port Range: All; Destination: 0.0.0.0/0;

RDS Security Group

• Go to the EC2 dashboard
• Go to security groups and create a new security group
• Inbound rules must include the following:
  • Type: MYSQL/Aurora; Port Range: 3306; Source: 0.0.0.0/0;
  • Add SSH rules for any IPs used to connect directly to the machine
• Outbound rules must include the following:
  • Type: All Traffic; Protocol: All; Port Range: All; Destination: 0.0.0.0/0;

Switch over the domain

• Add a file called maintenance.flag to the root of the new magento installation
• Update index.php to allow your own IP address to view the site while in maintenance mode

$ip = $_SERVER['REMOTE_ADDR'];
$allowed = array('xx.xxx.xxx.xx'); // Put your IP in here

if (file_exists($maintenanceFile) && !in_array($ip, $allowed)) {
    include_once dirname(__FILE__) . '/errors/503.php';
    exit;
}

• Update the DNS Zone file A record to point to the IP of the new application host
• Wait for the internet to reflect your changes

Setting up SSL

Generate the certificate signing request using a private key located on the server

• ``openssl req -new -newkey rsa:2048 --nodes -keyout {path/to_private_key/{your_domain}.key} -out {your_domain}.csr```
• Enter information, be sure to specify the Common Name as the url that you're securing
• Copy the contents of the outputted csr file to the proper form where the domain is listed (godaddy.com etc..)

Installing OpenSSL and Mod_SSL

• Install OpenSSL if it isn't already sudo apt-get install openssl
• Install mod_ssl if it isn't already
• Enable mod_ssl by running sudo a2enmod ssl
• Restart apache sudo service apache2 restart
• Verify that it is enabled by including an info.php file in the root of your installation that contains the following

<?php phpinfo(); ?>

• Make sure OpenSSL is installed and that mod_ssl is listen under enabled modules

Installing the SSL certificate on your server

• Download the certificate from the site where the SSL was purchased. It should be at least 2 files
  • A certificate file with a .crt extension
  • A certificate chain file with a .crt extension ('bundle' should be somewhere in the file name)
• Upload these two files to an appropriate location on the server
• Edit the ssl module config file to include the configuration of these certificate files
• The default config can be found at /etc/apache2/sites-enabled. Add the following to the bottom of the file

<VirtualHost *:443>
  DocumentRoot /var/www/html
  SSLEngine on
  SSLCertificateFile {path_to_certificate_file}.crt
  SSLCertificateKeyFile {path_to_key_used_to_generate_certificate_file}.key
  SSLCertificateChainFile {path_to_certificate_chain_file}.crt
</VirtualHost>

• Make sure that the changes you made didn't break the apache config by running apachectl configtest
• Restart Apache for the changes to take effect by running sudo service apache2 restart
• Enable secure URLs on the front end via the Magento admin panel - System -> Config -> General -> Web -> Secure
• SSL should be working and enabled. Double check from running your domain on a SSL Checker

Sending Emails with Amazon SES

SMTP Settings

• Go to SES in the aws dashboard and select SMTP settings
• Create SMTP Credentials if you haven't already and save these somewhere (They are available to download, but only visible once on the dashboard)
• Using ASchroder's SMTP Pro Email Extension, enter the credentials as follows
  • Email Connection: Amazon SES
  • SES Access Key: {access key from above when you generated smtp credentials}
  • SES Secret Key: {secret key from above when you generated smtp credentials}

Verify Email Addresses

• Under Identities select Email Addresses and add the addresses that will be used to send application emails, click the link in the emails that get sent.

Verify Domains

• Under Identities select Domains and add the domain name you will be using, check the box to generate DKIM settings.
• Add the Domain Verification Record Set it provides as a text record in the DNS zone file
• Add all the rows in the DKIM Record Set as CNAME records in the DNS zone file
• Once you receive a confirmation email that the DKIM settings have been verified, go back to Identities -> Domains and select the domain you're using. Scroll down to DKIM, make sure DKIM Verification Status is verified and that DKIM is Enabled. Click the link to enable it if it isn't.

Transferring Customer Data (If Neccessary)

Delete test data

• Run delete_users.sql and delete_orders against the new production database to clear it out before launching
• See this link for info on how to do this

Import Old Data

If moving an old site to a new site, we want to migrate customer records over. To do this we have to export customer and order tables, import them into the new database and then make sure the table with ID counters is updated.

• To export customer information, use mysqldump on the customer tables. The command is

mysqldump -u {user} -p {database_name} customer_address_entity customer_address_entity_datetime customer_address_entity_decimal customer_address_entity_int customer_address_entity_text customer_address_entity_varchar customer_entity customer_entity_datetime customer_entity_decimal customer_entity_int customer_entity_text customer_entity_varchar > customer_dump.sql

• To export orders, use mysqldump on the order tables. The command is

mysqldump -u {user} -p {database_name} sales_flat_creditmemo sales_flat_creditmemo_comment sales_flat_creditmemo_grid sales_flat_creditmemo_item sales_flat_invoice sales_flat_invoice_comment sales_flat_invoice_grid sales_flat_invoice_item sales_flat_order sales_flat_order_address sales_flat_order_grid sales_flat_order_item sales_flat_order_payment sales_flat_order_status_history sales_flat_quote sales_flat_quote_address sales_flat_quote_address_item sales_flat_quote_item sales_flat_quote_item_option sales_flat_quote_payment sales_flat_quote_shipping_rate sales_flat_shipment sales_flat_shipment_comment sales_flat_shipment_grid sales_flat_shipment_item sales_flat_shipment_track sales_invoiced_aggregated sales_invoiced_aggregated_order > order_dump.sql

• Then import this data by using the following commands

> mysql -u {user} -p {database_name} < customer_dump.sql
> mysql -u {user} -p {database_name} < order_dump.sql

Ensure entity type IDs have the correct starting value

• Check eav_entity_store table and ensure for each entity type, the last_increment_id is set to the last id in the corresponding entity type table. This will allow new orders to be placed by telling the system what number to auto increment from.
  • select increment_last_id from eav_entity_store where entity_type_id = 5; will give you the last increment id for an order
  • 1 = Customer
  • 2 = Customer Address
  • 3 = Catalog Category
  • 4 = Catalog Product
  • 5 = Order
  • 6 = Invoice
  • 7 = Credit Memo
  • 8 = Shipment
• I think you should be able to export the eav_entity_store table but I haven't done it yet so I'm not certain
• More information can be found at this link

Enable The Magento Cron

We have to enable the cron because Magento 1.9+ puts emails into a queue that it clears via a cron job

• Verify that the cron is running sudo service cron status
• Check for currently running cron jobs crontab -l
• If you haven't set one up, open the crontab to set up the cronjob crontab -e
• Use the following line to set up a cron that runs the cron.php file every 5 minutes
• */5 * * * * /usr/bin/php /var/www/html/cron.php
• Verify the path to the php install (i.e. /usr/bin/php) by running which php
• The third argument is the path to the cron file which is found in the root of the magento install
• More information can be found at this link

Trouble Shooting

• Make sure the script is executing correctly by running via CLI php /var/www/html/cron.php
• Make sure the script is executable (permissions 755+)

Setting File Permissions

Setting the correct file permissions is important to protect the installation from attacks and accidental file overwrites. Magento recommended instructions for setting file permissions can be found here.

DO NOT follow these instructions blindly, as they set all files and directories to fairly strict permissions. It changes the owner of all files and subdirectories to the apache user (www-data), with the majority of these files set to read-only permissions. This is a problem if you initialize the git repo on the server and run git pull as the normal logged in user (ubuntu), it will fail because ubunu doesn't have write permission (insert 3 hours of head -> keyboard banging here).

My recommendation is to:

1. Make all security changes to files contained in the repo locally first.
2. Do not change permissions on the live site, not only can this cause errors with files not being readable by the server, but you will run into git issues when trying to sync changes.
3. Enable stricter permissions a single directories at a time. Make sure if the owner of the file is changed to the ubuntu user, that it is still readable by the apache user (which should be part of the user group)

delete_orders.sql

SET FOREIGN_KEY_CHECKS=0;
 
##############################
# SALES RELATED TABLES
##############################
TRUNCATE `sales_flat_creditmemo`;
TRUNCATE `sales_flat_creditmemo_comment`;
TRUNCATE `sales_flat_creditmemo_grid`;
TRUNCATE `sales_flat_creditmemo_item`;
TRUNCATE `sales_flat_invoice`;
TRUNCATE `sales_flat_invoice_comment`;
TRUNCATE `sales_flat_invoice_grid`;
TRUNCATE `sales_flat_invoice_item`;
TRUNCATE `sales_flat_order`;
TRUNCATE `sales_flat_order_address`;
TRUNCATE `sales_flat_order_grid`;
TRUNCATE `sales_flat_order_item`;
TRUNCATE `sales_flat_order_payment`;
TRUNCATE `sales_flat_order_status_history`;
TRUNCATE `sales_flat_quote`;
TRUNCATE `sales_flat_quote_address`;
TRUNCATE `sales_flat_quote_address_item`;
TRUNCATE `sales_flat_quote_item`;
TRUNCATE `sales_flat_quote_item_option`;
TRUNCATE `sales_flat_quote_payment`;
TRUNCATE `sales_flat_quote_shipping_rate`;
TRUNCATE `sales_flat_shipment`;
TRUNCATE `sales_flat_shipment_comment`;
TRUNCATE `sales_flat_shipment_grid`;
TRUNCATE `sales_flat_shipment_item`;
TRUNCATE `sales_flat_shipment_track`;
TRUNCATE `sales_invoiced_aggregated`;            # ??
TRUNCATE `sales_invoiced_aggregated_order`;        # ??
TRUNCATE `log_quote`;
 
ALTER TABLE `sales_flat_creditmemo_comment` AUTO_INCREMENT=1;
ALTER TABLE `sales_flat_creditmemo_grid` AUTO_INCREMENT=1;
ALTER TABLE `sales_flat_creditmemo_item` AUTO_INCREMENT=1;
ALTER TABLE `sales_flat_invoice` AUTO_INCREMENT=1;
ALTER TABLE `sales_flat_invoice_comment` AUTO_INCREMENT=1;
ALTER TABLE `sales_flat_invoice_grid` AUTO_INCREMENT=1;
ALTER TABLE `sales_flat_invoice_item` AUTO_INCREMENT=1;
ALTER TABLE `sales_flat_order` AUTO_INCREMENT=1;
ALTER TABLE `sales_flat_order_address` AUTO_INCREMENT=1;
ALTER TABLE `sales_flat_order_grid` AUTO_INCREMENT=1;
ALTER TABLE `sales_flat_order_item` AUTO_INCREMENT=1;
ALTER TABLE `sales_flat_order_payment` AUTO_INCREMENT=1;
ALTER TABLE `sales_flat_order_status_history` AUTO_INCREMENT=1;
ALTER TABLE `sales_flat_quote` AUTO_INCREMENT=1;
ALTER TABLE `sales_flat_quote_address` AUTO_INCREMENT=1;
ALTER TABLE `sales_flat_quote_address_item` AUTO_INCREMENT=1;
ALTER TABLE `sales_flat_quote_item` AUTO_INCREMENT=1;
ALTER TABLE `sales_flat_quote_item_option` AUTO_INCREMENT=1;
ALTER TABLE `sales_flat_quote_payment` AUTO_INCREMENT=1;
ALTER TABLE `sales_flat_quote_shipping_rate` AUTO_INCREMENT=1;
ALTER TABLE `sales_flat_shipment` AUTO_INCREMENT=1;
ALTER TABLE `sales_flat_shipment_comment` AUTO_INCREMENT=1;
ALTER TABLE `sales_flat_shipment_grid` AUTO_INCREMENT=1;
ALTER TABLE `sales_flat_shipment_item` AUTO_INCREMENT=1;
ALTER TABLE `sales_flat_shipment_track` AUTO_INCREMENT=1;
ALTER TABLE `sales_invoiced_aggregated` AUTO_INCREMENT=1;
ALTER TABLE `sales_invoiced_aggregated_order` AUTO_INCREMENT=1;
ALTER TABLE `log_quote` AUTO_INCREMENT=1;

#########################################
# DOWNLOADABLE PURCHASED
#########################################
TRUNCATE `downloadable_link_purchased`;
TRUNCATE `downloadable_link_purchased_item`;
 
ALTER TABLE `downloadable_link_purchased` AUTO_INCREMENT=1;
ALTER TABLE `downloadable_link_purchased_item` AUTO_INCREMENT=1;

#########################################
# RESET ID COUNTERS
#########################################
TRUNCATE `eav_entity_store`;
ALTER TABLE  `eav_entity_store` AUTO_INCREMENT=1;

SET FOREIGN_KEY_CHECKS=0;

delete_users.sql

SET FOREIGN_KEY_CHECKS=0;

##############################
# CUSTOMER RELATED TABLES
##############################

TRUNCATE `customer_address_entity`;
TRUNCATE `customer_address_entity_datetime`;
TRUNCATE `customer_address_entity_decimal`;
TRUNCATE `customer_address_entity_int`;
TRUNCATE `customer_address_entity_text`;
TRUNCATE `customer_address_entity_varchar`;
TRUNCATE `customer_entity`;
TRUNCATE `customer_entity_datetime`;
TRUNCATE `customer_entity_decimal`;
TRUNCATE `customer_entity_int`;
TRUNCATE `customer_entity_text`;
TRUNCATE `customer_entity_varchar`;
TRUNCATE `tag`;
TRUNCATE `tag_relation`;
TRUNCATE `tag_summary`;
TRUNCATE `tag_properties`;            ## CHECK ME
TRUNCATE `wishlist`;
TRUNCATE `log_customer`;
 
ALTER TABLE `customer_address_entity` AUTO_INCREMENT=1;
ALTER TABLE `customer_address_entity_datetime` AUTO_INCREMENT=1;
ALTER TABLE `customer_address_entity_decimal` AUTO_INCREMENT=1;
ALTER TABLE `customer_address_entity_int` AUTO_INCREMENT=1;
ALTER TABLE `customer_address_entity_text` AUTO_INCREMENT=1;
ALTER TABLE `customer_address_entity_varchar` AUTO_INCREMENT=1;
ALTER TABLE `customer_entity` AUTO_INCREMENT=1;
ALTER TABLE `customer_entity_datetime` AUTO_INCREMENT=1;
ALTER TABLE `customer_entity_decimal` AUTO_INCREMENT=1;
ALTER TABLE `customer_entity_int` AUTO_INCREMENT=1;
ALTER TABLE `customer_entity_text` AUTO_INCREMENT=1;
ALTER TABLE `customer_entity_varchar` AUTO_INCREMENT=1;
ALTER TABLE `tag` AUTO_INCREMENT=1;
ALTER TABLE `tag_relation` AUTO_INCREMENT=1;
ALTER TABLE `tag_summary` AUTO_INCREMENT=1;
ALTER TABLE `tag_properties` AUTO_INCREMENT=1;
ALTER TABLE `wishlist` AUTO_INCREMENT=1;
ALTER TABLE `log_customer` AUTO_INCREMENT=1;

SET FOREIGN_KEY_CHECKS=1;