my random notes on using ec2 and s3

0 ec2 and s3 notes.md

About s3 and location syntax

bucket location in 's3:' notation is s3://<bucket name>. this is different than the public http URI, which will be https://<budket name>.s3.amazonaws.com/file

For the most part, getting/putting is done with s3 syntax.

getting and putting (ec2)

1. install s3cmd via apt-get
2. s3cmd --configure
3. to put: s3cmd put s3://<bucket name>/<file> <local file path>
4. to get: s3cmd get s3://<bucket name>/<file>

Re: authorization:

make sure s3 bucket is accessable to authorized users. alternatively, setup IAM so that ec2 instance has appropriate access.

1 boto stuff.py

# make sure boto is configured
# http://boto.readthedocs.org/en/latest/boto_config_tut.html
# default would be ~/.boto with appropriate access key and secret

from re import *
from boto import *
from datetime import *
from time import *
con=connect_ec2()
con.get_spot_price_history(instance_type='c4.large')
hist= con.get_spot_price_history(start_time=datetime.isoformat(datetime.now()),availability_zone='us-east-1c')
# get prices for all c4 instances that are compute optimized and usually reasonably priced
[m.string for m in [re.search('c4.large|c4.xlarge|c4.2xlarge',str(i)) for i in hist if i.price<.03] if m]
 
'''
Instance Type  vCPU  Memory (GiB)  Storage (GB)  Networking Performance  Physical Processor      Clock Speed (GHz)  Intel AVX†  Intel AVX2†  Intel Turbo  EBS OPT  Enhanced Networking†
t2.micro       1     1             EBS Only      Low to Moderate         Intel Xeon family       2.5                Yes         -            Yes          -        -
t2.small       1     2             EBS Only      Low to Moderate         Intel Xeon family       2.5                Yes         -            Yes          -        -
t2.medium      2     4             EBS Only      Low to Moderate         Intel Xeon family       2.5                Yes         -            Yes          -        -
m3.medium      1     3.75          1 x 4 SSD     Moderate                Intel Xeon E5-2670 v2*  2.5                Yes         -            Yes          -        -
m3.large       2     7.5           1 x 32 SSD    Moderate                Intel Xeon E5-2670 v2*  2.5                Yes         -            Yes          -        -
m3.xlarge      4     15            2 x 40 SSD    High                    Intel Xeon E5-2670 v2*  2.5                Yes         -            Yes          Yes      -
m3.2xlarge     8     30            2 x 80 SSD    High                    Intel Xeon E5-2670 v2*  2.5                Yes         -            Yes          Yes      -
c4.large       2     3.75          EBS Only      Moderate                Intel Xeon E5-2666 v3   2.9                Yes         Yes          Yes          Yes      Yes
c4.xlarge      4     7.5           EBS Only      High                    Intel Xeon E5-2666 v3   2.9                Yes         Yes          Yes          Yes      Yes
c4.2xlarge     8     15            EBS Only      High                    Intel Xeon E5-2666 v3   2.9                Yes         Yes          Yes          Yes      Yes
c4.4xlarge     16    30            EBS Only      High                    Intel Xeon E5-2666 v3   2.9                Yes         Yes          Yes          Yes      Yes
c4.8xlarge     36    60            EBS Only      10 Gigabit              Intel Xeon E5-2666 v3   2.9                Yes         Yes          Yes          Yes      Yes
c3.large       2     3.75          2 x 16 SSD    Moderate                Intel Xeon E5-2680 v2   2.8                Yes         -            Yes          -        Yes
c3.xlarge      4     7.5           2 x 40 SSD    Moderate                Intel Xeon E5-2680 v2   2.8                Yes         -            Yes          Yes      Yes
c3.2xlarge     8     15            2 x 80 SSD    High                    Intel Xeon E5-2680 v2   2.8                Yes         -            Yes          Yes      Yes
c3.4xlarge     16    30            2 x 160 SSD   High                    Intel Xeon E5-2680 v2   2.8                Yes         -            Yes          Yes      Yes
c3.8xlarge     32    60            2 x 320 SSD   10 Gigabit              Intel Xeon E5-2680 v2   2.8                Yes         -            Yes          -        Yes
g2.2xlarge     8     15            1 x 60 SSD    High                    Intel Xeon  E5-2670     2.6                -           -            -            Yes      -
r3.large       2     15.25         1 x 32 SSD    Moderate                Intel Xeon E5-2670 v2   2.5                Yes         -            Yes          -        Yes
r3.xlarge      4     30.5          1 x 80 SSD    Moderate                Intel Xeon E5-2670 v2   2.5                Yes         -            Yes          Yes      Yes
r3.2xlarge     8     61            1 x 160 SSD   High                    Intel Xeon E5-2670 v2   2.5                Yes         -            Yes          Yes      Yes
r3.4xlarge     16    122           1 x 320 SSD   High                    Intel Xeon E5-2670 v2   2.5                Yes         -            Yes          Yes      Yes
r3.8xlarge     32    244           2 x 320 SSD   10 Gigabit              Intel Xeon E5-2670 v2   2.5                Yes         -            Yes          -        Yes
i2.xlarge      4     30.5          1 x 800 SSD   Moderate                Intel Xeon E5-2670 v2   2.5                Yes         -            Yes          Yes      Yes
i2.2xlarge     8     61            2 x 800 SSD   High                    Intel Xeon E5-2670 v2   2.5                Yes         -            Yes          Yes      Yes
i2.4xlarge     16    122           4 x 800 SSD   High                    Intel Xeon E5-2670 v2   2.5                Yes         -            Yes          Yes      Yes
i2.8xlarge     32    244           8 x 800 SSD   10 Gigabit              Intel Xeon E5-2670 v2   2.5                Yes         -            Yes          -        Yes
hs1.8xlarge    16    117           24 x 2,000    10 Gigabit              Intel Xeon Family       2                  -           -            -            -        -

'''
# get instance types of first instance in first cluster:
con.get_all_instances()[0].instances[0].instance_type


# setup spot request:
iamcon=boto.connect_iam()
role=iamcon.list_instance_profiles_for_role('h2o')['list_instance_profiles_for_role_response']['list_instance_profiles_for_role_result']['instance_profiles'][0]['arn']
role['get_role_response']['get_role_result']['role']['arn']
# ami
ami='ami-027bea6a'
# instance type:
spot_inst_type='c4.large'
# security group:
[i for i in con.get_all_security_groups() if re.search('wizard',str(i))]
 group=[i for i in con.get_all_security_groups() if re.search('wizard',str(i))][-1]  # use group.id below

con.request_spot_instances(
 instance_type=spot_inst_type,
 price=.05,
 image_id=ami,
 instance_profile_arn=role,
 key_name='tcbook',
 count=1,
 security_group_ids=[group.id], # note that this needs to be in a list
 dry_run=True
 )