Exam Notes – EC2

Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud. Amazon EC2 reduces the time required to obtain and boot new server instances to minutes, allowing you to quickly scale capacity, both up and down, as your computing requirements change.

EC2 has changed the economics of computing by allowing you to pay only for capacity that you actually use. Amazon EC2 provides developers the tools to build failure resilient applications and isolate themselves from common failure scenarios

  • On Demand – allows you to pay a fixed rate by the hour (or by the second) with no commitment (by the hour or second currently depends on the technology chosed – linux is by the second – windos is by the hour – probably will change though)
    • perfect for users that want the low cost and flexibility of Amazon EC2 without any up_front payment or long-term commitment
    • applications with short term, spiky or unpredictable workloads that cannot be interrrupted
    • applications being developed or tested on Amazon AE2 for the first time
  • Reserved – contractual  – provides you with a capacity reservation and offer a significant discount on the hourly charge for an instance. 1 year or 3 year terms
    • application with steady state or predictable usage
    • applications that require reserved capacity
    • users can make up-front payments to reduce their total computing costs even fruther (the longer the contract the greater the discount)
      • Standard RIs (up to 75% off on-demand – pay everything up front for 3 years)
      • Convertable RIs (up to 54% off on-demand) feature the capability to change the attributes of the RI as long as the exchange results in the creation of Reserved Instances of equal or greater value. Basically you can you from a very CPU intensive instance to a memory intensive instance – you are not locked into an instance type
      • Scheduled RIs are available to launch within the time window you reserve. This option allows you to match your capacity reservation to a predictable recurring schedule that only requires a fraction of a day, week or month. So if you only need the capacity on a Friday you can reserve it for then and it will automatically scale back once the timeslot has passed. Cheaper than having the “always ready” on demand model
  • Spot – enables you to bid whatever price you want for instance capacity, providing for even greater savings if your applications have flexible start and end times
    • applications that have flexible start and end times – many companies do data crunching at 3am because it requires a lot of resources but is not realtime
    • applications that are only feasible at very low compute prices
    • users with an urgent need for large amounts of additional computing capacity
    • SPOT instances will be terminated if the instance price goes above the bid amount.
      • If a spot instance is terminated by Amazon EC2, you will not be charged for a partial hour of usage. However if you terminate the instance yourself, you will be charged for the complete hour in which the instance ran
  • Dedicated hosts – physical EC2 server dedicated for your use. Dedicated Hosts can help you reduce costs by allowing you to use your existing server-bound software licenses (VM-Ware, Oracle, etc)
    • useful for regulatory requirements that may not support multi-tenant virtualisation
    • great for licensing which does not support multi-tenancy or cloud deployments
    • can be purchased On-Demand (hourly)
    • can be purchased as a Reservation for up to 70% off the On-Demand price

How to remember all of the above?

  • F for FPGA
  • I for IOPS
  • G for graphics
  • H for High Disk Throughput
  • T for cheap general purpose
  • D for density
  • R for RAM
  • M for main chaoice for general purpose apps
  • C for compute
  • P for graphics (think Pics)
  • X for Extreme memory

FIGHTDRMCPX

Fight Doctor Mac Pix

If EC2 is a virtual server, EBS is a virtual disk.

Amazon EBS allows you to create storage volumes and attach them to Amazon EC2 instances. Once attached you can create a file system on top of these volumes, run a database or use them in any other way you would use a block device. Amazon EBS volumes are placed in a specific Availability Zone , where they are automatically replicated to protect you from the failure of a single component –> EBS volumes do not just exist on one physicl disk, it is spread across multiple inside the AZ.

Think – EBS – a disk in the cloud you attach to your EC2 instances. The place where your OS is installed is called the root volume. Then you can create more volumes on top of that.

EBS Volume Types

  • SSD – General Purpose SSD (GP2)
    • General purpose, balances both price and performance
    • Ratio of 3 IOPS (Stands for “Input/Output Operations Per Second.” … The IOPS value indicates how many different input or output operations a device or group of devicescan perform in one second. It may be used along with other metrics, such as latency and throughput, to measure overall performance) per GB with up to 10000 IOPS and the ability to burst to 3000 IOPS for extended periods of time for volumes at 3334 GiB and above
  • SSD – Provisioned IOPS SSD (IO1)
    • Designed for I/O intensive applications such as large relational or NoSQL databases
    • use if you need ,ore that 10000 IOPS
    • You can provision up to 20000 IOPS per volume
  • MAGNETIC – Throughput Optimized HDD (ST1)
    • Big data
    • Data warehouses
    • Log processing
    • Cannot be a boot volume <– important
      • it has to be an additional volume
  • MAGNETIC – Cold HDD (SC1)
    • Lowest cost storage for infrequently accessed workloads
    • like a file server
    • Cannot be a boot volume <– important
  • MAGNETIC – Magnetic (standard)
    • Lowest cost per gigabyte for all EBS volume types that is bootable. Magnetic volumes are ideal for workloads where data is accessed infrequently and applications where the lowest storage cost is important
    • Not included in the Amazon comparison tables anymore but it is still available

If an instance is running …..

Open the console

Click on EC2Running Instances

Click on a/the running instance and check out the console below it. There are 4 tabs (in my version anyway) …

Termination Protection == TRUE. This will not work

You need to do this …

There are 2 types of status checks. Need to understand these for the exam.

System Status Check – basically says that it’s verifying that your instance is reachable. We test that we are able to get network packets to your instance. If this check fails there may be an issue with the infrastructure hosting your instance such as the power, networking or software systems.

A system status check is actually just checking the underlying hypervisor, making sure the hypervisor is up, making sure the networking that gets to the hypervisor is up.

Instance Status Check – checking that you can get traffic to the operating system. If an instance status check fails you can reboot your instance and that will mean that it will probably come up on another host – it will come back up on another hypervisor.

Basic monitoring is included. Detailed Monitoring costs

  1. Click launch new instance
  2. Choose Amazon Linux AMI
  3. Choose t2-micro (depends on region. 2019 EU (Stockholm) has T3 micro)
  4. Leave all defaults
    • There are settings for VPC, subnet etc. When launching an EC2 instance, it needs to do be launched inside a VPC. A default one is always available.
    • Subnet – tied to an availability zone.
    • Cloudwatch detailed monitoring. By default monitoring is reported every 5 minutes. Click the checkbox if you want monitoring to be reported every 1 minute. This will cost more.
    • User Data – here you can pass bootstrap scripts to your EC2 instance. Example, tell the EC2 instance to download apache, install apache, set the apache service to remain on even if the EC2 instance reboots
  5. Choose EBS volumes (section lower down – all previous steps still relevant)
    • note that the root volume is not encrypted
    • There are tools to encrypt it
    • OR you can (1) provision them, (2) create a copy or AMI of the EC2 instance, (3) while creating the copy, encrypt the root device volume
  6. …..

yum update -y

yum install httpd

chkconfig httpd on

cd /var/www/html

nano index.html

  • Termination protection – ON/OFF — OFF by default
  • On an EBS backed instance, the default setting is that the device volume is deleted if the EC2 instance is terminated
  • EBS root volumes of your default AMIs cannot be encrypted. You can use 3rd party tools such as bitlocker to encrypt the root volume OR you can (1) provision them, (2) create a copy or AMI of the EC2 instance, (3) while creating the copy, encrypt the root device volume
  • Additional volumes can be encrypted

MAC

All you need to do is click on applications and then you scroll down to utilities and then just go to terminal and that will open up your terminal window.

Now what you need to do is go to the directory way you downloaded your private key, maybe the downloads directory. The file will have an extension .pem.

Now you need to change the permissions of your file. CHMOD 400 . This locks your private key down so you can use it to SSH into your EC2 instance.

Go back to the AWS console and get your public IP address we’re going to connect into this EC2 instance.

TYPE ssh ec2-user@ -i and then ENTER

AWS doesnt know this key authenticity so I will ask for confirmation – type YES

Make yourself a superuser – sudo su (username changes from ec2-user to root)

RUN sudo yum update -y  to apply all updates to your server.

Install Apache – yum install httpd -s

Go to cd /var/www/html

Make file in there called index.html and write something into it.

Start Apache – service httpd start

Then test – go to web browser, copy and paste the public IP address you used to SSH into your instance – the contents of index.html should be displayed.

Windows

As in the last part, you will download the key pair file. But windows doesnt understand the file in its raw format, we need to convert it using Putty. Putty doesnt support .pem files, putty needs its own .ppk files.

Download putty.exe and puttygen.exe. Install both.

Run puttygen.exe. –> click load –> change file type to *.* so you can see the .pem file. OK (imports it).

Follow instructions –> “to use this key you need to save it in Puttys own format“. Click Save Private key.

Set file name. Save it as a .ppk instead of a .pem

Open Putty. Go to AWS console and get the IP address of the EC2 instance.

In Putty –> Host name = ec2-user@

Go to SSH in the menu –> go to Auth –> browse for private key. Open

Then save session so you can open it easier afterwards.

Do the same as the previous for the rest.

Security group is a virtual firewall controlling traffic to your EC2 instances. When launching an EC2 instance you associate it with one or more security groups. That means your instance is behind multiple security groups. Each securuty has rules that are then applied to traffic to (or from) your instance. For example, if one of your security groups does not ALLOW access via SSH, it will not be possible to access the EC2 instance via the SSH channel/port.

Practical

Log in to AWS console –> services –> EC2 — you see all created EC2 instances.

To create a new one ….. click button Launch Instance

    1. Choose AMI (Amazon linux AMI is fine) –> Select
    1. Chose Instance type (t2.micro)
    1. Configure instance details (leave defaults)
    1. Add storage (leave defaults)
    1. Add tags (whatever you like)
  1. Security Group
    • Create a security group with HTTP, SSH and HTTPS and add this one
  2. Review and Launch

LAUNCH

    • Selecting existing key pair if applicable + I acknowledge …..
  • If no security group existed previously, you will need to create a new public/private key pairing
      • Public key = padlock (this can be used by many instances)
      • Private key = key to open padlock (single key can be used to open all padlocks in the cloud)
    • Create a new pair – give it a name – download the key pair – launch instances
      • You will need this downloaded file later to connect over CLI to your EC2 instances – keep it safe, no not distribute
  • Starts Launching …..
  • Once launched, open up a terminal window (MAC or windows, see above)
  • elavate priviliges to root sudo su
  • always run an update to get started – yum update -y
  • install apache yum install httpd -y
  • start the service – service httpd start
  • make sure the apache service starts on instance start – chkconfig httpd on

Test this –

Go back to security groups. Look at the tabs, there are two in particular – Inbound and Outbound

Inbound talks about the channels that we can accept traffic. NOTE: if there is an Inbound rule, an outbound rule is implicitly created for the same traffic type. No need to explicitly define it. Security groups are stateful. If itis allowed in, it will be automatically allowed back out via the same channel. By default Outbound says ALL traffic. By default, no trafic is allowed to an instance, essentially you need to specify a whitelist. You cannot deny traffic, you can only allow traffic. NOTE: Network Access Control Lists (VPC part) are stateless so there you need to define rules for both directions.

Any changes to a security group are applied immediatly. Any rule you add, any rule you remove – change is applied immediately.

Every EC2 instance has one security group by default. This is there to allow any instances in the security group to communicate with any other instances in the security group.

Summary – Key points

  • All inbound traffic is blocked by default
  • All outbound traffic is allowed by default
  • Changes to security groups take effect immediately
  • You can have any number of EC2 instances within a security group
  • You can have multiple security groups attached to a particular EC2 instance
  • Security groups are stateful
    • if you create and inbound rule allowing traffic in, that traffic is automatically allowed back out again
  • You cannot block specific IP addresses using security groups, instead you need to use network access control lists
  • You can specify allow rules, but not deny rules.

Create a new EC2 instance, leaving everything as default … except step 4, Add Storage.

Leave the root volume as is. Add the following:

  • EBS 8GB Magnetic
  • EBS 500GB Throughput optimised HDD (ST1)
  • EBS 500GB Cold HDD (SC1)

Next next next -> launch instance.

NOTE: An EBS volume needs to be in the samd AZ as the EC2 instance you want to mount it on.

You can modify volumes:

  • Volume type
  • Size

There will be no downtime but there may be a performance change depending on the destination volume type.

You cannot modify the volume for magnetic storage. This makes sense as magnetic cannot magically change to SSD or visa versa.

So what else can you do:

You could create a snapshot of a volume. From the snapshot you can create a volume. During this creation you can choose the volume type (does not need to be the same as the origin volume). You can choose the size, and also the AZ – this means you can, from this snapshot, create a new volume in a different availability zone and connect it to an EC2 instance in the destination zone.

You can copy the snapshot to another region.

You can create an image from the snapshot. When you do this, you are then able to create an EC2 instance directly from the image. When launching your EC2 instance, you can choose the instance type you want to launch from and you can launch it into another AZ (subnet).

You can also copy an AMI from one region to another.

Summary:

  • Volumes exist on EBS
    • virtual hard disk
  • snapshots exist on S3
  • snapshots are a point in time copy of the volume
  • snapshots are incremental – this means that only the blocks that have changed since your last snapshot are moved to S3
  • the first snapshot takes some time to create
  • to create a snapshot for Amazon EBS volumes that serve as root devices, you should stop the instance before taking the snapshot, however a snapshot can be taken while the machine is running
  • you can create AMIs from EBS-backed instances and snapshots
  • you can change EBS volumes on the fly including the volume storage type and the size
  • volumes will ALWAYS be in the same AZ as the EC2 instance (latency concerns)
  • to move an EC2 volume from one AZ/Resgion to another, take a snap or an image of it and copy it to the new AZ/Region (they might be using CRR to achieve this)
  • snapshots of encrypted volumes are encrypted automatically
  • volumes restored from encrypted snapshots are encrypted automatically
  • you can share snapshots with others but only if they are unencrypted
    • these snapshots can be shared with other AWS accounts or they can be made public

Redundant Array of Independent Disks

  • RAID 0 – striped, No redundancy, good performance
    • used in gaming PCs (example). Basically you will create a single volume across multiple disks. If one disk fails, you lose the entire array
  • RAID 1 – mirrored, redundancy
    • you have one disk and you mirror an exact copy to another disk. One disk fails – you can still use the other disk
  • RAID 5 – Good for reads, bad for writes (AWS recommends against putting RAID 5 on EBS)
  • RAID 10 – striped and mirrored, good redundancy, good performance

Why use RAID? If you are not getting the disk I/O that you want. You have a single volume, provisioned it to its max size but you still need higher disk I/O –> solution is to add multiple volumes, create RAID array to get the disk I/O you want.