Archive for the ‘aws’ Category

aws: awscli: s3 and s3api

July 31, 2017

The post discusses using AWS CLI to request S3 service.

environment

bash-3.2$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.11.6
BuildVersion:   15G1510
bash-3.2$ python --version
Python 2.7.10
bash-3.2$ aws --version
aws-cli/1.11.127 Python/2.7.10 Darwin/15.6.0 botocore/1.5.90

aws configure
AWS’s IAM(Identity and Access Management) service manages user, group and permissions. Each user granted AWS CLI permission is assigned a access key id and secret access key. aws configure can setup access key id, secret access key, and etc. In my case, the user is granted AmazonS3FullAccess permission.

bash-3.2$ aws configure
AWS Access Key ID [None]: $(access_key_id)
AWS Secret Access Key [None]: $(secrect_access_key)
Default region name [None]: 
Default output format [None]: 

s3 and s3api

  • aws s3 command provides high level s3 commands
  • aws s3api command provides low level s3 commands

list all buckets
Each data in S3 is in a bucket. The name of each bucket is globally unique for all users and regions.

bash-3.2$ aws s3 ls 
2017-01-18 14:24:52 bucket_name_01
2017-07-31 13:53:14 bucket_name_02
bash-3.2$ aws s3api list-buckets
{
    "Owner": {
        "DisplayName": $(DisplayName), 
        "ID": $(ID)
    }, 
    "Buckets": [
        {
            "CreationDate": "2017-01-18T06:24:52.000Z", 
            "Name": "bucket_name_01"
        }, 
        {
            "CreationDate": "2017-07-31T05:53:14.000Z", 
            "Name": "bucket_name_02"
        }
    ]
}

list all objects in a bucket

bash-3.2$ aws s3api list-objects --bucket bucket_name_01
{
    "Contents": [
        {
            "LastModified": "2017-07-30T20:46:02.000Z", 
            "ETag": "\"af8a4df66bd9ef2433b3f29e07fbbac4\"", 
            "StorageClass": "STANDARD", 
            "Key": "hello.txt", 
            "Owner": {
                "DisplayName": $(DisplayName), 
                "ID": $(ID)
            }, 
            "Size": 12
        }
    ]
}

conclusion
The post discusses using aws s3 and aws s3api to send requests to AWS s3 service.

Advertisements

aws: command line interface

July 31, 2017

The post discusses AWS command line interface.

environment

bash-3.2$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.11.6
BuildVersion:   15G1510
bash-3.2$ python --version
Python 2.7.10
bash-3.2$ pip --version
pip 9.0.1 from /Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg (python 2.7)

api and aws
The AWS service behaviours is controlled by API. Here shows using AWS CLI to control AWS service.

install aws cli
Use pip, python package manager, to install awscli package

bash-3.2$ sudo pip install --upgrade awscli --ignore-installed six
The directory '/Users/chengyihe/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/Users/chengyihe/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting awscli
  Downloading awscli-1.11.127-py2.py3-none-any.whl (1.2MB)
    100% |████████████████████████████████| 1.2MB 746kB/s 
Collecting six
  Downloading six-1.10.0-py2.py3-none-any.whl
Collecting docutils>=0.10 (from awscli)
  Downloading docutils-0.13.1-py2-none-any.whl (537kB)
    100% |████████████████████████████████| 542kB 818kB/s 
Collecting botocore==1.5.90 (from awscli)
  Downloading botocore-1.5.90-py2.py3-none-any.whl (3.6MB)
    100% |████████████████████████████████| 3.6MB 325kB/s 
Collecting s3transfer<0.2.0,>=0.1.9 (from awscli)
  Downloading s3transfer-0.1.10-py2.py3-none-any.whl (54kB)
    100% |████████████████████████████████| 61kB 1.4MB/s 
Collecting colorama<=0.3.7,>=0.2.5 (from awscli)
  Downloading colorama-0.3.7-py2.py3-none-any.whl
Collecting PyYAML<=3.12,>=3.10 (from awscli)
  Downloading PyYAML-3.12.tar.gz (253kB)
    100% |████████████████████████████████| 256kB 1.4MB/s 
Collecting rsa<=3.5.0,>=3.1.2 (from awscli)
  Downloading rsa-3.4.2-py2.py3-none-any.whl (46kB)
    100% |████████████████████████████████| 51kB 593kB/s 
Collecting python-dateutil<3.0.0,>=2.1 (from botocore==1.5.90->awscli)
  Downloading python_dateutil-2.6.1-py2.py3-none-any.whl (194kB)
    100% |████████████████████████████████| 194kB 877kB/s 
Collecting jmespath<1.0.0,>=0.7.1 (from botocore==1.5.90->awscli)
  Downloading jmespath-0.9.3-py2.py3-none-any.whl
Collecting futures<4.0.0,>=2.2.0; python_version == "2.6" or python_version == "2.7" (from s3transfer<0.2.0,>=0.1.9->awscli)
  Downloading futures-3.1.1-py2-none-any.whl
Collecting pyasn1>=0.1.3 (from rsa<=3.5.0,>=3.1.2->awscli)
  Downloading pyasn1-0.3.1-py2.py3-none-any.whl (61kB)
    100% |████████████████████████████████| 71kB 925kB/s 
Installing collected packages: docutils, six, python-dateutil, jmespath, botocore, futures, s3transfer, colorama, PyYAML, pyasn1, rsa, awscli
  Running setup.py install for PyYAML ... done
Successfully installed PyYAML-3.12 awscli-1.11.127 botocore-1.5.90 colorama-0.3.7 docutils-0.13.1 futures-3.1.1 jmespath-0.9.3 pyasn1-0.3.1 python-dateutil-2.6.1 rsa-3.4.2 s3transfer-0.1.10 six-1.10.0
bash-3.2$ aws --version
aws-cli/1.11.127 Python/2.7.10 Darwin/15.6.0 botocore/1.5.90

aws usage
The aws command can send client api all AWS service. For example EC2 and S3.

bash-3.2$ aws
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help
aws: error: too few arguments

show aws s3 document

bash-3.2$ aws s3 help

show aws ec2 document

bash-3.2$ aws ec2 help

conclusion
The post discusses how to install AWS CLI, and shows its usage.

reference
AWS Command Line Interface
Universal Command Line Environment for AWS.

aws: ec2: ebs and instance store

July 30, 2017

The post discusses ebs and instance store.

what is a ec2 instance
A running virtual machine in aws cloud.

instance and root device volume
Instances can be categorised into two groups according to their root device volume.

  • A ebs-backed instance is a running machine whose image is stored at ebs
  • A instance store-backed instance is a running machine whose image is stored at instance store
    • what is ebs
      EBS is an aws service. It stands for elastic block service.

      • Similar to s3, ebs is also a storage service. But it is only for image of ec2 instance.
      • The data in ebs is persistent even its corresponding instance is stopped.
      • The charge of ebs is additional. It is not included within instance’s hourly running cost.
      • When a instance is stopped, its corresponding ebs is still in use and charged.
      • When a instance is terminated, its instance is deleted. If the instance is ebs-backed, the ebs is also deleted at which time the ebs is stopped charged

      what is instance store
      A storage within the instance itself.

      • The physical storage of instance store is attached to the host computer.
      • The instance store could have many instance store volumes. Each instance volume could be represented as linux device, such as /dev/eupheumeral0, /dev/eupheumeral1, and so on.
      • The data in instance store is lost when its corresponding instance is stopped
      • The charge of instance store is included within instance’s hourly running cost.

      conclusion
      If persistent data is required, then choose ebs-backed instance type. But be careful that the ebs is still charged when its corresponding instance is stopped. Only when the instance is terminated does the ebs stop been charged.

      reference
      Amazon EC2 Root Device Volume
      Amazon EC2 Instance Store

aws: ec2: instance type

July 30, 2017

The post discusses ec2 instance and instance type.

what is a ec2 instace
A running ec2 instance is a virtual machine running in the aws cloud service.

instance types
Before a machine is running, we could decide which image to run on which machine. In aws ec2 context, we could choose which ami(Amazon machine image) to run on which instance type. In this analogy, choosing instance type is identical to choosing the machine on which the image is running.

more related to instance types
ec2 supports lots of instance types.

  • Instance types could be categorised into general, compute, memory, storage, and accelerated computing.
  • Each instance type category includes similar instance types with different power. For example, general purpose instance types have t2.nano, t2.micro, t2.small, t2.medium, t2.large and, t2.xlarge. As expected t2.micro is more powerful and expensive than t2.nano, t2.small is more powerful and expensive than t2.micro.
  • t2.micro is the only free tier eligible instance type.

conclusion
A brief note of ec2 instance type.

reference
Instance Types


%d bloggers like this: