Amazon AWS Reviews and Pricing

Independent Analyst and Advisory Consultant at Server StorageIO - www.storageio.com

Feb 22, 2017

Amazon cloud storage options enhanced with Glacier

In case you missed it, Amazon Web Services (AWS) has enhanced their cloud services (Elastic Cloud Compute or EC2) along with storage offerings. These include Relational Database Service (RDS), DynamoDB, Elastic Block Store (EBS), and Simple Storage Service (S3). Enhancements include new functionality along with availability or reliability in the wake of recent events (outages or service disruptions). Earlier this year AWS announced their Cloud Storage Gateway solution that you can read an analysis here. More recently AWS announced provisioned IOPS among other enhancements (see AWS whats new page here).

Before announcing Glacier, options for Amazon storage services relied on general purpose S3 or EBS with other Amazon services. S3 has provided users the ability to select different availability zones (e.g. geographical regions where data is stored) along with level of reliability for different price points for their applications or services being offered.

Note that AWS S3 flexibility lends itself to individuals or organizations using it for various purposes. This ranges from storing backup or file sharing data to being used as a target for other cloud services. S3 pricing options vary depending on which availability zones you select as well as if standard or reduced redundancy. As its name implies, reduced redundancy trades lower availability recovery time objective (RTO) in exchange for lower cost per given amount of space capacity.

AWS has now announced a new class or tier of storage service called Glacier, which as its name implies moves very slow and capable of supporting large amounts of data. In other words, targeting inactive or seldom accessed data where emphasis is on ultra-low cost in exchange for a longer RTO. In exchange for an RTO that AWS is stating that it can be measured in hours, your monthly storage cost can be as low as 1 cent per GByte or about 12 cents per year per GByte plus any extra fees (See here).

Here is a note that I received from the Amazon Web Services (AWS) team:
----------------------
Dear Amazon Web Services Customer,
We are excited to announce the immediate availability of Amazon Glacier – a secure, reliable and extremely low cost storage service designed for data archiving and backup. Amazon Glacier is designed for data that is infrequently accessed, yet still important to keep for future reference. Examples include digital media archives, financial and healthcare records, raw genomic sequence data, long-term database backups, and data that must be retained for regulatory compliance. With Amazon Glacier, customers can reliably and durably store large or small amounts of data for as little as $0.01/GB/month. As with all Amazon Web Services, you pay only for what you use, and there are no up-front expenses or long-term commitments.

Amazon Glacier is:

Low cost- Amazon Glacier is an extremely low-cost, pay-as-you-go storage service that can cost as little as $0.01 per gigabyte per month, irrespective of how much data you store.
Secure – Amazon Glacier supports secure transfer of your data over Secure Sockets Layer (SSL) and automatically stores data encrypted at rest using Advanced Encryption Standard (AES) 256, a secure symmetrix-key encryption standard using 256-bit encryption keys.
Durable- Amazon Glacier is designed to give average annual durability of 99.999999999% for each item stored.
Flexible -Amazon Glacier scales to meet your growing and often unpredictable storage requirements. There is no limit to the amount of data you can store in the service.
Simple- Amazon Glacier allows you to offload the administrative burdens of operating and scaling archival storage to AWS, and makes long term data archiving especially simple. You no longer need to worry about capacity planning, hardware provisioning, data replication, hardware failure detection and repair, or time-consuming hardware migrations.
Designed for use with other Amazon Web Services – You can use AWS Import/Export to accelerate moving large amounts of data into Amazon Glacier using portable storage devices for transport. In the coming months, Amazon Simple Storage Service (Amazon S3) plans to introduce an option that will allow you to seamlessly move data between Amazon S3 and Amazon Glacier using data lifecycle policies.

Amazon Glacier is currently available in the US-East (N. Virginia), US-West (N. California), US-West (Oregon), EU-West (Ireland), and Asia Pacific (Japan) Regions.

A few clicks in the AWS Management Console are all it takes to setup Amazon Glacier. You can learn more by visiting the Amazon Glacier detail page, reading Jeff Barrs blog post, or joining our September 19th webinar.
Sincerely,
The Amazon Web Services Team
----------------------

What is AWS Glacier?

Glacier is low-cost for lower performance (e.g. access time) storage suited to data applications including archiving, inactive or idle data that you are not in a hurry to retrieve. Pay as you go pricing that can be as low as $0.01 USD per GByte per month (and other optional fees may apply, see here) depending on availability zone. Availability zone or regions include US West coast (Oregon or Northern California), US East Coast (Northern Virginia), Europe (Ireland) and Asia (Tokyo).

Now what is understood should have to be discussed, however just to be safe, pity the fool who complains about signing up for AWS Glacier due to its penny per month per GByte cost and it being too slow for their iTunes or videos as you know its going to happen. Likewise, you know that some creative vendor or their surrogate is going to try to show a miss-match of AWS Glacier vs. their faster service that caters to a different usage model; it is just a matter of time.

Lets be clear, Glacier is designed for low-cost, high-capacity, slow access of infrequently accessed data such as an archive or other items. This means that you will be more than disappointed if you try to stream a video, or access a document or photo from Glacier as you would from S3 or EBS or any other cloud service. The reason being is that Glacier is designed with the premise of low-cost, high-capacity, high availability at the cost of slow access time or performance. How slow? AWS states that you may have to wait several hours to reach your data when needed, however that is the tradeoff. If you need faster access, pay more or find a different class and tier of storage service to meet that need, perhaps for those with the real need for speed, AWS SSD capabilities ;).

Here is a link to a good post over at Planforcloud.com comparing Glacier vs. S3, which is like comparing apples and oranges; however, it helps to put things into context.

In terms of functionality, Glacier security includes secure socket layer (SSL), advanced encryption standard (AES) 256 (256-bit encryption keys) data at rest encryption along with AWS identify and access management (IAM) policies.

Persistent storage designed for 99.999999999% durability with data automatically placed in different facilities on multiple devices for redundancy when data is ingested or uploaded. Self-healing is accomplished with automatic background data integrity checks and repair.

Scale and flexibility are bound by the size of your budget or credit card spending limit along with what availability zones and other options you choose. Integration with other AWS services including Import/Export where you can ship large amounts of data to Amazon using different media and mediums. Note that AWS has also made a statement of direction (SOD) that S3 will be enhanced to seamless move data in and out of Glacier using data policies.

Part of stretching budgets for organizations of all size is to avoid treating all data and applications the same (key theme of data protection modernization). This means classifying and addressing how and where different applications and data are placed on various types of servers, storage along with revisiting modernizing data protection.

While the low-cost of Amazon Glacier is an attention getter, I am looking for more than just the lowest cost, which means I am also looking for reliability, security among other things to gain and keep confidence in my cloud storage services providers. As an example, a few years ago I switched from one cloud backup provider to another not based on cost, rather functionality and ability to leverage the service more extensively. In fact, I could switch back to the other provider and save money on the monthly bills; however I would end up paying more in lost time, productivity and other costs.

What do I see as the barrier to AWS Glacier adoption?

Simple, getting vendors and other service providers to enhance their products or services to leverage the new AWS Glacier storage category. This means backup/restore, BC and DR vendors ranging from Amazon (e.g. releasing S3 to Glacier automated policy based migration), Commvault, Dell (via their acquisitions of Appassure and Quest), EMC (Avamar, Networker and other tools), HP, IBM/Tivoli, Jungledisk/Rackspace, NetApp, Symantec and others, not to mention cloud gateway providers will need to add support for this new capabilities, along with those from other providers.

As an Amazon EC2 and S3 customer, it is great to see Amazon continue to expand their cloud compute, storage, networking and application service offerings. I look forward to actually trying out Amazon Glacier for storing encrypted archive or inactive data to compliment what I am doing. Since I am not using the Amazon Cloud Storage Gateway, I am looking into how I can use Rackspace Jungledisk to manage an Amazon Glacier repository similar to how it manages my S3 stores.

Some more related reading:
Only you can prevent cloud data loss
Data protection modernization, more than swapping out media
Amazon Web Services (AWS) and the NetFlix Fix?
AWS (Amazon) storage gateway, first, second and third impressions

As of now, it looks like I will have to wait for either Jungledisk adds native support as they do today for managing my S3 storage pool today, or, the automated policy based movement between S3 and Glacier is transparently enabled.

[To view all of the links mentioned in this post, go to:storageioblog.com/amazon-cloud-storage-options-enhanced-with-glacier/ ]

Some updates:

storageioblog.com/november-2013-server-storageio-update-newsletter/

storageioblog.com/fall-2013-aws-cloud-storage-compute-enhancements/

Disclosure: My company does not have a business relationship with this vendor other than being a customer.

Walter WebbWorks

Report as inappropriate

Nov 10, 2018

I can help you guys with anything that you need to ask me about the consultancy things and all of them only on aws consulting. It is because it is a great paltform and people always help me there.
Website: www.clickittech.com/

it_user194427

Chief Technology Officer at a tech services company with 51-200 employees

Jan 9, 2016

Download

An amazing platform to build on but IAM policies, and cross account access needs improvement.

Pros and Cons

"The whole IaaS model is an invaluable service, and the ease of deployment, maintenance, and scalability, and pay as you go model make AWS an amazing platform to build on."

"Like anything, failures happen every once in a while."

What is most valuable?

The whole IaaS model is an invaluable service. The ease of deployment, maintenance, and scalability, and pay as you go model make AWS an amazing platform to build on.

How has it helped my organization?

AWS sitting at the core of our service, and we have been able to provide an amazing number of features, that were otherwise very expensive, and labor intensive to put in place, these include high availability, business continuity planning, disaster recovery, among others.

What needs improvement?

AWS has an amazing feature set but I have not used all of them to be able to have a well rounded opinion about improvement. However, of the features I have used, I would say IAM policies, and cross account access would probably be one of the main areas of improvement. Amazon is working on a "Service Catalog" which could potentially fill some of these holes.

For how long have I used the solution?

I've used it for three years.

What was my experience with deployment of the solution?

Surprisingly, since starting to use AWS, the process has been quite simple. The deployment was very smooth. Despite this, it does take a bit of getting used to when working with VPCs, and networking in an AWS context, but that's a fairly quick learning curve that can be attained easily.

What do I think about the stability of the solution?

Like anything, failures happen every once in a while. I have experienced some failed hardware under my instances, which caused a brief outage. The stability of the service, however, is also much more reliant on the architecture of the application than the stability of the AWS infrastructure. In any case, AWS has been quite stable over all.

What do I think about the scalability of the solution?

Scalability is one of AWS' strengths. Scaling resources, be it an AWS EC2 instance, or an RDS instance is a snap. Also, scaling into multiple geographic regions in the world is also possible, and quite a realistic view in that environment.

How are customer service and technical support?

Customer Service:

My experience with AWS customer service has been stellar. Everyone I come into contact with from Sales, to Technical Support are always friendly, and courteous.

Technical Support:

The technical support team is quite knowledgeable, and there is no question asked that doesn't get addressed with full attention, complete with references, examples, and a recap of conversations that were conducted.

Their technical support processes are clearly well thought out. I can always know what communication to expect, and the level of help that I can expect to receive. I have yet to call them on an issue where a resolution wasn't reached on the first or second contact.

Which solution did I use previously and why did I switch?

Previously, I used co-location services. The reason I switched is quite obvious:

Cost
Constant overheads
Constant challenge of meeting budgets with consistent cutting edge technology

AWS has removed all these variables, and allowed me to concentrate on growing my services without having to worry about aging servers, or under capacity hardware, etc.

How was the initial setup?

Understanding AWS is actually quite easy. There are some notions that require a bit of previous knowledge to grasp. The good news is that the documentation available about the different services is quite extensive, which can give anyone a head start in launching their AWS services. The complexity of using AWS is directly related to the robustness of the application/service that is being deployed. The more AWS services are integrated together, the more complex the deployment will become.

What about the implementation team?

All AWS services were deployed in-house, with assistance from AWS support teams.

What was our ROI?

Because there is no initial investment in AWS services (it's a pay as you go service in its basic form) the ROI is immediate. Because AWS costs are consistently being reduced, it is a great way to build services, offered at affordable prices, while still getting good returns on investment.

What's my experience with pricing, setup cost, and licensing?

As mentioned above, AWS does not really have initial setup costs. It's like a utility company; you use the service, and pay for your usage. The daily cost is dependent upon the service being deployed at that point in time. For the flexibility, and consistent cutting edge technology that AWS operates on, it's well worth the price.

Which other solutions did I evaluate?

I have evaluated Azure, and Google as IaaS. Quite honestly, Google was too convoluted for my purposes, and although Azure had some nice "Microsoft-y" features that AWS doesn't necessarily have, I still felt that it was much easier to get started with AWS, than it is with the other services.

What other advice do I have?

Don't be afraid of "The Cloud". As prominent as it is today, a lot of people, and small businesses, are still afraid of storing their data away from their physical office. There are a ton of advantages in using AWS for your infrastructure instead of on-premises equipment. Give it a serious look before dismissing it. There is a lot that can be added here, but that could be an article all on its own.

Disclosure: My company does not have a business relationship with this vendor other than being a customer.

it_user194427Chief Technology Officer at a tech services company with 51-200 employees

Report as inappropriate

Nov 1, 2016

Not necessarily anything that other products would lack, it would actually be more of a "nice to have". It's definitely not a deal breaker by any means. I take the idea from the concept that AWS has with AMIs, for instance, or places where there are public repositories of UDF for scripts... same type of thing, for IAMs. there are a lot of out-of-the-box IAM policies that user can benefit from, and rather than re-inventing the wheel, it would be nice if they're compiled in a central place. That said, there's nothing that a Google search can fix :)

See all 6 comments

it_user6186

Independent Analyst and Advisory Consultant at Server StorageIO - www.storageio.com

Sep 27, 2015

EFS is NFS version 4 based however it does not support Windows SMB/CIFS, HDFS or other NAS access protocols.

Pros and Cons

"Overall I like this announcement and look forward to moving beyond the preview."

Cloud Conversations: AWS EFS Elastic File System (Cloud NAS) First Preview Look

Amazon Web Services (AWS) recently announced new Elastic File System (EFS) providing Network File System (NFS) NAS (Network Attached Storage) capabilities for AWS Elastic Cloud Compute (EC2) instances. EFS AWS compliments other AWS storage offerings including Simple Storage Service (S3) along with Elastic Block Storage (EBS), Glacier and Relational Data Services (RDS) among others.

Ok, that’s a lot of buzzwords and acronyms so lets break this down a bit.

AWS EFS and Cloud Storage, Beyond Buzzword Bingo

EC2 – Instances exist in various Availability Zones (AZ’s) in different AWS Regions. Compute instance with various operating systems including Windows and Ubuntu among others that also can be pre-configured with applications such as SQL Server or web services among others. EC2 instances vary from low-cost to high-performance compute, memory, GPU, storage or general purposed optimized. For example, some EC2 instances rely solely on EBS, S3, RDS or other AWS storage offerings while others include on-board Solid State Disk (SSD) like DAS SSD found on traditional servers. EC2 instances on EBS volumes can be snapshot to S3 storage which in turn can be replicated to another region.
EBS – Scalable block accessible storage for EC2 instances that can be configured for performance or bulk storage, as well as for persistent images for EC2 instances (if you choose to configure your instance to be persistent)
EFS – New file (aka NAS) accessible storage service accessible from EC2 instances in various AZ’s in a given AWS region
Glacier – Cloud based near-line (or by some comparisons off-line) cold-storage archives.
RDS – Relational Database Services for SQL and other data repositories
S3 – Provides durable, scalable low-cost bulk (aka object) storage accessible from inside AWS as well as via externally. S3 can be used by EC2 instances for bulk durable storage as well as being used as a target for EBS snapshots.
Learn more about EC2, EBS, S3, Glacier, Regions, AZ’s and other AWS topics in this primer here

What is EFS

Implements NFS V4 (SNIA NFS V4 primer) providing network attached storage (NAS) meaning data sharing. AWS is indicating initial pricing for EFS at $0.30 per GByte per month. EFS is designed for storage and data sharing from multiple EC2 instances in different AZ’s in the same AWS region with scalability into the PBs.

What EFS is not

Currently it seems that EFS has an end-point inside AWS accessible via an EC2 instance like EBS. This appears to be like EBS where the storage service is accessible only to AWS EC2 instances unlike S3 which can be accessible from the out-side world as well as via EC2 instances.

Note however, that depending on how you configure your EC2 instance with different software, as well as configure a Virtual Private Cloud (VPC) and other settings, it is possible to have an application, software tool or operating system running on EC2 accessible from the outside world. For example, NAS software such as those from SoftNAS and NetApp among many others can be installed on an EC2 instance and with proper configuration, as well as being accessible to other EC2 instances, they can also be accessed from outside of AWS (with proper settings and security).

AWS EFS at this time is NFS version 4 based however does not support Windows SMB/CIFS, HDFS or other NAS access protocols. In addition AWS EFS is accessible from multiple AZ’s within a region. To share NAS data across regions some other software would be required.

EFS is not yet as of this writing released and AWS is currently accepting requests to join the EFS preview here.

Where to learn more

Here are some links to learn more about AWS S3 and related topics

What this all means and wrap-up

AWS continues to extend its cloud platform include both compute and storage offerings. EFS compliments EBS along with S3, Glacier and RDS. For many environments NFS support will be welcome while for others CIFS/SMB would be appreciated and others are starting to find that value in HDFS accessible NAS. In addition, AWS has also added a new tier for inactive data in S3 for nearline storage as opposed to having to use Glacier.

Overall I like this announcement and look forward to moving beyond the preview.

Disclosure: My company does not have a business relationship with this vendor other than being a customer.

it_user317559

Linux administrator with 10,001+ employees

Sep 25, 2015

Download

we have 750 hours of Amazon EC2 Linux t2.micro instance usage, but it's expensive.

Pros and Cons

"750 hours of Amazon EC2 Linux t2.micro instance usage (1 GiB of memory and 32-bit and 64-bit platform support) -- It's enough hours to run continuously each month."

"Charges are high at the moment."

Valuable Features:

750 hours of Amazon EC2 Linux t2.micro instance usage (1 GiB of memory and 32-bit and 64-bit platform support) -- It's enough hours to run continuously each month.

Improvements to My Organization:

It gives us the time to check.

Room for Improvement:

Charges are high at the moment.

Disclosure: My company does not have a business relationship with this vendor other than being a customer.

it_user57903Principal at a tech company with 51-200 employees

Report as inappropriate

Apr 16, 2017

Aws has trusted advisor. Also look at park my cloud. Also if you use spot instances it will help you reduce costs. TCO - total cost of ownership

See all 5 comments

it_user7707

Owner at a tech consulting company with 51-200 employees

Aug 23, 2015

Amazon Web Services: Security Processes in the EC2 Cloud

Customer trust and confidence is at the heart of Amazon’s business and with so many customers using Amazon’s platforms to run their business securely and efficiently, Amazon has gone to great lengths to operate and manage a comprehensive control environment. The environment supports secure Amazon Web Services cloud web offerings by ensuring that all necessary policies and processes are used in compliance with AWS certifications.

Within the last few years Amazon Web Services security has achieved notable certifications which include SAS70 Type II audits, PCI DSS Level 1 which involves meeting Payment Card Industry Data Security Standards, ISO 27001 for Information Security Management Systems, and compliance within the Federal Information Security Management Act (FISMA) to properly serve government agency FedRAMP requirements for AWS GovCloud on the Amazon platform.

When Amazon introduced Amazon EC2 it started a process rolling for business customers to run their applications in Amazon’s computing environment. EC2 is the Elastic Compute Cloud which allows business customers to access Amazon’s secure cloud environment through a virtual machine. The platform deploys EC2 security which also supports Amazon Web Services for FedRAMP compliance.

Using Amazon EC2 business customers can create an image of their operating system and applications which is known as an Amazon Machine Image. Once the image is created it is uploaded to Amazon S3 which is Amazon’s Simple Storage Service. The AMI is then registered in Amazon EC2 allowing the customer to summon virtual machines as they are needed. The result is an AWS Virtual Private Cloud for business customers to conduct operations without the exorbitant expense of IT infrastructure. For this reason, Amazon must ensure the environment meets all compliance and security standards hence the acquisition of the certification described earlier.

Amazon EC2 Security Processes

Amazon’s approach to AWS security involves layered security processes which maintain data integrity and provide secure EC2 instances while still maintaining configuration flexibility to meet the individual requirements of EC2 business customers.

Administration Hosts: For business customers who require access to the management platform, Amazon uses a level of security to accommodate administration hosts without posing a risk to data integrity and other users. Through the use of AWS Identity and Access Management, this is accomplished by auditing all access activity and using a log to track the activity. If the user accessing the management platform terminates their authentication privileges then the privileges are automatically discontinued which ensures secure AWS applications.
Customer Controlled Instances: Amazon EC2 allows for virtual instances which are solely controlled by the customer. Business customers exercise full control and at no time can Amazon intervene by logging in to the customer’s operating system. For this reason, a set of practices is in place to guide the customer on authentication processes for AWS VPC in order to access the virtual instances. This involves designing an authentication and privilege system which can be enabled and disabled according to changing needs of virtual machine users.
Firewall: As part of the AWS Security Center, EC2 Business customers have access to a complex firewall solution which can be configured to meet the individual needs of each business customer. For example, the firewall for Amazon EC2 is typically configured by default to block all traffic. If the customer wants to allow inbound traffic they must open the necessary ports to allow inbound traffic while blocking unwanted traffic. The firewall also provides a host of options for setting specific protocols for inbound traffic such as by IP address and other identifications. Added security is in place since the business customer must use their x.509 certificate to change firewall configurations.
Xen: Another layer of AWS security for EC2 is the Xen Hypervisor which separates different instances running on the same virtual machine. The firewall is situated in the Xen Hypervisor which means packets for instances must pass through the firewall thereby adding enhanced security to isolated instances.

Finally, Amazon Web Services Cloud uses a layer of security known as Amazon EBS or Elastic Block Storage which restricts access to data snapshots to the specific Amazon Web Services account which created it. Business customers can make the data snapshots available to other AWS accounts however; this process should be carefully considered since there may be files with sensitive information.

Prior to releasing Elastic Block Storage to the customer, Amazon wipes old data in accordance with the National Industrial Security Program guidelines. Plus EBS allows business customers to encrypt their data on the block device using algorithms that comply with individual security standards.

Disclosure: My company does not have a business relationship with this vendor other than being a customer.

it_user7707Owner at a tech consulting company with 51-200 employees

Report as inappropriate

Aug 23, 2015

Hi Henry,

we'll post something on S3 security as well soon. aws.amazon.com/s3/faqs/

See all 2 comments

it_user234747

Practice Manager - Cloud, Automation & DevOps at a tech services company with 501-1,000 employees

Jun 30, 2015

It has a massive library of services for you to use in developing cloud-based solutions.

Pros and Cons

"Here is the cool thing, as an Enterprise/Cloud Architect you have a MASSIVE library of services (40+ at time of writing) that you can use to develop Cloud-based solutions for your customers."

Originally posted at https://vcdx133.com/2015/06/12/tech101-amazon-web-services

As part of my NPX preparation (AWS Certified Solutions Architect – Professional is one of the recommended qualifications) and my RapidMatter GitHub project (will run from AWS), I have been delving into the world of Amazon Web Services. One statement: “Wow!” I can see why they are the world leader in Public Cloud services.

Here is the cool thing, as an Enterprise/Cloud Architect you have a MASSIVE library of services (40+ at time of writing) that you can use to develop Cloud-based solutions for your customers. As you read through the list below, you will see the fundamental building blocks for every solution. By having this service matrix, you do not have to reinvent the wheel; it already exists and is ready to go. Thus, you can focus on making sure your customer requirements are being met with elegant and innovative designs.

Getting Started (takes 5 minutes)

You have a PC with a responsive and usable Internet connection
Create an AWS account
Provide a valid Credit Card
Provide a valid phone number that must be verified
Start using AWS immediately – there is a free tier (1 year trial period) for some services in some regions (micro instances)
The UI is very intuitive and easy to use
WARNING: You can spin-up most of the service catalogue. Do not forget and leave them running, your credit card will be charged

Core Services of AWS

EC2 – Elastic Cloud Compute – Virtual Machines you can provision (Instances) from a massive library of templates (AMI – Amazon Machine Images free and paid from the AWS Community/Marketplace) no
EBS – Elastic Block Store – Persistent Virtual Disks for your VMs (Instances)
S3 – Simple Storage Service – Scalable, Object-based Storage in the Cloud
Glacier – Archive Storage in the Cloud

Under The Hood: AWS uses a heavily customised version of Xen as its hypervisor.

Pricing Models

On-Demand – Pay-as-you-go
Reserved Instances – Pay up front
Spot Requests – Bid for excess AWS resources against other AWS users

Compute

EC2 Container Service – Run and Manage Docker Containers
Lambda – Run Code in Response to Events

Storage & Content Delivery

Storage Gateway – Integrates On-Premises IT Environments with Cloud Storage
Elastic File System – Fully Managed File System for EC2

Edge Services (to be close to all of your customers around the world)

Route53 – Scalable DNS and Domain Name Registration
CloudFront – Global Content Delivery Network – Caches static content regionally

Simple Micro-services that just work

SQS – Simple Message Queue Service
SES – Simple Email Service
SWF – Simple Workflow Service
AppStream – Low Latency Application Streaming
Elastic Transcoder – Easy-to-use Scalable Media Transcoding
CloudSearch – Managed Search Service

Databases

RDS – Relational Database Service – MySQL, Oracle, SQL Server & Amazon Aurora
DynamoDB – Predictable and Scalable NoSQL Data Store
ElastiCache – In-Memory Cache
Redshift – Managed Petabyte-Scale Data Warehouse Service

Networking

VPC – Virtual Private Cloud – Isolated Cloud Resources
Direct Connect – Dedicated Network Connection to AWS

Administration & Security

Directory Service – Managed Directory Services in the cloud
Identity & Access Management – Access Control and Key Management
Trusted Advisor – AWS Cloud Optimisation Expert
CloudTrail – User Activity and Change Tracking
Config – Resource Configurations and Inventory
CloudWatch – Resource and Application Monitoring

Deployment & Management

CloudFormation – Templated AWS Resource Creation (for Sysadmins)
Elastic Beanstalk – AWS Application Container (for Developers)
OpsWorks – DevOps Application Management Service
CodeDeploy – Automated Deployments

Analytics

EMR – Managed Hadoop Framework
Kinesis – Real-time Processing of Streaming Big Data
Data Pipeline – Orchestration for Data-Driven Workflows
Machine Learning – Build Smart Applications Quickly and Easily

Mobile Services

Cognito – User Identity and App Data Synchronisation
Mobile Analytics – Understand App Usage Data at Scale
SNS – Simple Notification Service – Push Notification Service

Enterprise Applications

WorkSpaces – Desktops in the Cloud (VDI)
WorkDocs – Secure Enterprise Storage and Sharing
WorkMail – Secure Email and Calendaring Service

Disclosure: My company does not have a business relationship with this vendor other than being a customer.

it_user261489

Program and Project Manager at a tech services company with 10,001+ employees

Jun 30, 2015

As with all public clouds, there is still a dilemma with security, but provides a rich set of services for IaaS, PaaS, and SaaS.

Pros and Cons

"AWS is quite good for cloud services such as IaaS, PaaS, and SaaS, providing a rich set of services and integrated monitoring tools alongside a competitive pricing model."

"AWS is a good platform for non-MySQL, MySQL, and Hadoop databases, but it’s not as good for RDBMS ones like MS SQL."

AWS is a good platform for non-MySQL, MySQL, and Hadoop databases, but it’s not as good for RDBMS ones like MS SQL. It still has many missing features -- like replication, backup policy, and the ability to store/attach databases from a local drive -- but it has many good features for big data.

Its new AppStream service will pre-process graphics, including 3D renderings, and blast the results to mobile clients. Its Kinesis service for streaming data sets the stage for building big data apps on AWS, the basic architecture for the internet of things. As for its Hadoop capabilities, AWS launched its Elastic MapReduce (EMR) a long time back. It is the best cloud services provider for open source software for databases, operating systems hosting apps, and many other customized applications.

As discussed with many tech professionals, there is still a dilemma with security like there was a decade ago when online e-commerce business started and people weren’t prepared to share their credit card and bank details. Now, as online shopping is common, I am expecting the same trend will grow for public cloud very soon. AWS security is very good and they are following all the required security regulations as much other public cloud providers are doing, as they know any security breach could impact their entire business.

Pricing is another key concern when private cloud is used for big business and multiple growth on data. The price is a big debate and requires lot of analysis, as it is a question for big organizations. But no doubt, AWS is quite good for a small setup as it’s very cost effective and provides an eco-setup.

AWS is quite for good cloud services such as IaaS, PaaS, and SaaS. It provides a rich set of services and integrated monitoring tools alongside a competitive pricing model. AWS offers a full range of computer and storage offerings, including on-demand instances and specialized services such as Amazon EMR, and Cluster GPU instances. Amazon Cloud Trail and Amazon CloudWatch services are very good monitoring and AWS Identity and Access Management (IAM) is a good administration and security feature for administrators to use.

Disclosure: My company does not have a business relationship with this vendor other than being a customer.

it_user544134IT Coordinator at a non-tech company with 10,001+ employees

Report as inappropriate

Nov 1, 2016

Thank you for the information on the AWS solution.

it_user6186

Independent Analyst and Advisory Consultant at Server StorageIO - www.storageio.com

May 1, 2015

Lambda and other AWS enhancements

A few weeks ago I attended Amazon Web Service (AWS) re:Invent 2014 in Las Vegas for a few days. For those of you who have not yet attended this event, I recommend adding it to your agenda. If you have interest in compute servers, networking, storage, development tools or management of cloud (public, private, hybrid), virtualization and related topic themes, you should check out AWS re:invent.

AWS made several announcements at re:invent including many around development tools, compute and data storage services. One of those to keep an eye on is cloud based Aurora relational database service that complement existing RDS tools. Aurora is positioned as an alternative to traditional SQL based transactional databases commonly found in enterprise environments (e.g. SQL Server among others).

Some recent AWS announcements prior to re:Invent include

AWS Adds EU (Frankfurt) Region
Amazon Linux AMI Updates
AWS Systems Manager for Microsoft System Center Virtual Machine Manager
T2, the New Low-Cost, General Purpose Instance Type for Amazon EC2
Windows Server 2012 R2 AMI Updates
Zocalo Enterprise File Sync & Share updates (read more Zocalo here )
AWS Management Portal for vCenter Setup Enhancements

AWS vCenter Portal

Using the AWS Management Portal for vCenter adds a plug-in within your VMware vCenter to manage your AWS infrastructure. The vCenter for AWS plug-in includes support for AWS EC2 and Virtual Machine (VM) import to migrate your VMware VMs to AWS EC2, create VPC (Virtual Private Clouds) along with subnet’s. There is no cost for the plug-in, you simply pay for the underlying AWS resources consumed (e.g. EC2, EBS, S3). Learn more about AWS Management Portal for vCenter here, and download the OVA plug-in for vCenter here.

AWS re:invent content

November 12, 2014 (Day 1) Keynote (highlight video, full keynote). This is the session where AWS SVP Andy Jassy made several announcements including Aurora relational database that complements existing RDS (Relational Data Services). In addition to Andy, the key-note sessions also included various special guests ranging from AWS customers, partners and internal people in support of the various initiatives and announcements.

November 13, 2014 (Day 2) Keynote (highlight video, full keynote). In this session, Amazon.com CTO Werner Vogels appears making announcements about the new Container and Lambda services.

AWS re:Invent announcements

Announcements and enhancements made by AWS during re:Invent include:

Key Management Service (KMS)
Amazon RDS for Aurora
Amazon EC2 Container Service
AWS Lambda
Amazon EBS Enhancements
Application development, deployed and life-cycle management tools
AWS Service Catalog
AWS CodeDeploy
AWS CodeCommit
AWS CodePipeline

Key Management Service (KMS)

Hardware security module (HSM) based key managed service for creating and control of encryption keys to protect security of digital assets and their keys. Integration with AWS EBS and others services including S3 and Redshift along with CloudTrail logs for regulatory, compliance and management. Learn more about AWS KMS here

AWS Database

For those who are not familiar, AWS has a suite of database related services including SQL and no SQL based, simple to transactional to Petabyte (PB) scale data warehouses for big data and analytics. AWS offers the Relational Database Service (RDS) which is a suite of different database types, instances and services. RDS instance and types include SimpleDB, MySQL, Postgress, Oracle, SQL Server and the new AWS Aurora offering (read more below). Other little data database and big data repository related offerings include DynamoDB (a non-SQL database), ElasticCache (in memory cache repository) and Redshift (large-scale data warehouse and big data repository).

In addition to database services offered by AWS, you can also combine various AWS resources including EC2 compute, EBS and other storage offerings to create your own solution. For example there are various Amazon Machine Images (AMI’s) or pre-built operating systems and database tools available with EC2 as well as via the AWS Marketplace , such as MongoDB and Couchbase among others. For those not familiar with MongoDB, Couchbase, Cassandra, Riak along with other non SQL or alternative databases and key value repositories, check out Seven Databases in Seven Weeks in my book review of it here.

Amazon RDS for Aurora

Aurora is a new relational database offering part of the AWS RDS suite of services. Positioned as an alternative to commercial high-end database, Aurora is a cost-effective database engine compatible with MySQL. AWS is claiming 5x better performance than standard MySQL with Aurora while being resilient and durable. Learn more about Aurora which will be available in early 2015 and its current preview here.

Amazon EC2 C4 instances

AWS will be adding a new C4 instance as a next generation of EC2 compute instance based on Intel Xeon E5-2666 v3 (Haswell) processors. The Intel Xeon E5-2666 v3 processors run at a clock speed of 2.9 GHz providing the highest level of EC2 performance. AWS is targeting traditional High Performance Computing (HPC) along with other compute intensive workloads including analytics, gaming, and transcoding among others. Learn more AWS EC2 instances here, and view this Server and StorageIO EC2, EBS and associated AWS primer here.

Amazon EC2 Container Service

Containers such as those via Docker have become popular to support developers rapidly build as well as deploy scalable applications. AWS has added a new feature called EC2 Container Service that supports Docker using simple API’s. In addition to supporting Docker, EC2 Container Service is a high performance scalable container management service for distributed applications deployed on a cluster of EC2 instances. Similar to other EC2 services, EC2 Container Service leverages security groups, EBS volumes and Identity Access Management (IAM) roles along with scheduling placement of containers to meet your needs. Note that AWS is not alone in adding container and docker support with Microsoft Azure also having recently made some announcements, learn more about Azure and Docker here. Learn more about EC2 container service here and more about Docker here.

AWS Lambda

In addition to announcing new higher performance Elastic Cloud Compute (EC2) compute instances along with container service, another new service is AWS Lambda. Lambda is a service that automatically and quickly runs your applications code in response to events, activities, or other triggers. In addition to running your code, Lambda service is billed in 100 millisecond increments along with corresponding memory use vs. standard EC2 per hour billing. What this means is that instead of paying for an hour of time for your code to run, you can choose to use the Lambda service with more fine-grained consumption billing.

Lambda service can be used to have your code functions staged ready to execute. AWS Lambda can run your code in response to S3 bucket content (e.g. objects) changes, messages arriving via Kinesis streams or table updates in databases. Some examples include responding to event such as a web-site click, response to data upload (photo, image, audio, file or other object), index, stream or analyze data, receive output from a connected device (think Internet of Things IoT or Internet of Device IoD), trigger from an in-app event among others. The basic idea with Lambda is to be able to pay for only the amount of time needed to do a particular function without having to have an AWS EC2 instance dedicated to your application. Initially Lambda supports Node.js (JavaScript) based code that runs in its own isolated environment.

Various application code deployment models

Lambda service is a pay for what you consume, charges are based on the number of requests for your code function (e.g. application), amount of memory and execution time. There is a free tier for Lambda that includes 1 million requests and 400,000 GByte seconds of time per month. A GByte second is the amount of memory (e.g. DRAM vs. storage) consumed during a second. An example is your application is run 100,000 times and runs for 1 second consuming 128MB of memory = 128,000,000MB = 128,000GB seconds. View various pricing models here on the AWS Lambda site that show examples for different memory sizes, times a function runs and run time.

How much memory you select for your application code determines how it can run in the AWS free tier, which is available to both existing and new customers. Lambda fees are based on the total across all of your functions starting with the code when it runs. Note that you could have from one to thousands or more different functions running in Lambda service. As of this time, AWS is showing Lambda pricing as free for the first 1 million requests, and beyond that, $0.20 per 1 million request ($0.0000002 per request) per duration. Duration is from when you code runs until it ends or otherwise terminates rounded up to the nearest 100ms. The Lambda price also depends on the amount of memory you allocated for your code. Once past the 400,000 GByte second per month free tier the fee is $0.00001667 for every GB second used.

Why use AWS Lambda vs. an EC2 instance

Why would you use AWS Lambda vs. provisioning an Container, EC2 instance or running your application code function on a traditional or virtual machine?

If you need control and can leverage an entire physical server with its operating system (O.S.), application and support tools for your piece of code (e.g. JavaScript), that could be an option. If you simply need to have an isolated image instance (O.S., applications and tools) for your code on a shared virtual on-premise environment then that can be an option. Likewise if you have the need to move your application to an isolated cloud machine (CM) that hosts an O.S. along with your application paying for those resources such as on an hourly basis, that could be your option. Simply need a lighter-weight container to drop your application into that’s where Docker and containers comes into play to off-load some of the traditional application dependencies overhead.

However, if all you want to do is to add some code logic to support processing activity for example when an object, file or image is uploaded to AWS S3 without having to standup an EC2 instance along with associated server, O.S. and complete application activity, that’s where AWS Lambda comes into play. Simply create your code (initially JavaScript) and specify how much memory it needs, define what events or activities will trigger or invoke the event, and you have a solution.

View AWS Lambda pricing along with free tier information here.

Amazon EBS Enhancements

AWS is increasing the performance and size of General Purpose SSD and Provisioned IOP’s SSD volumes. This means that you can create volumes up to 16TB and 10,000 IOP’s for AWS EBS general-purpose SSD volumes. For EBS Provisioned IOP’s SSD volumes you can create up to 16TB for 20,000 IOP’s. General-purpose SSD volumes deliver a maximum throughput (bandwidth) of 160 MBps and Provisioned IOP SSD volumes have been specified by AWS at 320MBps when attached to EBS optimized instances. Learn more about EBS capabilities here. Verify your IO size and verify AWS sizing information to avoid surprises as all IO sizes are not considered to be the same. Learn more about Provisioned IOP’s, optimized instances, EBS and EC2 fundamentals in this StorageIO AWS primer here.

Application development, deployed and life-cycle management tools

In addition to compute and storage resource enhancements, AWS has also announced several tools to support application development, configuration along with deployment (life-cycle management). These include tools that AWS uses themselves as part of building and maintaining the AWS platform services.

AWS Config (Preview e.g. early access prior to full release)

Management, reporting and monitoring capabilities including Data center infrastructure management (DCIM) for monitoring your AWS resources, configuration (including history), governance, change management and notifications. AWS Config enables similar capabilities to support DCIM, Change Management Database (CMDB), trouble shooting and diagnostics, auditing, resource and configuration analysis among other activities. Learn more about AWS Config here.

AWS Service Catalog

AWS announced a new service catalog that will be available in early 2015. This new service capability will enable administrators to create and manage catalogs of approved resources for users to use via their personalized portal. Learn more about AWS service catalog here.

AWS CodeDeploy

To support code rapid deployment automation for EC2 instances, AWS has released CodeDeploy. CodeDeploy masks complexity associated with deployment when adding new features to your applications while reducing human error-prone operations. As part of the announcement, AWS mentioned that they are using CodeDeploy as part of their own applications development, maintenance, and change-management and deployment operations. While suited for at scale deployments across many instances, CodeDeploy works with as small as a single EC2 instance. Learn more about AWS CodeDeploy here.

AWS CodeCommit

For application code management, AWS will be making available in early 2015 a new service called CodeCommit. CodeCommit is a highly scalable secure source control service that host private Git repositories. Supporting standard functionalities of Git, including collaboration, you can store things from source code to binaries while working with your existing tools. Learn more about AWS CodeCommit here.

AWS CodePipeline

To support application delivery and release automation along with associated management tools, AWS is making available CodePipeline. CodePipeline is a tool (service) that supports build, checking workflow’s, code staging, testing and release to production including support for 3rd party tool integration. CodePipeline will be available in early 2015, learn more here.

What this all means

AWS continues to invest as well as re-invest into its environment both adding new feature functionality, as well as expanding the extensibility of those features. This means that AWS like other vendors or service providers adds new check-box features, however they also like some increase the depth extensibility of those capabilities.

Besides adding new features and increasing the extensibility of existing capabilities, AWS is addressing both the data and information infrastructure including compute (server), storage and database, networking along with associated management tools while also adding extra developer tools. Developer tools include life-cycle management supporting code creation, testing, tracking, testing, change management among other management activities.

Another observation is that while AWS continues to promote the public cloud such as those services they offer as the present and future, they are also talking hybrid cloud. Granted you have to listen carefully as you may not simply hear hybrid cloud used like some toss it around, however listen for and look into AWS Virtual Private Cloud (VPC), along with what you can do using various technologies via the AWS marketplace.

AWS is also speaking the language of enterprise and traditional IT from an applications and development to data and information infrastructure perspective while also walking the cloud talk. What this means is that AWS realizes that they need to help existing environments evolve and make the transition to the cloud which means speaking their language vs. converting them to cloud conversations to then be able to migrate them to the cloud. These steps should make AWS practical for many enterprise environments looking to make the transition to public and hybrid cloud at their pace, some faster than others. More on these and some related themes in future posts.

The AWS re:Invent event continues to grow year over year, I heard a figure of over 12,000 people however it was not clear if that included exhibiting vendors, AWS people, attendees, analyst, bloggers and media among others. However a simple validation is that the keynotes were in the larger rooms used by events such as EMCworld and VMworld when they hosted in Las Vegas as was the expo space vs. what I saw last year while at re:Invent. Unlike some large events such as VMworld where at best there is a waiting queue or line to get into sessions or hands on lab (HOL), while becoming more crowded, AWS re:Invent is still easy to get in and spend some time using the HOL which is of course powered by AWS meaning you can resume what you started while at re:Invent later. Overall a good event and nice series of enhancements by AWS, looking forward to next years AWS re:Invent.

Disclosure: My company does not have a business relationship with this vendor other than being a customer.

it_user6186

Independent Analyst and Advisory Consultant at Server StorageIO - www.storageio.com

Apr 27, 2015

I like the ability for moving S3 objects within AWS, however I will continue to use other tools such as S3motion and s3sfs for moving data in and out of AWS.

Pros and Cons

"For those who are looking for a way to streamline replicating data (e.g. objects) from an AWS bucket in one region with a bucket in a different region you now have a new option."

"However a catch with the above mentioned tools (among others) and approaches is that to replicate your data (e.g. objects in a bucket) can involve other AWS S3 fees."

Cloud Conversations: AWS S3 Cross Region Replication storage enhancements

Amazon Web Services (AWS) recently among other enhancements announced new Simple Storage Service (S3) cross-region replication of objects from a bucket (e.g. container) in one region to a bucket in another region. AWS also recently enhanced Elastic Block Storage (EBS) increasing maximum performance and size of Provisioned IOPS (SSD) and General Purpose (SSD) volumes. EBS enhancements included ability to store up to 16 TBytes of data in a single volume and do 20,000 input/output operations per second (IOPS). Read more about EBS and other recent AWS server, storage I/O and application enhancements here.

The Problem, Issue, Challenge, Opportunity and Need

The challenge is being able to move data (e.g. objects) stored in AWS buckets in one region to another in a safe, secure, timely, automated, cost-effective way.

Even though AWS has a global name-space, buckets and their objects (e.g. files, data, videos, images, bit and byte streams) are stored in a specific region designated by the customer or user (AWS S3, EBS, EC2, Glacier, Regions and Availability Zone primer can be found here).

Understanding the challenge and designing a strategy

The following diagram shows the challenge and how to copy or replicate objects in an S3 bucket in one region to a destination bucket in a different region. While objects can be copied or replicated without S3 cross-region replication, that involves essentially reading your objects pulling that data out via the internet and then writing to another place. The catch is that this can add extra costs, take time, consume network bandwidth and need extra tools (Cloudberry, Cyberduck, S3fuse, S3motion, S3browser, S3 tools (not AWS) and a long list of others).

What is AWS S3 Cross-region replication

Highlights of AWS S3 Cross-region replication include:

AWS S3 Cross region replication is as its name implies, replication of S3 objects from a bucket in one region to a destination bucket in another region.
S3 replication of new objects added to an existing or new bucket (note new objects get replicated)
Policy based replication tied into S3 versioning and life-cycle rules
Quick and easy to set up for use in a matter of minutes via S3 dashboard or other interfaces
Keeps region to region data replication and movement within AWS networks (potential cost advantage)

To activate, you simply enable versioning on a bucket, enable cross-region replication, indicate source bucket (or prefix of objects in bucket), specify destination region and target bucket name (or create one), then create or select an IAM (Identify Access Management) role and objects should be replicated.

Some AWS S3 cross-region replication things to keep in mind (e.g. considerations):
As with other forms of mirroring and replication if you add something on one side it gets replicated to other side
As with other forms of mirroring and replication if you deleted something from the other side it can be deleted on both (be careful and do some testing)
Keep costs in perspective as you still need to pay for your S3 storage at both locations as well as applicable internal data transfer and GET fees
Click here to see current AWS S3 fees for various regions

S3 Cross-region replication and alternative approaches

There are several regions around the world and up until today AWS customers could copy, sync or replicate S3 bucket contents between AWS regions manually (or via automation) using various tools such as Cloudberry, Cyberduck, S3browser and S3motion to name just a few as well as via various gateways and other technologies. Some of those tools and technologies are open-source or free, some are freemium and some are premium for a few that also vary by interface (some with GUI, others with CLI or APIs) including ability to mount an S3 bucket as a local network drive and use tools to sync or copy.

However a catch with the above mentioned tools (among others) and approaches is that to replicate your data (e.g. objects in a bucket) can involve other AWS S3 fees. For example reading data (e.g. a GET which has a fee) from one AWS region and then copying out to the internet has fees. Likewise when copying data into another AWS S3 region (e.g. a PUT which are free) there is also the cost of storage at the destination.

AWS S3 cross-region hands on experience (first look)

For my first hands on (first look) experience with AWS cross-region replication today I enabled a bucket in the US Standard region (e.g. Northern Virginia) and created a new target destination bucket in the EU Ireland. Setup and configuration was very quick, literally just a few minutes with most of the time spent reading the text on the new AWS S3 dashboard properties configuration displays.

I selected an existing test bucket to replicate and noticed that nothing had replicated over to the other bucket until I realized that new objects would be replicated. Once some new objects were added to the source bucket within a matter of moments (e.g. few minutes) they appeared across the pond in my EU Ireland bucket. When I deleted those replicated objects from my EU Ireland bucket and switched back to my view of the source bucket in the US, those new objects were already deleted from the source. Yes, just like regular mirroring or replication, pay attention to how you have things configured (e.g. synchronized vs. contribute vs. echo of changes etc.).

While I was not able to do a solid quantifiable performance test, simply based on some quick copies and my network speed moving via S3 cross-region replication was faster than using something like s3motion with my server in the middle.

It also appears from some initial testing today that a benefit of AWS S3 cross-region replication (besides being bundled and part of AWS) is that some fees to pull data out of AWS and transfer out via the internet can be avoided.

Where to learn more

Here are some links to learn more about AWS S3 and related topics

What this all means and wrap-up

For those who are looking for a way to streamline replicating data (e.g. objects) from an AWS bucket in one region with a bucket in a different region you now have a new option. There are potential cost savings if that is your goal along with performance benefits in addition to using what ever might be working in your environment. Replicating objects provides a way of expanding your business continuance (BC), business resiliency (BR) and disaster recovery (DR) involving S3 across regions as well as a means for content cache or distribution among other possible uses.

Overall, I like this ability for moving S3 objects within AWS, however I will continue to use other tools such as S3motion and s3sfs for moving data in and out of AWS as well as among other public cloud serves and local resources.

Disclosure: My company does not have a business relationship with this vendor other than being a customer.

it_user6186

Independent Analyst and Advisory Consultant at Server StorageIO - www.storageio.com

Mar 26, 2015

I like the new ability for moving S3 objects within AWS, however I will continue to use other tools for moving data in and out of AWS.

Pros and Cons

"For those who are looking for a way to streamline replicating data (e.g. objects) from an AWS bucket in one region with a bucket in a different region you now have a new option."

"However a catch with the above mentioned tools (among others) and approaches is that to replicate your data (e.g. objects in a bucket) can involve other AWS S3 fees."

Cloud Conversations: AWS S3 Cross Region Replication storage enhancements

Amazon Web Services (AWS) recently among other enhancements announced new Simple Storage Service (S3) cross-region replication of objects from a bucket (e.g. container) in one region to a bucket in another region. AWS also recently enhanced Elastic Block Storage (EBS) increasing maximum performance and size of Provisioned IOPS (SSD) and General Purpose (SSD) volumes. EBS enhancements included ability to store up to 16 TBytes of data in a single volume and do 20,000 input/output operations per second (IOPS). Read more about EBS and other recent AWS server, storage I/O and application enhancements here.

The Problem, Issue, Challenge, Opportunity and Need

The challenge is being able to move data (e.g. objects) stored in AWS buckets in one region to another in a safe, secure, timely, automated, cost-effective way.

Even though AWS has a global name-space, buckets and their objects (e.g. files, data, videos, images, bit and byte streams) are stored in a specific region designated by the customer or user (AWS S3, EBS, EC2, Glacier, Regions and Availability Zone primer can be found here).

Understanding the challenge and designing a strategy

The following diagram shows the challenge and how to copy or replicate objects in an S3 bucket in one region to a destination bucket in a different region. While objects can be copied or replicated without S3 cross-region replication, that involves essentially reading your objects pulling that data out via the internet and then writing to another place. The catch is that this can add extra costs, take time, consume network bandwidth and need extra tools (Cloudberry, Cyberduck, S3fuse, S3motion, S3browser, S3 tools (not AWS) and a long list of others).

What is AWS S3 Cross-region replication

Highlights of AWS S3 Cross-region replication include:

AWS S3 Cross region replication is as its name implies, replication of S3 objects from a bucket in one region to a destination bucket in another region.
S3 replication of new objects added to an existing or new bucket (note new objects get replicated)
Policy based replication tied into S3 versioning and life-cycle rules
Quick and easy to set up for use in a matter of minutes via S3 dashboard or other interfaces
Keeps region to region data replication and movement within AWS networks (potential cost advantage)

To activate, you simply enable versioning on a bucket, enable cross-region replication, indicate source bucket (or prefix of objects in bucket), specify destination region and target bucket name (or create one), then create or select an IAM (Identify Access Management) role and objects should be replicated.

Some AWS S3 cross-region replication things to keep in mind (e.g. considerations):
As with other forms of mirroring and replication if you add something on one side it gets replicated to other side
As with other forms of mirroring and replication if you deleted something from the other side it can be deleted on both (be careful and do some testing)
Keep costs in perspective as you still need to pay for your S3 storage at both locations as well as applicable internal data transfer and GET fees
Click here to see current AWS S3 fees for various regions

S3 Cross-region replication and alternative approaches

There are several regions around the world and up until today AWS customers could copy, sync or replicate S3 bucket contents between AWS regions manually (or via automation) using various tools such as Cloudberry, Cyberduck, S3browser and S3motion to name just a few as well as via various gateways and other technologies. Some of those tools and technologies are open-source or free, some are freemium and some are premium for a few that also vary by interface (some with GUI, others with CLI or APIs) including ability to mount an S3 bucket as a local network drive and use tools to sync or copy.

However a catch with the above mentioned tools (among others) and approaches is that to replicate your data (e.g. objects in a bucket) can involve other AWS S3 fees. For example reading data (e.g. a GET which has a fee) from one AWS region and then copying out to the internet has fees. Likewise when copying data into another AWS S3 region (e.g. a PUT which are free) there is also the cost of storage at the destination.

AWS S3 cross-region hands on experience (first look)

For my first hands on (first look) experience with AWS cross-region replication today I enabled a bucket in the US Standard region (e.g. Northern Virginia) and created a new target destination bucket in the EU Ireland. Setup and configuration was very quick, literally just a few minutes with most of the time spent reading the text on the new AWS S3 dashboard properties configuration displays.

I selected an existing test bucket to replicate and noticed that nothing had replicated over to the other bucket until I realized that new objects would be replicated. Once some new objects were added to the source bucket within a matter of moments (e.g. few minutes) they appeared across the pond in my EU Ireland bucket. When I deleted those replicated objects from my EU Ireland bucket and switched back to my view of the source bucket in the US, those new objects were already deleted from the source. Yes, just like regular mirroring or replication, pay attention to how you have things configured (e.g. synchronized vs. contribute vs. echo of changes etc.).

While I was not able to do a solid quantifiable performance test, simply based on some quick copies and my network speed moving via S3 cross-region replication was faster than using something like s3motion with my server in the middle.

It also appears from some initial testing today that a benefit of AWS S3 cross-region replication (besides being bundled and part of AWS) is that some fees to pull data out of AWS and transfer out via the internet can be avoided.

Where to learn more

Here are some links to learn more about AWS S3 and related topics

What this all means and wrap-up

For those who are looking for a way to streamline replicating data (e.g. objects) from an AWS bucket in one region with a bucket in a different region you now have a new option. There are potential cost savings if that is your goal along with performance benefits in addition to using what ever might be working in your environment. Replicating objects provides a way of expanding your business continuance (BC), business resiliency (BR) and disaster recovery (DR) involving S3 across regions as well as a means for content cache or distribution among other possible uses.

Overall, I like this ability for moving S3 objects within AWS, however I will continue to use other tools such as S3motion and s3sfs for moving data in and out of AWS as well as among other public cloud serves and local resources.

Disclosure: My company does not have a business relationship with this vendor other than being a customer.