Easily transfer data across AWS partitions
Overview
This solution provides secure, scalable, and trackable data transfer for HAQM Simple Storage Service (HAQM S3) objects and HAQM Elastic Container Registry (HAQM ECR) images. This data transfer helps customers easily create and manage different types (HAQM S3 object and HAQM ECR image) of transfer tasks between AWS partitions (for example, aws, aws-cn, aws-us-gov), and from other cloud providers to AWS.
What's new
- Provided support for HAQM S3 destination bucket being encrypted with HAQM S3 managed keys
- Provided the optional HAQM S3 bucket to hold prefix list file
- Added three supported Regions: Asia Pacific (Melbourne), Canada (Calgary), and Israel (Tel Aviv)
- Increased Finder capacity to 316GB & 512GB
To find out about other new features, refer to the Revisions page.
Benefits
The user interface allows customers to create and manage data transfer tasks for HAQM S3 objects and HAQM ECR images.
The solution supports data transfer to HAQM S3 from Alibaba Cloud OSS, Tencent COS, Qiniu Kodo, and HAQM S3 compatible cloud storage. Transfer to HAQM ECR from Docker Hub, Google gcr.io, and Red Hat Quay.io is also supported.
Transfer tasks are on-demand and pay-as-you-go. For more information, refer to the Cost section of the implementation guide.
Technical details
You can automatically deploy this architecture using the implementation guide and the AWS CloudFormation templates for AWS Regions or AWS China Regions.
Step 1
The solution’s static web assets (front end user interface) are stored in HAQM Simple Storage Service (HAQM S3) and made available through HAQM CloudFront.
Step 2
The backend APIs are provided via AWS AppSync GraphQL.
Step 3
Users are authenticated by either HAQM Cognito User Pool (in AWS Standard Regions) or by an OpenID connect provider (in AWS China Regions) such as Authing, Auth0, etc.
Step 4
AWS AppSync runs AWS Lambda to call backend APIs.
Step 5
Lambda starts an AWS Step Functions workflow that uses AWS CloudFormation to start or stop/delete the HAQM Elastic Container Registry (HAQM ECR) or HAQM S3 plugin template.
Step 6
The plugin templates are hosted in a centralized HAQM S3 bucket managed by AWS.
Step 7
The solution also provisions an HAQM Elastic Container Service (HAQM ECS) cluster that runs the container images used by the plugin template, and the container images are hosted in HAQM ECR.
Step 8
The data transfer task information is stored in HAQM DynamoDB.
Step 1
The solution’s static web assets (front end user interface) are stored in HAQM Simple Storage Service (HAQM S3) and made available through HAQM CloudFront.
Step 2
The backend APIs are provided via AWS AppSync GraphQL.
Step 3
Users are authenticated by either HAQM Cognito User Pool (in AWS Standard Regions) or by an OpenID connect provider (in AWS China Regions) such as Authing, Auth0, etc.
Step 4
AWS AppSync runs AWS Lambda to call backend APIs.
Step 5
Lambda starts an AWS Step Functions workflow that uses AWS CloudFormation to start or stop/delete the HAQM Elastic Container Registry (HAQM ECR) or HAQM S3 plugin template.
Step 6
The plugin templates are hosted in a centralized HAQM S3 bucket managed by AWS.
Step 7
The solution also provisions an HAQM Elastic Container Service (HAQM ECS) cluster that runs the container images used by the plugin template, and the container images are hosted in HAQM ECR.
Step 8
The data transfer task information is stored in HAQM DynamoDB.
- Publish Date
Important
If you deploy this solution in the AWS (Beijing) Region operated by Beijing Sinnet Technology Co., Ltd. (Sinnet), or the AWS (Ningxia) Region operated by Ningxia Western Cloud Data Technology Co., Ltd., you are required to provide a domain with ICP Recordal before you can access the web console.
The web console is a centralized place to create and manage all data transfer jobs. Each data type (for example, HAQM S3 or HAQM ECR) is a plugin for Data Transfer Hub, and is packaged as an AWS CloudFormation template hosted in an HAQM S3 bucket that AWS owns. When you create a transfer task, an AWS Lambda function initiates the AWS CloudFormation template, and state of each task is stored and displayed in the DynamoDB tables.
As of today, the solution supports two data transfer plugins: an HAQM S3 plugin and an HAQM ECR plugin.