[SEO Subhead]
This Guidance demonstrates how to securely transfer HAQM Simple Storage Service (HAQM S3) objects and HAQM Elastic Container Registry (HAQM ECR) images across AWS environments. Through an intuitive user interface, you can easily create and manage various types of transfer tasks between AWS partitions (aws, aws-cn, aws-us-gov) and from other cloud providers to AWS. The Guidance provides scalable and trackable data transfers, streamlining the process of moving both HAQM S3 objects and HAQM ECR images between environments.
Please note: If you deploy this solution in the AWS (Beijing) Region operated by Beijing Sinnet Technology Co., Ltd. (Sinnet), or the AWS (Ningxia) Region operated by Ningxia Western Cloud Data Technology Co., Ltd., you are required to provide a domain with ICP Recordal before you can access the web console.
The web console is a centralized place to create and manage all data transfer jobs. Each data type (for example, HAQM S3 or HAQM ECR) is a plugin for Data Transfer Hub, and is packaged as an AWS CloudFormation template hosted in an HAQM S3 bucket that AWS owns. When you create a transfer task, an AWS Lambda function initiates the AWS CloudFormation template, and state of each task is stored and displayed in the DynamoDB tables.
As of today, the solution supports two data transfer plugins: an HAQM S3 plugin and an HAQM ECR plugin.
Architecture Diagram
-
Overview
-
HAQM ECR transfer option
-
HAQM S3 transfer option
-
Overview
-
This architecture diagram illustrates how to secure, scale, and track data transfer for HAQM S3 objects and HAQM ECR images.
Click to enlarge
Step 1
HAQM Simple Storage Service (HAQM S3) stores static web assets (such as the frontend UI), which are made available through HAQM CloudFront.Step 2
AWS AppSync GraphQL provides backend APIs.Step 3
Users are authenticated by either HAQM Cognito user pools (in AWS Standard Regions) or by an OpenID connect provider (in AWS China Regions) such as Authing or Auth0.Step 4
AWS AppSync runs AWS Lambda to call backend APIs.Step 5
Lambda starts an AWS Step Functions workflow that uses AWS CloudFormation to start or stop or delete HAQM Elastic Container Registry (HAQM ECR) or the HAQM S3 plugin template.Step 6
A centralized S3 bucket, managed by AWS, hosts plugin templates.
Step 7
The solution also provisions an HAQM Elastic Container Service (HAQM ECS) cluster that runs the container images used by the plugin template, and the container images are hosted in HAQM ECR.Step 8
HAQM DynamoDB stores data transfer task information.Step 1
HAQM Simple Storage Service (HAQM S3) stores static web assets (such as the frontend UI), which are made available through HAQM CloudFront.Step 2
AWS AppSync GraphQL provides backend APIs.Step 3
Users are authenticated by either HAQM Cognito user pools (in AWS Standard Regions) or by an OpenID connect provider (in AWS China Regions) such as Authing or Auth0.Step 4
AWS AppSync runs AWS Lambda to call backend APIs.Step 5
Lambda starts an AWS Step Functions workflow that uses AWS CloudFormation to start or stop or delete HAQM Elastic Container Registry (HAQM ECR) or the HAQM S3 plugin template.Step 6
A centralized S3 bucket, managed by AWS, hosts plugin templates.
Step 7
The solution also provisions an HAQM Elastic Container Service (HAQM ECS) cluster that runs the container images used by the plugin template, and the container images are hosted in HAQM ECR.Step 8
HAQM DynamoDB stores data transfer task information. -
HAQM ECR transfer option
-
This architecture diagram illustrates how to run the HAQM ECR plugin to transfer container images from other container registries.
Click to enlarge
Step 1
An HAQM EventBridge rule runs a Step Functions workflow on a regular basis (by default, it runs daily).Step 2
Step Functions invokes Lambda to retrieve the list of images from the source.
Step 3
Lambda will either list all the repository content in the source HAQM ECR or get the stored image list from Parameter Store, a capability of AWS System Manager.Step 4
The transfer task will run within AWS Fargate in a maximum concurrency of 10. If a transfer task fails for some reason, it will automatically retry three times.Step 5
Each task uses skopeo to copy the images into the target HAQM ECR registry.Step 6
After the copy completes, the status (either success or fail) is logged into DynamoDB for tracking purposes.Step 1
An HAQM EventBridge rule runs a Step Functions workflow on a regular basis (by default, it runs daily).Step 2
Step Functions invokes Lambda to retrieve the list of images from the source.
Step 3
Lambda will either list all the repository content in the source HAQM ECR or get the stored image list from Parameter Store, a capability of AWS System Manager.Step 4
The transfer task will run within AWS Fargate in a maximum concurrency of 10. If a transfer task fails for some reason, it will automatically retry three times.Step 5
Each task uses skopeo to copy the images into the target HAQM ECR registry.Step 6
After the copy completes, the status (either success or fail) is logged into DynamoDB for tracking purposes. -
HAQM S3 transfer option
-
This architecture diagram illustrates how run the HAQM S3 plugin to transfer objects from their sources into S3 buckets.
Click to enlarge
Step 1
A time-based EventBridge rule initiates the Lambda function on an hourly basis.
Step 2
Lambda uses the launch template to launch a data comparison job (JobFinder) in HAQM Elastic Compute Cloud (HAQM EC2).Step 3
The job lists all the objects in the source and destination S3 buckets and makes comparisons among objects to determine which objects should be transferred.Step 4
HAQM EC2 sends a message for each object that will be transferred to HAQM Simple Queue Service (HAQM SQS). HAQM S3 event messages can also be supported for more real-time data transfer. Whenever an object is uploaded to the source bucket, the event message is sent to the same HAQM SQS queue.Step 5
A JobWorker node running in HAQM EC2 consumes the messages in HAQM SQS and transfers the object from the source bucket to the destination bucket. You can use an Auto Scaling group to control the number of EC2 instances to transfer the data based on business needs.
Step 6
DynamoDB stores a record with transfer status for each object.
Step 7
The EC2 instance will get (download) the object from the source bucket based on the HAQM SQS message.
Step 8
The EC2 instance will put (upload) the object to the destination bucket based on the HAQM SQS message.
Step 9
When the JobWorker node identifies a large file (with a default threshold of 1 GB) for the first time, a multipart upload task running in HAQM EC2 is initiated. The corresponding UploadId is then conveyed to Step Functions, which invokes a scheduled recurring task. Every minute, Step Functions verifies the successful transmission of the distributed shards associated with the UploadId across the entire cluster
Step 10
If all shards have been transmitted successfully, HAQM EC2 invokes the CompleteMultipartUpload API in HAQM S3 to finalize the consolidation of the shards. Otherwise, any invalid shards are discarded.
Step 1
A time-based EventBridge rule initiates the Lambda function on an hourly basis.
Step 2
Lambda uses the launch template to launch a data comparison job (JobFinder) in HAQM Elastic Compute Cloud (HAQM EC2).Step 3
The job lists all the objects in the source and destination S3 buckets and makes comparisons among objects to determine which objects should be transferred.Step 4
HAQM EC2 sends a message for each object that will be transferred to HAQM Simple Queue Service (HAQM SQS). HAQM S3 event messages can also be supported for more real-time data transfer. Whenever an object is uploaded to the source bucket, the event message is sent to the same HAQM SQS queue.Step 5
A JobWorker node running in HAQM EC2 consumes the messages in HAQM SQS and transfers the object from the source bucket to the destination bucket. You can use an Auto Scaling group to control the number of EC2 instances to transfer the data based on business needs.
Step 6
DynamoDB stores a record with transfer status for each object.
Step 7
The EC2 instance will get (download) the object from the source bucket based on the HAQM SQS message.
Step 8
The EC2 instance will put (upload) the object to the destination bucket based on the HAQM SQS message.
Step 9
When the JobWorker node identifies a large file (with a default threshold of 1 GB) for the first time, a multipart upload task running in HAQM EC2 is initiated. The corresponding UploadId is then conveyed to Step Functions, which invokes a scheduled recurring task. Every minute, Step Functions verifies the successful transmission of the distributed shards associated with the UploadId across the entire cluster
Step 10
If all shards have been transmitted successfully, HAQM EC2 invokes the CompleteMultipartUpload API in HAQM S3 to finalize the consolidation of the shards. Otherwise, any invalid shards are discarded.
Well-Architected Pillars
The AWS Well-Architected Framework helps you understand the pros and cons of the decisions you make when building systems in the cloud. The six pillars of the Framework allow you to learn architectural best practices for designing and operating reliable, secure, efficient, cost-effective, and sustainable systems. Using the AWS Well-Architected Tool, available at no charge in the AWS Management Console, you can review your workloads against these best practices by answering a set of questions for each pillar.
The architecture diagram above is an example of a Solution created with Well-Architected best practices in mind. To be fully Well-Architected, you should follow as many Well-Architected best practices as possible.
-
Operational ExcellenceRead the Operational Excellence whitepaper
Step Functions provides a reliable, scalable, and fault-tolerant workflow management system, helping ensure data transfer tasks are executed reliably. HAQM ECS and HAQM ECR simplify the deployment and management of containerized components, improving operational efficiency. These services enable DevOps best practices and promote a culture of continuous improvement and automation while reducing the operational overhead and manual effort required to manage infrastructure and data transfer workflows.
-
SecurityRead the Security whitepaper
HAQM Cognito and OpenID Connect providers help ensure that only authorized users can access and manage data transfer tasks. AWS AppSync provides a secure GraphQL interface to interact with the backend APIs, protecting against unauthorized access. DynamoDB offers secure storage for data transfer task details, with options for encryption at rest. These services provide a comprehensive, end-to-end secure workflow for managing data transfers across AWS partitions and external cloud providers, protecting sensitive data from unauthorized access or tampering.
-
ReliabilityRead the Reliability whitepaper
HAQM ECS and HAQM ECR offer a highly available and scalable way to run and manage container-based components. HAQM Simple Notification Service (HAQM SNS) provides a reliable notification mechanism. These services ensure you can reliably and consistently provision and manage the necessary infrastructure for data transfers, leveraging the scalability and high availability of containerized services like HAQM ECS and HAQM ECR.
-
Performance EfficiencyRead the Performance Efficiency whitepaper
Lambda provides a serverless, scalable, and highly performant compute service to run backend components. DynamoDB offers a fast and scalable NoSQL database to store data transfer task information, with low latency access. CloudFront improves performance and responsiveness of the web interface by caching and serving static assets from a global network of edge locations. These services leverage inherent performance and scalability benefits of serverless and managed AWS services, reducing operational overhead while ensuring the Guidance can handle increasing workloads without compromising performance.
-
Cost OptimizationRead the Cost Optimization whitepaper
Lambda and the serverless architecture allow for on-demand, pay-as-you-go compute resources, eliminating the need for always-on server infrastructure. DynamoDB provides a pay-per-request NoSQL database service, meaning you only pay for the resources consumed, without incurring costs of provisioning and maintaining a database. HAQM SNS and HAQM SQS provide cost-effective serverless notifications, while HAQM ECS on Fargate enables on-demand infrastructure for containers. This serverless approach helps reduce operational costs and overhead associated with managing and scaling infrastructure.
-
SustainabilityRead the Sustainability whitepaper
The serverless design using Lambda, HAQM SQS, and DynamoDB aims at reducing carbon footprint compared to continually operating on-premises servers. Step Functions, HAQM SNS, and HAQM SQS enable serverless notifications, while HAQM ECS on Fargate provides on-demand infrastructure for containers. These serverless and cloud-native services enable you to scale your infrastructure efficiently, reduce required resources and related carbon footprints as compared to on-premises servers, and optimize resource utilization.
Related Content
[Title]
Disclaimer
The sample code; software libraries; command line tools; proofs of concept; templates; or other related technology (including any of the foregoing that are provided by our personnel) is provided to you as AWS Content under the AWS Customer Agreement, or the relevant written agreement between you and AWS (whichever applies). You should not use this AWS Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the AWS Content, such as sample code, as appropriate for production grade use based on your specific quality control practices and standards. Deploying AWS Content may incur AWS charges for creating or using AWS chargeable resources, such as running HAQM EC2 instances or using HAQM S3 storage.
References to third-party services or organizations in this Guidance do not imply an endorsement, sponsorship, or affiliation between HAQM or AWS and the third party. Guidance from AWS is a technical starting point, and you can customize your integration with third-party services when you deploy the architecture.