AWS Storage Blog
Migrating files to HAQM FSx for Windows File Server using Robocopy
When migrating file shares to AWS, users need a direct and efficient solution for transferring their data. HAQM FSx for Windows File Server provides fully managed shared storage built on Windows Server, and delivers a wide range of data access, data management, and administrative capabilities.
There are a few different choices for migration. AWS DataSync is the most common tool for the job, however setting it up can introduce added complexity, especially for smaller migrations. In such cases, Robocopy (short for “Robust File Copy”) can be a powerful alternative. Robocopy is a command-line utility built into Windows, designed for copying and synchronizing files and directories, which offers control, scalability, and file verification.
In this post, we walk you through the process of migrating data from an existing Windows File Server to HAQM FSx for Windows File Server using Robocopy. We do recommend using DataSync for moving data to or from AWS storage services (such as HAQM S3, HAQM Elastic File System (HAQM EFS), FSx for Windows File Server) in most cases. This is because DataSync is a cross-platform service that can handle NFS and SMB protocols, while Robocopy is a Windows-only tool that works specifically with NTFS. Robocopy also does not have built-in encryption or compression, thus it can be detrimental for larger migrations and isn’t recommended when moving data across unsecured channels such as the internet.
Solution overview
Our objective is to migrate existing files to the cloud quickly and reliably. To do this, we use three key components: 1/source file server, 2/FSx for Windows file system, and 3/EC2 intermediary instance that acts as the bridge between the source file server and FSx for Windows file system. The solution is shown in figure 1.
We begin by creating the HAQM FSx for Windows file system. This acts as the final destination for our files. Then, we create an EC2 instance and mount the source file share (on-premises or otherwise) and the HAQM FSx for Windows file system target file shares to it. When the EC2 instance has access to both the source file server and FSx for Windows File Server, we can use Robocopy to migrate the data from one share to the other efficiently.
Figure 1: Solution overview
Although a t2.medium instance may suffice for testing, we recommend choosing a larger instance, especially with higher throughput SSD volumes such as gp2 or gp3 volumes for large-scale or time-sensitive migrations. Lower-tier instances could act as a performance bottleneck during high-data transfers.
Prerequisites
HAQM FSx for Windows File Server needs domain integration for authentication and access control. You can join FSx for Windows File Server to an external domain controller or use AWS Managed Microsoft AD. AWS Managed Microsoft AD is a simple, managed Active Directory service that can be set up with just a few clicks in the AWS console. You can follow this post on how to set it up. You also need an existing file server and established connectivity from it to AWS through a private tunnel, Direct Connect, or the internet.
Walkthrough
We will walk through the following steps to implement this solution.
- Set up an HAQM FSx for Windows File Server file system.
An FSx for Windows File Server file system is a specific folder (and its subfolders) within FSx for Windows File Server that is made accessible to compute instances over the network using the SMB protocol. Start by creating a new FSx for Windows File Server file system that you can access.
1.1. From the AWS Management Console, navigate to FSx and choose Create file system.
1.2. Choose HAQM FSx for Windows file server with a Multi-AZ deployment for high availability.
1.3. Use SSD storage for higher throughput and scalability. The file system size can be increased but not decreased after creation.
1.4. Configure VPC settings and Windows authentication through AWS Managed Microsoft AD or Self-managed AD. To create an AWS Managed Microsoft AD from scratch, you can follow this documentation. Make sure that the VPC selected is the VPC in which you want this file server to exist in the future. Depending on your scenario, fill out the information regarding your domain.
1.5. Complete the set up and save your DNS name and network configuration details when available. On the Review and create page, you can now see a summary of your selections for the FSx for Windows File Server file system. Note the settings that can and can’t be modified after the file system has been created, then choose the Create file system button.
1.6. When the file system has been created and shows as Available on the File systems overview page, choose File system ID and note the DNS name for the file system under Network and security.
Figure 2: Create a file system
2. Set up an EC2 Windows instance
Now that you have a FSx for Windows File Server running, you can spin up a client in HAQM EC2 to view and access the shares. For our example, we are using a t2.medium EC2 instance running Microsoft Windows Server 2022 Base.
2.1. From the search bar on the top of the console, type in “EC2” and choose EC2 from the Services drop-down.
2.2. On the EC2 Dashboard page, choose the Launch instance button.
2.3. From the Launch an instance page, find the Name field and enter a name for the new EC2 instance.
2.3.1. Under Quick Start choose Windows, then under HAQM Machine Image (AMI) choose Microsoft Server 2022 Base.
2.3.2. Under Instance type choose t2.medium
2.3.3. Under Key pair (login) there is a subsection called Key pair name. Choose Create new key pair on the right. Input a memorable key pair name, and make sure the Private key file format section has .pem chose, then choose the Create key pair button. You are prompted to download the Private Key. Save the Private Key on the local computer. This Private Key file is not used in this tutorial, but it is the only way to retrieve your local administrator password for this EC2 instance.
2.3.4. Under Network Settings, scroll down to Firewall (security groups) and choose a security group that allows incoming RDP connectivity. Make sure the Network VPC and Subnet are the same as the FSx for Windows File Server Share.
2.3.5. Under Configure storage you leave leave the default of 30 GiB of gp2 storage.
2.3.6. Expand the Advanced details section and locate the Domain join directory section. Expand the drop-down and choose your preferred domain.
2.3.7. Locate the IAM instance profile drop-down box and choose the AWS Identity and Access Management (IAM) role that can join a domain.
2.3.8. Choose the Launch Instance button on the right side of the page. The creation and launch of the EC2 instance take a few minutes. Expand the Hamburger Menu icon on the left and then choose Instances. You may need to press the refresh button to see the newly created instance.
2.4. In the Instances page, choose the newly created Instance ID from Step 5.3.8. On the Instance summary page, note the Public IPv4 DNS address.
3. RDP into the EC2 instance
The creation of the necessary resources is complete. You are now ready to connect to the EC2 instance and mount the FSx for Windows File Server file system as shown in figure 3.
3.1. Enter the Public IPv4 DNS address from Step 2.4 into the Microsoft Remote Desktop client and choose Connect.
3.2. You are prompted to log in to the EC2 instance. Enter the Domain Name, a backslash, and then a valid username. For testing purposes, we can use the Domain Admin username. Reference step 1.4. where the Active Directory settings were entered during the FSx for Windows File Server creation process. The full username should look something like the following: corp.example.com\admin.
3.3. Input the password and choose OK.
3.4. If you are prompted for a security dialog box about an untrusted certificate, then this can be ignored. Choose Yes to continue.
3.5. During the first log in, the initial log in process takes a few minutes. After the desktop has appeared, you can mount your file system created in Step 1.
Figure 3: Connect to EC2 instance
4. Mount the FSx for Windows File Server file system
Next, you mount the file shares on the EC2 instance. This instance acts as the intermediary between source file server and AWS.
4.1. After you’re connected, open File Explorer as in figure 4.
4.2. In the navigation panel, open the context (right-click) menu for Network, and choose Map Network Drive.
4.3. For Drive, choose a drive letter.
4.4. For Folder, enter either the file system’s DNS name or a DNS alias associated with the file system, and the share name.
Using an IP address instead of the DNS name could result in unavailability during the failover process of the Multi-Availability Zone (AZ) file system. Furthermore, DNS names or associated DNS aliases are necessary for Kerberos-based authentication in Multi-AZ and Single- AZ file systems. You can find the file system’s DNS name and any associated DNS aliases on the HAQM FSx console by choosing Windows File Server, Network & security.
4.5. To use a DNS alias associated with the file system, enter the following for Folder.
\\<fqdn-dns-alias>\share
Figure 4: Map network drive
5. Mount the source file share
5.1. After you’re connected, open File Explorer.
5.2. In the navigation panel, open the context (right-click) menu for Network, and choose Map Network Drive.
5.3. For Drive, choose a drive letter.
5.4. For Folder, enter either the file system’s DNS name or IP associated with the source file system, and the share name.
6. Using Robocopy
Here you use Robocopy to migrate the files from source to AWS using the created EC2 instance as the bridge between both environments.
6.1. Open Command Prompt or Windows PowerShell as an administrator, and run the following Robocopy command to copy the files from the source share to the target share as shown in figure 5.
robocopy
Y:\source-folder\ Z:\target-FSx-folder\ /copy:DATSOU /secfix /e /b /MT:8
/LOG:C:\migration.log /v /teeThis command uses the following elements and options:
- Y:\source-folder\ – Refers to the source share.
- Z:\target-FSx-folder\ – Refers to the target file system on FSx for Windows File Server. Ex: \\amznfsxabcdef1.mydata.com\share
- /copy – Specifies the following file properties to be copied:
- D – data
- A – attributes
- T – timestamps
- S – NTFS ACLs
- O – owner information
- U – auditing information.
- /secfix – Makes sure that file security is updated for all files, even if the file itself wasn’t copied during the operation.
- /e – Copies subdirectories, including empty ones.
- /b – Uses the backup and restore privilege in Windows to copy files even if their NTFS ACLs deny permissions to the current user.
- /MT:8 – Specifies how many threads to use for performing multithreaded copies.
- /LOG:C:\migration.log – Writes the status output to “migration” log file in the C disk.
- /v – Produces verbose output, and shows all skipped files.
- /tee – Writes the status output to the console window, and to the log file.
Figure 5: Windows command prompt
If you are copying large files over a slow or unreliable connection, then you can enable restartable mode by using the /zb option in place of the /b option. With restartable mode, if the transfer of a large file is interrupted, a subsequent Robocopy operation can pick up in the middle of the transfer instead of having to re-copy the entire file from the beginning. Enabling restartable mode can reduce the data transfer speed.
After completing the initial copy, you can maintain synchronization between directories using Robocopy with the /MIR switch. This command mirrors the source directory to the destination, ensuring that any updates, additions, or deletions are consistently reflected. This approach is particularly useful for directories requiring continuous updates until the final cutoff time.
Cleaning up
If you’re running this as a lab, then you may need to delete the resources created using this post. To do so, go to the HAQM FSx console and choose the FSx for Windows File Server file system that was created, then choose Delete file system on the Actions button drop-down on the top-right.
To clean up the EC2 instance, go to the HAQM EC2 console, choose Instances on the left, choose the instance that was created on the right side, then choose Terminate (delete) instance on the Instance state button drop-down on the top right.
Conclusion
In this post, we walk you through the process of migrating data from an existing Windows File Server to HAQM FSx for Windows File Server using Robocopy. Migrating files to HAQM FSx for Windows File Server using Robocopy offers a practical and efficient approach for smaller-scale migrations where AWS DataSync might be too complex. This method uses an intermediary EC2 instance to bridge the source file server and FSx for Windows File Server file system to provide flexibility and control over the migration process, making sure that file attributes, permissions, and other metadata are preserved.
Additional resources
Migrating files to HAQM Fsx with Datasync