Arcesium Uses HAQM EBS and HAQM FSR for Faster Database Refreshes

Arcesium, a global financial technology and professional services firm, recognized that speedier database refreshes would benefit its clients. The company ultimately turned to HAQM Web Services (AWS) to employ HAQM Elastic Block Store (HAQM EBS), a simple-to-use and high-performance block storage service, and HAQM EBS Fast Snapshot Restore (HAQM FSR). As a result, the company achieved rapid scaling, high performance, ample storage, and better resource management capabilities.

In 2014, Arcesium was in the process of spinning off from the D. E. Shaw group, a global investment and technology development firm. At that time, Arcesium considered whether to opt for an on-premises data center or a cloud provider for its computing needs. It ultimately made the decision to use AWS, largely because of the length of time AWS had been in the market and its available suite of dynamic services. “We serve the world’s most sophisticated financial institutions with very complex workflows, and we needed high-performing infrastructure that could scale and meet our data processing needs. We ran benchmarks with various providers,” says Manikantan Madhunapantula, associate director of Arcesium and head of the company’s database engineering group. “We compared on-premises and AWS instance sizes and throughput—input/output operations per second specifically. AWS had a significant advantage over other providers—especially for running our database workloads.”

One of the first challenges Arcesium faced was finding a way to refresh its multiterabyte nonproduction database servers on AWS without using expensive filer software, degrading performance, or incurring a lot of downtime. “Without the capability to refresh with the latest production data, clients would have to invest a lot of time and energy to create data to successfully test their daily, weekly, and monthly data-dependent workflows,” says Manikantan. “Whatever solution we embarked on had to provide a consistent, point-in-time view of multiple databases across the multiple volumes in our environment.”

As Arcesium started its cloud journey, it planned to run database workloads on HAQM Elastic Compute Cloud (HAQM EC2) instances, using HAQM EBS volumes to satisfy its performance requirements. Because Arcesium runs multiterabyte databases, it needed the most powerful instances available from HAQM EC2, a web service that provides secure, resizable compute capacity in the cloud.

Although HAQM EC2 and HAQM EBS offer high performance and ample storage, Arcesium recognized the need to develop an innovative solution to handle an increasing number of refreshes. Arcesium initially made copy-on-write backups for a consistent point-in-time view of databases, then mounted the shadow copy to an HAQM EC2 instance and copied all data over to another disk drive to accomplish refreshes. While this worked fine, Arcesium also wanted to provide the capability to manage multiple environment refreshes over the same weekend to enable clients to run numerous transformative projects simultaneously. That’s when Arcesium reached out to AWS and was introduced to what would become HAQM FSR, as it enables fast restores.

Arcesium wanted to achieve application-consistent snapshots for its databases, so it explored HAQM EBS snapshots using AWS Systems Manager—which provides visibility and control of a customer’s infrastructure on AWS—to discover a way to make a copy-only backup without affecting the backup chain. Arcesium was also able to further streamline the process by reducing the snapshots of all HAQM EBS volumes attached to the instance to just those that were needed. Using HAQM FSR, the company facilitated refreshes of up to 30 multiterabyte environments within the weekend time window while successfully reducing refresh costs by 58 percent and refresh time by 59 percent.

This innovative use of the technologies means that the company can now refresh multiple HAQM EBS databases over the weekend without the addition of filer-based software or other costly solutions. Refreshing client environments used to take the entire weekend, but now Arcesium only needs a few hours. This increase in both speed and performance has resulted in increased satisfaction for Arcesium’s clients, who can now successfully test their data-dependent workloads quickly and efficiently. “The capability to refresh multiple client environments provides clients the capacity to run multiple large transformation projects at the same time—for example, testing new system integration in one staging environment while testing an upgrade of an accounting system in another,” says Manikantan.

In the technology and professional services industry, it’s vital to have a plan in case of a disaster. For extensive databases, full backups take a lot of time, but Arcesium discovered that it could use application-consistent HAQM EBS snapshots and HAQM FSR for database backups as well. Using HAQM EBS snapshots, Arcesium can pause the input/output while taking the backups, and it no longer needs to stop the database. When the company uses HAQM FSR while restoring from the HAQM EBS snapshots, the target incurs a downtime of only 10–15 minutes, versus 15–20 hours for a traditional backup/restore for large databases. If the group wants to go back to a certain point in time, it can use the snapshot it has on hand—and, depending on the size, this can actually be done within 4 hours. “Using application-consistent HAQM EBS snapshots as database backups would be really helpful for any AWS customers running databases on top of HAQM EC2 instances,” says Manikantan.

By using HAQM EBS and HAQM FSR, Arcesium can provide faster database refresh times for more clients without compromising performance or incurring downtime. Arcesium is looking to further explore what it can do on AWS. “We’re looking to create a strategy in which we can quickly move hundreds of terabytes on AWS by using complete copy clones,” says Manikantan. That type of speed and data storage would mean even faster and more efficient services for Arcesium’s global clientele.