AWS Partner Network (APN) Blog
Building a Cloud Native File System with Cloud Native Qumulo and HAQM S3
By Bill Crew, Principal Technical Marketing Engineer – Qumulo
By Adam Provost, Senior Director – Qumulo
By Dylan Souvage, Solutions Architect – AWS
By John Giles, Senior Solutions Architect – AWS
 |
 |
Organizations are looking to modernize their file storage infrastructure by moving to the cloud. Enterprises seeking to modernize file-based workloads in the cloud need solutions that go beyond traditional migration approaches to fully realize cloud benefits. Cloud Native Qumulo (CNQ) on AWS delivers an approach that ensures workload compatibility, preserves performance, and controls costs while leveraging cloud-native capabilities. This solution helps businesses improve their cloud performance and build a better file storage system in the cloud, rather than moving their existing system.
Suppose your enterprise wants to expand beyond the on-premises data center into a cloud architecture, and you are looking to maximize the value of all of your unstructured data. CNQ on AWS provides a seamless path for organizations to harness the full potential of this diverse file data extended on-premises into AWS, enabling flexibility and operational efficiency. In that case, CNQ on AWS delivers the flexibility and capacity of object storage while being fully compatible with file-based workflows. As a data platform purpose-built for the cloud, CNQ has these characteristics: 1/Elasticity, meaning that performance and capacity can independently scale both up and down dynamically, 2/Boundless Scale with full multi-protocol support, and 3/ Utility base pricing, like HAQM Simple Storage Service (HAQM S3). CNQ operates on a pay-as-you-go model, charging only for the resources used with no need to pre-provision capacity or performance.
Qumulo’s cloud-native solution facilitates the effortless migration of diverse workflows, from massive archives to high performance computing (HPC) applications, seamlessly transitioning them from traditional data centers to AWS. With AWS’s robust cloud infrastructure, organizations can tailor a cloud file solution that scales to meet their size and performance requirement, while unlocking new possibilities in cloud-based AI and HPC workloads. CNQ’s scalability allows it to adapt to changing organizational needs, such as the addition or expansion of workloads.
About Cloud Native Qumulo
CNQ empowers organizations to leverage a fully customizable, multi-protocol solution that dynamically scales to meet workload performance requirements. CNQ, a cloud-native platform, uses HAQM Elastic Compute Cloud (HAQM EC2), AWS networking, and HAQM S3 to deliver scalability for modern workloads.
Cloud Native Qumulo on AWS delivers a fully dynamic file storage platform that is natively integrated with the AWS Cloud. Here’s what sets CNQ apart:
- Elastic Scalability: Each Cloud Native Qumulo (CNQ) instance on AWS can automatically scale to exabyte-level storage within a single namespace by simply adding data. CNQ on AWS offers straightforward performance adjustments—just add or remove EC2 compute instances to instantly boost throughput or IOPS, all without disruption and in just minutes. Plus, you only pay for the capacity and compute resources you use.
- Deployed in minutes: CNQ runs in your own HAQM Virtual Private Cloud (VPC), deployed via either Terraform or AWS CloudFormation, so you can choose the specific EC2 instance type you need to satisfy your workload’s performance requirements and build a complete file data platform on AWS in less than 6 minutes, for a three node cluster.
- Automatic TCO Management: Since CNQ can use HAQM S3 Intelligent Tiering for the persistent data layer, your data will automatically be optimized for your specific workload, balancing the storage cost against the data access cost. In addition, all data written to CNQ is compressed to ensure maximum cost efficiency.
CNQ’s fully customizable architecture can be configured for the specific throughput, and IOPS requirements of virtually any file or object-based workload. Purchase CNQ in a PayGo model, removing the need to pre-provision cloud-file services. Pay for what you use. CNQ delivers comparable performance and services to on-premises file storage at a similar TCO.
Qumulo’s cloud-native architecture redefines cloud storage by decoupling capacity from performance, allowing these to be adjusted independently and on-demand. This provides the flexibility to change underlying components, such as the compute instance type and count, and cache disk capacity, allowing for rapid and non-disruptive performance adjustments. This architecture, which includes the innovative Qumulo NeuralCache, optimizes data movement and access, especially for large-scale or file-based workloads. It provides an adaptive storage platform, ensuring that businesses can efficiently manage and scale their data storage as their needs grow, without compromising on performance or reliability.
CNQ retains all the core Qumulo functionalities, including real-time analytics, robust data protection, security, and global collaboration. Figure 1 architecture shows how CNQ integrates into the cloud’s elastic resource model, providing exceptional flexibility and efficiency. This makes CNQ ideal for both hybrid-cloud enterprises.
Figure 1: CNQ dis-aggregated cloud architecture with node overview and abstraction layer
Features and Benefits of CNQ
Beyond the inherent scalability and dynamic elasticity in every deployment, CNQ supports enterprise-class data-management features such as snapshots, replication, and quotas. CNQ also offers full multi-protocol support – NFS, SMB, REST, and S3 – for all your data. By letting you share the same data via both file and S3 protocols, CNQ enables collaborative and mixed-use workloads, eliminating the need to import file data into object storage with consistently low time-to-first-byte latencies of 1-2ms. CNQ delivers a combined file and object platform to satisfy your most performance-intensive AI and HPC workloads.
CNQ can run in most AWS Availability Zones as easily as it runs in any AWS region worldwide, allowing your on-premises data centers to take advantage of the AWS cloud’s scalability, reliability, and durability. CNQ can also be dynamically reconfigured without taking services offline, so you can boost or shrink performance temporarily or permanently if your workloads change. A CNQ instance initially deployed as a disaster recovery or archive target can be converted from a low-cost, high-capacity service to a high-performance data platform in seconds without redeploying the service or migrating any hosted data.
If you already use Qumulo storage on-premises or in other cloud platforms, Qumulo’s Cloud Data Fabric (CDF) enables seamless data movement between your on-premises, edge, and AWS-based deployments. Connect portals between locations to build a Global Namespace and instantly express your on-premises data to AWS’s portfolio of cloud-native applications, like AWS Deadline Cloud for burst rendering, or one of many HPC engines, with CDF instantly and seamlessly moving files through a large-scale data pipeline.
Use Qumulo’s continuous replication engine to enable disaster recovery scenarios, or combine replication with Qumulo’s cryptographically locked snapshot feature to protect older versions of critical data from loss or ransomware. CNQ leverages S3’s 11-nines to achieve a durable and highly available file system. CNQ may utilize multiple availability zones for even more availability—without the added costs typically associated with data replication required by other file systems.
Conclusion
CNQ on AWS offers a powerful, flexible, and scalable solution for enterprises looking to expand their data storage to the cloud without compromising on compatibility, performance, or reliability. By combining the best of cloud-native capabilities with robust data management features, CNQ enables organizations to seamlessly migrate their most demanding workloads to AWS, whether for archiving, high-performance computing, or AI. With its elastic scalability, cost-efficient pricing model, and seamless integration with S3, Qumulo enables enterprises to optimize their storage needs dynamically and securely, all while minimizing the complexities and risks of traditional cloud migrations. CNQ provides the future-proof solution that today’s organizations need to thrive in an increasingly cloud-driven world.
Ready to get started with CNQ on AWS? Check out Qumulo’s comprehensive AWS Administrator Guide for detailed implementation instructions, or go directly to the AWS Marketplace to deploy your solution today. For more insights on how Qumulo can transform your cloud storage strategy, visit the Qumulo Blog for the latest updates, use cases, and success stories. Check out the AWS Partner Network (APN) Blog for further reading.
.
.
Qumulo – AWS Partner Spotlight
Qumulo is an AWS Advanced Technology Partner and AWS Competency Partner, providing a cloud-native, scalable platform for managing unstructured data across on-premises, edge, and cloud environments. Its solution supports exabyte-scale data, hybrid cloud deployments, and multi-protocol compatibility (SMB, NFS, S3), simplifying data management for complex workloads and serving industries like media and entertainment, healthcare, life sciences, and more. Qumulo provides scalable data storage that can accommodate any data, be deployed across any location, and offers complete control over management and access.