Posted On: Sep 3, 2019
HAQM SageMaker customers can now use smaller HAQM FSx for Lustre file systems as the data source for training machine learning models. Until today, the smallest FSx for Lustre file system that could be created was 3.6 TBs. For training sets that are smaller than this size, customers can now create and use file systems as small as 1.2 TB.
HAQM FSx for Lustre is a high performance file system that works with HAQM S3 data and is optimized for workloads such as machine learning, analytics, and high performance computing.
With FSx for Lustre, customers can accelerate their HAQM SageMaker training jobs. Until support for HAQM FSx for Lustre was announced on August 27, 2019, HAQM SageMaker transparently downloaded a full training set from HAQM S3 to local file storage at the start of a training job, when using the File input mode. Now, with HAQM FSx for Lustre, customers can accelerate their File mode training jobs by avoiding the initial HAQM S3 download time. When linked to HAQM S3, HAQM FSx for Lustre file systems automatically copy objects from HAQM S3 to the file system when the objects are accessed for the first time. The same HAQM FSx for Lustre file system can also be used for iterative HAQM SageMaker training jobs running on the same datasets, preventing repeated downloading of common objects. With support for smaller file systems, HAQM FSx for Lustre can now be used for even more machine learning workloads.
Smaller HAQM FSx for Lustre file systems are available now in the following AWS Regions: US East (N. Virginia), US East (Ohio), US West (Oregon), US West (N. California), EU (Ireland), EU (Frankfurt), and Asia Pacific (Tokyo). For details on region availability please check the AWS region table.
Visit the documentation for more information and read the blog post for how to use the feature.