HAQM Robotics Uses HAQM SageMaker and AWS Inferentia to Enable ML Inferencing at Scale

HAQM Robotics uses its software and machinery to automate the flow of inventory in HAQM fulfillment centers. There are three main physical components to the company’s system: mobile shelving units, robots, and employee workstations. The robots deliver mobile shelving units to stations, and employees either put inventory in (stowing) or take it out (picking). “Our existing stow-and-pick workflows can sometimes create a bottleneck for downstream processing,” says Eli Gallaudet, a senior software manager at HAQM Robotics. “In 2017, we kicked off an initiative to figure out how to make some of those workflows simpler.”

Looking to reduce time-consuming bin scanning, HAQM Robotics built the Intent Detection System, a deep-learning-based computer vision system trained on millions of video examples of stowing actions. The company wanted to train the system to automatically identify where associates place inventory items. Knowing it would need cloud compute to deploy the deep-learning models to HAQM fulfillment centers, HAQM Robotics turned to AWS. The team deployed its models to Docker containers, hosting them using HAQM Elastic Container Service (HAQM ECS), a fully managed container orchestration service.

Once the team had collected enough video examples of stowing actions, it experimented with applying model architectures to the large annotated video dataset. After several iterations, the team could begin letting the deployed models automate the process.

Although HAQM Robotics could tap into ample compute resources on AWS, the company still had to handle hosting itself. When AWS announced the release of HAQM SageMaker at AWS re:Invent 2017, HAQM Robotics quickly adopted it, avoiding the need to build a costly hosting solution of its own. HAQM Robotics was the first company to deploy to HAQM SageMaker on a large scale and remains one of the largest deployments as of January 2021.

At first the team primarily used HAQM SageMaker to host models. HAQM Robotics adapted its service usage as needed, initially using a hybrid architecture and running some algorithms on premises and some on the cloud. “We built a core set of functionalities that enabled us to deliver the Intent Detection System,” says Tim Stallman, a senior software manager at HAQM Robotics. “And then as HAQM SageMaker features came online, we slowly started adopting those.” For example, the team adopted HAQM SageMaker Experiments—a capability that enabled the team to organize, track, compare, and evaluate ML experiments and model versions.

HAQM Robotics also used HAQM SageMaker automatic scaling. “HAQM SageMaker doesn’t just manage the hosts we use for inferencing,” says Gallaudet. “It also automatically adds or removes hosts as needed to support the workload.” Because it doesn’t need to procure or manage its own fleet of over 500 GPUs, the company has saved close to 50 percent on its inferencing costs.

HAQM Robotics has seen considerable success. The company has used HAQM SageMaker to reduce time spent on management and to balance the ratio of scientists to software development engineers. HAQM SageMaker also enabled the system to scale horizontally during its rollout across the HAQM fulfillment network—and the team is confident that HAQM SageMaker can handle its peak inference demands.

This solution is backed by HAQM Elastic Compute Cloud (HAQM EC2), which provides secure, resizable compute capacity in the cloud and enables users to quickly migrate host types as newer host types become available. The HAQM Robotics team was able to reduce their inference costs by 20 percent by migrating from HAQM EC2 P2 Instances to HAQM EC2 G4 Instances. Now utilizing AWS Inferentia, the HAQM Robotics team is able to further reduce inference costs by 35 percent over G4 instances (over 50 percent reduction from P2 instances) and Inferentia has delivered 20 percent higher throughput, allowing them to scan more packages a day without requiring more resources. “Our system will use more than 1000 SageMaker hosts in 2022 and AWS Inferentia helps us to serve the rapidly growing traffic at higher throughput without re-training our ML models," says Pei Wang, a software engineer at HAQM Robotics.

The HAQM SageMaker–powered solution grew rapidly after its initial deployment. The HAQM Robotics team started implementing the solution on a small scale at a fulfillment center in Wisconsin and rapidly expanded to dozens more. As the solution grew, HAQM SageMaker quickly and seamlessly scaled alongside it. “We expect to almost double our volume in 2022,” says Gallaudet.

The team sees many other opportunities to experiment on AWS, including running its models on the edge using HAQM SageMaker Edge Manager, which efficiently manages and monitors ML models across fleets of smart devices. HAQM Robotics also expects to build models that can further automate package tracking and help automate package damage assessment.

By experimenting with cutting-edge technology, HAQM Robotics continues to increase efficiency in fulfillment centers and improve the HAQM customer experience. “Many of the techniques that we’ve learned and experiences we’ve had with the Intent Detection System have directly enabled us to move quickly on these projects,” says Stallman.