HAQM EMR on HAQM EKS

Why EMR on EKS?

HAQM EMR on HAQM EKS enables you to submit Apache Spark jobs on demand on HAQM Elastic Kubernetes Service (EKS) without provisioning clusters. With EMR on EKS, you can consolidate analytical workloads with your other Kubernetes-based applications on the same HAQM EKS cluster to improve resource utilization and simplify infrastructure management.

Until now, you had to choose between using EMR to manage Apache Spark on EC2 or self-managing Apache Spark on HAQM EKS. When you use EMR on EC2, the EC2 instances are dedicated to EMR. When you self-manage Apache Spark on EKS, you need to manually install, manage, and optimize Apache Spark to run on Kubernetes.

With HAQM EMR on HAQM EKS, you can share compute and memory resources across all of your applications and use a single set of Kubernetes tools to centrally monitor and manage your infrastructure. You can also use a single EKS cluster to run applications that require different Apache Spark versions and configurations, and take advantage of automated provisioning, scaling, faster runtimes, and development and debugging tools that EMR provides.

Benefits

You get the same EMR benefits for Apache Spark on EKS that you get on EC2 today. This includes fully managed versions of Apache Spark 2.4 and 3.0, automatic provisioning, scaling, performance optimized runtime, and tools like EMR Studiofor authoring jobs and an Apache Spark UI for debugging.
With EMR on EKS, your compute resources can be shared between your Apache Spark applications and your other Kubernetes applications. Resources are allocated and removed on demand to eliminate over-provisioning or under-utilization of these resources, enabling you to lower costs as you only pay for the resources you use.
By running analytics applications on EKS, you can reuse existing EC2 instances in your shared Kubernetes cluster and avoid the startup time of creating a new cluster of EC2 instances dedicated for analytics. You can also get 3x faster performance running performance optimized Spark with EMR on EKS compared to standard Apache Spark on EKS.

Use cases

With EMR on EKS, you can automate the provisioning, management, and scaling of Apache Spark, and use a single set of tools to centrally manage and monitor your infrastructure.

Run multiple EMR workloads that require different frameworks, versions, and configurations on the same EKS cluster as your other application workloads.

EMR on EKS provides a managed experience for developing, troubleshooting, and optimizing your analytics. You can deploy configurations and start jobs in seconds to test new EMR versions on the same EKS cluster without allocating dedicated resources.