AWS Step Functions adds HAQM EMR service integration

Posted On: Nov 19, 2019

AWS Step Functions is now integrated with HAQM EMR, making it faster to build and easier to monitor EMR big data processing workflows.

AWS Step Functions allows you to build resilient workflows using AWS services such as HAQM EMR, HAQM SageMaker, and AWS Lambda. HAQM EMR is the industry leading cloud-native big data platform, allowing teams to process vast amounts of data quickly, and cost-effectively at scale. With Step Functions and HAQM EMR, you can orchestrate big data workflows while writing minimal additional code.  

With HAQM EMR and AWS Step Functions, you can now create efficient data processing workflows that order HAQM EMR steps, manage dependencies and run work in parallel. You can proactively scale a cluster up and down as part of an ETL workflow, right-sizing the cluster for the task at hand. You can also improve the resilience of your data processing workflows by choosing how exceptions are handled, retrying failed jobs and alerting users to failures.

By using the HAQM EMR service integration with the AWS Step Functions Data Science SDK you can build end-to-end data science workflows. The EMR service integration is available in all regions where both AWS Step Functions and HAQM EMR are available. For a complete list of regions and service offerings, see AWS Regions.

To get started, review the documentation and deploy a one-click sample project that demonstrates how to build a data processing workflow with HAQM EMR, then start building your first data processing workflow.

To learn more: