Tag: HAQM EMR

AWS IoT makes it easy to integrate and control your devices from other AWS services for even more powerful IoT applications. In particular, IoT provides tight integration with AWS Lambda, HAQM Kinesis, HAQM S3, HAQM Machine Learning, HAQM DynamoDB, HAQM CloudWatch, and HAQM OpenSearch Service.

In this post, I show you how to gain valuable insight into your network by using HAQM EMR and HAQM VPC Flow Logs. The walkthrough implements a pattern often found in network equipment called ‘Top Talkers’, an ordered list of the heaviest network users, but the model can also be used for many other types of network analysis.

This post shows you how to build a custom application for EMR for Apache Bigtop-based releases 4.x and greater. EMR nodes are based on the HAQM Linux AMI, so I will deploy on RPM packages and use Elasticsearch as the example application.

Jonathan Fritz is a Senior Product Manager for HAQM EMR We are excited to launch HAQM EMR release 5.0 today, giving customers the latest versions of 16 supported open-source applications in the big data ecosystem, including new major versions of Spark and Hive. Almost exactly a year ago, we shipped release 4.0, which brought significant […]

In this blog post, you will learn how to install JobServer on EMR using a bootstrap action (BA) derived from the JobServer GitHub repository. Then we’ll run JobServer using a sample dataset.

This is a guest post by Takumi Sakamoto, a software engineer at SmartNews. SmartNews in their own words: “SmartNews is a machine learning-based news discovery app that delivers the very best stories on the Web for more than 18 million users worldwide.” Data processing is one of the key technologies for SmartNews. Every team’s workload […]

In this post, I discuss an alternate solution; namely, running separate CPU and GPU clusters, and driving the end-to-end modeling process from Apache Spark.

In this post, I will show you how to transfer data using Apache Sqoop, which is a tool designed to transfer data between Hadoop and relational databases. Support for Apache Sqoop is available in HAQM EMR releases 4.4.0 and later.

This post shows you how you can use Spark Streaming to process data coming from HAQM Kinesis streams, build some graphs using Zeppelin, and then store the Zeppelin notebook in HAQM S3.

HAQM EMR has added Apache Tez version 0.8.3 as a supported application in release 4.7.0. Tez is an extensible framework for building batch and interactive data processing applications on top of Hadoop YARN.