A guide to data, analytics, and machine learning (ML) tools for AWS

Data residency in HAQM S3

• HAQM S3 allows for region bounded buckets, which ensure all data stored in that bucket will reside in a specific geographical location. The cool thing about using HAQM S3 as a landing location for the bulk of your data is its flexibility towards transforming and pushing that data out to other systems. You can use HAQM S3 as the landing zone while leveraging data residency capabilities of other service such as HAQM Relational Database Service (HAQM RDS) and HAQM Elastic Compute Cloud (HAQM EC2), which lie at the core of most applications today, including those running in container orchestrated environments. If you add AWS Local Zones to the mix, you have even more control over the location of your data and its various projections.

Security and Compliance Enhanced Delta Lake with Databricks

• With Databricks you can build a Delta Lake on top of your S3 stored data, and using the Enhanced Security and Compliance Add On you get improved security and controls that help you guarantee your compliance needs are met. This Add On includes hardened host OS, inter-node encryption, and enforced updates, together with compliance security profiles available for HIPAA, IRAP, PCI-DSS, and FedRAMP High and Moderate out of the box.

HAQM Macie for automated sensitive data discovery on S3

• Having your data on S3 also allows you to easily enable HAQM Macie, a service that will continually evaluate, discover, and scan your data on HAQM S3, and gives you a very quick path to automating remediation using HAQM EventBridge and AWS Security Hub.

Start by creating your S3 Buckets in the appropriate regions and setting up your Databricks Workspace to use the buckets with the desired data residency and enabling the Compliance Security Profile. Then configure HAQM Macie to handle automated discovery of sensitive data in the applicable S3 buckets.
 

HAQM FSx for ONTAP

• Fast performance, regional block storage for direct access by your compute nodes running Linux, Windows, and MacOS can be achieved using HAQM FSx. With HAQM FSx you can share file storage across HAQM EC2 instances, HAQM EKS nodes and even AWS Lambda functions, with very low latency and incredibly fast performance. HAQM FSx provides out-of-the-box support for multi-Availability Zone (AZ) deployment.
  

NetApp BlueXP

• With HAQM FSx for NetApp ONTAP and NetApp BlueXP, you get access to effortless cross region replication and data transport capabilities. Backup and cross-region disaster recovery are also readily available, which allows you to centrally manage your entire data estate across AWS regions, including automating through the criteria that you choose to apply to data being moved around or recovered in case of disaster using rules and policies.

You can start by creating your HAQM FSx ONTAP file systems directly in the AWS Console, which will allow you to discover them using NetApp BlueXP or directly create the environment from within BlueXP directly. In both scenarios you will need to grant the correct IAM permissions for the AWS resources to be accessible and managed through BlueXP.

From here, you can configure your high availability architecture, automatic capacity and throughput handling as well as all the other powerful capabilities available with NetApp BlueXP working with FSx volumes.
 

Snowflake

• Providing instantaneous capacity to handle any scale and making features available such as data stages and external tables, Snowflake provides out of the box capabilities to solve for all the challenges of working with data at scale we’ve discussed. Snowflake also provides an intuitive and familiar interface by supporting standard ANSI SQL to perform most operations. Additionally, Snowflake enables multi-region architectures off the shelf, by supporting multi-region deployment, data sharing and replication.

HAQM S3

• S3 Buckets can be directly used within Snowflake for data staging, which allows to load and unload data into your Snowflake deployment without having to build and maintain any complex data pipeline. You can also directly integrate buckets as external sources which enable you to query data stored in S3 without having to import it into the Snowflake data catalog at all.

Start by creating and loading data in your S3 buckets. Once you have your buckets created, you can use Snowflake storage integrations to allow it to read and write data from them. Storage integrations allow for authentication and authorization using AWS native IAM, and without using less secure secret keys and access tokens.
 

Confluent Kafka

• Offers incredibly powerful and unique capabilities built on top of the battle-proven Apache Kafka. Addressing the challenges outlined above, Confluent offers automated cluster scaling, dramatically reducing the complexity of handling data streams at scale, as well as features towards automated data deduplication. Relying on Confluent Kafka you get various message delivery, receipt, and ordering guarantees out of the box, which gives you a wide array of solutions when building your cloud-native architecture. To handle challenges around scale, Confluent Kafka offers infinite storage and infinite retention capabilities, which makes storage concerns a thing of the past.
  

HAQM EventBridge

• Confluent Kafka has native integrations with HAQM EventBridge, which allows you to send messages from one or more Kafka topics, including customized mapping and other handy capabilities to a specified event bus. HAQM EventBridge can handle infinite load with fully serverless scaling capabilities, and offers native integrations to most of AWS services, as well as custom capabilities to integrate with third-party services.
  

Start by creating a Confluent Kafka cluster, and configuring all your data sources to stream data to it, using Kafka client libraries. Once your data is flowing into Confluent, use the EventBridge Sink connector to send your records to one or more EventBridge buses, which will allow you to configure any downstream processing or handling using the huge array of AWS services supported natively or through SDK by EventBridge.
 

Matillion Data Productivity Cloud

• Building data pipelines with no code is one of the most powerful capabilities of Matillion, providing access to move and work with data to more diverse users, as well as reducing the effort in maintaining those pipelines over time. Matillion is powered by PipelineOS, which provides a cloud-native approach to scaling the underling infrastructure that runs the pipelines by using stateless agents capable of scaling to work in parallel and process massive amounts of data in a short period of time. Matillion also offers native and seamless integration with HAQM Redshift, dramatically simplifying storing processed and optimized data in a flexible and highly scalable repository.

HAQM Redshift

• Offering both configurable as well as machine learning driven fully automated performance tuning, HAQM Redshift serves as an ideal candidate to store voluminous data generated by your data pipelines. The capacity of HAQM Redshift to scale up to meet demands can easily adjust to the requirements of a dynamic and elastic range of data pipelines and serves as a flexible landing location where you can analyze, visualize and, if necessary, project that data to other target systems.
  
  Start by creating your Redshift cluster in AWS, and create a user for Matillion to have the necessary privileges to create databases and manage their data. Also, get the Redshift endpoint and port, which you will need when creating your Matillion pipelines.

Now you can start using Matillion’s wide range of capabilities to consume, extract or transform data from many different sources, including RDS data querying, which automatically stages data to S3, or any of the other data transfer sources including HDFS, SFTP, and many external third party services.

Astro by Astronomer

• As an enterprise-grade platform built on Apache Airflow®, Astro by Astronomer provides a unified solution for orchestrating, monitoring, and scaling data workflows across diverse cloud environments. It tackles the difficulties of maintaining reliability and efficiency in data operations by offering elastic scalability, high availability, and advanced security features. You can work with Astro using the Astro UI, Astro CLI or Astro API, to accelerate pipeline development and deployment capabilities (1500+ connectors) to ensure seamless connectivity across various data sources and services.

Start by signing up for Astro by Astronomer in AWS Marketplace. Create your first Deployment using either a pre-built template or your own custom project. From there, you can immediately begin developing, deploying, and running your data pipelines in a production-ready Airflow environment integrated with your AWS data storage services.

 

Grafana

• With over 26 different ways to present data, Grafana provides a powerful framework for business intelligence engineers, data scientists, and other engineers working to extract and share insights on data to build intuitive, simple yet comprehensive dashboards, while allowing for integration of many data sources and providing a common interface to query and inspect the data as the dashboard is being built. Grafana Cloud also provides enterprise-level capabilities, such as SSO and IAM integrations, providing robust ways to build security and access controls between your dashboard consumers and data.

AWS Identity and Access Management

• With AWS IAM you get access to incredibly granular controls and multiple ways to group, assign and managed those permissions to individuals as well as groups of users. These access controls interface directly with all AWS services and can be used to control access to dashboards and specific components within them.
  

Once you have subscribed to your Grafana Cloud account, simply use Grafana Data Sources to connect and consume data from your AWS resources, including HAQM CloudWatch, HAQM RDS. HAQM ElasticSearch Services and many others. Then use any of the built in Grafana Dashboards to visualize the health and status of each service, or build your own using Grafana’s query, exploration, visualization and alerting capabilities.
 

Neo4j AuraDB Professional

• Neo4j offers a Neo4j AuraDB, a fully managed graph database platform, that enables developers and data scientists to build scalable, AI-driven applications and perform complex data analysis. It allows for rapid development of intelligent, context-driven applications with real-time insights and built-in developer tools. For data scientists, Neo4j AuraDB provides a comprehensive data science as a service solution, integrating graph database capabilities with machine learning workflows. It offers a library of 65+ pre-tuned algorithms, data preparation techniques, and graph-native ML, enabling the analysis of relationships in big data and the creation of predictive models at scale.

• Neo4j Graph Database enables developers and data scientists to uncover hidden patterns and relationships in large datasets, addressing data quality issues through robust preparation and governance processes. Neo4j's unique property graph model facilitates real-time analytics on highly connected data, empowering businesses to make timely decisions in dynamic environments. For those seeking a cloud-native solution, Neo4j AuraDB offers the same capabilities in a fully managed format, allowing for rapid development of intelligent, context-driven applications with built-in developer tools.

To get started with Neo4j, sign up for Neo4j AuraDB Professional Pay-as-you-go in AWS Marketplace. Next, model your data as a graph and import your existing data using Neo4j's import tools. Then, start querying your graph using Cypher in the Neo4j Browser or your preferred development environment. Finally, integrate Neo4j with your chosen programming language to build graph-powered applications, leveraging Neo4j's driver libraries and APIs for seamless connectivity and data manipulation.

 

Databricks Data Intelligence Platform

• Databricks offers a strong set of capabilities in their MLFlow offering that manage the entire lifecycle of machine learning development workflows, from data preparation to training and notebook development to model deployment across a varied range of integrated model runtime environments.
  
  The Databricks Unity Catalog feature allows for storage of ML models, while enabling access controls, auditing, model discovery, and many other powerful capabilities when working with evolving versions of machine- learning projects.

HAQM SageMaker

• Databricks offers direct integration with HAQM SageMaker, allowing for direct and seamless deployment of models developed and managed using Databricks set of capabilities, while taking advantage of the powerful features built into HAQM SageMaker, including customizable scalability, API endpoint deployment and lifecycle management, and many other features required to put models into production environments (for example, tight IAM controls and direct access to data stored and managed within other AWS services).

You can use Databricks for the development and exploration of your model, and SageMaker to deploy it and make it available to users in a production ready capacity. Start by using Databricks by creating a workspace to work on the development of your model. Once you have a model ready to deploy, use the MLFlow API to deploy it to SageMaker, it is as simple as that.
 

MongoDB Atlas

• Developers are very familiar with MongoDB as a data repository for JSON documents, arguably one of the most common NoSQL data formats in use today, which also means a lot of data is already stored in these databases. MongoDB can be directly integrated as an HAQM Bedrock knowledge base , and it can serve both as document storage as well as storage for all the corresponding vector embeddings that will be used by the RAG implementation on top of your foundational LLM of choice. MongoDB provides an intuitive and familiar interface for developers to work with document and vector data, while enabling them to deploy RAG-based solutions with little effort by leveraging HAQM Bedrock’s fully managed and serverless access to foundational LLMs.

HAQM Bedrock

• HAQM Bedrock is a fully managed serverless offering that enables seamless access to a wide variety of high-performing foundational models. It also offers powerful and intuitive integration capabilities using HAQM Bedrock Agents and HAQM Bedrock Knowledge Bases to enrich and extend the context available to LLMs when generating responses to users.

Start by creating your MongoDB Atlas cluster and storing the credentials to access it in AWS Secrets Manager. Store the data you want to use for your Knowledge base in an S3 bucket, this can be anything from PDF files to other documents. Now, create an Atlas Vector Search Index in your Atlas Cluster. Now you are ready to configure your Knowledge Base in HAQM Bedrock, adding the Atlas vector store you created and the S3 bucket holding your data in the Knowledge Base configuration. Finally create and Agent that uses this knowledge base and you’ll be ready to use this for as RAG source for your LLM.

Datasaur LLM Labs and NLP Labeling

• Datasaur LLM Labs leverages HAQM SageMaker JumpStart to provide access to 80+ popular foundational models. It features a sandbox environment for LLM experimentation, allowing users to customize prompt templates, adjust hyperparameters, and evaluate multiple model responses. Datasaur's NLP Labeling solution provides a comprehensive interface for text, document, and audio data annotation. It supports various NLP tasks from entity recognition to advanced techniques like coreference resolution and dependency parsing. The platform offers automation features such as integration with popular NLP libraries, weakly supervised learning capabilities, and connections to AWS services like HAQM S3 and HAQM Textract. Both solutions work synergistically, allowing users to leverage LLMs for initial data labeling, incorporate reinforcement learning with human feedback (RLHF), and even set up LLM distillation workflows for more efficient, task-specific models.

HAQM Textract

• HAQM Textract is a machine learning service that automatically extracts text, handwriting, layout elements, and data from scanned documents. It goes beyond simple optical character recognition (OCR) to identify, understand, and extract specific data from various document types, enabling businesses to quickly automate document processing and extract valuable information from forms, invoices, and receipts.

Start by signing up for Datasaur LLM Labs in AWS Marketplace. Next, create your first project by importing your prepared dataset into the LLM Labs interface. Then, begin fine-tuning your language models using LLM Labs' comprehensive features, including prompt engineering tools and team collaboration capabilities. Finally, deploy your fine-tuned models through the platform's deployment options and integrate them into your applications using LLM Labs' APIs and SDKs.

 

Boomi AtomSphere Platform

• Boomi simplifies the process of creating and managing integrations across diverse systems with its low-code development environment and extensive library of pre-built connectors. The platform's intuitive drag-and-drop interface enables both technical and non-technical users to build complex integrations quickly. Boomi's Master Data Hub ensures data consistency and integrity across integrated systems, while its API Management capabilities allow for secure and scalable API creation and governance. The platform's cloud-native architecture provides the flexibility to run integrations anywhere - in the cloud, on-premises, or at the edge - meeting various deployment needs and compliance requirements.

HAQM service integration

• Boomi seamlessly integrates with various AWS services, including HAQM S3, HAQM RDS, and HAQM Redshift, allowing for efficient data movement and transformation between AWS and other environments. This integration capability enables organizations to leverage the power of AWS alongside their existing on-premises or other cloud-based systems, facilitating hybrid and multi-cloud architectures.

Sign up for Boomi Pay-As-You-Go in AWS Marketplace to access a flexible, low-cost integration platform. Quickly connect your applications, data, and APIs using pre-built connectors, visual workflows, and advanced features like Master Data Hub and API management. Scale up or down as needed with no long-term commitments, and leverage Boomi's extensive support resources to drive your digital transformation initiatives.