HAQM Detective – Rapid Security Investigation and Analysis

Almost five years ago, I blogged about a solution that automatically analyzes AWS CloudTrail data to generate alerts upon sensitive API usage. It was a simple and basic solution for security analysis and automation. But demanding AWS customers have multiple AWS accounts, collect data from multiple sources, and simple searches based on regular expressions are not enough to conduct in-depth analysis of suspected security-related events. Today, when a security issue is detected, such as compromised credentials or unauthorized access to a resource, security analysts cross-analyze several data logs to understand the root cause of the issue and its impact on the environment. In-depth analysis often requires scripting and ETL to connect the dots between data generated by multiple siloed systems. It requires skilled data engineers to answer basic questions such as “is this normal?”. Analysts use Security Information and Event Management (SIEM) tools, third-party libraries, and data visualization tools to validate, compare, and correlate data to reach their conclusions. To further complicate the matters, new AWS accounts and new applications are constantly introduced, forcing analysts to constantly reestablish baselines of normal behavior, and to understand new patterns of activities every time they evaluate a new security issue.

HAQM Detective is a fully managed service that empowers users to automate the heavy lifting involved in processing large quantities of AWS log data to determine the cause and impact of a security issue. Once enabled, Detective automatically begins distilling and organizing data from HAQM GuardDuty, AWS CloudTrail, and HAQM Virtual Private Cloud (HAQM VPC) Flow Logs into a graph model that summarizes the resource behaviors and interactions observed across your entire AWS environment.

At re:invent 2019, we announced a preview of HAQM Detective. Today, it is our pleasure to announce its availability for all AWS Customers.

HAQM Detective uses machine learning models to produce graphical representations of your account behavior and helps you to answer questions such as “is this an unusual API call for this role?” or “is this spike in traffic from this instance expected?”. You do not need to write code, to configure or to tune your own queries.

To get started with HAQM Detective, I open the AWS Management Console, I type “detective” in the search bar and I select HAQM Detective from the provided results to launch the service. I enable the service and I let the console guide me to configure “member” accounts to monitor and the “master” account in which to aggregate the data. After this one-time setup, HAQM Detective immediately starts analyzing AWS telemetry data and, within a few minutes, I have access to a set of visual interfaces that summarize my AWS resources and their associated behaviors such as logins, API calls, and network traffic. I search for a finding or resource from the HAQM Detective Search bar and, after a short while, I am able to visualize the baseline and current value for a set of metrics.

I select the resource type and ID and start to browse the various graphs.

I can also investigate a HAQM GuardDuty finding by using the native integrations within the GuardDuty and AWS Security Hub consoles. I click the “Investigate” link from any finding from HAQM GuardDuty and jump directly into a HAQM Detective console that provides related details, context, and guidance to investigate and to respond to the issue. In the example below, GuardDuty reports an unauthorized access that I decide to investigate:

HAQM Detective console opens:

I scroll down the page to check the graph of failed API calls. I click a bar in the graph to get the details, such as the IP addresses where the calls originated:

Once I know the source IP addresses, I click New behavior: AWS role and observe where these calls originated from to compare with the automatically discovered baseline.

HAQM Detective works across your AWS accounts, it is a multi-account solution that aggregates data and findings from up to 1000 AWS accounts into a single security-owned “master” account making it easy to view behavioral patterns and connections across your entire AWS environment.

There are no agents, sensors, or additional software to deploy in order to use the service. HAQM Detective retrieves, aggregates and analyzes data from HAQM GuardDuty, AWS CloudTrail and HAQM Virtual Private Cloud (HAQM VPC) Flow Logs. HAQM Detective collects existing logs directly from AWS without touching your infrastructure, thereby not causing any impact to cost or performance.

HAQM Detective can be administered via the AWS Management Console or via the HAQM Detective management APIs. The management APIs enable you to build HAQM Detective into your standard account registration, enablement, and deployment processes.

HAQM Detective is a regional service. I activate the service in every AWS Regions in which I want to analyze findings. All data are processed in the AWS Region where they are generated. HAQM Detective maintains data analytics and log summaries in the behavior graph for a 1-year rolling period from the date of log ingestion. This allows for visual analysis and deep dives over a large data set for a long period of time. When I disable the service, all data is expunged to ensure no data remains.

There are no additional charges or upfront commitments required to use HAQM Detective. We charge per GB of data ingested from AWS AWS CloudTrail, HAQM Virtual Private Cloud (HAQM VPC) Flow Logs, and HAQM GuardDuty findings. HAQM Detective offers a 30-day free trial. As usual, check the pricing page for the details.

HAQM Detective is available in these 14 AWS Regions : US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Pacific (Seoul), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Asia Pacific (Mumbai), Europe (Stockholm), Europe (Frankfurt), Europe (Ireland), Europe (London), Europe (Paris), and South America (São Paulo).

You can start to use it today.