Simplify AWS Cost Data Analysis with HAQM Q in QuickSight

Overview

Financial teams across industries are seeking agile solutions that provide rapid insights, enabling timely and informed decisions to manage costs effectively. AWS makes it easy to build and scale generative AI by adding capabilities to its suite of analytics tools. HAQM Q in QuickSight offers an in-depth analysis with additional granularity and dimensionality (e.g., hourly and resource-level data beyond 14 days of history etc.) more control, custom visualizations for your data, and the ability to integrate with other data sources. HAQM Q in QuickSight enables intuitive Q&A sessions, providing quick, actionable insights, comprehensive data storytelling, and advanced visualization features that are ideal for detailed financial analysis. For a high-level out of the box solution, you can use cost analysis capability in HAQM Q Developer.

In this post, we’ll demonstrate how users can effortlessly extract insights from their datasets and dashboards on-the-go simply by asking questions in natural language to their AWS billing and cost data generated by AWS Data Exports. Through four practical scenarios, we’ll showcase the power and versatility of this feature. We’ll also provide a range of sample prompts at the end of the blogpost, allowing you to experiment with Q in QuickSight using your own datasets. Whether you’re a data expert or a business user, this blogpost will help you harness the full potential of conversational AI for data analysis.

Transforming Questions into Insights: Real-World Scenarios

Following are some examples of scenarios we are going to explore in this blog post that you can ask Q in QuickSight to get insights from your cost data.

The What question?

• Scenario 1 – What usage types are driving highest HAQM service costs? – Exploring Spend by Service: Imagine you’re analyzing your AWS spend by service and discover a service with the highest expenditure. This what questions drills down further into the types of usage driving these costs within that service.
• Scenario 2 – What are the EC2 instance details based on different purchasing options? – Analyzing best combination of instances: Gauge the EC2 instance spread across different purchase options to optimize costs based on workload needs and leverage different pricing models for flexibility and budgeting

The Why question?

• Scenario 3 – Why did my spend drop in November? – Investigating Spend Fluctuations: You are reviewing a dashboard displaying total spend by month for a specific year. You notice a drop in spending from October to November and want to understand the reason behind it.

The How question?

• Scenario 4 – What trends or patterns can we identify in our AWS usage and spending over time to maximize savings opportunity? – Crafting Executive Summaries and Data Stories: Create compelling executive summaries and data narratives.

Pre-requisite:

This solution utilizes HAQM Q in QuickSight to transform the way AWS customers interact with and analyze their billing, cost, and usage data generated by AWS Data Exports. Once set up, this solution empowers users to ask detailed questions about their AWS spending, usage trends, and cost-saving opportunities using natural language.

To run this solution, you must complete the following steps:

1. Sign up for QuickSight Enterprise account if you don’t have an existing QuickSight account.
2. Create a cost and usage dashboard from Data Exports integrated in QuickSight. You can follow video tutorial or documentation for detailed steps.

Security and Access Control in HAQM QuickSight:

Securing your data and controlling access in HAQM QuickSight is crucial for protecting your organization’s sensitive information. By following the principle of least privilege and leveraging QuickSight’s built-in security features, you can minimize access risks and ensure compliance with security best practices. QuickSight offers predefined user roles to effectively manage user permissions. To implement a secure QuickSight environment:

1. Assign Reader role to users who only need to consume dashboards, reports, data stories and ask questions of data with HAQM Q without modifying data. This is the best “out-of-the-box” option, ensuring they don’t receive unnecessary permissions.
2. Carefully assign Author role only to users who require data preparation, dashboard creation and content sharing capabilities.
3. Strictly limit Admin access to users who require administrative capabilities for managing user groups, permissions, billing and services.

To further secure data access, consider implementing row-level security (RLS), column-level security (CLS) which restricts access to specific data based on user roles or group membership. Additionally, ensure your QuickSight account is configured with the required IAM policies to enforce granular access controls and prevent unauthorized data exposure.

Explore Q in QuickSight with Generative AI capabilities:

1. After creating cost and usage dashboard following the instructions from the pre-requisite video tutorial link, login to QuickSight Console and go to Topics. Select the topic you’ve created from the exported dashboard, review the topic settings and data fields.
2. Click “Start Review” (Fig 1). It is recommended to follow along Steps to review your topic prompt to learn better about your data fields (Fig 2) and Q functionalities related to it.

Figure 1: Cost and Usage topic- Summary

Figure 2: Cost and Usage topic- data fields and values

We want to make this topic more natural language-friendly and easy to interpret. It helps Q interpret your data and better answer your questions, so provide as much information about your datasets and their associated fields as possible. To do this, you can modify/update the topic settings by making the following changes –

1. Enable “include” field for the attributes you want to include in your dataset scope for Q. Exclude any unused fields.
2. Add “Synonyms” to your fields for Q to easily interpret these terms and map them to the correct fields. For instance – Field “Billing Period” can have synonyms such as “Invoice Period”, “Payment Period”, etc.
3. You can add synonyms to field values as well. For instance, you can change field value from “amazonrds” to “HAQM RDS” (HAQM Relational Database Service) for Field “Service” through Configure filed value synonyms option.
4. Add a Named Entity to group multiple columns of data that collectively represent a business concept without stating each column explicitly. For e.g. columns HAQM EC2 instance family, instance type, region, processor platform can all be addressed as named entity “EC2 instance details”. Q prioritizes higher ranked named entities when answering questions.

HAQM QuickSight Q topics documentation contains additional settings (optional) that you can add to enhance Q’s responses, making them even more intuitive and natural. These settings are designed to make HAQM QuickSight Q topics more natural-language-friendly and improve the overall user experience.

Once you’ve updated the topic settings per your requirements, you can start asking questions to Q from clicking the Q option provided at the top of the console as shown below –

Figure 3: Q in QuickSight Console

Upon clicking Ask a question about Cost and Usage Topic (Figure 3), a dialogue box will open that will present sample questions, available date range etc. You can get started with either the sample questions or ask your own.

FinOps Scenarios for Q:

In this section, we are going to explore in detail the scenarios we discussed briefly in the introduction section of the blog post above.

Scenario 1 – What usage types are driving highest HAQM service costs? Suppose you are exploring Spend by Service and come across a service (For this instance – RDS from the sample dataset, refer Fig 4) with highest spend.

Figure 4: Question – Spend by Service

Now you want to drill down further for the kind of usage within that service, you can ask Q – What usage types are driving highest HAQM RDS costs?  And you will be presented by something similar to this –

Figure 5: Question – What usage types are driving highest HAQM RDS costs?

For this first scenario we have highlighted fields/options in the visual (Fig 5). Please review the highlighted fields/options above to better understand how Q has generated the output of the question you have asked moving forward for the rest of the scenarios.

1. Visual options – change the visual (e.g. from bar chart to pie chart), View insights/forecasting, add to pinboard (to save for future exports), view explanation (step by step summary of how the visual is generated in terms of fields and aggregations)
2. Interpreted as – How your natural language question has been interpreted quantitively by Q
3. Mark as Verified – If you are satisfied with the explanation and accuracy, you can mark question as verified so readers/users can use it in future
4. Question Suggestion – Provides suggestion around the question you asked based on the data fields
5. Feedback – you can leave feedback around questions for topic owners which can be reviewed from Q topic User Activity page.

Scenario 2 – What are the EC2 instance details based on different purchasing options? You want to gauge the EC2 instance spread across different purchase options to optimize costs based on workload needs and to leverage different pricing models for flexibility and budgeting in cloud deployments. You can ask Q – Show EC2 instance details based on the different purchasing options?

Figure 6: Question – Show EC2 instance details based on the different purchasing options?

This will provide you with a list of EC2 instance details and purchasing options (Fig 6) to understand coverage and potential options for cost optimizing your EC2 instances across your regions of operations.

Scenario 3 – Why did my spend drop in November? You came across a dashboard chart for total spend by month for 2023 and notice spend in November 2023 dropped compared to October 2023 and you want to know why. Then you can ask Q – Why did my spend drop in November 2023? And you will be presented by something similar to this –

Figure 7: Question – Why did my spend drop in November 2023?

This will result in a list of key drivers and their contribution (Fig 7) instantly for drop in the spend which can help in faster analysis and in turn decision making.

Scenario 4 – Compelling Executive summaries and data stories – What trends or patterns can we identify in our AWS usage and spending over time to maximize savings opportunity?

 Data Story is a generative BI feature in QuickSight that lets users create narratives with data insights, eliminating the need to manually copy visuals. It integrates dashboards and descriptive narratives, addressing business problems and using LLMs for context. Users can customize visuals, add images, and format text. The data story can be shared according to data governance rules and is downloadable in formats like PDF and PPT.

To create a data story, go to QuickSight Dashboard and select CURQuicksightExport dashboard. Click BUILD option from top right and hit Data Story (Fig 8). It will ask you to pass the prompt and select visuals from the dashboard to generate a story. You can pass prompt for Story – What trends or patterns can we identify in our AWS usage and spending over time to maximize savings opportunity? Also select ADD VISUALS option to add visuals from your dashboard to support your storyline.

Figure 8: Data Story with prompt

  Figure 9: Data story output

Figure 10: Data story output

Figure 11: Data story output

Figure 12: Data story output

This will generate an editable data story (Figure 9-12) based on your dashboard visuals in either a document or presentation slide format with introduction, overview, facts and summary.

 Prompts for you to try:

1. What are the most popular regions by spend last month?
2. Show EC2 instance details covered by SavingsPlan?
3. Are there any product families that are more commonly used or contribute more to costs?
4. What is the average spend per service broken down by region?
5. Are there any seasonal or periodic trends in the cost that can be identified based on billing periods?
6. Forecast future unblended costs based on historical data and trends?
7. Top 10 services with increased usage and cost in the last 6 months?
8. Data Story prompt – How can we optimize our AWS spending based on the insights derived from the billing data

You could ask field specific questions as well. These questions aim to delve deeper into specific aspects of the data provided, allowing for a more comprehensive analysis of the dashboard metrics, such as:

9. For the “Instance Type” field:

• What is the distribution of instance types across different services and regions?
• Are there any instance types that are more commonly used or preferred for EC2?

Cost of running this solution:

This solution involves storing cost exports in HAQM Simple Storage Service (S3) at S3 standard rates. There is a $250/month per account HAQM Q enablement fee in HAQM QuickSight.

HAQM QuickSight pricing varies by user role:

Author Pro costs $50 per user/month and includes all Author capabilities plus generative BI with HAQM Q
Reader costs $3 per user/month and includes interactive dashboards and access to HAQM Q&A and shared generative data stories.
Author Pro role is required to build these dashboards. Reader Pro role would be required by business users to review insights. To learn more about costs for Q in QuickSight, see HAQM QuickSight pricing.

Conclusion

You now know how to integrate your cost data exports to HAQM Q in QuickSight with Generative AI capabilities for impromptu QnA experience, compelling executive summaries and data stories to expedite the efficient financial decision making. To learn further more about this, refer to HAQM Q in QuickSight and AWS Cost and Management data exports