HAQM SageMaker Lakehouse now supports attribute-based access control

HAQM SageMaker Lakehouse now supports attribute-based access control (ABAC) with AWS Lake Formation, using AWS Identity and Access Management (IAM) principals and session tags to simplify data access, grant creation, and maintenance. With ABAC, you can manage business attributes associated with user identities and enable organizations to create dynamic access control policies that adapt to the specific context.

SageMaker Lakehouse is a unified, open, and secure data lakehouse that now supports ABAC to provide unified access to general purpose HAQM S3 buckets, HAQM S3 Tables, HAQM Redshift data warehouses, and data sources such as HAQM DynamoDB or PostgreSQL. You can then query, analyze, and join the data using Redshift, HAQM Athena, HAQM EMR, and AWS Glue. You can secure and centrally manage your data in the lakehouse by defining fine-grained permissions with Lake Formation that are consistently applied across all analytics and machine learning(ML) tools and engines. In addition to its support for role-based and tag-based access control, Lake Formation extends support to attribute-based access to simplify data access management for SageMaker Lakehouse, with the following benefits:

• Flexibility – ABAC policies are flexible and can be updated to meet changing business needs. Instead of creating new rigid roles, ABAC systems allow access rules to be modified by simply changing user or resource attributes.
• Efficiency – Managing a smaller number of roles and policies is more straightforward than managing a large number of roles, reducing administrative overhead.
• Scalability – ABAC systems are more scalable for larger enterprises because they can handle a large number of users and resources without requiring a large number of roles.

Attribute-based access control overview

Previously, within SageMaker Lakehouse, Lake Formation granted access to resources based on the identity of a requesting user. Our customers were requesting the capability to express the full complexity required for access control rules in organizations. ABAC allows for more flexible and nuanced access policies that can better reflect real-world needs. Organizations can now grant permissions on a resource based on user attribute and is context-driven. This allows administrators to grant permissions on a resource with conditions that specify user attribute keys and values. IAM principals with matching IAM or session tag key-value pairs will gain access to the resource.

Instead of creating a separate role for each team member’s access to a specific project, you can set up ABAC policies to grant access based on attributes like membership and user role, reducing the number of roles required. For instance, without ABAC, a company with an account manager role that covers five different geographical territories needs to create five different IAM roles and grant data access for only the specific territory for which the IAM role is meant. With ABAC, they can simply add those territory attributes as keys/values to the principal tag and provide data access grants based on those attributes. If the value of the attribute for a user changes, access to the dataset will automatically be invalidated.

With ABAC, you can use attributes such as department or country and use IAM or sessions tags to determine access to data, making it more straightforward to create and maintain data access grants. Administrators can define fine-grained access permissions with ABAC to limit access to databases, tables, rows, columns, or table cells.

In this post, we demonstrate how to get started with ABAC in SageMaker Lakehouse and use with various analytics services.

Solution overview

To illustrate the solution, we are going to consider a fictional company called Example Retail Corp. Example Retail’s leadership is interested in analyzing sales data in HAQM S3 to determine in-demand products, understand customer behavior, and identify trends, for better decision-making and increased profitability. The sales department sets up a team for sales analysis with the following data access requirements:

• All data analysts in the Sales department in the US get access to only sales-specific data in only US regions
• All BI analysts in the Sales department have full access to data in only US regions
• All scientists in the Sales department get access to only sales-specific data across all regions
• Anyone outside of Sales department have no access to sales data

For this post, we consider the database salesdb, which contains the store_sales table that has store sales details. The table store_sales has the following schema.

To demonstrate the product sales analysis use case, we will consider the following personas from the Example Retail Corp:

• Ava is a data administrator in Example Retail Corp who is responsible for supporting team members with specific data permission policies
• Alice is a data analyst who should be able to access sales specific US store data to perform product sales analysis
• Bob is a BI analyst who should be able to access all data from US store sales to generate reports
• Charlie is a data scientist who should be able to access sales specific across all regions to explore and find patterns for trend analysis

Ava decides to use SageMaker Lakehouse to unify data across various data sources while setting up fine-grained access control using ABAC. Alice is excited about this decision as she can now build daily reports using her expertise with Athena. Bob now knows that he can quickly build HAQM QuickSight dashboards with queries that are optimized using Redshift’s cost-based optimizer. Charlie, being an open source Apache Spark contributor, is excited that he can build Spark based processing with HAQM EMR to build ML forecasting models.

Ava defines the user attributes as static IAM tags that could also include attributes stored in the identity provider (IdP) or as session tags dynamically to represent the user metadata. These tags are assigned to IAM users or roles and can be used to define or restrict access to specific resources or data. For more details, refer to Tags for AWS Identity and Access Management resources and Pass session tags in AWS STS.

For this post, Ava assigns users with static IAM tags to represent the user attributes, including their department membership, Region assignment, and current role relationship. The following table summarizes the tags that represent user attributes and user assignment.

Ava then defines access control policies in Lake Formation that grant or restrict access to certain resources based on predefined criteria (user attributes defined using IAM tags) being satisfied. This allows for flexible and context-aware security policies where access privileges can be adjusted dynamically by modifying the user attribute assignment without changing the policy rules. The following table summarizes the policies in the Sales department.

The following diagram illustrates the solution architecture.

Implementing this solution consists of the following high-level steps. For Example Retail, Ava as a data Administrator performs these steps:

1. Define the user attributes and assign them to the principal.
2. Grant permission on the resources (database and table) to the principal based on user attributes.
3. Verify the permissions by querying the data using various analytics services.

Prerequisites

To follow the steps in this post, you must complete the following prerequisites:

1. AWS account with access to the following AWS services:
   • HAQM S3
   • AWS Lake Formation and AWS Glue Data Catalog
   • HAQM Redshift
   • HAQM Athena
   • HAQM EMR
   • AWS Identity and Access Management (IAM)

2. Set up an admin user for Ava. For instructions, see Create a user with administrative access.
3. Setup S3 bucket for uploading script.
4. Set up a data lake admin. For instructions, see Create a data lake administrator.
5. Create IAM user named Alice and attach permissions for Athena access. For instructions, refer to Data analyst permissions.
6. Create IAM user Bob and attach permissions for Redshift access.
7. Create IAM user Charlie and attach permissions for EMR Serverless access.
8. Create job runtime role: scientist_role and that will be used by Charlie. For instruction refer to: Job runtime roles for HAQM EMR Serverless
9. Setup EMR Serverless application with Lake Formation enabled. For instruction refer to: Using EMR Serverless with AWS Lake Formation for fine-grained access control
10. Have an existing AWS Glue database or table and HAQM Simple Storage Service (HAQM) S3 bucket that holds the table data. For this post, we use salesdb as our database, store_sales as our table, and data is stored in an S3 bucket.
    • Register the S3 bucket with Lake Formation. For instructions, refer to Registering an HAQM S3 location.
    • Revoke IAMAllowedPrincipals group permission on both the database and table to enforce Lake Formation permission for access. For instructions, refer to Revoking permission using the Lake Formation console.

Define attributes for the IAM principals Alice, Bob, Charlie

Ava completes the following steps to define the attributes for the IAM principal:

1. Log in as an admin user and navigate to the IAM console.
2. Choose Users under Access management in the navigation pane and search for the user Alice.
3. Choose the user and choose the Tags tab.
4. Choose Add new tag and provide the following key pairs:
   • Key: Department and value: sales
   • Key: Region and value: US
   • Key: Role and value: Analyst
5. Choose Save changes.
   
6. Repeat the process for the user Bob and provide the following key pairs:
   • Key: Department and value: sales
   • Key: Region and value: US
   • Key: Role and value: BIAnalyst
7. Repeat the process for the user Charlie and IAM role scientist_role and provide the following key pairs:
   • Key: Department and value: sales
   • Key: Region and value: ALL
   • Key: Role and value: Scientist

Grant permissions to Alice, Bob, Charlie using ABAC

Ava now grants database and table permissions to users with ABAC.

Grant database permissions

Complete the following steps:

1. Ava logs in as data lake admin and navigate to the Lake Formation console.
2. In the navigation pane, under Permissions, choose Data lake permissions.
3. Choose Grant.
4. On the Grant permissions page, choose Principals by attribute.
5. Specify the following attributes:
   • Key: Department  and value: sales
   • Key: Role and value: Analyst,Scientist
6. Review the resulting policy expression.
7. For Permission scope, select This account.
   
8. Next, choose the catalog resources to grant access:
   • For Catalogs, enter the account ID.
   • For Databases, enter salesdb.
9. For Database permissions, select Describe.
10. Choose Grant.

Ava now verifies the database permission by navigating to the Databases tab under the Data Catalog and searching for salesdb. Select salesdb and choose View under Actions.

Grant table permissions to Alice

Complete the following steps to create a data filter to view sales specific columns in store_sales records whose country=US:

1. On the Lake Formation console, choose Data filters under Data Catalog in the navigation pane.
2. Choose Create new filter.
3. Provide the data filter name as us_sales_salesonlydata.
4. For Target catalog, enter the account ID.
5. For Target database, choose salesdb.
6. For Target table, choose store_sales.
7. For column-level access, choose Include columns: store_id, item_code, transaction_date, product_name, country, sales_price, and quantity.
8. For Row-level access, choose Filter rows and enter the row filter country='US'.
9. Choose Create data filter.
   

10. On the Grant permissions page, choose Principals by attribute.
11. Specify the attributes:
    • Key: Department and value: sales
    • Key: Role as value: Analyst
    • Key: Region and value: US
12. Review the resulting policy expression.
13. For Permission scope, select This account.
    
14. Choose the catalog resources to grant access:
    • Catalogs: Account ID
    • Databases: salesdb
    • Table: store_sales
    • Data filters: us_sales
15. For Data filter permissions, select Select.
16. Choose Grant.

Grant table permissions to Bob

Complete the following steps to create a data filter to view only store_sales records whose country=US:

1. On the Lake Formation console, choose Data filters under Data Catalog in the navigation pane.
2. Choose Create new filter.
3. Provide the data filter name as us_sales.
4. For Target catalog, enter the account ID.
5. For Target database, choose salesdb.
6. For Target table, choose store_sales.
7. Leave Column-level access as Access to all columns.
8. For Row-level access, enter the row filter country='US'.
9. Choose Create data filter.
   

Complete the following steps to grant table permissions to Bob:

1. On the Grant permissions page, choose Principals by attribute.
2. Specify the attributes:
   • Key: Department and value: sales
   • Key: Role as value: BIAnalyst
   • Key: Region and value: US
3. Review the resulting policy expression.
4. For Permission scope, select This account.
5. Choose the catalog resources to grant access:
   • Catalogs: Account ID
   • Databases: salesdb
   • Table: store_sales
6. For Data filter permissions, select Select.
7. Choose Grant.

Grant table permissions to Charlie

Complete the following steps to grant table permissions to Charlie:

1. On the Grant permissions page, choose Principals by attribute.
2. Specify the attributes:
   1. Key: Department and value: sales
   2. Key: Role as value: Scientist
   3. Key: Region and value: ALL
3. Review the resulting policy expression.
4. For Permission scope, select This account
5. Choose the catalog resources to grant access:
   1. Catalogs: Account ID
   2. Databases: salesdb
   3. Table: store_sales
6. For Table permissions, select Select.
7. For Data permissions, specify the following columns: store_id, transaction_date, product_name, country, sales_price, and quantity.
8. Choose Grant.
   

Alice now verifies the table permission by navigating to the Tables tab under the Data Catalog and searching for store_sales. Select store_sales and choose View under Actions. The following screenshots show the details for both sets of permissions.

Data Analyst uses Athena for building daily sales reports

Alice, the data analyst logs in to the Athena console and run the following query:

select * from "salesdb"."store_sales" limit 5

Alice has the user attributes as Department=sales, Role=Analyst, Region=US, and this attribute combination allows her access to US sales data to specific sales only column, without access to customer data as shown in the following screenshot.

BI Analyst uses Redshift for building sales dashboards

Bob, the BI Analyst, logs in to the Redshift console and run the following query:

select * from "salesdb"."store_sales" limit 10

Bob has the user attributes Department=sales, Role=BIAnalyst, Region=US, and this attribute combination allows him access to all columns including customer data for US sales data.

Data Scientist uses HAQM EMR to process sales data

Finally, Charlie logs in to the EMR console and submit the EMR job with runtime role as scientist_role. Charlie uses  the script sales_analysis.py that is uploaded to s3 bucket created for the script. He chooses the EMR Serverless application created with Lake Formation enabled.

Charlie submits batch job runs by choosing the following values:

• Name: sales_analysis_Charlie
• Runtime_role: scientist_role
• Script location: <s3_script_path>/sales_analysis.py
• For spark properties, provide key as spark.emr-serverless.lakeformation.enabled and value as true.
• Additional configurations: Under Metastore configuration select Use AWS Glue Data Catalog as metastore. Charlie keeps rest of the configuration as default.

Once the job run is completed, Charlie can view the output by selecting stdout under Driver log files.

Charlie uses scientist_role as job runtime role with the attributes Department=sales, Role=Scientist, Region=ALL, and this attribute combination allows him access to select columns of all sales data.

Clean up

Complete the following steps to delete the resources you created to avoid unexpected costs:

1. Delete the IAM users created.
2. Delete the AWS Glue database and table resources created for the post, if any.
3. Delete the Athena, Redshift and EMR resources created for the post.

Conclusion

In this post, we showcased how you can use SageMaker Lakehouse attribute-based access control, using IAM principals and session tags to simplify data access, grant creation, and maintenance. With attribute-based access control, you can manage permissions using dynamic business attributes associated with user identities and secure your data in the lakehouse by defining fine-grained permissions in the Lake Formation that are enforced across analytics and ML tools and engines.

For more information, refer to documentation. We encourage you to try out the SageMaker Lakehouse with ABAC and share your feedback with us.

About the authors

Sandeep Adwankar is a Senior Product Manager at AWS. Based in the California Bay Area, he works with customers around the globe to translate business and technical requirements into products that enable customers to improve how they manage, secure, and access data.

Srividya Parthasarathy is a Senior Big Data Architect on the AWS Lake Formation team. She enjoys building data mesh solutions and sharing them with the community.