[SEO Subhead]
This Guidance demonstrates how to conduct a semantic video search powered by generative artificial intelligence (AI). It streamlines the process of finding and extracting relevant video segments from vast amounts of video data. With natural language queries, it can quickly search for specific scenes, actions, concepts, people, objects, and more without the need to watch the full length of original videos. Designed to drive cost efficiency and scalability in media supply chain management, this Guidance is built for those who want to efficiently generate content such as trailers, sports highlights, captions, and descriptions from their existing video assets through automation.
Please note: [Disclaimer]
Architecture Diagram
[Architecture diagram description]
Step 1
HAQM Simple Storage Service (HAQM S3) hosts a static website for the semantic video search, served by an HAQM CloudFront distribution. HAQM Cognito provides customer identity and access management for the web application.
Step 2
Upload videos to HAQM S3 with HAQM S3 pre-signed URLs.
Step 3
After a video is uploaded successfully, an API call to HAQM API Gateway invokes AWS Lambda to queue new indexing-video requests in HAQM Simple Queue Service (HAQM SQS).
Step 4
Lambda processes new messages in the HAQM SQS queue, initiating an AWS Step Functions workflow.
Step 5
HAQM Rekognition detects multiple video shots from the original video, containing the start, end, and duration of each shot. Shot metadata is used to generate a sequence of frames, which are grouped by individual video shots and stored in HAQM S3.
Step 6
Step Functions uses the Map state to run a set of workflows for each frame stored in HAQM S3 in parallel.
Step 7
HAQM Rekognition detects celebrities and text in the frames.
Step 8
A foundation model (FM) in HAQM Bedrock generates image descriptions from the frames data and detected text or celebrities in the frames. The FM creates a description of the video shot based on a sequence of descriptions of its associated frames.
Step 9
An embedding model in HAQM Bedrock generates the embeddings of a video’s descriptions. HAQM OpenSearch Service stores the embeddings and other related information in a vector database.
Step 10
An embedding model in HAQM Bedrock generates the embedding of the users’ query, which is then used to perform semantic search for the videos from the OpenSearch Service vector database.
Step 11
HAQM DynamoDB tables store profiling and video indexing job metadata to keep track of the jobs’ status and other relevant information.
Step 1
HAQM Simple Storage Service (HAQM S3) hosts a static website for the semantic video search, served by an HAQM CloudFront distribution. HAQM Cognito provides customer identity and access management for the web application.
Step 2
Upload videos to HAQM S3 with HAQM S3 pre-signed URLs.
Step 3
After a video is uploaded successfully, an API call to HAQM API Gateway invokes AWS Lambda to queue new indexing-video requests in HAQM Simple Queue Service (HAQM SQS).
Step 4
Lambda processes new messages in the HAQM SQS queue, initiating an AWS Step Functions workflow.
Step 5
HAQM Rekognition detects multiple video shots from the original video, containing the start, end, and duration of each shot. Shot metadata is used to generate a sequence of frames, which are grouped by individual video shots and stored in HAQM S3.
Step 6
Step Functions uses the Map state to run a set of workflows for each frame stored in HAQM S3 in parallel.
Step 7
HAQM Rekognition detects celebrities and text in the frames.
Step 8
A foundation model (FM) in HAQM Bedrock generates image descriptions from the frames data and detected text or celebrities in the frames. The FM creates a description of the video shot based on a sequence of descriptions of its associated frames.
Step 9
An embedding model in HAQM Bedrock generates the embeddings of a video’s descriptions. HAQM OpenSearch Service stores the embeddings and other related information in a vector database.
Step 10
An embedding model in HAQM Bedrock generates the embedding of the users’ query, which is then used to perform semantic search for the videos from the OpenSearch Service vector database.
Step 11
HAQM DynamoDB tables store profiling and video indexing job metadata to keep track of the jobs’ status and other relevant information.
Well-Architected Pillars
The AWS Well-Architected Framework helps you understand the pros and cons of the decisions you make when building systems in the cloud. The six pillars of the Framework allow you to learn architectural best practices for designing and operating reliable, secure, efficient, cost-effective, and sustainable systems. Using the AWS Well-Architected Tool, available at no charge in the AWS Management Console, you can review your workloads against these best practices by answering a set of questions for each pillar.
The architecture diagram above is an example of a Solution created with Well-Architected best practices in mind. To be fully Well-Architected, you should follow as many Well-Architected best practices as possible.
-
Operational ExcellenceRead the Operational Excellence whitepaper
Use HAQM CloudWatch to monitor your workload and get insights into performance so that your application runs and responds effectively. Complement this with Step Functions to orchestrate your serverless workflow, handle errors automatically, and maintain consistency in processing. These services give you the observability and control you need to proactively identify and resolve issues.
-
SecurityRead the Security whitepaper
Utilize Cognito to authenticate users and authorize access to your resources. Pair this with CloudFront to securely deliver your content globally, with built-in distributed denial-of-service (DDoS) protection from AWS Shield Standard. These services ensure only valid users can access your application while safeguarding your data.
-
ReliabilityRead the Reliability whitepaper
HAQM S3 stores your videos, images, and other media objects with high availability and durability, minimizing the risk of data loss or downtime. It also supports features like versioning, cross-Region replication, and lifecycle policies, contributing to the integrity and availability of your data. Use this with CloudWatch to monitor your application and automatically respond to incidents when thresholds are breached. Finally, use Step Functions to orchestrate your serverless workflows, with built-in error handling and retry mechanisms that support consistent processing and fault tolerance.
-
Performance EfficiencyRead the Performance Efficiency whitepaper
Optimize your application's performance by taking advantage of serverless computing with Lambda. You can use this in tandem with HAQM Bedrock for high-performing machine learning inference, and benefit from the global reach of CloudFront to deliver content with low latency. These services work together to scale your application efficiently and deliver a responsive user experience.
-
Cost OptimizationRead the Cost Optimization whitepaper
Optimize costs by using HAQM S3 Intelligent-Tiering and HAQM S3 Lifecycle configuration to automatically store your media in the most cost-effective manner. HAQM S3 Lifecycle transitions files to cheaper storage tiers or deletes them based on rules, keeping your costs low. Lambda and HAQM Bedrock also help, as you only pay for the resources you use without provisioning servers. Lambda scales on demand, charging only for compute time used, and HAQM Bedrock is invoked as needed, avoiding idle infrastructure costs.
-
SustainabilityRead the Sustainability whitepaper
Lambda scales dynamically based on demand, improving efficiency and eliminating constant server provisioning. It also supports event filtering for HAQM SQS to reduce traffic to your Lambda functions. Complement this with HAQM S3 Intelligent-Tiering and HAQM S3 Lifecycle, which automatically shift data to the most suitable storage tiers based on usage. This minimizes expensive storage and unnecessary data. Finally, CloudFront caches content closer to users, lowering network traffic and energy consumption for data transfers.
Implementation Resources
A detailed guide is provided to experiment and use within your AWS account. Each stage of building the Guidance, including deployment, usage, and cleanup, is examined to prepare it for deployment.
The sample code is a starting point. It is industry validated, prescriptive but not definitive, and a peek under the hood to help you begin.
Related Content
[Title]
Disclaimer
The sample code; software libraries; command line tools; proofs of concept; templates; or other related technology (including any of the foregoing that are provided by our personnel) is provided to you as AWS Content under the AWS Customer Agreement, or the relevant written agreement between you and AWS (whichever applies). You should not use this AWS Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the AWS Content, such as sample code, as appropriate for production grade use based on your specific quality control practices and standards. Deploying AWS Content may incur AWS charges for creating or using AWS chargeable resources, such as running HAQM EC2 instances or using HAQM S3 storage.
References to third-party services or organizations in this Guidance do not imply an endorsement, sponsorship, or affiliation between HAQM or AWS and the third party. Guidance from AWS is a technical starting point, and you can customize your integration with third-party services when you deploy the architecture.