AWS Partner Network (APN) Blog
Transform Large Language Model Observability with Langfuse
By Marc Klingen, Co-Founder & CEO – Langfuse
By Clemens Rawert, Co-Founder – Langfuse
By Pyone Thant Win, Partner Solutions Architect – AWS
By Vaishali Taneja, Partner Solutions Architect – AWS
By Qiong (Jo) Zhang, Sr. WW Data & AI PSA – AWS
 |
| Langfuse |
 |
According to a 2024 Deloitte report, only 23% of organizations are prepared for risk governance associated with generative AI. With the emergence of generative AI, the use of Large Language Models (LLMs) presents challenges such as hallucination, and variable cost and performance. As companies accelerate their adoption of LLMs, the need for robust LLM observability has become critical. LLM observability is a proactive approach to monitoring, interpreting, and auditing LLM output generation. By providing visibility into the opaque system of LLM behavior, observability tools empower teams to detect errors, mitigate bias, and ensure reliable system performance.
Langfuse is an AWS Technology Partner offering a LLM engineering platform that helps developers collaboratively monitor, debug, analyze, and iterate on their LLM applications. With over 6 million SDK installs per month, 10,000 GitHub stars, and 4.7 million Docker pulls, Langfuse has established itself as a popular platform for LLM observability. With Langfuse, customers inspect and audit the source code, and freely transfer data in and out of the solution. This blog shows how Langfuse helps organizations take control of LLM observability and optimization by providing capabilities for real-time LLM tracing, monitoring, and prompt management.
Enterprise-Ready LLM Observability by Langfuse
Langfuse is an open-source project built to foster transparency and trust for organizations working with LLM. The Langfuse platform is available on Langfuse Cloud hosted by Langfuse on AWS. An option to self-host with AWS Fargate is also available for users who prefer to deploy and manage Langfuse in their own AWS account. Langfuse offers native integration with HAQM Bedrock, seamlessly providing observability for generative AI applications on HAQM Bedrock. Langfuse also supports popular generative AI tools and frameworks such as LangChain, LlamaIndex, and DSPy ensuring compatibility with existing AI workflows.
The platform provides essential tools for:
- Real-time tracing and monitoring: Gain instant visibility into your LLM’s behavior, allowing identification and resolution of issues before they impact users.
- Collaborative prompt management: Streamline your team’s workflow by centralizing prompt development, version control, and testing, leading to more efficient and effective LLM interactions.
- Performance evaluation: Objectively assess your LLM’s output quality and relevance, enabling data-driven decisions to improve model performance and user satisfaction.
- Comprehensive metrics tracking: Measure and optimize crucial KPIs such as response times, token usage, and cost efficiency, ensuring your LLM applications deliver maximum value.
Langfuse’s Architectural Evolution on AWS
Langfuse’s infrastructure journey illustrates the evolution from a simple prototype to an enterprise-ready observability platform. Figure 1 shows the current version-3 architecture hosted on Langfuse Cloud. This architecture uses AWS Fargate containers spanning across private subnets for enhanced security and scalability. The ingestion pipeline separates immediate API calls from background operations, using dedicated workers for asynchronous processing. This design ensures API calls achieve high throughput and low latency under varying workloads.
The architecture also incorporates a diverse storage strategy to maintain high performance and scalability. HAQM ElastiCache for Redis is used for caching and queue management. HAQM Aurora for Postgres stores transactional data. Clickhouse on AWS stores traces, observations, and scores. HAQM Simple Storage Service (HAQM S3) stores raw events and multi-modal features.
Figure 1 – Langfuse Architecture Diagram
Langfuse handles tens of thousands of events per minute while maintaining consistent low-latency responses (50-100ms on average to fetch prompts) across both cloud-hosted and self-hosted deployments. This allows organizations to adopt Langfuse at any stage of their AI observability journey. Security features such as subnet isolation, robust access controls, and data protection measures, and load balancing and auto-scaling capabilities ensure reliable performance during peak usage. The platform scales effectively for enterprise needs while remaining accessible to smaller deployments.
Langfuse’s Key Features and Benefits
Langfuse addresses the challenges of working with LLMs. Its features empower developers to monitor, debug, evaluate and improve their AI-driven solutions.
LLM tracing and observability:
Langfuse addresses the challenge of LLM unpredictability by providing granular observability and control flow tracing to clarify LLM behavior, as shown in Figure 2. Specifically, Langfuse offers features like:
- LLM inference tracing: Tracking every step of the LLM’s decision-making process, from input to output.
- Embedding retrieval tracking: Monitor the retrieval and usage of embeddings, ensuring accuracy and relevance.
- API usage audits: Log API calls and interactions to identify inefficiencies or errors.
Figure 2 – LLM Tracing and Observability Feature by Langfuse
LLM output evaluation:
LLM outputs vary in quality and alignment with user intent, but manually assessing them is time-consuming and subjective. Langfuse solves this with a score-based evaluation system that leverages:
- Model-Based Evaluations: Automated assessments using user-defined metrics such as toxicity, bias, hallucination as shown in figure 3.
- User feedback: Direct input from end-users such as user-defined LLM-as-a-judge template (shown under Prompt section in Figure 3) to gauge satisfaction and relevance.
- Manual labelling: Human-in-the-loop evaluations for nuanced or mission critical outputs.
- Implicit data signals: Behavioral data (e.g., click-through rates, dwell time) to infer output quality.
Figure 3 – LLM Output Evaluation Feature by Langfuse
Prompt Management:
Langfuse offers tools for prompt versioning, prompt testing and collaboration features, facilitating team feedback and refinement, as shown in Figure 4. By integrating seamlessly with existing workflows, Langfuse allows users to deploy and monitor prompts in real-world scenarios while leveraging data-driven insights to improve output quality and alignment with user intent. Structured prompt management is essential for scaling LLM applications. It enables teams to maintain quality at scale through version control, collaborative refinement, and data-driven optimization. By centralizing prompt management, teams automatically deploy updates, measure their impact, and maintain consistent performance across their entire application without disrupting production code.
Figure 4 – Prompt Management Feature by Langfuse
Real-World Impact through Customer Success Stories
Leading enterprises across industries are leveraging Langfuse’s LLM observability platform to enhance their generative AI applications. Samsara is an industry-leading Physical Operations technology provider that offers an integrated platform combining Internet of Things (IoT) devices, AI, and cloud-based software to help organizations improve their operations. They connect fleets, equipment, sites, and people, offering features like AI-powered dash cams, workforce apps, and equipment monitoring. Samsara has integrated Langfuse into their LLM infrastructure for comprehensive monitoring and optimization of the Samsara Assistant. This generative AI offering provides customers with simple answers to complex questions about maintenance, compliance, and safety in their fleet. Langfuse LLM Engineering platform’s observability features is helping Samsara maintain high performance and reliability in both text and multimodal AI applications.
Organizations such as Merck Group and Twilio trust Langfuse. Merck Group is a Germany-based science and technology company operating in the Healthcare, Life Sciences and Electronics businesses. The company uses LLM observability features offered by Langfuse such as real-time LLM tracing across their organization-wide AI platform. Similarly, Twilio, customer engagement solution provider, uses Langfuse for collaborative prompt management in their solutions.
Conclusion
In this blog, we covered how Langfuse’s LLM engineering platform is transforming the LLM observability space. As AI transforms businesses across industries, the importance of robust observability tools cannot be overstated. Langfuse, with its comprehensive feature set and deep integration with AWS, provides organizations with the tools they need to monitor, debug, evaluate and improve AI applications. Whether you are starting your AI journey or looking to scale existing applications, Langfuse offers the capabilities needed to succeed in today’s AI-driven landscape.
Organizations looking to enhance their AI observability capabilities can get started with Langfuse through several channels:
- Langfuse Enterprise Edition (EE) – Self Hosting on AWS Marketplace
- Langfuse Cloud offering on AWS Marketplace
- aws-sample deployment: Deploy Langfuse on ECS with Fargate
- Langfuse Demo
- GenAIOps Workshop with HAQM Bedrock and Langfuse
.
.
Langfuse – AWS Partner Spotlight
Langfuse is an AWS Advanced Technology Partner that offers an open-source LLM engineering platform designed to streamline the development, monitoring, and testing of LLM-based applications. It addresses the unique challenges posed by LLMs, such as complex control flows, non-deterministic outputs, and mixed user intents, by offering robust tools for tracing, debugging, and evaluating these applications.