AWS Cloud Operations Blog
Integrating Kubecost with HAQM Managed Service for Prometheus
This blog post was co-written by Linh Lam, Solution Architect, Kubecost
Customers can track their Kubernetes control plane and HAQM Elastic Compute Cloud (HAQM EC2) costs using AWS Cost and Usage Reports. However, they often need deeper insights to accurately track Kubernetes costs across namespaces, clusters, pods, and more. We recently announced that AWS and Kubecost collaborated to deliver cost monitoring for EKS customers. Today, in partnership with Kubecost, we are launching a streamlined integration between Kubecost and HAQM Managed Service for Prometheus. This approach provides cluster operators with HAQM Elastic Kubernetes Service (HAQM EKS) cost insights powered by Kubecost for a single HAQM EKS cluster and backed by a scalable HAQM Managed Service for Prometheus workspace.
HAQM Managed Service for Prometheus is a Prometheus-compatible monitoring and alerting service that makes it easy to monitor containerized applications and infrastructure at scale. You can use the open-source Prometheus query language to monitor and alert for the performance of containerized workloads without having to worry about scaling the underlying monitoring infrastructure. The service automatically scales the ingestion, storage, alerting, and querying of operational metrics as workloads grow or shrink. Furthermore, it’s integrated with AWS security services to enable fast and secure access to data. This lets you concentrate on your workloads instead of having to manage your monitoring stack.
Kubecost is built on OpenCost, which was recently accepted as a Cloud Native Computing Foundation (CNCF) Sandbox project, and is actively supported by AWS. Kubecost provides fine-grained visibility into your cluster, letting you break down costs by Kubernetes resources, such as pods, nodes, namespaces, and labels. This costs-visibility allows teams to have transparent and accurate cost data based on their actual AWS bill.
Now let’s configure Kubecost to be backed by an HAQM Managed Service for Prometheus workspace.
Prerequisites
To get started with this post, you should have the following:
- An HAQM Elastic Kubernetes Service (HAQM EKS) cluster.
- Helm 3.9+, kubectl, eksctl, jq, and the awscli.
- AWS Identity and Access Management (IAM) credentials with permissions to create IAM roles.
- An IAM OIDC provider created for the HAQM EKS cluster.
Installing Kubecost
If your cluster is running Kubernetes 1.23 or above, you will need to install the HAQM EBC CSI driver to your cluster. This driver allows your cluster to manage the lifecycle of HAQM EBS volumes for persistent volumes. Run the following command to create an IAM service account with the policies needed to use the HAQM EBS CSI Driver.
eksctl create iamserviceaccount \
--name ebs-csi-controller-sa \
--namespace kube-system \
--cluster <CLUSTER_NAME> \
--attach-policy-arn arn:aws:iam::aws:policy/service-role/HAQMEBSCSIDriverPolicy \
--approve \
--role-only \
--role-name HAQMEKS_EBS_CSI_DriverRole
Where <CLUSTER_NAME>
is the name of your HAQM EKS cluster.
Export the ARN of the HAQMEKS_EBS_CSI_DriverRole
, which was created by the previous command:
export SERVICE_ACCOUNT_ROLE_ARN=$(aws iam get-role --role-name HAQMEKS_EBS_CSI_DriverRole | jq -r '.Role.Arn')
Now install the HAQM EBS CSI add-on for EKS using the HAQMEKS_EBS_CSI_DriverRole
by issuing the following command:
eksctl create addon --name aws-ebs-csi-driver --cluster <CLUSTER_NAME> \
--service-account-role-arn $SERVICE_ACCOUNT_ROLE_ARN --force
Where <CLUSTER_NAME>
is the name of your HAQM EKS cluster. To verify that the HAQM EBS CSI driver add-on is working as expected, run the following command:
kubectl get pods -n kube-system | grep ebs
This command should list several running pods.
Now install Kubecost on the HAQM EKS cluster by running the following command:
helm upgrade -i kubecost \
oci://public.ecr.aws/kubecost/cost-analyzer --version <VERSION> \
--namespace kubecost --create-namespace \
-f http://tinyurl.com/kubecost-amazon-eks
Where <VERSION>
is the current version of the kubecost/cost-analyzer chart. At the time of publication, the latest version is 1.98.0. You can find all available versions of the EKS optimized Kubecost bundle here. We recommend finding and installing the latest available Kubecost cost analyzer chart version.
Next, you must set up IAM roles for Kubecost service accounts (IRSA). Using the OIDC provider for the cluster, you grant IAM permissions to your cluster’s service accounts. You must grant appropriate permissions to the kubecost-cost-analyzer
and kubecost-prometheus-server
service accounts. These will be used to send and retrieve metrics from the workspace. Run the following commands on the command line:
eksctl create iamserviceaccount \
--name kubecost-cost-analyzer \
--namespace kubecost \
--cluster <CLUSTER_NAME> --region <REGION> \
--attach-policy-arn arn:aws:iam::aws:policy/HAQMPrometheusQueryAccess \
--attach-policy-arn arn:aws:iam::aws:policy/HAQMPrometheusRemoteWriteAccess \
--override-existing-serviceaccounts \
--approve
eksctl create iamserviceaccount \
--name kubecost-prometheus-server \
--namespace kubecost \
--cluster <CLUSTER_NAME> --region <REGION> \
--attach-policy-arn arn:aws:iam::aws:policy/HAQMPrometheusQueryAccess \
--attach-policy-arn arn:aws:iam::aws:policy/HAQMPrometheusRemoteWriteAccess \
--override-existing-serviceaccounts \
--approve
<CLUSTER_NAME>
is the name of the HAQM EKS cluster where you want to install Kubecost and <REGION>
is the region of the HAQM EKS cluster. Note that these commands each generate an AWS CloudFormation stack that creates a new IAM role, assigns the HAQMPrometheusQueryAccess
and HAQMPrometheusRemoteWriteAccess
managed policies, and configures a trust relationship with the OIDC provider for the cluster. This allows the service account to assume the IAM role.
If you don’t yet have an HAQM Managed Service for Prometheus workspace, then the following AWS Command Line Interface (AWS CLI) command will create a workspace called kubecost-amp
in the current region. You may skip this step if you have a pre-existing workspace.
aws amp create-workspace --alias kubecost-amp --region <REGION>
<REGION>
is the region where you want the workspace to be created. Now you can update the Kubecost configuration so that it uses your HAQM Managed Service for Prometheus workspace as the source for your cluster metrics. Run the following command from the command line:
export AWS_REGION=<REGION>
export AMP_WORKSPACE_ID=<WORKSPACE-ID>
Where <WORKSPACE-ID>
is the Workspace ID of the cluster and <REGION>
is the current region of the workload. You can get the Workspace ID of the workspace from the Summary section of the HAQM Managed Service for Prometheus console. See the following figure.

Figure 1: The HAQM Managed Service for Prometheus workspace detail, which lists the workspaces ARN, Workspace ID, Endpoint – remote write URL, and the Endpoint – query URL
Run the following command to create a file called config-values.yaml
, which contains the defaults that Kubecost will use for connecting to your HAQM Managed Service for Prometheus workspace.
cat << EOF > config-values.yaml
global:
amp:
enabled: true
prometheusServerEndpoint: http://localhost:8005/workspaces/${AMP_WORKSPACE_ID}
remoteWriteService: http://aps-workspaces.${AWS_REGION}.amazonaws.com/workspaces/${AMP_WORKSPACE_ID}/api/v1/remote_write
sigv4:
region: ${AWS_REGION}
sigV4Proxy:
region: ${AWS_REGION}
host: aps-workspaces.${AWS_REGION}.amazonaws.com
EOF
Now, run the following command to configure Kubecost to begin using your workspace:
helm upgrade -i kubecost \
oci://public.ecr.aws/kubecost/cost-analyzer --version <VERSION> \
--namespace kubecost --create-namespace \
-f http://tinyurl.com/kubecost-amazon-eks \
-f config-values.yaml
<VERSION>
is the current version of the kubecost/cost-analyzer chart. At the time of publication, the latest version is 1.98.0.
Finally, restart the Prometheus deployment, which reloads the service account configuration.
kubectl rollout restart deployment/kubecost-prometheus-server -n kubecost
After a few minutes, your Kubecost cluster should be available and ready to view.
You can enable port-forwarding to expose the Kubecost dashboard:
kubectl port-forward deployment.apps/kubecost-cost-analyzer 9090:9090 -n kubecost
The dashboard will be available to view here.
![: The Kubecost dashboard, which shows monthly savings of $1,058.56, monthly Kubernetes costs of $1,627.16, and a 3.9% cost efficiency. The dashboard shows how costs are allocated across various Kubernetes resources.]](http://d2908q01vomqb2.cloudfront.net/972a67c48192728a34979d9a35164c1295401b71/2022/09/21/couldops_1085_3.png)
Figure 2: The Kubecost dashboard, backed by an HAQM Managed Service for Prometheus workspace
Conclusion
We’re excited that this release allows customers to store and analyze their Kubecost metrics using HAQM Managed Service for Prometheus as a backend. This release helps customers better support cost monitoring for their HAQM EKS workloads without having to deal with managing the monitoring stack.
Using Kubecost in your HAQM EKS workloads lets you better monitor costs associated with containerized workloads. Your HAQM EKS cost insights are powered by Kubecost and backed by a scalable HAQM Managed Service for Prometheus workspace.
If you need support, you can submit a support request via AWS Support.
If you would like to learn more from the Kubecost team, contact them here.
Get started today by integrating your workload with HAQM EKS cost monitoring.
About the authors: