AWS Cloud Operations Blog

Integrating Kubecost with HAQM Managed Service for Prometheus

This blog post was co-written by Linh Lam, Solution Architect, Kubecost

Customers can track their Kubernetes control plane and HAQM Elastic Compute Cloud (HAQM EC2) costs using AWS Cost and Usage Reports. However, they often need deeper insights to accurately track Kubernetes costs across namespaces, clusters, pods, and more. We recently announced that AWS and Kubecost collaborated to deliver cost monitoring for EKS customers. Today, in partnership with Kubecost, we are launching a streamlined integration between Kubecost and HAQM Managed Service for Prometheus. This approach provides cluster operators with HAQM Elastic Kubernetes Service (HAQM EKS) cost insights powered by Kubecost for a single HAQM EKS cluster and backed by a scalable HAQM Managed Service for Prometheus workspace.

HAQM Managed Service for Prometheus is a Prometheus-compatible monitoring and alerting service that makes it easy to monitor containerized applications and infrastructure at scale. You can use the open-source Prometheus query language to monitor and alert for the performance of containerized workloads without having to worry about scaling the underlying monitoring infrastructure. The service automatically scales the ingestion, storage, alerting, and querying of operational metrics as workloads grow or shrink. Furthermore, it’s integrated with AWS security services to enable fast and secure access to data. This lets you concentrate on your workloads instead of having to manage your monitoring stack.

Kubecost is built on OpenCost, which was recently accepted as a Cloud Native Computing Foundation (CNCF) Sandbox project, and is actively supported by AWS. Kubecost provides fine-grained visibility into your cluster, letting you break down costs by Kubernetes resources, such as pods, nodes, namespaces, and labels. This costs-visibility allows teams to have transparent and accurate cost data based on their actual AWS bill.

Now let’s configure Kubecost to be backed by an HAQM Managed Service for Prometheus workspace.

Prerequisites

To get started with this post, you should have the following:

Installing Kubecost

If your cluster is running Kubernetes 1.23 or above, you will need to install the HAQM EBC CSI driver to your cluster. This driver allows your cluster to manage the lifecycle of HAQM EBS volumes for persistent volumes. Run the following command to create an IAM service account with the policies needed to use the HAQM EBS CSI Driver.

eksctl create iamserviceaccount   \
    --name ebs-csi-controller-sa   \
    --namespace kube-system   \
    --cluster <CLUSTER_NAME>   \
    --attach-policy-arn arn:aws:iam::aws:policy/service-role/HAQMEBSCSIDriverPolicy  \
    --approve \
    --role-only \
    --role-name HAQMEKS_EBS_CSI_DriverRole

Where <CLUSTER_NAME> is the name of your HAQM EKS cluster.

Export the ARN of the HAQMEKS_EBS_CSI_DriverRole, which was created by the previous command:

export SERVICE_ACCOUNT_ROLE_ARN=$(aws iam get-role --role-name HAQMEKS_EBS_CSI_DriverRole | jq -r '.Role.Arn')

Now install the HAQM EBS CSI add-on for EKS using the HAQMEKS_EBS_CSI_DriverRole by issuing the following command:

eksctl create addon --name aws-ebs-csi-driver --cluster <CLUSTER_NAME> \
    --service-account-role-arn $SERVICE_ACCOUNT_ROLE_ARN --force

Where <CLUSTER_NAME> is the name of your HAQM EKS cluster. To verify that the HAQM EBS CSI driver add-on is working as expected, run the following command:

kubectl get pods -n kube-system | grep ebs

This command should list several running pods.

Now install Kubecost on the HAQM EKS cluster by running the following command:

helm upgrade -i kubecost \
oci://public.ecr.aws/kubecost/cost-analyzer --version <VERSION> \
--namespace kubecost --create-namespace \
-f http://tinyurl.com/kubecost-amazon-eks

Where <VERSION> is the current version of the kubecost/cost-analyzer chart. At the time of publication, the latest version is 1.98.0. You can find all available versions of the EKS optimized Kubecost bundle here. We recommend finding and installing the latest available Kubecost cost analyzer chart version.

Next, you must set up IAM roles for Kubecost service accounts (IRSA). Using the OIDC provider for the cluster, you grant IAM permissions to your cluster’s service accounts. You must grant appropriate permissions to the kubecost-cost-analyzer and kubecost-prometheus-server service accounts. These will be used to send and retrieve metrics from the workspace. Run the following commands on the command line:

eksctl create iamserviceaccount \
    --name kubecost-cost-analyzer \
    --namespace kubecost \
    --cluster <CLUSTER_NAME> --region <REGION> \
    --attach-policy-arn arn:aws:iam::aws:policy/HAQMPrometheusQueryAccess \
    --attach-policy-arn arn:aws:iam::aws:policy/HAQMPrometheusRemoteWriteAccess \
    --override-existing-serviceaccounts \
    --approve

eksctl create iamserviceaccount \
    --name kubecost-prometheus-server \
    --namespace kubecost \
    --cluster <CLUSTER_NAME> --region <REGION> \
    --attach-policy-arn arn:aws:iam::aws:policy/HAQMPrometheusQueryAccess \
    --attach-policy-arn arn:aws:iam::aws:policy/HAQMPrometheusRemoteWriteAccess \
    --override-existing-serviceaccounts \
    --approve

<CLUSTER_NAME> is the name of the HAQM EKS cluster where you want to install Kubecost and <REGION> is the region of the HAQM EKS cluster. Note that these commands each generate an AWS CloudFormation stack that creates a new IAM role, assigns the HAQMPrometheusQueryAccess and HAQMPrometheusRemoteWriteAccess managed policies, and configures a trust relationship with the OIDC provider for the cluster. This allows the service account to assume the IAM role.

If you don’t yet have an HAQM Managed Service for Prometheus workspace, then the following AWS Command Line Interface (AWS CLI) command will create a workspace called kubecost-amp in the current region. You may skip this step if you have a pre-existing workspace.

aws amp create-workspace --alias kubecost-amp --region <REGION>

<REGION> is the region where you want the workspace to be created. Now you can update the Kubecost configuration so that it uses your HAQM Managed Service for Prometheus workspace as the source for your cluster metrics. Run the following command from the command line:

export AWS_REGION=<REGION>
export AMP_WORKSPACE_ID=<WORKSPACE-ID>

Where <WORKSPACE-ID> is the Workspace ID of the cluster and <REGION> is the current region of the workload. You can get the Workspace ID of the workspace from the Summary section of the HAQM Managed Service for Prometheus console. See the following figure.

Figure 1: The HAQM Managed Service for Prometheus workspace detail, which lists the workspaces ARN, Workspace ID, Endpoint – remote write URL, and the Endpoint – query URL

Figure 1: The HAQM Managed Service for Prometheus workspace detail, which lists the workspaces ARN, Workspace ID, Endpoint – remote write URL, and the Endpoint – query URL

Run the following command to create a file called config-values.yaml, which contains the defaults that Kubecost will use for connecting to your HAQM Managed Service for Prometheus workspace.

cat << EOF > config-values.yaml
global:
  amp:
    enabled: true
    prometheusServerEndpoint: http://localhost:8005/workspaces/${AMP_WORKSPACE_ID}
    remoteWriteService: http://aps-workspaces.${AWS_REGION}.amazonaws.com/workspaces/${AMP_WORKSPACE_ID}/api/v1/remote_write
    sigv4:
      region: ${AWS_REGION}

sigV4Proxy:
  region: ${AWS_REGION}
  host: aps-workspaces.${AWS_REGION}.amazonaws.com
EOF

Now, run the following command to configure Kubecost to begin using your workspace:

helm upgrade -i kubecost \
oci://public.ecr.aws/kubecost/cost-analyzer --version <VERSION> \
--namespace kubecost --create-namespace \
-f http://tinyurl.com/kubecost-amazon-eks \
-f config-values.yaml

<VERSION> is the current version of the kubecost/cost-analyzer chart. At the time of publication, the latest version is 1.98.0.

Finally, restart the Prometheus deployment, which reloads the service account configuration.

kubectl rollout restart deployment/kubecost-prometheus-server -n kubecost

After a few minutes, your Kubecost cluster should be available and ready to view.

You can enable port-forwarding to expose the Kubecost dashboard:

kubectl port-forward deployment.apps/kubecost-cost-analyzer 9090:9090 -n kubecost

The dashboard will be available to view here.

: The Kubecost dashboard, which shows monthly savings of $1,058.56, monthly Kubernetes costs of $1,627.16, and a 3.9% cost efficiency. The dashboard shows how costs are allocated across various Kubernetes resources.]

Figure 2: The Kubecost dashboard, backed by an HAQM Managed Service for Prometheus workspace

Conclusion

We’re excited that this release allows customers to store and analyze their Kubecost metrics using HAQM Managed Service for Prometheus as a backend. This release helps customers better support cost monitoring for their HAQM EKS workloads without having to deal with managing the monitoring stack.

Using Kubecost in your HAQM EKS workloads lets you better monitor costs associated with containerized workloads. Your HAQM EKS cost insights are powered by Kubecost and backed by a scalable HAQM Managed Service for Prometheus workspace.

If you need support, you can submit a support request via AWS Support.

If you would like to learn more from the Kubecost team, contact them here.

Get started today by integrating your workload with HAQM EKS cost monitoring.

About the authors:

Mike George

Mike George is a Principal Solutions Architect based out of Salt Lake City, Utah. He enjoys helping customers solve their technology problems. His interests include software engineering, security, artificial intelligence (AI), and machine learning (ML).

Abhi Khanna

Abhi Khanna is a Senior Product Manager at AWS specializing in HAQM Managed Service for Prometheus. He has been involved with Observability products for the last 3 years, helping customers build towards more perfect visibility. He enjoys helping customers simplify their monitoring experience. His interests include software engineering, product management, and building things.

Linh Lam, Solution Architect, Kubecost

Linh Lam is a Kubecost Solution Architect, ISV, focusing on integration and building solutions for customers. He is also passionate about application modernization, serverless, and container technology. Outside of work he enjoys hiking, camping, and building his home audio systems.