For the complete documentation index, see llms.txt. Markdown versions of documentation pages are available by appending .md to page URLs.
Skip to content

Prometheus FAQs

FAQs cover the most common questions about Finout’s Prometheus integrations across both per-cluster and centralized Prometheus Monitoring tools setups. It provides clarity on how Prometheus metrics are collected and processed to support accurate Kubernetes cost attribution across your environments. If you need additional assistance, contact Finout at support@finout.io.

How do I grant Finout access to a dedicated S3 bucket for Kubernetes data?

If you’d like to export Kubernetes data to a dedicated S3 bucket, different from the one used for your AWS Cost and Usage Report (CUR), you can grant Finout access by creating an inline IAM policy:

  1. Go to your newly created IAM role.

  2. Click Add permissions → Create inline policy.

  3. Select the JSON tab, and paste the following policy. Replace <BUCKET_NAME> with the name of your dedicated S3 bucket (or use your existing CUR bucket):

    ​

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": ["tag:GetTagKeys"],
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": ["s3:Get*", "s3:List*"],
                "Resource": "arn:aws:s3:::<BUCKET_NAME>/*"
            },
            {
                "Effect": "Allow",
                "Action": ["s3:Get*", "s3:List*"],
                "Resource": "arn:aws:s3:::<BUCKET_NAME>"
            }
        ]
    }

  4. Click Next until the Review screen appears.

  5. Name the policy finout-access-policy_K8S.

  6. Click Create policy to attach it to your IAM role.

  7. In the Finout console, complete Step 1 – Connect S3 bucket using your dedicated bucket details.


Do I need a separate Cost Center for every Kubernetes cluster?

No. If all your Kubernetes clusters (or CronJobs) use the same configuration and write their Prometheus metrics to the same S3 bucket and prefix, you can use a single Cost Center in Finout.

You only need a new Cost Center when clusters write to different S3 buckets or prefixes, or when your organization requires separate ownership boundaries—for example, if different departments or business units must export their Prometheus data to different buckets for security, compliance, or budgeting reasons.


How do I integrate Prometheus with Finout using Google GKE or Azure AKS?

Finout supports enriching AWS (EKS), GCP (GKE), and Azure (AKS) Kubernetes costs using Kubernetes metrics. However, this integration currently requires the Kubernetes metrics to be uploaded to an AWS S3 bucket.

You have two options for uploading your metrics:

  1. Use your own AWS S3 bucket Push your Kubernetes monitoring metrics (from GKE, AKS, or EKS) into an S3 bucket within your own infrastructure, then follow the standard integration steps.

  2. Use Finout’s AWS S3 bucket Push your Prometheus metrics to an S3 bucket hosted by Finout. Once your metrics are in the bucket, contact Finout Support at support@finout.io to complete the onboarding and enrichment setup for your cluster.


My Kubernetes costs are missing for past periods - how do I recover them?

Kubernetes data gaps occur if the Finout cronjob was inactive during past timeframes. You can resolve this by performing a backfill, which retrieves your historical cluster's Prometheus metrics to ensure your past spending and reports are complete.

To get started, reach out to Finout Support at support@finout.io with the following details:

  • The time period you'd like to backfill (start date – end date)

  • Your current CronJob YAML configuration

  • The cluster(s) you need to be backfilled

Once our team prepares the configuration, you'll be asked to run a one-time Job in your relevant Kubernetes clusters. Finout will provide the exact YAML configuration.

After the Job completes, data will be available in Finout within approximately 2 days.

Before you request a backfill, note the following limitations:

  • Multiple clusters must be backfilled separately — each cluster requires its own Job.

  • Your CronJob image must be version 2.0.2 or above.


Where is the finout/finout-metrics-exporter image hosted?

The finout/finout-metrics-exporter image is hosted on Docker Hub.


What should I do if I'm using both Prometheus per-cluster and Centralized Prometheus Monitoring tools (e.g., Thanos, Mimir, VictoriaMetrics) in my account?

Finout supports both per-cluster Prometheus and Centralized Prometheus Monitoring tools or aggregated setups.

To support both configurations:

  1. Create two Kubernetes Prometheus cost centers and deploy two cronjobs, one for each type, per the instructions above. Contact support at support@finout.io for help.

  2. Configure exporters to write separate S3 paths using:

    1. S3_BUCKET

    2. S3_PREFIX

  3. Update security policies to allow access to all relevant prefixes or buckets (if using a single bucket).


I have multiple Prometheus-compatible endpoints (e.g., several Mimir instances). How should I connect them?

Deploy one CronJob per endpoint. You can then choose either setup:

  • One cost center for all endpoints: all CronJobs use the same <S3_PREFIX>, and each CronJob uses a different CLUSTER_NAME. In a centralized setup, CLUSTER_NAME sets the folder name in S3.

  • One cost center per endpoint: each CronJob gets its own cost center and a different <S3_PREFIX>.

Both setups are fully supported.

Does each integration need a completely different S3 prefix? No. The prefix only needs to be unique per endpoint. For example, k8s/staging and k8s/production can share the same <BUCKET_NAME>.


Do I need to add the Worker Node Role policy for every Kubernetes cluster?

Yes. You must add the Kubernetes Worker Node Role Policy to each cluster you integrate with Finout. If you have multiple clusters, repeat Steps 2 and 3 for each one to ensure all clusters are correctly connected and sending data to Finout.


What does CLUSTER_LABEL_NAME an environment variable actually do?

  • This variable is required for all Centralized Prometheus Monitoring integrations.

  • It's not a cluster name — it's a pointer to the label in your metrics store that holds the cluster identifier.

  • Finout reads that label dynamically per metric to attribute data to the right cluster.

  • Needed because centralized monitoring stores serve multiple clusters from one endpoint.

  • Different from CLUSTER_NAME: that just names the S3 folder where metrics get reported — it doesn't affect cluster attribution


Why don't I see my Kubernetes labels in Finout?

  • Finout only pulls labels from three metrics: kube_pod_labels, kube_node_labels, and kube_namespace_labels

  • On those metrics, only labels matching the prefix label_.* are ingested — labels that don't match this pattern are skipped

  • If your label isn't showing up, it's most likely not exposed under this prefix (see Troubleshooting guide).


When should I use an OVERRIDE_METRIC environment variable, and how do I configure one?

Use an OVERRIDE_METRIC override when your Prometheus setup doesn't match Finout's default query for a given metric — for example, your kube-state-metrics exposes an extra label (like exported_namespace instead of namespace), your cluster uses non-standard label names, or you rely on a custom recording rule that changes how a metric is aggregated.

How it works:

Add an environment variable to the CronJob configuration named:

Where <metric_name> is the exact query name Finout uses internally (for example, cpu_usage, memory_usage_v2, node_info — see the Prerequisites section for the full list). The value of the environment variable is the complete replacement PromQL query, written exactly as you want Finout to run it against your Prometheus endpoint.

Example — overriding memory_usage_v2 to also strip an exported_namespace label:


How does Finout handle Sidecar InitContainers in cost allocation?

Finout Metrics Exporter collects the CPU and memory usage and request metrics from Sidecar InitContainers for calculating Kubernetes costs, ensuring accurate workload costs, and reflecting their full resource utilization.

Do short-lived initContainers affect cost calculations?

Short-lived initContainers still have a negligible impact on cost. Their brief runtime and small resource usage mean they continue to have little or no effect on overall cost attribution.

When will I see Sidecar InitContainer costs in Finout after enabling the integration?

After the Prometheus CronJob begins exporting metrics to your S3 bucket, Kubernetes cost data generally appears in Finout within 24 hours, as part of the same end-to-end process used for calculating Kubernetes costs. Updated cost attribution for Sidecar InitContainers may take up to 2 days to fully populate. This enhanced behavior is available starting from Finout Metrics Exporter version 1.30.


Does Finout support hostNetwork pods in Kubernetes network cost allocation?

Yes. Finout automatically excludes network metrics for pods configured with hostNetwork: true from cost attribution across all K8s integrations: Prometheus, Agent, and Datadog

Why does this matter?

Pods with hostNetwork: true share the node's network namespace, which causes cAdvisor to report the full node-level network traffic for each such pod. On a node running 4 of these pods, the node's traffic was being counted 4 times over — inflating the reported network costs for those workloads.


Can I use the Finout Kubernetes agent and the Prometheus CronJob in the same environment?

Yes. You can run the agent on some clusters and the CronJob on others, for example the agent on dev EKS and the CronJob on prod EKS. Each cluster must be covered by only one integration. Please contact Finout Support to set up this combination.


How can I reduce AMP costs when using Finout for Kubernetes cost allocation?

To reduce AMP costs, the best practice is to have AMP scrape only the metrics required for Finout’s Kubernetes cost calculation, assuming this is your only monitoring use case. These metrics must be scraped at a one-minute sampling interval, as increasing the interval will reduce cost-allocation accuracy. The required metrics are:

For more information, please see the Troubleshooting section.

Last updated

Was this helpful?