# Prometheus Per-Cluster Integration

## Overview

Integrate Finout with the Prometheus instance in each monitored cluster to track that cluster’s Kubernetes costs and waste directly in Finout. This Per-Cluster Prometheus cost enrichment is supported across all major clouds, including AWS, GCP, and Azure. 

* To connect multiple clusters that share the same configuration, apply the YAML from Step 3 (“Create and Configure the CronJob”) to every cluster - this will send all metrics to the same S3 bucket and prefix. 
* If you need different S3 locations (buckets or prefixes), create additional Prometheus integration for those clusters by contacting support at <support@finout.io>.

**Prometheus Per-cluster Setup at a Glance**:

1. **Set Collection Method:** Name your integration and select your metrics collection method.
2. **Connect S3 Bucket**: The destination for Kubernetes metrics.
3. **Set cronjob permissions:** This allows the cronjob to write the metrics into your S3 bucket.
4. **Configure the create cronjob YAML** using the template, and deploy it in the cluster.
   1. Select the authentication method.
   2. Set the required environment variables in the YAML and make additional adjustments, then deploy it in your cluster.
5. **Select cost centers:** This enriches with the integrated Kubernetes metrics.
6. **Validate your Kubernetes Integration:** Ensure that the Prometheus metrics are exported correctly to S3.

**What happens next**: A Prometheus (per-cluster) Cost Center is created and automatically enriched to your AWS Cost Center.  Data appears in Finout within \~2 days (due to cloud billing delay).

## 1. Set Collection Method

<figure><img src="https://3858159242-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWqjB2puKXPDR7L86FX2e%2Fuploads%2FBixQ3CDbkbuYj3n2114C%2Fimage.png?alt=media&#x26;token=9bc6c6bb-a498-4c6b-ae06-59a55e867aba" alt=""><figcaption></figcaption></figure>

1. In Finout, navigate to **Settings** > **Cost Centers > Prometheus (Kubernetes).**\
   The **Set Collection Method** step appears.<br>

   <figure><img src="https://3858159242-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWqjB2puKXPDR7L86FX2e%2Fuploads%2Fb5UCPcSeJSYkH9N6HRuC%2Fimage.png?alt=media&#x26;token=1f4f899b-7ee8-4a78-9ab3-781d8bdb1ba9" alt=""><figcaption></figcaption></figure>
2. Name your Prometheus integration.&#x20;
3. Ensure that the Per-cluster collection method is selected.
4. Click **Next**.\
   You are brought to the **Connect S3 Bucket** step.

## 2. Connect S3 Bucket <a href="#h_8048bc22bc" id="h_8048bc22bc"></a>

<figure><img src="https://3858159242-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWqjB2puKXPDR7L86FX2e%2Fuploads%2FpIRN23TKEa9g1X37SJm2%2Fimage.png?alt=media&#x26;token=2f99249d-ad3e-466b-8a54-ed7370e82fd6" alt=""><figcaption></figcaption></figure>

1. Connect a S3 bucket that contains the Kubernetes metrics collected from Prometheus using this integration. Finout is granted read-only permissions to access the data.\
   Ensure that you have already connected an S3 bucket to Finout for the AWS Cost and Usage Report (CUR).\
   You can reuse the **same S3 bucket and IAM role**, and Finout will automatically populate the **Role ARN, Bucket name**, and **External ID** fields in the console.\
   \
   If you want to use a different S3 bucket or haven’t configured one yet, follow the steps in **Grant Finout Access to an S3 Bucket**.\
   \
   Fill in the following fields:
   1. **External ID** - this is taken from your existing AWS Cost Center and is filled in by default. Use this same External ID in the IAM role’s trust policy to grant Finout permissions to read from the S3 bucket that stores your Prometheus metrics.
   2. **ARN Role** - Provide the ARN of the IAM role that grants Finout read-only access to this S3 bucket. When creating or updating this role, make sure you use the External ID from the Finout console in the role’s trust policy.
   3. **Bucket Name** - Enter the name of the S3 bucket that stores your Prometheus metrics. Use the bucket name only (no `s3://` and no path). It must be in the Region you selected and readable by the Role ARN.
   4. **S3 prefix** - S3 prefix (folder path) in the bucket used to store Prometheus metrics (default is `k8s/prometheus`).
   5. **Region** - AWS region of the bucket (e.g., us-east-1). Must match the bucket’s actual region.
2. Click **Next**.

{% hint style="info" %}
**Note**: Finout supports enriching **AWS (EKS), GCP (GKE), and Azure (AKS)** Kubernetes costs using Kubernetes metrics. However, this integration currently requires the Kubernetes metrics to be **uploaded to an AWS S3 bucket**. To connect other cost centers, contact Finout Support at <support@finout.io>.
{% endhint %}

{% hint style="warning" %}
**Important**:&#x20;

* If you want to integrate Finout with more than one cluster, repeat Step 3 (Set CronJob Permissions) and Step 4 (Configure and Create the Cronjob) for each cluster. If the clusters belong to the same Cost Center, make sure they all use the same `S3_PREFIX`.
* The values  `BUCKET_NAME` and `S3_PREFIX` across steps 3 and 4 need to be identical to the values added in this step.&#x20;
  {% endhint %}

## 3. Set CronJob Permissions <a href="#h_5ef0375593" id="h_5ef0375593"></a>

In this step, you will grant the cronjob permissions to write the Kubernetes metrics into the bucket you configured in the previous step.<br>

<figure><img src="https://3858159242-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWqjB2puKXPDR7L86FX2e%2Fuploads%2F1QBIFP97ieHM6bQzeyWy%2Fimage.png?alt=media&#x26;token=3d5ed0b3-ad62-4ac6-9c3f-cd606a972112" alt=""><figcaption></figcaption></figure>

1. **Configure Policy:**\
   Attach this policy to the Kubernetes node role, or to the IAM role used by your CronJob, so the CronJob has the S3 access it needs:

   * **Write** to store exported metrics in your bucket.
   * **Read** to check its saved state in the bucket and know from which timestamp to continue.
   * **Delete** to remove files from the bucket older than the retention period (30 days by default, configurable).

   ```yaml
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "FinoutBucketPermissions",
         "Effect": "Allow",
         "Action": "s3:ListBucket",
         "Resource": "arn:aws:s3:::<BUCKET_NAME>",
         "Condition": {
           "StringEquals": {
             "s3:delimiter": "/"
           },
           "StringLike": {
             "s3:prefix": "<S3_PREFIX>*"
           }
         }
       },
       {
         "Sid": "FinoutMetricFilesPermissions",
         "Effect": "Allow",
         "Action": [
           "s3:PutObject",
           "s3:GetObject",
           "s3:DeleteObject"
         ],
         "Resource": "arn:aws:s3:::<BUCKET_NAME>/<S3_PREFIX>/*"
       }
     ]
   }

   ```

2. **Prepare kube-state-metrics (Prerequisite):**\
   Use kube-state-metrics version 2.0.2 or later.\
   If your cluster uses the prometheus-kube-state-metrics DaemonSet, add the flag below so kube-state-metrics exports all required labels to your Prometheus endpoint (you can adjust the pattern to match your setup):

   `--metric-labels-allowlist=pods=[*]`*`,nodes=`*`[*]`\
   \
   ​Do this by adding an arg to the kube-state-metrics container, for example:

   ```yaml
   spec:
   containers:
     args:
      --port=8080
      --metric-labels-allowlist=pods=[*],nodes=[*]
   ```

3. Click **Next**.

## 4. Configure and Create the CronJob <a href="#h_135c6f678f" id="h_135c6f678f"></a>

<figure><img src="https://3858159242-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWqjB2puKXPDR7L86FX2e%2Fuploads%2FhKY6k6OLkknB1ZLrXwb0%2Fimage.png?alt=media&#x26;token=c2826afb-eb24-4a89-a8a4-27cbd35d0f12" alt=""><figcaption></figcaption></figure>

1. Copy the CronJob configuration below to a file and modify the values of the relevant environment variables (for example, cronjob.yaml):|

Example YAML:

```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: finout-prometheus-exporter-job
spec:
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 1
  concurrencyPolicy: Forbid
  schedule: "*/30 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: finout-prometheus-exporter
              image: finout/finout-metrics-exporter:2.0.3
              imagePullPolicy: Always
              env:
                - name: S3_BUCKET
                  value: "<BUCKET_NAME>"
                - name: S3_PREFIX
                  value: "<S3_PREFIX>"
                - name: CLUSTER_NAME
                  value: "<CLUSTER_NAME>"
                - name: HOSTNAME
                  value: "<PROMETHEUS_SERVICE>.<NAMESPACE>.svc.cluster.local"
                - name: PORT
                  value: 9090
          restartPolicy: OnFailure
```

* This is an example of a CronJob that schedules a Job every 30 minutes.&#x20;
* The job queries Prometheus with a 5-second delay between queries to prevent overloading your Prometheus stack.

2. Modify the suggested YAML file above, if needed.

#### YAML Environment Variables:

<table><thead><tr><th width="134.19140625">Category</th><th width="132.20703125">Variable</th><th width="135.1875">Description</th><th width="116.16015625">Required / Optional</th><th width="89.65234375">Default</th><th>Notes</th></tr></thead><tbody><tr><td>Scope &#x26; Multi-Cluster Behavior</td><td><strong>CLUSTER_NAME</strong></td><td>The cluster name defines the folder where metrics are stored in S3 and also appears as the cluster name within the Finout app.</td><td><strong>Required</strong></td><td>None</td><td>Identifies the metrics source cluster.</td></tr></tbody></table>

<table data-header-hidden><thead><tr><th width="132.58984375">Category</th><th width="134.17578125">Variable</th><th>Description</th><th width="115.0390625">Required / Optional</th><th width="89.4375">Default Value</th><th>Notes</th></tr></thead><tbody><tr><td>Auth &#x26; Identity</td><td><strong>ROLE_ARN</strong></td><td>ARN for IAM role to assume for authorization.</td><td>Optional</td><td>None</td><td>Used if assuming a role instead of direct access.</td></tr><tr><td>Auth &#x26; Identity</td><td><strong>ROLE_EXTERNAL_ID</strong></td><td>External ID for assumed role.</td><td>Optional</td><td>None</td><td>Only needed if the IAM role requires an external ID.</td></tr><tr><td>Storage &#x26; Paths</td><td><strong>S3_BUCKET</strong></td><td>Customer’s S3 bucket to store the exported metrics.</td><td><strong>Required</strong></td><td>None</td><td><p>Must exist in customer’s environment.</p><p></p></td></tr><tr><td>Storage &#x26; Paths</td><td><strong>S3_PREFIX</strong></td><td>S3 prefix where metrics will be stored.</td><td><strong>Required</strong></td><td>None</td><td><p>For example, k8s/prometheus</p><p>has to be the same s3_prefix if multiple per cluster integrations within the same cost center config.</p></td></tr><tr><td>Endpoint &#x26; Connectivity</td><td><strong>SCHEME</strong></td><td>The protocol used when calling the Prometheus metrics endpoint: either <code>https</code> or <code>http</code></td><td>Optional</td><td><code>http</code> </td><td>-----</td></tr><tr><td>Endpoint &#x26; Connectivity</td><td><strong>PATH_PREFIX</strong></td><td>Optional sub-path between host/port and the Prometheus metrics API.</td><td>Optional</td><td>None</td><td>-----</td></tr><tr><td>Endpoint &#x26; Connectivity</td><td><strong>HOSTNAME</strong></td><td>The Prometheus compatible API endpoint.</td><td>Optional<br></td><td><code>localhost</code></td><td>Must be reachable from the pod running the Finout exporter cronjob.<br></td></tr><tr><td>Endpoint &#x26; Connectivity</td><td><strong>PORT</strong></td><td>API port</td><td>Optional</td><td><code>9090</code></td><td>Standard Prometheus port.</td></tr><tr><td>Query Window &#x26; Data Volume</td><td><strong>TIME_FRAME</strong></td><td>Time range per query in seconds.</td><td>Optional</td><td><code>3600</code></td><td>Lower values reduce query load and risk of OOM.</td></tr><tr><td>Query Window &#x26; Data Volume</td><td><strong>BACKFILL_DAYS</strong></td><td>Days of historical data to fetch on first run.</td><td>Optional</td><td><code>3d</code></td><td>Large values increase load and risk of slow queries.</td></tr><tr><td>Query Window &#x26; Data Volume</td><td><strong>INITIAL_BACKFILL_DATE</strong> </td><td>Defines the initial backfill start date</td><td>Optional</td><td>None</td><td><p></p><ul><li>Available starting from the <a href="prometheus-metrics-exporter-release-notes">Exporter’s v2.0.0 </a></li><li>value should be in YYYY-MM-DD format.</li><li>limited to maximum 3 days.</li></ul></td></tr><tr><td>Query Window &#x26; Data Volume</td><td><strong>QUERIES_BATCH_SIZE</strong></td><td>Controls how many queries are processed per batch to manage memory usage.</td><td>Optional</td><td>5</td><td><p></p><ul><li>Available starting from the <a href="prometheus-metrics-exporter-release-notes">Exporter’s v2.0.0 </a></li><li><strong>Default is 5</strong>. Lowering it reduces memory usage but may increase overall runtime.</li><li><strong>Minimum is 1</strong> </li><li>If a customer sees <strong>OOM kills / memory pressure</strong>, recommend reducing QUERIES_BATCH_SIZE before changing anything else.</li></ul></td></tr></tbody></table>

To configure these fields, add them to the configuration file under the env section and use the name/value format for each desired field.

For example:

```yaml
env:
  - name: TIME_FRAME
    value: "3600"
  - name: BACKFILL_DAYS
    value: "3d"
```

Ensure that the field names and the corresponding values are correctly specified to apply the desired configuration.

4. Run the command in a namespace of your choice, preferably the one where the Prometheus stack is deployed: ​`kubectl create -f <filename>`
5. Trigger the job (instead of waiting for it to start):\
   `kubectl create job --from=cronjob/finout-prometheus-exporter-job finout-prometheus-exporter-job -n <namespace>`

   The job automatically fetches Prometheus data from 3 days ago up to the current time.

   <div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p><strong>Note</strong>: This can be changed by modifying your <code>BACKFILL_DAYS</code> variable.</p></div>
6. Click **Save**.\
   The Prometheus cost center is created.

## 5. Select Cost Centers

<figure><img src="https://3858159242-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWqjB2puKXPDR7L86FX2e%2Fuploads%2FO9fzyRKlbHsv96LjpVJK%2Fimage.png?alt=media&#x26;token=2a12f8b0-4b3d-4970-a3f3-94eeab998997" alt=""><figcaption></figcaption></figure>

1. Select the cost centers that this integration will enrich with Kubernetes metrics. These cost centers can then attribute Kubernetes costs for supported services (EKS, GKE, and AKS) by namespace, workload, and label.
2. Click **Complete Integration**.\
   The cost center will be created in about 48 hours.<br>

   <figure><img src="https://3858159242-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWqjB2puKXPDR7L86FX2e%2Fuploads%2FcjQ0QjNtkQYVsuFx4fOQ%2Fimage.png?alt=media&#x26;token=219a776c-025f-437b-b9c6-2f50cd15afdf" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
**Note**:&#x20;

* If this cost center is already enriched by a Kubernetes integration, ensure there are no overlapping nodes between the integrations to avoid duplicated costs and resources.
* You can hover over every created cloud cost center and see all the Kubernetes cost centers that enrich it.<br>

  <figure><img src="https://3858159242-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWqjB2puKXPDR7L86FX2e%2Fuploads%2FCeKySfVSrCoxamJ1Nv8P%2Fimage.png?alt=media&#x26;token=8dface99-81fe-4b43-a9bc-f17c4acc97e3" alt=""><figcaption></figcaption></figure>

{% endhint %}

## 6. Validate Your Kubernetes Integration

**Confirm that your Prometheus Integration is working correctly:**

#### S3 Validation

To confirm that Prometheus metrics are being exported correctly to your S3 bucket:

1. **Navigate to the S3 path**, for example:\
   `s3://cur-bucket/k8s/prometheus/prod-cluster/end=20251101/day=5/`\
   You should see a list of metric folders, such as: `metric=cpu_requests/`
2. **Open a metric folder** and verify that `.json.gz` files are uploaded. Each file should have a timestamp prefix, for example:\
   `s3://cur-bucket/k8s/prometheus/prod-cluster/end=20251101/day=5/metric=cpu_requests/1759622400_cpu_requests.json.gz`\
   If these files appear, the CronJob ran successfully, and Prometheus metric files were generated and stored in the correct S3 structure.

#### Data Availability

Kubernetes cost and usage data will appear across Finout within 48 hours, matching the standard cloud billing data delivery window.

{% hint style="info" %}
**Note**: If any issues occur, share your exporter logs with Finout support at <support@finout.io> for further investigation.
{% endhint %}

For more information, please see the [FAQs](https://docs.finout.io/kubernetes-integrations/kubernetes/prometheus/prometheus-faqs) and [Troubleshooting](https://docs.finout.io/kubernetes-integrations/kubernetes/prometheus/prometheus-troubleshooting) section.\
For new updates to the Prometheus Exporter introduced between versions and their impact on cost calculation, refer to the [release notes](https://docs.finout.io/kubernetes-integrations/kubernetes/prometheus/prometheus-metrics-exporter-release-notes).