# Cost Archiver

## Overview

Finout offers a straightforward solution for managing long-term cost data storage by allowing you to archive data in an AWS S3 bucket you own. This setup consolidates your company’s cloud spending data from various cost centers into a structured and easily accessible format.

The Archiver feature enhances data management by introducing a partitioned data structure in S3, enabling faster access with reduced processing overhead. The data format is in Parquet, which improves processing efficiency. Additionally, a manifest file is included with each archiver run, providing visibility into data export status. This design supports a scalable system capable of handling large data volumes with minimal delays while maintaining alignment with Finout’s data delivery SLAs.

{% hint style="info" %}
**Note**: The Archiver provides all fields, enrichments, costs, virtual tags, and values, *except for virtual tag reallocation*.
{% endhint %}

**Cost Archiver Workflow:**

1. [Create a New Bucket](#id-1.-create-a-new-bucket)
2. [Obtain an External ID from Finout](#id-3.-obtain-an-external-id-from-finout)
3. [Grant Finout Access to your Archiver Bucket](#id-4.-grant-finout-access-to-your-cur-bucket)
4. [Add AWS Bucket Details to Finout](#id-5.-adding-aws-bucket-details-to-finout)

### Archiver files structure

Cost data will be updated once a day in the following structure: 

Archiver/\<archiver\_name>/v2/year=\<year>/month=\<month>/day=\<day>/\*.snappy.parquet

### Cost data structure

| Field name         | Type                            | Description                                                                                                                                                                               |
| ------------------ | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| time               | Date (YYYY-MM-DDT00:00:00.000Z) |                                                                                                                                                                                           |
| service            | String                          | The type of service the cost is related to, for example, “EC2” (in AWS) or “Compute Engine” (in GCP).                                                                                     |
| region             | String                          | The region the cost is associated with.                                                                                                                                                   |
| cost\_center\_type | String                          | AWS / GCP / K8s / Snowflake                                                                                                                                                               |
| costs              | dict\<string, double>           | All the following costs: amortized\_cost, net\_amortized\_cost, blended\_cost, unblended\_cost, net\_unblended\_cost, uncovered\_cost, reservation\_cost, savings\_plan\_cost, spot\_cost |
| metadata           | dict\<string, string>           | All metadata relevant to this cost line.                                                                                                                                                  |

### Manifest file structure &#x20;

The new manifest file will provide transparency and allow clients to monitor data export consistency. This file will be maintained under destination-archiver/version=2/manifest.json and will document each archiver run, detailing the date and scope of data exported, run ID, and the export timestamp.

```
{
"latest_export_time": 
"exports": [
	{	
		"archiver_id": "<archiver_id>", 
		"run_id": "<external_exporter_run_id>",
		"export_time": "<Current time in UTC in ISO format>", 
		"expored_partitions": ["<year=yyyy>/month=<mm>/day=<dd>"]
		},
	{	
			"archiver_id": "<archiver_id>", 
			"run_id": "<external_exporter_run_id>",
			"export_time": "<Current time in UTC in ISO format>",
		"	expored_partitions": ["<year=yyyy>/month=<mm>/day=<dd>"]
	}	,
		....
]	
}
```

## Setting Up Cost Archiver

Set up Cost Archiver in Finout to efficiently store and manage your organization’s cost data in an AWS S3 bucket, enabling structured, scalable, and easily accessible long-term storage.

### 1. Create a New Bucket

To output cost data, Finout requires write and delete permissions to an S3 bucket. To do so, create a new bucket or a new folder within an existing S3 bucket used for your Archiver.

{% hint style="info" %}
**Note**: Save your bucket name for step 3.
{% endhint %}

### 2. **Obtain an External ID** from Finout

Get the external ID to Grant Finout Access to Your Archiver Bucket.

1. In Finout, navigate to **Settings** > **Cost Centers** > **Add cost center**.\
   The **Connect Accounts** page appears.<br>

   <figure><img src="https://3858159242-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWqjB2puKXPDR7L86FX2e%2Fuploads%2FivN16BJIVqnl4N8pAHxa%2Fimage.png?alt=media&#x26;token=aa3208b1-0e72-48c6-a529-f4cc4042a428" alt=""><figcaption></figcaption></figure>
2. Click **Connect Now**.<br>

   <figure><img src="https://3858159242-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWqjB2puKXPDR7L86FX2e%2Fuploads%2Fe4cR0Q9D0eQBhA7QnQ3c%2Fimage.png?alt=media&#x26;token=81ef9299-4636-4366-b536-4816fb5739ef" alt=""><figcaption></figcaption></figure>
3. &#x20;Copy the **External ID** and continue to grant Finout access to your Archiver bucket (Save for step 4).&#x20;

### 3. Grant Finout Access to your Archiver Bucket

Once the Archiver bucket is created, grant Finout access to the bucket by creating an IAM role. This can be done manually or by using CloudFormation.

{% hint style="info" %}
**Note**: It is recommended to grant access through CloudFormation.
{% endhint %}

{% hint style="success" %}
**Prerequisite**: Obtain an External ID from Finout. (Step 2)
{% endhint %}

**To grant access using CloudFormation:**

1. Create a CloudFormation Stack after logging to your  [AWS account](https://eu-north-1.signin.aws.amazon.com/oauth?client_id=arn%3Aaws%3Asignin%3A%3A%3Aconsole%2Fcloudformation\&code_challenge=9at2qlfz2hoSdir6XSHnxUhVnR8-vrS0hI-RCxafu2o\&code_challenge_method=SHA-256\&response_type=code\&redirect_uri=https%3A%2F%2Fconsole.aws.amazon.com%2Fcloudformation%2Fhome%3FhashArgs%3D%2523%252Fstacks%252Fquickcreate%253Fparam_OrgEnabled%253DDisabled%2526param_OrgId%253D%2526param_ExternalID%253D%2526param_BucketName%253D%2526param_BucketPrefix%253D%2526stackName%253DFinout-Archiver-Connector%2526templateURL%253Dhttps%253A%252F%252Ffinout-public-assets.s3.us-east-1.amazonaws.com%252Farchiver%252Ffinout-archiver-connector-org.json%26isauthcode%3Dtrue%26oauthStart%3D1739268570916%26state%3DhashArgsFromTB_eu-north-1_50753c08f8ab9b15). <br>

   <figure><img src="https://lh7-rt.googleusercontent.com/docsz/AD_4nXd23d9Im4UcWtiagKhpMhgiH6TFeTiQntL22TN91DZtbgGP0sAsVS_J9U619_KtVsoDYSetQ5j7YG0zHBvD50rLqEQG1M80D091sPNffUyrYVDeFzkn8HRnkZDbYa4x2GUJyrNyvw?key=pacRQa4h1RWT_wxFaHKrY-C5" alt=""><figcaption></figcaption></figure>

   <figure><img src="https://3858159242-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWqjB2puKXPDR7L86FX2e%2Fuploads%2FKvqnLXV1CMzzbFtb7kYf%2Fimage.png?alt=media&#x26;token=acafa99e-e331-43c4-ac5b-fa8e8f926002" alt=""><figcaption></figcaption></figure>
2. Add the necessary information above, as well as the “external-id” (obtained in step 2) and the bucket name created for your Archiver (step 1).
3. Click **Next** and then **Submit**.\
   ​You are brought to the **Stack details page**.

   <figure><img src="https://finout.intercom-attachments.eu/i/o/16684371/172e0ce42bbed5ff50806cf1/AD_4nXc4icyPw8yeqwXIw6JeQdh7aaiAVQhQxHX9O-343TphGeFHmfhiFL4Bo-4lijpPnStHfabkb2SUvLn9Ovqn9rb2bVyojKE_4bUa2xlrwwfAPXItIhtnkX3zn1dDwTUVThO9dltSKZlXi0jajl5KZz1M2esQ?expires=1725348600&#x26;signature=e982db55d17db47f9efa522b7f2641fb8ac4d0b7b105dff7f3f04bc0ad0dfaf3&#x26;req=0dNpzVz5qzZk2hL085ZhoZeQw%2BseQEewACUAhEgjwraHInrQ3OnJZYbEe%2B3E%0AEg%3D%3D%0A" alt=""><figcaption></figcaption></figure>
4. Click **Output** and copy the ARN IAM role value to add in Finout (step 4).

**To grant access manually:**

1. Navigate to creating a new cross-account role in IAM , sign in the your[ AWS account](https://eu-north-1.signin.aws.amazon.com/oauth?client_id=arn%3Aaws%3Asignin%3A%3A%3Aconsole%2Fcloudformation\&code_challenge=9at2qlfz2hoSdir6XSHnxUhVnR8-vrS0hI-RCxafu2o\&code_challenge_method=SHA-256\&response_type=code\&redirect_uri=https%3A%2F%2Fconsole.aws.amazon.com%2Fcloudformation%2Fhome%3FhashArgs%3D%2523%252Fstacks%252Fquickcreate%253Fparam_OrgEnabled%253DDisabled%2526param_OrgId%253D%2526param_ExternalID%253D%2526param_BucketName%253D%2526param_BucketPrefix%253D%2526stackName%253DFinout-Archiver-Connector%2526templateURL%253Dhttps%253A%252F%252Ffinout-public-assets.s3.us-east-1.amazonaws.com%252Farchiver%252Ffinout-archiver-connector-org.json%26isauthcode%3Dtrue%26oauthStart%3D1739268570916%26state%3DhashArgsFromTB_eu-north-1_50753c08f8ab9b15), and then create a role for another AWS account.\ <br>

   <figure><img src="https://3858159242-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWqjB2puKXPDR7L86FX2e%2Fuploads%2FCovOaA9rXZNZ0YQ0MZMA%2Fimage.png?alt=media&#x26;token=9d1dee61-535b-4e92-8611-5277d6cd5656" alt=""><figcaption></figcaption></figure>

2. In the account ID, enter: 277411487094.

3. Paste the **External ID** and enter the “`external-id`” (Obtained in step 2). <br>

   <div align="left"><figure><img src="https://lh7-rt.googleusercontent.com/docsz/AD_4nXePL6_zRXTBVitm7ZYa3Oj16FreqlHrOCNXIrEqeVQOvn-O-CefHivic3st4ogy62aCHluJRVMzgtwFawj0BKLW0yzi2p-smrQ82LsWH0pVyE4y9pDxwpW3N5_ZKIjf-ZYil-Po?key=pacRQa4h1RWT_wxFaHKrY-C5" alt=""><figcaption></figcaption></figure></div>

4. Click **Next**. ​\
   The **Review** step appears.<br>

   <figure><img src="https://lh7-rt.googleusercontent.com/docsz/AD_4nXf9TkemkmHaNpd4bFhDtE9ckxGLKWWHJbsYnVRyJDRMK4Vn3M4HUKVr3VX7CUSv4qdQ-ydz9zJZAdLVDiUCaAHLDhof6e83UfQRsDExALtEoEAsinHIjrcX2G9YhckjxcqslL8FUw?key=pacRQa4h1RWT_wxFaHKrY-C5" alt=""><figcaption></figcaption></figure>

5. Add a Role name: `FinoutArchiverRole` and then configure the role. \
   A new role is created.

6. Go to your new role in **Summary**.<br>

   <figure><img src="https://3858159242-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FWqjB2puKXPDR7L86FX2e%2Fuploads%2F6Tu6p4tWZ798lAjTJGGy%2Fimage.png?alt=media&#x26;token=036585b1-4ea8-460a-8c00-8a86a583b22d" alt=""><figcaption></figcaption></figure>

7. Copy and save the **Role ARN**.

8. Click on **Add permissions** and choose **Create inline policy**.\
   Choose JSON format and paste the following JSON:

   ```
     {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject",
                   "s3:PutObjectAcl",
                   "s3:ListMultipartUploadParts",
                   "s3:ListBucket",
                   "s3:DeleteObject"
               ],
               "Resource": [
                   "arn:aws:s3:::finout-data-archiver/*",
                   "arn:aws:s3:::finout-data-archiver"
               ]
           }
       ]
   }

   ```

9. Click **Next** until the Review step, and name it `finout-access-policy`.

10. Click **Create policy**.\
    Your IAM role is finalized and created.

### 4. Adding AWS Bucket Details to Finout

After creating your bucket in AWS and granting Finout access, send the following bucket details to Finout support.

1. **Role ARN** -The Amazon Resource Name (ARN) specifies the role.&#x20;
2. **Bucket Name** -This is the name under which Finout will write archiver data.
3. **S3 Path Prefix** - This is the folder in S3 in which the archiver files are written.

## FAQs

**How does Finout ensure data accuracy and reliability with the manifest file?**

Finout ensures data accuracy and reliability through automated systems that monitor and reprocess any failed exports, eliminating the need for manual intervention. While the manifest file enhances transparency by providing visibility into data export processes, Finout’s robust validation mechanisms ensure consistent and accurate data without requiring clients to check the manifest manually.

**How does Cost Archiver handle data updates over time?**

Cost Archiver  is designed to handle retrospective data updates, which are common with sources like AWS CUR that may reflect changes for several months. It ensures that past data is updated to include late adjustments, with the manifest file clearly indicating which dates have been modified. Finout’s automation ensures data completeness and reliability throughout this process.

**What mechanisms are in place for maintaining data consistency with partitioned files?**

Cost Archiver performs automatic checks to identify and update partitioned files whenever source data changes. Finout updates relevant partitions and logs these updates in the manifest file.  This approach guarantees consistency and integrity across the partitioned data structure.

<br>


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.finout.io/billing-integrations/cloud-providers/connect-to-aws/cost-archiver.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.