# S3 Telemetry Integration
Integrating your telemetry data into Finout using a S3 bucket of your choosing, requires setting up a consistent export process to an S3 bucket, for which Finout has been granted read access. This ensures that Finout can retrieve and process your data, which should be structured to include a daily timestamp, the telemetry value for each day, and the granularity of the data collected.
#### How it works
Finout ingests telemetry data from an S3 bucket that you manage. The S3 endpoint grants Finout read-only access to specific files in your AWS account. Telemetry files must be exported on a consistent schedule and include a daily timestamp, the telemetry value for each day, and the data’s granularity.
Finout supports two ingestion patterns, depending on how you organize telemetry data in S3:
* **Multiple S3 buckets (multiple endpoints):**\
Use this option if you want to separate telemetry data across different buckets (for example, by team, environment, or data source). Each S3 bucket is configured as a separate **endpoint** in Finout and requires its own access permissions.
* **Single S3 bucket with multiple files (single endpoint):**\
Use this option if you prefer to centralize telemetry data in one bucket. Finout reads multiple CSV files from a specified folder path within the bucket. The bucket and folder path together are treated as a single **endpoint**, provided the correct permissions are configured.
**S3 Telemetry Configuration Workflow:**
1. [Generate CSV Export](#h_52fa740f9d)
2. [Prepare CSV Format](#h_df5b422546)
3. [Create a Folder in the Bucket](#h_b1310b2af3)
4. [Obtain External ID from Finout](#h_17fc5d10a0)
5. [Set up an S3 Bucket](#h_17fc5d10a0)
6. [Provide Data to Finout](#h_9e63e58804)
## 1. Generate CSV Export
Generate a daily CSV export by selecting from two approaches for your CSV exports:
1. Maintain a single CSV that receives updates daily with new data entries.
2. Create a new CSV file for each day's telemetry data.
We recommend opting for the second option, which is creating a distinct CSV file for each day's telemetry data.
## 2. Prepare CSV Format
When preparing your telemetry data for Finout, ensure your CSV files follow the required format, which includes three main column types: Date, Metadata, and Telemetry.
| **Type of column** | **How many columns can be used?** | **Mandatory/Optional** | **What is this used for?** | **Format / Notes** |
| ----------------------- | --------------------------------- | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- |
| **Date** | Single | Mandatory | Each data row input represents the telemetry for a specific date, accompanied by their metadata values, indicating the day on which the telemetry was recorded. |
YYYY-MM-DD
“date” column
|
| **Metadata (multiple)** | 0-10 | Optional | Any tagging attributes can be utilized to group and filter the telemetry data based on metadata such as team, group, or feature. | Optional |
| **Telemetry** | 1 | Mandatory | Represent the total of the telemetry for the given date and metadata. | Positive number |
{% hint style="danger" %}
**Important**:
1. The CSV must include at least two specific columns: date and \.
2. The Telemetry column name will be used as the unique identifier for this telemetry within Finout.
3. Column headers must not include spaces.
{% endhint %}
**Example**:
In this example, we will supply the mandatory columns date (“date") and telemetry (“bytes") together with an optional metadata column “teamname”.
"date","teamname","bytes"
"2022-06-07","Servers","12.12"
"2022-06-07","Storage","6.22"
"2022-06-08","Network","0.05"
Date = date
teamname = metadata
Bytes = telemetry
## 3. Create a Folder in the Bucket
Create a separate folder within the bucket for each telemetry integration. This setup allows you to easily reuse permissions for sending other telemetry using the same integration.
{% hint style="warning" %}
**Important - Before you continue:**
* If you *don’t have an endpoint* or want to *create additional endpoints*, continue to the next step.
* If you *already have an endpoint and want to add a file* to it, contact Finout Support and provide the new folder's prefix name. Finout will configure the new telemetry using the existing endpoint.
{% endhint %}
## 4. Obtain External ID from Finout
1. Copy your “external-id” from the Finout console, or use a random one, preferably in the format of finout-XXXXXXX.
2. Click on[ create a new cross-account role in IAM](https://console.aws.amazon.com/iam/home?region=us-east-1#/roles%24new?step=type\&roleType=crossAccount) to create a role for another AWS account.
3. Enter the **AWS Account ID** associated with the S3 bucket in the Account ID field.
4. Choose the option to **Require external ID** and input the “external-id” you got from the Finout console.
5. Click **Next** until you reach the review screen.
6. Configure the review as shown below, with one exception: the role name should be: FinoutMetricsReadOnlyRole (unlike in the screenshot below).
7. Go to your newly created role.
8. Copy the **Role ARN** and paste it into the Finout console.
9. Click **Add permissions** and select **Create inline policy**.
10. Select JSON, and insert the following. Replace the \ with the name of your newly created bucket or your existing CUR bucket:
```json
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"tag:GetTagKeys"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": "arn:aws:s3:::/*"
},
{
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": "arn:aws:s3:::"
}
]
}
```
11. Click Next until you reach the review screen, and name the policy finest-access-policy\_telemetry\_export (or choose your own name).
12. Navigate to **Settings > Endpoints**. \
The Endpoint page appears.
13. Click **Add Endpoint**, then select **Amazon S3**. \
The **Add Amazon S3 Bucket Endpoint** page appears.\
14. Bucket Access - Choose one of the following options:
1. Read Only - If you only want Finout to read the S3 bucket, for example, if you want to add telemetry.
2. Read and Write (Coming Soon) - If you want Finout to read and write in the S3 bucket. For example, if you are adding an archiver.
15. Copy the External ID.
## 5. Set up an S3 Bucket
{% hint style="warning" %}
**Important**: If you already have an Amazon S3 bucket connected to Finout, you can skip to **Step 3**. If you have not yet set up an S3 bucket, or if you want to connect additional S3 buckets within your organization, continue with this step.
{% endhint %}
1. Click on[ create a new cross-account role in IAM](https://console.aws.amazon.com/iam/home?region=us-east-1#/roles%24new?step=type\&roleType=crossAccount) to create a role for another AWS account.
2. Enter the **AWS Account ID** associated with the S3 bucket in the Account ID field.
3. Choose the option to **Require external ID** and input the “external-id” you got from the Finout console.
4. Click **Next** until you reach the review screen.
5. Configure the review as shown below, with one exception: the role name should be: FinoutMetricsReadOnlyRole (unlike in the screenshot below).
6. Go to your newly created role.
7. Copy the **Role ARN** and paste it into the Finout console.
8. Click **Add permissions** and select **Create inline policy**.
9. Select JSON, and insert the following. Replace the \ with the name of your newly created bucket or your existing bucket:
```json
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"tag:GetTagKeys"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": "arn:aws:s3:::/*"
},
{
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": "arn:aws:s3:::"
}
]
}
```
10. Click Next until you reach the review screen, and name the policy finest-access-policy\_telemetry\_export (or choose your own name).
## 6. Add Bucket Details to Finout
1. Endpoint Name and Description.
2. Add the following fields:
1. Add the **Role ARN.** The Amazon Resource Name (ARN) specifies the role.
2. Add the **Bucket Name**. This is the name under which AWS stores your cost and usage reports.
3. Add the **S3 Path Prefix** (Optional). This is the folder in S3 in which the files are located.
4. Add the **Region**.
3. Click **Test Endpoint** to create a test.
4. Click **Add Endpoint**.
5. Contact Finout Support and inform them that the endpoint was created.\
**Result**:\
The Amazon S3 Bucket endpoint is created and appears in the endpoints list.
---
# Agent Instructions: Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://docs.finout.io/telemetry-integrations/telemetry/s3-telemetry-integration.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
