Connect CostGuard for GCP

Overview

To use Finout's CostGuard for GCP, Finout needs access to your GCP resource's operational metrics.

Note: Finout uses a 'Monitoring Viewer' role - A read-only permissions role and will not access any sensitive data.

This guide assumes that your account is already integrated with GCP as a cost center and that Finout has access to your GCP billing data. If this is not the case, please refer to the GCP onboarding documentation.

The following steps provide instructions for granting Finout access to the necessary metrics, which will enable CostGuard to generate recommendations. The documentation is organized into two sections: one for onboarding an existing project and another for onboarding a new project.

How the Finout Integration Works

1. Finout utilizes a GCP scoping project, granting it access to essential metrics across all relevant projects in your GCP account, with read-only permissions.

   Note: A scoping project is a GCP project configured to access metric scopes from all projects that require monitoring.

2. You need to enable the Finout service account, which is responsible for monitoring your GCP billing, to access the metric scopes of the scoping project.

3. For this seamless integration, Finout has developed a Terraform script specifically designed for this purpose.

Functionality of Finout's Terraform Script

• Specifically applicable to new project creation: The Terraform script initiates by creating a new project within your account, designated as the scoping project, following GCP's recommended practices.

• Enabling monitoring API: It activates the Stackdriver Monitoring API for the newly created scoping project.

• Binding Finout's billing service account to the scoping project: The script assigns the Finout billing service account to the scoping project with the 'Monitoring Viewer' role.

• Linking monitored projects to the scoping project: It facilitates the binding of either all or selected projects in your account to the scoping project for monitoring purposes.

• Granting Finout access to monitored projects: The script assigns the Finout billing service account to all monitored projects, including the scoping project, with the 'Compute Viewer' role.

• Enabling compute engine API: It ensures the activation of the Compute Engine API for all monitored projects, which includes the scoping project as well.

Important: To run the Terraform script, the user must have an Editor role within GCP.

Onboarding an Existing Project

1. Clone Finout’s shared repository- git clone:

   Execute [e-mail]:finout-io/finout-onboarding.git to clone Finout’s shared repository.

2. Terraform script usage:

   Use the following Terraform script: gcp/cost-guard/existing-project-onboarding

3. Edit the Terraform state file configuration:

   Modify gcp-cost-guard-onboarding-existing-project/provider.tf file to save the Terraform state file.

   
4. Set up a bucket name:

   In <BUCKET_NAME> input the name of the bucket for storing Terraform state files. If it does not exist, create a new bucket.

5. Configure env.tfvars:

   Edit gcp-cost-guard-onboarding-existing-project/profiles/env.tfvars as follows:

   Use an existing project as the “scoping project” and bind Finout service account to this project:

   a. Edit the profiles/env.tfvars as follows:

   

   b. Run the script with additional parameters:

   • <FINOUT_SERVICE_ACCOUNT> -Retrieve the Finout service account details. Learn how to here.
   • <SCOPING_PROJECT_ID> - Add the existing project ID. Learn how to here.
   • <ADD_MONITORED_PROJECTS> - Add monitored projects to the scoping project. The default value is ‘false’. If you intend to add new projects to the monitoring scope, please change the value to ‘true’ and continue to the section below.

   • <MONITORED_PROJECTS> - Specify which projects will be monitored:

     1. If you wish to define all the organization projects as monitored projects - leave an empty list.

     2. If you wish to define specific projects - fill the "Monitored projected" list with the specific project IDs.

        • Defining monitored projects in this list will not affect the current projects that are being monitored.

6. Run the Terraform script.
7. Validate that the integration is working correctly:

   1. In the GCP Console, check the scoping project under IAM to ensure Finout's service account has the 'Monitoring Viewer' role.

   2. Under Monitoring in the scoping project, verify that all intended projects are added to the Metric Scope.

      
8. Send the scoping project ID and account service to Finout.

   Note: To send the project ID, please reach out to Finout's support team.

Additional help: setting up and running the Finout integration

Get a service account principal

To bind the Finout service account to the scoping project using the Terraform script, we need to be provided with the service account principle:

1. Navigate to the Google Console.

2. Choose IAM & Admin.

   
3. Provide Finout’s service account principal.

   

Get your “scoping project ID”

1. Go to the GCP console to locate your project.

2. Copy its ID (project_id).

   

Script functions

1. The script enables monitoring API (Stackdriver Monitoring API) for the scoping project.
2. Employ Finout’s service account with billing query permissions.

3. Adds 'Monitoring Viewer' role permissions to the desired project.

4. Binds projects as monitored to the scoping project, either fully or partially.

5. Links Finout's billing service account with the 'Compute Viewer' role to all monitored projects, including the scoping project.

6. Enables the compute engine API for all monitored projects, including the scoping project.

Running the script

To run the Terraform script, you must have an Editor role.

1. Login to your GCP Cloud with gcloud auth application-default login.

2. Run the script with the following commands:

• terraform init

• terraform workspace select <workspace>

• terraform apply -var-file "profiles/<environment>.tfvars"

Adding new monitored projects

New projects are not automatically added to the Finout scoping project.

After executing the script, you have the option to add more projects as monitored projects based on your preferred method:

1. If all the organization’s projects are to be monitored, leave the list empty and rerun the script. This will include any new projects automatically.

2. If specific projects are chosen, add their IDs to the “Monitored Projects” list.

   Note: Projects not included will not be monitored.

Onboarding a New Project

1. Clone Finout’s shared repository- git clone: git clone [e-mail]:finout-io/finout-onboarding.git

2. Terraform script usage:

   Use the following Terraform script: gcp/cost-guard/new-project-onboarding

3. Edit the Terraform state file configuration:

   Modify gcp-cost-guard-onboarding-new-project/provider.tf file to save the Terraform state file.

   
4. Set up a bucket name:

   In <BUCKET_NAME> - Input the name of the bucket for storing Terraform state files. If it does not exist, create a new bucket.

5. Configure env.tfvars:

   Edit gcp-cost-guard-onboarding-new-project/profiles/env.tfvars as follows:

   Create a new “scoping project” and bind the Finout service account to this project:

   a. Edit the profiles/env.tfvars as follows:

   

   b. Run the script with additional parameters:

   • <FINOUT_SERVICE_ACCOUNT> - Retrieve the Finout service account details. Learn how to here.
   • <SCOPING_PROJECT_ID> - Add a new ID for the scoping project to be created. Default new project id: finout-scoping-project-xxxxxx.

   • <ORG_ID> - Retrieve the organization ID. Show me how.
   • <SCOPING_PROJECT_NAME> - Define a new name for the scoping project to be created. Default new project name: finout-scoping-project.

   • <MONITORED_PROJECTS> - Specify which projects will be monitored:

     1. If you wish to define all the organization projects as monitored projects - leave an empty list.

     2. If you wish to define specific projects - fill the "Monitored projected" list with the specific project IDs.

6. Run the Terraform script.
7. Validate that the integration is working correctly:

   1. In the GCP Console, check the scoping project under IAM to ensure Finout's service account has the 'Monitoring Viewer' role.

   2. Under Monitoring in the scoping project, verify that all intended projects are added to the Metric Scope.

      
8. Send the scoping project ID and account service to Finout.

   Note: To send the project ID, please reach out to Finout's support team.

   Important: This script is designed for onboarding one organization at a time. For multiple organizations, you will need to run the script separately for each organization requiring monitoring.

Additional help: setting up and running the Finout integration

Get a service account principal

To bind the Finout service account to the scoping project using the Terraform script, we need to be provided with the service account principle:

1. Navigate to the Google Console.

2. Choose IAM & Admin.

   
3. Provide Finout’s service account principal (see attached screenshot).

   

Retrieve the organization ID

1. In the Google console, click on the Google Cloud icon in the left upper corner.

   
2. Click on the organization name.

   
3. Copy the organization ID.

   

Script functions

1. The script creates a new project as a scoping project.

2. Enables the monitoring API for the scoping project.
3. Employ Finout’s service account with billing query permissions.

4. Adds 'Monitoring Viewer' role permissions to the desired project.

5. Binds projects as monitored to the scoping project.

How to run the script

To run the Terraform script, you must have an Editor role within GCP.

1. Login to your GCP Cloud with gcloud auth application-default login

2. Run the script with the following commands:

• terraform init

• terraform workspace select <workspace>

• terraform apply -var-file "profiles/<environment>.tfvars"

Assigning Compute Viewer Role to the Service Account

To access the metadata of resources, like underutilized persistent disks, the service account must have the 'Compute Viewer' role.

Adding new monitored projects

New projects are not automatically added to the Finout scoping project.

After executing the script, you have the option to add more projects as monitored projects based on your preferred method:

1. If all the organization's projects are to be monitored, leave the list empty and rerun the script. This will include any new projects automatically.

2. If specific projects are chosen, add their IDs to the “Monitored Projects” list.

   Note: Projects not included will not be monitored.

Required Permissions for GCP Monitoring

cloudnotifications.activities.list

monitoring.alertPolicies.get

monitoring.alertPolicies.list

monitoring.dashboards.get

monitoring.dashboards.list

monitoring.groups.get

monitoring.groups.list

monitoring.metricDescriptors.get

monitoring.metricDescriptors.list

monitoring.monitoredResourceDescriptors.get

monitoring.monitoredResourceDescriptors.list

monitoring.notificationChannelDescriptors.get

monitoring.notificationChannelDescriptors.list

monitoring.notificationChannels.get

monitoring.notificationChannels.list

monitoring.publicWidgets.get

monitoring.publicWidgets.list

monitoring.services.get

monitoring.services.list

monitoring.slos.get

monitoring.slos.list

monitoring.snoozes.get

monitoring.snoozes.list

monitoring.timeSeries.list

monitoring.uptimeCheckConfigs.get

monitoring.uptimeCheckConfigs.list

opsconfigmonitoring.resourceMetadata.list

resourcemanager.projects.get

resourcemanager.projects.list

stackdriver.projects.get

stackdriver.resourceMetadata.list

Main Metrics

compute.googleapis.com/instance/cpu/utilization

compute.googleapis.com/instance/network/received_bytes_count

compute.googleapis.com/instance/network/sent_bytes_count

cloudsql.googleapis.com/database/memory/usage

cloudsql.googleapis.com/database/network/connections