# AWS EC2-GPU Cost Optimization Scans (Alpha) ## Overview GPU resources are among the most expensive in cloud infrastructure - yet GPU waste is largely invisible in billing data. AWS embeds GPU cost into EC2 instance pricing with no dedicated cost dimension, making underutilization hard to detect without usage metrics. Finout ingests GPU metrics from CloudWatch and runs scans in CostGuard to surface idle and GPU-backed EC2 instances that can be shut down or rightsized. ## Setup GPU usage metrics are not emitted by AWS CloudWatch by default. The CloudWatch Agent must be installed and configured on each target EC2 instance before Finout can ingest GPU metrics. Finout does not install agents or modify instance configurations. Once GPU metrics are present in your CloudWatch environment, Finout ingests them automatically, no additional configuration is required inside Finout. **This part covers what you need to configure in your AWS environment before GPU cost optimization scans can run in Finout.** ### Prerequisites Before you begin, confirm the following: * Your EC2 instances use NVIDIA GPUs (P or G instance families for rightsizing scans; any GPU-backed instance for idle scans) * You have IAM permissions to create and modify roles and policies * You have access to AWS Systems Manager *** ### Step 1: Install the NVIDIA Driver Install the appropriate NVIDIA driver for every relevant EC2 instance. Follow the[ official AWS documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/install-nvidia-driver.html). *** ### Step 2: Install and Configure the SSM Agent SSM (Systems Manager) lets you deploy and manage the CloudWatch Agent across instances without direct SSH access. Follow the[ official AWS documentation](https://docs.aws.amazon.com/systems-manager/latest/userguide/ssm-agent.html) to install and configure the SSM Agent. *** ### Step 3: Attach IAM Role Policies Each target EC2 instance must have an IAM role with both of the following managed policies attached: | AmazonSSMManagedInstanceCore | Allows the instance to communicate with SSM | | ---------------------------- | ---------------------------------------------------- | | CloudWatchAgentServerPolicy | Allows the instance to publish metrics to CloudWatch | Both policies are required. *** ### Step 4: Install and Configure the CloudWatch Agent #### 4a: Install the CloudWatch Agent via SSM Use[ AWS Systems Manager Run Command](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/installing-cloudwatch-agent-ssm.html) to install the CloudWatch Agent on your instances. #### 4b: Create the Agent Configuration File The agent requires a JSON configuration file that defines which metrics to collect. Store this configuration in AWS Systems Manager Parameter Store so it can be deployed to instances via SSM and collect all GPU metrics required by Finout. * Metrics are collected every 60 seconds, published under the CWAgent namespace, and dimensioned by InstanceId. {% hint style="warning" %} **Note**: The `append_dimensions` field with InstanceId is **required**. Metrics published without `InstanceId` cannot be linked to EC2 resources in Finout and will not be usable for scans. {% endhint %} ``` { "metrics": { "namespace": "CWAgent", "append_dimensions": { "InstanceId": "${aws:InstanceId}" }, "metrics_collected": { "nvidia_gpu": { "measurement": [ "utilization_gpu", "temperature_gpu", "power_draw", "utilization_memory", "fan_speed", "memory_total", "memory_used", "memory_free", "pcie_link_gen_current", "pcie_link_width_current", "encoder_stats_session_count", "encoder_stats_average_fps", "encoder_stats_average_latency", "clocks_current_graphics", "clocks_current_sm", "clocks_current_memory", "clocks_current_video" ], "metrics_collection_interval": 60 } } }, "force_flush_interval": 60 } ``` For full configuration reference, see the[ official AWS documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-cloudwatch-agent-configuration-file.html). *** ### Step 5: Verify Metrics Are Being Collected After deploying the agent configuration, confirm that GPU metrics are flowing into CloudWatch. 1. Open the CloudWatch console in the AWS Management Console.
2. Navigate to Metrics > All metrics and look for the CWAgent namespace.
3. Within CWAgent, you should see metrics like `utilization_gpu`, `temperature_gpu`, `memory_used`, etc., dimensioned by InstanceId. 4. If no metrics appear within a few minutes, check the CloudWatch Agent logs at: `/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log` *** ### Ingested Metrics Once setup is complete, Finout ingests the following GPU metrics from CloudWatch. All metrics are collected per physical GPU and preserved at instanceId + index granularity - this is critical for detecting partial waste on multi-GPU instances.
MetricMetric NameDescription
GPU Utilizationnvidia_smi_utilization_gpu% of time GPU cores are active - core waste signal
GPU Memory Usednvidia_smi_memory_usedAbsolute GPU memory consumption
GPU Memory Totalnvidia_smi_memory_totalTotal GPU memory - reference/metadata
GPU Memory Freenvidia_smi_memory_freeFree GPU memory
GPU Memory Utilizationnvidia_smi_utilization_memory% of memory bandwidth in use
CPU UtilizationCPUUtilizationHigh CPU + low GPU → misconfigured workload
GPU Power Drawnvidia_smi_power_drawConfirms powered on but idle
GPU Temperaturenvidia_smi_temperature_gpuSustained low temperature reinforces inactivity
Graphics Clocknvidia_smi_clocks_current_graphicsGPU clock speed - suppresses false positives on bursty workloads
SM Clocknvidia_smi_clocks_current_smStreaming multiprocessor clock - same use as above
*** ## EC2/GPU Cost Optimization Scans This section covers how the GPU cost optimization scans work in [CostGuard](https://docs.finout.io/user-guide/optimize/costguard) after the required metrics are ingested from CloudWatch. {% hint style="info" %} **Note**: A minimum of 7 days of GPU metrics data from CloudWatch must accumulate into Finout, after the configuration is updated, before the first CostGuard scan results appear. {% endhint %} ### Idle GPU Scan #### Goal Identify GPU-backed EC2 instances where both the GPU and CPU are consistently inactive, and recommend shutdown. Because GPU cost is embedded in EC2 instance pricing, a fully idle GPU instance represents 100% wasted spend. | Scan name | EC2 - GPU Idle | | -------------------- | ----------------------------------------- | | Sampling source | Amazon CloudWatch (CWAgent + EC2 metrics) | | Timeframe | 7 days back | | Calculation interval | 24 hours | | Cost type | Net Amortized | | Potential savings | Full EC2 instance cost | #### Logic All GPUs on the instance must meet every idle threshold below on every day in the 7-day lookback period. If any single GPU index is active on any single day, the instance is not flagged. #### Scan Thresholds
MetricConditionNotes
CPU Utilizationmax < 5%Required - aligned with existing EC2 idle scan
GPU Utilization per indexavg < 10%-
GPU Memory Utilization per indexavg < 20%-
NetworkInavg < 30 MBSupporting - aligned with existing EC2 idle scan
NetworkOutavg < 30 MBSupporting - aligned with existing EC2 idle scan
GPU Power Draw per indexavg < 15%Supporting - confirms the GPU is powered on but unused
*** ### GPU Rightsizing Scans #### Goal Identify GPU-backed EC2 instances where the GPU is idle but the workload is actively running. The recommendation is to migrate to an equivalent non-GPU instance, eliminating the GPU premium without disrupting the workload. {% hint style="info" %} **Notes:** * Applies exclusively to P and G EC2 instance families. * Potential savings reflect the cost delta between the current instance and the recommended instance — not the full instance cost. {% endhint %} #### How the Target Family Is Determined
Workload SignalRecommended FamilyRationale
High CPU, low memoryC familyCompute-optimized — high CPU throughput without GPU overhead
High CPU, high memoryR familyMemory-optimized — suited for in-memory analytics and large dataset processing
Moderate CPU, moderate memoryM familyGeneral purpose — balanced workload with no GPU requirement
#### Scan Thresholds CPU and memory signals determine which scan fires and which target family is recommended.
Scan NameThresholdsRecommended FamilyRecommendation
EC2 - GPU Rightsize to CPU Instance

GPU Utilization < 10%

AND

max CPU Utilization > 50%

AND Memory Used < 40%

C family - c6i instances

Migrate to compute-optimized C-family.

Workload is CPU-bound with low memory and no GPU activity.

EC2 - GPU Rightsize to Memory Instance

GPU Utilization < 10%

AND

Max CPU Utilization > 50%

AND

Memory Used > 60%

R family - r6i instances

Migrate to memory-optimized R-family.

Workload needs CPU and high memory but no GPU.

EC2 - GPU Rightsize to General Purpose Instance

GPU Utilization < 10%

AND

Max CPU Utilization = 20–50%

AND Memory Used = 40–60%

M family - m6i instances

Migrate to general purpose M-family.

Balanced CPU/memory with no GPU activity.

--- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.finout.io/user-guide/optimize/costguard/connect-costguard-for-aws/aws-ec2-gpu-cost-optimization-scans-alpha.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.