Machine Learning Cloud Costs 2026: Training, Inference & GPU Pricing

ML workloads have a more complex cost anatomy than standard web infrastructure. Five components account for essentially all of the bill.

1. GPU compute

GPU instances are the dominant cost for any non-trivial ML workload. A single H100 GPU costs roughly $2.00–$2.50/hr to rent on-demand across major clouds. An 8-GPU instance (the minimum for most serious training runs) therefore starts at $16–$20/hr just for the accelerators. Training a mid-sized model for 72 hours on a p5.48xlarge costs approximately $7,079 at on-demand rates.

GPU pricing is highly generation-sensitive. A100-based instances (p4d, NC A100 v4) cost roughly a third of H100-based instances (p5, NCads H100 v5, a3-highgpu) for the same GPU count. For fine-tuning and smaller experiments, A100 instances often provide better cost efficiency per token trained.

2. Training time

Training cost scales linearly with wall-clock time. Doubling batch size can reduce training time — but GPU memory constraints limit this. Multi-node distributed training (using 4–32 GPU instances simultaneously) cuts elapsed time but multiplies the per-hour spend. The total GPU-hours required is the invariant; the question is how fast you want to burn them.

3. Storage

Model checkpoints, training datasets, and intermediate activations can consume terabytes of storage. EBS gp3 in AWS costs $0.08/GB/month. A 5 TB dataset stored for one month costs $400. S3 at $0.023/GB/month is cheaper for infrequently accessed datasets ($115/month for 5 TB), but reading from S3 during training introduces I/O latency that can meaningfully increase total training time on bottlenecked pipelines.

High-performance parallel file systems like AWS FSx for Lustre ($0.145/GB/month for SSD-backed) or GCP Filestore are often used to feed GPUs fast enough to avoid idle time between batches. At $0.145/GB/month, 100 TB of Lustre costs $14,500/month — often more than the GPU compute itself for data-heavy workloads.

4. Data transfer (egress)

Pulling training data into a cloud region from on-premises or another region incurs egress charges. AWS charges $0.09/GB for internet egress out of us-east-1. Loading a 10 TB dataset from external storage costs $921 in egress alone. For cross-region training (data in us-west-2, training in us-east-1), inter-region transfer runs $0.02/GB.

See our guide on cloud egress costs for a full breakdown of transfer pricing across providers.

5. Inference serving

Once trained, models need to serve predictions. A real-time inference endpoint running 24/7 on a single ml.g5.2xlarge in SageMaker costs approximately $1,760/month. That is often more than the training run that produced the model. Inference costs are ongoing; training is episodic. For high-traffic production inference, the cost structure is fundamentally different from training — and demands its own optimization strategy.

All three major clouds use NVIDIA GPUs for ML workloads. The key differentiator is not GPU hardware (largely similar across providers) but pricing, availability, and the ecosystem around each instance family. Use the ML cost calculator to compare specific instance configurations side by side.

A100 GPU instances (40GB and 80GB)

H100 GPU instances (80GB)

H100 instances cost almost exactly 3x as much as their A100 equivalents across all three clouds — a remarkably consistent ratio. The H100 delivers roughly 3x the training throughput for most transformer workloads (per NVIDIA H100 benchmarks), which means cost-per-token-trained is roughly equivalent between A100 and H100 instances. The practical advantage of H100 is speed: finishing a training run in 24 hours instead of 72 hours, which matters for iteration velocity and spot-interruption risk.

GCP's a2-highgpu-8g stands out as the lowest-cost A100 option at $3.28/GPU-hour, roughly 20% cheaper than the equivalent AWS p4d. For long training runs where you can tolerate GCP's preemption model, this translates to material savings. H100 pricing is effectively identical across all three clouds.

Single-GPU and development instances

Not every ML workload needs 8 GPUs. For development, experimentation, and smaller fine-tuning jobs, single- and dual-GPU instances are often more cost-effective. AWS g5.xlarge (1x A10G, 24GB) costs $1.006/hr. Azure NC4as T4 v3 (1x T4, 16GB) costs $0.526/hr. GCP n1-standard-8 with 1x T4 costs $0.952/hr including GPU.

For iterative experimentation on smaller models, starting with T4-based instances and scaling up to A100/H100 only for final training runs can cut development-phase GPU spend by 60-80%.

For most teams, inference wins. Across AI-forward organizations, serving models now accounts for roughly 80% of AI infrastructure budgets, versus about 20% for training (GPUnex, 2026). The conventional wisdom that training dominates only holds for foundational pretraining from scratch — a GPT-scale run still costs tens of millions. But if you're fine-tuning an existing model and shipping it, inference usually overtakes training cost within six months of launch.

The crossover point happens when cumulative inference cost exceeds cumulative training cost. For a product serving 10,000 requests/day at 500ms average latency on a GPU instance, that crossover typically occurs within 3–6 months of launch. Plan for it.

Inference optimization: the bigger lever

Inference cost per request depends on three factors: instance type, model size, and batching efficiency. Switching from a dedicated GPU endpoint (ml.g5.xlarge at $0.528/hr) to a shared multi-model endpoint can cut inference cost per request by 60-80% for low-traffic models. Quantizing models from fp16 to int8 cuts memory requirements roughly in half, letting you run on smaller (cheaper) instances or pack more replicas onto the same instance.

For a detailed breakdown of how purchasing model affects inference costs, see the reserved vs spot vs on-demand guide. Real-time inference requires reserved or on-demand instances; batch inference is a strong candidate for spot.

The ML cost calculator lets you model both training and inference costs separately, including multi-region inference deployments.

GPU spot instances are the single highest-impact cost lever for ML training workloads. AWS p4d.24xlarge spot pricing runs approximately $9.83/hr — a 70% discount from the $32.77 on-demand rate (AWS Spot Instance Advisor). A 72-hour training run that costs $2,360 on spot versus $7,079 on demand is the same training run — just with checkpoint-aware fault tolerance built in.

GPU spot markets behave differently from general-purpose compute spot markets. The demand for p4d and p5 instances in us-east-1 is high enough that interruption rates are elevated, particularly during business hours. Teams running serious training workloads should distribute across multiple availability zones and use mixed instance type pools.

Making training fault-tolerant for spot

Every training framework supports checkpointing. PyTorch Lightning checkpoints every N steps natively. Hugging Face Trainer saves checkpoints at configurable intervals. Checkpointing every 30 minutes means you lose at most 30 minutes of compute on interruption — the cost of one checkpoint save is trivial compared to the spot savings on a multi-day run.

The practical approach for multi-node training: use Elastic Fabric Adapter (EFA) networking on AWS to maintain low-latency inter-node communication, set checkpoint intervals at 15–30 minutes, and configure your job runner (SageMaker, Ray, or custom scripts) to automatically resume from the latest checkpoint on a new spot allocation. This architecture eliminates essentially all the risk of spot interruption for training.

For context on the broader tradeoffs between spot, reserved, and on-demand purchasing across all workload types, see our reserved vs spot vs on-demand comparison, including the section specifically covering GPU spot availability patterns in 2026.

Azure Spot VMs and GCP Spot VMs for ML

Azure Spot VMs for NC-series (GPU) instances provide similar 60-80% discounts but use an eviction model with 30 seconds notice rather than AWS's 2-minute warning. This makes checkpoint intervals even more critical on Azure. GCP Preemptible/Spot VMs for A2 and A3 instances are capped at 24-hour maximum runtime, which is not a concern for most training runs but matters for very long runs that exceed the 24-hour mark without a restart.

Managed ML platforms add a markup on top of raw instance prices in exchange for managed infrastructure, experiment tracking, model registry, and deployment tooling. The surcharge is significant — and often invisible until you see the bill.

AWS SageMaker

SageMaker charges a surcharge on every instance type it manages. For GPU instances, the SageMaker Training price is approximately 30-40% higher than the underlying EC2 on-demand price. An ml.p4d.24xlarge in SageMaker Training costs $42.074/hr versus $32.77/hr for the equivalent EC2 p4d.24xlarge — a 28% surcharge (AWS SageMaker Pricing).

SageMaker Savings Plans (1-year and 3-year) cover the SageMaker-specific surcharge and can bring the effective rate down to near raw EC2 pricing. SageMaker also supports spot training natively through Managed Spot Training, which automatically handles checkpointing and restarts. For teams using SageMaker, Managed Spot Training + Savings Plans is typically the lowest-cost configuration.

Google Vertex AI

Vertex AI Training charges the underlying GCP Compute Engine rate plus a managed service surcharge of approximately 20-30% for GPU workloads. The Vertex AI Training price for an a2-highgpu-8g is $33.07/hr versus $26.27/hr for the raw GCP instance — a 26% surcharge. Vertex AI offers custom training with preemptible (spot) VMs, cutting the effective rate to approximately $9.92/hr for the same machine (Google Cloud Vertex AI Pricing).

GCP Committed Use Discounts (1-year or 3-year) apply to the underlying Compute Engine resource but not the Vertex AI management surcharge. Teams running continuous training pipelines should compare total cost between Vertex AI with CUDs versus raw GKE clusters with GPU node pools.

Azure Machine Learning

Azure ML computes prices at approximately 25% above raw Azure VM on-demand rates for GPU compute clusters. An NC96ads A100 v4 compute cluster in Azure ML costs $39.91/hr versus $31.93/hr for the raw VM — a 25% surcharge (Azure Machine Learning Pricing). Azure Reserved VM Instances reduce the underlying VM cost by up to 63% on a 3-year term, though the Azure ML layer is not covered by reserved pricing.

The managed platform surcharge is worth paying if it replaces significant MLOps engineering effort. For teams without dedicated infrastructure engineers, SageMaker and Vertex AI provide real value. For teams with strong platform engineering, running training jobs directly on EC2, GCE, or Azure VMs via Kubernetes or Ray clusters eliminates the 25-30% surcharge while maintaining full control over the training environment.

Keep going deeper on AI/ML costs: compare GPU prices across hyperscalers and specialist clouds in our GPU cloud pricing comparison, price out serving a model in LLM inference cost per 1M tokens, run the self-host vs OpenAI API break-even, and see the full strategy in cloud cost management for AI/ML startups. For broader cost context, our reserved vs spot vs on-demand guide covers the commitment-model decision, and the Databricks cost breakdown digs into managed ML platform pricing. For build economics, see what it costs to train an LLM in 2026, the monthly cost of a production RAG system, and a vector database pricing comparison.