Training ML Models on AWS EKS with H100 GPUs

2026-04-02

If you’re looking to train ML models on GPU-accelerated Kubernetes clusters, AWS EKS with NVIDIA H100s (P5 instances) is one of the most capable options available today. The ecosystem of tooling around it has matured significantly — but finding the right starting point can be overwhelming.

This post collects the best hands-on resources I’ve found, organized by source.

Which Cloud Providers Offer H100s for Kubernetes?

Before diving into EKS specifically, here’s the lay of the land:

AWS (EKS) — P5 instances (8x H100 80GB), plus P5e/P5en variants
Google Cloud (GKE) — A3 machine series (8x H100 80GB), A3 Mega/Ultra variants
Microsoft Azure (AKS) — ND H100 v5 series (8x H100 80GB)
Oracle Cloud (OKE) — BM.GPU.H100.8 bare metal instances
CoreWeave — Kubernetes-native, offers H100 SXM and PCIe
Lambda Cloud — H100 instances available

The rest of this post focuses on EKS.

AWS Official Resources

These come directly from AWS and are well-maintained:

aws-samples/aws-do-eks — End-to-end framework for running distributed training workloads on EKS, including GPU node provisioning and job orchestration.
awslabs/data-on-eks — Terraform blueprints for running distributed PyTorch and Ray training on EKS with P5 (H100) instances. One of the best starting points if you want infrastructure-as-code.
aws-ia/terraform-aws-eks-blueprints — IaC patterns including GPU node groups (p5.48xlarge) with EFA networking and NVIDIA device plugins.
AWS EKS Best Practices — Machine Learning — Prescriptive guidance on GPU scheduling, EFA networking, and training job patterns on EKS.
AWS HyperPod on EKS — Managed resilient training clusters with automatic node health checks and recovery, specifically targeting H100/P5 instances. Launched at re:Invent 2024.

NVIDIA Resources

NVIDIA maintains the GPU software stack that makes all of this work:

NVIDIA/NeMo-Framework-Launcher — Launcher for large-scale NeMo training jobs with Helm charts and Kubernetes deployment options for multi-node GPU training. This is the go-to for LLM training at scale.
NVIDIA GPU Operator — Automates NVIDIA driver, container toolkit, device plugin, and DCGM deployment on Kubernetes. This is the prerequisite for any GPU training on EKS.
NeMo Megatron on NGC — Pre-built containers and Helm charts for distributed LLM training with Megatron-LM on Kubernetes.

Kubeflow on EKS

Kubeflow is the most established ML platform for Kubernetes:

awslabs/kubeflow-manifests — AWS-maintained Kubeflow distribution with Terraform and Kustomize manifests for EKS deployment, including GPU training pipelines.
kubeflow/training-operator — The PyTorchJob and TFJob CRDs for launching multi-node distributed training. Works directly on GPU-enabled EKS clusters.

Community and OSS

ray-project/kuberay — KubeRay operator for running Ray clusters on EKS. Supports Ray Train for distributed GPU training with PyTorch and TensorFlow.
substratusai/lora-training — A practical example of fine-tuning LLMs with LoRA on Kubernetes GPU nodes. Great for understanding the end-to-end flow.

Recommended Starting Path

If you have a specific use case like fine-tuning an LLM on H100s, here’s the order I’d approach it:

Infrastructure — Use data-on-eks or terraform-aws-eks-blueprints to provision a P5 EKS cluster with EFA networking enabled.
GPU stack — Deploy the NVIDIA GPU Operator to handle drivers, toolkit, and device plugins automatically.
Training framework — Use Kubeflow Training Operator (PyTorchJob) for standard distributed training, or NeMo Framework Launcher if you’re training large language models.
Orchestration — Add Kubeflow Pipelines or Ray if you need pipeline-level orchestration across multiple training stages.