AI Models & Platforms
AWS Adds Training Validation and Hung-Job Detection to HyperPod

Key PointsHyperPod now verifies that GPUs actually compute correct training results rather than…
View more
- HyperPod now verifies that GPUs actually compute correct training results rather than only checking hardware health.
- A 15-minute hang detection rule cuts a stalled 32-instance job's waste from about $16,900 to about $530.
- Slurm hang detection stays off by default and a missing rules file leaves jobs silently unmonitored.
Amazon Web Services announced three reliability capabilities for Amazon SageMaker HyperPod on November 18, 2026, targeting numerical training faults, Kubernetes failures and stalled Slurm jobs that can consume GPU capacity while individual nodes still appear healthy.
HyperPod already monitors hardware errors, runs deep diagnostic checks and tests communication between instances, with automatic recovery for faulty nodes. Those controls do not show whether GPUs compute correct training results, whether a job is still progressing, or whether the Kubernetes services on which it depends are working.
AWS’s cost example shows the exposure: an eight-hour hang on 32 ml.p5.48xlarge instances, representing 256 NVIDIA H100 GPUs, wastes 2,048 GPU-hours and costs about $16,900 at the approximate on-demand HyperPod price of $66 per instance-hour in US East (N. Virginia). A 15-minute detection rule brings that scenario’s cost to about $530 by canceling the job so Slurm can requeue it.
Testing Training Correctness
MultiNodeTrainingValidation extends HyperPod’s deep health checks for clusters orchestrated by Amazon Elastic Kubernetes Service (EKS), using supported Elastic Fabric Adapter (EFA)-enabled NVIDIA GPU instance groups. Its Tiny Megatron test runs a fixed Megatron-LM workload across all GPUs on each instance, while the Megatron test runs the same workload across instances of the same type over EFA. An instance fails if training loss becomes not-a-number (NaN) or infinite, or if the run does not finish within its time limit.
The check can run when an instance group is created or updated, or on demand for an entire group or selected instances through the StartClusterHealthCheck API. Results appear in CloudWatch logs and node labels. A failure labels the instance and, with automatic node recovery enabled, leads to a reboot or replacement; during provisioning, this happens before customer jobs can use the capacity.
In AWS’s test on two ml.p5.48xlarge instances, the two training checks added about eight minutes to a full deep-health-check run lasting 1 hour and 28 minutes. Instances cannot accept workloads until the checks finish, making the additional provisioning time part of capacity planning. Passing the fixed workload establishes that it trained correctly, but does not guarantee that a customer’s own training job will converge.
Linking Cluster Failures to Jobs
The HyperPod Kubernetes Resource Monitoring Agent runs as one pod per EKS cluster and prefers CPU nodes so it does not consume GPU capacity. It watches node conditions and allocatable GPU or EFA devices, critical infrastructure pods, required DaemonSets, and HyperPodPyTorchJob and Kubeflow training jobs. Warning and Normal Kubernetes events mark failures and recoveries, while structured JSON CloudWatch logs retain history and connect failed jobs to their pods and nodes.
The agent only reports; it does not reboot nodes or restart jobs. Hardware remediation remains with the health monitoring agent and automatic node recovery, while the HyperPod training operator continues to handle job restarts.
The optional NCCL integration requires version 2.24 or later and, in AWS’s implementation, polls each participating pod’s reliability, availability and serviceability (RAS) listener every 30 seconds. It combines those reports into a healthy, degraded or unhealthy job-level state. Reaching the listener requires binding it to the pod IP, which exposes TCP port 28028 on the pod network. AWS’s guidance is to apply a Kubernetes NetworkPolicy that limits port access to the agent while preserving traffic among the job’s own pods before enabling monitoring.
Detecting Slurm Jobs That Stop Progressing
The Slurm capability requires a HyperPod Slurm Amazon Machine Image released on or after November 10, 2026. It is off by default and enabled per job with the HYPERPOD_HANG_RULES environment variable, which points to a rules file readable at the same path on every compute node. No training-script or container changes are needed, and the feature works with PyTorch, TensorFlow, JAX and custom scripts.
Default detection watches job output and GPU activity, looking for stopped output while GPUs are idle but retain memory, or for a longer output silence regardless of GPU state. Custom regular-expression rules can replace default detection, require periodic progress messages or declare a hang immediately when an error pattern appears. Each approach has blind spots: default detection can miss a job that keeps GPUs busy without progressing, while custom rules miss hangs their patterns do not describe.
A detection writes a HANG_DETECTED event to CloudWatch. The default notify action leaves the job running; cancel lets Slurm requeue it up to the cluster’s MaxBatchRequeue limit, although jobs submitted with --no-requeue remain canceled. This is separate from auto-resume, which restarts jobs that exit after hardware failures rather than jobs that remain allocated without making progress.
A missing or invalid rules file produces a CONFIG_ERROR event in CloudWatch and leaves the job unmonitored, without an error in sbatch, squeue or the job output. Because the agent does not check rules-file ownership, AWS recommends restrictive file and parent-directory permissions so another user cannot replace the file and alter detection or cancellation for jobs that use it.
Together, the additions make training correctness and job progress visible alongside hardware health, reducing reliance on manual log reconstruction when capacity goes to waste. The trade-off is operational: longer provisioning checks, CloudWatch monitoring and carefully tuned detection rules become part of protecting GPU time, particularly when automatic cancellation must distinguish a hang from normal checkpointing or evaluation gaps.












