AI Models & Platforms
AWS Adds Quality, Speed and Cost Rankings to SageMaker JumpStart

Key PointsAWS keeps benchmark scores separate so teams can prioritize the test that matters for…
View more
- AWS keeps benchmark scores separate so teams can prioritize the test that matters for their application.
- Rankings reflect a chosen deployment configuration rather than a universal ordering of model quality.
- The hourly cost estimate excludes data transfer, storage and discounts so it is not the full bill.
Amazon Web Services on November 18, 2026 introduced the Amazon SageMaker JumpStart Model Leaderboard, giving SageMaker Studio users a single view of benchmark results, response performance and estimated instance pricing across more than 70 models.
The launch addresses a selection problem created by JumpStart’s catalog of hundreds of open-weight foundation models: published results can come from different prompts, settings and hardware, while deployment behavior and cost vary by configuration. The new Leaderboard view sits beside Categories and All models on the JumpStart base-models page, opens directly to large language model rankings and retains JumpStart’s search and filters. Launch coverage includes Meta, Google, Qwen, IBM and Arcee, and the board has a shareable URL.
Consistent Tests and Deployment Controls
AWS says it runs each benchmark with a common evaluation harness and settings across the models it tests. The three tests are BIG-Bench Hard, covering multistep reasoning; GPQA Diamond, using graduate-level science questions that are difficult to answer through web searches; and MMLU-Pro, which tests broader knowledge and reasoning with harder multiple-choice questions.
Scores remain separate rather than being merged into a composite, allowing users to prioritize the test most relevant to their application. Higher scores are better on all three benchmarks.
For operational measurements, AWS sends a standardized workload to endpoints using the same JumpStart deployment configurations available to customers. The board reports time to first token, throughput per request in tokens per second, and estimated instance cost per hour. Those figures depend on two controls: a use-case profile, which defaults to Max Context, and an optimization goal, which defaults to Balanced. Changing the profile alters performance and cost results while leaving quality scores unchanged.
Balanced chooses the configuration with the lowest estimated cost per token, taking account of both hourly price and generation speed. The other goals select the lowest hourly instance price, lowest time to first token or highest throughput; some throughput configurations use speculative decoding. This makes the ranking specific to a deployment configuration rather than a universal ordering of models.
Selection Carries Into Deployment
Teams can first exclude models that fail requirements such as license, provider or input and output modality, then sort the remaining entries by a chosen benchmark. Blue heatmaps highlight stronger results within columns, while adjustable columns and frozen model names support wider comparisons.
Selecting a model opens deployment options and, where serverless customization is supported, customization controls. The SageMaker JumpStart documentation notes that customization, fine-tuning and evaluation are not available for every model.
AWS advises deploying with the same use-case profile and optimization goal used during comparison because a different configuration can change quality, latency, throughput and cost. The hourly estimate covers only the SageMaker instance at the on-demand rate for the configuration’s default instance type. Data transfer, storage, inference-component charges, Reserved Instance discounts and AWS Savings Plans sit outside the figure, so the leaderboard’s price column does not represent the full bill.
Benchmark Limits and Possible Expansion
AWS recommends using the board to build a shortlist, then testing the top two or three candidates on an organization’s own data and prompts before production. Scores should be compared within each benchmark, and close results treated as ties because small differences can reflect run-to-run variation. Reasoning and non-reasoning models use different generation parameters, while GPQA Diamond results for reasoning models are averaged over 10 passes, so comparisons between those model types are approximate.
AWS says it is exploring leaderboards for curated private hubs, additional task boards for object detection, text embeddings and image understanding, and more models and benchmarks. A missing score does not indicate poor quality; some models simply have not been benchmarked, and the launch benchmarks focus on LLMs.
For JumpStart users, model selection can now begin with deployment-specific evidence inside the service where the model will run. The remaining decision is narrower but still consequential: whether a model’s benchmark advantage holds on an organization’s own workload, and whether any gain justifies its measured response behavior and instance cost.












