Beyond FLOPS: Why Cost per Token Is the Key Metric for AI Infrastructure

The evolution of data centers from storage and processing hubs into AI token factories is fundamentally altering the economics of artificial intelligence infrastructure. As AI inference becomes the primary workload for these facilities, the industry is shifting its focus from raw hardware specifications to a more precise metric: cost per token.

Enterprises have traditionally evaluated AI infrastructure using input metrics such as peak chip specifications, compute cost, or floating point operations per second per dollar (FLOPS per dollar). However, these metrics fail to account for the actual output of the system—the intelligence delivered in the form of tokens.

Defining the New Metrics of AI Inference

To understand the shift in total cost of ownership (TCO), We see necessary to distinguish between three primary financial metrics used in AI deployment:

View this post on Instagram about Token Is, Inference

• Compute cost: The total amount an enterprise pays for infrastructure, whether through cloud rental or on-premises ownership.
• FLOPS per dollar: A measure of raw computing power acquired per dollar spent.
• Cost per token: The all-in cost to produce each delivered token, typically measured as the cost per million tokens.

While compute cost and FLOPS per dollar are input metrics, cost per token is an output metric. Optimizing for inputs while a business operates on output creates a mismatch that can hinder the ability to scale AI profitably.

The Inference Iceberg and the Role of the Denominator

The calculation for cost per million tokens involves a numerator—the cost per GPU per hour—and a denominator, which represents the delivered token output. Many enterprises focus on the numerator, which is the visible cost of cloud hourly rates or amortized hardware.

The true driver of efficiency, however, lies beneath the surface in the denominator. Increasing token output has two primary business implications: it minimizes the cost per token to grow profit margins on every interaction and maximizes revenue by delivering more tokens per megawatt of power.

Achieving a low cost per token requires a comprehensive stack of optimizations. If these elements are missing, the denominator collapses, and even a cheaper GPU can result in a higher cost per token.

• Hardware and Precision: Support for FP4 precision to maintain accuracy while increasing efficiency.
• Architecture: Scale-up interconnects capable of handling the all-to-all traffic required by mixture-of-experts (MoE) reasoning models.
• Software Optimizations: The use of speculative decoding, multi-token prediction, disaggregated serving, and KV-cache offloading.
• Platform Support: Ability to handle agentic AI requirements, including high throughput, ultralow latency, and large input sequence lengths.

Comparative Performance: Blackwell vs. Hopper

Data analyzing the DeepSeek-R1 AI model illustrates the divergence between theoretical compute costs and actual business outcomes. When comparing the NVIDIA Blackwell platform to the NVIDIA Hopper architecture, the differences in raw cost do not reflect the difference in output.

The NVIDIA Blackwell platform costs approximately 2x more per GPU per hour than NVIDIA Hopper. Similarly, the FLOPS per dollar advantage for Blackwell is 2x. However, the actual token output is orders of magnitude higher.

Blackwell delivers more than 65x the tokens per second per GPU compared to Hopper. In terms of energy efficiency, Blackwell provides over 50x greater token output per watt. This results in a cost per million tokens that is nearly 35x lower than that of the Hopper generation.

Infrastructure Deployment and Ecosystem

The reduction of token costs is achieved through extreme codesign across networking, memory, storage, software, and compute. The use of open-source inference software, including vLLM, SGLang, NVIDIA TensorRT-LLM, and NVIDIA Dynamo, allows token output to increase and costs to decline over time on existing infrastructure.

Several cloud providers and partners have already deployed NVIDIA Blackwell infrastructure to provide these efficiencies at scale. These include CoreWeave, Nebius, Nscale, and Together AI.