AI Computing Capacity Depends on Memory, Networks and Cooling

An accelerator's peak arithmetic speed describes one component of an AI system. Delivering useful work also requires enough memory, movement of data, coordination among devices and reliable power and cooling. A faster chip can spend time waiting if another part of the system cannot keep up.

This is why a processor specification cannot be converted directly into production capacity or investment return. The relevant result is an accepted workload completed within its quality and delivery requirements, at a measurable system cost. Different workloads can expose different constraints on the same hardware.

Separate model storage from working memory

Model parameters occupy memory, but they are not the only allocation. Runtime buffers, intermediate calculations and cached information from prior tokens also need space. NVIDIA's TensorRT-LLM documentation describes these separate memory demands and their dependence on serving configuration.

For a hypothetical dense model with 70 billion parameters stored at two bytes each, parameter storage alone is about 140 billion bytes, or 140 GB using decimal units. At an ideal packed four bits per parameter, the raw weight data would be about 35 GB. Quantization metadata and other allocations are excluded from both figures.

The arithmetic does not establish that a particular device can run the model at acceptable speed or quality. Lower precision changes implementation requirements and can affect accuracy. Fitting the weights is one condition; serving the intended context lengths and simultaneous requests requires additional memory and testing.

Longer conversations create another memory demand

Transformer serving can retain keys and values from earlier tokens in a KV cache to avoid repeating some calculations. Its size depends on the architecture, sequence length, number of active requests and storage format. NVIDIA documents techniques for managing and reusing those caches.

A deliberately hypothetical serving setup might use 40 GB for weights, 10 GB for runtime overhead and 1 GB of cache per active request. In an 80 GB memory budget, that leaves 30 GB for caches, or 30 requests under those assumptions. Doubling the per-request cache requirement would reduce the simple capacity estimate to 15.

These are invented allocations for explanation, not measurements of a commercial model. They show why a headline memory size or a short-prompt benchmark cannot establish capacity for much longer conversations. Sharing, eviction and other implementation choices can change the result.

Bandwidth and interconnect can limit useful speed

A calculation requires data as well as arithmetic. If supplying the necessary values from memory takes longer than the arithmetic, additional compute capability alone may deliver limited improvement. Splitting a model across devices adds communication requirements that depend on the parallelization strategy and network.

For a toy workload that must read 100 GB for one operation, an effective transfer rate of 2 TB per second implies a minimum of 0.05 seconds just for that data movement. Doubling arithmetic speed cannot remove that particular lower bound. Real models reuse data and combine operations, so this is a scale illustration rather than an inference-speed estimate.

The same distinction applies between servers. Adding devices can increase aggregate resources while introducing synchronization and transfer costs. Scaling should be measured on the target workload rather than inferred from the number of installed accelerators.

Throughput and responsiveness are different objectives

Batching can increase aggregate utilization by processing several requests together. The resulting trade-offs matter when users expect a fast first response and regular output afterward. MLCommons' inference resources distinguish online, interactive and offline measurements because request arrival patterns and latency constraints change the useful result.

Imagine a hypothetical service reporting 10,000 generated tokens per second across 100 requests. That average is 100 tokens per second per request only if output is shared evenly; it says nothing by itself about waiting before generation starts. A queue lasting 20 seconds could make the service unsuitable for an interactive application despite high aggregate throughput.

Benchmark comparisons need the model, quality target, input and output lengths, load pattern and latency requirement. Different tests can reward different system designs. A larger published number does not establish superiority for every use case or a lower cost per accepted task.

Installed equipment needs an operating environment

Electricity delivery and heat removal constrain how much hardware can operate simultaneously. The data-center power assessment must distinguish equipment capacity from sustained facility load. Networking, storage and maintenance also affect availability after installation.

Suppose a hypothetical cluster could perform 1 million accepted tasks per hour while operating but is available only 80% of scheduled time. Its time-averaged output is 800,000 tasks per scheduled hour under a constant-workload assumption. A system with a lower peak but higher availability could produce more useful work over the month.

Evaluate delivered throughput, quality, responsiveness, availability and total operating cost together. Compute, memory and power are complementary resources. The financially useful improvement is the one that relieves the actual bottleneck in the deployed system, not simply the component with the most impressive specification.