Beyond bandwidth: Network is the new compute

In high-performance computing (HPC), bandwidth moves the data, but predictable communication keeps the science moving.

HPC has always centered on processing power: faster CPUs, more powerful GPUs, higher memory bandwidth. With innovations in accelerators/XPUs and explosive growth in data processing and computational demands, infrastructure must now support HPC application workloads that are increasingly distributed across multiple nodes, racks and clusters. 

Today, the network isn’t just about speeds or feeds or connecting compute resources. It’s an extension of compute itself.

What does the world’s fastest supercomputer tell us?

A supercomputer’s performance is defined by how fast its processors compute and by how efficiently it can get thousands of processors to work together. Leading supercomputers have demonstrated the critical role of networks by supporting purpose-built, highly specialized interconnects. 

The challenge goes far beyond moving packets. At scale, the fabric should support communication across millions of processing cores that constantly generate highly synchronized MPI collectives, halo neighbor exchanges and bursty traffic patterns.  Leading supercomputing architectures strongly emphasize low-latency communication, high bidirectional bandwidth, adaptive routing, multiple redundant paths and sophisticated congestion control mechanisms to ensure distributed processors spend less time waiting for data. For example, communication performance is central to the value of compute investments for university research centers, national laboratories and enterprise AI teams.

HPC–AI convergence: Different workloads, shared challenges

HPC and AI solve different problems but share similar challenges, such as supporting thousands of processors that continuously exchange data, synchronizing their operations and ensuring that they act as one.

In the HPC context, consider a molecular dynamics simulation that is divided into smaller domains. Each domain is assigned to a GPU that independently processes atomic interactions and exchanges boundary information with neighboring domains. The GPUs rely on MPI collective operations like AllReduce and All-to-All to synchronize the calculations. These exchanges add communication overhead, as shown by the ghost atoms in Figure 1.

Figure 1. Ghost atoms exchange in distributed dynamics solutions

AI adds to the challenge of operating distributed systems, especially when LLMs outgrow the single GPU’s memory capacity. The models have to be distributed across the GPU cluster using parallelism techniques to slice or replicate models across GPUs, which are highly dependent on networking between them:

• Tensor parallelism partitions individual model layers and distributes its weights across multiple GPUs. 
• Pipeline parallelism splits a model’s neural layers into successive stages and hosts them on multiple GPUs for processing like an assembly line.
• Data parallelism replicates the workloads across multiple GPUs which enables simultaneous processing of data batches to improve training and inference throughput. 

There is a clear resemblance between HPC and AI networking. HPC halo exchanges and AI tensor parallel communications depend on the underlying network and require low latency, high bandwidth, congestion control mechanisms and consistent performance across nodes.

This convergence is the motivation behind efforts to combine high-performance Ethernet, RDMA over converged Ethernet and rail-based designs, advanced load balancing techniques, Multipath Reliable Connection (MRC), source-based routing SRv6 and to develop native transport, i.e., Ultra Ethernet.

From architectural theory to real-world validation 

We are conducting hands-on HPC benchmarking tests at our AI Networking Innovation Lab in California. Our teams continuously test designs and optimize parameters to ensure the network fabric for real-world HPC/AI performs exceptionally in distributed clusters. The test environment includes:

• A reproducible multi-node testbed: We use multiple eight-GPU nodes, each with dedicated NICs that run RoCEv2. Consistent NVIDIA NGC container images support repeatable tests across nodes. 
• A layered benchmark suite: Our goal is to ensure every layer runs at optimal capacity. We benchmark using several different workloads:
  • OSU Micro-Benchmarks, which provide point-to-point latency and bandwidth benchmarks for collectives
  • HPCG, widely used in the TOP500 community to rank supercomputers based on data movements
  • GROMACS, used for molecular dynamics benchmarking applications for simulating the movements of proteins. 
  • LAMMPS, widely used for classic domain decomposition for material science. 

We stress tested the fabric with HPCG, which exercises sparse matrix operations and memory-intensive workloads. In our test bed, scaling HPCG from one eight-GPU node to two nodes with 16 GPUs increased the aggregate throughput from approximately 5.49 to 10.0 TFLOP/s, a 1.82x improvement, or about 91% scaling efficiency over this two-node comparison.

The LAMMPS results show the impact of communication tuning. Using 16 NVIDIA H200 GPUs across two nodes connected through a Nokia 7220 IXR-H5 Ethernet fabric using RoCEv2, application throughput for a simulation with 2.048 million atoms improved from 821 to 1105 timesteps per second—almost 34%. We kept the compute resources constant and improved performance by changing how MPI and UCX communications access the network interfaces.

HPC and AI workloads are driving innovation in network topologies, such as rail-based designs that organize connectivity so that GPUs and their associated NIC interfaces can communicate in parallel across corresponding portions of the fabric. There is no one-size-fits-all network topology. While some topologies support fewer hops or more equal-cost paths, the ideal choice depends on the traffic patterns and demands of a given application.

Peak bandwidth is just the beginning    

The evolution of data centers has been remarkable, from connecting ports between two endpoints, to connecting applications through APIs and now connecting intelligence. As HPC and AI redefine the data center landscape, faster speeds are going mainstream. The next frontier of high-performance networking includes 1.6T, RoCEv2 and Ultra Ethernet features such as Packet Trimming and Congestion Signaling (CSIG), which meet the demands of workloads in unprecedented ways. 

Our benchmarking journey has taught us an important lesson on building high-performance computing networks. Faster network interfaces and increasing switch capacity are essential. But we also need to understand how applications communicate, how the traffic moves across the topology and how effectively the network supports distributed computation.

Through our benchmarking efforts, we have developed a deeper understanding of how distributed compute resources communicate using MPI, and how GPU-to-NIC locality, NUMA affinity and multi-rail configurations affect their interaction with the fabric. The goal is to bring application-level performance and network telemetry together, correlating the metrics, GPU utilization with fabric congestion, interface utilization, ECN marking and routing behavior.

A university research team might measure simulation time; an enterprise AI team might track training time. Both need application-level evidence.

Connecting intelligence at Nokia

Nokia is focused on translating these architectural principles into practical, resilient and high-performance Ethernet fabrics for AI and HPC workloads. Our data center portfolio, including Nokia SR Linux and Nokia Event-Driven Automation, brings together scalable fabric design, deep network visibility and simplified operations. The objective is to build HPC and AI fabrics for which performance is measured by how fast packets move across the network and how efficiently the network enables distributed computation.

Interested in learning more? Visit the Nokia booth at Supercomputing 2026 to speak with our HPC and AI networking experts.