The era of distributed inference

As AI moves closer to users, devices and data, the network must enable distributed compute to behave like one seamless system.

For the past several years, the AI infrastructure story has been dominated by ever-larger training clusters. Massive facilities filled with accelerators have enabled increasingly capable foundation models, while applications such as chatbots have introduced millions of people to generative AI. But the next phase will look very different. As AI evolves from human-to-AI exchanges toward agentic systems and, ultimately, physical AI, demand will shift from building models to running them: everywhere, continuously and at scale.

That transition will make inference the center of gravity for AI infrastructure. Goldman Sachs Research projects token processing rising from roughly five quadrillion tokens per month today to more than 120 quadrillion by 2030. At the same time, a McKinsey report shows inference compute capacity will grow from about 35 GW in 2026 to 117 GW by 2030. Whether the final numbers land above or below those estimates, the direction is unmistakable: AI inference workloads are becoming more numerous, more distributed and more sensitive to where processing occurs.

Inference changes the geography of compute

Training and inference have different geographic requirements. Training can be concentrated in remote locations where power and land are available because the work is planned, centralized and less dependent on proximity to the end user. Inference serves live applications. Its performance, economics and regulatory profile can improve when compute is closer to the people, machines and data sources that generate the demand.

Consider AI applications that process thousands of high-definition video streams, analyze large medical images or coordinate fleets of autonomous machines. Moving all of that raw data to a distant cloud can be costly and slow. Local or regional processing reduces transport requirements, improves responsiveness and helps organizations address data-sovereignty, privacy and sector-specific rules. For many emerging applications, proximity will be a defining element of the service design.

The metropolitan deployment challenge

Bringing significant compute capacity into metropolitan areas is easier said than done. Large new data centers require scarce power, suitable land, lengthy permitting and substantial capital with development timelines of three to five years. Projects near population centers may also face environmental scrutiny and community opposition. Those constraints can delay the point at which expensive AI infrastructure begins serving customers and generating revenue. 

A practical alternative is to distribute capacity across many smaller sites. Instead of constructing a single 100 MW facility, an operator could place 1–5 MW inference clusters in available warehouses, telecom central offices, commercial buildings or other locations with usable space and power. This distributed infrastructure approach can accelerate deployment, reuse existing assets, bring compute closer to demand, and spread risk by minimizing the impact of a single site failure.

But distribution creates a new challenge. AI applications frequently depend on multiple agents, models and data sources operating across different sites. A single workflow may need to retrieve information from one location, invoke specialized models in another and maintain context across several clusters. Without the right network, a collection of small sites remains exactly that: a collection of isolated resources.

The evolution to inference connectivity

This is where telecommunication providers can play a defining role. They already possess many of the ingredients required for distributed AI: physical sites, fiber infrastructure, operational expertise and established connections to enterprises and consumers. The opportunity is to combine those assets into specialized services that enable geographically separate inference clusters to operate as one federated environment.

Traditional best-effort connectivity will not be enough. Distributed inference will generate large flows of prompts, model context, telemetry, workload state and cached information between sites. Agentic and physical AI applications will require predictable response times, while active sessions must continue through failures and traffic shifts. Workloads may move dynamically among enterprises, neoclouds, hyperscalers and service providers, often crossing organizational or geographic boundaries.

Inference connectivity provides scalable, efficient and deterministic networking. It must deliver strong resilience, security, tenant isolation and sovereignty controls. It also needs open programmability and multi-domain automation so connectivity can adapt as workloads move. Just as cloud platforms abstracted pools of computing resources, inference connectivity must abstract network distance and operational complexity without hiding performance or risk from the customer.

Inference connectivity includes the following aspects,

• Connectivity between distributed inference clusters

• Connectivity to share distributed inference cluster with other service providers

• Connectivity to distributed inference clusters to end users (fixed and mobile)

An ecosystem opportunity

No single company will deliver this model alone. Telecommunication providers can provide sites, connectivity and managed services. AI and cloud providers can define workload behavior and service requirements. Network technology providers can supply the routing, optical, security, assurance and automation capabilities that make the system work at scale. The strongest commercial models will be open enough to support multiple clouds and providers while consistent enough to deliver measurable service-level outcomes.

As inference becomes more distributed, network capabilities that were once treated as supporting infrastructure become part of the AI service itself. Reliability, latency, security and programmability will directly shape application quality and economics.

How Nokia can help build the distributed AI network

Nokia brings a broad set of capabilities to this transition. High-performance IP routing can connect AI clusters across metro and wide-area networks. Optical transport and coherent pluggable optics can deliver high-capacity links between sites. Security technologies, including encryption and DDoS protection, can safeguard infrastructure and data. End-to-end automation and assurance can provide the visibility and operational control needed to support demanding AI services.

Equally important, distributed inference depends on secure segmentation across customers, clouds and domains. Technologies such as MPLS, Segment Routing, SRv6, EVPN, VPN services and network slicing provide a foundation for workload isolation, policy enforcement and multi-tenancy. Combined with open architectures, they can help transform separate compute locations into a consistent platform spanning regional, metro and edge sites.

AI factories create intelligence, networks deliver it

AI is shifting from training models to supporting a wide range of agentic and physical applications. The future, empowered by inferencing compute resources, will be highly distributed and connected with advanced networking solutions. These solutions will make the distributed clusters predictable, resilient and secure enough for critical production workloads. They will enable many distributed clusters to be operated as a single federated system.

The future of AI will require a network that is more than mere connectivity. It will require a trusted platform that is an integral part of enabling coordination of distributed compute and supporting AI applications with predictable performance and advanced security.

Learn how inference connectivity turns distributed AI clusters into one platform