Building a Core Router for the Next Decade
The need for more in the core
As service providers bolster their networks with 100 Gigabit Ethernet links to stay ahead of rapidly growing bandwidth requirements, existing core routers struggle to cost efficiently scale capacity. They are challenged to deliver high volumes of 100G links while keeping power and space consumption down, not to mention incremental capabilities that will be required as services and the applications that drive them evolve. We saw these challenges as an important opportunity to reshape the core router market. Core routers of the past decade had a simple and singular mission – deliver scalable bandwidth in support of Internet traffic growth. Today this simple picture has become far more nuanced as the scope and nature of applications and the distribution of content are driving an evolution in metro networks and cloud infrastructure (Figures 1 and 2). Some implementations within core networks may solely focus on IP core routing and Internet peering, while others will benefit from the reduced complexity of Multiprotocol Label Switching (MPLS). Still others will look to interconnect datacenters or incorporate Virtual Private Network (VPN) infrastructure services to best serve the range of applications delivered by their services. The flexibility to address the broader set of core network requirements is increasingly valuable, in addition to higher capacity and efficiency.
Pushing beyond traditional trade-offs
Developing dedicated platforms with single-purpose designs and limited feature sets has been suggested as one way to support the requirements of the core network in a scalable and cost-efficient way. You remove functionality to reduce costs, but in doing so you lose flexibility, and may force service providers to maintain multiple platforms to meet their full scope of needs. The historical alternative has been to add chassis to existing deployments, but this escalates space and power costs. Traditional industry thinking views scalability and span of capability as desirable attributes that require trade-offs. This is a classic product design challenge that has grown more intense as core router capacity tries to keep pace with Internet traffic growth that has been doubling every 18 months . We believed we could push the boundaries in each of these areas simultaneously: to scale density and maximize efficiency without sacrificing the versatility to support the range of present and future core network needs. We had to rethink some critical aspects of the router, but rather than start from scratch, we chose to base our design on our field-proven techniques and technologies:
- Our 400G FP3 Network Processing Unit (NPU) silicon allowed us to massively scale bandwidth, not only at 10G, 40G and 100G, but also with a clear path to 400GE.
- Our Service Router Operating System (SR OS) provides robust and proven feature and protocol capabilities, having been operationalized in service provider IP networks for a decade. The flexibility to add features on a solid base, and to deliver them through software licenses as needed to ensure superior economics is an inherent advantage of the 7950 XRS family.
Our multi-faceted design mandate
To enter a new market, and be successful, you need to have both a disruptive change and a product that can significantly leapfrog the status quo. The transition to 100G is a disruptive change, but to scale on bandwidth alone is not enough. We also had to scale features — a capability that incumbent core routers struggle to achieve. We needed to:
- Deliver massive scale, but with the versatility to evolve and address different sets of core networking requirements.
- Ensure our platform could scale capacity and features while maintaining greater efficiency.
Designing for scalability and versatility
The 7950 XRS is the industry’s first network processor-based core router platform. To deliver the combination of capacity and capabilities that are needed, we focused on silicon design and achieving 400G forwarding in our third-generation FP3 chipset. In addition to best-in-class port scaling at 100G, it paves the way for support for 400G clear-channel Ethernet on the existing chipset. The FP3 scales the span of feature capabilities s as well. Resembling the FP and FP2 before it, the FP3 is a fully programmable network processor. Unlike off-the-shelf approaches that hardcode "engines" with predetermined tasks, no feature or function is hardcoded in the FP3. This allows us to add and change features through software as customers’ needs and requirements evolve. The SR OS operating system represents years of accumulated features, design experience and field testing. Rather than port the SR OS to the 7950 XRS platform, the software team pulled the 7950 XRS into the SR OS and scaled it for bandwidth capacity, features and routing table size. This approach allows the 7950 XRS to run the same SR OS binary as our line of IP/MPLS products. It wasn’t easy, but we can truly say we use a single operating system across all of our products. To build a router that can last for at least a decade, we realized the importance of delivering a very flexible platform that lets service providers make changes in place and avoid forklift upgrades. Most routers let you upgrade the forwarding card, central control processor or central switch fabric. We designed the 7950 XRS chassis to include 21 field-upgradeable units that can be removed and upgraded without rebooting the system. It supports the standard upgrades described above, as well as upgrades to the:
- Central processor
- Distributed control processors on each forwarding complex
- Distributed forwarding complex fabric taps
- Large unit link power modules
An individual power lug can even be replaced if one is damaged during installation.
Designing for efficiency
Power efficiency is a growing concern as core router scale and density are constantly driven to new heights.. There are strict limits to the amount of power that can be dissipated in a rack and that can be fed to a central office. If the 7950 XRS was going to provide five times the capacity of existing routers, we didn’t want it to require five times the power. This concern was particularly important because we wanted to scale features along with capacity. Saving power is only one aspect of efficiency. Space and cost efficiencies are also important. The 7950 XRS had to offer a relatively small form factor to make it cost effective for customers that want maximum bandwidth with the right mix of features for their service delivery needs.. Efficiency could not be an afterthought. It had to be incorporated in the system’s development at all levels. We designed every component of the data plane for optimal forwarding capacity. Scale allowed us to condense more capabilities in a smaller footprint. In addition, a higher capacity of components meant that fewer components were needed and less power consumed. We developed dynamic control functions to minimize power draw on the individual chips, sections of chips and on complete line cards based on utilization at any given point in time. At the chassis level, we employed subsystems that can power down a complete forwarding line card when placed in a redundancy or off-line mode. Using advanced cooling techniques, the system keeps critical chips at temperatures where they provide optimal performance with minimal power leakage. To make the best use of available resources, we also made sure that our hardware had the perfect geometry to support Ethernet interface speeds. Simply put, a forwarding plane that is dimensioned on an odd baseline, such as 140G or 240G, is not best suited to efficiently handle the required Ethernet speeds. It leaves a significant amount of available line card capacity unused. With the FP3, we designed 200G and 400G line cards with a perfect geometry for 10G, 40G and 100G speeds and the ability to support 400G as the technology emerges. Support for 400G is crucial for the future, but it also helps to reduce the number of components for accelerating broad deployments of 100GE right now. If you design a 400G card with 4 100G interfaces and use a 400G forwarding plane, those 4 interfaces are tied to the same forwarding plane. All packets go through the same FP3 chip, the same memory surrounding the chip and the same fabric. Conversely, a line card design using 100G chips will need 4 chips instead of 1. It will also need 4 times the memory because each memory complex has to address the full forwarding plane of the entire network. That’s a lot of redundant memory. More redundant memory leads to higher power draw and more elements that can fail. Our 400G silicon makes the 7950 XRS a less complex product, with higher quality and lower power consumption, as well as favorable economics.
Evolution instead of revolution
When introducing a new product, it always sounds great to say: “We’ve developed 12 revolutionary new ways of doing things.” However, you may also have at least 12 new bugs that you don’t know about yet because your new methods haven’t been fully tested in the field. Core routers need to be extremely reliable and our goal was to launch a new platform that lives up to these expectations. Reuse and evolution can accelerate development and ensure higher reliability. The FP3 chipset and SR OS are building blocks that have been widely used in our routing and switching portfolio for about a decade. Our extensive experience at the IP network edge has paid off too because the core now requires a degree of flexibility that was already provided within the scope of capabilities of the IP service edge.
By combining these assets, we are able to bring a complete set of features into the 7950 XRS platform in a very short time, while maintaining high quality. Because the OS running on the 7950 XRS is the same as the Alcatel-Lucent 7750 Service Router, we are able to test just one binary that delivered the accumulated benefits of ten years of testing and fault elimination. Similarly, the FP3 is the first and only 400G network processor chipset in the market¬place. It represents the third generation of NPU silicon innovation from our team. Since we built the FP3 in-house with close cooperation from our service provider customers, we were able to tune the processor to their unique needs. As a result, the FP3 capabilities have been thoroughly tested in many different ways on many different platforms. With the 7950 XRS, we now add another platform to our routing portfolio, this one designed to optimize the core, both internet backbones and metro cores. This new platform and the experience the team gained in developing it will serve as the basis for the next phase of evolution in our routing platforms. To contact the author or request additional information, please send an e-mail to firstname.lastname@example.org.
Ken Kutzler, Vice President of Hardware Engineering for the Alcatel-Lucent IP Division and the new 7950 Extensible Routing System (XRS[PW1]), talks about the philosophy and considerations that drove the development of the core router platform.
[PW1]Link to press release