Skip to main content

Seamless IP data center interconnect


IP data center interconnect (DCI) has emerged as a key router application for operators who need to connect cloud-based data centers across the existing IP/MPLS WAN. And the market is growing. In its June 2015 Router and Switch Vendor Leadership Global Service Provider Survey, Infonetics estimates that routers for DCI will make up between 5% and 10% of the overall router market over the next few years.


Most of the data center interconnect discussion focus is on connectivity between data centers, particularly from an enterprise perspective. As enterprises move to cloud IT, they need solutions that interconnect on-premises data centers in the private cloud with hosted data center resources in the hybrid or public cloud (see the TechZine article DCI solutions for the new cloud ecosystem). However, these discussions omit a key aspect related to how enterprise data center applications are connected to the end user.

Most enterprise end users are connected using IP/MPLS-based WAN technologies like IP-VPN or carrier Ethernet. Furthermore, traditional enterprise data centers are evolving to adopt SDN for network virtualization which introduces several emerging technologies such as network virtualization overlay (NVO) technology, Virtual Extensible LAN (VXLAN) and Ethernet VPN within the data center. It is important therefore to seamlessly extend connectivity not only between data centers but also from the data center to the end users.

Enter: the Data Center Gateway

Sitting at the edge of the data center network, the router functions as a data center gateway -- providing the interface to the IP/MPLS WAN for interworking Layer 2 and Layer 3 VPN services to remote centers and branch locations. These services provide seamless connectivity between multiple data centers on the same or different IP subnets using the same or different Layer 2 or Layer 3 encapsulation mechanisms.

It also enables full integration of data center and VPN services for seamless connectivity between data center and branch locations.

Figure 1: DCI using the data center gateway
Figure 1: DCI using the data center gateway

Integration challenges

There are a number of challenges to integrating data center and IP/MPLS WAN networks. The networking technologies used in the data center and the IP/MPLS WAN are typically different, making interworking complex. While VLAN technology is typically used in the data center, end-user branch sites are typically connected using Layer 2 VPN and Layer 3 VPN. For integration, it’s imperative that the control plane and data plane interwork seamlessly.

VLAN handoffs to the IP/MPLS WAN require manual provisioning, often in multiple places and without network or service auto-discovery. Service provisioning needs to be automated and instantaneous so the cloud user can get instant connectivity to the cloud resources.

Virtualizing the data center brings about massive scaling requirements, with a 50X to 100X increase in the number of endpoints, which are highly dynamic. As virtual machines (VMs) are turned up and released from one data center to another, the network services that interconnect them must follow instantaneously and evolve just as dynamically.

Finally, there are often routing complexities across the IP/MPLS WAN to reach the end users. Centralized IP routes can lead to:

  • Tromboning
  • Variable network performance
  • Extra latency
  • Link congestion

Limitations such as these make VM mobility with scalable and automatic movement of service connectivity very complex.

Seamless and Unrestricted Networking

The data center gateway plays a critical role for seamless and unrestricted networking across the IP/MPLS WAN. It must have the flexibility to seamlessly integrate the control, data and management planes, automate service delivery, and be highly scalable to ensure that the IP/MPLS WAN does not become a bottleneck to cloud service delivery.

Integration Requires Flexibility

For seamless integration of the control and data planes, the full breadth of standards-based Layer 2 and Layer 3 networking technologies used in traditional and SDN-based data centers and in IP/MPLS WAN networks needs to be supported.

As the service demarcation point, the data center gateway must have the flexibility to interwork connectivity between the data center and IP/MPLS WAN network using EVPN, VXLAN, IEEE 802.1q/ad, IP, MPLS, VPLS, and PBB protocols.

It also needs to support flexible administrative models for Layer 2 and Layer 3 connectivity. A decoupled model is used when the administrator managing the data center and IP/MPLS WAN are different entities. An integrated model is used when the administrative entities are the same.

Evolutionary Path to Virtualization

The advancement of technologies such as NVO, VXLAN and EVPNs provides an evolutionary path as traditional data centers leverage SDN to virtualize. In the SDN-based data center, NVO technology is being adopted for data center networking. Overlays provide a number of benefits such as VPNs for multi-tenancy, network virtualization for VM mobility of resources, improved resource allocation, and protection from topology or technology changes.

VXLAN Brings Scale

VXLAN has emerged as the de facto NVO standard. It addresses data plane needs using overlay networks within virtualized data centers that accommodate multiple tenants. VXLAN encapsulates the Ethernet MAC frame using UDP. To meet the massive tenant endpoint scaling requirements, its 24-bit VPN identifier provides more than 16 million VXLAN IDs as opposed to only 4K VLANs from IEEE 802.1q/ad.

Each VXLAN frame can be easily transported across an IP network to deliver a VPN to the hypervisor attached to a VM. VXLAN avoids the Layer 2 MAC explosion problem because VM MAC addresses are learned only at the edge of the network. Since VXLAN is routable with IP, it allows the use of existing underlay IP network resiliency and load balancing mechanisms such as Equal Cost Multi-Path and IP Fast Reroute.

EVPN Brings Simplicity

EVPN provides a single protocol instance for all tenants with integrated Layer 2 and Layer 3 connectivity over the same interface or VLAN (see EVPN TechZine article for more details). For data plane encapsulation, it supports network virtualization overlays including VXLAN along with Multiprotocol Label Switching (MPLS) and Provider Backbone Bridge (PBB) (for details refer to draft-ietf-bess-dci-evpn-overlay).

For the control plane, SDN-based data centers are moving to cloud-friendly EVPN technology. In EVPN, Multiprotocol Border Gateway Protocol is used as the control plane protocol. This brings proven and inherent BGP control plane scalability to MAC routes, and can even be extended with hierarchy and route reflection (for details refer to RFC 7432).

In EVPN, the control plane and data plane are abstracted and separated, so IP/MAC learning is performed in the control plane instead of the data plane. IP/MAC mobility simplifies VM mobility, providing faster moves of VM’s between data centers while maintaining the correct forwarding data base on all routers.

Because it uses User Datagram Protocol (UDP), EVPN can also run over a basic IP network. EVPN attributes simplify routing across subnets, ensuring connectivity to every end user. These may include:

  • IP/MAC mobility
  • MAC protection
  • ARP suppression
  • All-active multihoming
  • IP prefix advertisement


To address speed and efficiency constraints, cloud providers are implementing SDN-based solutions to virtualize and automate the data center network. Next-generation cloud services require service delivery to be automatic and instantaneous (i.e., auto-instantiation) across the IP/MPLS WAN to remote data centers and end users.

For SDN-based data centers, the data center gateway ensures that the IP/MPLS WAN can be used to extend an elastic pool of resources between SDN-based data centers that can be consumed and repurposed on demand.

Unrestricted Networking

Unrestricted networking requires seamless management plane integration between the data center gateway and data center SDN platform. Extensible Messaging and Presence Protocol (XMPP) provides an efficient protocol for near real-time instant communication between them.

Once the cloud service manager receives a service request that a tenant requires VMs for deployment across 2 data centers, it receives the service parameters and policies from the data center SDN platform to automatically provide a policy-based attachment of IP/MPLS WAN VPN services to the remote tenant.

The policy manager in charge of automating the services and policies in the data center SDN platform must be able to manage the gateway and provide an abstraction for the IT admin users. This allows them to manage connectivity simply, without the need to understand IP/MPLS WAN networking technologies.

For unrestricted networking without manual intervention, auto-instantiation must be independent of the administrator managing the IP/MPLS WAN. This is relevant in scenarios where the data center and IP/MPLS WAN are operated by different entities.

If the cloud service provider owns the IP/MPLS WAN, the service needs to be fully dynamic and initiated by the data center SDN platform through the data center gateway. Otherwise, the cloud operator needs to pre-provision the IP/MPLS WAN attachments to interwork with the IP/MPLS WAN operator’s provider edge router.

Figure 2: Seamless and unrestricted DCI across the IP/MPLS WAN
Figure 2: Seamless and unrestricted DCI across the IP/MPLS WAN

High Scalability Needed

Many communication service providers offer carrier Ethernet and IP-VPN services supporting thousands of VPN instances with hundreds of end points within the VPN. For the enterprise end user, there is significant value in having data center services inherently part of the VPN, for seamless connectivity between the data center and branch offices.

The data center gateway platform must be able to scale. First, to meet the massive number of end points for cloud service and VPN service instances with enhanced SLAs and tailored policies. This requires advanced QoS mechanisms supporting per-flow bandwidth, per-service availability and other service metrics - all at line-rate without impacting application performance.

Second, it needs to be able to scale bandwidth over high density GE, 10GE, 40GE and 100GE interfaces with various form-factor options to right-size data center placement and service delivery. For some deployment scenarios, such as adding on-the-fly encryption and support for non-Ethernet technologies, the platform also needs to streamline operations across a converged IP-optical network.

Related Material

Seamless Data Center Interconnect application note

7750 SR web page

Nuage VSP web page

Our authors look forward to your questions and comments.

Jorge Rabadan

About Jorge Rabadan

Senior Product Line Manager

Jorge is a product line manager responsible for the development of EVPN and Data Center Gateway technologies in Nokia’s SROS. Jorge has more than 20 years in the telecom industry (working for different networking vendors), being part of Nokia’s IP Division for more than 12 years and is an active member in the IETF community, contributing and authoring more than 30 drafts and RFCs. When he is not working he enjoys traveling, reading a good book or working out.

Tweet me @jorabada

Rod Hoekman

About Rod Hoekman

Sr. Manager IP Product Marketing

Rod is a senior product marketing manager with 25+ years of experience in the telecom industry. His focus is marketing for Nokia’s IP Routing products and service provider solutions for the data center, business, mobile and residential segments. When he isn’t working, you can find Rod at the rink, on the slopes or cycling with family.

Tweet me @RodHoekman

Article tags