IP/MPLS-based Networks for Mission-Critical Services
AltaLink, the only fully independent transmission system operator in Canada, is responsible for a transmission grid with a total length of 12’000km (240kv and 500kv) and around 300 substations. The company decided in 2007 to migrate their traditional Asynchronous Data Transfer (ATM) and Time Division Multiplexing (TDM) based network for operational communications into a next-generation, packet-based Wide Area Network (WAN). Communications has always played a critical role for utilities because they depend on their networks to help manage generation and transmission of electricity efficiently and reliably. However, today both the growing energy demand and the pressure on utilities to integrate renewable energy sources into their grids further increases the importance of smart communications to make smart grids possible. Their role is to deliver specialized services to manage the energy flows of which many are very time-critical. AltaLink’s existing network has been used for 10 different services including Supervisory Control and Data Acquisition (SCADA), different forms of teleprotection (TPR), mobile radio, substation metering, office Local Area Network (LAN) and operational voice (PBX).
Towards a next generation network
AltaLink’s process for rethinking their operational communications network was triggered by the decline of the market availability of ATM technology. The plans soon focused on Internet Protocol/Multi-Protocol Label Switching (IP/MPLS) and a next generation Synchronous Digital Hierarchy/Synchronous Optical Network (SDH/SONET) infrastructure as the only possible alternatives. There are several differences between TDM networks and packet-oriented networks like IP/MPLS including the requirement of TDM to send data in dedicated end-to-end circuits using 64 Kbps channels with fixed bandwidth. In contrast, IP transmits the various applications’ data that has been divided into packets as it appears over the same pipe not using circuits dedicated to each application and thus making more efficient use of the bandwidth. When AltaLink was looking at its options, SDH/SONET was seen as adding minimal additional value except for extending the lifespan of the existing network. Consequently, AltaLink decided to bypass SDH/SONET technology. An important driver of AltaLink’s decision was the future of smart-grid services with the regulator putting more emphasis on the ability to provision, manage and operate critical services, and at the same time report actual operating statistics. Regardless of whether it is TPR, SCADA, or other services, AltaLink needed to be able to prove its adherence to reliability benchmarks. The combination of cost-effective IP/MPLS technology with a visually-oriented network management system provided AltaLink with a compliant and cost-effective solution. Another part of the equation was the fact that most of the services AltaLink was carrying over their network would likely transition to IP. A roadmap of all existing and future services revealed that teleprotection vendors like ABB or Siemens already offered Ethernet-compatible equipment. IEC 61850 is largely based on Ethernet, and SCADA will go over IP or eventually evolve into a 61850-type of application (eScada). In addition, synchrophasors, metering, and almost everything one encounters in a substation has an IP/MPLS-compatible evolution. Furthermore, an advantage of IP/MPLS compared with TDM-based communication is the flexibility provided by dynamic bandwidth allocation. In contrast to a fixed bandwidth per channel, with IP/MPLS the available bandwidth can be distributed between services as and when needed. Moreover, IP/MPLS provides backward compatibility to a number of traditional communications technologies (see also Table 1) and there is access equipment available with a number of existing interfaces (e.g. RS-232, 4W E&M, V.21 etc.).
Preparing for change
First, AltaLink reviewed the available telecom technology. In late 2008, the company evaluated several IP/MPLS vendors and discussed solutions from the largest manufacturers. In October 2009, AltaLink built a lab for testing various interfaces, circuits, and settings (teleprotection testing, latency, interoperability testing with TDM equipment). Six months later, 24 IP/MPLS nodes had been deployed for a comprehensive field pilot test. In January 2010, the supplier, Alcatel-Lucent, was selected. The actual rollout began in January 2011, with the complete migration planned for 2013. A proper IP transformation is a multi-level task and requires more than just deploying IP/MPLS routers. To ensure a smooth migration, AltaLink identified several measures to facilitate the implementation concerning physical media, services, and cyber security (see Table 1).
Although the rollout is still underway, the two main hurdles AltaLink had to address of how to migrate time-critical services to the new network and the issue of cyber security have been addressed.
Low latency and cyber security
A major concern for utilities is whether the IP/MPLS technology can meet the strict latency requirements for protection signals between transmission substations known as teleprotection (TPR). The doubts over IP/MPLS usually concern the ability to guarantee low latency service and cyber security. First, IP/MPLS is sometimes and incorrectly still perceived as connection-less IP-technology that can provide very cost-efficient data transport but only with a “best-effort” like quality of service (QoS). This is the case for IP only however in contrast, the MPLS part of IP/MPLS makes the solution connection-oriented and capable of multiple guaranteed QoS levels. Second, IP/MPLS is used by many Swiss energy companies for office-related data and other services with moderate latency requirements but its implementation as an alternative for TDM based networks is still rare. While AltaLink belongs to the pioneering companies within the energy sector, IP/MPLS has been used by telecommunication carriers for time-sensitive applications like backhauling of mobile data traffic for many years. Moreover, the Norwegian air traffic controller firm Avinor, along with hundreds of other mission-critical industry networks, successfully have selected IP/MPLS. Between April and September 2010, AltaLink deployed 24 nodes and migrated important services such as teleprotection onto its pilot IP/MPLS network. Teleprotection is the most stringent application that can be transported over utility networks due to its very low delay requirements and the resulting impacts of a failure (e.g. electricity outages or potential equipment damage, both generating huge costs and associated liability). Delay or latency is an area with important differences between TDM and packet-oriented networks such as IP/MPLS. In a TDM world, the latency is fixed (deterministic). With IP/MPLS, latency is not only dependent on the telecom equipment but also on network design. In order to migrate permissive and current differential teleprotection applications on to the IP/MPLS network, the Netcom team first asked the application owners (Asset Management, System Operations or field force personnel) what is the required latency? It turned out that today’s teleprotection applications are typically developed using a total fault clearing time of five to seven 60Hz cycles. As Figure 1 indicates, the fault inception (1) and fault resolution delay (4) take between one and three-to-five cycles, respectively. This leaves one cycle or 16.6ms (20ms in a 50Hz grid) for total end-to-end delay comprised of TPR equipment delay (2) and telecom network delay (3). The TPR delay was found to be around 3ms at each terminal, leaving approximately 10ms for an acceptable total telecom network delay. Telecom network delay consists of packetization delay, network transport delay, and jitter buffer/de-packetization delay. Packetization delay relates to the process of transforming TDM traffic into packet data. For network delay, there is fixed delay based on physical link speed and distances involved. A variable delay depending on the number of hops (nodes) in between. Thereby each node adds a maximum of 150µs equipment latency, 10µs transmission of data latency (for 1500 bytes over a GigE link) and 5µs/km for speed of light (over fibre). For example the network delay (one-way) for a connection between two TPR shelves traversing 10 nodes and 1000km is around 6ms. Finally, what is labeled in Figure 1 jitter buffer and de-packetization delay relates to the time required for a data packet to move out of the (jitter) buffer and to get de-packetized into a TDM stream connecting to the TPR equipment.
The general benchmark of 10ms for telecom network delay enabled Netcom to quickly design and deploy the network and to even have some room to shave off additional milliseconds where needed. This is possible by selecting different settings for MPLS C-pipe service jitter-buffer and payload-size. But reduced service latency does not come for free – it requires a higher bandwidth. For example, reducing latency to approximately 6ms (G.703) one-way consumes nearly 6 times of the available bandwidth on the transport-side of the network (i.e. 384 kbps instead only 1x64kbps). Since the emergence of the buzz-word "cyberwar" and the appearance of the Stuxnet virus in Iran, the topic of cyber security has increased as a priority on many utilities’ agendas. Before the transformation, AltaLink had only a minimal network security strategy in place. There were firewalls protecting the core networks, but nothing at the edge and only minimal password security policies. In the course of the IP/MPLS deployment, the company aimed toward aligning their security up to the level carriers apply for MPLS networks. For example, isolating and securing the control plane was important because if it had been compromised, somebody could gain significant control of the network. To achieve full control plane protection, AltaLink implemented a multi-tier security concept with multiple security layers and intrusion-detection check-points even for administrators in order to have no direct access to the control plane. First, there is a comprehensive password protection at different levels including centralized authentication and logging, allowing users to be quickly isolated and locked out, if necessary. Second, there are security policies for each service (through access control lists, MAC-pinning, IP and bandwidth filters). Moreover, AltaLink also aligns with the North American Electric Reliability Corporation cyber-security policy for critical infrastructure. So far the best input they received and implemented from this regulatory authority is to deploy a centrally-managed and monitored firewall at every substation. As the third layer, this enables protection from malicious traffic close to the source at the edge of the network. Thereby a centralized firewall management solution is used to create and distribute firewall policies on a per service basis by the click of a mouse button. The fourth important security barrier is to use MPLS nodes only to bring the services into the substations via a VLAN trunk, and then hand them to a firewall to apply the per-service firewall policy. Finally, the service connects to a Layer-2 switch representing the inexpensive and simple access port that a field service employee at the site can use to interact with a specific service. This also creates a clear demarcation point for network management which ends at the Layer-2 switch. This means that the security between a given service and the access-switch is under the accountability of the site personnel and thus limiting their ability to affect (local) services used within the specific site.
Made to stick: Takeaways from the AltaLink case
As part of the proof that IP/MPLS works well as a multiservice communications platform for a mission-critical service provider, there are at least three important issues utilities should consider when considering investing in this promising technology. Those issues concern services, the configuration and development of the telecom team, and platforms for facilitating the network change. First, any step towards replacing a communications technology should start by analyzing current and future services to be supported. A technology change for the sake of technology is often the wrong approach. AltaLink made an assessment and opted for IP/MPLS, based on the service portfolio. However, depending on the type and size of the utility, other technologies might be more suitable. Second, the example of AltaLink shows that the implementation of IP/MPLS triggers drastic changes for the employees responsible for the management of the telecom network(s). Generally, these changes can lead to two different scenarios. Either all telecom resources are centralized and dedicated IP/MPLS skill sets are built up or the operations of the network are outsourced. AltaLink clearly decided for the centralization of the skills option. The employees have received significant technology and equipment training. Today, NETCOM consists of 9 employees compared to a total of 4 before the transformation. Third, AltaLink created a good experience by supporting the transformation through several platforms. A first important action was to build up a lab for testing services and to help employees acquire IP skills in a risk-free environment. Another helpful measure was to nurture a strong relationship with the equipment vendor to gain knowledge efficiently. Finally, the maintenance of on-going communication with service-owners was important to ensure concerns are addressed and to demonstrate ongoing performance of the new IP/MPLS network.
The management team of AltaLink has taken an innovative step as one of the first electric utilities worldwide to build an IP/MPLS infrastructure to enable a better management of the transmission infrastructure to ensure more reliability, safety and cost efficiency. By successfully engineering the network to support critical applications such as SCADA and TPR, this next-generation network can not only replace existing TDM networks but is flexible enough for a smart grid future. To contact the authors or request additional information, please send an e-mail to firstname.lastname@example.org. Editor's note: This is a translation of an article that originally appeared in the Swiss electrotechnical and energy magazine called Bulletin SEV/VSE.
- Other companies using IP/MPLS for the transport of time-sensitive grid applications are Energy Australia, or Grand River Dam Authority. Many European energy providers such as Elia, Eskom or Red Electrica are currently doing proofs of concept for implementing the IP/MPLS technology for this purpose.