COVID-19: The network must go on
Podcast episode 21
COVID-19 has created a massive spike in global internet traffic. Telecoms that prepared for a 50 percent increase year-over-year are seeing that growth in just the last month. How do we maintain our networks as usage soars during COVID-19 lockdowns? Nokia’s Rafael de Fermín has answers.
Below is a transcript of this episode of the Futurithmic podcast. Some parts have been condensed for clarity.
Michael Hainsworth: COVID-19 has created unprecedented growth in global internet traffic. While most networks see a year over year increase in demand of 50 percent, we're seeing as much in just the last four weeks in regions shut down by coronavirus. Busy-hour capacity was designed to address the congestion typical of a Sunday night at 9:00PM thanks to online streaming, but now even weekend over the top streaming peaks are significantly higher. And latency-sensitive applications during the weekday, like teleconferencing, are spiking 300 percent.
But to paraphrase Broadway, the network must go on. For insight into the state of the network and the people behind it, we turn to the senior VP of IP and Optics for Europe, the Middle East, and Africa at Nokia, Rafael de Fermín. We began by discussing that COVID-19 has revealed the critical nature of telecom infrastructure.
Rafael de Fermín: Those of us who work in telecoms, we always thought that that was a very critical asset and that, in fact, it is at the core of the building blocks of our society. But we've seen that telecoms perhaps is not on the radar of the society, and politicians, and people in general. Because we sort of take it for granted, and perhaps we don't appreciate its importance day after day.
When you come to a crisis ideation, I think that the important things actually shine on their own. And now everybody understands how crucial it is to have a good telecommunication infrastructure now that everybody's working from home or following education from home. Now everybody understands how crucial telecommunications are. In my opinion, they always were. We just didn't realize because we took it for granted.
Like, we are now appreciating how important it is to be able to walk down the street, go for a walk, go for a run. We never thought those things were that important because, again, we took them for granted.
MH: The EU commissioner is talking about net neutrality legislation as least one streaming video provider throttles its outbound transmissions by 25 percent. What are the implications?
RDF: Multiple. One of the things that we need to think about is, while the highways are open for everybody, and I think that it is good that they open, and to some extent, that would be the case for net neutrality. But what we also need to understand is that some services are more important than others. I think we all understand that when an ambulance has to go, everybody goes to the left-hand side or to the right-hand side of the road so that the ambulance has a clear path. That is probably an extreme example, but there are ambulances running through our networks. And there's a full classification of "not-that-important" services and super-important services.
Maybe we need to start thinking about, while we open the networks to everybody, how do we guarantee that less important services, like the example you mentioned, that perhaps is now flooding networks because people are streaming from home, that it's not doing so in detriment of crucial services that are running through the networks and are absolutely fundamental to keep countries running.
MH: I would have thought quality-of-service type rules within network routers would already ensure that a video streaming that takes place for a conference call at a corporate level gets higher priority than an email or a Facebook post.
RDF: We do have those capabilities, and we can make sure that this happens. Of course, whether those policies are applied or not depend on, "Do all routers out there have all the capabilities to do that?" The ones that Nokia sells would have. And are the service providers willing to impose these different levels of services and to guarantee that certain packets, certain communications, are more important than others and get higher priority? But technically speaking, this can be done today.
During the weekdays, it’s approaching the traffic of the weekends.
MH: Social media and messaging, for example. On the first day of the lockdown versus the previous Sunday, WhatsApp saw... what, a 500 percent increase in activity, network traffic on it? For Netflix, it was up by about 100 percent or so. Total network traffic on the whole over the previous week was up by about 80 percent or so. I understand that a lot of this insight into the traffic flows in the COVID-19 world come courtesy of this thing called Nokia Deepfield. What is that?
RDF: Yes, they do. We have a tool called Nokia Deepfield that is able to understand the individual flows that run through the IP networks of the world. So rather than just looking at the total amount of information that goes through a pipe, or through a network, or towards an end user, we are able to have a view of the different conversations, communications, and flows that make up the total.
This tool, which is very helpful in a number of circumstances and that has been used for network planning for quite some time by many service providers around the world is proving extremely important in crisis. Because it is allowing the service providers not only to see what is the total amount of increase but also why it is increasing, what the components are that represent a significant increase in a significant area of the network, at what times, and what the possible solutions are to avoid a congestion in those areas that are being stretched the most.
MH: So we're seeing unprecedented growth in latency-sensitive applications during business hours. I suppose that’s no surprise with the kids being home. A 400 percent increase in the growth in gaming. But also a 300 percent growth in teleconferencing apps in the United States such as Zoom, as well as Skype. If weekday network load peaks are like what telecom operators generally only expect over the weekend, what does this mean for the network?
RDF: This is indeed what we are seeing at this point of time. First, we see that because people are working from home, there’s an increase in VPNs and in certain other tools, and also because the kids who are home from school are also streaming a lot more on a Wednesday than they used to, right?
Putting all of that together, and by the way, the streaming of the kids is a lot more important in volume than the VPNs of their fathers. Because even though you experience a 400 percent increase in VPN, the traffic is still a fraction of what Netflix alone would represent.
During the weekdays, it's approaching the traffic of the weekends. That is okay. In principle, the network is supposed to be able to handle it, but, and there is an interesting ‘but’, it's not actually happening in the same places. Traffic on the weekends is basically streaming and all kind of enjoyment-related kind of traffic, if I may say so. Traffic during the working days is happening in other places. Part of it is streaming, but maybe you also start to get some kind of loads in connections to data centers and things like that.
So the first thing that service providers need to take care of is what's happening during the weekdays. Because while the totality of the network may still be okay, there could be links, routers, or locations that are approaching saturation. Now on the weekends, what we see is that the kids, and the parents perhaps as well, used to do things outside. Used to go and play a basketball game. Used to go to the theater. They can't do those things anymore, so guess what? They stream a lot. This represents a significant increase in the weekend traffic now that’s very consistent in terms of shapes. What's happening during the weekends is basically the same thing qualitatively that was happening a few weekends before. Quantitatively, there's a lot more of that.
It will now be alleviated temporarily a little bit by companies like Netflix coming down by 25 percent in their bitrate in Europe, as an example, but I think it will continue to grow. Because we only just started, and people will learn, and people will get into the different tools that some of them have just discovered. As such, they will be able to exploit more of the capabilities that this new world is offering them. So I do expect that weekend traffic is also going to continue growing.
MH: I want to talk a little bit about exploitation. First though, help me understand what staffing and network operation centers and security operation centers looks like right now amid COVID-19 shutdown.
RDF: Well, that's a very interesting thing. Some of those network operations people need to go to the network operation center, which is the first level of stress because in some countries, that’s not as easy. Some of the activities and tasks can be done remotely, but some of them actually need to happen in the network operation center.
The second point of stress is to make sure that network operation center has the best possible tools to understand what's happening in the network and to be able to react quickly.
MH: So, bad actors take advantage of bad situations. We're seeing crisis profiteers buying up toilet paper. What about network security at a time when teams are working with skeleton crews?
RDF: You always have that, and to be honest that is really sad. But you've got to be prepared for that. In terms of crisis, there will always be a bad guy who will try to profit from the crisis. I'm afraid we are starting to see scams. We are starting to see attacks. I even read about an organized attack to the hospitals in a European country. Which to be honest, if you have even a little bit of humanity, that would be probably the last thing you would be thinking about these days.
But we can't relax. We need to understand that there's some people out there that are willing to try and profit from these difficult times. So we need to be prepared to fight against them to the best of our abilities.
MH: And how do we go about that?
RDF: There's multiple ways. There's all kinds of security tools that many service providers are putting in place. But one thing that we have started doing very recently is we've started to take the abilities of this Nokia Deepfield tool to use it in a different way. It was initially created to understand the flows in a network and be able to make some decisions in network planning. That was the original idea. But we soon realized that if you understand what's happening inside the network, you also have a tool that is able to give you a lot of information about potential attacks in that network. That gives you extreme capabilities of understanding when the attack is happening, and then blocking the attack quickly.
Today, the way most of those attacks are handled is that suspicious traffic is routed to our scrubbing center, and then it is treated there. Of course, that means that we introduce a significant load in the network from wherever those packets are detected all the way through to the scrubbing center for treatment. What if we could drop the packets whenever they are identified? That is something that we are starting to put in place in some networks around Europe, and it's in cases like that, allowing you to quickly offload the network from that massive amount of traffic, that constitutes the attack.
And all of sudden, the growth that you have planned to happen in a year, maybe a year and a half, actually happened last week. It’s there now.
MH: With the growth that was expected to happen over one to two years now happening over one to two weeks, the CDC and others are talking about this pandemic being with us for months upon months. How do CSPs plan for sustained strain on the networks?
RDF: They have network planning that basically was going to cope for the coming two, three years. You always plan ahead. And all of sudden, the growth that you have planned to happen in a year, maybe a year and a half, actually happened last week. It's there now. Obviously you're never running your network so hot that you can't cope with this kind of traffic.
As I said before, it might require a tweak or two in different places because maybe the traffic is not coming from exactly where you expected, etc., but nothing that people cannot cope with short-term. But CSPs are already starting to think, "Okay, but what if the peak on the first weekend after the lockout is not the peak? It's just the first peak. What about the second weekend, and the third weekend, and the fourth weekend? Am I just about to experience a tsunami of data in my network?"
Potentially, that can happen. So what we see is telcos bringing forward some of the expansions that they already had planned because they expected that kind of traffic to happen. They just didn't expect it to happen so soon. So we see a lot of companies bringing that forward in time and making sure that they are able to react appropriately to this amount of data that is getting to them way earlier than they ever anticipated.
MH: CSPs have rollout plans for expansion and building out their infrastructure. But because the demand as we see it now is coming from more residential communities, more homes, we deal with last-mile issues more so than, "Well, we have to increase the build out in city centers where the corporate world was going to use it." How do we shift gears like that? How do we change streams? How do we ensure that we can build out where the build-out needs to be but still maintain a long-term focus? Because at some point, this is going away.
RDF: First of all, I don't really think that you need to change gears. I think that the traffic was expected over 18 months that has happened in 18 days is more or less where we predicted it would be. Because over 18 months, we thought that there was going to be a lot more streaming, that more people would sign up to Netflix, and Hulu, etc. So it basically is happening where you thought it was going to happen. Maybe, as I said, there are a couple of tweaks because you never anticipated this big amount of traffic from home with people teleworking. But again, that is a very small part of the totality of the traffic. The big thing is not fundamentally changing gears from what you had planned if you're a service provider.
The second is the question of bringing forward all that investment. “Do I need to throw it away?" Because you know, when the whole COVID thing goes away - and god knows when, but we hope it'll be sooner rather than later - do I now have extra capacity in my network that I won't use? Well, first of all they were planning to use it in a year from now anyhow. So the maximum that could really happen is that maybe the network is slightly empty for a quarter or something like that, but to be honest I don't even think that will be the case.
There's a lot of people that have jumped onto some of those platforms that they didn't need to. They had other kinds of pastimes. They spent their spare time in a different way, and they might not have found Netflix yet, if you know what I mean. They found it now, and it is addictive. Once you get into streaming, once you appreciate all the things that they can offer to you, you're probably not going to sign off. You're probably going to stay.
So actually, I don't really anticipate to see that volley. I think that the traffic that has come is there to stay. The CSPs already planned that this was going to happen in 12-18 months from now. It happened faster, but I don't anticipate we're going to see a drop in network traffic when COVID19 goes out by any means. I think that we advanced some of those expansions, but this is a good investment.
Perhaps the third part of this question, is the challenge we're going to find in our ability to actually rollout those expansions if COVID19 gets more and more serious and it is more difficult for the installers or whoever to go to sites and plug the boards and build the routers. I think many governments are already recognizing how important telecommunications are, and many governments have already declared that telecoms is strategic. Like medicines, and food, and other supplies chains that you need to maintain under a moment of crisis like this one.
MH: If Nokia Deepfield reveals that the stress is mostly on the aggregation networks and the service edge routers where demand is reaching capacity maximums, telecom operators may need to make fresh choices about their gear. What roles do commercial pressures, quality, reliability, and availability play in those decisions?
RDF: I think that's a very good question, and I think that also comes back to the beginning of our conversation that in times of crisis, the really important things show up naturally. Only last year when we thought the world was “normal” and we never thought that anything like this pandemic would happen, service providers out there selected a router that was balanced in multiple areas. Clearly, the quality, the stability, the reliability of that router was very important. But the commercials of that router were very important as well.
People make choices in different ways. And you have the whole scale. You have people who selected, taking into account mostly the quality of the product; you have people who were somewhat in the middle. And you have service providers that selected more, taking into account the commercials of the transaction.
I think today, the ones who build solid, very reliable networks sleep better than the ones who build a cheaper, but potentially more challenged IP network. I think perhaps moving forward, people again will understand that those priorities were always there. We just didn't see how important being able to run in the street was in those days.
So equally, we didn't appreciate how important it is to have a good, solid, reliable network that doesn't fail you. I think moving forward perhaps people will edge a little bit more on that side when making these kind of decisions. Because I think reality now has proven to us how important those decisions are.