The journey to automated and autonomous networks

 

The TM Forum has been instrumental in defining the journey towards automation and autonomous telco networks. 

As telco revenues from consumers continue to decline and the 5G promise to create connectivity products that enterprises, governments and large organizations will be able to discover, program and consume remains elusive, telecom operators are under tremendous pressure to maintain profitability.

The network evolution started with Software Defined Networks, Network Functions Virtualization and more recently Cloud Native evolution aims to deliver network programmability for the creation of innovative on-demand connectivity services. Many of these services require deterministic connectivity parameters in terms of availability, bandwidth, latency, which necessitate end to end cloud native fabric and separation of control and data plane. A centralized control of the cloud native functions allow to abstract resource and allocate them on demand as topology and demand evolve.

A benefit of a cloud native network is that, as software becomes more open and standardized in a multi vendor environment, many tasks that were either manual or relied on proprietary interfaces can now be automated at scale. As layers of software expose interfaces and APIs that can be discovered and managed by sophisticated orchestration systems, the network can evolve from manual, to assisted, to automated, to autonomous functions.

TM Forum defines 5 evolution stages from full manual operation to full autonomous networks.

• Condition 0 - Manual operation and maintenance: The system delivers assisted monitoring capabilities, but all dynamic tasks must be 0 executed manually
• Step 1 - Assisted operations and maintenance: The system executes a specific, repetitive subtask based on pre-configuration, which can be recorded online and traced, in order to increase execution efficiency.
• Step 2: - Partial autonomous network: The system enables closed-loop operations and maintenance for specific units under certain external environments via statically configured rules.
• Step 3 - Conditional autonomous network: The system senses real-time environmental changes and in certain network domains will optimize and adjust itself to the external environment to enable, closed-loop management via dynamically programmable policies.
• Step 4 - Highly autonomous network: In a more complicated cross-domain environment, the system enables decision-making based on predictive analysis or active closed-loop management of service-driven and customer experience-driven networks via AI modeling and continuous learning.
• Step 5 - Fully autonomous network: The system has closed-loop automation capabilities across multiple services, multiple domains (including partners’ domains) and the entire lifecycle via cognitive self-adaptation.

After describing the framework and conditions for the first 3 steps, the TM Forum has recently published a white paper describing the Level 4 industry blueprints.

The stated goals of level 4 are to enable the creation and roll out of new services within 1 week with deterministic SLAs and the delivery of Network as a service. Furthermore, this level should allow fewer personnel to manage the network (1000's of person-year) while reducing energy consumption and improving service availability.

These are certainly very ambitious objectives. The paper goes on to describe "high value scenarios" to guide level 4 development. This is where we start to see cognitive dissonance creeping in between the stated objectives and the methodology.  After all, much of what is described here exists today in cloud and enterprise environments and I wonder whether Telco is once again reinventing the wheel in trying to adapt / modify existing concepts and technologies that are already successful in other environments.

First, the creation of deterministic connectivity is not (only) the product of automation. Telco networks, in particular mobile networks are composed of a daisy chain of network elements that see customer traffic, signaling, data repository, look up, authentication, authorization, accounting, policy management functions being coordinated. On the mobile front, the signal effectiveness varies over time, as weather, power, demand, interferences, devices... impact the effective transmission. Furthermore, the load on the base station, the backhaul, the core network and the  internet peering point also vary over time and have an impact on its overall capacity. As you understand, creating a connectivity product with deterministic speed, latency capacity to enact Network as a Service requires a systemic approach. In a multi vendor environment, the RAN, the transport, the core must be virtualized, relying on solid fiber connectivity as much as possible to enable the capacity and speed. The low latency requires multiple computing points, all the way to the edge or on premise. The deterministic performance requires not only virtualization and orchestration of the RAN, but also the PON fiber and end to end slicing support and orchestration. This is something that I led at Telefonica with an open compute edge computing platform, a virtualized (XGS) PON on a ONF ONOS VOLTHA architecture with an open virtualized RAN. This was not automated yet, as most of these elements were advanced prototype at that stage, but the automation is the "easy" part once you have assembled the elements and operated them manually for enough time. The point here is that deterministic network performances is attainable but still a far objective for most operators and it is a necessary condition to enact NaaS, before even automation and autonomous networks.

Second, the high value scenarios described in the paper are all network-related. Ranging from network troubleshooting, to optimization and service assurance, these are all worthy objectives, but still do not feel "high value" in terms of creation of new services. While it is natural that automation first focuses on cost reduction for roll out, operation, maintenance, healing of network, one would have expected more ambitious "new services" description.

All in all, the vision is ambitious, but there is still much work to do in fleshing out the details and linking the promised benefits to concrete services beyond network optimization.

The AI-native telco network

AI, and more particularly generative AI has been a big buzzword since the public launch of GTP. The promises of AI to automate and operate complex tasks and systems are pervading every industry and telecom is not impervious to it. 

Most telecom equipment vendors have started incorporating AI or brushed up their big data / analytics skills at least in their marketing positioning. 

We have even seen a few market acquisitions where AI / automation has been an important part of the investment narrative / thesis (HPE / Juniper Networks). 

Concurrently, many startups are being founded or are pivoting towards AI /ML to take advantage of this investment cycle. 

In telecoms, there has been use for big data, machine learning, deep learning and other similar methods for a long time. I was leading such a project at Telefonica on 2016, using advanced prediction algorithms to detect alarming patterns, infer root cause analysis and suggest automated resolutions. 

Other projects used machine vision to count houses and settlements at the scale of a country and superimpose telco networks to divine where are the best deployment strategies and technologies for radio access networks. 

While generative AI is somewhat new, the use of data to analyze, represent, predict network conditions is well known. 

AI in telecoms is starting to show some promises, particularly when it comes to network planning, operation, spectrum optimization, traffic prediction, and power efficiency. It comes with a lot of preconditions that are often glossed over by vendors and operators alike. 

Like all data dependent technologies, one has first to have the ability to collect, normalize, sanitize and clean data before storing it for useful analysis. In an environment as idiosyncratic as a telecoms network, this is not an easy task. Not only networks are composed of a mix of appliances, virtual machines and cloud native functions, they have had successive technological generations deployed along each other, with different data schema, protocols, interface, repository which makes the extraction arduous. After that step, normalization is necessary to ensure that the data is represented the same way, with the same attributes, headers, … so that it can be exploited. Most vendors have their proprietary data schemes or “augment” standard with “enhanced” headers and metadata. In many case the data need to be translated in a format that can be normalized for ingestion. The cleaning and sanitizing is necessary to ensure that redundant or outlying data points do not overweight the data set. As always, “garbage in / garbage out” is an important concept to keep in mind. 

These difficult steps are unfortunately not the only prerequisite for an AI native network. The part that is often overlooked is that the network has to be somewhat cloud native to take full advantage of AI. The automation in telecoms networks requires interfaces and APIs to be defined, open and available at every layer, from access to transport to the core, from the physical to the virtual and cloud native infrastructure. NFV, SDN, network disaggregation, open optical, open RAN, service based architecture, … are some of the components that can enable a network to take full advantage of AI. 

Cloud networks and data centers seem to be the first to adopt AI, both for the hosting of the voracious GPUs necessary to train the Large Language Models and for the resale / enablement of AI oriented companies. 

For that reason, the more recent greenfield networks that have been recently deployed with the state of the art cloud native technologies should be the prime candidates for AI / ML based network planning, deployment and optimization. The amount of work necessary for the integration and deployment of AI native functions is objectively much lower than their incumbent competitors. 

We haven’t really seen sufficient evidence that this level of cloud "nativeness" enables mass optimization and automation with AI/ML that would result in massive cost savings in at least OPEX, creating a unfair competitive advantage against their incumbents. 

As the industry approaches Mobile World Congress 2024, with companies poised to showcase their AI capabilities, it is crucial to remain cognizant of the necessary prerequisites for these technologies to deliver tangible benefits. Understanding the time and effort required for networks to truly benefit from AI is essential in assessing the realistic impact of these advancements in the telecom sector.

Telco grade or Cloud grade?

 

For as long as I can remember, working in Telco, there has been the assumption that Telco networks were special. 

They are regulated, they are critical infrastructure, they require a level of engineering and control that goes beyond traditional IT. This has often been the reason why some technologies and vendors haven't been that successful in that space, despite having stellar records in other equally (more?) demanding industries such as energy, finance, space, defence...

Being Telco grade, when I cut my teeth as a telco supplier, meant high availability (5x9's), scalability and performance (100's of millions of simultaneous streams, connections, calls, ...), security, achieved with multiple vertical and horizontal redundancies, and deployed of highly specialized appliances.

Along comes the Cloud, with its fancy economics, underpinned by separation of hardware and software, virtualization, then decomposition, then disaggregation of software elements into microservices. Add to it some control / user plane separation, centralized control, management, configuration, deployment, roll out, scalability rules... a little decentralized telemetry and systematic automation through radical opening of API between layers... That's the recipe for Cloud grade networks.

At the beginning, the Telco-natives looked at these upstarters with a little disdain, "that's good for web traffic. If a request fail, you just retry, it will never be enough for Telco grade...". 

Then with some interest "maybe we can use that Cloud stuff for low networking, low compute stuff like databases, inventory management... It's not going to enable real telco grade stuff, but maybe there is some savings".

Then, more seriously "we need to harness the benefits of the cloud for ourselves. We need to build a Telco cloud". This is about the time the seminal white paper on Telco virtualization launched NFV and a flurry of activities to take IT designed cloud fabric (read Openstack) and make it Telco grade (read pay traditional Telco vendors who have never developed or deployed a cloud fabric at scale and make proprietary branches of an open source project hardened with memorable features such as DPDK SR-IOV, CPU pinning so that the porting of their proprietary software on hypervisor does not die under the performance SLA...). 

Fast forward a few years, orchestration and automation become the latest targets, and a zoo of competing proprietary-turned-open-source projects start to emerge, whereas large communities of traditional telco vendors are invited to contribute charitably time and code on behalf of Telcos for projects that they have no interest in developing or selling.

In the meantime, Cloud grade has grown in coverage, capacity, ecosystem, revenues, use cases, flexibility, availability, scalability... by almost any metrics you can imagine, while reducing costs and prices. Additionally, we are seeing new "cloud native" vendors emerge with Telco products that are very close to the Telco grade ideal in terms of performance, availability, scalability, at a fraction of the cost of the Telco-natives. Telco functions that the Telco-native swore could never find their way to the cloud are being deployed there, for security, connectivity, core networks, even RAN...

I think it is about time that the Telco-natives accept and embrace that it is probably faster, more cost efficient and more scalable to take a Cloud-native function and make it Telco-grade than trying to take the whole legacy Telco network and trying to make it Cloud grade. It doesn't mean to throw away all the legacy investment, but at least to consider sunsetting strategy and cap and grow. Of course, it means also being comfortable with the fact that the current dependencies of traditional Telco vendors might have to be traded for dependencies on hyperscalers, who might, or not become competitors down the line. Not engaging with them, si not going to change that fact. 5G stand alone, Open RAN or MEC are probably good places to start, because they are greenfield. This is where the smart money is these days, as entry strategy into Telco world goes...

Announcing Edge computing and hybrid clouds workshops

After working 5 years on edge computing and potentially being one of the only analysts having evaluated, then developed and deployed the technology in a telco networks, I am happy to announce immediate availability of the following workshops:

Hybrid and edge computing strategy

• Hybrid cloud and Edge computing opportunity 

• Demand for hybrid and edge services (internal and external)

• Wholesale or retail business?

• Edge strategies: what, where, when, how?

• Hyperscalers strategies, positions, risks and opportunities

• Operators strategies

• Conclusions and recommendations

Edge computing Technology

• Technological trends

• SDN, NFV, container, lifecycle management

• Open source, ONF, TIP, Akraino, MobiledgeX, Ori

• Networks disaggregation, Open RAN, Open OLT

• Edge computing: Build or buy?

• Nokia, Ericsson, Huawei

• Dell, Intel, …

• Open compute, CORD

• Conclusions and recommendations

Innovation and transformation processes

• Innovation process and methodology 

• How to jumpstart technological and commercial innovation

• Labs, skills, headcount and budget

• How to transition from innovation to commercial deployment

• How to scale up sustainably

Drop me a line if you are interested.

Telefonica and the edge computing opportunity

Real world use cases of edge Computing With Intel, Affirmed Networks and Telefonica

State of NFV at MWC18

Video of the presentation at TIP 2017: Telefonica's Internet para todos

This is the video describing the project "internet para todos", connecting the unconnected in LatAm.
I present the industry trends and constraints that force telcos reeaxamine their model and the necessary changes in the value chain and the technology to enable ultra low cost versatile networks to connect the unconnected

Internet Para Todos: Connecting the Unconnected in LATAM

Patrick Lopez, VP, Networks Innovation, Telefonica

Telefonica´s Internet para todos

Presented today at #TIPSUMMIT , on the need for the #Telco industry to evolve to connect the unconnected, what @Telefinnovation is doing about it, from applying #DataScience to HD population density modeling with @facebook , and @TelecomInfraP, to assembling innovative networks and commercial and operating models with LatAm partners, #SDN #NFV #edgecomputing #opencellular

Telefonica internet para todos from Patrick Lopez

Operators transformation strategies - TELECOMtv

Telecom TV interview at Networks Functions Virtualization World Congress

Customer Centric Networks and SDN NFV

These slides were presented in May 2017 at the NFV World Congress in San Jose.
They illustrate how we are looking at deploying cloud microservices at the edge of our networks to provide unique experiences using SDN, open source and NFV.

Telefonica CCN and SDN / NFV from Patrick Lopez

Telefonica's innovation framework

I have received many requests over the last months to explain in more details our innovation process. Now that our innovation methodology is a widely commented Harvard Business Review Case Study, I thought it was a good time to shed some light on how a large telco such as Telefonica can innovate in a fast paced environment.
Innovation is not only a decision, it's a process, a methodology. In our case we have different teams looking after external innovation, through business ventures and venture capital and internal looking after networks, data, and moonshots. The teams that I support, focusing on networks innovation are adapting the lean elephant methodology to invent tomorrow's mobile, fixed and TV networks.

Ideation

The process starts with directed ideation, informed by our corporate customer segmentation, customer sentiment studies and selected themes. An innovation call centered around specific themes such as "imagine tomorrow's TV" or "Artificial intelligence and networks QoE" is launched across the group, with local briefings including our selection parameters. A jury is convened to review the hundreds of ideas and shortlist the most interesting. The selected intrapreneurs have a month to prepare a formal pitch for their ideas. They are assisted by customer experience specialists who help them refine the problem they seek to resolve, its applicability and market appeal.

Feasibility

After the pitch and selection, the intrapreneurs are transitioned to the innovation team full time and given a few weeks to create a feasibility plan and preliminary resource budget for prototyping. Once ready, the successful applicants present the plan in details to the jury.

Prototyping

The lucky few that pass this gate are given 3 to 8 months to prototype their project, together with commensurate resource. At this stage, the project must have strong internal sponsorship, with verticals or markets within Telefonica who are committing to take the prototype in their labs for functional testing. The resulting prototype, together with the value proposition and addressable market are reviewed before passing to the next phase.

Market trial

The prototype is then hardened and deployed in a commercial network for friendly and limited A/B testing and refinement. This phase can last 2 to 6 months, with increasing number of users and sophistication in measurement of the value proposition's effectiveness. During this phase as well, a full product / service business case is finalized, using the data collected during the market trial.

Productization and transfer

The project meets customer needs? It is innovative and provides differentiation? It is profitable and Telefonica has an unfair advantage in solving real market problems? These are some of the tough questions the intrapreneur and his team must be able to answer before the solution can be productized and eventually transferred to one of our verticals or to create a new one.


This process has been the source of Telefonica's early advances in IoT, big data, smart cities... It has also killed, merged, pivoted and spun off hundreds of projects. The network innovations teams I support are aiming at radically changing networks topology, deployment and value chain using software defined networks, virtualization, containerization and lambda computing all the way to the edge of our networks. We are developers, network hackers, user experience experts, computer scientists, devops engineers,....

The next months will see some exciting announcements on this. Stay tuned.

You can catch me and we can chat about it at the upcoming NFV world congress or TM Forum live.

What is left for operator to enable SDN and NFV?

Debate: What is left for operator to enable SDN and NFV?

In a live debate held last week at Mobile World Congress, Patrick Lopez, VP Networks Innovation, Telefonica, and Manish Singh, VP Product Management, SDN & NFV, Tech Mahindra, joined TMN editor Keith Dyer to discuss what operators are hoping to achieve with the adoption of NFV and SDN.

The panel asked what the end goals are, and looked at the progress operators have made so far, picking out key challenges that operators still face around integration, certification and onboarding of VNFs, interoperability, the role of orchestration and the different Open Source approaches to NFV MANO.

The panel also looked at how operators can adapt their own cultures to act in a more agile way, adopting continuous integration and DevOps models.

Key quotes:

Lopez: “The end game is the ability to create services that are more customer-centric and enable operators to provide real value to consumers, things and enterprises by providing experiences that are tailored for them. And to be able to do that you need to have an infrastructure that is very elastic and very agile – that’s where SDN and NFV comes in.”

Singh: “As we dis-aggregate the hardware from the software, and get to this virtualised infrastructure layer where different network functions are orchestrated – integration, performance characterisation, capacity planning and onboarding all become challenges that need to be addressed

Singh: “There has been ecosystem fragmentation in the orchestration layer and for the VNF vendors that was creating challenges in terms of, ‘How many orchestrators, how many VIMs on the infrastructure layer do I support?'”

Lopez: “It’s really hard to create an industry that is going to grow if we don’t all share the same DNA.”

Singh: “The good news is there is a vibrant ecosystem, and I think having a couple of key alternatives as we drive forward is a good thing. And we see an inflection point where a new way of standardising things is coming up, and that really sets the way for 5G.”

Lopez: “You cannot implement automation well if you don’t understand how you have deployed that NFV-SDN technology. You need to implement that itself to understand the gotchas to be able to automate.”

Singh: “As we look at SDN NFV the other key aspect is the ability to bring new player, VNFs and components into the fold and we are enabling that to be done cost effectively, efficiently and rapidly.”

Lopez: “It [SDN-NFV] works, we can achieve the main core benefits of the technology. It can do what we were planning to do – to run a software defined network. We are there, now it is about optimising it and making it run better and automating it.

World's first ETSI NFV Plugfest

As all know in the telecom industry, the transition from standard to implementation can be painful, as vendors and operators translate technical requirements and specifications into code. There are always room for interpretation and desires to innovate or differentiate that can lead to integration issues. Open source initiatives have been able to provide viable source code for implementation of elements and interfaces and they are a great starting point. The specific vendors and operators’ implementations still need to be validated and it is necessary to test that integration needs are minimal.

Networks Function Virtualization (NFV) is an ETSI standard that is a crucial element of telecom networks evolution as operators are looking at their necessary transformation to accommodate the hyper growth resulting from video services moving to online and mobile.

As a member of the organization’s steering committee, I am happy to announce that the 5G open lab 5Tonic will be hosting the world’s first ETSI NFV plugfest from January 23 to February 3, 2017 with the technical support of Telefonica and IMDEA Networks Institute.  

5Tonic is opening its doors to the NFV community, comprising network operators, vendors and open source collaboration initiatives to assert and compare their implementations of Virtual Network Functions (VNFs), NFV Infrastructure and Virtual Infrastructure Manager. Additionally, implementations of Management and Orchestrations (MANO) functions will also be available.

43 companies and organizations have registered to make this event the largest in NFV interoperability in the world.

Companies:

•           Telefonica

•           A10

•           Cisco

•           Canonical

•           EANTC

•           EHU

•           Ensemble

•           Ericsson

•           F5

•           Fortinet

•           Fraunhofer

•           HPE

•           Huawei

•           Inritsu

•           Intel

•           Italtel

•           Ixia

•           Keynetic

•           Lenovo

•           Mahindra

•           Openet

•           Palo Alto

•           Radware

•           RIFT.io

•           Sandvine

•           Sonus

•           Spirent

•           RedHat

•           VMWare

•           WIND

Open source projects:

•           OSM (Open Source MANO)

•           Open Baton

•           Open-O

•           OPNFV

 OSM is delivering an open source MANO stack aligned with ETSI NFV Information Models. As an operator-led community, OSM is offering a production-quality open source MANO stack that meets the requirements of commercial NFV networks.

Testing will take place on site at the 5TONIC lab near Madrid, as well as virtually for remote participants.

Innovation and transformation, micro segments and strands

When I first met the CEO of Telefonica Research and Development, David Del Val, he asked me what I thought of the direction the industry was taking. I have not been shy on this blog and other public forum about my opinion on operators' lack of innovation and transformation. My comments went something like that:

"I think that in a time very soon, I don´t know if it´s going to be in 3 years, 5 or 10, voice will be free, texts will be free, data will be free or as close to a monthly utility price as you can think. Already, countries are writing access to broadband in their citizens´ fundamental rights. Most operators are talking about innovation and new services, but let´s face it, they have had a pretty poor track record. MMS was to be the killer app for GPRS/EDGE, push to talk for 3G,video calling for HSPA, VoLTE for 4G... There is no shame in being an operator of a very good, solid, inexpensive connectivity service. Some companies are very successful doing that and there will be more in the future. But you don't need hundreds of thousands of people for that. If operators' ambition is to "monetize", "launch new services", "open new revenue streams", "innovate", they have to transform first. And it's gonna hurt."

At that point, I wasn't sure I had made the best first impression, but as you know now, that discussion ended up turning into a full time collaboration. 

The industry is undergoing changes that will accelerate and break companies that are not adaptable or capable of rethinking their approach. 

4G wasn’t designed as a video network capable of doing other things like browsing and voice; the telecoms industry designed 4G to be a multipurpose mobile broadband network, capable of carrying VoIP, browsing, messaging, … but really, it wasn’t so hard to see that video would be the dominant part of traffic and cost and growing. I don´t have a crystal ball but I had identified publicly the problem more than 7 years ago.

The industry’s failure to realize this has led us in a situation where we have not engaged video providers early enough to create a mutually profitable business model. The result is traffic is increasing dramatically across all networks, while revenues are stagnating or decreasing because video services are mostly encrypted. At the same time, our traditional revenues from voice and messaging are eroded by other providers. 

As the industry is gearing up towards 5G and we start swimming in massive MIMO, beam-forming, edge computing, millimeter wave, IoT, drone and autonomous vehicles, I think it is wise to understand what it will take to really deliver on these promises.

Agile, lean, smart, open, software-defined, self organizing, autoscalable, virtualized, deep learning, DevOps, orchestrated, open-source... my head hurts from all the trappings of 2016´s trendy telco hipster lingo. 

This is not going to get better in 2017.

The pressure to generate new revenues and to decrease costs drastically will dramatically increase on operators. There are opportunities to create new revenue streams (fintech, premium video, IoT…) or reduce costs (SDN, NFV, DevOps, Open source…) but they require initial investments that are unsure from a business case perspective because they are unproven. We are only starting to see operators who have made these investments over the last 3 years announcing results now. These investments are hard to make for any operator, because they are not following our traditional model. Operators for the last 20 years have been conditioned to work in standards to invent the future collectively and then buy technology solutions from large vendors. The key for that model was not innovation, it was sustainability, interoperability.

The internet has broken that model.

·      

I think that operators who want to be more than a bit pipe provider need to create unique experiences for consumers, enterprises, verticals and things. Unique experiences can only be generated from context (understanding the customer, his desire, intent, capacity, limitations...), adaptation (we don't need slices, we need strands) and control (end to end performance, QoS and QoE per strand). Micro segmentation has technical, but more importantly operational and organizational impacts.

Operators can't hope to control, adapt, contextualize and innovate if they can't control their network. Today, many have progressively vacated the field of engineering to be network administrators, writing RFPs to select vendors, or better, mandate integrators to select and deploy solutions. The result is networks that are very undifferentiated, where a potential "innovation" from one can be rolled out by another with a purchase order, where a change in a tariff, a new enterprise customer on-boarding, a new service takes years to deploy, hundreds of people, and millions of euros. 

Most operators can't launch a service if it has less than 10 million people addressable market, or it won't make the business case, right off the bat.

There are solutions, though, but they are tough medicine. You can't really rip the rewards of SDN or NFV if you don't control their implementation. It's useless to have a programmable network, if you can't program. Large integrators and vendors have made the effort to retool, hire and train. Operators must do the same unless they want to be MVNOs on their own networks. 

Innovation is trying. Projects can fail, technology evolves, but transformation is sustainable.

SDN / NFV: Enemy of the state

Extracted from my SDN and NFV in wireless workshop.

I want to talk today about an interesting subject I have seen popping up over the last six months or so and in many presentations in the stream I chaired at the NFV world congress a couple of months ago.

In NFV and to a certain extent in SDN as well, service availability is achieved through a combination of functions redundancy and fast failover routing whenever a failure is detected in the physical or virtual fabric. Availability is a generic term, though and covers different expectations whether you are a consumer, operator or enterprise. The telecom industry has heralded the mythical 99.999% or five nines availability as the target to reach for telecoms equipment vendors.

This goal has led to networks and appliances that are super redundant, at the silicon, server, rack and geographical levels, with complex routing, load balancing and clustering capabilities to guarantee that element failures do not impact catastrophically services. In today's cloud networks, one arrives to the conclusion that a single cloud, even tweaked can't performed beyond three nines availability and that you need a multi-cloud strategy to attain five nines of service availability...

Consumers, over the last ten years have proven increasingly ready to accept a service that might not be always of the best quality if the price point is low enough. We all remember the start of skype when we would complain of failed and dropped calls or voice distortions, but we all put up with it mostly because it was free-ish. As the service quality improved, new features and subscriptions schemes were added, allowing for new revenues as consumers adopted new services.
One could think from that example that maybe it is time to relax the five nines edict from telecoms networks but there are two data points that run counter to that assumption.

1. The first and most prominent reason to keep a high level of availability is actually a regulatory mandate. Network operators operate not only a commercial network but also a series of critical infrastructure for emergency and government services. It is easy to think that 95 or 99% availability is sufficient until you have to deliver 911 calls, where that percentage difference means loss of life.
2. The second reason is more innate to network operators themselves. Year after year, polls show that network operators believe that the way they outcompete each others and OTTs in the future is quality of service, where service availability is one of the first table stakes. 

As I am writing this blog, SDN and NFV in wireless have struggled through demonstrating basic load balancing and static traffic routing, to functions virtualization and auto scaling over the last years. What is left to get commercial grade (and telco grade) offerings is resolving the orchestration bit (I'll write another post on the battles in this segment) and creating a service that is both scalable and portable.

The portable bit is important, as a large part of the value proposition is to be able to place functions and services closer to the user or the edge of the network. To do that, an orchestration system has to be able to detect what needs to be consumed where and to place and chain relevant functions there.
Many vendors can demonstrate that part. The difficulty arises when it becomes necessary to scale in or down a function or when there is a failure.

Physical and virtual functions failure are to be expected. When they arise in today's systems, there is a loss of service, at least for the users that were using these functions. In some case, the loss is transient and a new request / call will be routed to another element the second time around, in other cases, it is permanent and the session / service cannot continue until another one is started.

In the case of scaling in or down, most vendors today will starve the virtual function and route all new requests to other VMs until this function can be shut down without impact to live traffic. It is not the fastest or the most efficient way to manage traffic. You essentially lose all the elasticity benefits on the scale down if you have to manage these moribund zombie-VNFs until they are ready to die.

Vendors and operators who have been looking at these issues have come to a conclusion. Beyond the separation of control and data plane, it is necessary to separate further the state of each machine, function service and to centralize it in order to achieve consistent availability, true elasticity and manage disaster recovery scenarios.

In most cases, this is a complete redesign for vendors. Many of them have already struggled to port their product to software, then port it to hypervisor, then optimized for performance... separating state from the execution environment is not going to be just another port. It is going to require redesign and re architecting.

The cloud-native vendors who have designed their platform with microservices and modularity in mind have a better chance, but there is still a series of challenges to be addressed. Namely, collecting state information from every call in every function, centralizing it and then redistribute it is going to create a lot of signalling traffic. Some vendors are advocating some inline signalling capabilities to convey the state information in a tokenized fashion, others are looking at more sophisticated approaches, including state controllers that will collect, transfer and synchronize relevant controllers across clouds.
In any case, it looks like there is still quite a lot of work to be done in creating truly elastic and highly available virtualized, software defined network.

Time to get out of consumer market for MNOs?

I was delivering a workshop on SDN / NFV in wireless, last week, at a major pan-european tier one operator group and the questions of encryption and net neutrality were put again on the table.

How much clever, elastic, agile software-defined traffic management can we really expect when "best effort" dictates the extent of traffic management and encryption renders many efforts to just understand traffic composition and velocity difficult?

There is no easy answer. I have spoken at length on both subjects (here and here, for instance) and the challenges have not changed much. Encryption is still a large part of traffic and although it is not growing as fast as initially planned after Google, Netflix, Snapchat or Facebook's announcements it is still a dominant part of data traffic. Many start to think that HTTPS / SSL is a first world solution, as many small and medium scale content or service providers that live on a freemium or ad-sponsored models can't afford the additional cost and latency unless they are forced to. Some think that encryption levels will hover around 50-60% of the total until mass adoption of HTTP/2 which could take 5+ years. We have seen, with T-Mobile's binge on  a first service launch that actively manages traffic, even encrypted to an agreed upon quality level. The net neutrality activists cried fool at the launch of the service, but quickly retreated when they saw the popularity and the first tangible signs of collaboration between content providers, aggregators and operators for customers' benefit.

As mentioned in the past, the problem is not technical, moral or academic. Encryption and net neutrality are just symptoms of an evolving value chain where the players are attempting to position themselves for dominance. The solution with be commercial and will involve collaboration in the form of content metadata exchange, to monitor, control and manage traffic. Mobile Edge Computing can be a good enabler in this. Mobile advertising, which is still missing over 20b$ in investment in the US alone when compared to other media and time spent / eyeball engagement will likely be part of the equation as well.

...but what happens in the meantime, until the value chain realigns? We have seen consumer postpaid ARPU declining in most mature markets for the last few years, while we seen engagement and usage of so-called OTT services explode. Many operators continue to keep their head in the sand and thinking of "business as usual" while timidly investigating new potential "revenue streams".

I think that the time has come for many to wake up and take hard decisions. In many cases, operators are not equipped organizationally or culturally for the transition that is necessary to flourish in a fluid environment where consumer flock to services that are free, freemium, or ad sponsored. What operators know best, subscription services see their price under intense pressure because OTTs are looking at usage and penetration at global levels, rather than per country. For these operators who understand the situation and are changing their ways, the road is still long and with many obstacles, particularly on the regulatory front, where they are not playing by the same rules as their OTT competition.

I suggest here that for many operators, it is time to get out. You had a good run, made lots of money on consumer services through 2G, 3G and early 4G, the next dollars or euros are going to be tremendously more expensive to get than the earlier.
At this point, I think there are emerging and underdeveloped verticals (such as enterprise and IoT) that are easier to penetrate (less regulatory barriers, more need for managed network capabilities and at least in the case of enterprise, more investment possibilities).
I think that at this stage, any operator who derives most of its revenue from consumer services should assume that these will likely dwindle to nothing unless drastic operational, organizational and cultural changes occur.
Some operator see the writing on the wall and have started the effort. There is no guarantee that it will work, but certainly having a software defined, virtualized elastic network will help if they are betting the farm on service agility. Others are looking at new technologies, open source and standards as they have done in the past. Aligning little boxes from industry vendors in neat powerpoint roadmap presentations, hiring a head of network transformation or virtualization... for them, the reality, I am afraid will come hard and fast. You don't invest in technologies to build services. You build services first and then look at whether you need more or new technologies to enable them.

MEC: The 7B$ opportunity

Extracted from Mobile Edge Computing 2016.

Executive summary

Table of contents

Defining an addressable market for an emerging product or technology is always an interesting challenge. On one hand, you have to evaluate the problems the technology solves and their value to the market, and on the other hand, appreciate the possible cost structure and psychological price expectations from the potential buyer / users.

This warrants a top down and bottoms up approach to look at how the technology can contribute or substitute some current radio and core networks spending, together with a cost based review of the potential physical and virtual infrastructure. [...]

The cost analysis is comparatively easy, as it relies on well understood current cost structure for physical hardware and virtual functions. The assumptions surrounding the costs of the hardware has been reviewed with main x86 based hardware vendors. The VNFs pricing relies on discussions with large and emerging telecom equipment vendors for standard VNFs such as EPC, IMS, encoding, load balancers, DPI… price structure. Traditional telco professional services, maintenance and support costs are apportioned and included in the calculations.

The overall assumption is that MEC will become part of the fabric of 5G networks and that MEC equipment will cover up to 20% of a network (coverage or population) when fully deployed.
The report features total addressable market, cumulative and incremental for MEC equipment vendors and integrator, broken down by CAPEX / OPEX, consumer, enterprises and IoT services.
It then provides a review of operators opportunities and revenue model for each segment.