An IP over DWDM renaissance at 400G

A convergence of technology innovations is giving rise to a new concept of "routed optical networking" in which Layer 3 becomes the switching layer across a DWDM transport layer. The key advances are the commercialization of 400Gbps pluggable coherent optics, the emergence of multi-terabit network processing units (NPUs) for routers, and continuing advances in telemetry and software automation for IP networks. While sharing some of the same heritage, the new routed optical network is not your father's IP over DWDM.

Defined as the physical integration of DWDM optics into a router, the concept of IP over DWDM (or IPoDWDM) has been around for decades — with continuing interest from operators that never translated into large-scale deployments. Explaining the appeal is simple: placing DWDM optics directly into the router eliminates the transponder shelf and optics between routers and DWDM systems — a capex win.

Back to the future

Several challenges have stood in the way, the biggest of which has been the "faceplate tradeoff." At any given data rate, DWDM optics modules have always been larger than client optics, so outfitting routers with DWDM optics meant sacrificing roughly half the capacity of the router. This was a deal-breaker for most telecom operators. But with the advent of the Optical Internetworking Forum (OIF) 400ZR pluggable optics, for the first time in history, the faceplate tradeoff is eliminated. 400ZR DWDM optics have been designed to fit the dominant 400GE form factors (using QSFP-DD or OSFP modules) with no capacity/ footprint penalty.

Vendors have also realized they can use excess power margins to boost transmission into regional distances using those same QSFP-DD and OSFP modules but with different forward error correction algorithms (boosting distances up to ~1,500km compared to the <120km defined in OIF 400ZR). The extended reach pluggable modules are typically called 400ZR+. The tradeoff here is that the extended distances are not OIF specified and thus (at least today) are not fully standardized. (The OpenZR+ MSA's work to standardize coherent 400G extended distances is a step in the right direction, but it is still early.)

Key innovation for routed optical network architecture

In terms of routed optical networks, the resurgence of integrated DWDM optics on routers, enabled by 400ZR and OpenZR+, is a crucial starting point. The next key innovation is the capacity of the routers themselves, which enable router scaling from 10s to 100s of terabits of capacity using the newest-generation NPUs. This is far more capacity than is actually required today. The development is significant because constrained router capacity was at the heart of the router bypass trend that drove telecom transport architectures over the past decade. Operators sought to keep traffic in the optical layer (i.e., optical transport networking [OTN] or wavelengths) as much as possible and only bumped up to the routed layer when IP packet processing was required.

The final pillar in the new routed optical network architecture is network automation, which is accelerating rapidly for IP networks and will surely continue over the next decade. Much of this innovation is being defined within the Internet Engineering Task Force (IETF) and includes Path Computation Element Protocol (PCEP), segment routing and Bit Indexed Explicit Replication (BIER), among many others. Automation will drive network agility while also hitting the opex component of the total cost of ownership for transport networks.

In a future of pluggable OpenZR+ optics, abundant router capacity and IP network automation, "router bypass" need not be the default transport network approach. And elevating more of the switching role to the IP layer has implications for the optical layers — particularly for OTN switches and ROADM nodes. In essence, "optical switching bypass" could save both capex and opex in segments in which the Layer 1 and wavelength switching functions become redundant. Segment routing, in particular, is critical for optical switching bypass. Segment routing can be used to deliver the sub-50ms protection that has historically been provided by the optical layers. It can also be used to deliver circuit-style emulation over an IP layer, thus removing the need for OTN switching for operators that have high-capacity private line services.

Widespread appeal, but …

Significantly, although 400ZR work within the OIF began with hyperscalers, there is already evidence that the architecture also appeals to communications service providers. Sweden's Telia Carrier, Ethiopian internet service provider WebSprix and Windstream in the US have all announced plans to commercial deploy routed optical networks using 400G pluggable coherent optics beginning this year.

To be clear, Heavy Reading's point is not that the optical industry will be turned on its head. Rather, the combination of technology innovations opens up new transport architecture opportunities that did not exist prior to 400G. There are limitations in performance, and there is no question that highest performance optics will continue to drive long haul and subsea networks. Additionally, some operators will always prefer chassis-based DWDM systems and switched OTN. But with the combination of 400G pluggable coherent optics, next-gen NPUs and mature software-defined networking (SDN) automation, a viable alternative optical architecture has emerged. This trend, which has already begun, will accelerate throughout 2021 and beyond.

This blog is sponsored by Cisco Systems.