Don’t Call it a Comeback: The Role of SD-WAN in the Generative AI EraDon’t Call it a Comeback: The Role of SD-WAN in the Generative AI Era

Today's AI applications often require massive uploads of context and data, followed by streaming responses that can vary dramatically in size and urgency. These new patterns demand networks that can adapt in real-time, making Software-Defined Wide Area Network (SD-WAN) technology more relevant than ever. By intelligently managing traffic flows, understanding application behavior, and automatically adjusting to changing conditions, SD-WAN is emerging as a crucial technology for enterprises looking to optimize their networks for the AI era.

The rise of distributed GenAI applications

Recently, a new category of applications has emerged: distributed generative AI (GenAI) applications. The trend is moving towards a highly federated learning model for GenAI where large language models (LLMs) with trillions of parameters and inference engines are spread across different locations, working together in a more interconnected way.

LLMs are primarily hosted in data centers, but they're increasingly being complemented by small language models (SLMs) that operate at edge locations. These SLMs are specialized, much smaller in size (typically a few billion parameters), and can run on compact hardware. They often lack the full context or complete capabilities to fully answer a user's query on their own.

In a typical workflow, a user submits a prompt, the SLM examines it, and then delegates the task to an LLM. The SLM essentially acts as a preliminary filter, handing the prompt to the more comprehensive LLM to determine the most effective way to provide an answer.

This is like a highly interconnected network, with a central data center and multiple “edge" locations that communicate extensively with each other. These components constantly exchange information, aiming to generate the best possible results for various applications.

How AI apps change network traffic patterns

To see how this affects networks, look at the example of web applications, typically designed with a strong bias towards downloading. In these applications, the initial request/upload is usually small, while the response/download is much larger. Network infrastructures have traditionally been built to support this pattern, with upload capacities being significantly lower than download capacities.

GenAI applications are fundamentally changing this model. For example, to summarize a conversation or a website, you need to upload the entire content first. This means the data transmission becomes more bursty and requires much higher upload capacities. The nature of connectivity is also transforming. Networking is no longer just about people or organizations who need to communicate, but also about how different application components interact and exchange information.

Introducing VeloRAIN

To help organizations seize the AI networking opportunity, Broadcom has introduced VeloRAIN (VeloCloud Robust Artificial Intelligence Networking). VeloRAIN is an AI networking architecture that enhances the performance, security, and scalability of distributed AI workloads beyond traditional data centers. It builds on the leading VeloCloud SD-WAN platform to optimize AI workloads across distributed enterprise networks.

VeloRAIN uses several strategies to focus on creating more intelligent network management.

AI-driven application profiling: The first and perhaps most crucial pillar is context — not just knowing what an application is, but truly understanding how it behaves. Consider the difference between traditional web applications and AI-driven platforms. A GenAI platform like ChatGPT represents an entirely different network challenge compared to a standard chat application. It's not enough to simply identify the application type. Network engineers must dive deep, developing application signatures that capture the intricate behavioral patterns of these sophisticated systems.

Traditional SD-WAN approaches simply identify an application by its type. VeloRAIN goes much deeper, analyzing the application's specific behavioral patterns and internal workflows to understand not just what the application is, but how it functions. For example, in an AI application, there's a significant difference between a model download and a real-time query. The goal is to recognize unique application signatures and understand nuanced internal flows so network traffic can be prioritized and optimized accordingly.

Most AI traffic is encrypted, making traditional inspection methods ineffective. To overcome this, VeloRAIN uses AI and machine learning technologies to help analyze and prioritize encrypted traffic, ultimately delivering better network outcomes for AI applications.

AI-based network optimization: VeloCloud Dynamic Multipath Optimization™ (DMPO) is designed to optimize network connectivity across different underlying network technologies for the best possible user experience. Existing VeloCloud SD-WAN capabilities are being augmented with AI to better serve AI applications, taking into account the significant shifts in network usage patterns. This means reimagining optimization strategies to accommodate new communication dynamics and developing flexible network routing that can handle the more complex, bi-directional data flows characteristic of modern AI platforms.

An exciting development in networking is the increase in market share for underlying technologies like 5G and satellite connections. While these technologies benefit all applications, they're particularly important for AI applications due to their specific performance requirements. 5G networks, for example, can create dedicated "network slices" — essentially reserved bandwidth for specific purposes. The challenge is using these slices effectively. When an AI application makes a query with specific performance requirements, those requirements must be matched with the appropriate network resources. If a dedicated slice isn't available, VeloCloud SD-WAN can create a priority channel, using its understanding of the application's needs to ensure necessary network performance.

Here's how it works: SD-WAN policies define what an application needs in terms of performance. These requirements are then mapped to a specific network slice that can guarantee certain levels of performance, including specific delays, latency, and data loss rates. The system then routes the traffic to the nearest edge location that has the appropriate AI model to handle the request. With this approach, application developers don't need to worry about network details or deployment locations. They can simply build their applications to meet certain performance standards (SLAs), and the network operators can use SD-WAN technology to deliver on these requirements.

AIOps with real-time data: Managing AI networking at scale is challenging. Concepts like intent-based networking or AIOps help automatically map application needs to network resources, adapting to changing conditions and network behavior over time. This automated, intelligent management will be crucial for building the next generation of SD-WAN infrastructure that can properly support AI applications. The goal is to create a network that can automatically understand and adapt to application needs, ensuring consistent performance without requiring constant manual intervention.

The role of SD-WAN in AI networking

In recent years, some have suggested that SD-WAN technology is becoming less relevant as the industry evolves. However, having an SD-WAN layer in AI networking is becoming more valuable because SD-WAN can:

This flexibility is crucial. Whether an application is running over wireless, wired, or satellite connections becomes less important. Application developers can focus on building their software to meet specific performance standards, and SD-WAN handles the complexities of delivering that performance across any type of network connection.

This creates a powerful framework where applications can maintain consistent performance regardless of the underlying network infrastructure. Rather than becoming obsolete, SD-WAN is evolving into an essential bridge between sophisticated AI applications and the diverse network technologies that support them.