TechPrufer
Get started
Cloudflare
Cloudflare logoCloudflare · Feature trail

Kernel Bypass for High Packet Rate Networking

1 post · Sep 2015 – Oct 2015

What this trail covers

Cloudflare has developed and contributed a 'single RX queue mode' to the Netmap project. This allows for kernel bypass on selected RX queues of network interfaces, enabling user-space applications to handle high packet rates for specific traffic flows while leaving other queues to the standard Linux kernel network stack. This is crucial for mitigating large packet floods and improving performance on NICs that do not natively support advanced kernel bypass features.

All Cloudflare trails
  1. Single RX queue kernel bypass in Netmap for high packet rate networkingThis post introduces a modification to the Netmap project to enable a 'single RX queue mode'. This feature allows Netmap applications to selectively take over a single receive queue on a network interface, bypassing the Linux kernel for that queue. This is achieved by modifying Netmap's driver-specific code and exposing a minimal API for opening interfaces in this mode. The post also details the necessary steps for configuring flow steering and disabling hardware offloads to ensure proper isolation of the selected queue. Example code and setup instructions for IXGBE NICs are provided, demonstrating the ability to handle millions of packets per second on a single core.
  2. Kernel bypassThis post introduces the concept of kernel bypass for high packet rate networking, explaining why the standard Linux kernel networking stack is insufficient for workloads like handling large packet floods. It details experiments demonstrating the performance drop of the kernel stack as more CPU cores are utilized. The post then surveys various kernel bypass techniques including PACKET_MMAP, PF_RING, Snabbswitch, DPDK, and Netmap, highlighting their limitations for Cloudflare's use case (dedicating a whole NIC). It then introduces Solarflare's EF_VI as a proprietary solution and discusses the concept of a 'bifurcated driver' and a 'virtualization approach' using Intel's SR-IOV (VF) interfaces as alternatives that allow for partial NIC takeover.