BGP Traffic Engineering · Cloudflare Feature Trail

Aug 2026BGP Role model: tracking the adoption of RFC 9234This post details the analysis of RFC 9234 (BGP Roles and OTC attribute) adoption across the internet. It explains how BGP Roles and the OTC attribute work to prevent route leaks and describes Cloudflare's methodology for tracking adoption by monitoring OTC attribute propagation from peer ASes. The analysis reveals that while adoption is growing, two major Tier-1 networks are stripping the OTC attribute, which is a significant impediment to widespread route leak prevention. The post also provides guidance on how to enable BGP Roles in one's own network.

Jul 2026BGP ORIGIN attribute manipulation and its impact on the InternetThis post details Cloudflare's investigation into the widespread manipulation of the BGP ORIGIN attribute. It describes experiments using Anycast announcements with different ORIGIN values (IGP, EGP, INCOMPLETE) and analysis of BGP Update messages from RIPE RIS and RouteViews collectors. The findings indicate that approximately 70% of observed paths have a manipulated ORIGIN attribute, with nearly 10% of direct peers and six Tier-1 ASes rewriting the ORIGIN to IGP. The post also highlights discrepancies in manipulation behavior between IPv4 and IPv6 and discusses the operational history and impact of this attribute manipulation on internet routing and revenue.

Jun 2026Enforcing the First AS in BGP AS PATHsThis post details the problem of forged AS_PATHs in BGP, where attackers create fake routes to intercept traffic. It explains how this bypasses RPKI-ROV and ASPA in certain scenarios. The core contribution is the emphasis on enforcing the 'First AS' rule in BGP AS_PATHs, as outlined in RFC 4271, as a critical mitigation. The post provides concrete examples of forged paths, including one that incorrectly included Cloudflare's ASN. Cloudflare's own experimental measurements, where they intentionally violated the First AS rule, are presented to demonstrate the widespread acceptance of such violations by major networks. This highlights a significant gap in current BGP security practices and advocates for universal enforcement of the First AS rule.

Feb 2026ASPA: making Internet routing more secureThis post introduces ASPA (Autonomous System Provider Authorization) as a new cryptographic standard for validating the entire path of network traffic, building upon RPKI and ROAs. It explains how ASPA works by verifying the 'chain of relationships' from both ends of the AS path (up-ramp and down-ramp) to detect route leaks and forged-origin hijacks. The post details the validation process with examples and demonstrates the ease of creating ASPA objects through RIPE and ARIN dashboards. It also notes limitations of ASPA against certain types of provider-forged path advertisements. Cloudflare Radar's new ASPA deployment monitoring feature is also highlighted.

Feb 2026Cloudflare outage on February 20, 2026This post details a critical incident where a bug in an automated BYOIP prefix cleanup sub-task led to the unintentional withdrawal of approximately 1,100 BYOIP prefixes, causing widespread service outages. The root cause was identified as an API query bug where `pending_delete` with an empty value was interpreted as a request for all BYOIP prefixes, leading to their deletion. The incident also revealed insufficient staging environment data and incomplete test coverage for scenarios involving automated task-runner services modifying user data. Recovery was complicated by different states of impact across customers, including withdrawn prefixes, withdrawn prefixes with removed service bindings, and prefixes with all service bindings removed. The post outlines the immediate steps taken, including reverting the change, enabling customer self-remediation, and manual restoration, and highlights the need for improved operational state management systems, such as health-mediated deployments and the removal of manual actions in the Addressing API, to prevent future occurrences.

Jan 2026Route leak incident on January 22, 2026This post details a specific incident of a BGP route leak caused by an accidental misconfiguration in Cloudflare's network automation code. The misconfiguration involved a policy change that inadvertently allowed internal routes to be advertised externally to peers and providers in Miami. The post provides a timeline of the incident, the specific configuration diff that caused the issue, and the technical explanation of how the 'route-type internal' match in JunOS/Junos EVO became too permissive. It also includes monocle output showing anomalous AS paths indicative of the route leak and a graph illustrating backbone congestion. The resolution involved manual reversion of the configuration, pausing automation, reverting the code change, and re-enabling automation after confirming the fix.

Jan 2026A closer look at a BGP anomaly in VenezuelaThis post analyzes a specific BGP route leak event involving AS8048 (CANTV) in Venezuela. It details the observed route leak, explains BGP route leaks and valley-free routing principles, and investigates the relationship between AS8048 and its customer AS21980. The analysis suggests that the leak was likely caused by insufficient routing export/import policies by AS8048, evidenced by route prepending and repeated occurrences of similar leaks. It contrasts this with potential malicious intent and highlights the limitations of current validation methods, referencing RFC9234 as a potential solution.

Nov 2025DIY BYOIP- a new way to Bring Your Own IP prefixes to CloudflareIntroduced a self-serve BYOIP API that automates prefix onboarding using RPKI ROAs and IRR/rDNS record modification, significantly reducing manual effort and onboarding time. This new process replaces the manual Letter of Authorization (LOA) system and leverages cryptographically-strong authorization mechanisms for routing security.

Oct 2025BGP zombies and excessive path huntingThis post details the formation and impact of BGP zombies, which are routes stuck in the Default-Free Zone (DFZ) due to slow convergence or buggy software. It explains how 'path hunting' (exhaustive search for the best path) and the Minimum Route Advertisement Interval (MRAI) contribute to these delays. The post provides examples of BGP zombies occurring in upstream ISP loops and within Cloudflare's internal network, noting that they can persist for several minutes. It also observes that IPv4 zombies have a longer impact than IPv6 zombies, potentially due to the larger IPv4 routing table size. The post demonstrates how to intentionally spawn BGP zombies using on-demand BYOIP announcements and withdrawals.

Oct 2025A framework for measuring Internet resilienceThis post introduces a data-driven framework for measuring Internet resilience, moving beyond abstract definitions to quantifiable metrics. It defines Internet resilience as the measurable capability of a national or regional network ecosystem to maintain diverse and secure routing paths in the face of challenges, and to rapidly restore connectivity following a disruption. The framework emphasizes the importance of public data sources and introduces metrics related to IXPs, colocation facilities, peering participation rates, and the diversity of IXP membership. It also highlights the role of route hygiene, RPKI, ROV, and diverse interconnections in building a resilient Internet.

Oct 2025A deep dive into BPF LPM trie performance and optimizationThis post details the performance bottlenecks and limitations of the BPF LPM trie implementation in the Linux kernel, specifically its two-child node design and lack of level compression, which lead to slow lookup, update, and delete operations, especially with millions of entries. It also discusses the impact of these limitations on production systems, causing soft lockups and traffic packet loss. The post proposes optimizations and explores the trade-offs involved in improving trie performance.

Sep 2025Monitoring AS-SETs and why they matterThis post introduces Cloudflare Radar's new feature for monitoring Internet Routing Registry (IRR) AS-SET information. It details how AS-SETs are used for BGP route filtering and preventing route leaks, providing examples of their application with tools like bgpq4. The post also highlights the importance of accurate AS-SET data and the potential risks of misuse, showcasing how Cloudflare Radar enhances visibility into AS-SET memberships with additional metadata such as inferred ASN, IRR sources, member counts, and AS Cone size. It also introduces a tree-view visualization for AS-SET hierarchies. The post concludes by looking ahead to the adoption of RFC9234 and ASPA for more robust BGP security.

May 2025Bringing connections into view: real-time BGP route visibility on Cloudflare RadarIntroduced a real-time BGP route lookup service on Cloudflare Radar and via the Radar API. This service provides near instantaneous global BGP routing data, visualized using Sankey diagrams and accessible programmatically. It includes prefix origin tables with visibility percentages and RPKI validation outcomes, aiding in the debugging of BGP misconfigurations. The service leverages RIPE RIS and RouteViews' real-time stream capabilities to reduce data delay from minutes to seconds.

Nov 2024Resilient Internet connectivity in Europe mitigates impact from multiple cable cutsThis post analyzes the impact of two submarine cable cuts in the Baltic Sea on internet connectivity in Lithuania, Sweden, Finland, and Germany. It uses Cloudflare Radar data (NetFlows, bandwidth, latency, BGP announcements) to demonstrate that despite significant cable damage, there were no observable disruptions to traffic volumes or announced IP address space in the affected countries. The analysis highlights the role of redundant submarine and terrestrial fiber connections, as well as effective BGP routing, in maintaining resilient internet connectivity. It also notes minor, likely unrelated, fluctuations in internet quality metrics for Finland and Germany.

Oct 2024Cloudflare’s perspective of the October 30, 2024, OVHcloud outageThis post details how Cloudflare's BGP monitoring and configuration, specifically the implementation of maximum prefix-limits on peering sessions, prevented a significant route leak originating from Worldstream (AS49981) from impacting Cloudflare's network and subsequently OVHcloud (AS16276). It analyzes the traffic drop observed during the OVHcloud outage, the shift in traffic routing patterns, and the evidence of Worldstream exceeding its prefix limits, leading to the shutdown of the BGP session. The post concludes by recommending MANRS guidelines and specific preventative measures like max prefix-limits, ASPA, and RFC9234 for improved Internet resilience.

Oct 2024Impact of Verizon's September 30 outage on Internet trafficThis post analyzes the impact of a Verizon mobile network outage on internet traffic, using Cloudflare Radar data to observe HTTP request volume changes. It details traffic declines and recovery patterns across Verizon's Autonomous System (AS6167) and specific impacted cities, noting that voice services may have been more affected than data. The analysis provides quantitative data on traffic drops (up to 9% overall, 30% in Omaha) and recovery timelines, correlating them with Verizon's public statements.

Sep 2024Cloudflare incident on September 17, 2024This post details an incident where an internal software error led to the inadvertent withdrawal of fifteen IPv4 prefixes from Cloudflare's network, affecting some Business plan websites for approximately one hour. The root cause was a bug in a legacy system's synchronization function that failed to renumber all websites, leading to the incorrect assumption that associated IP addresses were unused. The incident was resolved by rolling back the change and re-announcing the prefixes. Preventative measures include adding tests to verify outstanding assignments before prefix withdrawal, improving processes for prefix withdrawal with automated and manual traffic verification, and re-implementing legacy system functions with better documentation and test coverage.

Sep 2024Making progress on routing security: the new White House roadmapThis post highlights the White House's new roadmap for enhancing internet routing security, with a strong emphasis on the adoption and enforcement of Resource Public Key Infrastructure (RPKI) and Route Origin Authorization (ROA) records. It explains the fundamental concepts of BGP and the security vulnerabilities it presents, such as route hijacking and leaks. The post details real-world examples of BGP incidents (KLAYswap, RTComm, 1.1.1.1 leaks) and how RPKI/ROAs could have mitigated them. It also references past work by NIST and Cloudflare's own advocacy for RPKI adoption, positioning the White House roadmap as a culmination of these efforts.

Jul 2024Cloudflare 1.1.1.1 incident on June 27, 2024This post details a specific incident where a BGP hijack of 1.1.1.1/32 by AS267613 and a route leak of 1.1.1.0/24 by AS262504, propagated by AS1031, caused temporary unreachability and high latency for Cloudflare's 1.1.1.1 DNS resolver. It explains the mechanisms of BGP hijacks (Longest Prefix Matching) and route leaks (RFC7908 Type 1), and the role of BGP local-preference. The incident timeline and impact on users in Germany and the US are provided. Cloudflare's response involved engaging with peering partners and disabling peering locations. The post emphasizes the continued importance of RPKI, ASPA, and improved detection methods for BGP security.

Jun 2024Exam-ining recent Internet shutdowns in Syria, Iraq, and AlgeriaThis post details the analysis of recent Internet shutdowns in Syria, Iraq, and Algeria, focusing on their technical implementation via BGP route announcements and their impact on Internet traffic. It examines how Syria's shutdowns involve not announcing IP address space, leading to asymmetrical routing and traffic spikes to DNS resolvers. It also analyzes Iraq's shutdowns and Algeria's content blocking strategy. The post uses Cloudflare Radar data to illustrate traffic drops, prefix announcements, and protocol-specific traffic patterns during these events.

Mar 2024Undersea cable failures cause Internet disruptions for multiple African countriesThis post details Cloudflare's observation and analysis of widespread internet disruptions in Africa caused by multiple undersea cable failures. It leverages Cloudflare Radar data to visualize the temporal and geographical spread of these outages, identifying specific countries and networks impacted, including Vodacom in South Africa. The post also references reports from network providers and Microsoft Azure, highlighting the specific submarine cables affected (WACS, MainOne, SAT-3/WASC, ACE) and the cumulative impact of these failures combined with Red Sea cable issues. It further discusses the operational impact on a specific submarine cable operator (MainOne) and its DNS/website reachability. The post serves as a data-driven report on a significant network event, illustrating Cloudflare's role in monitoring and reporting on global internet health.

Mar 2024Simplifying how enterprises connect to Cloudflare with Express Cloudflare Network InterconnectIntroduces 'Express CNI' which simplifies the ordering and deployment of Cloudflare Network Interconnects (CNIs) to under 3 minutes. This new process eliminates the need for GRE tunnels for Magic Transit and Magic WAN, allowing direct connection with standard 1500-byte packets. It also details the cost savings compared to other cloud providers' direct connect services and outlines future plans for expanding Express CNI features to partners and more global locations, including BGP support.

Sep 2023How Cloudflare’s systems dynamically route traffic across the globeThis post introduces Traffic Manager, an automated system for dynamically routing traffic across Cloudflare's global network to handle data center issues. It details the 'world before Traffic Manager' where manual intervention by network engineers was required, leading to sub-optimal customer experiences and scalability issues. The post explains the development of Traffic Manager to automate detection and route withdrawal. It also introduces Traffic Predictor, a system that simulates data center failures to predict traffic failover and inform Traffic Manager policies. The post discusses the limitations of prefix advertisement-based traffic management and the need for more granular control to avoid 'stranded capacity'.

Jul 2023Cloudflare Radar's new BGP origin hijack detection systemThis post details the design and implementation of Cloudflare Radar's new BGP origin hijack detection system. It explains the problem of BGP hijacks, the limitations of current prevention mechanisms like RPKI due to low adoption rates, and the architecture of Cloudflare's detection system. The system includes a prefix origin change detection module using a custom Rust-based IP prefix trie to process BGP update messages, a hijack detection module, and a storage and notification module. It highlights the use of live-stream and historical BGP data from sources like RouteViews and RIPE RIS, and the process of generating 'Origin Change Signals' when prefix origin ASes change.

Jul 2023Routing information now on Cloudflare RadarIntroduced a new Routing page on Cloudflare Radar to monitor BGP messages and present insights on IP prefixes, networks, countries, and the Internet overall. This page provides statistics on ASes, prefixes, routes, and RPKI validation results, along with BGP announcement volume charts and network connectivity information. It also consolidates detection results for BGP route leaks, BGP hijacks, and RPKI Invalid Multiple Origin AS (MOAS) events. The Routing data is made available through a developer API.

Jun 2023Network performance update: Speed Week 2023This post details Cloudflare's ongoing efforts to improve network performance by enhancing its traffic engineering capabilities. It quantifies improvements in network rankings, showing Cloudflare is now the fastest provider for 46% of networks. It also describes the tooling and processes used for this monitoring, including Real User Measurements (RUM) and a Performance Benchmarks Dashboard. The actions taken to improve performance are categorized into establishing peering, expanding compute capacity, and applying traffic engineering actions.

Jun 2023How Orpheus automatically routes around bad Internet weatherThis post introduces Orpheus, a system that automatically routes around bad Internet weather by identifying connectivity outages in real time and leveraging Cloudflare's network to find alternative paths. It details how Orpheus saves requests that would otherwise fail due to BGP limitations or hijacking by using synthetic probes and real-time traffic analysis to test route validity. Orpheus reuses components of Argo Smart Routing's control plane, focusing on failover paths and a robust Internet monitoring system. The post highlights Orpheus's impact, saving 132 billion requests and preventing over 50 incidents, and explains how it reduces 522 errors by ensuring origin reachability even when direct paths are down.

May 2023The European Network Usage Fees proposal is about much more than a fight between Big Tech and Big European telcosThis post discusses the implications of the European Commission's proposed network usage fees on Internet interconnection. It explains the current model of peering and transit, highlights the benefits of settlement-free peering, and argues that regulatory intervention would lead to higher prices for consumers and smaller businesses by creating fast lanes for Big Tech and slow lanes for everything else. The post details Cloudflare's extensive peering network and its stance against the proposed fees, emphasizing the importance of free and frequent network connections for a faster and more reliable Internet.

Apr 2023Cloudflare’s view of the Virgin Media outage in the UKThis post analyzes a Virgin Media outage by correlating traffic data from Cloudflare Radar with BGP announcement and withdrawal activity. It details the observed traffic patterns, including multiple outages and partial recoveries, and maps these to specific BGP events, suggesting a BGP-related or core network infrastructure issue as the root cause. The post also highlights the unavailability of Virgin Media's status page due to its reliance on the affected network and the inability to perform DNS lookups.

Dec 2022Helping build a safer Internet by measuring BGP RPKI Route Origin ValidationThis post introduces a new method for measuring Route Origin Validation (ROV) deployment by observing the reachability of RPKI-invalid origins from measurement points within an AS. It details the implementation of this technique on isbgpsafeyet.com, which uses browser-based probes to continuously collect data on ROV deployment. The post also presents the initial findings on the global state of ROV deployment, highlighting significant variations by country and the long way to go before the internet is fully protected from routing problems.

Dec 2022Cloudflare expands Project Pangea to connect and protect (even) more community networksProject Pangea has been expanded to remove the requirement for participants to bring their own /24 IP address block. This change allows community networks to access Cloudflare's Magic Transit and Magic Firewall services for Internet connectivity, DDoS protection, network firewalling, and traffic acceleration without needing to provide their own IP space. This is enabled by Cloudflare's ability to advertise IP addresses on behalf of participants, and by extending Magic Transit to support customers without their own IP address space.

Dec 2022The Montgomery, Alabama Internet Exchange is making the Internet faster. We’re happy to be there.This post details Cloudflare's participation in the Montgomery Internet Exchange (MGMix) and its impact on regional internet performance. It explains the role of Internet Exchanges (IXs) in connecting networks, contrasting them with transit and Private Network Interconnections (PNIs). The post highlights the historical underperformance of the internet in Alabama and Mississippi and how MGMix, with Cloudflare's participation, aims to improve this by facilitating local traffic exchange. It also touches upon the challenges of IX growth and the importance of transit providers for networks like Cloudflare. The post includes traffic data from MGMix and mentions the expansion of MGMix to Auburn, Alabama.

Nov 2022Cloudflare servers don't own IPs anymore – so how do they connect to the Internet?This post details the evolution of Cloudflare's egress network architecture, moving from a simple unicast IP per server model to a complex system requiring multiple IPs per server to support traffic segmentation (trusted vs. untrusted) and country-specific geofencing for services like WARP. It highlights the challenges of IP address cardinality and cost, especially with IPv4, and contrasts two extreme solutions (each server owns all IPs vs. one server owns all IPs for a tag). The post sets the stage for a new, more efficient solution that leverages IPv6's address space and a refined IPv4 strategy, aiming to allow egress from any data center with any required IP tag while maintaining performance and cost-effectiveness.

Nov 2022Why BGP communities are better than AS-path prependsThis post details the limitations of AS-path prepending for inbound traffic engineering, highlighting its reliance on equal path attributes and its ineffectiveness against sophisticated routing policies. It introduces BGP communities as a superior alternative, offering a more flexible and granular mechanism for influencing routing decisions. The post explains how communities can be used to signal business policies and preferences to neighboring ASes, enabling more effective traffic shaping and management.

Nov 2022How we detect route leaks and our new Cloudflare Radar route leak serviceThis post introduces Cloudflare Radar's route leak data and API, enabling public access to information about route leaks across the Internet. It details the system developed to detect route leak events automatically by ingesting data from public sources and Cloudflare's global network view. The post also provides a primer on BGP and route leaks, defining route leaks according to RFC7908 and illustrating the six types of route leaks (Type 1, 2, 3, and 4 are detailed with examples). The system's design is informed by these RFC7908 definitions and the categorization of AS relationships (customer, provider, peer).

Oct 2022Making peering easy with the new Cloudflare Peering PortalIntroduced PeeringDB login for the Cloudflare Peering Portal, allowing users to leverage existing credentials. Enabled direct peering session requests within the portal for Internet Exchanges, streamlining the process of establishing BGP sessions with Cloudflare's network team.

Jul 2022Cloudflare’s view of the Rogers Communications outage in CanadaThis post details Cloudflare's real-time analysis of the Rogers Communications outage, focusing on the impact of BGP update spikes and prefix withdrawals. It uses Cloudflare Radar data to illustrate the loss and partial recovery of traffic, correlating it with BGP announcements and withdrawals. The analysis provides a technical breakdown of how BGP functions and its role in internet routing, offering insights into the likely cause of the outage (internal error) and the ongoing recovery efforts.

Jun 2022Cloudflare outage on June 21, 2022This post details a specific outage caused by an incorrect BGP policy change that led to the withdrawal of critical site-local prefixes. It explains the technical error in re-ordering BGP policy terms, the impact on internal routing and load balancing (Multimog), and the incident timeline. Remediation steps include improving change management processes for MCP data centers, redesigning policy statements to prevent incorrect ordering, and enhancing automation for staggered rollouts and automated rollback.

Jun 2022AAE-1 & SMW5 cable cuts impact millions of users across multiple countriesThis post details the impact of simultaneous submarine cable cuts (AAE-1 and SMW-5) on global internet connectivity, using Cloudflare Radar data to illustrate traffic drops and recovery. It explains how these physical infrastructure failures, occurring in Egypt, affected millions of users and cloud providers across the Middle East, Africa, and Asia. The post also discusses the role of network redundancy and the location of cable landing points in determining the severity of impact, and highlights Cloudflare's distributed network architecture and Orpheus system as mechanisms for mitigating such disruptions.

May 2022Tracking shifts in Internet connectivity in Kherson, UkraineThis post details a specific incident in Kherson, Ukraine, where an internet outage was accompanied by a shift in BGP routing from Ukrainian providers to Russian networks (AS201776 - Miranda, AS12389 - Rostelecom). This shift impacted Cloudflare's data center selection, with traffic being routed through Moscow instead of Kyiv and Frankfurt. The post provides BGP path data and traffic graphs illustrating these changes. It also discusses the geopolitical implications of controlling internet resources during conflict and references the annexation of Crimea in 2014 as a precedent.

Apr 2022Internet disruptions overview for Q1 2022This post provides a Q1 2022 overview of internet disruptions observed by Cloudflare, categorizing them by cause: physical events (volcanic eruption, earthquake, submarine cable cuts, data center fires), infrastructure damage, and power outages. It details specific incidents in Tonga, Japan, The Gambia, Yemen, Tasmania, Iran, Cuba, Venezuela, Pakistan, Kazakhstan, Uzbekistan, Kyrgyzstan, and Taiwan, illustrating the impact with traffic graphs and linking to relevant Cloudflare Radar data and external reports. The analysis highlights the interconnectedness of physical infrastructure, power grids, and internet connectivity.

Mar 2022Cloudflare Radar’s new ASN pagesThis post introduces new Autonomous System (AS) pages on Cloudflare Radar. These pages provide detailed traffic statistics, protocol usage, security information (application and network-level DDoS attacks), geographical traffic distribution, and BGP announcement volume for individual ASes. This enhances transparency into the Internet's infrastructure and helps in understanding unusual events like outages and attacks.

Mar 2022Network performance update: Security WeekThis post details Cloudflare's efforts to improve network performance by addressing congestion and network hairpins. It presents new performance metrics showing Cloudflare is the fastest provider in 71% of the top 1,000 networks globally. It provides two case studies: one in Canada where insufficient peering led to congestion, and Cloudflare mitigated it by spreading traffic, upgrading IX ports, and setting up direct peering; and another in Australia where users experienced network hairpins, resolved by establishing peering in Sydney. The post also updates performance maps showing improvements across various regions.

Feb 2022BGP security and confirmation biasesThis post details a specific incident where a configuration error during a BGP hot-cut led to a route leak lasting 32 seconds. The root cause was the removal of BGP filters without adding explicit filters to only export Cloudflare and customer prefixes. The incident was exacerbated by confirmation bias, where the low prefix limit was initially accepted as the sole cause. In response, Cloudflare implemented an implicit reject policy for BGP sessions when no import/export policy is configured, and emphasized the importance of network automation and bias awareness in operational procedures. It also highlights the value of RFC8212 for default BGP route propagation behavior.

Feb 2022Internet is back in Tonga after 38 days of outageThis post details the restoration of internet connectivity to Tonga after a 38-day outage caused by a damaged undersea cable. It analyzes the traffic evolution before and after the repair, highlighting the critical role of submarine cables in global internet infrastructure and the challenges of single points of failure. The post also touches upon the repair process, the use of satellite services as a temporary measure, and the ongoing work to restore connectivity to outlying islands.

Jan 2022Internet shut down in Kazakhstan amid unrestThis post details the analysis of a nationwide internet shutdown in Kazakhstan, observed through Cloudflare Radar data. It highlights the impact on ASNs, particularly Kazakhtelecom and mobile providers, and correlates traffic drops with BGP updates indicating route unreachability. The post also notes the pattern of brief internet restorations coinciding with official government announcements, and discusses the broader context of state-imposed shutdowns for civil unrest, elections, or exams.

Jan 2022How The Gambia lost access to the Internet for more than 8 hoursThis post details the analysis of a major internet outage in The Gambia caused by a failure in the ACE submarine cable and a subsequent issue with terrestrial backup links via Senegal. It uses BGP data to illustrate the country's reliance on a single network path and highlights the critical role of submarine cables and terrestrial connectivity in maintaining global internet access. The analysis demonstrates how BGP updates can signal widespread network disruptions and emphasizes the interconnectedness of global networks.

Dec 2021Argo for Packets is Generally AvailableThis post announces the general availability of Argo for Packets, which extends Cloudflare's Argo traffic optimization to the IP layer. It details how Argo for Packets dynamically selects the best possible path through Cloudflare's network, going beyond traditional BGP routing which prioritizes hop count over latency. The system leverages health check probes and Layer 4 traffic analytics to understand one-way latencies and construct a map of interconnected data centers. This allows for personalized dynamic routes for customers, resulting in an average of 10% latency reduction. Examples are provided showing significant latency improvements for specific customer network topologies, particularly in reducing long-tail latencies. The integration with Magic WAN and Cloudflare for Offices is also highlighted.

Nov 2021Sudan was cut off from the Internet for 25 daysThis post details the analysis of a 25-day internet shutdown in Sudan, observing traffic patterns through Cloudflare Radar. It highlights the impact of the shutdown on ISPs like Mobitel, MTN, and Sudatel, and notes the resurgence of mobile traffic. The post also contextualizes the event within Sudan's history of internet disruptions and social media restrictions.

Oct 2021Understanding how Facebook disappeared from the InternetThis post details Cloudflare's external observation of the October 4, 2021, Facebook outage. It explains how Facebook's withdrawal of BGP routes led to their services becoming unreachable, impacting DNS resolution globally. The post uses BGP data and DNS query logs to illustrate the event's progression and its cascading effects on other internet services. It also references Facebook's internal post-mortem for a more complete picture.

Oct 2021Two Weeks Later: Finding and Eliminating Long Tail LatenciesThis post details Cloudflare's efforts to improve network performance in specific geographies by addressing long-tail latencies. It explains how the company identified underperforming networks, diagnosed issues such as route advertisement problems and tromboning in Brazil, lack of local peering in Spain, and suboptimal upstream selection in Africa. The post outlines the technical steps taken, including BGP configuration changes (LOCAL_PREF), establishing new peering sessions at IXPs, enabling anycast IP addresses, and adjusting routing policies. These actions resulted in significant latency reductions and improved rankings in the affected networks.

Sep 2021Project Myriagon: Cloudflare Passes 10,000 Connected NetworksThis post details the technical achievement of Cloudflare's Project Myriagon, which successfully connected to over 10,000 autonomous systems (ASNs). It explains the fundamental concepts of ASNs and BGP routing, highlighting the impact of direct peering connections on reducing internet latency (up to 90% reduction observed) and avoiding congestion. The post categorizes ASNs into Transit Only, Origin Only, and Mixed types, and discusses Cloudflare's strategic prioritization of connections to improve global reach and performance. It also introduces Project CatchG as the next initiative to further expand network connectivity.

Sep 2021Magic makes your network fasterThis post details how Cloudflare's Magic Transit, Magic WAN, and Magic Firewall products leverage Anycast architecture, extensive interconnections, and data-driven routing decisions to improve network performance and reduce latency. It explains how the 'hub is everywhere' model eliminates traffic tromboning common in traditional hub-and-spoke networks, leading to significant latency improvements (up to 70% in lab tests). The post also introduces the concept of 'getting smarter' by using network-wide insights to optimize routing beyond standard BGP, mentioning Argo Smart Routing for Packets as a further performance enhancement.

Sep 2021Cloudflare Backbone: A Fast Lane on the Busy Internet HighwayThis post details the construction and benefits of Cloudflare's dedicated global backbone, comprising long-distance fiber optic cables and metro fiber networks. It explains how this backbone provides guaranteed network capacity and consistent latency, acting as a 'fast lane' on the Internet. Performance benchmarks are presented, comparing traffic routed through the backbone versus the public Internet, demonstrating up to a 44.8% improvement in average response times from locations like Tokyo and São Paulo. The post also acknowledges instances where public Internet routes were faster due to geographical distance and highlights the role of other Cloudflare technologies like Argo Smart Routing and WARP+ in dynamically selecting the fastest routes.

Sep 2021Argo 2.0: Smart Routing Learns New TricksThis post announces significant upgrades to Argo Smart Routing. Key enhancements include: 1. Extending Argo to optimize traffic from clients to Cloudflare (last-mile optimization), reducing end-user RTT by up to 40% and overall TTFB by 39%. 2. Adding support for accelerating pure IP workloads for Magic Transit and Magic WAN customers, enabling performance benefits for IP networks. This is achieved by leveraging health check probes from Cloudflare data centers to customer origins to construct a map of interconnected data centers and their latencies, allowing for dynamic routing. Packet-level latency improvements of 10% on average are reported. Argo Smart Routing for Packets is available in closed beta for Magic Transit customers with a Cloudflare Network Interconnect.

Aug 2021Making Magic Transit health checks faster and more responsiveThis post details the evolution of the Magic Transit GRE tunnel health check system. The initial 'shared-nothing' approach suffered from inconsistent health reports and slow response times. Version 2 introduced multicast sharing of health check results within data centers, improving consistency but leading to scaling issues with message volume. Version 3, the focus of this post, implements consistent hashing to distribute health check responsibilities across servers in a data center. Each server now periodically sends heartbeats, constructs a sorted list of active peers, and is assigned responsibility for checking specific tunnels based on hashing. This significantly reduces CPU and memory usage at the edge, enabling faster detection of issues and supporting future growth.

Jul 2021Cloudflare customers can now use Microsoft Azure Data Transfer Routing Preference to enjoy lower data transfer costsThis post details the integration of Cloudflare's network with Microsoft Azure's Routing Preference service. This integration allows mutual customers to leverage Cloudflare's global network and Azure's optimized routing solution for lower data transfer costs and predictable performance. The post highlights the extensive peering Cloudflare has with Azure (over 9,500 networks globally, including private interconnections at key locations and over 108 public peering links). It explains how traffic is routed over Cloudflare's network to Azure origins and back, avoiding ISP involvement for the egress path. The post also provides a three-step guide for customers to enable this integration and includes testimonials from customers who have benefited from it.

May 2021Branch predictor: How many "if"s are too many? Including x86 and M1 benchmarks!This post investigates the performance impact of branch prediction, specifically the Branch Target Buffer (BTB), on modern CPUs. It demonstrates that the number of branches in hot code loops can significantly degrade performance once a certain threshold (around 4096 branches) is reached, regardless of code density. The experiments show that unpredicted branches incur a substantial performance penalty, highlighting the importance of optimizing code to minimize branch mispredictions.

Mar 2021Protecting Cloudflare Customers from BGP Insecurity with Route Leak DetectionIntroduced Route Leak Detection, a new feature that monitors BGP advertisements for customer-owned prefixes onboarded to Cloudflare. This feature ingests data from RIPE RIS, RouteViews, and Caida's BMP feed to detect unauthorized route advertisements. Alerts are triggered based on changes in the number of ASNs seeing routes, BGP update frequency, and advertised subnets. The feature supports notification via email, webhooks, and PagerDuty. The post also highlights the importance of RPKI for preventing route leaks and provides statistics on its adoption.

Aug 2020August 30th 2020: Analysis of CenturyLink/Level(3) outageThis post analyzes a CenturyLink/Level(3) outage, detailing how Cloudflare's automated systems detected increased 522 errors, rerouted traffic across alternative network providers (Cogent, NTT, GTT, Telia, Tata), and disabled the affected provider in affected cities. It explains why some customers remained impacted due to single-homed connectivity and CenturyLink's network not honoring route withdrawals. The analysis points to a Flowspec rule as the likely root cause of the outage, causing BGP instability and a sustained increase in BGP updates, potentially due to a loop in rule processing. The post also speculates on the reasons for the extended resolution time, including potential login difficulties for CenturyLink's NOC and the impact of de-peering actions by other tier-1 providers.

Jul 2020Bringing Your Own IPs to Cloudflare (BYOIP)This post announces the general availability of Bring Your Own IP (BYOIP) for Cloudflare's Layer 7 products, Spectrum, and Magic Transit. It details the reasons customers choose BYOIP (allow-listing, DNS updates), the scale of onboarding, and the benefits of using BYOIP with Cloudflare (global announcement, protection, acceleration, regional deployments). It outlines the required documentation (LOA, IRR records) and configuration options (dynamic advertisement toggle, delegations) for Layer 7 and Spectrum services. It also mentions RPKI support for BYOIP prefixes.

Jul 2020Cloudflare outage on July 17, 2020This post details a specific incident where a configuration error in Cloudflare's backbone network caused a widespread outage. The error involved misconfiguring a router in Atlanta, leading to all backbone traffic being routed there and overwhelming the router. The post explains the BGP local-preference mechanism and how the incorrect configuration led to the traffic misrouting. It also outlines the immediate changes made: a maximum-prefix limit on backbone BGP sessions and a change to BGP local-preference for local server routes to prevent recurrence.

Apr 2020Internet performance during the COVID-19 emergencyThis post analyzes the impact of increased Internet traffic due to the COVID-19 pandemic on Cloudflare's network performance. It presents data on global Internet usage growth, showing significant increases in countries like Portugal, Spain, and the UK. It also details TCP RTT measurements in the UK and Italy, demonstrating network stability in the UK despite traffic surges, and identifying strain in Italy that has since improved. The post concludes that the Internet infrastructure is robust and capable of handling sudden demand increases.

Apr 2020Remote work, regional lockdowns and migration of Internet usageThis post details the analysis of a network outage caused by a configuration error in Cloudflare's backbone network, leading to traffic misrouting. It explains how edge router flow records, combined with geolocation data, were used to identify a geographical migration of Internet usage from business districts to residential areas during lockdowns. The post visualizes this migration with heatmaps for various cities and discusses the overall increase in traffic volume. It also touches on the resilience of the Internet and Cloudflare's role in maintaining service during increased strain.

Mar 2020RPKI and the RTR protocolThis post introduces and details the Resource Public Key Infrastructure (RPKI) to Router Protocol (RTR), explaining its role in communicating validated route information from RPKI validators to internet routers. It highlights the lightweight nature of RTR, its separation from cryptographic processing, and the operational security considerations for the transport layer. Cloudflare's open-source GoRTR implementation of the RTR protocol and a Python-based rpki-rtr-client are also presented.

Oct 2019Dogfooding Magic Transit by DDoSing our Austin officeThis post details the dogfooding of Magic Transit by simulating a DDoS attack on the Cloudflare Austin office. It explains how Magic Transit uses BGP to attract traffic to Cloudflare's network edge, where it is protected by multi-layered defenses including XDP, eBPF, iptables, Gatebot, and DoSD. The experiment involved onboarding the Austin office to Magic Transit in an always-on routing topology and then launching a simulated DDoS attack. The post highlights the effectiveness of the mitigation systems in blocking attacks quickly and without service degradation, and discusses lessons learned for engineering and procedural aspects, including fine-tuning mitigations, optimizing routings, and refining run-books for faster customer onboarding.

Jun 2019The deep-dive into how Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline MondayThis post provides a detailed technical analysis of a BGP route leak event that impacted large parts of the internet. It demonstrates how to use RIPE NCC archived data and shell commands to extract and analyze BGP routing information, specifically focusing on the routes leaked by Verizon and their impact on Cloudflare's network. The analysis includes extracting route announcements, AS-PATH data, and timing information to understand the duration and scope of the leak. It also highlights the importance of RPKI ROAs and IRR objects in validating route announcements and discusses the role of BGP communities like NO_EXPORT in preventing such incidents. The post includes code snippets for data collection and processing, making the analysis reproducible.

Jun 2019How Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline TodayThis post details a major internet outage caused by a BGP route leak originating from DQE Communications, amplified by Verizon's lack of filtering. It explains the mechanics of BGP route leaks, the role of BGP optimizers in creating more-specific routes, and the failure of Verizon to implement best practices like prefix limits, IRR filtering, and RPKI validation. Cloudflare's network team contacted DQE to resolve the issue. The post strongly advocates for the adoption of RPKI and MANRS principles to prevent future incidents.

May 2019Argo and the Cloudflare Global Private BackboneThis post introduces the Cloudflare Global Private Backbone, a physical network of private fiber connections linking Cloudflare locations. It details how this backbone, in conjunction with Argo Smart Routing, improves performance by providing more control over end-to-end latency, capacity management, and significantly reduces jitter compared to transit provider links. Argo's transit selection logic is enhanced by the backbone, allowing for more precise routing decisions at each hop.

Nov 2018How a Nigerian ISP Accidentally Knocked Google OfflineThis post details a specific incident where a Nigerian ISP (MainOne) misconfigured their BGP routing, causing a route leak that overwhelmed intermediary networks and led to a widespread outage affecting Google and other services. It explains the concept of route leaks, AS paths, and the trust-based nature of internet routing. The post highlights the lack of filtering by China Telecom CN2 and TransTelecom as a key factor in the propagation of the leak. Cloudflare's internal systems detected and auto-remediated the issue. The post concludes by emphasizing the need for better filtering, RPKI adoption, and adherence to best practices like BCP-38 to mitigate such incidents.

Oct 2018Cloudflare Peering Portal - BetaThis post introduces the Cloudflare Peering Portal, a new tool for ISPs and hosting providers to view detailed statistics on traffic exchanged with Cloudflare. It explains the problems peering solves (bandwidth costs, slow websites, limited network resources) and the different methods of peering (PNI, IXP, hosting a node). The portal provides time series traffic statistics, peering session statistics, prefix data, and POP relative traffic weighting to help partners estimate benefits and make informed decisions about peering.

Sep 2018Bandwidth Alliance: powered by smart routing on Cloudflare’s networkThis post introduces the Bandwidth Alliance, a group of cloud providers aiming to reduce data transfer fees for mutual customers. It details how Cloudflare's Argo traffic routing engine is the technical underpinning of this initiative. Argo's control plane analyzes network performance data and topology to generate optimal routes, prioritizing low-cost or free connections (like Private Network Interconnects) over expensive paid transit, which standard BGP does not inherently do. A limited, routing-only version of Argo is automatically enabled for Bandwidth Alliance partners to ensure traffic is routed to the nearest Cloudflare location for cost savings.

Apr 2018BGP leaks and cryptocurrenciesThis post details a specific BGP leak incident where Amazon's Route53 IP space was announced by eNet Inc. (AS10297) to Hurricane Electric (AS6939), affecting Cloudflare's 1.1.1.1 DNS resolver in several regions. The leak caused DNS poisoning for myetherwallet.com, leading to potential cryptocurrency theft via fake TLS certificates. The post explains the mechanics of BGP leaks, the actors involved, the affected regions, and discusses potential solutions including RPKI/ROA, DNSSEC, HSTS, DANE, and DNS over HTTPS.

Nov 2016Not one, not two, but three undersea cables cut in JerseyThis post details how Cloudflare's BGP traffic engineering automatically rerouted traffic from London to Paris when three undersea cables serving Jersey were cut. The traffic shifted seamlessly to a backup path, demonstrating the resilience of the network and the control provided by BGP routing.

Nov 2016The Internet is Hostile: Building a More Resilient NetworkThis post details Cloudflare's proactive approach to building a more resilient network by actively monitoring internet quality, detecting brownouts (e.g., packet loss), and implementing automatic mitigation mechanisms. It introduces the use of thousands of probes across its global network to collect real-time data on internet infrastructure health, stored in a metrics pipeline using ZeroMQ, Prometheus, and OpenTSDB. The post explains the development of a detection and alerting system based on defined triggers and the application of learnings from DDoS mitigation to automate responses to external network events. It highlights the use of NAPALM and Salt for multi-vendor network device automation to disable faulty links and improve routing. The post quantifies the impact of these improvements on reducing origin reachability errors and improving overall performance.

Jun 2016A Post Mortem on this Morning's IncidentThis post details an incident where significant packet loss occurred due to issues with a transit provider (Telia Carrier). Cloudflare's systems detected the packet loss instantly, but manual intervention was required to disable ports with the problematic provider. The incident resulted in a spike in 522 errors for customers. The post highlights the need for improved communication during incidents and outlines plans to extend automated detection and mitigation systems to all POPs to proactively detect packet loss and move traffic away from problematic providers.

Jan 2016Think Global, Peer Local. Peer with CloudFlare at 100 Internet Exchange PointsThis post details Cloudflare's expansion to 100 Internet Exchange Points (IXPs) and the technical rationale behind this strategy. It explains the concept of peering, the role of IXPs as Layer 2 LANs, and how BGP is used to establish direct connections between networks. The post highlights the performance benefits of local traffic exchange, the improvement in redundancy, and the advantages for DDoS mitigation by having direct paths for attack traffic. It also discusses different IXP operating models (membership-based, commercial, volunteer-based).

May 2015CloudFlare "Interview Questions"This post shares a collection of TCP/IP quirks and trivia, framed as interview questions, to encourage engineers to explore the inner workings of the network stack. It covers topics such as TCP port numbers, URG pointer usage, RST packet payloads, IPv6 flow fields, IP_FREEBIND socket option, PSH flag functionality, TCP timestamp usage in SYN cookies, UDP checksums, simultaneous TCP open, stupid window syndrome, CWE/ECE flags, IP ID and DF bit interaction, SYN packet payloads, SYN+ACK packet payloads, ICMP packet-too-big messages, Linux sysctl settings like tcp_no_metrics_save, SYN queue length and overflow behavior, BGP bogons, TCP MD5 checksums, and differences in IPv4/IPv6 checksumming algorithms. The post aims to spark discussion and knowledge sharing about these networking fundamentals.

Oct 2014Route leak incident on October 2, 2014This post details a BGP route leak incident caused by Internexa, an ISP in Latin America, which accidentally directed large amounts of traffic destined for Cloudflare data centers to a single data center in Medellín, Colombia. This miscommunication caused traffic floods and temporary downtime for customers. Cloudflare worked with Internexa to resolve the leak, quarantined the affected data center, and disabled connectivity with Internexa while investigating the root cause. The incident lasted 49 minutes, impacting traffic by 50% in North America and 12% in Europe. This incident highlights the inherent fragility of the Internet's core routing system and the problem of route leakage.

Jul 2014CloudFlare Joins Three More Peering Exchanges in AustraliaCloudflare established connections to three new Internet exchanges in Australia (Megaport in Sydney, Brisbane, and Melbourne), doubling its reach and enabling direct traffic exchange with ISPs. This peering reduces latency and congestion by bypassing third-party transit providers, leading to improved service performance and a faster web experience for users on peered networks.

May 2013Today's Network IssueThis post details an incident where a merger between transit providers (nLayer, GTT, Inteliquent) and subsequent network consolidation led to incorrect BGP routing. Traffic intended for East Coast and European data centers was misrouted to the Los Angeles data center, causing overload and performance degradation for approximately 10% of connections. Cloudflare worked with the transit provider to rebalance traffic and has since added this scenario to their routing safeguards to prevent future disruptions from upstream network changes.

May 2013Syrian Internet RestoredThis post analyzes the BGP route withdrawal and restoration patterns during the Syrian internet outage, concluding it was a deliberate shutdown rather than a cable cut. It also identifies a specific IP range that maintained connectivity, providing evidence against the official explanation. The post includes a BGPlay video of route restoration and a graph of Syrian traffic to Cloudflare.

May 2013How Syria Turned Off the Internet (Again)This post documents a specific instance of a country (Syria) intentionally disabling its internet connectivity by withdrawing BGP routes. It provides technical evidence of this event through network monitoring data, including a BGPlay visualization of route withdrawal and a graph of traffic to Cloudflare's network from the affected region. This serves as a case study for understanding how BGP can be used to control or sever internet access.

Mar 2013Load Balancing without Load BalancersThis post introduces Cloudflare's use of Anycast routing for load balancing and fault tolerance. It details how Anycast is applied at both the Wide Area Network (WAN) and Local Area Network (LAN) levels. WAN Anycast uses BGP to announce IP addresses from all data centers, directing traffic via the shortest path. LAN Anycast uses BGP and a monitor daemon (Bird) to manage server routes within data centers, allowing for dynamic route withdrawal based on process/server failures and load. It also discusses early experiments with BGP for true load balancing by adjusting route weights and using hashing for consistent routing.

Dec 2012Syrian Internet access reestablished starting 1432 UTCThis post details the reestablishment of Syrian Internet access following an outage. It documents the observed BGP connectivity changes with global carriers (PCCW, TATA, Turk Telecom) and the return of traffic levels to pre-outage levels. The post uses BGP maps and traffic monitoring data to illustrate the recovery process.

Nov 2012Why Google Went Offline Today and a Bit about How the Internet WorksThis post details a specific incident of BGP route leakage caused by a misconfiguration at Moratel, an Indonesian ISP, which led to a widespread Google outage. It explains the BGP protocol, Autonomous Systems, and how route leakage occurs. The author, a Cloudflare network engineer, describes the troubleshooting process, including using `dig +trace` and `show route` commands, and the resolution achieved by contacting Moratel engineers. The post emphasizes the trust-based nature of BGP and the importance of network engineering teams monitoring routes and managing connectivity.

May 2012Today's Outage Post MortemThis post details a specific incident where an incorrect route announcement via BGP caused a 15-minute outage affecting 75% of Cloudflare's network. The root cause was the accidental entry of out-bound routes into the in-bound interface during maintenance in Hong Kong. The post-mortem outlines the mechanism of BGP route propagation and the impact of the incorrect announcement. Future prevention measures include implementing a verification layer for all routing changes and engaging with upstream providers for additional confirmation checks on major routing changes.

Apr 201230% More Traffic in Less Than a Blink of an EyeThis post details how optimizing routing to the Singapore data center resulted in an average latency reduction of 25ms for traffic from Thailand, Malaysia, and Indonesia. This latency reduction led to a 30% increase in bandwidth, hits, and page views in the region, demonstrating the amplification effect of performance benefits on user engagement and the impact of network routing on traffic volume.

Oct 2011A Brief Primer on AnycastThis post introduces and explains the concept of Anycast routing, contrasting it with Unicast. It details how Cloudflare utilizes Anycast across its globally distributed DNS infrastructure and its network of data centers. The benefits of Anycast, including speed, resiliency, and attack mitigation, are explained with examples of traceroutes and network latency.

What this trail covers