Biggest DDoSes of all time generated by protocol 0-day in HTTP/2

In August and September, threat actors unleashed the biggest distributed denial-of-service attacks in Internet history by exploiting a previously unknown vulnerability in a key technical protocol. Unlike other high-severity zerodays in recent years—Heartbleed or log4j, for example—which caused chaos from a torrent of indiscriminate exploits, the more recent attacks, dubbed HTTP/2 Rapid Reset, were barely noticeable to all but a select few engineers.

HTTP2/Rapid Reset is a novel technique for waging DDoS, or distributed denial-of-service attacks, of an unprecedented magnitude. It wasn’t discovered until after it was already being exploited to deliver record-breaking DDoSes. One attack on a customer using the Cloudflare content delivery network peaked at 201 million requests per second, almost triple the previous record Cloudflare had seen of 71 million rps. An attack on a site using Google’s cloud infrastructure topped out at 398 million rps, more than 7.5 times bigger than the previous record Google recorded of 46 million rps.

Doing more with less

The DDoSes hitting Cloudflare came from a network of roughly 20,000 malicious machines, a relatively small number compared with many so-called botnets. The attack was all the more impressive because, unlike many DDoSes directed at Cloudflare customers, this one resulted in intermittent 4xx and 5xx errors when legitimate users attempted to connect to some websites.

“Cloudflare regularly detects botnets that are orders of magnitude larger than this—comprising hundreds of thousands and even millions of machines,” Cloudflare Chief Security Officer Grant Bourzikas wrote. “For a relatively small botnet to output such a large volume of requests, with the potential to incapacitate nearly any server or application supporting HTTP/2, underscores how menacing this vulnerability is for unprotected networks.”

The vulnerability that HTTP/2 Rapid Reset exploits resides in HTTP/2, which went into effect in 2015 and has undergone several overhauls since then. Compared to the HTTP/1 and HTTP/1.1 protocols that predated it, HTTP/2 provided the ability for a single HTTP request to carry 100 or more “streams” that a server can receive all at once. The resulting throughput can lead to almost 100 times higher utilization of each connection, compared with the earlier HTTP protocols.

The increased efficiency wasn’t just useful for distributing video, audio, and other sorts of benign content. DDoSers began leveraging HTTP/2 to deliver attacks that were orders of magnitude larger. There are two properties in the protocol allowing for these new efficient DDoSes. Before discussing them, it’s useful to review how DDoS attacks work in general and then move on to the way HTTP protocols prior to 2.0 worked.

There are several types of DDoS attacks. The best known forms are volumetric and network protocol attacks. Volumetric attacks stuff incoming connections to a targeted site with more bits than the connection can carry. This is akin to routing more vehicles onto a highway than it can accommodate. Eventually, the traffic comes to a standstill. As of last year, the biggest recorded volumetric DDoS was 3.47 terabits per second.

Network protocol DDoSes work to overwhelm routers and other devices found in layers 3 and 4 of the network stack. Because they work on these network layers they’re measured in packets per second. One of the largest protocol attacks was one blocked by security firm Imperva that peaked at 500 million packets per second.

The type of attack carried out by HTTP/2 Rapid Reset falls into a third form of DDoS known as Application Layer attacks. Rather than trying to overwhelm the incoming connection (volumetric) or exhaust the routing infrastructure (network protocol), application-level DDOSes attempt to exhaust the computing resources available in layer 7 of a target’s infrastructure. Floods to server applications for HTTP, HTTPS, and SIP voice are among the most common means for exhausting a target’s computing resources.

In August 2022, Google said it blocked such a DDoS that peaked at 46 million requests per second. In February, Cloudflare blocked an application-level DDoS that reached 71 million requests per second.

The power of HTTP/2 Rapid Reset DDoSes stems from its ability to do two things. The first is to take advantage of the increased throughput and funnel 100 more requests into a single round-trip TCP connection. As Google explained earlier this week:

One of the main constraints when mounting a Layer 7 DoS attack is the number of concurrent transport connections. Each connection carries a cost, including operating system memory for socket records and buffers, CPU time for the TLS handshake, as well as each connection needing a unique four-tuple, the IP address and port pair for each side of the connection, constraining the number of concurrent connections between two IP addresses.

 

In HTTP/1.1, each request is processed serially. The server will read a request, process it, write a response, and only then read and process the next request. In practice, this means that the rate of requests that can be sent over a single connection is one request per round trip, where a round trip includes the network latency, proxy processing time and backend request processing time. While HTTP/1.1 pipelining is available in some clients and servers to increase a connection's throughput, it is not prevalent amongst legitimate clients.

 

With HTTP/2, the client can open multiple concurrent streams on a single TCP connection, each stream corresponding to one HTTP request. The maximum number of concurrent open streams is, in theory, controllable by the server, but in practice clients may open 100 streams per request and the servers process these requests in parallel. It’s important to note that server limits can not be unilaterally adjusted.

 

For example, the client can open 100 streams and send a request on each of them in a single round trip; the proxy will read and process each stream serially, but the requests to the backend servers can again be parallelized. The client can then open new streams as it receives responses to the previous ones. This gives an effective throughput for a single connection of 100 requests per round trip, with similar round trip timing constants to HTTP/1.1 requests. This will typically lead to almost 100 times higher utilization of each connection.

In an email Thursday researcher Pascal Geenens of security firm Radware said that what sets the Rapid Reset attack apart from others is its unique ability to reuse the same TCP/TLS session to send a near-infinite number of requests.

"The HTTP/2 multiplexing leverages streams between client and server for the request/response and these streams are implemented in layer 7, on top of the TCP and TLS sessions," he explained. "The client and server can perform a theoretically infinite number of requests/response transactions without having to redo TCP or TLS handshakes."

In an earlier post Thursday, Geenens explained it this way and provided a helpful graphic for visualizing how HTTP/2.0 works:

In HTTP/1.0, each TCP connection handles one request and response. HTTP/1.1, outlined in RFC2616, introduced pipelining. Though not commonly adopted, malicious actors have exploited this feature to boost the rate at which they can send requests over a single TLS session, aiming to overwhelm a web server or its backend infrastructure.

 

Although HTTP/1.1 pipelining allows a new request to be initiated even before the previous request receives a response, the responses will always be delivered in order. That means that a single large response, such as a download, can delay responses to requests with small responses made after the large request. This phenomenon is commonly referred to as head-of-line blocking.

 

HTTP/2, as defined in RFC7540, introduced a solution for the head-of-line blocking issue by implementing a new binary framing layer that encapsulates and transfers HTTP messages between client and server.

 

HTTP/2 binary framing layer

Ilya Grigorik

 

Figure 1: HTTP/2 binary framing layer (image:Ilya Grigorik* )

 

The TLS session between client and servers basically serves as a tunnel allowing multiple concurrent and independent streams to be transferred between client and server. Responses to requests are no longer guaranteed to be received in order and longer requests will not block the transmission of responses for smaller requests.

 

The head-of-line blocking is no concern to the attacker. Their goal is to flood the server with as many requests as possible in the shortest amount of time while keeping the request generation as efficient as possible, preferably without the need to receive and process too many responses from the server.

Enter reset stream

The 100-fold increase in requests is only the first of two things that make HTTP/2 protocol such an efficient DDoS vehicle. The second HTTP/2.0 property allowing the record-breaking DDoSes is the capability it gives to end-user clients and servers alike to unilaterally terminate a particular stream. Technically known as the RST_STREAM, this reset stream takes effect immediately, freeing the attacker to send a new stream of malicious requests. The attacker then repeats the sending and almost instantaneous resetting of the stream over and over, hence the phrase “rapid reset” in the attack name.

As the Google post explained:

The ability to reset streams immediately allows each connection to have an indefinite number of requests in flight. By explicitly canceling the requests, the attacker never exceeds the limit on the number of concurrent open streams. The number of in-flight requests is no longer dependent on the round-trip time (RTT), but only on the available network bandwidth.

 

In a typical HTTP/2 server implementation, the server will still have to do significant amounts of work for canceled requests, such as allocating new stream data structures, parsing the query and doing header decompression, and mapping the URL to a resource. For reverse proxy implementations, the request may be proxied to the backend server before the RST_STREAM frame is processed. The client on the other hand paid almost no costs for sending the requests. This creates an exploitable cost asymmetry between the server and the client.

 

Another advantage the attacker gains is that the explicit cancellation of requests immediately after creation means that a reverse proxy server won't send a response to any of the requests. Canceling the requests before a response is written reduces downlink (server/proxy to attacker) bandwidth.

HTTP/2 Rapid Reset attacks mostly affect the large infrastructure providers. Software smaller providers use, such as NGINX, Apache Server, and HAProxy, largely had defenses already in place, although many have patched their systems to make them more effective.

“Google and Cloudflare should have written ‘the typical web server is not affected, but the typical Cloud implementation is,’” Stephen Eissing, an engineer who works on an Apache team and also works on HTTP/2, HTTP/3 implementation, wrote on Mastodon. “Blaming the exploit on the protocol standard is lame. When you implement 100% of an RFC you are not done with your work.”

Alex Forster, the lead engineer in Cloudflare's DDoS Mitigation team, pushed back on the idea the Rapid Reset attack wasn't viable on an array of other server apps that implement HTTP/2.

"It is estimated that at least 60% of public-facing websites and APIs use HTTP/2, meaning all are / were vulnerable to this threat–underscoring that this was not hyperbolic and represents a real existential threat to the Internet," he wrote in an email. "While the worst-case scenarios have been eliminated with the patching by the biggest players, there will be a long-tail of vendors who do not implement patches, and an even longer-tail of organizations who do not apply those patches."

Geenens of Radware further observed that the new attack, if it had remained viable, could have tipped the DDoS scales in the favor of attackers in a way that has never happened before. He explained:

The significance of Rapid Reset is in the efficiency with which attackers can generate requests. With the same resources the attackers can significantly increase their attack rates by eliminating overhead related to TCP and TLS handshakes. Evolution in technologies have always allowed us to get more capable over time. Attackers gain from this evolution, but so do businesses. As attackers got more capable over the years, businesses equally gained in resources and scale to counter those attack rates. There is a direct relation between the cost and resources available to attackers and the cost and resources available for effective mitigations. This flaw in HTTP/2 that was abused by an attacker and discovered by Google could have disturbed this balance and allowed the attackers to gain an advantage on their victims.

Despite the diminished impact HTTP/2 Rapid Reset has on smaller HTTP/2 server applications, anyone who administers one should check with the developer for mitigation guidance. A large number of developers of smaller applications, listed here, are providing fixes for the vulnerability, which is tracked as CVE-2023-44487. All large cloud providers are currently patched.