kelmenhorst / quic-censorship Goto Github PK

I have collected HTTP/3 censorship measurements on a VPS connected to AS45090 repeatedly (21 times) between 14.01. and 21.02.2022. The test list had around 300 entries and The analysis of the measurements suggests that the Chinese firewall does not explicitly handle QUIC traffic but HTTP/3 traffic is nevertheless impaired.

Not all websites that were blocked over HTTPS were also blocked over HTTP/3. The result figure shows that the Great Firewall uses different methods to block HTTPS connections. Different censorship methods cause different failure types when attempting to connect.

To around 24% of tested websites, not even the TCP handshake could be completed and the connection attempt timed out in the first roundtrip. The same set of websites was also unavailable over HTTP/3.
This observed pattern suggests that the corresponding IP endpoints are blocked: When the Great Firewall detects a blacklisted IP address it seems to block the connection no matter whether the request used TCP or QUIC.

                                  HTTPS                                                   HTTP/3

Figure 1: Failure rates of hosts tried over HTTPS vs. HTTP/3. AS45090. Endpoints that fail during the TCP handshake when using HTTPS also fail over HTTP/3. Hosts that are blocked with TCP/TLS-specific methods stay available over HTTP/3.

Follow-up measurements with quicping support this assumption: Attempting to ping the unavailable HTTP/3 hosts with quicping failed in nearly all cases which shows that the censorship is independent from the announced QUIC version and not based on any TLS feature in the handshake. Given the correspondence with failing TCP handshakes, it is most likely that the IP endpoints are blocked.
                                  HTTP/3                                                 quicping

Figure 2: Failure rates of hosts tried over HTTP/3 vs. pinged with quicping. AS45090. Hosts that time out during an HTTP/3 request do not respond to a QUIC Ping packet either. Thus, the filtering of QUIC connections is not based on TLS properties negotiated in the handshake.

Figure 2: The results consistency of the hosts between multiple measurements was high which means that hosts were either always blocked or always available. I define the result consistency as the percentage of the most frequently encountered measurement result out of all measurements directed at this host.

China applies IP blocking which likely causes the TCP handshake timeouts when using HTTPS. Since both TCP and QUIC run over IP, IP-blocklisted hosts are not available via both protocols.
These findings show that there was no development in regards to HTTP/3 censorship in China since last year (Elmenhorst et al. 2021). A possible explanation: The censor is not motivated to censor QUIC specifically, since browsers mostly use TCP to explore HTTP/3 support anyways which means that it is enough to block HTTP(S).

In March 2021 we have collected HTTP/3 censorship measurements in Uganda. The tests were run from mobile devices in two different networks: Airtel Uganda (AS37075, 10 repetitions) and MTN (AS20294, 4 repetitions). The test list had 16 hosts.

In both networks, HTTP/3 endpoints are targeted by censorship.
The blocks in AS20294 and AS37075 have a lot of the same characteristics:

Separate HTTPS and HTTP/3 blocking

Both for HTTPS and HTTP/3 the censors seem to apply multiple different blocking techniques which cause a variety of different failure types when trying to connect to blocked hosts.

Overall, it is very likely that HTTPS and HTTP/3 traffic are handled differently by the censorship system (in contrast to IP endpoint blocking which treats all IP-based traffic the same). A strong indication for this are the conn-refused and conn-reset failures that happen during HTTPS requests: These error types suggest the injection of TCP RST packets which, of course, is only applicable to TCP. Still, these same websites are blocked over QUIC. Since QUIC connections cannot be terminated by TCP RSTs, there needs to be a different mechanism in place to block QUIC.

                        HTTPS                                 HTTP/3                                             HTTPS                                 HTTP/3

Figure 1: Failure rates of hosts tried over HTTPS vs. HTTP/3. Left: AS37075, right: AS2029 (details below).

QUIC connection timeouts

On some days, QUIC connections did not timeout during the handshake but on the working connection. I can think of 2 explanations for this:

a) The censorship system of the ISPs is too slow to interfere immediately during the handshake. This seems unlikely because on other days and. Moreover, such an efficiency issue would be sensitive to different times of day (peak and off-peak traffic volumes) but this sensitivity is not visible in the data.

b) The filter is intentionally only applied to application data. Such an interference could be implemented by keeping state about connections or by applying the filter only to smaller data packets (handshake packets are at least 1200 Bytes).

QUIC Ping

When HTTP/3 transactions time out during the QUIC handshake, these HTTP/3 hosts did also not respond to quicping. This observation indicates that the blocking already happens during the first roundtrip. There is probably no DPI of the payload of the QUIC Initial packet since the blocking also happens with the random payload generated by quicping.
On the other hand, when the blocking only happens after a few roundtrips (conn-to), quicping works (quicping only elicits 1 roundtrip).

                        HTTP/3                                 quicping                                         HTTP/3                                 quicping

Figure 2: Failure rates of hosts tried over HTTP/3 vs. pinged with quicping. Left: AS37075, right: AS2029.

No SNI blocking

When changing the SNI to an allowed domain name, the corresponding websites remain blocked.

                        HTTP/3                           w/ SNI spoof                                       HTTP/3                             w/ SNI spoof

Figure 3: Failure rates of hosts tried over HTTP/3 vs. HTTP/3 with spoofed SNI. Left: AS37075, right: AS2029.

Temporal variance

There is a rather strong temporal variance of failure types when running HTTPS and HTTP/3 requests in the network of Airtel Uganda (AS37075). The result consistency is low for all measured protocols (HTTPS over TCP, HTTP/3 over QUIC and quicping): Around 50% of hosts have the same test result in only 80% or less of measurements.
It's unclear to me why the censorship is this inconsistent. It will take further continuous measurements to find out more about what is going on here.

Figure 4: Cumulative distributions of result consistency, calculated for each host as: no. of measurements with most frequent result / no. of all measurements. Left: AS37075, right: AS20294.

Airtel Uganda, AS37075

                                      HTTPS                                                   HTTP/3

Figure 5: Failure rates of hosts tried over HTTPS vs. HTTP/3. AS37075.

To each protocol, multiple different blocking methods are applied which cause multiple different error types:

conn-refused and TCP-hs-to indicate that the blocking happened during the TCP handshake (likely no DPI on TLS). These HTTPS error types correspond to QUIC-hs-to failures which also indicate that the blocking happened early on, during the QUIC handshake. While TCP-hs-to and QUIC-hs-to could be caused by the same IP blocking method, this is not true for conn-refused which is caused by an injected TCP RST.

The above described variance of censorship is very striking in AS37075, especially when comparing the measurements from individual, even consecutive, days:

Figure 6: Temporal variance of blocking is the network of Airtel Uganda during March, 3 and March, 15 2022.

MTN Uganda, AS20294

                                      HTTPS                                                   HTTP/3

Figure 7: Failure rates of hosts tried over HTTPS vs. HTTP/3. AS20294.

For each HTTPS error type, there is a corresponding HTTP/3 error type, i.e. if an HTTPS request failed with the error type A, the corresponding HTTP/3 request to the same URL caused error type B.
Websites that failed over HTTPS with a conn-refused message, where unavailable over HTTP/3 due to a QUIC handshake timeout (QUIC-hs-to). Both error types occur during the first roundtrip. HTTPS conn-reset failures corresponded to HTTP/3 conn-to failures which both indicate that the blocking happened after a few roundtrips.

Reports suggest that Russia started to block HTTP/3 traffic nationwide on the 4th of March 2022. With the increasing restrictions of media since the beginning of the invasion of Ukraine, internet censorship in Russia is rapidly changing and getting more restrictive. In this context, people reported that HTTP/3 doesn't work anymore.

• https://ntc.party/t/http-3-quic/1823/
• net4people/bbs#108

We were able to run 2 rounds of HTTP/3 measurements in a mobile (PJSC MegaFon Yota, AS31213), and a landline network (PJSC Rostelecom, AS12389) in Russia, on March 27 and 31. The test list contained 11 websites, i.e. 11 HTTPS endpoints and 11 HTTP/3 endpoints were measured.

Yota, AS31213

                                  HTTPS                                                   HTTP/3

Figure 1: Failure rates of hosts tried over HTTPS vs. HTTP/3. AS31213.

Table 1 Summary of the results.

HTTP/3 overblocking

The second column of Table 1 shows that all tested websites, except for vk.com were blocked over HTTP/3. The first column shows that only 4 websites (6 endpoints) were blocked over HTTPS. This observation is consistent with reports saying that all foreign websites would be blocked over HTTP/3.

Immediate timeouts

HTTP/3 blocking appears as timeouts during the QUIC handshake. The timeout always occurs during the very first roundtrip, i.e. the client never receives any response from the server and doesn't read a single byte.

quicping works

quicping works in all measured cases which means that the blocking method is more fine grained than just blocking UDP endpoints (see the fifth column). Explanation: If the UDP endpoint was blocked, quicping (which only elicits one roundtrip) should also be affected by the blocking because the blocking of HTTP/3 traffic happens during the very first roundtrip.

So why is quicping not impaired? Most likely, quicping packets pass the filter because they do not have the most commonly used QUIC version (1). Instead, they carry special version strings which are reserved for version negotiation. Supporting this consideration, users in Russia have experienced that HTTP/3 works in some networks when they disabled QUIC version 1 and used QUIC version draft-29 instead.

SNI blocking (?)

We measured all failed endpoints again, this time with the SNI set to vk.com (see the third column). Blocked endpoints remain blocked when using an allowed SNI. In contrast, users from other networks in Russia have reported that blocked hosts become available when using a fake SNI which shows that HTTP/3 blocking varies between ISPs.

Lastly, we used the domain names of the blocked HTTP/3 hosts as SNI in requests to vk.com (see the fourth column). Unexpectedly, vk.com was suddenly unavailable when using one of the following blocked domain names as SNI: navalny.com, www.instagram.com, censor.net, www.facebook.com - which is precisely the set of hosts that are also blocked over HTTPS. This indicates that certain SNIs trigger blocking and that the censor parses the Initial QUIC packet in order to inspect the SNI. However, since using a fake SNI is not enough to evade blocking in Yota's network, there is probably an additional layer of censorship which drops packets to unwanted destination endpoints regardless of the SNI.

QUIC version filter as an indication for DPI

QUIC uses Initial encryption which means that Initial packets in the handshake are already encrypted. While this is better than no encryption, it is only a weak protection because the keys can be derived from the connection IDs and the used QUIC version, so an observer of the connection can decrypt Initial packets.

It's possible that censors in Russia use a shortcut to decrypt and parse QUIC Initials by assuming that most traffic is using QUIC version 1. When they apply the decryption algorithm to a QUIC packet of version draft-29 rather than version 1, the decryption fails. This would explain the observation that draft-29 packets do not trigger the filter and that quicping works.

Blocking scenario

A possible blocking scenario which would explain the above described observations is the following:

• The ISP manages a blacklist of SNIs. Given the measurements, we assume that at least navalny.com, www.instagram.com, censor.net, www.facebook.com are on the blacklist.
• All HTTP/3 requests with version 1 are null routed. Either, there is a whitelist to this rule, excluding the IP endpoints of at least vk.com from the blocking, or, this null routing is only applied to international network traffic.
• There is a second filter inspecting the SNI which seems to be applied to all HTTP/3 traffic (national and international): If the SNI of a given packet is on the blacklist, it cannot pass the filter.

Rostelecom, AS12389

                                  HTTPS                                                   HTTP/3

Figure 2: Failure rates of hosts tried over HTTPS vs. HTTP/3. AS12389.

I have discovered seemingly systematic HTTP/3 blocking in AS133694. AS133694 is downstream from an ISP belonging to previously government-owned ISP Tata Communications (AS4755). The measurements were taken using a VPN, and the input list of 226 HTTPS and 226 HTTP/3 endpoints was tested 8 times between the 21st and 25th of February.

                            HTTPS                                                   HTTP/3

Nearly all requests that were blocked over HTTPS were also blocked over HTTP/3. In fact, more requests failed over HTTP/3 than HTTPS so there is some over-blocking of QUIC traffic.

HTTPS requests to unwanted hosts terminate after the TCP connect handshake, during the TLS handshake, and the client receives an End-of-File error. The EOF-err causes the TCP connection to terminate immediately.

The HTTP/3 blocking occurs as a QUIC handshake timeout failure and it is already applied during the very first roundtrip, which means that the client does not receive any response from the server (0 bytes read). Since the QUIC client re-transmits the Initial packets, it takes several seconds (quic-go has a default handshake idle timeout of 5 seconds) until the client quits. Thus, the censor has to continue blocking every single retransmit packet until the client gives up.

Inconsistency

QUIC blocking was, in comparison to HTTPS, rather unstable which means that usually blocked HTTP/3 hosts were sometimes, randomly it seems, unblocked.

No SNI blocking

We changed the SNI in the TLS ClientHello to an allowed domain and measured the HTTP/3 hosts again: There is no difference in the results. While some requests were suddenly successful, this was in the scope of the general inconsistency of HTTP/3 blocking described above. Additionally, when we measured uncensored endpoints and changed the SNI to the names of censored hosts, the requests were successful 100% of the time. It's therefore safe to say that there is no parsing of the SNI in HTTP/3 traffic.

UDP endpoint blocking

When HTTP/3 fails, quicping also fails (even though quicping results do have the same level of inconsistency as HTTP/3 results).
Since HTTPS requests to blocked hosts do not timeout but are actively blocked (EOF-err), the two protocols are most likely handled differently by the censor. Thus, IP blocking is not probable.
If IP blocking is not in place the failures of quicping can best be explained by UDP endpoint blocking targeting HTTP/3 traffic.

We re-ran follow-up measurements after all HTTP/3 requests that failed:

As of the time of writing this, Firefox also has a Boolean config alt-svc-mapping-for-testing in addition to the network.http.http3.alt-svc-mapping-for-testing setting. It is unclear if the Boolean setting needs to be enabled to true for the http3 setting to work.

QUIC separate packet types between those using a long header (initial, handshake, 0RTT) and those using a short header (1RTT) -- the key being the first bit set to 1 for long headers. The "QUIC Ping" is based on long headers, and tests whether an initial packet makes it all the way to the server. There are however usages of QUIC in which the first packet seen on a path is a short header packet -- for example, when probing a new path before migration or multipath establishment. I think it would be interesting to test whether networks would carry such "1RTT" packets.

Many QUIC servers implement the "stateless reset" function. When they receive a 1RTT with a destination CID that they don't understand, they respond with a "stateless reset" -- a 1RTT packet ending with the reset signature associated with the CID. That function might be used in a variant of "QUIC Ping" that sends 1RTT packet. Of course, there is no guarantee that servers send such resets -- but there is also no guarantee that servers send a version negotiation packet in response to a handshake packet. It might be interesting to try.