Glad that it's published, I'd been following it since ESNI draft days. Was pretty useful back when I was in India since Jio randomly blocked websites, and cloudflare adopted the ESNI draft on its servers as did Firefox client side which made their SNI based blocking easy to bypass.
There was a period where I think both disabled ESNI support as work was made on ECH, which now is pretty far along. I was even able to setup a forked nginx w/ ECH support to build a client(browser) tester[0].
Hopefully now ECH can get more mainstream in HTTPS servers allowing for some fun configs.
A pretty interesting feature of ECH is that the server does not need to validate the public name (it MAY) , so clients can use public_name's that middleboxes (read: censors) approve to connect to other websites. I'm trying to get this added to the RustTLS client[1], now might be a good time to pick that back up.
[0] https://rfc9849.mywaifu.best:3443/ [1] https://github.com/rustls/rustls/issues/2741
Why didn't the Indian government block traffics based on IP instead? That would make it much harder to bypass.
If i'm not mistaken its because IPs are actually much easier to rotate than domains.
E.g. all the users will remember `example.com` , underlying it doesn't matter what IP it resolves to. If the IP gets "burned" , then the providers can rotate to a new IP (if their provider allows).
Vs. telling your users to use a new domain `example.org` , fake websites etc.
Also sensible ISPs usually don't block IPs since for services behind a CDN it could lead to other websites being blocked, though of course sometimes this is ignored. See also: https://blog.cloudflare.com/consequences-of-ip-blocking/
That's why you have a strictly legal domain that enables a convoluted redirect with plausible deniability (not 302)
It'll still eventually stick, but a lot slower
> Was pretty useful back when I was in India since Jio randomly blocked websites
With Jio, you don't really need ECH at all. The blocks are mostly rudimentary and bypassed with encrypted DNS (DoH / DoT / DNSCrypt) and Firefox (which fragments the TLS ClientHello packets into two).
Should've added this was back in like 2018 or so. Setting up DoH was harder than enabling SNI, and from my testing back then they were hard filtering on SNI (e.g. I used OpenSSL CLI to set the SNI to `pornhub.com` and connect to "known good" IPs, it'd still get reset).
Funnily enough, not setting the SNI and connecting the the origin IP, and then requesting the page worked fine.
I wrote about ECH a couple of months ago, when the specs were still in draft but already approved for publication. It's a short read, if you're not already familiar with ECH and its history: https://www.feistyduck.com/newsletter/issue_127_encrypted_cl...
In addition to the main RFC 9849, there is also RFC 9848 - "Bootstrapping TLS Encrypted ClientHello with DNS Service Bindings": https://datatracker.ietf.org/doc/rfc9848/
There's an example of how it's used in the article.
Thanks for the writeup, Ivan, I am a great fan of your work!
Now we need to get Qualys to cap SSL Labs ratings at B for servers that don't support ECH. Also those that don't have HSTS and HSTS Preload while we're at it.
Thanks! Sadly, SSL Labs doesn't appear to be actively maintained. I've noticed increasing gaps in its coverage and inspection quality. I left quite a while ago (2016) and can't influence its grading any more, sadly.
Is there a well-maintained alternative to SSL Labs you can recommend?
I use testssl.sh [1] mostly because I can test things not publicly accessible.
Yes, there is! After I left SSL Labs, I built Hardenize, which was an attempt to go wider and handle more of network configuration, not just TLS and PKI. It covers a range of standards, from DNS, over email, TLS and PKI, and application security.
Although Hardenize was a commercial product (it was acquired in 2022 by another company, Red Sift), it has a public report that's always been free. For example:
https://www.hardenize.com/report/feistyduck.com
The CSP inspection in Hardenize could use a refresh, but the TLS and PKI aspects are well maintained [at the time of writing].
> There's an example of how it's used in the article
A bit tricky in Go, but nothing too complicated. We implemented ECH in Aug 2024 for our DNS Android app and it has worked nicely since: https://github.com/celzero/firestack/blob/09b26631a2eac2cf9c...
(Disclosure: I'm a Caddy maintainer), Caddy already supports ECH, leaning on the DNS plugins to automate setting the DNS HTTPS records to wire it up. Here's a lot of technical detail about it https://caddyserver.com/docs/automatic-https#encrypted-clien...
one angle that hasn't come up here yet: ECH basically kills TLS fingerprinting as a bot detection signal
right now tools like Cloudflare Bot Management rely heavily on JA3/JA4 hashes - they fingerprint the ClientHello to identify scrapers vs real browsers. if the ClientHello is encrypted, that whole detection layer collapses. you can still do behavioral analysis and JS challenges, but the pre-HTTP layer that currently catches a huge chunk of naive bots - gone
curious how Cloudflare handles this internally given they're one of the biggest ECH adopters but also one of the biggest bot detection vendors. seems like they're eating their own lunch on this one, or they've already shifted their detection stack to not rely on it as much
Cloudflare can and must decrypt the ClientHello for the sites it serves in order to actually serve the traffic. Using ECH with CF means you use their ECH domain and their keys.
If you control the domain you're fingerprinting clients on, you can decrypt the inner ClientHello and fingerprint on that.
If you're not in control of the domain you're fingerprinting, then ECH is working as intended.
I don't expect naive bots to implement ECH any time soon, though. If a bot can't be bothered to download curl-impersonate, they won't pass any ECH flags either.
the naive bot point is true but the threat model that actually matters is the other end - the sophisticated bots that already do implement TLS spoofing. those are the ones using got-scraping, curl-impersonate, or custom TLS stacks specifically to pass JA3/JA4 checks. they're already past the "can't be bothered" threshold.
for that tier, ECH flips the dynamic a bit. right now detection can use JA3/JA4 as a positive signal - "this fingerprint matches Chrome 120, looks clean". with ECH, if the bot is running behind a CDN that terminates ECH (like Cloudflare), the server sees a decrypted ClientHello that looks like... a real Chrome on Cloudflare's infrastructure. the fingerprint is clean by construction.
so paradoxically ECH might make things harder for the sophisticated bot detection case while doing nothing about the naive case, which is sort of backwards from what you'd expect.
It doesn't prevent fingerprinting, stop spreading misinformation. It only prevents your ISP from knowing what website you're connecting to.
fair point, I should have been more precise. the server (Cloudflare in this case) still decrypts the inner ClientHello and can fingerprint it - jannesan and jeroenhd are right about that.
the part that changes is passive fingerprinting from third parties - network middleboxes, ISPs, DPI systems that have historically been able to read ClientHello parameters in transit and build behavioral profiles. that layer goes away. for bot detection specifically that matters less since detection happens at the server, so your correction stands for that use case.
the Cloudflare paradox I was gesturing at is maybe better framed as: for sites NOT on Cloudflare, ECH makes it harder for Cloudflare (as a network observer) to do pre-connection fingerprinting. but for their own CDN customers, they decrypt it anyway so nothing changes for them. the conflict is more theoretical than practical for their current product.
> the part that changes is passive fingerprinting from third parties
That's exactly what I said:
> It only prevents your ISP from knowing what website you're connecting to.
Why would Clownflare ever see traffic to sites not on Clownflare?
They do routing. Even if you're connecting to a non Cloudflare server, the traffic may still be routed through their servers.
Why would they want to peek traffic? Most likely for statistics (most frequently visited websites etc).
Can you give an example of a BGP route or traceroute to a site not on Clownflare that was routed through Clownflare?
Since most ISPs also maintain their own DNS resolver, they could always reverse lookup the IP address AFAIK.
The whole idea behind ECH is one IP hosts tons of sites (eg. CDN) so you have no idea which one it is.
Also reverse lookup has nothing to do with hosting own DNS resolver.
What you're describing is a SNI, not ECH. Those two serve very different purposes.
> Also reverse lookup has nothing to do with hosting own DNS resolver.
It has everything to do with that. Had you used two brain cells, you would've known that they can memorize the IP address and the domain name, and if you connect to that IP in a short period of time, most likely you visited that domain name.
SNI is unencrypted, so your ISP can see it. ECH encrypts it.
How does this relate to my comment?
True. ECH is useless if you're using plain DNS. DNS over TLS or HTTPS is the way to go.
What OP wrote seems correct:
> ECH basically kills TLS fingerprinting as a bot detection signal
They are not talking about fingerprinting in general. Please elaborate how else TLS fingerprinting can be done.
I am talking about TLS fingerprinting, not JS fingerprinting.
> Please elaborate how else TLS fingerprinting can be done.
By doing everything as it is right now?
How would you (an arbitrary web server) fingerprint a TLS connection if the Client Hello is encrypted?
By decrypting it? I don't think you know how TLS, or E2E works in general. ISP doesn't perform the fingerprinting, the server does.
Of course! My bad, thanks for engaging.
The website owner (or cloudflare in this case) has the keys to decrypt the client hello. That's necessary for routing information.
You're right, sorry! I got confused myself.
ECH is great from a privacy perspective, but I’m curious how well this will actually work in practice.every time the web encrypts more metadata there’s pushback from middleboxes and network operators.
The tension is that Security and Dev parts of the stack remove the actual troubleshooting capabilities of the Network layer without opening up the tools that are supposed to replace them.
It's not a problem if Network can still do their job. It's a whole other matter to expect Network to do their job through another layer. You end up with organizations that can't maintain their applications and expect magic fixes.
Orgs that are cooperative probably don't have this issue but there are definitely parts of some organizations that when one part takes capability from another they don't give it back in some sort of weird headcount game despite not really wanting to understand Network to a Network level.
This feels like a recurring pattern in the stack. abstraction removes visibility faster than tooling replaces it.
Encryption and higher-level platforms are great for security and productivity, but the debugging surface keeps shrinking. Eventually when something breaks, nobody actually has the layer-by-layer visibility needed to reason about it.
Something that puzzles me about ECH:
> In verifying the client-facing server certificate, the client MUST interpret the public name as a DNS-based reference identity [RFC9525]. Clients that incorporate DNS names and IP addresses into the same syntax (e.g. Section 7.4 of [RFC3986] and [WHATWG-IPV4]) MUST reject names that would be interpreted as IPv4 addresses.
Aside from apparently not considering the existence of IPv6, why are IP-based certificates explicitly ruled out? This makes the spec entirely meaningless for small servers and basically requires shifting hosting to shared hosts/massive CDNs to provide any protection against SNI snooping.
\> This makes the spec entirely meaningless for small servers and basically requires shifting hosting to shared hosts/massive CDNs to provide any protection against SNI snooping.
Actually you can setup ECH on your server, and configure the public_name to be something like `cloudflare-ech.com` , so clients would indeed use that in the OuterSNI, connect to you, without you needing to use CF. And middleboxes might think they are indeed connecting to CF (though CF publishes their IP ranges so this could be checked elsewhere).
IPv6 addresses aren't confusable with a DNS name in these syntaxes AFAIU so it's not that they didn't consider IPv6 but that it's not relevant to the issue.
Yes, "Don't stand out" technologies like ECH aren't useful if you inherently stand out anyway. They're intended to make broad surveillance and similar undirected attacks less effective, they aren't magic invisibility cloaks and won't protect you if you're a singular target.
I think it's saying that you can't make the name look like an IP address; i.e. if the syntax were www.google.com[142.250.117.139] (I'm making this syntax up) you couldn't put 142.250.117.139[142.250.117.139].
The syntax being referred to includes some obscure, outdated addressing formats (IPv4 addresses represented as two or three number groups in dotted notation rather than the normal 4).
However, "DNS-based reference identity [RFC9525]" seems to explicitly disallow IP-based certificates by requiring a DNS name. I can only interpret the sentence I quoted as written to say "make sure you never ever accidentally validate an IP address".
I don't think your interpretation is right. If it were,
> Clients that incorporate DNS names and IP addresses into the same syntax
They wouldn't mention the IP addresses at all. Also, notice the word "and".
The colon isn’t a valid character in DNS, so there’s just no risk of confusing IPv6 addresses (which contain at least one colon in all notations I’ve seen).
For IPv4, there’s room for ambiguity.
And how are IP certificates required for small servers?
> For IPv4, there’s room for ambiguity.
I can't think of a single numeric TLD, so I don't think anyone is confusing IP literals with domain names, unless they're doing so extremely lazily.
> And how are IP certificates required for small servers?
You need a valid certificate as the outer certificate which contains an SNI that will still be readable. For cloudflare.com and google.com that's easy; you can't tell what website Cloudflare is proxying and whether Google is serving you Youtube, Gmail, or Google Search content.
For an independently-hosted myhumanrightsblog.net, that's not as easy. They'd need another domain reachable on that server to set up the ECH connection to hide the risky TLD. Clients being snooped on still get specific domains logged.
IP certificates work around that issue by validating the security of the underlying connection rather than any specific hostname. Any server could be serving any hostname over an IP-address-validated connection. For snooped-on clients, the IP address is already part of the network traffic anyway, but no domains ever hit the traffic logs at all.
But then your underlying issue is that you're microhosting and can't hide behind a large cloud provider's domain front, so isn't that inherent to anything you might do?
In other words, blocking solutions that know your small blog is hosted exclusively on 1.2.3.4, without any collateral damage to other blogs the blocking government cares about will just block your IP.
Conversely, if you're hosting importedgoodsecommercesitegovernmentofficialslove.com next to myhumanrightsblog.net on the same IP, ECH is for you and solves your problem: Just register mycoolagnostichosting.net and do ECH to that.
"Just buy a second domain exclusively to work around the arbitrary restrictions put onto the protocol" works as a solution, but it's a silly solution that shouldn't be necessary.
ECH doesn't benefit you if you're connecting directly to one IP. Middleboxes can track that you're connecting to this IP.
ECH prevents tracking through routing layers where your ClientHello might contain foo.example.com or bar.example.com but route via the same IP (Cloudflare). A middlebox can see you are using a cloudflare hosted website, but not know what cloudflare website.
There's no benefit encrypting the SNI with 10.20.30.40 if they can see you're connecting to 10 20.30.40 anyway
THe benefit is that the SNI is not being logged. Resolving an IP to a domain name is pretty hard for a small actor who doesn't have a record of all DNS records.
It's an interesting feature, but it's pushing my buttons lately. Specifically on Cloudflare it is on by default, and on the free tier you can't disable it and you need a business plan for it. Which I think is stupid, but never the less it's causing us problems. We are trying some split-dns setup for a company "intranet", and if the site's have be accessed before, the ECH is remembered. So the browser tries and it eventually fails with ECH Error or on Firefox it just hangs like it's loading all the time. And it's so frustrating, because sometimes it works, sometimes it doesn't, you can clear cache and stuff and it still won't work, sometimes it works in Incognito sometimes it doesn't. This is not a real problem, but since we haven't fully switched to the "intranet" and we use some of the WAF features of Cloudflare sometimes it is so frustrating.
Split dns is such an ugly hack, it causes no end of hard to debug problems.
It is yeah, but it is a temporary solution while we are working out the setup.
If only we had some technology that would relieve us of the need to share IP addresses among multiple servers, this might have been unnecessary.
Will this have an impact on Loadbalancers? Like does one have to do client side load balancing like in gRPC?
If your load balanced doesn't support ECH, don't tell clients to use ECH.
My understanding is that you can use split mode to only have the load balancer decrypt the server name section, and forward the actual session and key exchange down to the backend without doing double layer encryption.
The loadbalancer can force a downgrade .
If the load balancer can force a downgrade, an attacker can do it as well.
Only if the attacker has a valid certificate for the domain to complete the handshake with.
Relying on HTTPS and SVCB records will probably allow a downgrade for some attackers, but if browsers roll out something akin to the HSTS preload list, then downgrade attacks become pretty difficult.
DNSSEC can also protect against malicious SVCB/HTTPS records and the spec recommends DoT/DoH against local MitM attacks to prevent this.
> but if browsers roll out something akin to the HSTS preload list, then downgrade attacks become pretty difficult.
Can you explain why, considering it is at the client's side ("browsers")?
If browsers remember which domains do ECH and refuse to downgrade to non-ECH connections after, the way the HSTS cache forces browsers to connect over HTTPS despite direct attempts to load over HTTP, then you only need an entry in the browser database to make downgrade attacks to accomplish SNI-snooping impossible.
For HSTS, browsers come with a preloaded list of known-HTTPS domains that requests are matched against. That means they will never connect over HTTP, rather than connect over HTTP and upgrade+maintain a cache when the HSTS header is present. If ECH comes with a preload list, then browsers connecting to ECH domains will simply fail to connect rather than permit the network to downgrade their connection to non-ECH TLS.