Much has already been written on how moving to centralised DNS is bad for our privacy in 2019, and on that basis alone centralizing our DNS on a few large cloud providers seems like a bad idea.
In this post, I want to look at the business and commercial consequences of moving DNS from the Internet Service Provider to a centralised place in the cloud, paying special attention to network neutrality, (anti-)competitive & regulatory aspects.
I hope that afterwards, it will be clear that when service providers argue against DoH, this does not have to mean they were spying on their users and hope to continue doing so – there are other major problems as well.
The lay of the land
As of 2019, the internet roughly looks like this:
This is sampling of the big guns of content distribution. Most of these are reached directly from the ISP, with some content providers hosting their servers within the network service providers. The biggest Content Distribution Networks (CDNs) shift so much data it even makes sense to have regional caches spread out throughout an ISPs service area.
In this layout, the ISP is completely in charge of distributing traffic. If it does a bad job, it will make its customers unhappy. If an ISP decides to prioritize one content provider over another, this is called a network neutrality violation, and various countries and regions (including the EU) have regulated the networking industry to outlaw this practice. Despite this fact, ISPs can sometimes wield significant power and for this reason they are under constant regulatory scrutiny.
Note that some countries have an underdeveloped ISP market, with large fractions of the population having no choice of broadband service provider. Regulation is then of the utmost importance to keep everyone honest, but in some of these countries the regulator has been captured by industry and is no longer very effective. This mostly goes for the US.
Technical details
Gaining access to content is a two-step process. Users, Apps and browsers almost exclusively connect to domain names (like ‘apple.com’) to retrieve content or perform actions. Such domain names can not be accessed directly on the internet because devices and servers talk to each other using IP addresses. DNS is used to find an IP address associated with a domain name, and then a connection can be made. Currently this mostly looks like this:
First (1) a device (computer, phone, tablet, tv, set-top box, streaming device) requests the IP address for ‘server1.apple.com’ from the ISP DNS server. This server either has the answer already (likely), or (2) it will talk to the CDN DNS server, which then (3) responds with the best IP address for the request, which is then (4) relayed to the original client. In the final step (5), the client device sets up a connection to that IP address.
Of note is that steps 1 through 4 are essentially spent “waiting”. If this process is slow, ISP subscribers experience bad performance and the internet feels sluggish.
Also noteworthy is that in step ‘2’, the customer’s network number (AS) is shared with the CDN. This can allow the CDN to pick the “best IP address” based on where the user is connected to the network, so content can be served to them from a well-placed cache.
In this (the existing) configuration, ISPs and CDNs have very well aligned incentives – providing end-users with rapid and snappy access to content.
The brave new world of centralised DNS over HTTPS (DoH)
Centralised DoH is where browsers, operating systems, phones, tables or computers no longer send their DNS lookups to the network-provided (ISP) DNS server, but transmit the query to a server hosted by a third party (in this case, the first party is the customer, the second party is the ISP).
The narrative behind centralised DoH is that regular DNS is unencrypted. In addition, Internet service providers are presumed to be profiling their customers and selling their browsing behaviour, and DoH is claimed to stop this (although it doesn’t). DoH operators vow (with differing specificity) not to sell customer data. They will however keep 24 hour logs of all queries for analysis, for some reason.
So far three companies have been entertaining the idea of centralised DoH, Google, Mozilla (Firefox) and Cloudflare. Google has recently decided their browsers and phones will not use centralised DoH for now, but they are however doing it for their Google Home Wifi products.
Cloudflare is pushing heavily for the world to centralise DNS on Cloudflare. While their CEO tweets from time to time that he’d be happiest if other people also offered DoH, they are expending significant lobbying efforts in convincing (some) browser vendors, governments and regulators that it is a good idea to move DNS from regulated network providers to Cloudflare.
Specifically, in the US, these efforts have been successful, with Mozilla deciding all Firefox DNS traffic should be sent to Cloudflare by default. Firefox users there receive a notification about the move, but do not have to opt-in. If they want to go back to their network provided DNS, they have to click a scary button called “Disable Protection”:
Flow of control with Centralised DoH
Let’s say a Firefox user in the US wants to visit some Akamai hosted content. With centralised DoH, the DNS lookup bypasses the local ISP DNS and instead goes to a Cloudflare server. This server may have to in turn ask the Akamai nameserver for the IP address, and once this is returned to the user, the actual connection to Akamai can be established, providing access to the content.
We have to keep in mind that if a DNS lookup is slow, the entire internet feels sluggish. Slow DNS = Slow internet. In this new scenario, Cloudflare, an Akamai competitor, is responsible for making Akamai service snappy. In addition, for this to work, connectivity from the ISP to Cloudflare needs to be perfect, and the same goes for the connection between Akamai and Cloudflare – companies who previously did not exchange a lot of data, nor had much of an interest in doing so.
In addition, where previously CDN operators could provide optimized DNS answers, because they could see where the query was coming from, Cloudflare has vowed not to provide such details to CDNs, ostensibly for privacy reasons. A CDN nameserver will henceforth only see that a query came from “Cloudflare”, and no longer from which ISP. This leads to sub-optimal routing, which I have personally experienced as “dog slow internet” when trying to access Akamai-hosted content through Cloudflare DNS.
Cloudflare, and connectivity to Cloudflare, now determine how quickly sites load to such an extent that we can well change our initial ‘Internet lay of the land’ diagram to this:
Every website visit, every lookup of every domain name now passes through Cloudflare. If Cloudflare has a bad day, the internet has a bad day. If Cloudflare and the ISP have a mutual network issue, instead of this only impacting Cloudflare, it now impacts all sites a subscriber would like to visit.
In addition, because of the flow of packets, not only does the ISP need to have top-notch perfect connectivity to Cloudflare, from now on, so must EVERY content provider in the world – the moment there is any congestion on the link, lookups slow down, and with that access to all content from that CDN.
Of special note is that regular ISPs are highly regulated precisely because they are in such a crucial position. Meanwhile, in its new position, Cloudflare has become critical internet infrastructure, but has somehow completely evaded regulation.
Why this is problematic
Within Cloudflare, there is no department called “Keeping Competitors’ Services Snappy”. In fact, Cloudflare lists many of the content providers above (and their suppliers) as outright competitors in their S-1 filing with the SEC:
Whenever ISPs have complained about Cloudflare inserting itself in the lookup chain, this has been framed as providers whining about no longer being able to violate their customers’ privacy. But for example in Europe where ISPs are not in the business of selling their user data, this rings hollow.
The real problem is that an unregulated entity is attempting to take over highly regulated services while gaining significant market power over both ISPs and content providers.
The nature of ISPs is comparable to that of utilities and it is therefore proper to regulate them as such. It is hugely problematic if some of their indeed considerable market power is then usurped by a new third party that has managed to completely escape regulation.
Why are Cloudflare and others pushing for centralised DoH?
This is indeed somewhat of a mystery. Like many websites that claim to care about our privacy before stuffing our browsers with cookies and trackers, Cloudflare (and Google and Mozilla) tell us they are in it to improve our privacy. Only one of these three is actually a non-profit though. It is pretty hard to see Google or Cloudflare as publicly traded charities heavily invested in improving our privacy. Mozilla is a very credible privacy advocate (even if I disagree with how they want to improve my privacy).
When questioned, Cloudflare states they are doing it because 1) it does not cost them real money and 2) users of the Cloudflare DoH service get slightly faster access to Cloudflare-hosted content.
The first bit could technically be true, although providing high speed encrypted DNS service does cost tons of CPU cycles. It appears however that Cloudflare is spending serious time lobbying governments in Europe and the US to get them behind centralised DoH – and unless there is a new pro-bono trend in lobbying I am not aware of, such efforts cost real money.
The second part is also interesting and somewhat revealing. If the impetus to centralise DoH on Cloudflare is indeed to speed up Cloudflare services relative to competitors, that is a clear network neutrality violation. It has also been claimed that this effect is in fact tiny, but if so, there is no good faith explanation left anymore why the company is attempting to centralise the internet on itself.
In the absence of good explanations the mind wanders to bad explanations. A crucial fact is that some CDNs that compete with Cloudflare face immediate challenges if DNS moves away from the ISP – CDNs will lose sight of where DNS queries are actually coming from, leading streaming video to (initially) be served from potentially sub-optimal locations.
What could be done
From a European perspective, it is quite clear that any centralised DoH provider that manages to become the new default for lookups is, in fact, also a telecommunications service provider. With this comes all the fun of the NIS directive and the full force of the EU telecommunications framework directives. Governments here would do well to recognize this fact and regulate accordingly.
Meanwhile, Mozilla has negotiated a privacy contract with Cloudflare for the DoH services, and we can find the promises in that contract here. There is no trace of network neutrality in there, nor is there a commitment from Cloudflare to actively work on establishing top-notch service to relevant content delivery networks. Life would be a lot better if Mozilla required such commitments from Cloudflare.
If regulated as such, centralised DNS over HTTPS could be made more palatable – but it might also make running a DoH service for free unattractive enough that operators will no longer bother.
Summarising
Centralised DNS over HTTPS is pushed to keep ISP’s presumed prying eyes away from our DNS traffic, and grant such access to other parties like Cloudflare that then promise not do anything bad with our data. There are good reasons to assume centralised DoH is bad for privacy.
In addition, by moving crucial telecommunication network functionality from the regulated ISP to unregulated cloud providers, there is significant risk of network neutrality violations. This is because the centralised DNS over HTTPs provider is now in charge of providing snappy service, including to its documented competitors. All this without regulation.
Governments should recognize centralised DoH operators that take over DNS lookups by default for what they are: providers that need to be regulated because of their systemic position. And finally, it would behoove Mozilla (who are strident fighters for a free and open internet) to make sure their contract with Cloudflare includes provisions that make sure all CDNs are equally well served by their chosen DoH providers.
