This weekend at the excellent FOSDEM gathering there were no less than three presentations on DNS over HTTPs. Daniel Stenberg presented a keynote session “DNS over HTTPS – the good, the bad and the ugly” (video), Vittorio Bertola discussed “The DoH Dilemma” while Daniel, Stéphane Bortzmeyer and I formed a DNS Privacy Panel expertly moderated by Jan-Piet Mens. I want to thank Daniel, Jan-Piet, Rudolf van der Berg, Stéphane & Vittorio for proofreading & improving this post, but I should add this does not imply an endorsement from anyone!
In what follows, I will attempt to give a neutral description of what I think we learned, and where we now are on DoH, with a focus on the European perspective. If you find a noticeable bias, please let me know urgently and I’ll address it. But to be clear, I’m no fan of centralizing DNS on a small number of cloud providers.
After the neutral description you will find some strong opinions on if “DNS over Cloud” is a good thing or not.
Daniel Stenberg’s interpretation of what worries people about DoH
Words & definitions
During the FOSDEM presentations, various visions on the desirability of DNS over HTTPS were discussed. We were sadly rather hampered by messy definitions. There are two definitions that sound the same but are different in practice. Firstly, there is “DNS over HTTPS” (DoH) which is a transport protocol so you can securely ask DNS queries over HTTPS.
Secondly, Google, Firefox and Cloudflare are working on using DoH to move DNS queries from the network service provider straight onto the cloud. In other words, where previously your service provider could see (and answer) your DNS queries, in this proposed future you would send your DNS requests to a “free-as-in-beer” cloud provider.
As Daniel pointed out well during his keynote, both of these things have been called DoH, which is highly confusing. “The Resistance” as Daniel labels it complains about “DoH” when in fact they are mostly complaining about centralizing DNS on cloud providers. We should not blame the protocol for what operators might do with it.
It may be that the greatest benefit we get out of the hours of FOSDEM DoH presentations is that we now know we should separate our concerns with DoH (the protocol) from our concerns about the application of this protocol to deliver what I propose we henceforth call DNS over Cloud (DoC).
DNS over HTTPs (the protocol, DoH)
The DoH protocol is designed to use the HTTP and TLS infrastructure to deliver encrypted and authenticated DNS answers that (crucially) are hard to block by network operators. An earlier protocol called DNS over TLS was already available but since it runs on port 853 and “does not look like HTTPS”, network operators that dislike DoT can easily block it. Most corporate networks will in fact do this by default.
DoH shares the benefits and downsides of HTTPS. It can send out more trackable data than regular DNS, simply because HTTP supports things like headers & cookies. TLS session resumption functions as another tracking mechanism. On the plus side, anything that can cache or redistribute HTTPS can now also be used to improve or proxy DNS. Also, DNS over HTTPS makes it possible to push DNS answers even before they are asked, which could increase page load performance.
It may be seen as good or bad that HTTPS can be made undetectable and unblockable, depending on who you are and what you worry about. If Google were to colocate a DNS over HTTPS service on the IP address also used for ‘Google.com’, countries and network operators would face a Solomon’s choice if they wanted to block DoH: give up Google searches or keep DoH alive.
Update: Google turns out to do exactly this, you can get DNS answers over an https://google.com/ request.
Network operators that feel they should be in control of their network will not like this standoff, while users that think their network operators should have no power over them will rejoice. For the second group, at FOSDEM we discussed the proverbial Turkish dissident that would benefit from unblockable DNS.
Finally, because DoH uses authenticated HTTPS (just when like visiting any website), we know we are talking to the nameserver we want to talk to. It protects against rogue nameservers, possibly injected by hijacking the DHCP request, or simply by spoofing IP packets.
DNS over Cloud (DoC)
As it stands, network operators (ISPs, service providers, your WiFi providing coffee shop) can see your DNS traffic. In addition they could (and actually often do) manipulate or block certain queries or responses. This is an intrinsic property of providing DNS service – everyone that provides DNS service to you can do these things, cloud based or not.
One concrete difference between typical network DNS and DNS over Cloud is that network DNS tends to be unencrypted while DoC can encrypt the transport component. And encryption is good.
Much like using a VPN to access the internet only moves your traffic from one place to another, choosing a different DNS service does not magically make your DNS more secure. It does change who you want to trust though. And if you are lucky your trusted provider is more secure in ways that are relevant to you.
Currently, that trust is not very intentional. Internet users often have little choice in what ISP to use. In many cases they may not even know. While local regulations (like GDPR, NIS, ePrivacy and EU telecommunications directives) may limit what a provider is allowed to do, users may not be sure if their actual network operator adheres to these legislations.
DNS over Cloud proponents advocate that the user did however consciously choose a browser and that the browser is therefore in a good position to suggest or even pick a DNS provider for their users. Users sometimes also can’t pick a browser either, but they may have freedom to select a phone and different brands of phones include different browsers. Cheaper phones all ship with the same browser however.
During our DNS Privacy Panel it was also established that we estimate that most users do not care very much about their DNS privacy, and are in any case not well informed about the tradeoffs. The choice of DNS provider therefore needs to be made for them, either by their phone, their operating system or their browser.
A brief interlude on DNS encryption
Everyone agreed more encryption is good. This can happen between client equipment and the nameserver, or between the resolving nameserver and authoritative nameservers. Warren Kumari from Google tested the waters for our thoughts on opportunistic DNS over TLS (DoT) between phone/browser and DNS service, and this went down well, except that it was noted it is subject to downgrade attacks. I noted that PowerDNS has been stimulating its customers to enable DoT already because Android Pie will attempt to use your DoT service if it is there. There was also a brief discussion on the efforts to encrypt traffic between resolver and authoritative servers, something that is also good.
What does DoH protect against? Because we use HTTPS anyhow for protection?
At the very end of Daniel’s keynote a question was asked what the point is even of protecting DNS queries and responses. The DNS response leads to the setup of a TLS connection and this TLS connection is itself already encrypted and private. We don’t need DNS for that. In addition, a TLS connection setup will typically include the name of the site being visited in plaintext, even with TLS 1.3 (the Server Name Indication or SNI field). Finally, the IP address we eventually end up connecting to may give a very good indication who this connection is going to. So it is generally possible to tell where a TLS connection is going – even without looking at DNS. Stéphane’s RFC 7626 discusses many of these tradeoffs.
As of February 2019, there is little privacy differential when using DNS over HTTPS since the name still travels in plaintext. It may however be more expensive for a snooping network provider to extract the SNI from packets. Also, work is ongoing to use encrypted DNS to encrypt the SNI field too, in which case DNS over HTTPS would actually give us more additional privacy.
What DoH does however deliver today is protection against DNS-based censorship.
Censorship & things that break
The PowerDNS DoC service quickly gained thousands of users, many of whom are in Indonesia. PowerDNS learned that Indonesian ISPs perform a lot of blocking and DoH servers are a great way around such blocking. It may be that doh.powerdns.org is small enough to fly under the radar of the Indonesian censors.
Separately there was a brief discussion on how DoC can break things like VPNs and split horizon. We did not explore this much further except that it was noted it actually breaks things in production. An open question is if the encryption is worth the amount of breakage observed, and if we could maybe work around it.
Differences between Cloud and Network DNS Providers
The highly regulated nature of service providers, at least here in Europe, is a double edged sword. It restricts what ISPs can do with your data but it also means they respond to court orders that block content and may implement blocking of child pornography even without such orders. Internet users may not be happy with such blocking, either because they want easy access to Torrents, or simply because they object to the very principle of communications being blocked.
In addition, while (European) service providers are under legal obligation not to monetize or otherwise sell your traffic (without very explicit permission), that does not mean they don’t do it. Specifically, all service providers here will respond to government (bulk) interception orders, and provide police & spies with a full copy of all your traffic, including the unencrypted DNS parts.
Cloud providers meanwhile are very adept at navigating the GDPR waters and are able to simultaneously promise you they won’t sell your data but also power most of their bottom line selling advertising based on what you do online. In addition, they are relatively out of reach of government interception or blocking orders, which take many months to travel to a foreign jurisdiction, and frequently never arrive.
Differing views on the panel
We lucked out with three speakers with three informed but still different opinions on the subject.
I (Bert) make my living selling software and services to telecommunication service providers. I know many European ISPs intimately and I do not believe they are engaging in secret user profiling. We have enough trouble with GDPR as it is to get any kind of DNS debugging data out of our customers. So my belief is that while service providers may not be “a force for good”, I do predict they’d have a very hard time breaking regulations to secretly run a surveillance economy. But, these people pay my company good money so I am biased to like them. I do not believe it is a good idea however to send a record of every website I visit to cloud providers like Google or Cloudflare.
Stéphane meanwhile is highly knowledgeable on how governments actually regulate the Internet. He even wrote a book on it that is subtitled “The Internet – a political space”. In his opinion, GDPR and other regulations may be great, but enforcement is scarce as data protection agencies do not understand DNS and do not prioritize it. This leaves room for even European service providers to sell and monetize DNS data. In addition, Stéphane is worried that when governments DO finally get interested in DNS, it is for censorship purposes.
Daniel offers a perspective inspired by his background in HTTPS – he sees the obvious benefits of not only encrypting DNS data but also authenticating its server. “You know who you are talking to”. He furthermore observes correctly that users spend time on different networks, and that we can’t possibly expect them to study the privacy practices and reputation of every school or coffee shop where they use Wifi. If users picked a suitable DoH provider that worked over all networks, they’d receive a constant level of trust – no matter what network they are on.
Daniel has separately argued that he regards an explicit promise from a cloud provider not to sell your traffic as a stronger guarantee than passively trusting that a provider will stick to the applicable laws. Finally, Daniel notes correctly that GDPR does not protect you if you are connected to a rogue nameserver (so not the one you were expecting to use). It may not be the service provider that spies on you but someone else on the path TO that provider. DoH protects against that scenario.
Who gets to pick who we should trust?
If a browser decides to use DoC for its lookups, which provider should it offer? Early in the discussion it was noted that there should be a transparent process for deciding who could be offered as a provider, where it was also noted that this process for Firefox has been far from transparent or even operational so far. A member of the audience spotted an interesting analogy with the CA/Browser Forum which has been used to determine which certificate authorities are to be trusted. Daniel noted that this is however also similar to search engine selection in browsers “and everyone picks the default, and that is the one that pays most”.
Stéphane opined that there should be many DoC providers to choose from, but since picking one is hard, the browser should present a list with a random one at the top. This allows choice but also prevents needless concentration if a user picks the default.
Why are cloud companies so anxious to host our DNS?
Warren Kumari from Google gave a very clear response – a better and faster internet leads to more internet use and more searches and therefore more advertisements and thus more money for Google. Such honesty is rare & it is appreciated. Warren also reconfirmed (as happened at FOSDEM 2018) that 8.8.8.8 sticks to its privacy policy, it is not being mined. As an aside, when 8.8.8.8 was launched, the state of ISP DNS was indeed dire. DNS in many service providers did not not have a good home – fitting awkwardly between network and application departments. The creation of 8.8.8.8 contributed to the vastly improved DNS we see today, and at the time it was necessary.
But why 1.1.1.1 or 9.9.9.9? Christian Elmerot from the Cloudflare (1.1.1.1) resolver team offered the explanation that people on 1.1.1.1 will get (slightly) faster answers from 1.1.1.1 for Cloudflare domains than when using other resolvers, and this makes their services more attractive.
It may however be that public DoC providers are not entirely disinterested in getting a copy of the world’s browsing behaviour.
Is it any faster?
DoH (or more precisely, DoC provided by Cloudflare) is actually 7 milliseconds slower on average than the system resolver, according to measurements performed at Mozilla back when Daniel still worked there. He does note however that the worst case performance of the Cloudflare DoC is much better than the worst case system resolver performance.
Should you run your own resolver?
It may be slower, but Stéphane noted that having your own resolver on your own machine, is actually also not good for your privacy, since authoritative servers will now see your personal IP address, instead of the service provider IP address. However, it does offer full control – at a possible performance and privacy penalty. Stéphane notes that a mixed mode local resolver, that uses a DoH provider for cache misses, may be an optimum. Some further thoughts on the benefits of a local resolver can be found in this post “Benefits of DNS service locality” by Paul Vixie.
What about EDNS Client Subnet?
There was a brief and somewhat angry discussion between me and Daniel that somehow got cut from the end of the video recording. This discussion was about EDNS Client Subnet, and how it impacts your privacy when used by a service provider.
Some large scale internet service providers include part of a customer’s IP address when sending queries to (for example) Akamai or Level3. This is currently necessary because these large scale CDNs perform load balancing via DNS and they need to see 24 or sometimes even 25 bits of the IPv4 address to determine the right server for a user. This is sometimes reported as a privacy problem, and in the general case it could be. However, when used between a hosting provider like Akamai and a service provider, in reality there is no loss of privacy – the customer is attempting to connect to an Akamai service, and Akamai will always see the subscriber IP address in that case.
Noteworthy is that DoC-providers that do not implement EDNS Client Subnet (ECS) may disadvantage competing cloud operators since they will send internet users to sub-optimal content distribution nodes. It may not be wise to rely on one CDN to connect to a competing CDN.
The balance
This is where the attempt at impartiality in this post ends.
First, we should separate a few things: DoH, DoC and “DoC-by-default”. It seems clear that the first two of these are not problematic. It is good that we have a secure DNS transport mechanism and some DoC providers may truly be a step up in privacy and security for users in some countries or places.
Our discussion should be about what we think of “DoC-by-default”, that is, any attempt by browser vendors to default people into moving their DNS to Cloudflare or themselves. My concern also extends to a weaker form where you get DoC-by-nudge if you press a little ‘Got it’ button when prompted if you want to benefit from the ‘Google Secure Lookup Service’.
Who should we believe, the highly regulated (European) service provider that says it is not allowed to spy on its users using DNS, and also says they aren’t doing it?
Or should we believe the cloud provider that claims those service providers are spying on their users and then asks us for our DNS traffic while promising not to sell it, although they will log each and every query?
Is it credible for DNS over Cloud providers to spend huge efforts on pushing their DNS services but also to claim they won’t do anything impactful with the results? Google has advanced that faster DNS means more revenues for them, which is likely true, but DNS over HTTPS will first slow things down!
Cloudflare meanwhile opines that if you use their DoC-service, this makes accessing Cloudflare domains a tiny bit faster compared to visiting competitor’s domain names. While this may be true, first the effect is tiny and second, it is not that great a sell for an actual Internet user.
It is currently true though that your coffee shop WiFi may be spying on you, or may enable an attacker to do so. However, as noted above, the names of sites you visit are still sent out unencrypted by TLS connections these days, so DoC does not even deliver on saving you from such spying right now.
Finally, much is made of the users in repressive regimes which might benefit greatly from unblockable DNS, and this may indeed be so. But as noble as it is to help the Turkish dissident to communicate with the world, it seems odd that to help her we need to send Cloudflare or Google a record of every website visited by 500 million Europeans!
Other arguments for DoC-by-default may be more appealing – if you are a cloud provider. Despite all promises to not sell the DNS data, not log it very long etc, the fact remains that a DoC operator gets sent a copy of every server and site name a user visits. Somehow someday that data is going to be monetized, and this will happen in ways users will not be consulted about.
This leaves us with other explanations for the DoH push, and none of these are very good. For one, it is plain and simple an attempt to fully control the Internet experience. As an example, there have been ISPs that have pondered adding ad-filtering as a network service. This does not sit well with advertising companies like Google, so they’d love to be sure such blocking is not possible. DoC-by-default gives that to them.
We’ve previously established that users do not have strong or informed opinions on the source of their DNS, so whatever happens will be decided by browser vendors, on behalf of internet users.
Given what we now know about the relative risks and benefits of DoC, it seems utterly unwarranted to decide that users should give their DNS to Google or Cloudflare because there is no credible claim it will actually improve their
lives.
Thanks for making it to the end of this long post!
- Some more thoughts on DoH & specifically the Firefox plans can be found here.
- Open-Xchange and several other groups are actively informing governments and data protection authorities about what is going on & why we feel DoC-by-default is a bad thing. Please contact vittorio.bertola@open-xchange.com if you are interested in joining in!
