PowerDNS Recursor 4.1.7 Released

Today we have released the PowerDNS Recursor 4.1.7. It is an update to relax EDNS compliance requirements from upstream authoritative servers.

Recursor version 4.1.5 (and, by extension, 4.1.6), contains a fix for Security Advisory 2018-07. One part of that fix is a stricter fallback to non-EDNS queries when EDNS queries fail. It turns out that there are several authoritative servers on the Internet that have such bad EDNS handling, that the domains hosted on them stop resolving with 4.1.5. The 4.1.7 release has relaxed the EDNS compliance requirement and includes an alternative fix for 2018-07.

Since reports of this started coming in yesterday, some domains have been fixed by their owners, but a long tail of broken zones remains for now.

We have decided to release this increase in strictness in the PowerDNS Recursor 4.2.0, so that domain owners can work on their server’s compliance. We urge operators of authoritative servers to check their domains and servers with the EDNS compliance tool and act upon its results. Increased EDNS compliance strictness will be added to many DNS resolvers coming next February.

The changelog is as follows:

  • #7172: Revert ‘Keep the EDNS status of a server on FormErr with EDNS’
  • #7174: Refuse queries for all meta-types

As always, the tarball(sig) can be found on the downloads website and packages for CentOS 6 and 7, Ubuntu Trusty, Xenial and Bionic and Debian Jessie and Stretch can be found on repo.powerdns.com.

dnsdist 1.3.3 released

We are very happy to announce the 1.3.3 release of dnsdist. This release contains a few new features, but is mostly fixing a security issue reported since the release of dnsdist 1.3.2.

Security fix

While working on a new feature, Richard Gibson noticed that it was possible for a remote attacker to craft a DNS query with trailing data such that the addition of a record by dnsdist, for example an OPT record when adding EDNS Client Subnet, might result in the trailing data being smuggled to the backend as a valid record while not seen by dnsdist. This is an issue when dnsdist is deployed as a DNS Firewall and used to filter some records that should not be received by the backend. This issue occurs only when either the ‘useClientSubnet’ or the experimental ‘addXPF’ parameters are used when declaring a new backend.

While dnsdist has not had any important security issue until now, we decided this was a good time to implement the same security polling mechanism that the authoritative server and the recursor have had for years. Starting with this release, dnsdist will regularly perform a security check using a DNS query to determine whether the current version is up-to-date security-wise, and let the administrator know otherwise.

Important changes

It is sometimes very useful to be able to generate answers directly from dnsdist, to quickly return a “No such domain” answer, spoof an “A” or “AAAA” answer, or even just reply with the TC bit set so that legitimate clients retry over TCP. Until now, answers generated that way were mirroring the flags and EDNS options, if any, of the initial query. This was not great because it could mislead the client into thinking that dnsdist, or the server behind it, was supporting features or a UDP payload size it did not.

Starting with this release, dnsdist is now generating a proper EDNS payload if the query had one, and responding without EDNS otherwise. This behavior can be turned off using the new setAddEDNSToSelfGeneratedResponses() directive if needed.

We must, however, provide a responder’s maximum payload size in this record, and we can’t easily know the maximum payload size of the actual backend so we picked a safe default value of 1500, which can be overridden using the new  setPayloadSizeOnSelfGeneratedAnswers() directive.

New features and improvements

A new load-balancing policy named “chashed” has been introduced, based on consistent hashing. This new policy load-balances the incoming queries based on a hash of the requested name, like the existing “whashed” one, but has the interesting property that adding or removing a server will only cause a very small portion of the incoming queries to be mapped to a different server than they were before, keeping the caches warm.

While we have been supporting the export of metrics using the well-known carbon protocol from day one, we have seen an increasing demand for supporting the emerging Prometheus protocol. Thanks to the work of Pavel Odintsov and Kai S, dnsdist now supports it natively.

Very large installations of the DNS over TLS feature introduced in 1.3.0 reported several issues that we addressed in this release:

  • dnsdist did not set TCP_NODELAY on its TLS sockets, causing needless latency ;
  • it was not possible to configure the number of stored TLS sessions ;
  • our OpenSSL implementation had a memory leak when some clients aborted prematurely because of a negotiation error during the TLS handshake.

We seized the opportunity to refactor the part of the code handling TLS connections with the use of smart pointers while fixing that last issue, making sure that this kind of memory leak will not happen again.

In 1.3.2, the optimized DynblockRulesGroup introduced in 1.3.0 gained the ability to whitelist and blacklist ranges from dynamic rules, for example to prevent some clients from ever being blocked by a rate-limiting rule. This feature has now been made available when our in-kernel eBPF filtering feature is used as well. At the same time, we introduced the ability to set up warning rates to the dynamic rules, making it possible to get an alert without blocking clients when they reach a configured rate, and to block them should they reach a higher rate.

Finally, we introduced several new rules to our existing set:

  • EDNSOptionRule, to be able to filter based on the presence of a given EDNS option ;
  • DSTPortRule, offering the ability to route queries by looking at their destination port ;
  • PoolAvailableRule, to be able to route queries based on whether a pool has at least one usable backend.

Please see the dnsdist website for the more complete changelog and the current documentation.

Release tarballs are available on the downloads website.

Several packages are also available on our repository. Please be aware that we have enabled a few additional features in our packages, like DNS over TLS and DNSTap support, on distributions where the required dependencies were available.

PowerDNS Recursor 4.1.6 Released

We have just released the PowerDNS Recursor 4.1.6. This release fixes an issue with DNSSEC validation introduced in Recursor 4.1.5 by reverting a code-change related to the acceptation of ADDITIONAL records. We will investigate this issue in detail the coming days, but found it necessary to issue an update with a fix early.

The full changelog is very short:

  • Revert “rec: Authority records in AA=1 CNAME answer are authoritative”. References: #7158, pull request 7159

The tarball(signature) is available at downloads.powerdns.com and packages for CentOS 6 and 7, Debian Jessie and Stretch, Ubuntu Bionic, Trusty and Xenial are available from repo.powerdns.com.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

PowerDNS Authoritative Server 4.0.6 & 4.1.5 and Recursor 4.0.9 & 4.1.5 Released

We’ve released PowerDNS Authoritative Server 4.0.6 & 4.1.5 and Recursor 4.0.9 & 4.1.5.

These are security releases with additional minor improvements and bug fixes.

Minimal patches for the releases are available at https://downloads.powerdns.com/patches/.

The changelogs look as follows:

Authoritative Server 4.1.5

This release fixes the following security advisories:

  • PowerDNS Security Advisory 2018-03 (CVE-2018-10851)
  • PowerDNS Security Advisory 2018-05 (CVE-2018-14626)

Improvements

  • Apply alias scopemask after chasing (#6976)
  • Release memory in case of error in the openssl ecdsa constructor (#6917)
  • Switch to devtoolset 7 for el6 (#7118, #7040)

Bug Fixes

  • Crafted zone record can cause a denial of service (CVE-2018-10851, #7149)
  • Packet cache pollution via crafted query (CVE-2018-14626, #7149)
  • Fix compilation with libressl 2.7.0+ (#6948, #6943)
  • Actually truncate truncated responses (#6913)

Authoritative Server 4.0.6

This release fixes PowerDNS Security Advisory 2018-03 (CVE-2018-10851).

Bug fixes

  • Crafted zone record can cause a denial of service (CVE-2018-10851, #7150)
  • Skip v6-dependent test when pdns_test_no_ipv6 is set in environment (#6013)
  • Fix el6 builds (#7135)

Improvements

  • Prevent cname + other data with dnsupdate (#6315)
  • Switch to devtoolset 7 for el6 (#7119)

Recursor 4.1.5

This release fixes the following security advisories:

  • PowerDNS Security Advisory 2018-04 (CVE-2018-10851)
  • PowerDNS Security Advisory 2018-06 (CVE-2018-14626)
  • PowerDNS Security Advisory 2018-07 (CVE-2018-14644)

Improvements

  • Add pdnslog to lua configuration scripts (Chris Hofstaedtler) (#6919, #6848)
  • Fix compilation with libressl 2.7.0+ (#6948, #6943)
  • Export outgoing ECS value and server ID in protobuf (if any) (#7004, #6991, #6989)
  • Switch to devtoolset 7 for el6 (#7122, #7040)
  • Allow the signature inception to be off by a number of seconds (Kees Monshouwer) (#7125, #7081)

Bug Fixes

  • Crafted answer can cause a denial of service (CVE-2018-10851, #7151)
  • Packet cache pollution via crafted query (CVE-2018-14626, #7151)
  • Crafted query for meta-types can cause a denial of service (CVE-2018-14644, #7151)
  • Delay the creation of rpz threads until we have dropped privileges (#6984 #6792)
  • Cleanup the netmask trees used for the ecs index on removals (#6961 #6960)
  • Make sure that the ecs scope from the auth is < to the source (#6963, #6605)
  • Authority records in aa=1 cname answer are authoritative (#6980, #6979)
  • Avoid a memory leak in catch-all exception handler (#7073)
  • Don’t require authoritative answers for forward-recurse zones (#6741, #6340)
  • Release memory in case of error in the openssl ecdsa constructor (#6917)
  • Convert a few uses to toLogString to print DNSName’s that may be empty in a safer manner (#6925, #6924)
  • Avoid a crash on DEC Alpha systems (#6945)
  • Clear all caches on (N)TA changes (#6951, #6949)

Recursor 4.0.9

This release fixes the following security advisories:

  • PowerDNS Security Advisory 2018-04 (CVE-2018-10851)
  • PowerDNS Security Advisory 2018-06 (CVE-2018-14626)
  • PowerDNS Security Advisory 2018-07 (CVE-2018-14644)

Bug fixes

  • Crafted answer can cause a denial of service (CVE-2018-10851, #7152)
  • Packet cache pollution via crafted query (CVE-2018-14626, #7152)
  • Crafted query for meta-types can cause a denial of service (CVE-2018-14644, #7152)

The tarballs and signatures are available at downloads.powerdns.com and packages for CentOS 6 and 7, Debian Jessie and Stretch, Ubuntu Bionic, Trusty and Xenial are available from repo.powerdns.com.  Raspberry PI packages will follow tomorrow.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

FOSDEM 2019 DNS devroom CfP

Hello DNS enthusiasts and other developers,

After a successful and packed half-day DNS devroom at FOSDEM 2018, we are happy to announce a full-day DNS devroom at FOSDEM 2019.

As with last year, we hope to host talks anywhere from hardcore protocol stuff, to practical sessions for programmers that are not directly involved with DNS but may have to deal with DNS in their day to day coding or system administrators responsible for DNS infrastructure.

We have been allotted a room on Sunday 3 February 2019. We expect to schedule 30 minutes per talk, including questions, but if you need more or less time, we can discuss this.

If you have something you’d like to share with your fellow developers, please head to pentabarf at https://penta.fosdem.org/submission/FOSDEM19. Examples of topics are measuring, monitoring, DNS libraries, and anecdotes on how you’ve (ab)used the DNS. Here’s the 2018 schedule, for your inspiration: https://archive.fosdem.org/2018/schedule/track/dns/ .

The deadline for submission is December 1st. If you have a FOSDEM pentabarf account from a previous year, please use that account. Reach out to dns-devroom-manager@fosdem.org if you run into any trouble.

See you there!

Cheers,
Peter van Dijk, Shane Kerr, Pieter Lexis, and Kees Monshouwer

PowerDNS and the ICANN KSK roll

The root KSK rollover is currently planned for 1600 UTC on the 11th of October 2018 – a few days from now. If you are using PowerDNS Recursor for DNSSEC validation, please keep reading!

During the KSK rollover, the root zone will stop using the old root Key Signing Key, known as KSK-2010 or 19036, and will start using the new Key Signing Key, known as KSK-2017 or 20326. Your Recursor needs to be aware of both keys to make sure validation keeps working after the rollover event.

If you are running Recursor 4.0.5 or up, both keys come preconfigured. If you are running an older 4.0.x version, it is possible your distribution has added the key for you.

In case of any doubt, verify you are ready:

# rec_control --socket-dir=. get-tas
Configured Trust Anchors:
.
19036 8 2 49aac11d7b6f6446702e54a1607371607a1a41855200fd2ce1cdde32f24e8fb5
20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d

The output should have both the 19036 and 20326 lines. If 20326 is missing, please upgrade your Recursor. If for some reason upgrading is not feasible for you right now, please follow the PowerDNS Recursor instructions that ICANN published. Those instructions involve a restart; if you want to avoid a restart this week, please see Runtime Configuration of Trust Anchors in the PowerDNS documentation.

In case of panic (in the unlikely event ICANN botches the roll, or the roll finds a bug in our software), you can run rec_control add-nta . DNSSEC on root is broken to disable DNSSEC immediately without restarting your daemon.

Should you have any trouble: if you are a supported customer, please reach out through the usual channels. Otherwise, contact us via our community channels.

Spoofing DNS with fragments

With some care, it turns out to be possible to spoof fake DNS responses using fragmented datagrams. While preparing a presentation for XS4ALL back in 2009, I found out how this could be done, but I never got round to formally publishing the technique. The presentation was however made available.

Update: this “discovery” has now been dated back to at least 2008 when Florian Weimer knew about it & tells us it was communicated clearly and widely back then.

In 2013, Amir Herzberg & Haya Shulman (while at Bar Ilan University) published a paper called Fragmentation Considered Poisonous. In this paper they explain how fragmented DNS responses can be used for cache poisoning. Later that year CZNIC presented about this paper and its techniques at RIPE 67.

A stunning 72 papers cite the original article, but as of 2018 not too many people know about this cache poisoning method.

More recently, The Register reported that another team, also involving Dr Shulman (now at Fraunhofer Institute for Secure Information Technology), has been able to use fragmented DNS responses to acquire certificates for domain names whose nameservers they do not control. They were able to demonstrate this in real life, which is a remarkable achievement. Incidentally, this team includes Amit Klein who in 2008 discovered & reported a weakness in PowerDNS.

Full details will be presented at the ACM Conference on Computer and Communications Security in Toronto, October 18. This presentation will also propose countermeasures.

Meanwhile, in this post, I hope to explain a (likely) part of their technique.

Whole datagram DNS spoofing

To match bona fide DNS responses to their corresponding queries, resolvers and operating system check:

  • Name of the query
  • Type of the query
  • Source/destination address
  • Destination port (16 bits)
  • DNS transaction ID (16 bits)

The first three items can be predictable, the last two aren’t supposed to be. To spoof in a false response therefore means we need to guess 32 bits of random. To do so, the attacker needs to send the resolver lots and lots of fake answers with guesses for destination port and the transaction ID. Over (prolonged) time, their chosen response arrives ahead of the authentic response, is accepted, and they are able to spoof a domain name. Profit.

Screenshot from 2018-09-10 22-12-57

In practice this turns out to be very hard to do. The 32 bit requirement plus the short timeframe in which to send false responses means that as far as I know, this has been demonstrated in a lab setting just once. Anecdotal reports of blindly spoofing a fully randomized source port resolver have not been substantiated.

Fragments

DNS queries and responses can be carried in UDP datagrams. A UDP datagram can be many kilobytes in size – far larger than most UDP packets. This means that a sufficiently large UDP response datagram can get split up into multiple packets. These are then called fragments.

Such fragments travel the network separately, to be joined together again on receipt.

Fragmented DNS responses happen occasionally with DNSSEC, for example in this case:

$ dig -t mx  isc.org @ams.sns-pb.isc.org +dnssec -4 +bufsize=16000
43.028963 IP 192.168.1.228.44751 > 199.6.1.30.53: 20903+ [1au] MX? isc.org. (48)
43.035379 IP 199.6.1.30.53 > 192.168.1.228.44751: 20903*- 3/5/21 
  MX mx.ams1.isc.org. 20, MX mx.pao1.isc.org. 10, RRSIG (1472)
43.035391 IP 199.6.1.30 > 192.168.1.228: ip-proto-17

The final line represents a fragment, which only notes it is UDP (protocol 17).

Matching fragments together is quite comparable to matching DNS queries to responses. Every IP packet, even a fragment, carries a 16 bit number called an IPID. This IPID is not copied from the query to the response, it is picked by the DNS responder.

Screenshot from 2018-09-10 22-13-30

On receipt, fragments are grouped by IPID, after which the checksum of the reassembled datagram is checked. If correct, the DNS response gets forwarded to the resolver process.

If we want to spoof a DNS response, we could pick a DNS query that leads to a fragmented datagram, and then try to spoof only the second fragment. On first sight, this does not appear to be much easier as we now need to guess the IPID (16 bits) and we also need to make sure the checksum of the whole datagram matches (another 16 bits). This then also requires a 32 bit guess to succeed.

However, if we send a server a DNS query, it will most of the time send the same DNS response to everyone who asks (also for fragmented answers). In other words, if the attacker wants to spoof a certain response, it will know exactly what that response looks like – with the exception of the destination port and the DNS transaction ID (32 bits).

But note that both of these unpredictable parts are in the first fragment. The second fragment is completely static, except for the IPID. Now for the clever bit.

The ‘internet checksum’ is literally .. a sum. So the checksum of the entire datagram consists of the checksum of the first fragment plus the checksum of the second fragment (modulo 16 bits).

Screenshot from 2018-09-10 22-13-49

This means that to make sure the whole reassembled datagram passes the checksum test, all we have to do is make sure that our fake second fragment has the same known partial checksum as the original. We can pick the checksum of our fake second segment easily through the TTL of the our chosen response record.

This leaves us with only 16 bits to guess, which given the birthday paradox is not that hard.

 

Randomness of the IPID

So how random is the IPID, does it even represent a 16-bit challenge? According to the 2013 paper, some operating systems pick the IPID from a global counter. This means an attacker can learn the currently used IPID and predict the one used for the next response with pretty good accuracy.

Other operating systems use an IPID that increments per destination which means we can’t remotely guess the IPID. It turns out however that through clever use of multiple fragments, this still allows an attacker to “capture” one of these. See the original paper for details.

Is that it?

Definitely not. In order to get a certificate issued falsely using this technique requires several additional elements. First we must be able to force many questions. Secondly, we must make sure that the original authoritative server fragments the answer just right. There are ways to do both, but they are not easy.

I await the presentation at the ACM conference in October eagerly – but I’m pretty sure it will build on the technique outlined above.

Countermeasures

In the meantime, DNSSEC does actually protect against this vulnerability, but it does require that your domain is signed and that your CA validates. This may not yet be the case.