Changes in the PowerDNS Recursor 4.2.0

The 4.2.0 release of the PowerDNS Recursor brings a lot of small, incremental changes over the 4.1.x releases. We expect little operational impact when upgrading from 4.1.x. However, several new features have been implemented and some features have changed.

This release was made possible by contributions from: Gibheer, cclauss, Aki Tuomi, Ruben, Doug Freed, Richard Gibson, Peter Gervai, Oli, Josh Soref, Rens Houben, Kirill Ponomarev, Kees Monshouwer, Matt Nordhoff, OSSO B.V., phonedph1, Rafael Buchbinder, Ruben Kerkhof, spirillen, Tom Ivar Helbekkmo and Chris Hofstaedtler.  Thanks!

DNS Flag Day

The 4.2.0 release of the PowerDNS Recursor removes several workarounds for authoritative servers that respond badly to EDNS(0) queries. This is part of a multi-vendor effort known as DNS flag day to move the DNS ecosystem forward by being less lenient on non-conforming implementations.

XPF Support

This release adds support for DNS X-Proxied-For (draft-bellis-dnsop-xpf-04). This technique is roughly equivalent to HTTP’s X-Forwarded-For header, it can communicate the IP address and port of the original requestor from a loadbalancer/frontend (like dnsdist) to the backend server. This can allow the backend server to make decisions regarding that specific client. XPF is disabled by default and can be enabled by setting the xpf-allow-from setting to the source IP address of the front-end proxy and setting xpf-rr-code to the code of the resource record used by the frontend.

EDNS Client Subnet Improvements

More granularity has been added for the users of EDNS Client Subnet. The new ecs-add-for setting can be set to a list of netmasks for which the requestor’s IP address should be used as the EDNS Client Subnet for outgoing queries. For IP addresses not on this list, the PowerDNS Recursor will use the ecs-scope-zero-address instead, which matches the behavior of 4.1.x. Valid incoming ECS values from use-incoming-edns-subnet are not replaced.

New and Updated Settings

Sites that process large numbers of queries per second (100k+), may benefit from the new distributor-threads setting. This can be used in combination with pdns-distributes-queries=yes to spawn multiple threads that will pick up incoming queries and distribute them over the worker threads.

For several statistics, the PowerDNS Recursor uses a public suffix list to group queries. Before, this list was built into the binary and only updated for every release. This release adds the public-suffix-list-file setting that allows operators to supply their own public suffix list. This option is unset by default, which means the built-in list is used.

Over the last years it has become clear that many networks on the internet lose large UDP packets, leading to authoritative servers being seen as dead from the recursor’s perspective. To ensure return packets from authoritative servers have a better chance of reaching the recursor, the edns-outgoing-bufsize setting’s default has changed from 1680 to 1232. 1232 was chosen because it is the largest DNS response that can be carried on an IPv6 link with the IPv6 minimal MTU (1280). In tandem with this change, the udp-truncation-threshold that decides when to truncate responses to clients has also been changed from 1680 to 1232.

Looking Forward

After the release of 4.2.0, the regular bugfix and improvement processes will happen.

At the same time, we will be working on the next major release of the PowerDNS Recursor (probably numbered 5.0) for which we are planning several new and exciting features aimed at moving the DNS ecosystem to a more privacy-centric and secure place. To do this, we would like to implement QNAME Minimisation and support for (longlived) TLS connections to authoritatives.

Other improvements we’d like to implement is an experimental feature where the cache is shared between the worker threads.

If you have any ideas that should be in the PowerDNS Recursor in the future, you’re welcome to open a feature request on GitHub. And if you would want to help write these features, we are still looking for people! Have a look at our careers page or send you CV and motivation to powerdns.careers@powerdns.com.

PowerDNS Recursor 4.1.11 Released

Since Spectre / Meltdown, system calls have become more expensive. In
addition, relevant versions of glibc turn out to implement pthread_cond_wait
and pthread_cond_signal in such a way that they use multiple system calls always.
There is an optimization in glibc to improve this but it is disabled.

This new setup changes our protobuf logging so it amortizes system calls so we perform
far less than one call per message.

Note that our previous RemoteLogger was configured in terms of how many
messages it would buffer. Our new code is configured in terms of how many
bytes. I have multiplied the configured numbers by 100 elsewhere (recursor
config, dnsdist config) to sort of maintain parity.

In addition, the old RemoteLogger would buffer messages while there was no
connection available. We no longer do this.

Finally new, every ‘reconnectTimeout’ seconds we will flush our buffers
opportunistically to not keep people waiting.

The changelog:

  • #7434: Add an option to export only responses over protobuf
  • #7430: Reduce systemcall usage in protobuf logging

The tarball (signature) is available at downloads.powerdns.com and packages for CentOS 6 and 7, Debian Jessie and Stretch, Ubuntu Bionic, Trusty and Xenial are available from repo.powerdns.com.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

PowerDNS Authoritative Server 4.1.6 Released

This release fixes a single issue: the PowerDNS API would accept more than one CNAME record for the same name attribute and would return 204 instead of refusing and returning a 4xx error.

The changelog:

  • #7279: Prevent more than one CNAME/SOA record in the same RRset

The tarball (signature) is available at downloads.powerdns.com and packages for CentOS 6 and 7, Debian Jessie and Stretch, Ubuntu Bionic, Trusty and Xenial are available from repo.powerdns.com.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

PowerDNS Recursor 4.1.10 Released

This release is only relevant when building PowerDNS Recursor from source with protobuf support disabled.

Release 4.1.10 fixes a bug where the recursor would not build when one had protobuf support disabled.

The changelog:

  • #7403: Fix compilation in handleRunningTCPQuestion without protobuf support

The tarball (signature) is available at downloads.powerdns.com and packages for CentOS 6 and 7, Debian Jessie and Stretch, Ubuntu Bionic, Trusty and Xenial are available from repo.powerdns.com.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

Domain security outside of DNS: Getting hacked administratively

This is a brief blogpost on the news that has been sent to us by many people, namely that there is a suspected Iranian group that is “hijacking DNS”. I was about to be interviewed on this subject but sadly that fell through. I did however prepare notes already, so please find some possibly useful things on the subject here.

Briefly: the weakest part of your DNS security currently likely isn’t actually DNS. It is the login (and password reset mailbox) where you manage your domains & nameserver settings. 

In general, if an attacker wants to take over a service you provide (a website, email or whatever), this requires them to change or redirect traffic between users and the targeted service.

There are four “gates” that determine how information flows from/to a named service:

  1. The nameserver configuration for a domain name (“the names of the nameservers”)
  2. What those configured nameserver names respond with.
  3. Which cables those IP addresses are routed to: the Border Gateway Protocol
  4. The actual servers and the software they run

Many attacks have historically focused on item 4, hacking either servers or software. For decades this was the easiest way. Recently however, the most used pieces of software have become more secure, and operators also update their software far more faithfully. Often this is done on their behalf by cloud providers.

Meanwhile, we are seeing a lot more attacks that involve BGP hijacks to route the (correct) IP addresses to incorrect locations.

The currently discussed attack involves items one and two. Why attack a server if you can reroute all traffic with a simple login? Once an attacker is logged in to a domain management solution they can change whatever they want.

So, how are these systems attacked? In the simplest case, an important domain is hosted at a registrar and protected only by a weak or leaked password. We may wonder how this is possible but it happens a lot. The “most important domain” for many companies was frequently registered by the founder ages ago. And unlike all new domains, it still languishes at a relatively unknown provider where it was registered back in the 1990s. This for example is the case for our own ‘powerdns.com’ domain.

And this original founder may not have been a security professional and picked ‘password123’ as his password. Or perhaps he did pick a good password, but it has now leaked in one of the massive breaches over the past decades.

Secondly, almost all DNS control panels come with a password reset solution that typically sends a reset mail.. to a preconfigured email address. This email address again might not be a well secured Gmail account but some ancient Yahoo mailbox that hasn’t been touched in years – again likely with a 1990s-era  password.

If that doesn’t work, an attacker has further options. If gaining access to the control panel of ‘importantdomain.com’ fails on the first try, and the password recovery email address is ‘john@companyfounder.com’, we can repeat the whole process over at ‘companyfounder.com’!

Perhaps someone over time improved the password for ‘importantdomain.com’ but not for the control panel of ‘companyfounder.com’. And if we control the ‘companyfounder.com’ domain, we can hijack the account recovery email, and thence take over ‘importantdomain.com’.

Beyond, even if that fails, we can repeat the whole process for the accounts not of ‘importantdomain.com’, but for the domain that contains the names of the nameservers themselves. Once an atttacker changes those, they can substitute nameservers that give answers that are implement the hijack.

The options to attack a domain administratively go on and on and on. It is therefore indeed very plausible that attackers have been able to acquire control of large numbers of domains.

So, what should operators do? The standing recommendation is of course to enable two-factor authentication for all control panels and to make sure that any remaining account recovery mailboxes are very well secured. Despite our very low opinions of Google’s stance on privacy, currently almost nothing beats a GMail account that itself is two-factor secured.

In addition, some domains can be “registry locked”, which is also highly recommended for high-profile domains.

But the most important recommendation is to audit each and every domain name of the company to see if these security measures have actually been taken. Because through the hops as described above, ‘importantdomain.com’ may in fact be hijacked via the nameserver of the domain of the mailbox of the company founder.

As a final note, it is often claimed that DNSSEC and TLS will protect against these attacks. While adding cryptography does raise the bar, and sometimes significantly, the control panels we have discussed so far include options to disable DNSSEC, while a DNS hijack enables an attacker to get fresh all new TLS certificates under their control within seconds. Using DNSSEC and TLS judiciously requires attackers to work harder, but it is no guarantee.

I hope this has been helpful!

PowerDNS Recursor 4.1.9 Released

We are very happy to announce the 4.1.9 release of the PowerDNS Recursor. This release is fixing two security issues, and addressing a shortcoming in the way incoming queries are distributed to threads under heavy load.This release fixes the following security issues:

  • PowerDNS Security Advisory 2019-01 (CVE-2019-3806): Lua hooks are not called over TCP
  • PowerDNS Security Advisory 2019-02 (CVE-2019-3807): DNSSEC validation is not performed for AA=0 responses

These issues respectively affect PowerDNS Recursor from 4.1.4 and 4.1.0, up to and including 4.1.8.  PowerDNS Recursor 4.0.x and below are not affected.

Minimal patches are available at https://downloads.powerdns.com/patches/2019-01/ and https://downloads.powerdns.com/patches/2019-02/.

The changelog:

  • #7397: Load the Lua script in the distributor thread, check signature for AA=0 answers (CVE-2019-3806, CVE-2019-3807)
  • #7377: Try another worker before failing if the first pipe was full

The tarball (signature) is available at downloads.powerdns.com and packages for CentOS 6 and 7, Debian Jessie and Stretch, Ubuntu Bionic, Trusty and Xenial are available from repo.powerdns.com.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

 

PowerDNS Authoritative Server 4.2.0-alpha1: Lua records, ixfrdist, swagger

We’re proud to release the first alpha version of the PowerDNS Authoritative Server 4.2 series. While some users have already deployed this version straight from our package builders or master repositories, this is still a very fresh release.
4.2 represents almost a year of development over 4.1 and contains some major new features and improvements, while deprecating some functionality you may have been relying on (autoserial, for example).

Lua records

An important new feature is the support for Lua Records, which make the following possible, from any backend (even BIND!):

@ IN LUA A "ifportup(443, {'52.48.64.3', '45.55.10.200'})"

This will poll the named IP addresses (in the background) and only serve up hosts that are available. Far more powerful constructs are possible, for example to pick servers from regional pools close to the user, except if all servers in that pool are down. It is also possible to do traffic engineering based on subnets or AS numbers. A simple example:
@    IN   LUA A ( "ifportup(443, {'52.48.64.3', '45.55.10.200'}, "
                  "{selector='closest'})
For more about this feature, please head to the documentation.

Deprecations

4.2 will see the removal of the poorly documented ‘autoserial’ feature. This removal decision was not taken lightly but as noted, its removal allows us to fix other bugs. Autoserial was holding us back. We realise it is no fun when a feature disappears, but since Authoritative Server 4.1 is still around, you can still use that if you require ‘autoserial’.
Following RFC6986 and anticipating the publication of Algorithm Implementation Requirements and Usage Guidance for DNSSEC, support for both ECC-GOST signing and GOST DS digests have been removed.

ixfrdist

A new tool ixfrdist transfers zones from an authoritative server and re-serves these zones over AXFR and IXFR. It checks the SOA serial for all configured domains and downloads new versions to disk. This makes it possible for hundreds of PowerDNS Recursors (or authoritative servers) to slave an (RPZ) zone from a single server, without overwhelming providers like our friends over at Spamhaus/Deteque and Farsight.
Inspired by our Open-Xchange colleagues our API is now described by a Swagger spec!

Log-log histograms

Over at PowerDNS, we love statistics. Making sense of DNS performance is not that easy however – most queries get answered very quickly, but it is the outliers that determine how users “experience the internet”. It turns out that log-log histograms make it possible to fully capture the quality of a DNS service. As explained in this blog post, PowerDNS now comes with tooling to make such histograms:
log-full-avg

Note that this tooling is not specific to PowerDNS Authoritative or even PowerDNS: it will analyse any PCAP file with DNS in there.

Improvements, fixes

Much more

The changelog lists many more improvements and bug fixes.