PowerDNS Authoritative Server 4.2.0-alpha1: Lua records, ixfrdist, swagger

We’re proud to release the first alpha version of the PowerDNS Authoritative Server 4.2 series. While some users have already deployed this version straight from our package builders or master repositories, this is still a very fresh release.
4.2 represents almost a year of development over 4.1 and contains some major new features and improvements, while deprecating some functionality you may have been relying on (autoserial, for example).

Lua records

An important new feature is the support for Lua Records, which make the following possible, from any backend (even BIND!):

@ IN LUA A "ifportup(443, {'52.48.64.3', '45.55.10.200'})"

This will poll the named IP addresses (in the background) and only serve up hosts that are available. Far more powerful constructs are possible, for example to pick servers from regional pools close to the user, except if all servers in that pool are down. It is also possible to do traffic engineering based on subnets or AS numbers. A simple example:
@    IN   LUA A ( "ifportup(443, {'52.48.64.3', '45.55.10.200'}, "
                  "{selector='closest'})
For more about this feature, please head to the documentation.

Deprecations

4.2 will see the removal of the poorly documented ‘autoserial’ feature. This removal decision was not taken lightly but as noted, its removal allows us to fix other bugs. Autoserial was holding us back. We realise it is no fun when a feature disappears, but since Authoritative Server 4.1 is still around, you can still use that if you require ‘autoserial’.
Following RFC6986 and anticipating the publication of Algorithm Implementation Requirements and Usage Guidance for DNSSEC, support for both ECC-GOST signing and GOST DS digests have been removed.

ixfrdist

A new tool ixfrdist transfers zones from an authoritative server and re-serves these zones over AXFR and IXFR. It checks the SOA serial for all configured domains and downloads new versions to disk. This makes it possible for hundreds of PowerDNS Recursors (or authoritative servers) to slave an (RPZ) zone from a single server, without overwhelming providers like our friends over at Spamhaus/Deteque and Farsight.
Inspired by our Open-Xchange colleagues our API is now described by a Swagger spec!

Log-log histograms

Over at PowerDNS, we love statistics. Making sense of DNS performance is not that easy however – most queries get answered very quickly, but it is the outliers that determine how users “experience the internet”. It turns out that log-log histograms make it possible to fully capture the quality of a DNS service. As explained in this blog post, PowerDNS now comes with tooling to make such histograms:
log-full-avg

Note that this tooling is not specific to PowerDNS Authoritative or even PowerDNS: it will analyse any PCAP file with DNS in there.

Improvements, fixes

Much more

The changelog lists many more improvements and bug fixes.

Enabling continuous fuzzing of PowerDNS products

We are very happy to announce that PowerDNS recently joined the OSS-Fuzz initiative, enabling continuous fuzzing for critical parts of our products. We are taking this opportunity to share some of our existing security processes, because we are proud of the progress we have made.

For quite some time now, we have been working on making our products safer by identifying and fixing existing security weaknesses, and deploying processes to prevent new ones from being introduced while reducing the impact should one slip through the cracks.

These efforts have led to the discovery of quite a few security issues in the past years, some of them present in the code base for several years. Nowadays, we are finding these issues because we are actively looking for them, using manual review, automated code analysis tools and fuzzing. But more on that later.

Very often, the line between a bug and a possible security issue is very thin, and vendors have been known to try to downplay the possible impact of an issue whenever possible. At PowerDNS, we choose instead to be forthcoming and open about the possible impact of the bugs that are discovered in our products. For one, there is no point in trying to downplay or obfuscate the real impact, as anyone can read the source code of our open-source products and decide that for themselves. But most of all, we believe our users should be as informed as possible regarding the severity of the issues fixed in a release, allowing them to decide for themselves how quickly they need to upgrade.

Now, let’s review the processes we have in place to prevent and find issues in our code base.

Secure development practices

In 4.0, we switched to a new version of the C++ language, C++11, which introduced several new features that allow us to avoid common pitfalls often associated with C and C++: for example, when they are properly used, smart pointers kill whole classes of bugs and security issues, including memory leaks and use-after-free.

At the same time, we also regularly stop using features that have proven brittle. For example, we recently removed our last usage of variable-length arrays (VLA), because we noticed it was very easy to shoot yourself in the foot and smash the stack by not properly checking the maximum size of such an array. We now rely on a specific compiler flag to prevent us from ever introducing them back by mistake, -Werror=vla.

Peer review and continuous testing

Every change to the code base goes through the same process:

  • a pull request is opened ;
  • our continuous testing tools make sure that it builds properly, and passes both our unit and functional tests ;
  • the change is reviewed by at least one person from our team ;
  • finally, it is merged into the appropriate branch.

To increase the likelihood of catching bugs during the continuous testing phase, our products are built with various sanitizing options enabled whenever applicable:

  • Address Sanitizer: catching out-of-bounds accesses, use-after-free, double-free, invalid free and memory leaks ;
  • Undefined Behavior Sanitizer: catching the use of misaligned or null pointers, signed integer overflow, and conversion to, from, or between floating-point types which would overflow the destination.

We also use several static analysis tools to detect mistakes that eluded both the original author and the reviewer:

Independent audit of the code base

Last year, we contracted a third-party security company, Nixu, to perform an independent audit of our code base, mainly focused on dnsdist and the Recursor but also covering a large part of the Authoritative server. Their assessment included a manual code review, supported by threat analysis using the STRIDE method, and the use of various static analysis tools, fuzzing and manual testing with ad-hoc tools built for that project.
A total of 11 findings were reported to us. A few of them could have allowed an authenticated user to escalate privileges, using defects in our API and web management interface. The remaining issues were denial of service, most of them in the DNSSEC handling code in the Recursor.
All of these issues have been fixed shortly after the report was handed to us, and we published the relevant security advisories about them.

Bug Bounty Program

We believe that working with skilled security researchers across the globe is crucial in identifying weaknesses in any technology. Therefore, in addition to our website advertising clear and simple ways of securely report any concern to us, we have also been running a public bug bounty program on the Hacker One platform for more than two years now: https://hackerone.com/powerdns

This program covers all our open-source products and every type of vulnerability, allowing everyone finding a security issue in our products to do the right thing by reporting it to us, while still getting financially rewarded.

Hardening measures

To further restrict the impact of a security issue in one of our products, we enable several hardening measures by default.
Some of these try to detect pathological behavior as early as possible, make it very hard to exploit it and almost always turn it into a crash, which is mitigated by supervisors like systemd:

  • buffer overflow, format string detection and prevention (stack-protector, FORTIFY_SOURCE=2) ;
  • full address space layout randomization via Position-Independent Executable (PIE) ;
  • fully read-only relocations (-Wl,-z,relro,-z,now).

Other measures restrict the privileges of the process, thus limiting what an attacker could do even if it could take full control of the process-execution path:

  • filesystem protection (chroot; systemd’s PrivateTmp, PrivateDevices, ProtectSystem and ProtectHome) ;
  • privileges dropping (uid/gid switching; systemd’s CapabilityBoundingSet, NoNewPrivileges and RestrictAddressFamilies).

Fuzzing

We used to run fuzzing sessions manually using AFL and Address Sanitizer after every important change to one of our parsers, since they deal with user-provided input and are therefore the more exposed part of our code. While we discovered a few issues that way, catching them before they could make it into a release, we were not very thrilled with this method, because it required human interaction.

That’s why we are so happy to have joined the OSS-fuzz project, enabling continuous fuzzing of our parsers. Continuous fuzzing means that every change to the code will get automatically tested and that, should an issue be found, a ticket will be automatically generated and we will be notified.
The OSS-fuzz infrastructure will not only run our targets with AFL as we used to do manually, it will also use the other well-known fuzzing engine, libFuzzer. Both of them will be run with Address and Undefined Behavior sanitizers enabled, catching issues that could otherwise be completely undetected.

Security issue handling

Now that we have explained how we prevent, catch and mitigate security issues, a few words about the way we handle them when they occur in a supported version.

As soon as a security concern has been reported to us, be it via our website, in a private e-mail or our program on HackerOne, we immediately acknowledge the reception of the message and start working on reproducing and assessing the severity of the issue. We then begin working on a patch for all supported versions, and communicate a timeline and a draft of the security advisory to the reporter when applicable.

Every security issue is then communicated to our customers in advance, at least a week before the public release, along with patched packages. We also notify various vendors via the distros security mailing-list. After the release, every security issue is the subject of a public security announcement, and these are publicly available on our website:

Important Changes in PowerDNS Authoritative Server 4.2.0

Hello PowerDNS user,

as the year draws to an end, we are preparing for the release of a new version of the Authoritative Server (and the Recursor – stay tuned for more news in that area). This release (4.2.0) will see some changes that may affect your usage of PowerDNS. Please read, or at least skim, this article to make sure you will not be surprised when you upgrade.

If, after reading this, you have questions, please reach out to us via our mailing list or IRC channel. If you feel you’ve found a bug or other problem, we’d love a ticket on our GitHub repository.

Autoserial

Since forever, PowerDNS has supported a badly-documented feature called ‘autoserial’ which allowed a database backend to generate a serial on demand, when the serial inside the SOA record was set to zero. This feature was never documented properly and most deployments we have seen of it turned out to be broken in key areas. Supporting this feature also means we have bugs around the handling of the valid serial number zero in zones that do not want autoserial.

Over the last few releases, PowerDNS has gained robust HTTP REST support, and a mature implementation of RFC 2136 DNS Update. For the command line user, pdnsutil increase-serial is a simple way to increase serials on zones after updates.

Because autoserial was hard to use, rarely did what people wanted, and because its presence prevents us from fixing other bugs, we have decided to retire the feature now that several alternatives are available.

Builder/packaging changes

Between 4.1 and 4.2, we have replaced our old package building infrastructure with a new one, based on pdns-builder. While the idea is that most users will not notice this transition, there may be unintended changes.

Our Debian/Ubuntu packages no longer use ucf.

Our ./configure script inconsistently used --enable and --with. This has been fixed; downstream packagers may need to adjust their packaging.

Lua

Around the development of the LUA record type (more on that in a separate post), some of the Lua handling in the Authoritative Server has been refactored. If you have any existing Lua scripts in your Auth server, please make sure they still work correctly after upgrading to 4.2.

GOST

Following RFC6986 which deprecates the usage of GOST R 34.11-2012 generally, and anticipating the publication of Algorithm Implementation Requirements and Usage Guidance for DNSSEC which intends to move DNSSEC ECC-GOST support in signers to the ‘MUST NOT’ category, support for both ECC-GOST signing and GOST DS digests has been removed. In the unlikely situation that you have domains signed with ECC-GOST, you will need to roll their algorithms before upgrading to PowerDNS Authoritative Server 4.2.

 

PowerDNS Recursor 4.1.8 Released

We’ve released PowerDNS Recursor 4.1.8.

This release fixes Security Advisory 2018-09 that we recently discovered, affecting PowerDNS Recursor from 4.1.0 up to and including 4.1.7.  PowerDNS Recursor 4.0.x and below are not affected.

The issue is that a remote attacker can trigger an out-of-bounds memory read via a crafted query, while computing the hash of the query for a packet cache lookup, possibly leading to a crash.

When the PowerDNS Recursor is run inside a supervisor like supervisord or systemd, a crash will lead to an automatic restart, limiting the impact to a somewhat degraded service.

A minimal patch is available at https://downloads.powerdns.com/patches/2018-09/.

The changelog:

  • #7221: Crafted query can cause a denial of service (CVE-2018-16855)

The tarball (signature) is available at downloads.powerdns.com and packages for CentOS 6 and 7, Debian Jessie and Stretch, Ubuntu Bionic, Trusty and Xenial are available from repo.powerdns.com.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

PowerDNS Recursor 4.1.7 Released

Today we have released the PowerDNS Recursor 4.1.7. It is an update to relax EDNS compliance requirements from upstream authoritative servers.

Recursor version 4.1.5 (and, by extension, 4.1.6), contains a fix for Security Advisory 2018-07. One part of that fix is a stricter fallback to non-EDNS queries when EDNS queries fail. It turns out that there are several authoritative servers on the Internet that have such bad EDNS handling, that the domains hosted on them stop resolving with 4.1.5. The 4.1.7 release has relaxed the EDNS compliance requirement and includes an alternative fix for 2018-07.

Since reports of this started coming in yesterday, some domains have been fixed by their owners, but a long tail of broken zones remains for now.

We have decided to release this increase in strictness in the PowerDNS Recursor 4.2.0, so that domain owners can work on their server’s compliance. We urge operators of authoritative servers to check their domains and servers with the EDNS compliance tool and act upon its results. Increased EDNS compliance strictness will be added to many DNS resolvers coming next February.

The changelog is as follows:

  • #7172: Revert ‘Keep the EDNS status of a server on FormErr with EDNS’
  • #7174: Refuse queries for all meta-types

As always, the tarball(sig) can be found on the downloads website and packages for CentOS 6 and 7, Ubuntu Trusty, Xenial and Bionic and Debian Jessie and Stretch can be found on repo.powerdns.com.

dnsdist 1.3.3 released

We are very happy to announce the 1.3.3 release of dnsdist. This release contains a few new features, but is mostly fixing a security issue reported since the release of dnsdist 1.3.2.

Security fix

While working on a new feature, Richard Gibson noticed that it was possible for a remote attacker to craft a DNS query with trailing data such that the addition of a record by dnsdist, for example an OPT record when adding EDNS Client Subnet, might result in the trailing data being smuggled to the backend as a valid record while not seen by dnsdist. This is an issue when dnsdist is deployed as a DNS Firewall and used to filter some records that should not be received by the backend. This issue occurs only when either the ‘useClientSubnet’ or the experimental ‘addXPF’ parameters are used when declaring a new backend.

While dnsdist has not had any important security issue until now, we decided this was a good time to implement the same security polling mechanism that the authoritative server and the recursor have had for years. Starting with this release, dnsdist will regularly perform a security check using a DNS query to determine whether the current version is up-to-date security-wise, and let the administrator know otherwise.

Important changes

It is sometimes very useful to be able to generate answers directly from dnsdist, to quickly return a “No such domain” answer, spoof an “A” or “AAAA” answer, or even just reply with the TC bit set so that legitimate clients retry over TCP. Until now, answers generated that way were mirroring the flags and EDNS options, if any, of the initial query. This was not great because it could mislead the client into thinking that dnsdist, or the server behind it, was supporting features or a UDP payload size it did not.

Starting with this release, dnsdist is now generating a proper EDNS payload if the query had one, and responding without EDNS otherwise. This behavior can be turned off using the new setAddEDNSToSelfGeneratedResponses() directive if needed.

We must, however, provide a responder’s maximum payload size in this record, and we can’t easily know the maximum payload size of the actual backend so we picked a safe default value of 1500, which can be overridden using the new  setPayloadSizeOnSelfGeneratedAnswers() directive.

New features and improvements

A new load-balancing policy named “chashed” has been introduced, based on consistent hashing. This new policy load-balances the incoming queries based on a hash of the requested name, like the existing “whashed” one, but has the interesting property that adding or removing a server will only cause a very small portion of the incoming queries to be mapped to a different server than they were before, keeping the caches warm.

While we have been supporting the export of metrics using the well-known carbon protocol from day one, we have seen an increasing demand for supporting the emerging Prometheus protocol. Thanks to the work of Pavel Odintsov and Kai S, dnsdist now supports it natively.

Very large installations of the DNS over TLS feature introduced in 1.3.0 reported several issues that we addressed in this release:

  • dnsdist did not set TCP_NODELAY on its TLS sockets, causing needless latency ;
  • it was not possible to configure the number of stored TLS sessions ;
  • our OpenSSL implementation had a memory leak when some clients aborted prematurely because of a negotiation error during the TLS handshake.

We seized the opportunity to refactor the part of the code handling TLS connections with the use of smart pointers while fixing that last issue, making sure that this kind of memory leak will not happen again.

In 1.3.2, the optimized DynblockRulesGroup introduced in 1.3.0 gained the ability to whitelist and blacklist ranges from dynamic rules, for example to prevent some clients from ever being blocked by a rate-limiting rule. This feature has now been made available when our in-kernel eBPF filtering feature is used as well. At the same time, we introduced the ability to set up warning rates to the dynamic rules, making it possible to get an alert without blocking clients when they reach a configured rate, and to block them should they reach a higher rate.

Finally, we introduced several new rules to our existing set:

  • EDNSOptionRule, to be able to filter based on the presence of a given EDNS option ;
  • DSTPortRule, offering the ability to route queries by looking at their destination port ;
  • PoolAvailableRule, to be able to route queries based on whether a pool has at least one usable backend.

Please see the dnsdist website for the more complete changelog and the current documentation.

Release tarballs are available on the downloads website.

Several packages are also available on our repository. Please be aware that we have enabled a few additional features in our packages, like DNS over TLS and DNSTap support, on distributions where the required dependencies were available.

PowerDNS Recursor 4.1.6 Released

We have just released the PowerDNS Recursor 4.1.6. This release fixes an issue with DNSSEC validation introduced in Recursor 4.1.5 by reverting a code-change related to the acceptation of ADDITIONAL records. We will investigate this issue in detail the coming days, but found it necessary to issue an update with a fix early.

The full changelog is very short:

  • Revert “rec: Authority records in AA=1 CNAME answer are authoritative”. References: #7158, pull request 7159

The tarball(signature) is available at downloads.powerdns.com and packages for CentOS 6 and 7, Debian Jessie and Stretch, Ubuntu Bionic, Trusty and Xenial are available from repo.powerdns.com.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.