Third Release Candidate of PowerDNS DNSdist 1.8.0

We are very happy to release the third candidate of what will become dnsdist 1.8.0!

This release contains fixes for several issues that were found in the second release candidate.

  • #12641: Use the correct source address when harvesting failed
  • #12639: Fix a race when a cross-protocol query triggers an IO error
  • #12638: Report the TCP latency for TCP-only Do53, DoT and DoH backends
  • #12648: Report per-incoming transport latencies in the web interface

The first one is actually a follow-up to the “dnsdist is responding from the wrong source IP address in some setups” which was incompletely corrected in rc2. The second one is a race condition that might have been occurring in very specific cases of network errors during asynchronous processing of cross-protocol queries. The last issue is a bit more complicated: in 1.8.0 we decided to break down the latency metrics to provide a more accurate view of what was actually going on:

  • global latency metrics are now per incoming protocol (Do53 UDP, Do53 TCP, DoT, DoH)
  • backend latency metrics are split between UDP (Do53) and TCP (Do53 TCP, DoT, DoH)

This change brought some adjustment in the interfaces consuming these metrics, and it was reported that this was not quite right. The web interface, for example, was only reporting the UDP-based metrics and not the other ones.

Please see the dnsdist website for the more complete changelog and the current documentation. The upgrade guide is also available there.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

We are immensely grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

The release tarball and its signature are available on the downloads website, and packages for several distributions are available from our repository.

Second Release Candidate of PowerDNS DNSdist 1.8.0

We are very happy to release the second candidate of what will become dnsdist 1.8.0!

This release contains fixes for a few issues that were found in the first release candidate, the most important one being that dnsdist was responding from the wrong source IP address in some setups, which was reported by multiple users. Many thanks to them!

  • #12586: Fix the harvesting of destination addresses, so we reply from the correct source IP in all cases
  • #12587: Skip signal-unsafe logging when we are about to exit, with TSAN
  • #12588: Fix compilation with DoH disabled (Adam Majer)
  • #12589: YaHTTP: Better detection of whether C++11 features are available
  • #12592: Only increment the ‘servfail-responses’ metric on backend responses (phonedph1)
  • #12593: Clean up the fortify and LTO m4 by not directly editing flags
  • #12615: Add Lua bindings for PB requestorID, deviceName and deviceID

Please see the dnsdist website for the more complete changelog and the current documentation. The upgrade guide is also available there.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

We are immensely grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

The release tarball and its signature are available on the downloads website, and packages for several distributions are available from our repository.

PowerDNS Recursor 4.8.3 Released

We are proud to announce the release of PowerDNS Recursor 4.8.3

This release is a maintenance release. The most important fixes concern the serve-stale functionality which could cause intermittent high CPU load. The serve-stale function is disabled by default.

Please refer to the change log for the 4.8.3 release for additional details.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

The tarball and signature are available from our download server and packages for several distributions are available from our repository. The Ubuntu Jammy package will be published soon.

The 4.5.x releases are EOL and the 4.6.x and 4.7.x releases are in critical fixes only mode. Consult the EOL policy for more details.

We would also like to repeat that starting with the 4.5 release branch we stopped supporting systems using 32-bit time. This includes most 32-bit Linux platforms.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

First Release Candidate of PowerDNS DNSdist 1.8.0

Hello!

We are very happy to release the first candidate of what will become dnsdist 1.8.0!

This release contains a significant amount of changes since the last major release, 1.7.0, which was released a bit over a year ago. We try to stick to a major release every six months, but this one took a bit longer than expected as we tackled a few challenges:

Low-end devices friendly

We know, based on the feedback we get from the users that interact with us, that dnsdist is used in a lot of different environments, from very large installations dealing with millions of queries per second to very small computers running in a closet somewhere! While we have until now been more focused on the first case, we have been getting a lot of interest coming from the very-low end of the spectrum: low-end devices, like customer premises equipment (CPEs), with very few resources. We realized that while other open-source components do a good job of providing traditional DNS services in that world, there is a need for software providing DNS over TLS and DNS over HTTPS support, to protect the confidentiality and integrity in the first mile of the internet access.

We knew that dnsdist was already successfully used on small devices, like raspberry pis, and that our memory and CPU usage was quite low, so we were surprised to learn that people were struggling to meet the very stringent requirements of some devices, and decided to have a look. This was a very interesting journey into flash-based filesystems of a few dozen megabytes, proportional set size memory usage, and low-powered CPUs.

Long story short, we managed to drastically reduce our memory usage and our CPU consumption, especially with very low QPS rates. We developed a new way of doing health-checking for these environments, only doing an actual active health-check after detecting failures from normal traffic. We also introduced a few options to reduce our binary size where it matters, like on OpenWrt builds.

OpenWrt integration

We wrote the necessary code to make dnsdist play nicely with OpenWrt’s native configuration format, Unified Configuration Interface (UCI), so that it is easy to set up dnsdist via the usual interfaces, including the Web UI.

We also provide DHCP integration, so that dnsdist can learn about devices on the local network and provide native DNS resolution for these devices.

This integration is not yet merged into the OpenWrt tree as it requires some feature that will only be available once 1.8.0 final has been released. Stay tuned, or reach out if you want a quick peek!

Hostile networks

We also realized that we could no longer rely on the network path between dnsdist and its backend to be trusted: while this is true when dnsdist is deployed on the same box, rack or datacenter as the backend, this no longer is when it is deployed on a CPE and instructed to forward its queries to a remote recursive resolver like Quad9.

Of course we strongly advise using DNS over TLS and/or DNS over HTTPS to secure that path, but this is unfortunately not always possible. We learned the hard way that in some countries ISPs are not only providing DNS over plain UDP only, without even supporting plain TCP, they are also still blocking attempts to connect to an external resolver via a more secure channel.

To work around that issue, we implemented new features to make dnsdist suitable as a proxy with an untrusted network path to the resolver, using well-known methods: random ports and random IDs. These are not enabled by default because they come at a cost, which we don’t want to impose when it is not necessary.

Discovery of Designated Resolvers

It’s one thing to support DNS over TLS and DNS over HTTPS both inbound and outbound, but it really does not help if the client does not know that you do, or if the configuration does not tell dnsdist that the backend does.

The IETF has been working for quite some time now on a new mechanism that leverages the SVCB record type to actually advertise that a secure, encrypted endpoint is available for use: Discovery of Designated Resolvers (DDR).

Since 1.7.0 dnsdist has been able to advertise DoT and DoH support to the client via SVCB records, but that requires writing a few lines of Lua to configure it. In 1.8.0, we have integrated that process into the OpenWrt configuration, requiring a single click to enable DDR advertisement to all the local clients, allowing Android and iOS devices to automatically upgrade to a secure channel.

We also taught dnsdist how to use DDR to detect whether a given backend can be upgraded from plain Do53 to DoT and DoH, so that we switch to a secure channel as soon as it becomes available, and fallback to Do53 if needed.

Faster TLS

To be able to keep pushing for broader adoption of DoT and DoH, it is crucial to reduce the overhead of the encryption compared to plain old Do53. To do so, we have added support for:

The technologies are still evolving quickly, and for now are marked as experimental in dnsdist but yields very promising results.

Second-chance lookups

The ability to act on a Server Failure, Refused, or any specific type of responses to trigger a second DNS lookup is a feature that regularly came up. It was not easy to implement given the existing design of dnsdist, but we refactored a fair amount of code in this release to be able to process queries and responses in an asynchronous way, paving the way for external lookups without blocking dnsdist and degrading performance.

This refactoring allowed us to finally implement that second-chance lookup, so that a query can be re-sent to a different pool of servers if the obtained response is not good enough.

User-defined metrics

It is now possible to define custom counters and gauges, that can be manipulated via the Lua API and are exported via the API and prometheus like built-in metrics.

New compilations options, Link-Time optimizations

We introduced several new compile-time options:

  • Link-Time Optimizations (LTO): GCC, clang and the associated linkers now support a new mode of building a binary, where information about all the individuals components, called compilation units, is made available to the linker so that it can make better optimization decisions. We have now enabled these optimizations in our own packages, via the –enable-lto option.
  • For a long time, we have been automatically detecting if the compiler has support for the FORTIFY_SOURCE=2 hardening option, enabling it whenever possible. Recently a stronger version of that option has been supported by GCC and clang, FORTIFY_SOURCE=3. This stronger version can be enabled by passing either –enable-fortify-source=3 or –enable-fortify-source=auto to our configure, with the latter always selecting the best supported version. We have enabled the stronger version in our test suites, but not yet in our production builds, as we are not yet sure of the actual impact
  • C++, as opposed to other languages, does not initialize its variables by default. This had led to a fair amount of security issues in the past, ranging from information disclosure to the ability to execute arbitrary code. We now have a new option, –enable-auto-var-init=zero, that can be used to zero-initialize all variables that are allocated on the stack. We have not yet enabled this option in our production builds, but we have enabled instead, in our test suites, a variant that increases the likelihood of detecting bugs by initializing the variables with specific patterns: –enable-auto-var-init=pattern

Users that can trade a bit of performance for stronger security guarantees are invited to enable both –enable-fortify-source=auto and –enable-auto-var-init=zero.

And many other improvements

  • A lot of new functionalities are now accessible via Lua: helpers to interface with the system network configuration, to get the MAC address of a client, to inspect and edit queries and responses
  • The scalability of MaxQPSIPRule has been improved on multi-core setups
  • The handling of multiple Carbon servers was lacking, allowing a misbehaving Carbon server to impact other servers: this has now been fixed
  • We introduced a new chain of rules, triggered after cache insertion
  • Our eBPF and XDP code has been greatly improved by Pierre Grié, Y7n05h and Yogesh Singh

Security audit

In the second part of 2022 we have mandated a security audit from the Nixu team, to have a strong look at the new features we introduced in 1.8.0 in particular (DDR, DNS over HTTPS, OpenWrt integration). This is the second audit of the dnsdist code-base realized by Nixu, and they were able to quickly focus on the new features. They went above and beyond what we expected, as they did last time, and found a potential issue in the way our ACL interacted with the OpenWrt system, in our not yet released UCI integration. In short, we were relying a bit too much on the OpenWrt firewall, and it might have opened access to the Do53, DoT and DoH ports from unintended network interfaces in some deployment scenarios where the firewall was not effective. We fixed that by being more restrictive in our default ACL.

Please see the dnsdist website for the more complete changelog and the current documentation. The upgrade guide is also available there.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

We are immensely grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

The release tarball and its signature are available on the downloads website, and packages for several distributions are available from our repository.

PowerDNS Recursor 4.8.2 Released

We are proud to announce the release of PowerDNS Recursor 4.8.2.

This release is a maintenance release, fixing some issues, in particular:

  • Record and negative cache cleaning now maintains balance between shards in a better way
  • A case where the wrong EDNS Client Subnet scope could be applied to outgoing queries has been fixed
  • A few other minor issues

Please refer to the change log for the 4.8.2 release for additional details.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

The tarball and signature are available from our download server and packages for several distributions are available from our repository. The Ubuntu Jammy package will be published soon.

The 4.5.x releases are EOL and the 4.6.x and 4.7.x releases are in critical fixes only mode. Consult the EOL policy for more details.

We would also like to repeat that starting with the 4.5 release branch we stopped supporting systems using 32-bit time. This includes most 32-bit Linux platforms.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

Security Advisory 2023-01 for PowerDNS Recursor 4.8.0

Hello,

Today we have released PowerDNS Recursor 4.8.1 due to a high severity issue found.

Please find the full text of the advisory below.

The changelog is available.

The tarball (signature) is available from our download server. Patches are available at patches. Packages for various distributions are available from our repository.

Note that PowerDNS Recursor 4.5.x and older releases are End of Life. Consult the EOL policy for more details.


PowerDNS Security Advisory 2023-01: unbounded recursion results in program termination

  • CVE: CVE-2023-22617
  • Date: 20th of January 2023
  • Affects: PowerDNS Recursor 4.8.0
  • Not affected: PowerDNS Recursor < 4.8.0, PowerDNS Recursor 4.8.1
  • Severity: High
  • Impact: Denial of service
  • Exploit: This problem can be triggered by a remote attacker with access to the recursor by querying names from specific mis-configured domains
  • Risk of system compromise: None
  • Solution: Upgrade to patched version

An issue in the processing of queries for misconfigured domains has been found in PowerDNS Recursor 4.8.0, allowing a remote attacker to crash the recursor by sending a DNS query for one of these domains. The issue happens because the recursor enters a unbounded loop, exceeding its stack memory. Because of the specific way in which this issue happens, we do not believe this issue to be exploitable for code execution.

PowerDNS Recursor versions before 4.8.0 are not affected.

Note that when the PowerDNS Recursor is run inside a supervisor like supervisord or systemd, a crash will lead to an automatic restart, limiting the impact to a somewhat degraded service.

CVSS 3.0 score: 8.2 (High)
https://www.first.org/cvss/calculator/3.0#CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:L/A:H/E:H/RL:U/RC:C

Thanks to applied-privacy.net for reporting this issue and their assistance in diagnosing it.

PowerDNS Recursor: writing to a file you did not open

This is the sixth part of a series of blog posts we are publishing, mostly around recent developments with respect to PowerDNS Recursor:

This post is about how the Recursor can write to files even when its permissions to access the file system are restricted.

When PowerDNS Recursor is running it mostly does not need to access files. In many runtime environments its access to the file system is restricted to limit the impact of potential security issues. When reconfiguring the Recursor, we need to make sure the files it needs to read are accessible in this restricted runtime environment. But in some cases, we also want to be able to write files.

A common case for writing to a file is extracting status information from the Recursor, for example, a dump of the Recursor caches. In older releases, the Recursor would need to have write access to a directory where it could write those dumps. This is not ideal, as this restricted runtime environment might not allow for opening files with write permission and even if it allows writing, we would rather run the Recursor without that ability.

Depending on the runtime environment, the place where files are written may be surprising. For example, using chroot or systemd‘s RuntimeDirectory and PrivateTmp changes the view of the file system from the Recursor’s perspective. As questions asked on IRC and other help channels reveal, this can be confusing and hard to diagnose for a system administrator.

From a general security perspective it is good to restrict a networking daemon to not be able to create and write to files, so how can we solve the problem of where to write the cache dumps?

Passing information between processes

One way would be to dump to information via a socket to a client process. That would involve the Recursor writing to a socket, the client process then reads from the socket and writes the information to a file or pipe.

We chose another approach: file descriptor passing. The main advantage is that this allows the information to be written immediately to a file or pipe without an intermediate networking step transferring the data between the Recursor and a client process. This simplifies the code and avoids work for both the client and the Recursor.

File descriptor passing

File descriptor passing is a technique that can be used between two processes communicating over a local (also know as UNIX) domain socket. The Recursor’s command client program rec_control already uses a local socket to communicate with the Recursor.

When a file, pipe or socket is opened it is assigned a file descriptor by the kernel. The file descriptor is an integer and its scope is per-process. The kernel keeps track of open file descriptors per process and has a mechanism to translate the per process file descriptor to an actual kernel object, which represents the actual file, pipe or socket. File descriptor passing transfers a file descriptor from one process to another while keeping the kernel file object reference the same, only the per-process kernel data are changing: the sending process loses a file descriptor slot while the receiving one gains one.

The effect is that one process can open a file (or other object) for writing and then transfer the file descriptor–which is just a reference–to another process. The second process can then write to the file (or other object) without having the permissions to create or open files for writing itself.

The code to do the actual file descriptor passing is a bit arcane, but well documented. See for example https://www.man7.org/linux/man-pages/man3/cmsg.3.html or http://man.openbsd.org/man3/CMSG_DATA.3.

Consequences for system administrators

The rec_control command was modified to use file descriptor passing with the release of Recursor 4.4.0. From the system administrator’s perspective, the command still looks like this:

rec_control dump-cache cache_dump.txt

The difference is that the cache_dump.txt file will now be created using the credentials and current working directory of the rec_control process and not the credentials and current working directory of the Recursor process. The rec_control command will create the file, pass the file descriptor to the Recursor and wait until the Recursor signals it is done writing. This part of the communication between rec_control and the Recursor did not change.

It now is also possible to let the Recursor write to standard output of the rec_control command:

rec_control dump-cache - | grep some_pattern

Other rec_control subcommands writing files were converted in a similar way.

Conclusion

By using file descriptor passing, we are able to run the Recursor process with more restricted privileges and provide a flexible way to write diagnostic information.

PowerDNS Recursor 4.8.0 Released

We are proud to announce the release of PowerDNS Recursor 4.8.0.

Compared to the previous major (4.7) release of PowerDNS Recursor, this release contains the following major changes:

  • Structured Logging has been implemented for almost all subsystems. This allows for improved (automated) analysis of logging information. We’ve posted a blog about this feature recently.
  • Optional Serve Stale functionality has been implemented, providing resilience against connectivity problems towards authoritative servers.
  • Optional Record Locking has been implemented, providing an extra layer of protection against spoofing attempts at the price of reduced cache efficiency.
  • Internal tables used to track information about authoritative servers are now shared instead of per-thread, resulting in better performance and lower memory usage.
  • EDNS padding of outgoing DoT queries has been implemented, providing better privacy protection.
  • Metrics have been added about the protobuf and dnstap logging subsystems and the rcodes received from authoritative servers.

As always, there are also many smaller bug fixes and improvements, please refer to the changelog for additional details. When upgrading do not forget to check the upgrade guide.

We are also announcing the removal of XPF support. If you are using this feature, switch to the Proxy Protocol.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

The tarball (signature) is available from our download server and packages for several distributions are available from our repository.

With the 4.8.0 release, the 4.5.x releases will be marked “End of Life” and the 4.6.x and 4.7.x releases will go into critical fixes only mode. Consult the EOL policy for more details.

We would also like to mention that with the 4.5 release we stopped supporting systems using 32-bit time. This includes many 32-bit Linux platforms.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

PowerDNS Authoritative Server 4.5.5, 4.6.4 and 4.7.3 Released

Hello,

Today we have released maintenance updates of PowerDNS Authoritative Server 4.5.5, 4.6.4 and 4.7.3, containing fixes for a few minor issues. For more details on the other fixes, consult the changelogs available at 4.5.5, 4.6.4, 4.7.3.

The source tarballs (4.5.5, 4.6.4, 4.7.3) and signatures (4.5.5, 4.6.4, 4.7.3) are available from our download server. Packages for various distributions are available from our repository.

Note that PowerDNS Authoritative Server 4.4.x and older releases are End of Life. Consult the EOL policy for more details.

We would also like to repeat that starting with the 4.5 release branch we stopped supporting systems using 32-bit time. This includes most 32-bit Linux platforms.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

PowerDNS Recursor 4.5.12, 4.6.5 and 4.7.4 Released

Hello,

Today we have released a maintenance release of PowerDNS Recursor 4.5.12, 4.6.5 and 4.7.4, containing fixes for a few minor issues. In particular, RPZ IXFRs now time out if the server becomes unresponsive. For more details on the other fixes, consult the changelogs available at 4.5.12, 4.6.5, 4.7.4.

The source tarballs (4.5.12, 4.6.5, 4.7.4) and signatures (4.5.12, 4.6.5, 4.7.4) are available from our download server. Packages for various distributions are available from our repository.

Note that PowerDNS Recursor 4.4.x and older releases are End of Life. Consult the EOL policy for more details.

We would also like to repeat that starting with the 4.5 release branch we stopped supporting systems using 32-bit time. This includes most 32-bit Linux platforms.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.