Category: Uncategorized

First Release Candidate of PowerDNS Recursor 4.7.0

We are proud to announce the first release candidate of PowerDNS Recursor 4.7.0. Testing of this release candidate is much appreciated!

The most important change compared to the 4.7.0-beta1 release is a fix for the experimental DoT to authoritative server probing code.

Compared to the previous major (4.6) release of PowerDNS Recursor, this release contains the following major changes:

  • A configurable way of adding Additional records to answers sent to the client, so the client does not have to ask for these records.
  • The step sizes for Query Minimization are now computed following to guidelines in RFC 9156.
  • The Recursor now schedules tasks to resolve IPv6 addresses of name servers not learned by glue records. This has the consequence that, if applicable, name servers will be contacted over IPv6 more often.
  • An experimental implementation of unilateral DoT probing. This allows the Recursor to learn if a an authoritative servers supports DoT.
  • Recursor has gained a way to fall back to parent NS set if contacting servers in the child NS set does not lead to an answer. This works around some broken authoritative servers configurations.
  • ZONEMD validation of the zones retrieved by the Zone to Cache, providing integrity guarantees for the zone retrieved.
  • The table recording round trip times of authoritative server IP addresses is now shared between threads to make it more effective and to reduce its memory footprint.
  • A Lua FFI hook for post-resolve interception: postresolve_ffi, providing a very fast way to do post-resolve Lua scripting.

As always, there are also many smaller bug fixes and improvements, please refer to the changelog for additional details. When upgrading do not forget to check the upgrade guide.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

The tarball (signature) is available from our download server and packages for several distributions are available from our repository.

With the final 4.7 release, the 4.4.x releases will be EOL and the 4.5.x and 4.6.x releases will go into critical fixes only mode. Consult the EOL policy for more details.

We would also like to mention that with the 4.5 release we stopped supporting systems using 32-bit time. This includes most 32-bit Linux platforms.

We also like to announce the upcoming removal of XPF support. If you are using this feature, plan switching to the proxy protocol.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

Refreshing Of Almost Expired Records: Keeping The Cache Hot

We’re planning to do a series of posts to highlight some of the features we have been working in recent releases or the Recursor. We also plan to discuss some C++ programming techniques we used in the Recursor that might be interesting. The first post in this series describes a technique that was introduced in PowerDNS Recursor 4.5: Refresh Almost Expired.

Introduction

When the Recursor receives a query from a client it answers it by first finding the name server that is authoritative for the domain in question. It does that by starting at a root name server and then walking DNS delegations to find the name servers (and their addresses) that are authoritative for the question asked. Once it knows which name servers are authoritative for the domain, it picks a specific name server address and asks the question to that name server. After the Recursor receives the answer, it will pass it on the the client asking the question.

To be able to answer fast, the Recursor caches information it receives from authoritative name servers. Both information about delegations and specific queries is stored in the record cache, which can be seen as a local in-memory copy of parts of the global DNS tree.

The recursor also has a few other caches, for example the Packet Cache and the Negative Cache, but these are not the subject of this post.

The Record Cache

The Record Cache contains entries of the following form

name, type, time to die, record set, ...

The entries are indexed and searchable by a name and type combination; time to die (TTD) is the time the entry will expire and record set is the set of DNS records associated with the name and type. When inserting an entry into the Record Cache the TTD is computed by taking the time to live (TTL) of the record set received from the authoritative name server and adding the current time to it. That way, it becomes easy to see if a record in the Record Cache is expired: just compare the TTD to the current time.

Query Processing

With the Record Cache, query processing becomes a bit more complicated:

1. Check to see if the Record Cache contains an entry (name, type) 
2. If an entry is present and not expired: return the answer using the
   record set
3. If the entry is not present, continue by finding the authoritative server 
   (using the Record Cache or the internet) as before and asking it the 
   question.
4. If an answer is received from an authoritative server, insert it into the 
   Record Cache and return it to the client.

Clients will have a better experience for cached record sets: the Recursor will be able to answer cached queries much faster than when it has to go to the internet to contact one or more authoritative servers to get an answer.

Caching Issues

When using a cache, it is always important to keep consistency in mind: the cache should reflect the contents of the records of the authoritative name servers on the internet. Here we have an issue: we keep a record set in the cache for a maximum period equal to the TTL. If an authoritative server changes a record’s content in between, we do not know that until we re-fetch the record set and we won’t do that until the record expires. The general consensus is that we accept this issue, since caching DNS records make the Recursor very fast compared to always asking authoritative name servers. This caching period is the main reason why it takes time for an updated DNS record to appear in the caches of resolvers and returned to clients.

Another issue is making sure the cache is effective: in an ideal world, we only want to store records that are queried again within the TTL period. Otherwise we waste resources storing data that just expires after a while without being used. Sadly we do not have a way to know beforehand if a record is going to be retrieved from the cache after we have stored it. We somewhat solve this issue by making sure that if records need to be evicted from the cache because it is too full, we pick the records using a least recently used (LRU) method in addition to having a preference to evict expired records.

The Client Experience

When a client ask a “popular” question, often it will get an answer from the cache, as popular record sets tend to be present in the cache. But once in a while an unlucky client has to wait a bit, as the record in the cache is expired and the Recursor has to go out on the net to find the answer. The chart below illustrates what happens when querying a record set with a short TTL 60 times with a one second interval, starting with a recently filled cache:

Of the 60 queries done, most are answered within about 1ms, but three take significantly more .The peaks are 20s apart, the TTL of the record set in question.

Wouldn’t it be nice if there was a way to avoid that some clients will have to wait much longer for an answer than other clients? What if we would be able to fetch an updated record set before it expires from the record cache?

Keeping the Cache Hot

The jargon word for a cache that contains data you are looking for is a hot cache.

To try to make sure that entries in the cache are up to date the Recursor can be configured to issue queries to refresh them in the background when it sees that a record set is popular and almost expired. The Recursor uses a rather simple way to decide if a record is popular: if a query comes in and it has an entry in the record cache that is almost expired, it is marked as popular and a task is scheduled to refresh it. The client asking the question will still get the answer from the cache quickly and when a client then asks for the same record, it will find it refreshed (if the scheduled task has been run in the meantime).

Below we show a new graph adding the query times with the “Refresh Almost Expired” feature enabled by setting refresh-on-ttl-perc = 10. This means that if a cache entry is re-queried with less than 10% of its original TTL remaining, it will be re-fetched. This feature is available since Recursor 4.5.

What we see is that the yellow bars of the graph do not have the high peaks that the orange ones have. The relative small price we have to pay is that we do slightly more queries to authoritative name servers.

To be able to do decide when to refresh a Record Cache entry, the Record Cache has been extended to include the original TTL. That way, The Recursor can decide if a cache entry is almost expired: only a set percentage of the original TTL remains. We have chosen to use this simple method since it does not require the Recursor to keep track of statistical data to see which questions are popular, while still being able to not refresh cache entries that are not re-queried within the sample period that is a percentage of the TTL. Doing that would be a waste of time and resources.

An asynchronous task subsystem was built to implement background refreshing. This subsystem is also used for other tasks in the Recursor that do not require the Recursor to wait for them. We will likely see the task subsystem in future posts again.

Short TTL values become more and more popular, as they allow for quick updates of record sets, but they decrease cache effectiveness, put more load on resolvers and authoritative name servers and cause more clients having to wait for resolvers fetching record sets. The Refresh Almost Expired functionality mostly solves the issue of clients having to wait on answers but we would prefer domain administrators to use large TTL values as it reduces load on the whole of the DNS infrastructure.

First Beta Release of PowerDNS Recursor 4.7.0

We are proud to announce the first beta release of PowerDNS Recursor 4.7.0.

Compared to the previous major (4.6) release of PowerDNS Recursor, this release contains the following major changes:

  • A configurable way of adding Additional records to answers sent to the client, so the client does not have to ask for these records.
  • The step sizes for Query Minimization are now computed following to guidelines in RFC 9156.
  • The Recursor now schedules tasks to resolve IPv6 addresses of name servers not learned by glue records. This has the consequence that, if applicable, name servers will be contacted over IPv6 more often.
  • An experimental implementation of unilateral DoT probing. This allows the Recursor to learn if a an authoritative servers supports DoT.
  • Recursor has gained a way to fall back to parent NS set if contacting servers in the child NS set does not lead to an answer. This works around some broken authoritative servers configurations.
  • ZONEMD validation of the zones retrieved by the Zone to Cache, providing integrity guarantees for the zone retrieved.
  • The table recording round trip times of authoritative server IP addresses is now shared between threads to make it more effective and to reduce its memory footprint.
  • A Lua FFI hook for post-resolve interception: postresolve_ffi, providing a very fast way to do post-resolve Lua scripting.

As always, there are also many smaller bug fixes and improvements, please refer to the changelog for additional details. When upgrading do not forget to check the upgrade guide.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

The tarball (signature) is available from our download server and packages for several distributions are available from our repository.

With the final 4.7 release, the 4.4.x releases will be EOL and the 4.5.x and 4.6.x releases will go into critical fixes only mode. Consult the EOL policy for more details.

We would also like to mention that with the 4.5 release we stopped supporting systems using 32-bit time. This includes most 32-bit Linux platforms.

We also like to announce the upcoming removal of XPF support. If you are using this feature, plan switching to the proxy protocol.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

PowerDNS Authoritative Server 4.6.2

Hello!

Today we published release 4.6.2 of the Authoritative Server. It contains a carefully selected set of new features, plus a few bug fixes.

Please find a full list in the changelog.

Please make sure to read the Upgrade Notes before upgrading.

The tarball (signature) is available at downloads.powerdns.com and packages for various Linux distributions are available from repo.powerdns.com.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

PowerDNS Recursor 4.6.2 and 4.5.9 Released

We are proud to announce the release of PowerDNS Recursor 4.6.2 and 4.5.9.

Both releases are maintenance releases correcting an issue where a reload of a Lua script could cause in-flight queries to fail and an improvement in the caching of negative results. The 4.6.2 release contains a feature to disable processing of root-hints and improvements to the refreshing of “almost expired” records and processing of native dns64.

Please refer to the change logs for the 4.6.2 and 4.5.9 releases for additional details.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

The tarballs (4.6.2, 4.5.9) and signatures (4.6.2, 4.5.9) are available from our download server and packages for several distributions are available from our repository.

The 4.3.x releases are EOL and the 4.4.x and 4.5.x releases are in critical fixes only mode. Consult the EOL policy for more details.

We would also like to repeat that starting with the 4.5 release branch we will stop supporting systems using 32-bit time. This includes most 32-bit Linux platforms.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

Security Advisory 2022-01 for PowerDNS Authoritative Server 4.4.2, 4.5.3, 4.6.0 and PowerDNS Recursor 4.4.7, 4.5.7, 4.6.0

Hello,

Today we have released PowerDNS Authoritative Server 4.4.3, 4.5.4 and 4.6.1, and PowerDNS Recursor 4.4.8, 4.5.8 and 4.6.1 due to a low severity issue found in both products.

  • In the Authoritative server this issue only applies to secondary zones for which IXFR transfers have been enabled and the network path to the primary server is not trusted. Note that IXFR transfers are not enabled by default.
  • In the Recursor it applies to setups retrieving one or more RPZ zones from a remote server if the network path to the server is not trusted.

Tarballs and signatures are available at https://downloads.powerdns.com/releases/, and patches are available at https://downloads.powerdns.com/patches/2022-01/. However, the releases contain no other changes, with the exception of our EL8 builds, which were switched from CentOS 8 to Oracle Linux 8.

Please find the full text of the advisory below.


PowerDNS Security Advisory 2022-01: incomplete validation of incoming IXFR transfer in Authoritative Server and Recursor.

  • CVE: CVE-2022-27227
  • Date: 25th of March 2022.
  • Affects: PowerDNS Authoritative version 4.4.2, 4.5.3, 4.6.0 and PowerDNS Recursor 4.4.7, 4.5.7 and 4.6.0
  • Not affected: PowerDNS Authoritative Server 4.4.3, 4.5.4, 4.6.1 and PowerDNS Recursor 4.4.8, 4.5.8 and 4.6.1
  • Severity: Low
  • Impact: Denial of service
  • Exploit: This problem can be triggered by an attacker controlling the network path for IXFR transfers
  • Risk of system compromise: None
  • Solution: Upgrade to patched version, do not use IXFR in Authoritative Server

In the Authoritative server this issue only applies to secondary zones for which IXFR transfers have been enabled and the network path to the primary server is not trusted. Note that IXFR transfers are not enabled by default.

In the Recursor it applies to setups retrieving one or more RPZ zones from a remote server if the network path to the server is not trusted.

IXFR usually exchanges only the modifications between two versions of a zone, but sometimes needs to fall back to a full transfer of the current version.

When IXFR falls back to a full zone transfer, an attacker in position of man-in-the-middle can cause the transfer to be prematurely interrupted. This interrupted transfer is mistakenly interpreted as a complete transfer, causing an incomplete zone to be processed.

For the Authoritative Server, IXFR transfers are not enabled by default.
The Recursor only uses IXFR for retrieving RPZ zones. An incomplete RPZ transfer results in missing policy entries, potentially causing some DNS names and IP addresses to not be properly intercepted.

We would like to thank Nicolas Dehaine and Dmitry Shabanov from ThreatSTOP for reporting and initial analysis of this issue.

First Alpha Release of PowerDNS Recursor 4.7.0

We are proud to announce the first alpha release of PowerDNS Recursor 4.7.0.

Compared to the previous major (4.6) release of PowerDNS Recursor, this release contains the following major changes:

  • ZONEMD validation of the zones retrieved by the Zone to Cache feature.
  • A configurable way of adding Additional records to answers sent to the client.
  • The step sizes for Query Minimization are now computed following to guidelines in RFC 9156.
  • A Lua FFI hook for post-resolve interception: postresolve_ffi, to be documented before the final release.

There are also improvements in root-hints handling and resolving of IPv6 address of nameservers not learned by glue records. The latter has the consequence that, if applicable, nameservers will be contacted over IPv6 more often.

As always, there are also many smaller bug fixes and improvements, please refer to the changelog for additional details. When upgrading do not forget to check the upgrade guide.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

The tarball (signature) is available from our download server and packages for several distributions are available from our repository.

With the final 4.7 release, the 4.4.x releases will be EOL and the 4.5.x and 4.6.x releases will go into critical fixes only mode. Consult the EOL policy for more details.

We would also like to mention that with the 4.5 release we stopped supporting systems using 32-bit time. This includes 32-bit Linux platforms like arm6, arm7, and i386.

We also like to announce the upcoming removal of XPF support. If you are using this feature, plan switching to the proxy protocol.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

Authoritative Server 4.7.0-alpha1

Hello!

this is the first Alpha release for Authoritative Server 4.7.0. It brings a couple of new features into the hands of our users early.

New features:

  • lmdbbackend databases now get a UUID assigned, making it easy for external software to spot if a database was completely replaced
  • lmdbbackend databases now optionally use random IDs for objects
  • a new LUA function called ifurlextup
  • autoprimary management in pdnsutil and the HTTP API

A full list of changes can be found in the changelog.

Please make sure to read the Upgrade Notes before upgrading.

The tarball (signature) is available at downloads.powerdns.com. Packages for various distributions are available from repo.powerdns.com.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

Moving CentOS 8 builds to Oracle Linux 8

As you might be aware, CentOS 8 has reached End of Life on December 31st 2021. Furthermore, yesterday, CentOS 8 actually disappeared from the distribution mirrors. While we had made plans for this, we failed to execute those plans until now. This means we will need to switch build environments on some of our supported branches (Recursor and Authoritative 4.4/4.5/4.6, and dnsdist 1.5/1.6/1.7) mid release cycle. We are making those changes this week.

In mid-2021, we did extensive testing of building and running on the various CentOS alternatives, and came to one very clear conclusion – while the resulting binaries were not always bit for bit identical, the differences were uninteresting. Because of this, we believe users will not notice this change in our build environment at all and can continue to run our packages on their RHEL-derivative of choice.

However, just in case incompatible changes pop up, we are not switching the 7 build environment at this time.

Authoritative Server 4.6.0

Hello!

after a very useful beta/RC period in which we received some excellent bug reports, we released Authoritative Server version 4.6.0 today.

Version 4.6.0 mostly brings small improvements and fixes, but there are three notable new features:

  • support for incoming PROXY headers
  • support for EDNS cookies
  • autoprimary management via pdnsutil and the API

Support for PROXY headers allows you to put a load balancer (such as dnsdist) in front of the Authoritative Server, while still having the Auth see the actual IPs of clients talking to it.

EDNS Cookies allow resolvers that support it to have an extra layer of authentication on their communication with the Authoritative Server.

Compared to 4.6.0-alpha1, the major user visible change is the new NSEC3PARAM settings – check the upgrade docs below for more information. Besides that, various bugs have been fixed.

A full list of changes can be found in the changelog.

Please make sure to read the Upgrade Notes before upgrading.

The tarball (signature) is available at downloads.powerdns.com. Packages for various distributions are available from repo.powerdns.com.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.