Category: Uncategorized

PowerDNS Authoritative Server 4.6.3

Hello!

Today we published release 4.6.3 of the Authoritative Server. It contains two bug fixes, and marks the arrival of Ubuntu Jammy packages for the 4.6 branch.

Please find a full list in the changelog.

Please make sure to read the Upgrade Notes before upgrading.

The tarball (signature) is available at downloads.powerdns.com and packages for various Linux distributions are available from repo.powerdns.com.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

PowerDNS Recursor 4.7.1 Released

We are proud to announce the release of PowerDNS Recursor 4.7.1.

This release is a maintenance releases correcting an issue where asynchronous tasks would not be executed promptly. It also allows the generic record format in zone files loaded using the ZoneToCache function.

Please refer to the change log for the 4.7.1 release for additional details.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

The tarball and signature are available from our download server and packages for several distributions are available from our repository. The Ubuntu Jammy package will be published soon.

The 4.4.x releases are EOL and the 4.5.x and 4.6.x releases are in critical fixes only mode. Consult the EOL policy for more details.

We would also like to repeat that starting with the 4.5 release branch we stopped supporting systems using 32-bit time. This includes most 32-bit Linux platforms.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

Probing DoT Support of Authoritative Servers: Just Try It

This is the second part of a series of blog posts we are publishing, mostly around recent developments with respect to PowerDNS Recursor. The first blog post was Refreshing Of Almost Expired Records: Keeping The Cache Hot.

In PowerDNS Recursor 4.6.0 we introduced DNS over TLS (DoT) support for outgoing connections. Starting with that version, DoT is used by default for (forwarded) connections to port 853 and for a configurable list of authoritative servers. On the client-resolver side there has been developments to make DoT discovery easy. In this post we will discuss the DoT discovery from the resolver to authoritative server perspective. PowerDNS Recursor 4.7.0 has an experimental feature implementing this.

DoT

These days many DNS clients have the ability to switch to a DoT (or DNS over HTTPS, DoH) connection under certain circumstances, providing better security for the client-resolver channel. While DNSSEC provides authenticity proof of data received from an authoritative server, it does not provide confidentiality and in general it does not protect the channel between the resolver and the client: most clients do not support DNSSEC validation. DoT (and DoH) fill parts of this gap. It is less well known that DoT can also be used to secure the channels between a resolver and the authoritative servers. Common to both applications of DoT is the question when to use DoT instead of regular DNS queries over UDP or TCP to port 53, commonly called Do53.

Client-Resolver DoT Discovery

In the case of client-resolver communication there is a (draft) standard that specifies how an client can discover if a resolver supports DoT: Discovery of Designated Resolvers (DDR). In short, using DNS queries with name _dns.example.net (or _dns.resolver.arpa if it does not know the name) and query type SVCB, a client can learn if a resolver supports DoT. The draft also contains rules to make sure a validated DoT connection can be set up if needed. All this allows a secure upgrade to DoT without user or administrator intervention. The major public resolvers all support DDR and if you run a resolver yourself, it is not difficult to set it up to return the proper record(s) for DDR queries.

Resolver-Authoritative DoT Discovery

Sadly, the case for resolver to authoritative server traffic is not as simple. At the moment of writing, the committee concerned with these things was not able to come up with anything better than Unilateral Opportunistic Deployment of Recursive-to-Authoritative DNS, which in essence boils down to: let the resolver find out if DoT is available by trying a DoT Connection to the IP address of the authoritative server and remembering the results for a while. At this moment most authoritative servers do not support DoT. A DoT connection is likely to time out, using resources during the attempt. There are also no guidelines on certificate checking of the DoT connection, the draft says any certificate must be accepted. Even if this is not very satisfying we set out to implement this mechanism as an experimental feature in PowerDNS Recursor 4.7.0. We hope in the future a better mechanism to discover DoT support of authoritative servers will be drafted. If so, we certainly intend to implement this better method.

Implementing DoT of authoritative server discovery

PowerDNS Recursor maintains several in-memory tables with characteristics of authoritative servers. An example of this information is the speed of each authoritative server contacted. The speed information is used to select the authoritative expected to be fastest when a query to an authoritative server of a domain needs to be done. Following the guidelines in the draft mentioned above, we introduced a new table to record DoT probe activity and results.

When an authoritative server needs to be contacted, this table is consulted to see if we already know if this server supports DoT. There are several cases:

  • If the server IP is not in the table because we did not contact it recently, we create an entry in the table and mark the DoT status Unknown and do a Do53 query
  • If the server is known to not support DoT, we do a Do53 query.
  • If the server is known to support DoT, we do a DoT query.
  • If we find the status is Unknown, we do a regular Do53 query and we schedule an asynchronous task to probe for DoT support. The table entry is marked Busy in this case.

if a task is submitted, a separate task mechanism in the recursor picks up the task and tries a DoT connection to the associated IP. If that works and a DNS query is also successful over the newly established DoT connection, the IP is marked as DoT capable (Good). If the DoT connections times out or the query is not successful, the IP is marked as Bad.

While the DoT status is not yet determined, we use regular DNS queries to contact a specific authoritative server, since we do not want to burden clients with the long (a couple of seconds) time outs involved in unsuccessful DoT probes.

The housekeeping of the table involves a bit more work, so that old information is expired or refreshed properly and to make sure that we do not overwhelm ourselves or authoritative servers with a lot of simultaneous DoT probes. The draft has some guidelines for doing this properly. There are also more complex issues: what do we do if a DoT query fails to a server we marked as DoT capable? Do we mark the IP as not DoT capable or not? It is likely not good to downgrade a server that has been serving DoT for many queries on the failure of a single query. These open questions and the hope that a draft that provides a better method will appear are the reasons this feature is marked as experimental.

The draft mentioned above also recommends the information learned to be persistent, the resolver should store the information while running and re-read the information on start-up. We do not implement that part yet.

The mechanism we implemented decouples the use of DoT from the discovery process. This means that in the future, alternative discovery mechanisms (specifying how) or policies (specifying when and in what order) can be implemented without interfering with other parts of the recursor. At the moment the probing process is lazy: tasks will be executed when the recursor is processing queries. An idle recursor will execute tasks only sporadically. Because of this, probe tasks executed will have no immediate relation to the queries executed on behalf of clients. This will be improved upon in the near future.

Trying it yourself

Using PowerDNS Recursor 4.7.0, you can try this yourself by setting max-busy-dot-probes to a non-zero value. This configuration governs the maximum number of simultaneous DoT probes. After the recursor has been busy for a while, you can look at the status of the DoT probes by using

rec_control dump-dot-probe-map -

To see a list of know authoritative servers and their DoT support status. A few example lines are shown below

23.61.199.67    akam.net.               1       Bad     2022-05-26T14:44:10
2.22.230.67     akam.net.               1       Busy    2022-05-26T14:44:21
184.85.248.67   akam.net.               1       Bad     2022-05-26T14:43:59
88.221.25.85    dscx.akamaiedge.net.    0       Unknown 2022-05-26T14:40:46
188.166.104.87  powerdns.com.           3       Good    2022-05-28T15:21:06
2.19.195.108    dscx.akamaiedge.net.    0       Unknown 2022-05-26T14:40:26
23.74.25.128    akagtm.org.             1       Bad     2022-05-26T14:41:16

A server marked Unknown is not probed yet, only servers that are visited at least twice within an interval get probed. The status can also be Bad (no DoT support), Good (DoT support) or Busy (DoT probe scheduled or running).

Conclusion

While we’ve implemented the DoT discovery mechanism as an experimental feature in PowerDNS Recursor 4.7.0, we consider it a half-measure, a waypoint if you will. We hope a future draft will provide more guidance on the how and when of the discovery process of DoT support between resolver and authoritative servers, which should lead to better security for DNS users.

PowerDNS Recursor 4.7.0 Released

We are proud to announce the release of PowerDNS Recursor 4.7.0.

Compared to the previous major (4.6) release of PowerDNS Recursor, this release contains the following major changes:

  • A configurable way of adding Additional records to answers sent to the client, so the client does not have to ask for these records.
  • The step sizes for Query Minimization are now computed following to guidelines in RFC 9156.
  • The Recursor now schedules tasks to resolve IPv6 addresses of name servers not learned by glue records. This has the consequence that, if applicable, name servers will be contacted over IPv6 more often.
  • An experimental implementation of unilateral DoT probing. This allows the Recursor to learn if a an authoritative servers supports DoT.
  • Recursor has gained a way to fall back to the parent NS set if contacting servers in the child NS set does not lead to an answer. This works around some broken authoritative servers configurations.
  • ZONEMD validation of the zones retrieved by the Zone to Cache, providing integrity guarantees for the zone retrieved.
  • The table recording round trip times of authoritative server IP addresses is now shared between threads to make it more effective and to reduce its memory footprint.
  • A Lua FFI hook for post-resolve interception: postresolve_ffi, providing a very fast way to do post-resolve Lua scripting.

As always, there are also many smaller bug fixes and improvements, please refer to the changelog for additional details. When upgrading do not forget to check the upgrade guide.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

The tarball (signature) is available from our download server and packages for several distributions are available from our repository.

With this 4.7.0 release, the 4.4.x releases will be EOL and the 4.5.x and 4.6.x releases will go into critical fixes only mode. Consult the EOL policy for more details.

We would also like to mention that with the 4.5 release we stopped supporting systems using 32-bit time. This includes most 32-bit Linux platforms.

We also like to announce the upcoming removal of XPF support. If you are using this feature, plan switching to the proxy protocol.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

First Release Candidate of PowerDNS Recursor 4.7.0

We are proud to announce the first release candidate of PowerDNS Recursor 4.7.0. Testing of this release candidate is much appreciated!

The most important change compared to the 4.7.0-beta1 release is a fix for the experimental DoT to authoritative server probing code.

Compared to the previous major (4.6) release of PowerDNS Recursor, this release contains the following major changes:

  • A configurable way of adding Additional records to answers sent to the client, so the client does not have to ask for these records.
  • The step sizes for Query Minimization are now computed following to guidelines in RFC 9156.
  • The Recursor now schedules tasks to resolve IPv6 addresses of name servers not learned by glue records. This has the consequence that, if applicable, name servers will be contacted over IPv6 more often.
  • An experimental implementation of unilateral DoT probing. This allows the Recursor to learn if a an authoritative servers supports DoT.
  • Recursor has gained a way to fall back to parent NS set if contacting servers in the child NS set does not lead to an answer. This works around some broken authoritative servers configurations.
  • ZONEMD validation of the zones retrieved by the Zone to Cache, providing integrity guarantees for the zone retrieved.
  • The table recording round trip times of authoritative server IP addresses is now shared between threads to make it more effective and to reduce its memory footprint.
  • A Lua FFI hook for post-resolve interception: postresolve_ffi, providing a very fast way to do post-resolve Lua scripting.

As always, there are also many smaller bug fixes and improvements, please refer to the changelog for additional details. When upgrading do not forget to check the upgrade guide.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

The tarball (signature) is available from our download server and packages for several distributions are available from our repository.

With the final 4.7 release, the 4.4.x releases will be EOL and the 4.5.x and 4.6.x releases will go into critical fixes only mode. Consult the EOL policy for more details.

We would also like to mention that with the 4.5 release we stopped supporting systems using 32-bit time. This includes most 32-bit Linux platforms.

We also like to announce the upcoming removal of XPF support. If you are using this feature, plan switching to the proxy protocol.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

Refreshing Of Almost Expired Records: Keeping The Cache Hot

We’re planning to do a series of posts to highlight some of the features we have been working in recent releases or the Recursor. We also plan to discuss some C++ programming techniques we used in the Recursor that might be interesting. The first post in this series describes a technique that was introduced in PowerDNS Recursor 4.5: Refresh Almost Expired.

Introduction

When the Recursor receives a query from a client it answers it by first finding the name server that is authoritative for the domain in question. It does that by starting at a root name server and then walking DNS delegations to find the name servers (and their addresses) that are authoritative for the question asked. Once it knows which name servers are authoritative for the domain, it picks a specific name server address and asks the question to that name server. After the Recursor receives the answer, it will pass it on the the client asking the question.

To be able to answer fast, the Recursor caches information it receives from authoritative name servers. Both information about delegations and specific queries is stored in the record cache, which can be seen as a local in-memory copy of parts of the global DNS tree.

The recursor also has a few other caches, for example the Packet Cache and the Negative Cache, but these are not the subject of this post.

The Record Cache

The Record Cache contains entries of the following form

name, type, time to die, record set, ...

The entries are indexed and searchable by a name and type combination; time to die (TTD) is the time the entry will expire and record set is the set of DNS records associated with the name and type. When inserting an entry into the Record Cache the TTD is computed by taking the time to live (TTL) of the record set received from the authoritative name server and adding the current time to it. That way, it becomes easy to see if a record in the Record Cache is expired: just compare the TTD to the current time.

Query Processing

With the Record Cache, query processing becomes a bit more complicated:

1. Check to see if the Record Cache contains an entry (name, type) 
2. If an entry is present and not expired: return the answer using the
   record set
3. If the entry is not present, continue by finding the authoritative server 
   (using the Record Cache or the internet) as before and asking it the 
   question.
4. If an answer is received from an authoritative server, insert it into the 
   Record Cache and return it to the client.

Clients will have a better experience for cached record sets: the Recursor will be able to answer cached queries much faster than when it has to go to the internet to contact one or more authoritative servers to get an answer.

Caching Issues

When using a cache, it is always important to keep consistency in mind: the cache should reflect the contents of the records of the authoritative name servers on the internet. Here we have an issue: we keep a record set in the cache for a maximum period equal to the TTL. If an authoritative server changes a record’s content in between, we do not know that until we re-fetch the record set and we won’t do that until the record expires. The general consensus is that we accept this issue, since caching DNS records make the Recursor very fast compared to always asking authoritative name servers. This caching period is the main reason why it takes time for an updated DNS record to appear in the caches of resolvers and returned to clients.

Another issue is making sure the cache is effective: in an ideal world, we only want to store records that are queried again within the TTL period. Otherwise we waste resources storing data that just expires after a while without being used. Sadly we do not have a way to know beforehand if a record is going to be retrieved from the cache after we have stored it. We somewhat solve this issue by making sure that if records need to be evicted from the cache because it is too full, we pick the records using a least recently used (LRU) method in addition to having a preference to evict expired records.

The Client Experience

When a client ask a “popular” question, often it will get an answer from the cache, as popular record sets tend to be present in the cache. But once in a while an unlucky client has to wait a bit, as the record in the cache is expired and the Recursor has to go out on the net to find the answer. The chart below illustrates what happens when querying a record set with a short TTL 60 times with a one second interval, starting with a recently filled cache:

Of the 60 queries done, most are answered within about 1ms, but three take significantly more .The peaks are 20s apart, the TTL of the record set in question.

Wouldn’t it be nice if there was a way to avoid that some clients will have to wait much longer for an answer than other clients? What if we would be able to fetch an updated record set before it expires from the record cache?

Keeping the Cache Hot

The jargon word for a cache that contains data you are looking for is a hot cache.

To try to make sure that entries in the cache are up to date the Recursor can be configured to issue queries to refresh them in the background when it sees that a record set is popular and almost expired. The Recursor uses a rather simple way to decide if a record is popular: if a query comes in and it has an entry in the record cache that is almost expired, it is marked as popular and a task is scheduled to refresh it. The client asking the question will still get the answer from the cache quickly and when a client then asks for the same record, it will find it refreshed (if the scheduled task has been run in the meantime).

Below we show a new graph adding the query times with the “Refresh Almost Expired” feature enabled by setting refresh-on-ttl-perc = 10. This means that if a cache entry is re-queried with less than 10% of its original TTL remaining, it will be re-fetched. This feature is available since Recursor 4.5.

What we see is that the yellow bars of the graph do not have the high peaks that the orange ones have. The relative small price we have to pay is that we do slightly more queries to authoritative name servers.

To be able to do decide when to refresh a Record Cache entry, the Record Cache has been extended to include the original TTL. That way, The Recursor can decide if a cache entry is almost expired: only a set percentage of the original TTL remains. We have chosen to use this simple method since it does not require the Recursor to keep track of statistical data to see which questions are popular, while still being able to not refresh cache entries that are not re-queried within the sample period that is a percentage of the TTL. Doing that would be a waste of time and resources.

An asynchronous task subsystem was built to implement background refreshing. This subsystem is also used for other tasks in the Recursor that do not require the Recursor to wait for them. We will likely see the task subsystem in future posts again.

Short TTL values become more and more popular, as they allow for quick updates of record sets, but they decrease cache effectiveness, put more load on resolvers and authoritative name servers and cause more clients having to wait for resolvers fetching record sets. The Refresh Almost Expired functionality mostly solves the issue of clients having to wait on answers but we would prefer domain administrators to use large TTL values as it reduces load on the whole of the DNS infrastructure.

First Beta Release of PowerDNS Recursor 4.7.0

We are proud to announce the first beta release of PowerDNS Recursor 4.7.0.

Compared to the previous major (4.6) release of PowerDNS Recursor, this release contains the following major changes:

  • A configurable way of adding Additional records to answers sent to the client, so the client does not have to ask for these records.
  • The step sizes for Query Minimization are now computed following to guidelines in RFC 9156.
  • The Recursor now schedules tasks to resolve IPv6 addresses of name servers not learned by glue records. This has the consequence that, if applicable, name servers will be contacted over IPv6 more often.
  • An experimental implementation of unilateral DoT probing. This allows the Recursor to learn if a an authoritative servers supports DoT.
  • Recursor has gained a way to fall back to parent NS set if contacting servers in the child NS set does not lead to an answer. This works around some broken authoritative servers configurations.
  • ZONEMD validation of the zones retrieved by the Zone to Cache, providing integrity guarantees for the zone retrieved.
  • The table recording round trip times of authoritative server IP addresses is now shared between threads to make it more effective and to reduce its memory footprint.
  • A Lua FFI hook for post-resolve interception: postresolve_ffi, providing a very fast way to do post-resolve Lua scripting.

As always, there are also many smaller bug fixes and improvements, please refer to the changelog for additional details. When upgrading do not forget to check the upgrade guide.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

The tarball (signature) is available from our download server and packages for several distributions are available from our repository.

With the final 4.7 release, the 4.4.x releases will be EOL and the 4.5.x and 4.6.x releases will go into critical fixes only mode. Consult the EOL policy for more details.

We would also like to mention that with the 4.5 release we stopped supporting systems using 32-bit time. This includes most 32-bit Linux platforms.

We also like to announce the upcoming removal of XPF support. If you are using this feature, plan switching to the proxy protocol.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

PowerDNS Authoritative Server 4.6.2

Hello!

Today we published release 4.6.2 of the Authoritative Server. It contains a carefully selected set of new features, plus a few bug fixes.

Please find a full list in the changelog.

Please make sure to read the Upgrade Notes before upgrading.

The tarball (signature) is available at downloads.powerdns.com and packages for various Linux distributions are available from repo.powerdns.com.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

PowerDNS Recursor 4.6.2 and 4.5.9 Released

We are proud to announce the release of PowerDNS Recursor 4.6.2 and 4.5.9.

Both releases are maintenance releases correcting an issue where a reload of a Lua script could cause in-flight queries to fail and an improvement in the caching of negative results. The 4.6.2 release contains a feature to disable processing of root-hints and improvements to the refreshing of “almost expired” records and processing of native dns64.

Please refer to the change logs for the 4.6.2 and 4.5.9 releases for additional details.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

The tarballs (4.6.2, 4.5.9) and signatures (4.6.2, 4.5.9) are available from our download server and packages for several distributions are available from our repository.

The 4.3.x releases are EOL and the 4.4.x and 4.5.x releases are in critical fixes only mode. Consult the EOL policy for more details.

We would also like to repeat that starting with the 4.5 release branch we will stop supporting systems using 32-bit time. This includes most 32-bit Linux platforms.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

Security Advisory 2022-01 for PowerDNS Authoritative Server 4.4.2, 4.5.3, 4.6.0 and PowerDNS Recursor 4.4.7, 4.5.7, 4.6.0

Hello,

Today we have released PowerDNS Authoritative Server 4.4.3, 4.5.4 and 4.6.1, and PowerDNS Recursor 4.4.8, 4.5.8 and 4.6.1 due to a low severity issue found in both products.

  • In the Authoritative server this issue only applies to secondary zones for which IXFR transfers have been enabled and the network path to the primary server is not trusted. Note that IXFR transfers are not enabled by default.
  • In the Recursor it applies to setups retrieving one or more RPZ zones from a remote server if the network path to the server is not trusted.

Tarballs and signatures are available at https://downloads.powerdns.com/releases/, and patches are available at https://downloads.powerdns.com/patches/2022-01/. However, the releases contain no other changes, with the exception of our EL8 builds, which were switched from CentOS 8 to Oracle Linux 8.

Please find the full text of the advisory below.


PowerDNS Security Advisory 2022-01: incomplete validation of incoming IXFR transfer in Authoritative Server and Recursor.

  • CVE: CVE-2022-27227
  • Date: 25th of March 2022.
  • Affects: PowerDNS Authoritative version 4.4.2, 4.5.3, 4.6.0 and PowerDNS Recursor 4.4.7, 4.5.7 and 4.6.0
  • Not affected: PowerDNS Authoritative Server 4.4.3, 4.5.4, 4.6.1 and PowerDNS Recursor 4.4.8, 4.5.8 and 4.6.1
  • Severity: Low
  • Impact: Denial of service
  • Exploit: This problem can be triggered by an attacker controlling the network path for IXFR transfers
  • Risk of system compromise: None
  • Solution: Upgrade to patched version, do not use IXFR in Authoritative Server

In the Authoritative server this issue only applies to secondary zones for which IXFR transfers have been enabled and the network path to the primary server is not trusted. Note that IXFR transfers are not enabled by default.

In the Recursor it applies to setups retrieving one or more RPZ zones from a remote server if the network path to the server is not trusted.

IXFR usually exchanges only the modifications between two versions of a zone, but sometimes needs to fall back to a full transfer of the current version.

When IXFR falls back to a full zone transfer, an attacker in position of man-in-the-middle can cause the transfer to be prematurely interrupted. This interrupted transfer is mistakenly interpreted as a complete transfer, causing an incomplete zone to be processed.

For the Authoritative Server, IXFR transfers are not enabled by default.
The Recursor only uses IXFR for retrieving RPZ zones. An incomplete RPZ transfer results in missing policy entries, potentially causing some DNS names and IP addresses to not be properly intercepted.

We would like to thank Nicolas Dehaine and Dmitry Shabanov from ThreatSTOP for reporting and initial analysis of this issue.