PowerDNS Authoritative Server 4.6.3

Hello!

Today we published release 4.6.3 of the Authoritative Server. It contains two bug fixes, and marks the arrival of Ubuntu Jammy packages for the 4.6 branch.

Please find a full list in the changelog.

Please make sure to read the Upgrade Notes before upgrading.

The tarball (signature) is available at downloads.powerdns.com and packages for various Linux distributions are available from repo.powerdns.com.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

PowerDNS Recursor 4.7.1 Released

We are proud to announce the release of PowerDNS Recursor 4.7.1.

This release is a maintenance releases correcting an issue where asynchronous tasks would not be executed promptly. It also allows the generic record format in zone files loaded using the ZoneToCache function.

Please refer to the change log for the 4.7.1 release for additional details.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

The tarball and signature are available from our download server and packages for several distributions are available from our repository. The Ubuntu Jammy package will be published soon.

The 4.4.x releases are EOL and the 4.5.x and 4.6.x releases are in critical fixes only mode. Consult the EOL policy for more details.

We would also like to repeat that starting with the 4.5 release branch we stopped supporting systems using 32-bit time. This includes most 32-bit Linux platforms.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

dnsdist-1.7.2 released

Hello!

We are very happy to release dnsdist 1.7.2 today, a maintenance release fixing a few bugs reported since 1.7.1:

  • An unhandled exception could happen when an invalid protocol was used in an incoming DNS over HTTPS forwarded-for header and passed to the backend via the proxy protocol, leading to a use-after-free and a crash. Forwarded-for headers are not used by default and should only be used if the client can be trusted (#11667)
  • An invalid proxy-protocol was sent to the backend, over TCP, if a query received via DNS over HTTPS resulted in a truncated UDP response from the backend (#11665)
  • Some metrics lacked a proper description in our Prometheus endpoint (#11664)
  • A side-effect of fixing the health-check timeout in 1.7.1 was leading to a CPU usage increase on devices that are mostly idle. We improved that situation, reducing the CPU usage even below what it was in 1.7.0 (#11579, #11580)

We also added a couple Lua bindings to make it easier to look into the DNS payload from custom Lua rules and actions (#11666).

Please see the dnsdist website for the more complete changelog and the current documentation.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

The release tarball and its signature are available on the downloads website, and packages for several distributions are available from our repository.

Probing DoT Support of Authoritative Servers: Just Try It

This is the second part of a series of blog posts we are publishing, mostly around recent developments with respect to PowerDNS Recursor. The first blog post was Refreshing Of Almost Expired Records: Keeping The Cache Hot.

In PowerDNS Recursor 4.6.0 we introduced DNS over TLS (DoT) support for outgoing connections. Starting with that version, DoT is used by default for (forwarded) connections to port 853 and for a configurable list of authoritative servers. On the client-resolver side there has been developments to make DoT discovery easy. In this post we will discuss the DoT discovery from the resolver to authoritative server perspective. PowerDNS Recursor 4.7.0 has an experimental feature implementing this.

DoT

These days many DNS clients have the ability to switch to a DoT (or DNS over HTTPS, DoH) connection under certain circumstances, providing better security for the client-resolver channel. While DNSSEC provides authenticity proof of data received from an authoritative server, it does not provide confidentiality and in general it does not protect the channel between the resolver and the client: most clients do not support DNSSEC validation. DoT (and DoH) fill parts of this gap. It is less well known that DoT can also be used to secure the channels between a resolver and the authoritative servers. Common to both applications of DoT is the question when to use DoT instead of regular DNS queries over UDP or TCP to port 53, commonly called Do53.

Client-Resolver DoT Discovery

In the case of client-resolver communication there is a (draft) standard that specifies how an client can discover if a resolver supports DoT: Discovery of Designated Resolvers (DDR). In short, using DNS queries with name _dns.example.net (or _dns.resolver.arpa if it does not know the name) and query type SVCB, a client can learn if a resolver supports DoT. The draft also contains rules to make sure a validated DoT connection can be set up if needed. All this allows a secure upgrade to DoT without user or administrator intervention. The major public resolvers all support DDR and if you run a resolver yourself, it is not difficult to set it up to return the proper record(s) for DDR queries.

Resolver-Authoritative DoT Discovery

Sadly, the case for resolver to authoritative server traffic is not as simple. At the moment of writing, the committee concerned with these things was not able to come up with anything better than Unilateral Opportunistic Deployment of Recursive-to-Authoritative DNS, which in essence boils down to: let the resolver find out if DoT is available by trying a DoT Connection to the IP address of the authoritative server and remembering the results for a while. At this moment most authoritative servers do not support DoT. A DoT connection is likely to time out, using resources during the attempt. There are also no guidelines on certificate checking of the DoT connection, the draft says any certificate must be accepted. Even if this is not very satisfying we set out to implement this mechanism as an experimental feature in PowerDNS Recursor 4.7.0. We hope in the future a better mechanism to discover DoT support of authoritative servers will be drafted. If so, we certainly intend to implement this better method.

Implementing DoT of authoritative server discovery

PowerDNS Recursor maintains several in-memory tables with characteristics of authoritative servers. An example of this information is the speed of each authoritative server contacted. The speed information is used to select the authoritative expected to be fastest when a query to an authoritative server of a domain needs to be done. Following the guidelines in the draft mentioned above, we introduced a new table to record DoT probe activity and results.

When an authoritative server needs to be contacted, this table is consulted to see if we already know if this server supports DoT. There are several cases:

  • If the server IP is not in the table because we did not contact it recently, we create an entry in the table and mark the DoT status Unknown and do a Do53 query
  • If the server is known to not support DoT, we do a Do53 query.
  • If the server is known to support DoT, we do a DoT query.
  • If we find the status is Unknown, we do a regular Do53 query and we schedule an asynchronous task to probe for DoT support. The table entry is marked Busy in this case.

if a task is submitted, a separate task mechanism in the recursor picks up the task and tries a DoT connection to the associated IP. If that works and a DNS query is also successful over the newly established DoT connection, the IP is marked as DoT capable (Good). If the DoT connections times out or the query is not successful, the IP is marked as Bad.

While the DoT status is not yet determined, we use regular DNS queries to contact a specific authoritative server, since we do not want to burden clients with the long (a couple of seconds) time outs involved in unsuccessful DoT probes.

The housekeeping of the table involves a bit more work, so that old information is expired or refreshed properly and to make sure that we do not overwhelm ourselves or authoritative servers with a lot of simultaneous DoT probes. The draft has some guidelines for doing this properly. There are also more complex issues: what do we do if a DoT query fails to a server we marked as DoT capable? Do we mark the IP as not DoT capable or not? It is likely not good to downgrade a server that has been serving DoT for many queries on the failure of a single query. These open questions and the hope that a draft that provides a better method will appear are the reasons this feature is marked as experimental.

The draft mentioned above also recommends the information learned to be persistent, the resolver should store the information while running and re-read the information on start-up. We do not implement that part yet.

The mechanism we implemented decouples the use of DoT from the discovery process. This means that in the future, alternative discovery mechanisms (specifying how) or policies (specifying when and in what order) can be implemented without interfering with other parts of the recursor. At the moment the probing process is lazy: tasks will be executed when the recursor is processing queries. An idle recursor will execute tasks only sporadically. Because of this, probe tasks executed will have no immediate relation to the queries executed on behalf of clients. This will be improved upon in the near future.

Trying it yourself

Using PowerDNS Recursor 4.7.0, you can try this yourself by setting max-busy-dot-probes to a non-zero value. This configuration governs the maximum number of simultaneous DoT probes. After the recursor has been busy for a while, you can look at the status of the DoT probes by using

rec_control dump-dot-probe-map -

To see a list of know authoritative servers and their DoT support status. A few example lines are shown below

23.61.199.67    akam.net.               1       Bad     2022-05-26T14:44:10
2.22.230.67     akam.net.               1       Busy    2022-05-26T14:44:21
184.85.248.67   akam.net.               1       Bad     2022-05-26T14:43:59
88.221.25.85    dscx.akamaiedge.net.    0       Unknown 2022-05-26T14:40:46
188.166.104.87  powerdns.com.           3       Good    2022-05-28T15:21:06
2.19.195.108    dscx.akamaiedge.net.    0       Unknown 2022-05-26T14:40:26
23.74.25.128    akagtm.org.             1       Bad     2022-05-26T14:41:16

A server marked Unknown is not probed yet, only servers that are visited at least twice within an interval get probed. The status can also be Bad (no DoT support), Good (DoT support) or Busy (DoT probe scheduled or running).

Conclusion

While we’ve implemented the DoT discovery mechanism as an experimental feature in PowerDNS Recursor 4.7.0, we consider it a half-measure, a waypoint if you will. We hope a future draft will provide more guidance on the how and when of the discovery process of DoT support between resolver and authoritative servers, which should lead to better security for DNS users.

PowerDNS Recursor 4.7.0 Released

We are proud to announce the release of PowerDNS Recursor 4.7.0.

Compared to the previous major (4.6) release of PowerDNS Recursor, this release contains the following major changes:

  • A configurable way of adding Additional records to answers sent to the client, so the client does not have to ask for these records.
  • The step sizes for Query Minimization are now computed following to guidelines in RFC 9156.
  • The Recursor now schedules tasks to resolve IPv6 addresses of name servers not learned by glue records. This has the consequence that, if applicable, name servers will be contacted over IPv6 more often.
  • An experimental implementation of unilateral DoT probing. This allows the Recursor to learn if a an authoritative servers supports DoT.
  • Recursor has gained a way to fall back to the parent NS set if contacting servers in the child NS set does not lead to an answer. This works around some broken authoritative servers configurations.
  • ZONEMD validation of the zones retrieved by the Zone to Cache, providing integrity guarantees for the zone retrieved.
  • The table recording round trip times of authoritative server IP addresses is now shared between threads to make it more effective and to reduce its memory footprint.
  • A Lua FFI hook for post-resolve interception: postresolve_ffi, providing a very fast way to do post-resolve Lua scripting.

As always, there are also many smaller bug fixes and improvements, please refer to the changelog for additional details. When upgrading do not forget to check the upgrade guide.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

The tarball (signature) is available from our download server and packages for several distributions are available from our repository.

With this 4.7.0 release, the 4.4.x releases will be EOL and the 4.5.x and 4.6.x releases will go into critical fixes only mode. Consult the EOL policy for more details.

We would also like to mention that with the 4.5 release we stopped supporting systems using 32-bit time. This includes most 32-bit Linux platforms.

We also like to announce the upcoming removal of XPF support. If you are using this feature, plan switching to the proxy protocol.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

First Release Candidate of PowerDNS Recursor 4.7.0

We are proud to announce the first release candidate of PowerDNS Recursor 4.7.0. Testing of this release candidate is much appreciated!

The most important change compared to the 4.7.0-beta1 release is a fix for the experimental DoT to authoritative server probing code.

Compared to the previous major (4.6) release of PowerDNS Recursor, this release contains the following major changes:

  • A configurable way of adding Additional records to answers sent to the client, so the client does not have to ask for these records.
  • The step sizes for Query Minimization are now computed following to guidelines in RFC 9156.
  • The Recursor now schedules tasks to resolve IPv6 addresses of name servers not learned by glue records. This has the consequence that, if applicable, name servers will be contacted over IPv6 more often.
  • An experimental implementation of unilateral DoT probing. This allows the Recursor to learn if a an authoritative servers supports DoT.
  • Recursor has gained a way to fall back to parent NS set if contacting servers in the child NS set does not lead to an answer. This works around some broken authoritative servers configurations.
  • ZONEMD validation of the zones retrieved by the Zone to Cache, providing integrity guarantees for the zone retrieved.
  • The table recording round trip times of authoritative server IP addresses is now shared between threads to make it more effective and to reduce its memory footprint.
  • A Lua FFI hook for post-resolve interception: postresolve_ffi, providing a very fast way to do post-resolve Lua scripting.

As always, there are also many smaller bug fixes and improvements, please refer to the changelog for additional details. When upgrading do not forget to check the upgrade guide.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

The tarball (signature) is available from our download server and packages for several distributions are available from our repository.

With the final 4.7 release, the 4.4.x releases will be EOL and the 4.5.x and 4.6.x releases will go into critical fixes only mode. Consult the EOL policy for more details.

We would also like to mention that with the 4.5 release we stopped supporting systems using 32-bit time. This includes most 32-bit Linux platforms.

We also like to announce the upcoming removal of XPF support. If you are using this feature, plan switching to the proxy protocol.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

Refreshing Of Almost Expired Records: Keeping The Cache Hot

We’re planning to do a series of posts to highlight some of the features we have been working in recent releases or the Recursor. We also plan to discuss some C++ programming techniques we used in the Recursor that might be interesting. The first post in this series describes a technique that was introduced in PowerDNS Recursor 4.5: Refresh Almost Expired.

Introduction

When the Recursor receives a query from a client it answers it by first finding the name server that is authoritative for the domain in question. It does that by starting at a root name server and then walking DNS delegations to find the name servers (and their addresses) that are authoritative for the question asked. Once it knows which name servers are authoritative for the domain, it picks a specific name server address and asks the question to that name server. After the Recursor receives the answer, it will pass it on the the client asking the question.

To be able to answer fast, the Recursor caches information it receives from authoritative name servers. Both information about delegations and specific queries is stored in the record cache, which can be seen as a local in-memory copy of parts of the global DNS tree.

The recursor also has a few other caches, for example the Packet Cache and the Negative Cache, but these are not the subject of this post.

The Record Cache

The Record Cache contains entries of the following form

name, type, time to die, record set, ...

The entries are indexed and searchable by a name and type combination; time to die (TTD) is the time the entry will expire and record set is the set of DNS records associated with the name and type. When inserting an entry into the Record Cache the TTD is computed by taking the time to live (TTL) of the record set received from the authoritative name server and adding the current time to it. That way, it becomes easy to see if a record in the Record Cache is expired: just compare the TTD to the current time.

Query Processing

With the Record Cache, query processing becomes a bit more complicated:

1. Check to see if the Record Cache contains an entry (name, type) 
2. If an entry is present and not expired: return the answer using the
   record set
3. If the entry is not present, continue by finding the authoritative server 
   (using the Record Cache or the internet) as before and asking it the 
   question.
4. If an answer is received from an authoritative server, insert it into the 
   Record Cache and return it to the client.

Clients will have a better experience for cached record sets: the Recursor will be able to answer cached queries much faster than when it has to go to the internet to contact one or more authoritative servers to get an answer.

Caching Issues

When using a cache, it is always important to keep consistency in mind: the cache should reflect the contents of the records of the authoritative name servers on the internet. Here we have an issue: we keep a record set in the cache for a maximum period equal to the TTL. If an authoritative server changes a record’s content in between, we do not know that until we re-fetch the record set and we won’t do that until the record expires. The general consensus is that we accept this issue, since caching DNS records make the Recursor very fast compared to always asking authoritative name servers. This caching period is the main reason why it takes time for an updated DNS record to appear in the caches of resolvers and returned to clients.

Another issue is making sure the cache is effective: in an ideal world, we only want to store records that are queried again within the TTL period. Otherwise we waste resources storing data that just expires after a while without being used. Sadly we do not have a way to know beforehand if a record is going to be retrieved from the cache after we have stored it. We somewhat solve this issue by making sure that if records need to be evicted from the cache because it is too full, we pick the records using a least recently used (LRU) method in addition to having a preference to evict expired records.

The Client Experience

When a client ask a “popular” question, often it will get an answer from the cache, as popular record sets tend to be present in the cache. But once in a while an unlucky client has to wait a bit, as the record in the cache is expired and the Recursor has to go out on the net to find the answer. The chart below illustrates what happens when querying a record set with a short TTL 60 times with a one second interval, starting with a recently filled cache:

Of the 60 queries done, most are answered within about 1ms, but three take significantly more .The peaks are 20s apart, the TTL of the record set in question.

Wouldn’t it be nice if there was a way to avoid that some clients will have to wait much longer for an answer than other clients? What if we would be able to fetch an updated record set before it expires from the record cache?

Keeping the Cache Hot

The jargon word for a cache that contains data you are looking for is a hot cache.

To try to make sure that entries in the cache are up to date the Recursor can be configured to issue queries to refresh them in the background when it sees that a record set is popular and almost expired. The Recursor uses a rather simple way to decide if a record is popular: if a query comes in and it has an entry in the record cache that is almost expired, it is marked as popular and a task is scheduled to refresh it. The client asking the question will still get the answer from the cache quickly and when a client then asks for the same record, it will find it refreshed (if the scheduled task has been run in the meantime).

Below we show a new graph adding the query times with the “Refresh Almost Expired” feature enabled by setting refresh-on-ttl-perc = 10. This means that if a cache entry is re-queried with less than 10% of its original TTL remaining, it will be re-fetched. This feature is available since Recursor 4.5.

What we see is that the yellow bars of the graph do not have the high peaks that the orange ones have. The relative small price we have to pay is that we do slightly more queries to authoritative name servers.

To be able to do decide when to refresh a Record Cache entry, the Record Cache has been extended to include the original TTL. That way, The Recursor can decide if a cache entry is almost expired: only a set percentage of the original TTL remains. We have chosen to use this simple method since it does not require the Recursor to keep track of statistical data to see which questions are popular, while still being able to not refresh cache entries that are not re-queried within the sample period that is a percentage of the TTL. Doing that would be a waste of time and resources.

An asynchronous task subsystem was built to implement background refreshing. This subsystem is also used for other tasks in the Recursor that do not require the Recursor to wait for them. We will likely see the task subsystem in future posts again.

Short TTL values become more and more popular, as they allow for quick updates of record sets, but they decrease cache effectiveness, put more load on resolvers and authoritative name servers and cause more clients having to wait for resolvers fetching record sets. The Refresh Almost Expired functionality mostly solves the issue of clients having to wait on answers but we would prefer domain administrators to use large TTL values as it reduces load on the whole of the DNS infrastructure.

dnsdist 1.7.1 released

Hello!

We are very happy to release dnsdist 1.7.1 today, a maintenance release fixing a few bugs reported since 1.7.0:

  • A use-after-free error could happen if a network error occurred in the middle of a XFR query, for a proxy-protocol-enabled backend, leading to a crash
  • The TLS Server Name Indication was not properly set on outgoing DNS over HTTPS or DNS over TLS connections to a backend
  • The health-check timeout was not properly set for outgoing DNS over HTTPS connections, leading to a very long timeout
  • The outgoing protocol was not always properly set in our in-memory ring buffers
  • Outgoing UDP timeouts were sometimes processed a bit too late when the health-check interval was set to more than one second
  • Filtering qnames via eBPF was broken
  • The dynamic block mechanism was not properly switching to eBPF filtering, when available, if the block action was not explicitly set
  • The latency histogram was broken in our prometheus metrics
  • Trying to create a 0-sized packet cache would lead to a crash

In addition to these fixes, our Docker images no longer have capability requirements. More information on that topic is available in our upgrade guide.

We also improved our compatibility with OpenSSL 3.0.0’s API.

As usual there were also other smaller enhancements and fixes, please see the dnsdist website for the more complete changelog and the current documentation.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

The release tarball and its signature are available on the downloads website, and packages for several distributions are available from our repository.

First Beta Release of PowerDNS Recursor 4.7.0

We are proud to announce the first beta release of PowerDNS Recursor 4.7.0.

Compared to the previous major (4.6) release of PowerDNS Recursor, this release contains the following major changes:

  • A configurable way of adding Additional records to answers sent to the client, so the client does not have to ask for these records.
  • The step sizes for Query Minimization are now computed following to guidelines in RFC 9156.
  • The Recursor now schedules tasks to resolve IPv6 addresses of name servers not learned by glue records. This has the consequence that, if applicable, name servers will be contacted over IPv6 more often.
  • An experimental implementation of unilateral DoT probing. This allows the Recursor to learn if a an authoritative servers supports DoT.
  • Recursor has gained a way to fall back to parent NS set if contacting servers in the child NS set does not lead to an answer. This works around some broken authoritative servers configurations.
  • ZONEMD validation of the zones retrieved by the Zone to Cache, providing integrity guarantees for the zone retrieved.
  • The table recording round trip times of authoritative server IP addresses is now shared between threads to make it more effective and to reduce its memory footprint.
  • A Lua FFI hook for post-resolve interception: postresolve_ffi, providing a very fast way to do post-resolve Lua scripting.

As always, there are also many smaller bug fixes and improvements, please refer to the changelog for additional details. When upgrading do not forget to check the upgrade guide.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

The tarball (signature) is available from our download server and packages for several distributions are available from our repository.

With the final 4.7 release, the 4.4.x releases will be EOL and the 4.5.x and 4.6.x releases will go into critical fixes only mode. Consult the EOL policy for more details.

We would also like to mention that with the 4.5 release we stopped supporting systems using 32-bit time. This includes most 32-bit Linux platforms.

We also like to announce the upcoming removal of XPF support. If you are using this feature, plan switching to the proxy protocol.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

PowerDNS Authoritative Server 4.6.2

Hello!

Today we published release 4.6.2 of the Authoritative Server. It contains a carefully selected set of new features, plus a few bug fixes.

Please find a full list in the changelog.

Please make sure to read the Upgrade Notes before upgrading.

The tarball (signature) is available at downloads.powerdns.com and packages for various Linux distributions are available from repo.powerdns.com.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.