Category: Uncategorized

First Alpha Release of PowerDNS Recursor 4.8.0

We are proud to announce the first alpha release of PowerDNS Recursor 4.8.0.

Compared to the previous major (4.7) release of PowerDNS Recursor, this release contains the following major changes:

  • Structured Logging has been implemented for almost all subsystems. This allows for improved (automated) analysis of logging information.
  • Optional Serve Stale functionality has been implemented, providing resilience against connectivity problems towards authoritative servers.
  • Optional Record Locking has been implemented, providing an extra layer of protection against spoofing attempts at the price of reduced cache efficiency.
  • Internal tables used to track information about authoritative servers are now shared instead of per-thread, resulting in better performance and lower memory usage.
  • EDNS padding of outgoing DoT queries has been implemented, providing better privacy protection.

As always, there are also many smaller bug fixes and improvements, please refer to the changelog for additional details. When upgrading do not forget to check the upgrade guide.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

The tarball (signature) is available from our download server and packages for several distributions are available from our repository.

With the final 4.8 release, the 4.5.x releases will be EOL and the 4.6.x and 4.7.x releases will go into critical fixes only mode. Consult the EOL policy for more details.

We would also like to mention that with the 4.5 release we stopped supporting systems using 32-bit time. This includes many 32-bit Linux platforms.

We also like to announce the upcoming removal of XPF support. If you are using this feature, plan switching to the proxy protocol.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

PowerDNS Recursor 4.5.11, 4.6.4 and 4.7.3 Released

Hello,

Today we have released a maintenance release of PowerDNS Recursor 4.5.11, 4.6.4 and 4.7.3, containing fixes for a few minor issues and performance enhancements in the case Recursor is confronted with connectivity issues to authoritative servers.

The changelogs are available at 4.5.11, 4.6.4, 4.7.3.

The source tarballs (4.5.11, 4.6.4, 4.7.3) and signatures (4.5.11, 4.6.4, 4.7.3) are available from our download server. Packages for various distributions are available from our repository.

Note that PowerDNS Recursor 4.4.x and older releases are End of Life. Consult the EOL policy for more details.

We would also like to repeat that starting with the 4.5 release branch we stopped supporting systems using 32-bit time. This includes most 32-bit Linux platforms.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

Authoritative Server 4.7.0-beta2

Hello!

this is the first Beta release for Authoritative Server 4.7.0, even though it is called beta2. (beta1 was never released because of bugs found during the release process).

4.7.0 brings support for Catalog Zones, developed by Kees Monshouwer. As part of that development, the freshness checks in the Primary code were reworked, reducing them from doing potentially thousands of SQL queries (if you have thousands of domains) to only a few. Installations with lots of domains will benefit greatly from this, even without using catalog zones.

4.7.0 also brings back GSS-TSIG support, previously removed for quality reasons, now reworked with many stability improvements.

Other things of note:

  • LUA records, when queried over TCP, can now re-use a Lua state, giving a serious performance boost.
  • lmdbbackend databases now get a UUID assigned, making it easy for external software to spot if a database was completely replaced
  • lmdbbackend databases now optionally use random IDs for objects
  • a new LUA function called ifurlextup, and improvements in other LUA record functions
  • autoprimary management in pdnsutil and the HTTP API

A full list of changes can be found in the changelog.

Please make sure to read the Upgrade Notes before upgrading.

The tarball (signature) is available at downloads.powerdns.com. Packages for various distributions are available from repo.powerdns.com.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

Sharing data between threads in PowerDNS Recursor

This is the third part of a series of blog posts we are publishing, mostly around recent developments with respect to PowerDNS Recursor. The first blog post was Refreshing Of Almost Expired Records: Keeping The Cache Hot, the second Probing DoT Support of Authoritative Servers: Just Try It.

In PowerDNS Recursor the actual resolving is done using mthreads: a lightweight cooperative thread switching mechanism. This allows us to write the resolving code in a straightforward manner, we can program it as if we are resolving in a synchronous way. The mthreads abstraction takes care of running another mthread when the resolving process for a particular query has to wait for incoming data from the network. Mthreads are mapped to Posix threads, the thread abstraction the C++ runtime provides. Typically a handful of Posix threads run many mthreads, one for each query in-progress. Mthread switching happens only at specific points in the resolving process, basically whenever I/O is done.

Process and Data

The resolving process uses a lot of data: for example cached DNS records and speed measurements of authoritative servers. When the Recursor was originally written this data was associated to the Posix threads as thread-local data. Other Posix threads use other data (because the data is thread-local) and other mthreads only have a chance to modify the data if the current mthread does network I/O. With some care around the steps doing I/O, an mthread can assume it owns the data, and no protection against concurrent access or modification is needed.

This approach is nice and simplifies things but it has a big drawback: each Posix thread has its own copy of the record cache and other data. Several copies of each data item can exist next to each other. This is inefficient: it uses more memory than needed and an mthread run by one Posix thread cannot take advantage of information learned by another mthread that happens to have been run by another Posix thread. We partially mitigated this drawback using our distributor, making sure that all queries for a given name would be served by the same Posix thread, and thus would have access to the cached data.

Sharing data

A few releases ago, we started to solve the ‘lack of data sharing’ problem by gradually moving to a shared data model. Sharing data is nice, but a lot of coordination has to be taken care of, as Posix threads run concurrently: one thread might change the data while another thread is looking at it or is also changing it. To solve this, some form of coordination has to be implemented. A general approach to do this is using a mutual exclusion primitive called mutex, which protects the accessing and modification of data and allows only one thread at a time to operate on the data. This is nice but there are two potential issues with this:

  1. If multiple threads are accessing the data, others must wait, this is called lock contention
  2. Making sure the code has acquired the right mutex while accessing the data is hard, it is easy to forget this.

Lets take a look how we solved both these issues.

Avoiding lock contention

To avoid lock contention, the record cache is split up in many caches called shards. The name of a record determines in which shard the entry is to be found if present. This has the consequence that two mthreads only experience contention if they happen to access records in the same shard. Even on a busy Recursor this chance is quite low, as the metrics the record cache keeps show. The cache also is more effective: as it does not store multiple copies of a single record, there is more room for unique records, and the cache hit ratio can be larger with the same amount of memory used.

For other data an important observation is that data is mostly used to decide if and which authoritative server to send a query to. At that point we know we are going to do I/O to send out a request and we expect having to wait for a reply. The cost of locking and accessing a shared data structure might be high for CPU bound code, but when already setting things up to do I/O the cost of locking is relatively low while the benefits of sharing the data between Posix threads is big: e.g. if one thread found a particular authoritative server to be unresponsive, it helps a lot if another thread can use that information to its advantage and choose to contact an alternative authoritative server.

Getting locking right

Sharing information can be beneficial, but only if you do it right. Making sure shared data is only accessed and modified while holding the proper mutex is a job you do not want to leave to the developer only as it is too easy to make mistakes. Both forgetting to acquire a mutex as well as forgetting to release a mutex (for example in a not so often executed error path) can happen easily. Tools like ThreadSanitizer (TSAN) can help to detect these errors, but have the drawback they can only detect errors while running the program and exercising the particular error path.

Luckily C++ has some facilities to make locking easier. My colleague Remi Gacogne used these primitives to create a locking mechanism that is easy to use and which prevents locking mistakes.

Lets first look at the basic mechanism provided by C++:

struct Data {
  int field;
  std::mutex mutex;
};

Example code using this class:

void f(Data& data)
{
  const std::lock_guard<std::mutex> lock(data.mutex);
  data.field = 0;
}

The lock guard local variable is initialised with an acquired mutex and when the scope is left the mutex held by the guard will be released so other threads can try to acquire the mutex.

This is nice, but it means we still have to remember to use this construct when accessing the field and we also have to to document or remember which mutex protects which data. With complex data-structures that becomes hard, forgetting to acquire the mutex or using the wrong mutex will happen.

The solution is to use a helper C++ template that takes care of the locking:

struct Data {
  int field;
};
using ProtectedData = LockGuarded<Data>;

void f(ProtectedData& protectedData)
{
  auto guard = protectedData.lock();
  guard->field = 0;
}

The nice trick is here that the fields of protectedData can only be accessed via the guard obtained using the lock() method. The scoping rules of C++ guarantee that. If I would try to access data it guarded without holding the proper lock, the compiler will not like it:

x.cc:12:17: error: no member named 'field' in 'LockGuarded<Data>'
  protectedData.field = 0;
  ~~~~~~~~~~~~~ ^

Compile time errors are much nicer than runtime errors only detected with extra sanitation! The implementation of the LockGuarded template and related classes can be found in lock.hh. We use them everywhere in the PoweDNS code base.

Status

The shared record cache was implemented in version 4.4.0 of PowerDNS Recursor. Implementing sharing of various tables related to authoritative servers was started in 4.7.0 and will be finished in the upcoming 4.8.0 version. In our testing we observe very low lock contention, a more effective cache and lower memory consumption.

Security Advisory 2022-02 for PowerDNS Recursor up to and including 4.5.9, 4.6.2, 4.7.1

Hello,

Today we have released PowerDNS Recursor 4.5.10, 4.6.3 and 4.7.2 due to a medium severity issue found. The security advisory only applies to Recursors running with protobuf logging enabled.

Please find the full text of the advisory below.

The changelogs are available at 4.5.10, 4.6.3, 4.7.2.

The source tarballs (4.5.10, 4.6.3, 4.7.2) and signatures (4.5.10, 4.6.3, 4.7.2) are available from our download server. Patches are available at patches. Packages for various distributions are available from our repository.

Note that PowerDNS Recursor 4.4.x and older releases are End of Life. Consult the EOL policy for more details.


PowerDNS Security Advisory 2022-02: incomplete exception handling related to protobuf message generation

CVE: CVE-2022-37428
Date: 23th of August 2022.
Affects: PowerDNS Recursor up to and including 4.5.9, 4.6.2 and 4.7.1
Not affected: PowerDNS Recursor 4.5.10, 4.6.3 and 4.7.2
Severity: Medium
Impact: Denial of service
Exploit: This problem can be triggered by a remote attacker with access to the recursor if protobuf logging is enabled
Risk of system compromise: None
Solution: Upgrade to patched version, disable protobuf logging of responses

This issue only affects recursors which have protobuf logging enabled using the

  • protobufServer function with logResponses=true or
  • outgoingProtobufServer function with logResponses=true

If either of these functions is used without specifying logResponses, its value is true.
An attacker needs to have access to the recursor, i.e. the remote IP must be in the access control list.
If an attacker queries a name that leads to an answer with specific properties, a protobuf message might be generated that causes an exception. The code does not handle this exception correctly, causing a denial of service.

PowerDNS Authoritative Server 4.6.3

Hello!

Today we published release 4.6.3 of the Authoritative Server. It contains two bug fixes, and marks the arrival of Ubuntu Jammy packages for the 4.6 branch.

Please find a full list in the changelog.

Please make sure to read the Upgrade Notes before upgrading.

The tarball (signature) is available at downloads.powerdns.com and packages for various Linux distributions are available from repo.powerdns.com.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

PowerDNS Recursor 4.7.1 Released

We are proud to announce the release of PowerDNS Recursor 4.7.1.

This release is a maintenance releases correcting an issue where asynchronous tasks would not be executed promptly. It also allows the generic record format in zone files loaded using the ZoneToCache function.

Please refer to the change log for the 4.7.1 release for additional details.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

The tarball and signature are available from our download server and packages for several distributions are available from our repository. The Ubuntu Jammy package will be published soon.

The 4.4.x releases are EOL and the 4.5.x and 4.6.x releases are in critical fixes only mode. Consult the EOL policy for more details.

We would also like to repeat that starting with the 4.5 release branch we stopped supporting systems using 32-bit time. This includes most 32-bit Linux platforms.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

Probing DoT Support of Authoritative Servers: Just Try It

This is the second part of a series of blog posts we are publishing, mostly around recent developments with respect to PowerDNS Recursor. The first blog post was Refreshing Of Almost Expired Records: Keeping The Cache Hot.

In PowerDNS Recursor 4.6.0 we introduced DNS over TLS (DoT) support for outgoing connections. Starting with that version, DoT is used by default for (forwarded) connections to port 853 and for a configurable list of authoritative servers. On the client-resolver side there has been developments to make DoT discovery easy. In this post we will discuss the DoT discovery from the resolver to authoritative server perspective. PowerDNS Recursor 4.7.0 has an experimental feature implementing this.

DoT

These days many DNS clients have the ability to switch to a DoT (or DNS over HTTPS, DoH) connection under certain circumstances, providing better security for the client-resolver channel. While DNSSEC provides authenticity proof of data received from an authoritative server, it does not provide confidentiality and in general it does not protect the channel between the resolver and the client: most clients do not support DNSSEC validation. DoT (and DoH) fill parts of this gap. It is less well known that DoT can also be used to secure the channels between a resolver and the authoritative servers. Common to both applications of DoT is the question when to use DoT instead of regular DNS queries over UDP or TCP to port 53, commonly called Do53.

Client-Resolver DoT Discovery

In the case of client-resolver communication there is a (draft) standard that specifies how an client can discover if a resolver supports DoT: Discovery of Designated Resolvers (DDR). In short, using DNS queries with name _dns.example.net (or _dns.resolver.arpa if it does not know the name) and query type SVCB, a client can learn if a resolver supports DoT. The draft also contains rules to make sure a validated DoT connection can be set up if needed. All this allows a secure upgrade to DoT without user or administrator intervention. The major public resolvers all support DDR and if you run a resolver yourself, it is not difficult to set it up to return the proper record(s) for DDR queries.

Resolver-Authoritative DoT Discovery

Sadly, the case for resolver to authoritative server traffic is not as simple. At the moment of writing, the committee concerned with these things was not able to come up with anything better than Unilateral Opportunistic Deployment of Recursive-to-Authoritative DNS, which in essence boils down to: let the resolver find out if DoT is available by trying a DoT Connection to the IP address of the authoritative server and remembering the results for a while. At this moment most authoritative servers do not support DoT. A DoT connection is likely to time out, using resources during the attempt. There are also no guidelines on certificate checking of the DoT connection, the draft says any certificate must be accepted. Even if this is not very satisfying we set out to implement this mechanism as an experimental feature in PowerDNS Recursor 4.7.0. We hope in the future a better mechanism to discover DoT support of authoritative servers will be drafted. If so, we certainly intend to implement this better method.

Implementing DoT of authoritative server discovery

PowerDNS Recursor maintains several in-memory tables with characteristics of authoritative servers. An example of this information is the speed of each authoritative server contacted. The speed information is used to select the authoritative expected to be fastest when a query to an authoritative server of a domain needs to be done. Following the guidelines in the draft mentioned above, we introduced a new table to record DoT probe activity and results.

When an authoritative server needs to be contacted, this table is consulted to see if we already know if this server supports DoT. There are several cases:

  • If the server IP is not in the table because we did not contact it recently, we create an entry in the table and mark the DoT status Unknown and do a Do53 query
  • If the server is known to not support DoT, we do a Do53 query.
  • If the server is known to support DoT, we do a DoT query.
  • If we find the status is Unknown, we do a regular Do53 query and we schedule an asynchronous task to probe for DoT support. The table entry is marked Busy in this case.

if a task is submitted, a separate task mechanism in the recursor picks up the task and tries a DoT connection to the associated IP. If that works and a DNS query is also successful over the newly established DoT connection, the IP is marked as DoT capable (Good). If the DoT connections times out or the query is not successful, the IP is marked as Bad.

While the DoT status is not yet determined, we use regular DNS queries to contact a specific authoritative server, since we do not want to burden clients with the long (a couple of seconds) time outs involved in unsuccessful DoT probes.

The housekeeping of the table involves a bit more work, so that old information is expired or refreshed properly and to make sure that we do not overwhelm ourselves or authoritative servers with a lot of simultaneous DoT probes. The draft has some guidelines for doing this properly. There are also more complex issues: what do we do if a DoT query fails to a server we marked as DoT capable? Do we mark the IP as not DoT capable or not? It is likely not good to downgrade a server that has been serving DoT for many queries on the failure of a single query. These open questions and the hope that a draft that provides a better method will appear are the reasons this feature is marked as experimental.

The draft mentioned above also recommends the information learned to be persistent, the resolver should store the information while running and re-read the information on start-up. We do not implement that part yet.

The mechanism we implemented decouples the use of DoT from the discovery process. This means that in the future, alternative discovery mechanisms (specifying how) or policies (specifying when and in what order) can be implemented without interfering with other parts of the recursor. At the moment the probing process is lazy: tasks will be executed when the recursor is processing queries. An idle recursor will execute tasks only sporadically. Because of this, probe tasks executed will have no immediate relation to the queries executed on behalf of clients. This will be improved upon in the near future.

Trying it yourself

Using PowerDNS Recursor 4.7.0, you can try this yourself by setting max-busy-dot-probes to a non-zero value. This configuration governs the maximum number of simultaneous DoT probes. After the recursor has been busy for a while, you can look at the status of the DoT probes by using

rec_control dump-dot-probe-map -

To see a list of know authoritative servers and their DoT support status. A few example lines are shown below

23.61.199.67    akam.net.               1       Bad     2022-05-26T14:44:10
2.22.230.67     akam.net.               1       Busy    2022-05-26T14:44:21
184.85.248.67   akam.net.               1       Bad     2022-05-26T14:43:59
88.221.25.85    dscx.akamaiedge.net.    0       Unknown 2022-05-26T14:40:46
188.166.104.87  powerdns.com.           3       Good    2022-05-28T15:21:06
2.19.195.108    dscx.akamaiedge.net.    0       Unknown 2022-05-26T14:40:26
23.74.25.128    akagtm.org.             1       Bad     2022-05-26T14:41:16

A server marked Unknown is not probed yet, only servers that are visited at least twice within an interval get probed. The status can also be Bad (no DoT support), Good (DoT support) or Busy (DoT probe scheduled or running).

Conclusion

While we’ve implemented the DoT discovery mechanism as an experimental feature in PowerDNS Recursor 4.7.0, we consider it a half-measure, a waypoint if you will. We hope a future draft will provide more guidance on the how and when of the discovery process of DoT support between resolver and authoritative servers, which should lead to better security for DNS users.

PowerDNS Recursor 4.7.0 Released

We are proud to announce the release of PowerDNS Recursor 4.7.0.

Compared to the previous major (4.6) release of PowerDNS Recursor, this release contains the following major changes:

  • A configurable way of adding Additional records to answers sent to the client, so the client does not have to ask for these records.
  • The step sizes for Query Minimization are now computed following to guidelines in RFC 9156.
  • The Recursor now schedules tasks to resolve IPv6 addresses of name servers not learned by glue records. This has the consequence that, if applicable, name servers will be contacted over IPv6 more often.
  • An experimental implementation of unilateral DoT probing. This allows the Recursor to learn if a an authoritative servers supports DoT.
  • Recursor has gained a way to fall back to the parent NS set if contacting servers in the child NS set does not lead to an answer. This works around some broken authoritative servers configurations.
  • ZONEMD validation of the zones retrieved by the Zone to Cache, providing integrity guarantees for the zone retrieved.
  • The table recording round trip times of authoritative server IP addresses is now shared between threads to make it more effective and to reduce its memory footprint.
  • A Lua FFI hook for post-resolve interception: postresolve_ffi, providing a very fast way to do post-resolve Lua scripting.

As always, there are also many smaller bug fixes and improvements, please refer to the changelog for additional details. When upgrading do not forget to check the upgrade guide.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

The tarball (signature) is available from our download server and packages for several distributions are available from our repository.

With this 4.7.0 release, the 4.4.x releases will be EOL and the 4.5.x and 4.6.x releases will go into critical fixes only mode. Consult the EOL policy for more details.

We would also like to mention that with the 4.5 release we stopped supporting systems using 32-bit time. This includes most 32-bit Linux platforms.

We also like to announce the upcoming removal of XPF support. If you are using this feature, plan switching to the proxy protocol.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

First Release Candidate of PowerDNS Recursor 4.7.0

We are proud to announce the first release candidate of PowerDNS Recursor 4.7.0. Testing of this release candidate is much appreciated!

The most important change compared to the 4.7.0-beta1 release is a fix for the experimental DoT to authoritative server probing code.

Compared to the previous major (4.6) release of PowerDNS Recursor, this release contains the following major changes:

  • A configurable way of adding Additional records to answers sent to the client, so the client does not have to ask for these records.
  • The step sizes for Query Minimization are now computed following to guidelines in RFC 9156.
  • The Recursor now schedules tasks to resolve IPv6 addresses of name servers not learned by glue records. This has the consequence that, if applicable, name servers will be contacted over IPv6 more often.
  • An experimental implementation of unilateral DoT probing. This allows the Recursor to learn if a an authoritative servers supports DoT.
  • Recursor has gained a way to fall back to parent NS set if contacting servers in the child NS set does not lead to an answer. This works around some broken authoritative servers configurations.
  • ZONEMD validation of the zones retrieved by the Zone to Cache, providing integrity guarantees for the zone retrieved.
  • The table recording round trip times of authoritative server IP addresses is now shared between threads to make it more effective and to reduce its memory footprint.
  • A Lua FFI hook for post-resolve interception: postresolve_ffi, providing a very fast way to do post-resolve Lua scripting.

As always, there are also many smaller bug fixes and improvements, please refer to the changelog for additional details. When upgrading do not forget to check the upgrade guide.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

The tarball (signature) is available from our download server and packages for several distributions are available from our repository.

With the final 4.7 release, the 4.4.x releases will be EOL and the 4.5.x and 4.6.x releases will go into critical fixes only mode. Consult the EOL policy for more details.

We would also like to mention that with the 4.5 release we stopped supporting systems using 32-bit time. This includes most 32-bit Linux platforms.

We also like to announce the upcoming removal of XPF support. If you are using this feature, plan switching to the proxy protocol.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.