First release candidate for Authoritative Server 4.7.0

This is the first release candidate for Authoritative Server 4.7.0. We hope it will also be the last šŸ™‚

4.7.0 brings support for Catalog Zones, developed by Kees Monshouwer. As part of that development, the freshness checks in the Primary code were reworked, reducing them from doing potentially thousands of SQL queries (if you have thousands of domains) to only a few. Installations with lots of domains will benefit greatly from this, even without using catalog zones.

4.7.0 also brings back GSS-TSIG support, previously removed for quality reasons, now reworked with many stability improvements.

Other things of note:

  • LUA records, when queried over TCP, can now re-use a Lua state, giving a serious performance boost.
  • lmdbbackend databases now get a UUID assigned, making it easy for external software to spot if a database was completely replaced
  • lmdbbackend databases now optionally use random IDs for objects
  • a new LUA function called ifurlextup, and improvements in other LUA record functions
  • autoprimary management in pdnsutil and the HTTP API
  • in beta, a key roller daemon, currently not packaged

A full list of changes can be found in the changelog.

Please make sure to read the Upgrade Notes before upgrading.

The tarball (signature) is available at downloads.powerdns.com. Packages for various distributions are available from repo.powerdns.com.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

Structured Logging in PowerDNS Recursor

This is the fourth part of a series of blog posts we are publishing, mostly around recent developments with respect to PowerDNS Recursor. The first blog post was Refreshing Of Almost Expired Records: Keeping The Cache Hot, the second Probing DoT Support of Authoritative Servers: Just Try It and the third Sharing data between threads in PowerDNS Recursor.

Complex programs like PowerDNS Recursor need to log information about their operation. In itself this is a delicate balance: too much logging will slow the process down, too little will make it harder to diagnose issues. The question “what and how to log” is also important: up until now we have used free-format text logging. The messages itself can go to a logging daemon (e.g. systemd-journald or syslogd), a text file or the console.

The problem with free text logging is that is it very hard to keep it consistent and hard to process automatically in a reliable way. It is also not always clear (unless the creator of the message is very careful) what information is actually logged and quoting of special values is often problematic. A typical example of a free format log message looks like this (set quiet=no when running Recursor to see these detailed messages, this is not recommended in a production environment as it generates an awful lot of messages):

3 [1/1] question for 'www.powerdns.com|A' from 127.0.0.1:53408

To illustrate, unless you are very familiar with the PowerDNS code base it is unclear what the 3 1/1 part means. These are: the Posix thread-id, the mthread-id (mthreads are the user level threads used by PowerDNS Recursor) and the number of queries being resolved concurrently. Observe that automated log analysis will probably fail if the name queried contains characters like a quote or vertical bar.

Structured Logging

A structured log message consists of a set of key-value pairs. The pairs can be exported in various formats, one of them being text lines. In the text representation, a message like the one above would look like:

msg="Question" subsystem="syncres" level="0" prio="Info" tid="3" ts="1664192640.282" ecs="" mtid="1" proto="udp" qname="www.powerdns.com" qtype="A" remote="127.0.0.1:63684"

Note that each value is labelled. That is a good start to make sure we know what each part means.

There are several keys that are common to each structured log entry. These are:

  • msg a short text message indicating the subject of the message instance. This part is fixed to allow for easily selection by log analysis tools.
  • subsystem the component which generated the message
  • level the detail level
  • prio the priority
  • tid the Posix thread-id
  • ts a timestamp
  • mtid the mthread-id

The message instance specific parts are ecs, proto, qname, qtype and remote.

The timestamp is produced by Recursor itself, which means that it is not dependent in the system used to export the log messages The structured message also adds extra information left out in the original (e.g. proto), and leaves out one item, as it was deemed to be non-relevant (the number of concurrent queries being processed).

The level value indicates the amount of detail. Higher levels indicated very detailed information, only relevant when developing or when doing very deep analysis of logs. Priority indicates if action by the administrator is required and is the same as the priority value used by e.g. syslog.

From an implementation point of view, the code that implements structured logging has a big advantage that information associated with several related log lines only has to be set up once, each generated log line will share the common information. The existing logging code would need to repeat that information with each generated log line. The structured logging code also has methods to log specific value types, like internet addresses, making sure they are always logged in the same manner.

Text format and key-value format

By default, PowerDNS Recursor sends its structured information as a formatted text string containing key-value pairs with proper quoting and escaping of quotes. On Linux system using systemd, it can also send information to systemd-journald as structured log entries. In this case systemd-journald stores the entry as a sequence of key-value pairs in a log database.

To use this logging backend, add the --structured-logging-backend=systemd-journal to the command line in the systemd unit file. Note that adding it to the recursor configuration file does not work as expected, as this file is processed after the logging has been set up.

To query the log, use a command similar to

# journalctl -r -n 1 -o json-pretty -u pdns-recursor.service 
{
"_SYSTEMD_INVOCATION_ID" : "568cd5479a5f4ea29e3c793939cd31f6",
"REMOTE" : "127.0.0.1:40908",
"PRIORITY" : "6",
"_UID" : "107",
"ECS" : "",
"_SYSTEMD_UNIT" : "pdns-recursor.service",
"__CURSOR" : "s=3e46e7865e524e19bada5d3e56e08b69;i=100129;b=8a57b91cdb5346559d48
2f5fae75bd92;m=b09c7370;t=5e99425342dfd;x=d050d34430fdca60",
"QNAME" : "www.powerdns.com",
"__MONOTONIC_TIMESTAMP" : "2963043184",
"_CAP_EFFECTIVE" : "0",
"_HOSTNAME" : "pepper",
"_COMM" : "pdns_recursor",
"_GID" : "114",
"_MACHINE_ID" : "8b32d617439149049d62119e02e2fa10",
"_SYSTEMD_CGROUP" : "/system.slice/pdns-recursor.service",
"MTID" : "1",
"_BOOT_ID" : "8a57b91cdb5346559d482f5fae75bd92",
"SUBSYSTEM" : "syncres",
"TID" : "2",
"_EXE" : "/usr/sbin/pdns_recursor",
"_PID" : "23684",
"QTYPE" : "A",
"MESSAGE" : "Question",
"LEVEL" : "0",
"_SOURCE_REALTIME_TIMESTAMP" : "1664197372161506",
"__REALTIME_TIMESTAMP" : "1664197372161533",
"_CMDLINE" : "/usr/sbin/pdns_recursor --daemon=no --write-pid=no --disable-syslo
g --log-timestamp=no --structured-logging-backend=systemd-journal",
"TIMESTAMP" : "1664197372.161",
"PROTO" : "udp",
"_SYSTEMD_SLICE" : "system.slice",
"_TRANSPORT" : "journal",
"_SELINUX_CONTEXT" : "unconfined\n",
"SYSLOG_IDENTIFIER" : "pdns_recursor"
}

As can be seen, systemd-journald adds quite a bit of extra key-value pairs and capitalises the keys. A drawback is that the information in this form is hard to parse for humans, but programs that are used for log analysis process JSON easily.

The future

In the future we will add more logging backends, to make it easy to send the structured logging information to a program used to process logs.

Note that trace information (the very detailed information on each step of the resolving process collected when debugging specific query processing using rec_control trace-regex) is not converted to structured format yet. This will be done in the near future, together with functionality to send trace logs to a specific destination for further analysis. Until then trace logs will end up in the general log, as they do now.

Reverting to old-style logging

If you already have log analysis in place, the changing of the logging information might disrupt your log processing. Switch back to tradition logging by setting structured-logging to no. New functionality requiring logging will implement structured logging only. When structured-logging is set to no, these new subsystems will output a textual representation of the key-value pairs.

Status and Conclusion

The structured logging system has been used by a few subsystems already in recent releases. The full conversion will appear in version 4.8.0, to be released in the near future. Pre-releases already allow you to see structured logging in action. With structured logging, log analysis of PowerDNS Recursor will become more easy, as the information is structured in a uniform way.

First Alpha Release of PowerDNS Recursor 4.8.0

We are proud to announce the first alpha release of PowerDNS Recursor 4.8.0.

Compared to the previous major (4.7) release of PowerDNS Recursor, this release contains the following major changes:

  • Structured Logging has been implemented for almost all subsystems. This allows for improved (automated) analysis of logging information.
  • Optional Serve Stale functionality has been implemented, providing resilience against connectivity problems towards authoritative servers.
  • Optional Record Locking has been implemented, providing an extra layer of protection against spoofing attempts at the price of reduced cache efficiency.
  • Internal tables used to track information about authoritative servers are now shared instead of per-thread, resulting in better performance and lower memory usage.
  • EDNS padding of outgoing DoT queries has been implemented, providing better privacy protection.

As always, there are also many smaller bug fixes and improvements, please refer to the changelog for additional details. When upgrading do not forget to check the upgrade guide.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

The tarball (signature) is available from our download server and packages for several distributions are available from our repository.

With the final 4.8 release, the 4.5.x releases will be EOL and the 4.6.x and 4.7.x releases will go into critical fixes only mode. Consult the EOL policy for more details.

We would also like to mention that with the 4.5 release we stopped supporting systems using 32-bit time. This includes many 32-bit Linux platforms.

We also like to announce the upcoming removal of XPF support. If you are using this feature, plan switching to the proxy protocol.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

PowerDNS Recursor 4.5.11, 4.6.4 and 4.7.3 Released

Hello,

Today we have released a maintenance release of PowerDNS Recursor 4.5.11, 4.6.4 and 4.7.3, containing fixes for a few minor issues and performance enhancements in the case Recursor is confronted with connectivity issues to authoritative servers.

The changelogs are available at 4.5.11, 4.6.4, 4.7.3.

The source tarballs (4.5.11, 4.6.4, 4.7.3) and signatures (4.5.11, 4.6.4, 4.7.3) are available from our download server. Packages for various distributions are available from our repository.

Note that PowerDNS Recursor 4.4.x and older releases are End of Life. Consult the EOL policy for more details.

We would also like to repeat that starting with the 4.5 release branch we stopped supporting systems using 32-bit time. This includes most 32-bit Linux platforms.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

Authoritative Server 4.7.0-beta2

Hello!

this is the first Beta release for Authoritative Server 4.7.0, even though it is called beta2. (beta1 was never released because of bugs found during the release process).

4.7.0 brings support for Catalog Zones, developed by Kees Monshouwer. As part of that development, the freshness checks in the Primary code were reworked, reducing them from doing potentially thousands of SQL queries (if you have thousands of domains) to only a few. Installations with lots of domains will benefit greatly from this, even without using catalog zones.

4.7.0 also brings back GSS-TSIG support, previously removed for quality reasons, now reworked with many stability improvements.

Other things of note:

  • LUA records, when queried over TCP, can now re-use a Lua state, giving a serious performance boost.
  • lmdbbackend databases now get a UUID assigned, making it easy for external software to spot if a database was completely replaced
  • lmdbbackend databases now optionally use random IDs for objects
  • a new LUA function called ifurlextup, and improvements in other LUA record functions
  • autoprimary management in pdnsutil and the HTTP API

A full list of changes can be found in the changelog.

Please make sure to read the Upgrade Notes before upgrading.

The tarball (signature) is available at downloads.powerdns.com. Packages for various distributions are available from repo.powerdns.com.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

Sharing data between threads in PowerDNS Recursor

This is the third part of a series of blog posts we are publishing, mostly around recent developments with respect to PowerDNS Recursor. The first blog post was Refreshing Of Almost Expired Records: Keeping The Cache Hot, the second Probing DoT Support of Authoritative Servers: Just Try It.

In PowerDNS Recursor the actual resolving is done using mthreads: a lightweight cooperative thread switching mechanism. This allows us to write the resolving code in a straightforward manner, we can program it as if we are resolving in a synchronous way. The mthreads abstraction takes care of running another mthread when the resolving process for a particular query has to wait for incoming data from the network. Mthreads are mapped to Posix threads, the thread abstraction the C++ runtime provides. Typically a handful of Posix threads run many mthreads, one for each query in-progress. Mthread switching happens only at specific points in the resolving process, basically whenever I/O is done.

Process and Data

The resolving process uses a lot of data: for example cached DNS records and speed measurements of authoritative servers. When the Recursor was originally written this data was associated to the Posix threads as thread-local data. Other Posix threads use other data (because the data is thread-local) and other mthreads only have a chance to modify the data if the current mthread does network I/O. With some care around the steps doing I/O, an mthread can assume it owns the data, and no protection against concurrent access or modification is needed.

This approach is nice and simplifies things but it has a big drawback: each Posix thread has its own copy of the record cache and other data. Several copies of each data item can exist next to each other. This is inefficient: it uses more memory than needed and an mthread run by one Posix thread cannot take advantage of information learned by another mthread that happens to have been run by another Posix thread. We partially mitigated this drawback using our distributor, making sure that all queries for a given name would be served by the same Posix thread, and thus would have access to the cached data.

Sharing data

A few releases ago, we started to solve the ‘lack of data sharing’ problem by gradually moving to a shared data model. Sharing data is nice, but a lot of coordination has to be taken care of, as Posix threads run concurrently: one thread might change the data while another thread is looking at it or is also changing it. To solve this, some form of coordination has to be implemented. A general approach to do this is using a mutual exclusion primitive called mutex, which protects the accessing and modification of data and allows only one thread at a time to operate on the data. This is nice but there are two potential issues with this:

  1. If multiple threads are accessing the data, others must wait, this is called lock contention
  2. Making sure the code has acquired the right mutex while accessing the data is hard, it is easy to forget this.

Lets take a look how we solved both these issues.

Avoiding lock contention

To avoid lock contention, the record cache is split up in many caches called shards. The name of a record determines in which shard the entry is to be found if present. This has the consequence that two mthreads only experience contention if they happen to access records in the same shard. Even on a busy Recursor this chance is quite low, as the metrics the record cache keeps show. The cache also is more effective: as it does not store multiple copies of a single record, there is more room for unique records, and the cache hit ratio can be larger with the same amount of memory used.

For other data an important observation is that data is mostly used to decide if and which authoritative server to send a query to. At that point we know we are going to do I/O to send out a request and we expect having to wait for a reply. The cost of locking and accessing a shared data structure might be high for CPU bound code, but when already setting things up to do I/O the cost of locking is relatively low while the benefits of sharing the data between Posix threads is big: e.g. if one thread found a particular authoritative server to be unresponsive, it helps a lot if another thread can use that information to its advantage and choose to contact an alternative authoritative server.

Getting locking right

Sharing information can be beneficial, but only if you do it right. Making sure shared data is only accessed and modified while holding the proper mutex is a job you do not want to leave to the developer only as it is too easy to make mistakes. Both forgetting to acquire a mutex as well as forgetting to release a mutex (for example in a not so often executed error path) can happen easily. Tools like ThreadSanitizer (TSAN) can help to detect these errors, but have the drawback they can only detect errors while running the program and exercising the particular error path.

Luckily C++ has some facilities to make locking easier. My colleague Remi Gacogne used these primitives to create a locking mechanism that is easy to use and which prevents locking mistakes.

Lets first look at the basic mechanism provided by C++:

struct Data {
  int field;
  std::mutex mutex;
};

Example code using this class:

void f(Data& data)
{
  const std::lock_guard<std::mutex> lock(data.mutex);
  data.field = 0;
}

The lock guard local variable is initialised with an acquired mutex and when the scope is left the mutex held by the guard will be released so other threads can try to acquire the mutex.

This is nice, but it means we still have to remember to use this construct when accessing the field and we also have to to document or remember which mutex protects which data. With complex data-structures that becomes hard, forgetting to acquire the mutex or using the wrong mutex will happen.

The solution is to use a helper C++ template that takes care of the locking:

struct Data {
  int field;
};
using ProtectedData = LockGuarded<Data>;

void f(ProtectedData& protectedData)
{
  auto guard = protectedData.lock();
  guard->field = 0;
}

The nice trick is here that the fields of protectedData can only be accessed via the guard obtained using the lock() method. The scoping rules of C++ guarantee that. If I would try to access data it guarded without holding the proper lock, the compiler will not like it:

x.cc:12:17: error: no member named 'field' in 'LockGuarded<Data>'
  protectedData.field = 0;
  ~~~~~~~~~~~~~ ^

Compile time errors are much nicer than runtime errors only detected with extra sanitation! The implementation of the LockGuarded template and related classes can be found in lock.hh. We use them everywhere in the PoweDNS code base.

Status

The shared record cache was implemented in version 4.4.0 of PowerDNS Recursor. Implementing sharing of various tables related to authoritative servers was started in 4.7.0 and will be finished in the upcoming 4.8.0 version. In our testing we observe very low lock contention, a more effective cache and lower memory consumption.

Security Advisory 2022-02 for PowerDNS Recursor up to and including 4.5.9, 4.6.2, 4.7.1

Hello,

Today we have released PowerDNS Recursor 4.5.10, 4.6.3 and 4.7.2 due to a medium severity issue found. The security advisory only applies to Recursors running with protobuf logging enabled.

Please find the full text of the advisory below.

The changelogs are available at 4.5.10, 4.6.3, 4.7.2.

The source tarballs (4.5.10, 4.6.3, 4.7.2) and signatures (4.5.10, 4.6.3, 4.7.2) are available from our download server. Patches are available at patches. Packages for various distributions are available from our repository.

Note that PowerDNS Recursor 4.4.x and older releases are End of Life. Consult the EOL policy for more details.


PowerDNS Security Advisory 2022-02: incomplete exception handling related to protobuf message generation

CVE: CVE-2022-37428
Date: 23th of August 2022.
Affects: PowerDNS Recursor up to and including 4.5.9, 4.6.2 and 4.7.1
Not affected: PowerDNS Recursor 4.5.10, 4.6.3 and 4.7.2
Severity: Medium
Impact: Denial of service
Exploit: This problem can be triggered by a remote attacker with access to the recursor if protobuf logging is enabled
Risk of system compromise: None
Solution: Upgrade to patched version, disable protobuf logging of responses

This issue only affects recursors which have protobuf logging enabled using the

  • protobufServer function with logResponses=true or
  • outgoingProtobufServer function with logResponses=true

If either of these functions is used without specifying logResponses, its value is true.
An attacker needs to have access to the recursor, i.e. the remote IP must be in the access control list.
If an attacker queries a name that leads to an answer with specific properties, a protobuf message might be generated that causes an exception. The code does not handle this exception correctly, causing a denial of service.

PowerDNS Authoritative Server 4.6.3

Hello!

Today we published release 4.6.3 of the Authoritative Server. It contains two bug fixes, and marks the arrival of Ubuntu Jammy packages for the 4.6 branch.

Please find a full list in the changelog.

Please make sure to read the Upgrade Notes before upgrading.

TheĀ tarballĀ (signature) is available atĀ downloads.powerdns.comĀ and packages for various Linux distributions are available fromĀ repo.powerdns.com.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

PowerDNS Recursor 4.7.1 Released

We are proud to announce the release of PowerDNS Recursor 4.7.1.

This release is a maintenance releases correcting an issue where asynchronous tasks would not be executed promptly. It also allows the generic record format in zone files loaded using the ZoneToCache function.

Please refer to the change log for the 4.7.1 release for additional details.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

The tarball and signature are available from our download server and packages for several distributions are available from our repository. The Ubuntu Jammy package will be published soon.

The 4.4.x releases are EOL and the 4.5.x and 4.6.x releases are in critical fixes only mode. Consult the EOL policy for more details.

We would also like to repeat that starting with the 4.5 release branch we stopped supporting systems using 32-bit time. This includes most 32-bit Linux platforms.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

dnsdist-1.7.2 released

Hello!

We are very happy to release dnsdist 1.7.2 today, a maintenance release fixing a few bugs reported since 1.7.1:

  • An unhandled exception could happen when an invalid protocol was used in an incoming DNS over HTTPS forwarded-for header and passed to the backend via the proxy protocol, leading to a use-after-free and a crash. Forwarded-for headers are not used by default and should only be used if the client can be trusted (#11667)
  • An invalid proxy-protocol was sent to the backend, over TCP, if a query received via DNS over HTTPS resulted in a truncated UDP response from the backend (#11665)
  • Some metrics lacked a proper description in our Prometheus endpoint (#11664)
  • A side-effect of fixing the health-check timeout in 1.7.1 was leading to a CPU usage increase on devices that are mostly idle. We improved that situation, reducing the CPU usage even below what it was in 1.7.0 (#11579, #11580)

We also added a couple Lua bindings to make it easier to look into the DNS payload from custom Lua rules and actions (#11666).

Please see the dnsdist website for the more complete changelog and the current documentation.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

The release tarball and its signature are available on the downloads website, and packages for several distributions are available from our repository.