Category: dnsdist

First release candidate for dnsdist 1.4.0

We are proud to announce the first release candidate of the 1.4.0 version of dnsdist. 1.4.0 brings a much more scalable way of handling DNS over TCP and DNS over TLS connections since the first alpha release. A major new feature since alpha2, and marquee feature of 1.4.0 compared to 1.3.x, is the new DNS-over-HTTPS functionality.

Following a round of testing from several large scale users, this version fixes several issues, most of them related to DNS over HTTPS (7894, 7917, 7927, 8112), DNS over TCP (7974, 7979, 8003, 8030, 8067, 8078, 8079, 8113), or both (7915).

In addition to minor improvements, it also introduces several new features:

  • a new ContinueAction allowing to keep processing rules even after calling a normally terminal action, like PoolAction (8117) ;
  • OCSP stapling for DNS over TLS and DNS over HTTPS (8141) ;
  • custom HTTP headers for DNS over HTTPS responses (contributed by Melissa Voegeli, 8148) ;
  • actions, rules and Lua binding to interact with DNS over HTTPS queries and generate responses from dnsdist (8153).

We want to thank everyone that contributed to the testing of the beta release, and invite you to contribute to the testing of this release candidate!

Please see the dnsdist website for the more complete changelog and the current documentation.

Release tarballs are available on the downloads website.

Several packages are also available on our repository.

dnsdist 1.4.0-beta1

We are very happy to announce the first beta release of the 1.4.0 version of dnsdist. This version fixes a crash in the DNS over HTTPS (DoH) implementation and adds a new rule to route queries based on the incoming TLS Server Name Indication (SNI) value. It also adds latency histograms to the Prometheus export, courtesy of Marlin Cremers.

As with the alpha releases, your feedback will be much appreciated so we can deliver a stable 1.4.0 final release!

Please see the dnsdist website for the more complete changelog and the current documentation.

Release tarballs are available on the downloads website.

Several packages are also available on our repository.

dnsdist 1.4.0-alpha2 with DNS over HTTPS support

We are very happy to announce the second alpha release of the 1.4.0 version of dnsdist. This version keeps up the DNS privacy improvements with the addition of a new major feature, DNS over HTTPS (DoH), and contains very few changes apart from that.

As with the first alpha, your feedback will be much appreciated so we can deliver a stable 1.4.0 final release!

Please see the dnsdist website for the more complete changelog and the current documentation.

Release tarballs are available on the downloads website.

Several packages are also available on our repository.

First alpha release of dnsdist 1.4.0

We are very happy to announce the 1.4.0 alpha 1 release of dnsdist. This version contains a few new features, but is mostly focused on DNS privacy improvements. We are introducing a new, much more scalable way of handling DNS over TCP and DNS over TLS connections. It will be followed quite quickly by a new alpha including experimental DNS over HTTPS support.

In older versions of dnsdist, a TCP worker could only handle one incoming connection at a time, which was not very efficient when dealing with a larger number of mostly inactive connections, as we are beginning to see with DNS over TLS. Starting with this release, TCP workers are now event-based and each one of them can handle a very large number of incoming connections simultaneously.

Your feedback will be much appreciated so we can deliver a stable 1.4.0 final release!

Important changes

We took the opportunity of this new release to clean up a few things that might require updating your existing configuration. First, the number of parameters to the newPacketCache command was getting out of hand, so we switched it to a table-based syntax as we already did with newServer a while ago.

addLuaAction and addLuaResponseAction have been removed. Instead, use addAction with a LuaAction, or addResponseAction with a LuaResponseAction.

Lua constants for DNS response codes and QTypes have been moved from the ‘dnsdist’ prefix to, respectively, the DNSQType and DNSRCode prefixes.

To improve security, all ambient capabilities are now dropped after the startup phase, which might prevent launching the webserver on a privileged port at run-time, or impact some custom Lua code. In addition, systemd’s sandboxing features are now determined at compile-time, resulting in more restrictions on recent distributions. See pull requests 7138 and 6634 for more information.

And finally, if you are compiling dnsdist, note that several ./configure options have been renamed to provide a more consistent experience. Features that depend on an external component have been prefixed with –with while internal features use –enable. This has lead to the following changes:

  • –enable-fstrm to –enable-dnstap
  • –enable-gnutls to –with-gnutls
  • –enable-libsodium to –with-libsodium
  • –enable-libssl to –with-libssl
  • –enable-re2 to –with-re2

New features and improvements

Dynamic blocks and Lua rules can now use the NoRecurse action, thanks to phonedph1.

Richard Gibson added the possibility to inspect and alter trailing data.

Dmitry Alenichev implemented new rules and actions to deal with unexpected EDNS versions, and to optionally accept completely empty (qdcount=0) responses from a backend.

Andrey Domas added the new QNameSetRule rule, along with the DNSNameSet object, to match exact qnames instead of doing suffix matching.

The health check mechanism has been improved with the new checkInterval, checkTimeout and rise parameters, thanks notably to “1848”.

We added a few convenience functions to pseudonymize IP addresses, as several users reported that they needed it to be GDPR-compliant.

We noticed that, on some specific versions of the Linux kernel, the code we used to measure our memory usage could be quite expensive so we switched to an alternative, cheaper method. You might notice that the memory usage reported by this new version does not exactly match the one reported by older versions, but it should be close enough.

Finally the cost of exporting queries and responses using our remote logging solution based on protobuf has been reduced by a huge margin. System calls that used to be cheap before the Spectre and Meltdown mitigations were introduced are now having a very visible impact, and we designed a new way of sending messages to work around that.

Please see the dnsdist website for the more complete changelog and the current documentation.

Release tarballs are available on the downloads website.

Several packages are also available on our repository. Please be aware that we have enabled a few additional features in our packages, like DNS over TLS and DNSTap support, on distributions where the required dependencies were available.

dnsdist 1.3.3 released

We are very happy to announce the 1.3.3 release of dnsdist. This release contains a few new features, but is mostly fixing a security issue reported since the release of dnsdist 1.3.2.

Security fix

While working on a new feature, Richard Gibson noticed that it was possible for a remote attacker to craft a DNS query with trailing data such that the addition of a record by dnsdist, for example an OPT record when adding EDNS Client Subnet, might result in the trailing data being smuggled to the backend as a valid record while not seen by dnsdist. This is an issue when dnsdist is deployed as a DNS Firewall and used to filter some records that should not be received by the backend. This issue occurs only when either the ‘useClientSubnet’ or the experimental ‘addXPF’ parameters are used when declaring a new backend.

While dnsdist has not had any important security issue until now, we decided this was a good time to implement the same security polling mechanism that the authoritative server and the recursor have had for years. Starting with this release, dnsdist will regularly perform a security check using a DNS query to determine whether the current version is up-to-date security-wise, and let the administrator know otherwise.

Important changes

It is sometimes very useful to be able to generate answers directly from dnsdist, to quickly return a “No such domain” answer, spoof an “A” or “AAAA” answer, or even just reply with the TC bit set so that legitimate clients retry over TCP. Until now, answers generated that way were mirroring the flags and EDNS options, if any, of the initial query. This was not great because it could mislead the client into thinking that dnsdist, or the server behind it, was supporting features or a UDP payload size it did not.

Starting with this release, dnsdist is now generating a proper EDNS payload if the query had one, and responding without EDNS otherwise. This behavior can be turned off using the new setAddEDNSToSelfGeneratedResponses() directive if needed.

We must, however, provide a responder’s maximum payload size in this record, and we can’t easily know the maximum payload size of the actual backend so we picked a safe default value of 1500, which can be overridden using the new  setPayloadSizeOnSelfGeneratedAnswers() directive.

New features and improvements

A new load-balancing policy named “chashed” has been introduced, based on consistent hashing. This new policy load-balances the incoming queries based on a hash of the requested name, like the existing “whashed” one, but has the interesting property that adding or removing a server will only cause a very small portion of the incoming queries to be mapped to a different server than they were before, keeping the caches warm.

While we have been supporting the export of metrics using the well-known carbon protocol from day one, we have seen an increasing demand for supporting the emerging Prometheus protocol. Thanks to the work of Pavel Odintsov and Kai S, dnsdist now supports it natively.

Very large installations of the DNS over TLS feature introduced in 1.3.0 reported several issues that we addressed in this release:

  • dnsdist did not set TCP_NODELAY on its TLS sockets, causing needless latency ;
  • it was not possible to configure the number of stored TLS sessions ;
  • our OpenSSL implementation had a memory leak when some clients aborted prematurely because of a negotiation error during the TLS handshake.

We seized the opportunity to refactor the part of the code handling TLS connections with the use of smart pointers while fixing that last issue, making sure that this kind of memory leak will not happen again.

In 1.3.2, the optimized DynblockRulesGroup introduced in 1.3.0 gained the ability to whitelist and blacklist ranges from dynamic rules, for example to prevent some clients from ever being blocked by a rate-limiting rule. This feature has now been made available when our in-kernel eBPF filtering feature is used as well. At the same time, we introduced the ability to set up warning rates to the dynamic rules, making it possible to get an alert without blocking clients when they reach a configured rate, and to block them should they reach a higher rate.

Finally, we introduced several new rules to our existing set:

  • EDNSOptionRule, to be able to filter based on the presence of a given EDNS option ;
  • DSTPortRule, offering the ability to route queries by looking at their destination port ;
  • PoolAvailableRule, to be able to route queries based on whether a pool has at least one usable backend.

Please see the dnsdist website for the more complete changelog and the current documentation.

Release tarballs are available on the downloads website.

Several packages are also available on our repository. Please be aware that we have enabled a few additional features in our packages, like DNS over TLS and DNSTap support, on distributions where the required dependencies were available.

dnsdist 1.3.2 released

We are very happy to announce the 1.3.2 release of dnsdist. This release contains a few new features, but is mostly fixing bugs and documentation issues reported since the release of dnsdist 1.3.0. You might be wondering why this release is not numbered 1.3.1, we discovered a build issue on some platforms right after tagging 1.3.1 and therefore decided to release 1.3.2 right away.

Breaking changes

After discussing with several users, we noticed that quite a lot of them were not aware that enabling the dnsdist’s console without a key, even restricted to the local host, could be a security issue and allow privilege escalation by allowing an unprivileged user to connect to the console and execute Lua code as the dnsdist user. We therefore decided to refuse any connection to the console until a key has been set, so please check that you do set a key before upgrading if you use the console.

New features

The DNS over TLS feature introduced in 1.3.0 was missing the ability to support both an RSA and an ECDSA certificate at the same time, and it was not possible to switch to a new certificate without restarting dnsdist. This has now been fixed.

The packet cache has also been improved in this release, with the addition of a negative TTL option to be able to specify how long NODATA and NXDOMAIN answers should be cached, as well as a way to dump the content of the cache. We also made the detection of ECS collisions more robust, preventing two queries for the same name, type and class but a different ECS subnet from colliding even if they did hash to the same value.

This version gained the ability to insert dynamic rules that do nothing, and do not stop the processing of subsequent rules, which is very useful for testing purposes. The optimized DynblockRulesGroup introduced in 1.3.0 also gained the ability to whitelist and blacklist ranges from dynamic rules, for example to prevent some clients from ever being blocked by a rate-limiting rule.

Finally, we introduced the new SetECSAction directive to be able to force the ECS value sent to a downstream server for some or all queries.

Bug fixes

In addition to various documentation and cosmetics fixes, a few annoying bugs have been fixed in this release:

  • If the first connection attempt to a given backend failed, dnsdist didn’t properly reconnect even when the backend became available ;
  • Dynamic blocks were sometimes created with the wrong duration ;
  • The ability to iterate over the results of the Lua exceed*() functions was broken in 1.3.0, preventing manual whitelisting from Lua ;
  • Some statistics were displayed with too many decimals in the web interface ;
  • A backend outstanding queries counter could become wrong if it dropped a lot of queries for a while.

 

Please see the dnsdist website for the more complete changelog and the current documentation.

Release tarballs are available on the downloads website.

Several packages are also available on our repository.

dnsdist 1.3.0 released

We are very happy to announce the 1.3.0 release of dnsdist, with a huge emphasis on privacy and scalability.

Privacy

A lot of users were interested in DNS over TLS support in dnsdist, to protect the privacy and integrity of queries and responses in transit between the client and dnsdist. We have been supporting DNSCrypt since 1.0.0, and improved it in this release by adding support for multiple active certificates and the new xchacha20 algorithm, but DNS over TLS is getting more traction and it made complete sense to support it as well in dnsdist. Our implementation can use either OpenSSL or GnuTLS, and we advise to enable both backends during compilation in order to be able to quickly switch from one to another should a serious vulnerability in one of them be found.

Scalability

As dnsdist is deployed on huge setups, we noticed that it did not scale as well as we expected over a large number of CPU cores. We investigated and found several points of contention, which we addressed by going lock-less whenever possible, and by reducing the granularity of the involved locks when it was not. This led to the optional sharding of the packet cache and our in-memory ring buffers, as well as a new per-pool mutex replacing the global Lua one for non-Lua load-balancing policies.

We had known for a while that dnsdist opening a single socket towards each backend was not performing too well in some scenarios, for example in front of a PowerDNS Recursor with multiple threads, reuseport support enabled and pdns-distribute-queries set to no, because the kernel would then not distribute queries evenly over the different threads. A known work-around was to add the same backend several times in the configuration, but it made metrics hard to understand and caused an unnecessary amount of contexts switching. Starting with 1.3.0, dnsdist supports opening a configurable amount of sockets towards a single backend.

Finally we observed that CPU pinning made a huge difference on some setups, especially on NUMA architecture, so we added the possibility to pin client and backend facing threads to specific CPU cores.

XPF

The solution to pass the client IP on to the backend in dnsdist has always been to add an EDNS Client Subnet option to the query. While it does work nicely, ECS was not designed for this use case and thus lacks some relevant information like the original source and destination ports, as well as the original destination IP. It also makes it impossible to keep any existing ECS information and forward the original source IP.
In coordination with the nice people from ISC, PowerDNS is working on a new solution called XPF, whose current draft is now implemented in dnsdist.

dnstap

In addition to our existing protocol buffer-based solution to export live information on queries and responses processed by dnsdist, Justin Valentini and Chris Hofstaedtler contributed support for exporting queries and responses over the dnstap protocol, which is supported by several other open source DNS servers and can be processed by third party tools.

Older versions

With the release of 1.3.0 today, we are also announcing that the 1.0 and 1.1 branches of dnsdist are now end of life and will not receive any updates, not even security fixes.
Note: Users with a commercial agreement with PowerDNS.COM BV or Open-Xchange can receive extended support for releases which are End Of Life. If you are such a user, these EOL statements do not apply to you.

Other Changes

As a final note, please be aware of three noteworthy changes in this new version:

  • First we removed the –daemon option, in which we kept finding new bugs. Very few users were actually using it, and since most OS provide at least one supervisor we decided to simply remove it ;
  •  Secondly we added the possibility to restrict access to the console using an ACL when it’s bound to a non-loopback IP. The default ACL allows connections from 127.0.0.1 and ::1 only, so you might need to update it to keep using the console over the network. Please make sure that you have enabled encryption before doing so ;
  • We finally removed some functions that were deprecated in 1.2.0 because they were redundant and made it harder to understand how the rules and actions actually work. Please have a look at the documentation to update your configuration.

Please see the dnsdist website for the more complete changelog and the current documentation.

Release tarballs are available on the downloads website.

Several packages are also available on our repository.