Category: powerdns

dnsdist 1.6.1 released

Hello!

We are happy to release dnsdist 1.6.1 today, a maintenance release fixing a few bugs reported since 1.6.0:

  • Adding ECS failed for queries with records in the answer or additional section (Dimitrios Mavrommatis)
  • The transport was not properly set in dnstap and protobuf messages for DoH queries
  • The outstanding queries counter was not properly reset when some TCP I/O errors occurred
  • The ability to load a new certificate on a DoH frontend was missing
  • A missing header could have caused a compilation issue on some platforms

As usual there were also other smaller enhancements and fixes, please see the dnsdist website for the more complete changelog and the current documentation.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

The release tarball (signature) is available on the downloads website, and packages for CentOS 7 and 8, Debian Buster and Bullseye, and Ubuntu Bionic and Focal are available from our repository.

DNS cache snooping attack

We have been getting questions about “DNS Server Cache Snooping Remote Information Disclosure” attacks lately, mostly coming from reports generated by one very popular security scanner:

DNS Server Cache Snooping Remote Information Disclosure

Synopsis:
The remote DNS server is vulnerable to cache snooping attacks.

Description:
The remote DNS server responds to queries for third-party domains that do not have the recursion bit set.
This may allow a remote attacker to determine which domains have recently been resolved via this name server, and therefore which hosts have been recently visited.
For instance, if an attacker was interested in whether your company utilizes the online services of a particular financial institution, they would be able to use this attack to build a statistical model regarding company usage of that financial institution.
Of course, the attack can also be used to find B2B partners, web-surfing patterns, external mail servers, and more.

Note: If this is an internal DNS server not accessible to outside networks, attacks would be limited to the internal network.

This may include employees, consultants and potentially users on a guest network or WiFi connection if supported.

Risk factor:
Medium

CVSS Base Score:
5.0

CVSS2#AV:
N/AC:L/Au:N/C:P/I:N/A:N

See also:
http://www.rootsecure.net/content/downloads/pdf/dns_cache_snooping.pdf

Solution:
Contact the vendor of the DNS software for a fix.

So what is this about?

The idea behind this “attack” is to find out whether a given recursive DNS server has been asked to resolve a given domain name recently. This might in theory be used by an authorized user to know which domains would be worth targeting for mis-typing attacks.

It cannot, however, be used to get the whole content of the cache since one needs to know which domains to query, and of course can’t be used either to know who requested that domain.

How does this work?

It works by exploiting the fact that DNS resolvers do not perform actual resolution for every query they get, instead they all rely on one or several caches, allowing them to remember the responses they have recently received for a certain time, up to the “TTL” value of the response. So if we can determine that a given domain is in the cache, we know that it was queried at most “TTL” seconds ago.

The easiest way to determine if a recursive server has a given domain in cache is to ask: sending a DNS query with the recursion desired bit cleared (RD=0) will only return results from the resolver cache, as per RFC 1034. If information is available about the requested name and type, it is returned, otherwise an empty ‘No Error’ answer is sent.

Should the server refuse to answer these queries, it is also possible to know whether an answer comes from the cache by looking at the TTL returned by the recursive server for a RD=1 request, and comparing it to the original TTL returned by a name server authoritative for that domain. If the TTL returned by the recursor is equal to the one returned by the authoritative server, it was likely not in cache, or was cached less than a second ago. Otherwise it comes from the recursor cache.

We could also precisely measure the time taken by the server to respond, since in the absence of a cached answer the recursive server would have to contact an authoritative server over the network, likely increasing the response time by several milliseconds.

Why does the Recursion Desired feature exist in the first place?

That feature is an integral part of the DNS protocol, as described in RFC 1034, in particular for communication between a recursive server and an authoritative server, where recursion is indeed not desired. As some servers historically provided both authoritative and recursive services, it still makes sense today to be able to distinguish the client’s expectations and to advertise the server capabilities.

It should also be noted that it is extremely useful to be able to use RD=0 queries to remotely inspect the content of the cache for a given name when troubleshooting operational issues in production. The alternative would require connecting to the running recursor process using rec_control and dumping the whole cache.

Mitigations

The first method might be mitigated by refusing RD=0 queries, for example using dnsdist:

addAction(NotRule(RDRule()), RCodeAction(DNSRCode.REFUSED))

It might however break existing clients and setups, since the RD=0 behaviour has been relied on for decades. Moreover, there is no way to mitigate the second and third ones without violating fundamental DNS specifications and impacting performance.

Therefore our recommendation is to simply ignore this “issue”, since there is no clear threat-model where it is actually relevant. This is also the conclusion that the fine folks of ISC reached regarding their BIND DNS server.

dnsdist 1.6.0 released

Hello!

We are proud to announce the final release of dnsdist 1.6.0, with no changes since the second release candidate. Compared to 1.5.x, this release contains several new exciting features, as well as improvements and bug fixes.

In our view, the most exciting new feature is the support of out-of-order processing for TCP and DNS over TLS connections. Out-of-order processing makes it possible to have several concurrent queries on the same TCP connection, and to receive the answers to these queries as soon as they are ready. Along with connection reuse, this reduces the overhead of TCP by a huge factor. Starting with 1.6.0, dnsdist will accept up to 65536 concurrent queries on the same incoming TCP connection, and will pass all of these to the backend over a single connection as well, provided that the backend supports it. This feature is not enabled by default, and can be enabled via the maxInFlight parameter of the addLocal/addTLSLocal (client-side) and the newServer (backend-side) commands.

This new version also brings support for accepting a Proxy Protocol header on incoming connections, making it possible for a frontend to provide dnsdist with the initial source and destination ports and addresses, as well as custom values. dnsdist can then process, add and remove values before passing the information to the backend. Chaining two dnsdist instances has never been this easy!

Other new features include the ability to define custom web endpoints in Lua, to extend the existing API, as well as the ability to create blazing-fast, lock-less per-thread custom load-balancing policies using the Lua foreign function interface (FFI).

Among the many improvements, dnsdist’s packet cache no longer hashes EDNS Cookies by default, which means that two queries that are identical except for the content of their cookies will now be served the same answer. Note that it might be necessary to restore the existing behaviour when dnsdist is in front of a backend actually using EDNS Cookies, which can be done via the cookieHashing parameter to newPacketCache.

Users of our own protocol buffer logging mechanism, or of dnstap, will be happy to learn that we replaced our implementation based on Google’s protocol buffer library by a tremendously faster one, based on the protozero library. This change results in much lower CPU utilization and increased scalability in a transparent way.

The memory usage of idle DNS over HTTPS and DNS over TLS connections has also been significantly reduced when the OpenSSL provider is used.

If you are upgrading from a previous version, please be aware that a few actions and commands have been renamed to clear some ambiguities. Almost all actions that allow further processing of rules now start with ‘Set’, to prevent mistakes:

Some commands changing the order of the rules could have easily been confused with the ones providing insight into the current traffic, and have therefore also been renamed:

Please also note that the use of additional parameters on the webserver command has been deprecated in favor of using setWebserverConfig.

Regular users should not be impacted by this change, but packagers should be aware that since 1.6.0 dnsdist now uses the C++17 standard instead of the C++11 one it was previously using.

Please see the dnsdist website for the more complete changelog and the current documentation.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

The release tarball (signature) is available on the downloads website, and packages for CentOS 7 and 8, Debian Buster and Ubuntu Bionic and Focal are available from our repository.

With this release, the 1.3.x releases are EOL and the 1.4.x releases go into critical security fixes only mode.

We would also like to take this opportunity to announce that we will stop supporting systems using 32-bit time. This includes 32-bit Linux platforms like arm and i386 before kernel version 5.1.

Finally, we would like to thank the PowerDNS community and all external contributors for their great work in this release, and in particular Stephane Bakhos, Stéphane Bortzmeyer, Georgeto, Matti Hiljanen, Andreas Jakum, Nuitari, Oli Schacher, Sukhbir Singh, Thibmac and Mischan Toosarani-Hausberger!

dnsdist 1.5.2 released

Hello!

We are happy to release dnsdist 1.5.2 today, a maintenance release fixing a few bugs reported since 1.5.1:

  • A typo in prometheus metrics dnsdist_frontend_tlshandshakefailures #9728 (AppliedPrivacy)
  • A hang when removing a server with more than one socket
  • SNI availability on resumed sessions, by acknowledging the name sent by the client
  • A crash when a DoH responses map is updated at runtime
  • Dynamic Block RCode rules messing up the queries count
  • EDNS in ServFail generated when no server is available
  • A crash with DynBPF objects in client mode
  • Add missing getEDNSOptions and getDO bindings for DNSResponse

As usual there were also other smaller enhancements and fixes, please see the dnsdist website for the more complete changelog and the current documentation.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

The release tarball (signature) is available on the downloads website, and packages for CentOS 7 and 8, Debian Buster and Ubuntu Bionic and Focal are available from our repository.

Second release candidate for dnsdist 1.6.0

Hi everyone,

We are happy to announce the second release candidate of what should become dnsdist 1.6.0. This release contains very few changes since the first release candidate, and thanks to the great feedback we received on previous versions we expect to be able to release 1.6.0 final very soon. The changed bits since -rc1 are:

  • Only use eBPF for “drop” actions, and clean up the eBPF rules more often
  • Fix missing locks in DNSCrypt certificates management
  • Make the backend queryLoad and dropRate values atomic

Please see the dnsdist website for the more complete changelog and the current documentation.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

The release tarball (signature) is available on the downloads website, and packages for CentOS 7 and 8, Debian Buster and Ubuntu Bionic and Focal are available from our repository.

With the future 1.6.0 final release, the 1.3.x releases will be EOL and the 1.4.x releases will go into critical security fixes only mode.

We would also like to take this opportunity to announce that we will stop supporting systems using 32-bit time. This includes 32-bit Linux platforms like arm and i386 before kernel version 5.1.

First release candidate for dnsdist 1.6.0

Hi everyone,

We are happy to announce the first release candidate of what should become dnsdist 1.6.0. This release contains very few changes since the third alpha:

  • Add missing getEDNSOptions and getDO bindings for DNSResponse
  • Fix some issues reported by Thread Sanitizer
  • Lua: don’t destroy keys during table iteration
  • Disable PMTU for IPv6 as well
  • Replace pthread_rwlock with std::shared_mutex

Please see the dnsdist website for the more complete changelog and the current documentation.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

We are grateful to the PowerDNS community for the reporting of bugs, issues, feature requests, and especially to the submitters of fixes and implementations of features.

Release tarballs are available on the downloads website, and packages for CentOS 7 and 8, Debian Buster and Ubuntu Bionic and Focal are available from our repository.

With the future 1.6.0 final release, the 1.3.x releases will be EOL and the 1.4.x releases will go into critical security fixes only mode.

We would also like to take this opportunity to announce that we will stop supporting systems using 32-bit time. This includes 32-bit Linux platforms like arm and i386 before kernel version 5.1.

Third Alpha Release of DNSDist 1.6.0

Hi everyone,

We are happy to announce the third alpha release of dnsdist 1.6.0. This release contains a few fixes for issues reported in the second alpha:

  • DNS over HTTPS queries with a non-zero ID were not properly handled. Very few DoH clients actually send an ID with a value different than 0 but it does happen and is allowed by RFC 8484. Many thanks to Frank Denis for reporting the issue !
  • The connect timeout was not used for outgoing TCP connections, and the write timeout was used instead.

In addition to these fixes, several improvements were made:

  • Reduced memory usage for idle DNS over HTTPS and DNS over TLS connections, saving roughly 35 kB per connection.
  • Smarter caching of outgoing TCP connections, ability to configure the number of concurrent incoming TCP connections per frontend, with more metrics.
  • Sharding has been enabled in the ring buffers and the packet cache by default, leading to better performance in the default configuration.
  • TLS renegotiation is now disabled by default, to prevent issues like CVE-2021-3449 in the future.

Please see the dnsdist website for the more complete changelog and the current documentation.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

Release tarballs are available on the downloads website, and packages for CentOS 7 and 8, Debian Buster and Ubuntu Bionic and Focal are available from our repository.

With the future 1.6.0 final release, the 1.3.x releases will be EOL and the 1.4.x releases will go into critical security fixes only mode.

We would also like to take this opportunity to announce that we will stop supporting systems using 32-bit time. This includes 32-bit Linux platforms like arm and i386 before kernel version 5.1.

Second Alpha Release of DNSDist 1.6.0

Hi everyone,

We are happy to announce the second alpha release of dnsdist 1.6.0. This release contains mostly fixes for issues reported in the first release candidate:

  • A race condition was found to sometimes occur at startup, making it possible for the first TCP connection to happen before the creation of TCP workers and lead to a crash.
  • Stéphane Bortzmeyer reported many TCP timeouts with the first alpha that did not happen with 1.5.x. We unfortunately did not manage to reproduce these timeouts, but we spent quite some time expanding the coverage of our TCP code, uncovering several bugs in the process. Although we unfortunately cannot be sure that the issue experienced by Stéphane has been fixed, the resulting code has seen much more testing and we have received excellent feedback from other users in the meantime, leading to this second alpha candidate.
  • The cache cleaning algorithm did not properly remove expired entries from all shards, when more than one shard was used and setCacheCleaningPercentage set below 100%. This led to a drop in the cache efficiency in the long run.
  • A null pointer dereference has been found when accessing a dynamic BPF block (DynBPF) object in client mode.
  • A debug line was not properly removed in the web server code, logging a new line for every HTTP query.

In addition to these fixes, Sander Hoentjen contributed several improvements to allow spoofing answers with multiple records, and Aki Tuomi introduced automatic conversion to string for several objects in Lua. Many thanks to them!

Please see the dnsdist website for the more complete changelog and the current documentation.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

Release tarballs are available on the downloads website, and packages for CentOS 7 and 8, Debian Buster and Ubuntu Bionic and Focal are available from our repository.

With the future 1.6.0 final release, the 1.3.x releases will be EOL and the 1.4.x releases will go into critical security fixes only mode.

We would also like to take this opportunity to announce that we will stop supporting systems using 32-bit time. This includes 32-bit Linux platforms like arm and i386 before kernel version 5.1.

First Alpha Release of DNSDist 1.6.0

Hello!

We are proud to announce the first alpha release of dnsdist 1.6.0. This release contains several new exciting features, as well as improvements and bug fixes.

In our view, the most exciting new feature is the support of out-of-order processing for TCP and DNS over TLS connections. Out-of-order processing makes it possible to have several concurrent queries on the same TCP connection, and to receive the answers to these queries as soon as they are ready. Along with connection reuse, this reduces the overhead of TCP by a huge factor. Starting with 1.6.0, dnsdist will accept up to 65536 concurrent queries on the same incoming TCP connection, and will pass all of these to the backend over a single connection as well, provided that the backend supports it. This feature is not enabled by default, and can be enabled via the maxInFlight parameter of the addLocal/addTLSLocal (client-side) and the newServer (backend-side) commands.

This new version also brings support for accepting a Proxy Protocol header on incoming connections, making it possible for a frontend to provide dnsdist with the initial source and destination ports and addresses, as well as custom values. dnsdist can then process, add and remove values before passing the information to the backend. Chaining two dnsdist instances has never been this easy!

Other new features include the ability to define custom web endpoints in Lua, to extend the existing API, as well as the ability to create blazing-fast, lock-less per-thread custom load-balancing policies using the Lua foreign function interface (FFI).

Among the many improvements, dnsdist’s packet cache no longer hashes EDNS Cookies by default, which means that two queries that are identical except for the content of their cookies will now be served the same answer. Note that it might necessary to restore the existing behaviour when dnsdist is in front of a backend actually using EDNS Cookies, which can be done via the cookieHashing parameter to newPacketCache.

Users of our own protocol buffer logging mechanism, or of dnstap, will be happy to learn that we replaced our implementation based on Google’s protocol buffer library by a tremendously faster one, based on the protozero library. This change results in much lower CPU utilization and increased scalability in a transparent way.

If you intend to test this alpha release, for which we would be very grateful, please be aware that a few actions and commands have been renamed to clear some ambiguities. Almost all actions that allow further processing of rules now start with ‘Set’, to prevent mistakes:

Some commands changing the order of the rules could have easily been confused with the ones providing insight into the current traffic, and have therefore also been renamed:

Please also note that the use of additional parameters on the webserver command has been deprecated in favor of using setWebserverConfig.

Regular users should not be impacted by this change, but packagers should be aware that since 1.6.0 dnsdist now uses the C++17 standard instead of the C++11 one it was previously using.

Please see the dnsdist website for the more complete changelog and the current documentation.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

Release tarballs are available on the downloads website, and packages for CentOS 7 and 8, Debian Buster and Ubuntu Bionic and Focal are available from our repository.

With the future 1.6.0 final release, the 1.3.x releases will be EOL and the 1.4.x releases will go into critical security fixes only mode.

We would also like to take this opportunity to announce that we will stop supporting systems using 32-bit time. This includes 32-bit Linux platforms like arm and i386 before kernel version 5.1.

Finally, we would like to thank the PowerDNS community and all external contributors for their great work in this release, and in particular Stephane Bakhos, Georgeto, Matti Hiljanen, Nuitari, Sukhbir Singh and Mischan Toosarani-Hausberger!

First alpha release of dnsdist 1.5.0

We are very happy to announce the 1.5.0 alpha 1 release of dnsdist. This version contains several new exciting features detailed below, but also a few breaking changes so please take the time to read the next section.

Your feedback will be much appreciated so we can deliver a stable 1.5.0 final release!

Important changes

We took the opportunity of this new release to clean up a few things that might require updating your existing configuration.

First, in systemd environments, dnsdist used to be started as root before dropping privileges and switching to an unprivileged user, which could lead to weird issues where files where readable during startup but not after, or the other way around. This is no longer the case, and dnsdist is now directly started as an unprivileged user. This might require updating the permissions on the files accessed during startup. It is therefore recommended to recursively chown directories used by dnsdist:

chown -R root:dnsdist /etc/dnsdist

Packages provided on the PowerDNS Repository will chown directories created by them accordingly in the post-installation steps.

We also updated the default behavior of our DNS over HTTPS implementation. DoH endpoints specified in the fourth parameter of addDOHLocal are now specified as exact paths instead of path prefixes.

For example,

addDOHLocal('2001:db8:1:f00::1', '/etc/ssl/certs/example.com.pem', '/etc/ssl/private/example.com.key', { "/dns-query" })

will now only accept queries for /dns-query and no longer for /dns-query/foo/bar.

The default endpoint also switched from / to /dns-query. That can be overridden through the fourth parameter of addDOHLocal().

Finally the default SSL/TLS library used for DNS over TLS was changed from GnuTLS to OpenSSL / LibreSSL, based on the feedback we received from our users.

Please see the upgrade guide for more information.

New features and improvements

The most exciting new feature is the implementation of the Proxy Protocol between dnsdist and its backends. Aimed to replace the use of EDNS Client Subnet and our own XPF, the Proxy Protocol is an existing standard where a small header is prepended to the query, passing not only the source and destination addresses and ports along to the backend, but also custom values. Support for parsing the Proxy Protocol is already available in the development tree of the PowerDNS Recursor.

We implemented a new spoofRawAction(), which makes it possible to spoof any kind of response from dnsdist, instead of the existing limitation to A, AAAA and CNAME records. This new action requires submitting the response in DNS wire-format.

While it has always been possible to write custom selectors and actions in Lua, there was a huge performance gap between built-in rules written in C++ and the Lua ones. This release adds the ability to use the Lua FFI interface available in LuaJIT to write high-performance selectors and rules, as well as load-balancing policies. With carefully written Lua, this delivers performances almost on par with the built-in C++ rules and actions, with greater flexibility.

Several very large-scale users reported that the load-balancing policies based on a hash of the qname could lack a bit of fairness when the traffic was heavily skewed toward a few names, leading to some backends receiving much more traffic than others. In order to address this shortcoming, we added the ability to set load bounds to the chashed and whashed policies so that queries will be dispatched to a different backend if the one selected based on the qname is already handling more queries than it should.

Our DNS over HTTPS implementation received several improvements, including the ability to send cache-control headers, and to parse X-Forwarded-For headers sent by a frontend.

Users with a large number of backends will be happy to know that we refactored the handling of health checks so that they can now be performed in parallel instead of sequentially, leading to a huge performance improvement.

Finally our remote logging features using DNSTAP or our own protobuf saw several performance enhancements, a better handling of re-connection events, and the addition of the source and destination ports of the query whenever possible.

We want to once again thank everyone that contributed to the testing of the previous release candidates!

Please see the dnsdist website for the more complete changelog and the current documentation.

Release tarballs are available on the downloads website.

Several packages are also available in our repository. Please be aware that we have enabled a few additional features in our packages, like DNS over HTTPS, DNS over TLS and DNSTAP support, on distributions where the required dependencies were available.