Category: dnsdist

dnsdist 1.3.3 released

We are very happy to announce the 1.3.3 release of dnsdist. This release contains a few new features, but is mostly fixing a security issue reported since the release of dnsdist 1.3.2.

Security fix

While working on a new feature, Richard Gibson noticed that it was possible for a remote attacker to craft a DNS query with trailing data such that the addition of a record by dnsdist, for example an OPT record when adding EDNS Client Subnet, might result in the trailing data being smuggled to the backend as a valid record while not seen by dnsdist. This is an issue when dnsdist is deployed as a DNS Firewall and used to filter some records that should not be received by the backend. This issue occurs only when either the ‘useClientSubnet’ or the experimental ‘addXPF’ parameters are used when declaring a new backend.

While dnsdist has not had any important security issue until now, we decided this was a good time to implement the same security polling mechanism that the authoritative server and the recursor have had for years. Starting with this release, dnsdist will regularly perform a security check using a DNS query to determine whether the current version is up-to-date security-wise, and let the administrator know otherwise.

Important changes

It is sometimes very useful to be able to generate answers directly from dnsdist, to quickly return a “No such domain” answer, spoof an “A” or “AAAA” answer, or even just reply with the TC bit set so that legitimate clients retry over TCP. Until now, answers generated that way were mirroring the flags and EDNS options, if any, of the initial query. This was not great because it could mislead the client into thinking that dnsdist, or the server behind it, was supporting features or a UDP payload size it did not.

Starting with this release, dnsdist is now generating a proper EDNS payload if the query had one, and responding without EDNS otherwise. This behavior can be turned off using the new setAddEDNSToSelfGeneratedResponses() directive if needed.

We must, however, provide a responder’s maximum payload size in this record, and we can’t easily know the maximum payload size of the actual backend so we picked a safe default value of 1500, which can be overridden using the new  setPayloadSizeOnSelfGeneratedAnswers() directive.

New features and improvements

A new load-balancing policy named “chashed” has been introduced, based on consistent hashing. This new policy load-balances the incoming queries based on a hash of the requested name, like the existing “whashed” one, but has the interesting property that adding or removing a server will only cause a very small portion of the incoming queries to be mapped to a different server than they were before, keeping the caches warm.

While we have been supporting the export of metrics using the well-known carbon protocol from day one, we have seen an increasing demand for supporting the emerging Prometheus protocol. Thanks to the work of Pavel Odintsov and Kai S, dnsdist now supports it natively.

Very large installations of the DNS over TLS feature introduced in 1.3.0 reported several issues that we addressed in this release:

  • dnsdist did not set TCP_NODELAY on its TLS sockets, causing needless latency ;
  • it was not possible to configure the number of stored TLS sessions ;
  • our OpenSSL implementation had a memory leak when some clients aborted prematurely because of a negotiation error during the TLS handshake.

We seized the opportunity to refactor the part of the code handling TLS connections with the use of smart pointers while fixing that last issue, making sure that this kind of memory leak will not happen again.

In 1.3.2, the optimized DynblockRulesGroup introduced in 1.3.0 gained the ability to whitelist and blacklist ranges from dynamic rules, for example to prevent some clients from ever being blocked by a rate-limiting rule. This feature has now been made available when our in-kernel eBPF filtering feature is used as well. At the same time, we introduced the ability to set up warning rates to the dynamic rules, making it possible to get an alert without blocking clients when they reach a configured rate, and to block them should they reach a higher rate.

Finally, we introduced several new rules to our existing set:

  • EDNSOptionRule, to be able to filter based on the presence of a given EDNS option ;
  • DSTPortRule, offering the ability to route queries by looking at their destination port ;
  • PoolAvailableRule, to be able to route queries based on whether a pool has at least one usable backend.

Please see the dnsdist website for the more complete changelog and the current documentation.

Release tarballs are available on the downloads website.

Several packages are also available on our repository. Please be aware that we have enabled a few additional features in our packages, like DNS over TLS and DNSTap support, on distributions where the required dependencies were available.

dnsdist 1.3.2 released

We are very happy to announce the 1.3.2 release of dnsdist. This release contains a few new features, but is mostly fixing bugs and documentation issues reported since the release of dnsdist 1.3.0. You might be wondering why this release is not numbered 1.3.1, we discovered a build issue on some platforms right after tagging 1.3.1 and therefore decided to release 1.3.2 right away.

Breaking changes

After discussing with several users, we noticed that quite a lot of them were not aware that enabling the dnsdist’s console without a key, even restricted to the local host, could be a security issue and allow privilege escalation by allowing an unprivileged user to connect to the console and execute Lua code as the dnsdist user. We therefore decided to refuse any connection to the console until a key has been set, so please check that you do set a key before upgrading if you use the console.

New features

The DNS over TLS feature introduced in 1.3.0 was missing the ability to support both an RSA and an ECDSA certificate at the same time, and it was not possible to switch to a new certificate without restarting dnsdist. This has now been fixed.

The packet cache has also been improved in this release, with the addition of a negative TTL option to be able to specify how long NODATA and NXDOMAIN answers should be cached, as well as a way to dump the content of the cache. We also made the detection of ECS collisions more robust, preventing two queries for the same name, type and class but a different ECS subnet from colliding even if they did hash to the same value.

This version gained the ability to insert dynamic rules that do nothing, and do not stop the processing of subsequent rules, which is very useful for testing purposes. The optimized DynblockRulesGroup introduced in 1.3.0 also gained the ability to whitelist and blacklist ranges from dynamic rules, for example to prevent some clients from ever being blocked by a rate-limiting rule.

Finally, we introduced the new SetECSAction directive to be able to force the ECS value sent to a downstream server for some or all queries.

Bug fixes

In addition to various documentation and cosmetics fixes, a few annoying bugs have been fixed in this release:

  • If the first connection attempt to a given backend failed, dnsdist didn’t properly reconnect even when the backend became available ;
  • Dynamic blocks were sometimes created with the wrong duration ;
  • The ability to iterate over the results of the Lua exceed*() functions was broken in 1.3.0, preventing manual whitelisting from Lua ;
  • Some statistics were displayed with too many decimals in the web interface ;
  • A backend outstanding queries counter could become wrong if it dropped a lot of queries for a while.

 

Please see the dnsdist website for the more complete changelog and the current documentation.

Release tarballs are available on the downloads website.

Several packages are also available on our repository.

dnsdist 1.3.0 released

We are very happy to announce the 1.3.0 release of dnsdist, with a huge emphasis on privacy and scalability.

Privacy

A lot of users were interested in DNS over TLS support in dnsdist, to protect the privacy and integrity of queries and responses in transit between the client and dnsdist. We have been supporting DNSCrypt since 1.0.0, and improved it in this release by adding support for multiple active certificates and the new xchacha20 algorithm, but DNS over TLS is getting more traction and it made complete sense to support it as well in dnsdist. Our implementation can use either OpenSSL or GnuTLS, and we advise to enable both backends during compilation in order to be able to quickly switch from one to another should a serious vulnerability in one of them be found.

Scalability

As dnsdist is deployed on huge setups, we noticed that it did not scale as well as we expected over a large number of CPU cores. We investigated and found several points of contention, which we addressed by going lock-less whenever possible, and by reducing the granularity of the involved locks when it was not. This led to the optional sharding of the packet cache and our in-memory ring buffers, as well as a new per-pool mutex replacing the global Lua one for non-Lua load-balancing policies.

We had known for a while that dnsdist opening a single socket towards each backend was not performing too well in some scenarios, for example in front of a PowerDNS Recursor with multiple threads, reuseport support enabled and pdns-distribute-queries set to no, because the kernel would then not distribute queries evenly over the different threads. A known work-around was to add the same backend several times in the configuration, but it made metrics hard to understand and caused an unnecessary amount of contexts switching. Starting with 1.3.0, dnsdist supports opening a configurable amount of sockets towards a single backend.

Finally we observed that CPU pinning made a huge difference on some setups, especially on NUMA architecture, so we added the possibility to pin client and backend facing threads to specific CPU cores.

XPF

The solution to pass the client IP on to the backend in dnsdist has always been to add an EDNS Client Subnet option to the query. While it does work nicely, ECS was not designed for this use case and thus lacks some relevant information like the original source and destination ports, as well as the original destination IP. It also makes it impossible to keep any existing ECS information and forward the original source IP.
In coordination with the nice people from ISC, PowerDNS is working on a new solution called XPF, whose current draft is now implemented in dnsdist.

dnstap

In addition to our existing protocol buffer-based solution to export live information on queries and responses processed by dnsdist, Justin Valentini and Chris Hofstaedtler contributed support for exporting queries and responses over the dnstap protocol, which is supported by several other open source DNS servers and can be processed by third party tools.

Older versions

With the release of 1.3.0 today, we are also announcing that the 1.0 and 1.1 branches of dnsdist are now end of life and will not receive any updates, not even security fixes.
Note: Users with a commercial agreement with PowerDNS.COM BV or Open-Xchange can receive extended support for releases which are End Of Life. If you are such a user, these EOL statements do not apply to you.

Other Changes

As a final note, please be aware of three noteworthy changes in this new version:

  • First we removed the –daemon option, in which we kept finding new bugs. Very few users were actually using it, and since most OS provide at least one supervisor we decided to simply remove it ;
  •  Secondly we added the possibility to restrict access to the console using an ACL when it’s bound to a non-loopback IP. The default ACL allows connections from 127.0.0.1 and ::1 only, so you might need to update it to keep using the console over the network. Please make sure that you have enabled encryption before doing so ;
  • We finally removed some functions that were deprecated in 1.2.0 because they were redundant and made it harder to understand how the rules and actions actually work. Please have a look at the documentation to update your configuration.

Please see the dnsdist website for the more complete changelog and the current documentation.

Release tarballs are available on the downloads website.

Several packages are also available on our repository.

dnsdist 1.2.1 released

We are very pleased to announce the availability of dnsdist 1.2.1, fixing several issues that were found in 1.2.0:

  • #5647: Make dnsdist dynamic truncate do right thing on TCP/IP
  • #5686: Add missing QPSAction
  • #5847: Don’t create a Remote Logger in client mode
  • #5858: Use libsodium’s CFLAGS, we might need them to find the includes
  • #6012: Keep the TCP connection open on cache hit, generated answers
  • #6041: Add the missing <sys/time.h> include to mplexer.hh for struct timeval
  • #6043: Sort the servers based on their ‘order’ after it has been set
  • #6073: Quiet unused variable warning on macOS (Chris Hofstaedtler)
  • #6094: Fix the outstanding counter when an exception is raised
  • #6164: Do not connect the snmpAgent from a dnsdist client

One new feature has also been added by Dan McCombs, allowing to work around an issue when dnsdist is compiled with IP_BIND_ADDRESS_NO_PORT enabled but run on a kernel that does not support it:

  • #5880: Add configuration option to disable IP_BIND_ADDRESS_NO_PORT

Finally, the handling of bracketed IPv6 addresses without port has been improved by Chris Hofstaedtler:

  • #6057: Handle bracketed IPv6 addresses without ports

Please see the dnsdist website for the more complete changelog and the current documentation.

Release tarballs are available on the downloads website.

Several packages are also available on our repository.

dnsdist 1.2.0 released

We are very pleased to announce the availability of dnsdist 1.2.0, bringing a lot of new features and fixes since 1.1.0.

This release also addresses two security issues of low severity, CVE-2016-7069 and CVE-2017-7557. The first issue can lead to a denial of service on 32-bit if a backend sends crafted answers, and the second to an alteration of dnsdist’s ACL if the API is enabled, writable and an authenticated user is tricked into visiting a crafted website. More information can be found in our security advisories 2017-01 and 2017-02.

Highlights include:

  • applying rules on cache hits
  • addition of runtime changeable rules that matches IP address for a certain time: TimedIPSetRule
  • SNMP support, exporting statistics and sending traps
  • preventing the packet cache from ageing responses when deployed in front of authoritative servers
  • TTL alteration capabilities
  • consistent hash results over multiple deployments
  • exporting CNAME records over protobuf
  • tuning the size of the ringbuffers used to keep track of recent queries and responses
  • various DNSCrypt-related fixes and improvements, including automatic key rotation

Users upgrading from a previous version should be aware that:

  •  the truncateTC option is now off by default, to follow the principle of least astonishment
  • the signature of the addLocal() and setLocal() functions has been changed, to make it easier to add new parameters without breaking existing configurations
  • the packet cache does not cache answers without any TTL anymore, to prevent them from being cached forever
  • blockfilter has been removed, since it was completely redundant

This release also deprecates a number of functions, which will be removed in 1.3.0. Those functions had the drawback of making dnsdist’s configuration less consistent by hiding the fact that each rule is composed of a selector and an action. They are still supported in 1.2.0 but a warning is displayed whenever they are used, and a replacement suggested.

For the many other new features, improvements and bug fixes, please see the dnsdist website for the more complete changelog, the current documentation, and the upgrade guide.

Release tarballs are available on the downloads website.

Several packages are also available on our repository.