Category: powerdns

Third Alpha Release of DNSDist 1.6.0

Hi everyone,

We are happy to announce the third alpha release of dnsdist 1.6.0. This release contains a few fixes for issues reported in the second alpha:

  • DNS over HTTPS queries with a non-zero ID were not properly handled. Very few DoH clients actually send an ID with a value different than 0 but it does happen and is allowed by RFC 8484. Many thanks to Frank Denis for reporting the issue !
  • The connect timeout was not used for outgoing TCP connections, and the write timeout was used instead.

In addition to these fixes, several improvements were made:

  • Reduced memory usage for idle DNS over HTTPS and DNS over TLS connections, saving roughly 35 kB per connection.
  • Smarter caching of outgoing TCP connections, ability to configure the number of concurrent incoming TCP connections per frontend, with more metrics.
  • Sharding has been enabled in the ring buffers and the packet cache by default, leading to better performance in the default configuration.
  • TLS renegotiation is now disabled by default, to prevent issues like CVE-2021-3449 in the future.

Please see the dnsdist website for the more complete changelog and the current documentation.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

Release tarballs are available on the downloads website, and packages for CentOS 7 and 8, Debian Buster and Ubuntu Bionic and Focal are available from our repository.

With the future 1.6.0 final release, the 1.3.x releases will be EOL and the 1.4.x releases will go into critical security fixes only mode.

We would also like to take this opportunity to announce that we will stop supporting systems using 32-bit time. This includes 32-bit Linux platforms like arm and i386 before kernel version 5.1.

Second Alpha Release of DNSDist 1.6.0

Hi everyone,

We are happy to announce the second alpha release of dnsdist 1.6.0. This release contains mostly fixes for issues reported in the first release candidate:

  • A race condition was found to sometimes occur at startup, making it possible for the first TCP connection to happen before the creation of TCP workers and lead to a crash.
  • Stéphane Bortzmeyer reported many TCP timeouts with the first alpha that did not happen with 1.5.x. We unfortunately did not manage to reproduce these timeouts, but we spent quite some time expanding the coverage of our TCP code, uncovering several bugs in the process. Although we unfortunately cannot be sure that the issue experienced by Stéphane has been fixed, the resulting code has seen much more testing and we have received excellent feedback from other users in the meantime, leading to this second alpha candidate.
  • The cache cleaning algorithm did not properly remove expired entries from all shards, when more than one shard was used and setCacheCleaningPercentage set below 100%. This led to a drop in the cache efficiency in the long run.
  • A null pointer dereference has been found when accessing a dynamic BPF block (DynBPF) object in client mode.
  • A debug line was not properly removed in the web server code, logging a new line for every HTTP query.

In addition to these fixes, Sander Hoentjen contributed several improvements to allow spoofing answers with multiple records, and Aki Tuomi introduced automatic conversion to string for several objects in Lua. Many thanks to them!

Please see the dnsdist website for the more complete changelog and the current documentation.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

Release tarballs are available on the downloads website, and packages for CentOS 7 and 8, Debian Buster and Ubuntu Bionic and Focal are available from our repository.

With the future 1.6.0 final release, the 1.3.x releases will be EOL and the 1.4.x releases will go into critical security fixes only mode.

We would also like to take this opportunity to announce that we will stop supporting systems using 32-bit time. This includes 32-bit Linux platforms like arm and i386 before kernel version 5.1.

First Alpha Release of DNSDist 1.6.0


We are proud to announce the first alpha release of dnsdist 1.6.0. This release contains several new exciting features, as well as improvements and bug fixes.

In our view, the most exciting new feature is the support of out-of-order processing for TCP and DNS over TLS connections. Out-of-order processing makes it possible to have several concurrent queries on the same TCP connection, and to receive the answers to these queries as soon as they are ready. Along with connection reuse, this reduces the overhead of TCP by a huge factor. Starting with 1.6.0, dnsdist will accept up to 65536 concurrent queries on the same incoming TCP connection, and will pass all of these to the backend over a single connection as well, provided that the backend supports it. This feature is not enabled by default, and can be enabled via the maxInFlight parameter of the addLocal/addTLSLocal (client-side) and the newServer (backend-side) commands.

This new version also brings support for accepting a Proxy Protocol header on incoming connections, making it possible for a frontend to provide dnsdist with the initial source and destination ports and addresses, as well as custom values. dnsdist can then process, add and remove values before passing the information to the backend. Chaining two dnsdist instances has never been this easy!

Other new features include the ability to define custom web endpoints in Lua, to extend the existing API, as well as the ability to create blazing-fast, lock-less per-thread custom load-balancing policies using the Lua foreign function interface (FFI).

Among the many improvements, dnsdist’s packet cache no longer hashes EDNS Cookies by default, which means that two queries that are identical except for the content of their cookies will now be served the same answer. Note that it might necessary to restore the existing behaviour when dnsdist is in front of a backend actually using EDNS Cookies, which can be done via the cookieHashing parameter to newPacketCache.

Users of our own protocol buffer logging mechanism, or of dnstap, will be happy to learn that we replaced our implementation based on Google’s protocol buffer library by a tremendously faster one, based on the protozero library. This change results in much lower CPU utilization and increased scalability in a transparent way.

If you intend to test this alpha release, for which we would be very grateful, please be aware that a few actions and commands have been renamed to clear some ambiguities. Almost all actions that allow further processing of rules now start with ‘Set’, to prevent mistakes:

Some commands changing the order of the rules could have easily been confused with the ones providing insight into the current traffic, and have therefore also been renamed:

Please also note that the use of additional parameters on the webserver command has been deprecated in favor of using setWebserverConfig.

Regular users should not be impacted by this change, but packagers should be aware that since 1.6.0 dnsdist now uses the C++17 standard instead of the C++11 one it was previously using.

Please see the dnsdist website for the more complete changelog and the current documentation.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

Release tarballs are available on the downloads website, and packages for CentOS 7 and 8, Debian Buster and Ubuntu Bionic and Focal are available from our repository.

With the future 1.6.0 final release, the 1.3.x releases will be EOL and the 1.4.x releases will go into critical security fixes only mode.

We would also like to take this opportunity to announce that we will stop supporting systems using 32-bit time. This includes 32-bit Linux platforms like arm and i386 before kernel version 5.1.

Finally, we would like to thank the PowerDNS community and all external contributors for their great work in this release, and in particular Stephane Bakhos, Georgeto, Matti Hiljanen, Nuitari, Sukhbir Singh and Mischan Toosarani-Hausberger!

First alpha release of dnsdist 1.5.0

We are very happy to announce the 1.5.0 alpha 1 release of dnsdist. This version contains several new exciting features detailed below, but also a few breaking changes so please take the time to read the next section.

Your feedback will be much appreciated so we can deliver a stable 1.5.0 final release!

Important changes

We took the opportunity of this new release to clean up a few things that might require updating your existing configuration.

First, in systemd environments, dnsdist used to be started as root before dropping privileges and switching to an unprivileged user, which could lead to weird issues where files where readable during startup but not after, or the other way around. This is no longer the case, and dnsdist is now directly started as an unprivileged user. This might require updating the permissions on the files accessed during startup. It is therefore recommended to recursively chown directories used by dnsdist:

chown -R root:dnsdist /etc/dnsdist

Packages provided on the PowerDNS Repository will chown directories created by them accordingly in the post-installation steps.

We also updated the default behavior of our DNS over HTTPS implementation. DoH endpoints specified in the fourth parameter of addDOHLocal are now specified as exact paths instead of path prefixes.

For example,

addDOHLocal('2001:db8:1:f00::1', '/etc/ssl/certs/', '/etc/ssl/private/', { "/dns-query" })

will now only accept queries for /dns-query and no longer for /dns-query/foo/bar.

The default endpoint also switched from / to /dns-query. That can be overridden through the fourth parameter of addDOHLocal().

Finally the default SSL/TLS library used for DNS over TLS was changed from GnuTLS to OpenSSL / LibreSSL, based on the feedback we received from our users.

Please see the upgrade guide for more information.

New features and improvements

The most exciting new feature is the implementation of the Proxy Protocol between dnsdist and its backends. Aimed to replace the use of EDNS Client Subnet and our own XPF, the Proxy Protocol is an existing standard where a small header is prepended to the query, passing not only the source and destination addresses and ports along to the backend, but also custom values. Support for parsing the Proxy Protocol is already available in the development tree of the PowerDNS Recursor.

We implemented a new spoofRawAction(), which makes it possible to spoof any kind of response from dnsdist, instead of the existing limitation to A, AAAA and CNAME records. This new action requires submitting the response in DNS wire-format.

While it has always been possible to write custom selectors and actions in Lua, there was a huge performance gap between built-in rules written in C++ and the Lua ones. This release adds the ability to use the Lua FFI interface available in LuaJIT to write high-performance selectors and rules, as well as load-balancing policies. With carefully written Lua, this delivers performances almost on par with the built-in C++ rules and actions, with greater flexibility.

Several very large-scale users reported that the load-balancing policies based on a hash of the qname could lack a bit of fairness when the traffic was heavily skewed toward a few names, leading to some backends receiving much more traffic than others. In order to address this shortcoming, we added the ability to set load bounds to the chashed and whashed policies so that queries will be dispatched to a different backend if the one selected based on the qname is already handling more queries than it should.

Our DNS over HTTPS implementation received several improvements, including the ability to send cache-control headers, and to parse X-Forwarded-For headers sent by a frontend.

Users with a large number of backends will be happy to know that we refactored the handling of health checks so that they can now be performed in parallel instead of sequentially, leading to a huge performance improvement.

Finally our remote logging features using DNSTAP or our own protobuf saw several performance enhancements, a better handling of re-connection events, and the addition of the source and destination ports of the query whenever possible.

We want to once again thank everyone that contributed to the testing of the previous release candidates!

Please see the dnsdist website for the more complete changelog and the current documentation.

Release tarballs are available on the downloads website.

Several packages are also available in our repository. Please be aware that we have enabled a few additional features in our packages, like DNS over HTTPS, DNS over TLS and DNSTAP support, on distributions where the required dependencies were available.

PowerDNS Recursor 4.1.9 Released

We are very happy to announce the 4.1.9 release of the PowerDNS Recursor. This release is fixing two security issues, and addressing a shortcoming in the way incoming queries are distributed to threads under heavy load.This release fixes the following security issues:

  • PowerDNS Security Advisory 2019-01 (CVE-2019-3806): Lua hooks are not called over TCP
  • PowerDNS Security Advisory 2019-02 (CVE-2019-3807): DNSSEC validation is not performed for AA=0 responses

These issues respectively affect PowerDNS Recursor from 4.1.4 and 4.1.0, up to and including 4.1.8.  PowerDNS Recursor 4.0.x and below are not affected.

Minimal patches are available at and

The changelog:

  • #7397: Load the Lua script in the distributor thread, check signature for AA=0 answers (CVE-2019-3806, CVE-2019-3807)
  • #7377: Try another worker before failing if the first pipe was full

The tarball (signature) is available at and packages for CentOS 6 and 7, Debian Jessie and Stretch, Ubuntu Bionic, Trusty and Xenial are available from

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.


PowerDNS Authoritative Server 4.1.4 released

We’re happy to announce the release of the PowerDNS Authoritative server version 4.1.4. This maintenance release features several bug fixes. The two most important being a fix in the handling of child-zones with DNS UPDATE and one where orphaned record comments would mess up records returned by the API.

Here is the list of changes, the full changelog is available in the documentation:


  • #6590: Fix warnings reported by gcc 8.1.0.
  • #6632, #6844, #6842, #6848: Make the gmysql backend future-proof. (Chris Hofstaedtler)
  • #6685, #6686: Initialize some missed qtypes. (Chris Hofstaedtler)

Bug Fixes

  • #6780: Avoid concurrent records/comments iteration from running out of sync.
  • #6816: Fix a crash in the API when adding records.
  • #4457, #6691: pdns_control notify: handle slave without renotify properly. (Chris Hofstaedtler)
  • #6736, #6738: Reset the TSIG state between queries.
  • #6857: Remove SOA-check backoff on incoming notify and fix lock handling. (Klaus Darilion)
  • #6858: Fix an issue where updating a record via DNS-UPDATE in a child zone that also exists in the parent zone, we would incorrectly apply the update to the parent zone.
  • #6676, #6677: Geoipbackend: check geoip_id_by_addr_gl and geoip_id_by_addr_v6_gl return value. (Aki Tuomi)

The tarball is here (sig) and packages for CentOS 6 and 7, Ubuntu Trusty, Xenial and Bionic and Debian Jessie and Stretch are available from the repositories.

PowerDNS Recursor 4.1.3 Released

This release improves the stability and resiliency of the RPZ implementation, prevents metrics gathering from slowing down the processing of DNS queries and fixes an issue related to the cleaning of EDNS Client Subnet entries from the cache.

The full changelog looks like this:


  • #6550, #6562: Add a subtree option to the API cache flush endpoint.
  • #6566: Use a separate, non-blocking pipe to distribute queries.
  • #6567: Move carbon/webserver/control/stats handling to a separate thread.
  • #6583: Add _raw versions for QName / ComboAddresses to the FFI API.
  • #6611, #6130: Update copyright years to 2018 (Matt Nordhoff).
  • #6474, #6596, #6478: Fix a warning on botan >= 2.5.0.

Bug Fixes

  • #6313: Count a lookup into an internal auth zone as a cache miss.
  • #6467: Don’t increase the DNSSEC validations counters when running with process-no-validate.
  • #6469: Respect the AXFR timeout while connecting to the RPZ server.
  • #6418, #6179: Increase MTasker stacksize to avoid crash in exception unwinding (Chris Hofstaedtler).
  • #6419, #6086: Use the SyncRes time in our unit tests when checking cache validity (Chris Hofstaedtler).
  • #6514, #6630: Add -rdynamic to C{,XX}FLAGS when we build with LuaJIT.
  • #6588, #6237: Delay the loading of RPZ zones until the parsing is done, fixing a race condition.
  • #6595, #6542, #6516, #6358, #6517: Reorder includes to avoid boost L conflict.

The tarball is available on (signature) and packages for CentOS 6 and 7, Debian Jessie and Stretch, Ubuntu Artful, Bionic, Trusty and Xenial are available from

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

Technical Preview Releases of Authoritative Server, Recursor and dnsdist

Hi everybody!

As recently announced, we have finished the great PowerDNS 4.x Spring Cleaning. And it was indeed kind of grand. We consciously set out to fix many things that had been waiting for years to be addressed. We took the liberty to change many things that we could not change (break) within 3.x.  However, it was breaking for the better.

As noted in our previous post, we are very grateful to our community, users, developers and customers that we were able to devote significant time to cleaning up past mistakes. This is very rare in the world of software. Additionally, as usual a specific shout-out to Aki Tuomi (these days working for our sister-company Dovecot), our certified consultants Kees Monshouwer, Christian Hofstaedtler and Jan-Piet Mens, our independent code-contributors Ruben Kerkhof, Ruben d’Arco, Mark Zealey, Pavel Boldin, Mark Schouten and all the others who contributed ideas, code and GitHub issues.

With this message, we bring good news and bad news just in time for our holidays. We promised 4.0 releases of PowerDNS Recursor, PowerDNS Authoritative and even a 1.0 release of dnsdist, in “December 2015”. The bad news is that we did not make it. The good news however is that we do have a set of Technology Preview releases that contain everything that 4.0 will.

In other words: the features are done, but we can’t yet sign off on the quality. However! Since most people won’t be deploying x.0 releases in December anyhow, we felt it was worthwhile to launch the 4.x series now with a strong technology preview. This preview will allow you to test our features, both to see if they work and to see if they actually fit in with your needs. And please do test, since that will speed up the advent of the actual 4.x release date!

In terms of roadmap, we consulted PowerDNS customers, community and developers, and out came a plan for 4.x. A few months into the development, various users and customers suddenly chimed in on absolutely mandatory features we had somehow missed. Because of that, 4.x both under- and overdelivers.

In addition to the huge internal cleanup, here are visible changes that did make it:


  • Fully-featured load balancer with a number of DNS-relevant load balancing policies. The default policy favours servers with the least amount of queries in flight and the fastest response times. This turns out to deliver tangible user experience improvements
  • Comes with a host of rules to block, change, or redirect traffic based on your needs. For example, use dnsdist to implement ‘views’, or what has been called ‘Advanced DNS Protection’ by some closed source resellers of open source.
  • dnscrypt, EDNS Client Subnet adding (for CG-NAT traversal, for example)
  • Realtime insights via HTTP/JSON/RESTful API & built-in live graphing website
  • For more about this new product, please see


  • GeoIP backend has gained many features, and can now run based on explicit netmasks not present in the GeoIP databases
  • Caches are now fully canonically ordered, which means entries can be wiped on suffix in all places
  • Old geobackend has been deprecated and is no longer part of PowerDNS
  • Newly revived ODBC backend for talking to Microsoft SQL Server & Azure, and with some tweaking, any other ODBC-database we do not support natively.
  • pdnssec tool does far more than DNSSEC, and has thus been renamed into ‘pdnsutil’.
  • ECDSA signing is now supported without external dependencies, and a single combined ECDSA signing key is the new default for securing zones.
  • Experimental ed25519 signing support based on draft-sury-dnskey-ed25519-03.


  • DNSSEC processing: if you ask for DNSSEC records, you will get them
  • DNSSEC validation: if so configured, PowerDNS will attempt to perform DNSSEC validation of your answers
  • Completely revamped Lua scripting API that is “DNSName” native and therefore far less error prone, and likely faster for most commonly used scenarios. Loads and indexes a 1 million domain custom policy list in a few seconds
  • New asynchronous per-domain, per-ip address, query engine. This allows PowerDNS to consult an external service in realtime to determine client or domain status. This could for example mean looking up actual customer identity from a DHCP server based on IP address (option 82 for example).
  • RPZ (from file, over AXFR or IXFR) support. This loads the largest Spamhaus zone in 5 seconds on our hardware, containing around 2 million instructions.
  • All caches can now be wiped on suffixes, because of canonical ordering
  • Many many more relevant performance metrics, including upstream authoritative performance measurements (‘is it me or the network that is slow’)
  • EDNS Client Subnet support, including cache awareness of subnet-varying answers

More technical details are available in the changelog.

Finally – the big question is of course: when will the actual 4.0.0 releases (and 1.0 for dnsdist) happen. The answer is that all this depends on what you find out during testing. We may be closer or further from the goal. As of now we can’t tell. We will report back to you in January to let you know when we expect to be able to do a release that meets our standards. But the more you test, the sooner this will be!

You can download tarballs:

Packages for several distributions are available from our repositories.

Once again, thank you everyone for working with us on this release. Happy holidays and a splendid new year!

The PowerDNS development & automation team:  Peter, PieterRemi (and Bert, who spent this release week on a sunny island, and not helping much!).

PowerDNS Recursor 3.7.1 Released

Released February 12th, 2015.

Download page.

This version contains a mix of speedups and improvements, the combined effect of which is vastly improved resilience against traffic spikes and malicious query overloads.

Of further note is the massive community contribution, mostly over Christmas. Especially Ruben Kerkhof, Pieter Lexis, Kees Monshouwer and Aki Tuomi delivered a lot of love. Thanks!

Minor changes:

  • Removal of dead code here and there 04dc6d618
  • Per-qtype response counters are now 64 bit 297bb6acf on 64 bit systems
  • Add IPv6 addresses for b and hints efc259542
  • Add IP address to logging about terminated queries 37aa9904d
  • Improve qtype name logging fab3ed345 (Aki Tuomi)
  • Redefine ‘BAD_NETS’ for dont-query based on newer IANA guidance 12cd44ee0 (lochiiconnectivity)
  • Add documentation links to systemd unit eb154adfd (Ruben Kerkhof)


  • Upgrade embedded PolarSSL to 1.3.9: d330a2ea1
  • yahttp upgrade c29097577 c65a57e88 (Aki Tuomi)
  • Replace . in hostnames by – for Carbon so as not to confuse Metronome 46541751e
  • Manpages got a lot of love and are now built from Markdown (Pieter Lexis)
  • Move to PolarSSL base64 488360551 (Kees Monshouwer)
  • The quiet=no query logging is now more informative 461df9d20
  • We can finally bind to and :: and guarantee answers from the correct source b71b60ee7
  • We use per-packet timestamps to drop ancient traffic in case of overload b71b60ee7, non-Linux portability ind63f0d836
  • Builtin webserver can be queried with the API key in the URL again c89f8cd02
  • Ringbuffers are now available via API c89f8cd02
  • Lua 5.3 compatibility 59c6fc3e3 (Kees Monshouwer)
  • No longer leave a stale UNIX domain socket around from rec_control if the recursor was down 524e4f4d8, ticket #2061
  • Running with ‘quiet=no’ would strangely actually prevent debug messages from being logged f48d7b657
  • Webserver now implements CORS for the API ea89a97e8, fixing ticket #1984
  • Houskeeping thread would sometimes run multiple times simultaneously, which worked, but was odd cc59bce67

New features:

  • New root-nx-trust flag makes PowerDNS generalize NXDOMAIN responses from the root-servers 01402d568
  • getregisteredname() for Lua, which turns ‘’ into ‘’ 8cd4851be
  • Lua preoutquery filter 3457a2a0e
  • Lua IP-based filter (ipfilter) before parsing packets 4ea949413
  • iputils class for Lua, to quickly process IP addresses and netmasks in their native format
  • getregisteredname function for Lua, to find the registered domain for a given name
  • Various new ringbuffers: top-servfail-remotes, top-largeanswer-remotes, top-servfail-queries


  • Remove unneeded malloc traffic 93d4a8909 8682c32bc a903b39cf
  • Our nameserver-loop detection carried around a lot of baggage for complex domain names, plus did not differentiate IPv4 and IPv6 well enough 891fbf888
  • Prioritize new queries over nameserver responses, improving latency under query bursts bf3b0cec3
  • Remove escaping in case there was nothing to escape 83b746fd1
  • Our logging infrastructure had a lot of locking d1449e4d0
  • Reduce logging level of certain common messages, which locked up synchronously logging systems 854d44e31
  • Add limit on total wall-clock time spent on a query 9de3e0340
  • Packet cache is now case-insensitive, which increases hitrate 90974597a

Security relevant:

  • Check for PIE, RELRO and stack protector during configure 8d0354b18 (Aki Tuomi)
  • Testing for support of PIE etc was improved in b2053c28c and beyond, fixes #2125 (Ruben Kerkhof)
  • Max query-per-query limit (max-qperq) is now configurable 173d790ea

Bugs fixed:

  • IPv6 outgoing queries had a disproportionate effect on our query load. Fixed in 76f190f2a and beyond.
  • rec_control gave incorrect output on a timeout 12997e9d8
  • When using the webserver AND having an error in the Lua script, recursor could crash during startup 62f0ae629
  • Hugely long version strings would trip up security polling 18b733382 (Kees Monshouwer)
  • The ‘remotes’ ringbuffer was sized incorrectly f8f243b01
  • Cache sizes had an off-by-one scaling problem, with the wrong number of entries allocated per thread f8f243b01
  • Our automatic file descriptor limit raising was attempted after setuid, which made it a lot less effective. Found and fixed by Aki Tuomi a6414fdce
  • Timestamps used for dropping packets were occasionaly wrong 183eb8774 and 4c4765c10 (RC2) with thanks to Winfried for debugging.
  • In RC1, our new DoS protection measures would crash the Recursor if too many root servers were unreachable.6a6fb05ad. Debugging and testing by Fusl.

Various other documentation changes by Christian Hofstaedtler and Ruben Kerkhof. Lots of improvements all over the place by Kees Monshouwer.

Diverting recursor-to-auth attacks

We get frequent reports from users/customers about various DNS-related attacks they are facing on either their authoritatives or recursors. This post focuses on one kind of attack that involves both. (Cloudmark wrote about this some time ago as well).

The attack works like this: given a target domain of, the attacker takes his botnet, and has it fire off high amounts of $ queries. These queries go from the infected hosts to their recursors (i.e. normal ISP recursors). The recursors then need to go out to the auths, after all they don’t have any random string in cache.

When this attack starts, there is no packet count amplification – bot sends query to recursor, recursor sends to auth, answer flows back, done. However, if enough of this happens, one or more of the auths for may get overloaded, and start responding more slowly, or not respond at all. The recursors will then either retry or move on to other auths, spreading the attack in the most effective and destructive way over the auths.

These attacks are painful, especially for authoritatives backed by (SQL) databases, like many PowerDNS users are running. Legitimate traffic for existing names gets cached very well inside pdns_server, but even if you put a wildcard in your database, these random queries will cause an SQL query, and those are costly.

Because SQL and random names are a bad fit, we get requests for being able to combine the bindbackend and an SQL backend in one pdns_server process. This works, but does not have the desired effect of offloading the SQL – we query both before sending out a response. So, something else needs to happen. While pondering that question this week, a few ideas came up:

  1. use IPTables u32 to match queries for the victim domain, and redirect them (I understand this can be done without generating a lot of state)
  2. teach dnsdist to pick backends based on domain name
  3. somehow get the recursors to redirect their traffic

I did not try ideas 1 and 2; I trust they will work in practice, and will effectively remove load from the SQL backends, but they still involve handling the whole malicious query load on the same server pipe. Luckily, it turns out idea 3 is feasible.

The idea behind 3 is to convince a recursor it is talking to the wrong machines, by virtue of sending it a new NSset in the AUTHORITY section of a response. Some authoritative servers will include the NSset from the zone in every response, but PowerDNS does not do this – so we need another trick.

Some time ago we added an experimental, internal-use-only feature to the Authoritative Server called lua-prequery, to be used specifically for our Recursor regression tests. While never designed for production usage, we can abuse it to make idea 3 work.

require 'posix'

function endswith(s, send)
 return #s >= #send and s:find(send, #s-#send+1, true) and true or false

function prequery ( dnspacket )
 qname, qtype = dnspacket:getQuestion()
 print(os.time(), qname,qtype)
 if endswith(qname, '') and posix.stat('/etc/powerdns/dropit')
   ret = {}
   ret[1] = {qname='', qtype=pdns.NS, content="", place=2, ttl=30}
   ret[2] = {qname='', qtype=pdns.NS, content="", place=2, ttl=30}
   return true
 return false

(A careful reader noted that the stat() call, while cached, may not be the most efficient way to enable/disable this thing. Caveat emptor.)

This piece of code, combined with a reference to it in pdns.conf (‘lua-prequery-script=/etc/powerdns/prequery.lua‘), will cause pdns_server to send authoritative NXDOMAINs for any query ending in, and include a new NSset, suggesting the recursor go look ‘over there’.

In our testing, BIND simply ignored the new NSset (we did not investigate why). PowerDNS Recursor believes what you tell it, and will stick to it until the TTL (30 seconds in this example) runs out. Unbound will also believe you, but if none of the machines you redirect it to actually work, it will come back. So, in general we recommend you point the traffic to a set of machines that can give valid replies.

In a lab setting, we found that with both Unbound and PowerDNS Recursor, this approach can move -all- traffic from your normal nameservers to the offload hosts, except for a few packets every TTL seconds. Depending on attack rate and TTL, this easily means offloading >99.9% of traffic, assuming no BIND is involved. In the real world, where some ISPs do use BIND for recursion, you won’t hit 99% or 90% but this approach may still help a lot.

We have not tried this on a real world attack, yet.

What’s next?

If you are under such an attack, and would like to give this a shot, please contact us, we’d love to try this on a real attack!

If you feel like toying around with this (I really want to find out how to make BIND cooperate, but I ran out of time), please get in touch (IRC preferred), I want to talk to you 🙂