PowerDNS and Open-Xchange agree to merge

team

(Peter van Dijk, Bert Hubert, Rafael Laguna, Mikko Linnamäki, Markku Kenttä, Timo Sirainen)

Hi everybody,

We’re currently at World Hosting Days in Rust Germany, where we just announced that PowerDNS will be joining the Open-Xchange family of companies.  Last week it was also announced that the famous Dovecot IMAP server project is now a part of OX too.

Update: The Register and The WHIR covered the news!

We’ve been working with Timo and his team at Dovecot and with the OX Team in Email Security projects and are already sharing personnel and infrastructure with each other and the cooperation works really well for all of us.

From the Open-Xchange website: “With over a decade of developing open-source software, Open-Xchange believes that only by engineering ruthlessly open products and services can the next generation of innovation emerge on the web. “Stay Open” contains many aspects of how we develop, engineer and deploy our products together with and for client-partners.”

We fully believe in that mission, and are glad that PowerDNS will become part of the Open-Xchange family. It will be great to have Timo and friends from Dovecot as cousins!

We’ll share more details of what the merger will and will not mean, but rest assured PowerDNS will stay as open and as community friendly as it has ever been.

Meanwhile, if you are at WHD, please come meet us at the Open-Xchange booth!

Bert, Peter and Pieter

World Hosting Days & Private Graphs as a Service!

Hi everybody,

Two announcements in one: First, like 7000 others, we’ll be visiting World Hosting Days in Rust, Germany next week. Peter, Pieter and I will be there, as will be two of our wonderful Certified Consultants (Kees Monshouwer and Christian Hofstaedtler).

If you want to meet up, please email any of us (or powerdns.ideas@powerdns.com) and we can coordinate. The PowerDNS team will in any case be available for drinks! We always like to  hear from users since you have more experience running PowerDNS than we do, and can help us guide new features.

Secondly, last year we made our ‘public graphing as a service’ available, as described on
http://blog.powerdns.com/2014/12/11/powerdns-graphing-as-a-service/

Today, we’re happy to announce that we now also have a private variant for supported customers and selected users. This means you can benefit from a one-line setup in PowerDNS (simply set the ‘carbon-server’ variable and you are done), and view all your PowerDNS instances from one single interface, and in private.

If you’d like to use our private graphing service, please contact us for details.

The public graphing instance is now receiving over 1 gigabyte of graphs every week, so we think we are fulfilling a need!

Cheers,

Bert

Introducing dnsdist: DNS, abuse- and DoS-aware query distribution for optimal performance

Over the years, PowerDNS users have frequently asked us about our preferred DNS load balancing solution, and we’ve never had a satisfying answer for that. Users of dedicated hardware often tell us that vendors spend most of their time and effort on balancing HTTP, and frequently deliver substandard or even buggy DNS functionality.

In terms of software, one big PowerDNS deployment is happy with OpenBSD relayd, and it indeed does look powerful. Other operators have deployed keepalived, which is very strong from a networking standpoint, but does not offer a lot of DNS specifics.

But all in all, we never found any load balancing solution for DNS that made people truly happy.

Simultaneously, there is now a lot of focus on providing the very best DNS performance, even in the face of the ongoing reflection attacks (one of which has been dubbed the “Chinese water torture attack“).

Putting these three things together (no really satisfying DNS aware load balancer, drive for the very best performance, ongoing attacks) led us to pollute the waters of the internet with yet another piece of software: dnsdist.

From its README:

“dnsdist is a highly DNS-, DoS- and abuse-aware loadbalancer. Its goal in life is to route traffic to the best server, delivering top performance to legitimate users while shunting or blocking abusive traffic.”

This is quite a mission statement, but we’ve tried to keep things simple. The simplest possible invocation:

# dnsdist --local 0.0.0.0:53 192.168.1.2 192.168.1.3 192.168.1.4

Delivers sensible and smart distribution of queries over three downstream servers (.2, .3 and .4). All three are polled for availability, and traffic is delivered to the server with the lowest amount of outstanding queries. If this is all you need, you should be done. However, much more power is available.

For example, the following configuration does something quite special:

newServer{address="192.168.1.2", qps=10000, order=1}
newServer{address="127.0.0.1:5300", qps=5000, order=2}
newServer{address="192.168.1.79:5300", qps=7000, order=3}
setServerPolicy(firstAvailable)

This configuration again defines three downstream servers, but it also configures the preferred QPS limit of each. Finally, it changes the default server selection policy to ‘firstAvailable’. In production, as viewed from the dnsdist console, this looks like this:

qps

(This console is available either locally or remotely.)

It has frequently been observed that a busy nameserver is a happy nameserver, since this means that all caches are hot and that more cache hits are generated per TTL. The configuration above will send traffic to the first server that has not hit its configured QPS limit. If all have exceeded their limit, it decays to the standard ‘leastOutstanding’ policy.

Built in, two further load balancing policies are available:

  • wrandom: weighted random, where traffic is distributed by the weight parameter
  • roundrobin: as the name implies

In the configuration, custom policies can be written in Lua (see below).

Traffic inspection

Now, if all traffic were bona fide, and everything worked as planned, we’d be done with the feature set above. In the real world however, we frequently suffer degraded performance from malicious or clearly illegitimate traffic. Dnsdist offers quite some features to inspect live traffic:
topqueries

This shows us the ‘top 5′ of incoming queries, showing a rather ‘flat’ profile for this server. Nothing really jumps out. Potentially however, this is because unwanted traffic consists of many unique names, so let’s see:

topqueries-2
This shows traffic grouped by the last 2 components of query names, and now ‘t-ipnet.de.’ jumps out. Let’s say we determine these queries as unwanted (which would be HIGHLY unlikely in this case), dnsdist offers three options:

  1. Block outright: addDomainBlock("t-ipnet.de.")
  2. Rate limit to (say) 10/s: addQPSLimit("t-ipnet.de.", 10)
  3. Shunt to a dedicated abuse pool:
    • newServer{address="192.168.1.30:5300", pool="abuse")
    • newPoolRule("t-ipnet.de.", "abuse")

All three options have their merits, but we’re especially excited about this last option. Operators usually face a stark choice with traffic – block it, and potentially upset lots of bona fide users, or allow and degrade performance for everyone. By dedicating a bunch of servers to dubious traffic, it is possible to isolate worrying traffic, and only have it impact the users of the impacted domain names.

Further analysis is possible based on response codes. This for example shows the top-5 servfail responses:

topresponses

Analogous inspection based on IP addresses is possible via:

topclients

And the three options outlined above for blocking, limiting and shunting traffic also accept IP addresses as parameters.

Full flexibility

While dnsdist natively comes with a broad set of primitives to route, shape and block traffic, ultimate flexibility is possible with the built-in Lua support.

For example, to block all domain names with 5 consecutive numbers in them, try:

function blockFilter(remote, qname, qtype, dh)
return string.match(qname:tostring(), "%d%d%d%d%d") ~= nil
end

Similarly, blockFilter can implement for example returning TC=1 answers for all ANY queries, or if you are so inclined, rate limit all ANY queries. A example of this can be found in the sample configuration file.

Server load balancing policies can also be written entirely in Lua, and direct traffic to specific servers, or to a specific built-in load balancing policy, but based from a different pool. This last service is great for providing split-horizon service, for example.

Practical details

dnsdist is a daemon that can partially be configured from the commandline, but otherwise has a configuration file. It can be queried at runtime via an optionally encrypted TCP/IP connection. At this console, which is readline() based and offers history search etc, every setting can be displayed and changed.

The Lua parser is the console commandline and configuration parser. In practice, most configurations can run without involving Lua for every packet. However, even in very dynamic setups where this is required, the practical overhead is minimal, especially when compiled against luajit.

Note that dnsdist is not PowerDNS specific, and will happily balance traffic between nameserver implementations or even third party services!

What dnsdist is not

dnsdist tries to be high performance, but is fundamentally limited by the fact that it is a general purpose handler of DNS packets. Practically speaking, this means that dnsdist can’t be used to scale to many hundreds of thousands of forwarded packets per second, as this tends to stress out commodity servers and operating systems. However, care has been taken to allow dnsdist to itself be hooked up to network-based load balancing solutions, for example by allowing non-local binds, and proper behavior on binding to 0.0.0.0.

Similarly, dnsdist is not an alternative to many gigabit capable DoS scrubbing devices or services.

It is however a great way to deal with “the last gigabit” of an attack, or simply smaller scale attacks that algorithmically kill your performance, instead of simply flooding your pipe. And by distributing queries wisely, your users experience better performance.

Some further cool things

  • showResponseLatency(): prints a histogram of response times
  • showServers(): show statistics for all configured downstreams
  • getServer(0):setDown(): force server 0 down administratively
  • showPoolRules(): show configured pool rules
  • showQPSLimiters(): show configured QPS limiters
  • setServerPolicy(wrandom): set weighted random server selection policy
  • setServerPolicyLua(): configure Lua-based server policies
  • .. more in the README

Current status & how to get it

dnsdist is still very fresh software. However, we are actively seeking feedback, but not quite yet production users. Please let us know your thoughts on powerdns.ideas@powerdns.com!

Snapshot releases can be found on https://xs.powerdns.com/dnsdist/. Compiling dnsdist requires a comparatively recent compiler (gcc 4.8, clang 3.5) and Lua. More details can be found in the README.

Actual production releases should happen shortly.

Authoritative Server 3.4.3 released

Warning: Version 3.4.3 of the PowerDNS Authoritative Server is a major upgrade if you are coming from 2.9.x. Additionally, if you are coming from any 3.x version (including 3.3.1), there is a mandatory SQL schema upgrade. Please refer to the Upgrade documentation for important information on correct and stable operation, as well as notes on performance and memory use.

Released March 2nd, 2015

Find the downloads on our download page.

Bug fixes:

  • commit ceb49ce: pdns_control: exit 1 on unknown command (Ruben Kerkhof)
  • commit 1406891: evaluate KSK ZSK pairs per algorithm (Kees Monshouwer)
  • commit 3ca050f: always set di.notified_serial in getAllDomains (Kees Monshouwer)
  • commit d9d09e1: pdns_control: don’t open socket in /tmp (Ruben Kerkhof)

New features:

  • commit 2f67952: Limit who can send us AXFR notify queries (Ruben Kerkhof)

Improvements:

  • commit d7bec64: respond REFUSED instead of NOERROR for “unknown zone” situations
  • commit ebeb9d7: Check for Lua 5.3 (Ruben Kerkhof)
  • commit d09931d: Check compiler for relro support instead of linker (Ruben Kerkhof)
  • commit c4b0d0c: Replace PacketHandler with UeberBackend where possible (Christian Hofstaedtler)
  • commit 5a85152: PacketHandler: Share UeberBackend with DNSSECKeeper (Christian Hofstaedtler)
  • commit 97bd444: fix building with GCC 5

Experimental API changes (Christian Hofstaedtler):

From NOERROR to REFUSED

Back in 2011, in the work leading up to the biggest release of the Authoritative Server so far (3.0, with DNSSEC), in an attempt to bring the rcode for a ‘dangling CNAME’ in line with BIND, by accident the rcode for ‘we have no idea about this zone at all’ was also changed to NOERROR. This mostly went unnoticed; we got the occasional question about this behaviour, and always reassured people that this new behaviour was correct. We are aware of other (minority) auths that also do this. We still hold the position that this behaviour is correct, by the way.

However, the DNS landscape is changing. More and more parties are doing their own authoritative DNS implementations, or are buying expensive load balancers on which DNS was an afterthought. Many of these products send bogus replies to any questions that are weird to them (AAAA; EDNS; uppercase names; you name it, some vendor will have broken it). Specifically, it is now common for broken auths to respond with an empty, non-authoritative (AA=0), NOERROR reply when asked for AAAA. This reply is indistinguishable from a PowerDNS auth saying ‘this zone is unknown to me’!

As a result of this, some implementers of recursive servers (notably Google Public DNS, notable not the PowerDNS Recursor) have chosen to treat this reply as NODATA instead of ‘this server is lame’. This means that if one of your PowerDNS auths loses a zone (or a whole database, or any other number of operator errors), Google Public DNS will take your ‘i dunno’ for ‘definitely not’, breaking your zone!

Given all this, while we are confident that our approach since 3.0 is valid, we have decided to change our behaviour, and from now on the PowerDNS Authoritative Server will send REFUSED replies to any questions for unknown zones. This change will also be in Authoritative Server 3.4.3, released today.

Should you currently be affected by this incompatibility between your pre-3.4.3 auth and Google Public DNS or another recursor that misunderstands these replies, then you can use send-root-referral=lean to confuse the resolver into thinking you are lame for a zone. Do note that OARC recommends against this, and indeed recommends REFUSED, which PowerDNS has now switched to.

PowerDNS development plans: 4.x DNSSEC, C++ 2011!

In this post, we’d like to share our current plans for .. PowerDNS 4.x! We shared this first with the PowerDNS-development community, and after we gathered feedback, we’re now announcing it more broadly.

The tl;dr: For the next few months we will be spring cleaning git master, and stable code and releases can be found in the auth-3.4 and rec-3.7 branches. We’ll also be moving to C++ 2011. Please read on for the whole story.

First some background. PowerDNS is a 15 year old software project, and over these 1.5 decades, we have built up some ‘technical debt’, and it is time for a spring cleaning in our code.

Meanwhile, we are broadening what our code does, to include for example smart, DNS-native, load balancing and further denial of service mitigation. And of course, the major work of bringing carrier-grade DNSSEC to the recursor.

Finally, we’ve fallen in love with C++ 2011, and we would like to start taking advantage of this now 4 year old revision of C++.

All this means some important changes. For one, where it used to be the case that our git ‘master’ was usually fit to run in production (and people actually did this), for the coming few months please consider our master branch a ‘heavy development zone’. While we’ll try to keep things working, it might break for hours or even days at a time. Even though there will be somewhat of a wild-west aspect to development, major changes will be implemented as pull requests from separate branches that can be studied by the community.

Meanwhile, PowerDNS 3.x development and maintenance will continue on separate release branches. The latest 3.x releases will remain actively supported until 4.x is more powerful, more stable, and can be compiled on Debian Stable (more about this later). Active support means more than passive maintenance, if there are pressing things that need to happen, they will happen. But the focus for new things will shift to 4.x.

(as an example of what active support means, we are currently gathering the patches for auth-3.4.3, see here)

Things we will be addressing during our spring cleaning include:

  • We treat DNS names as ASCII strings, which we escape and unescape repeatedly. DNS names are not ascii strings, and we keep finding issues related to us treating them like strings.
  • The PowerDNS Authoritative Server distributes queries to multiple backends inefficiently
  • The PowerDNS Recursor cache is both slower and less memory efficient than it could be
  • DNSSEC in the PowerDNS Recursor
  • Move our own atomic, locking and semaphore infrastructure to C++ 2011 native
  • The Lua APIs use an ascii based interface for domain names and IP addresses, and this could be faster
  • One thing we are probably not going to do is change the database format, by the way.

The somewhat bad news about the spring cleaning is that we’ll come out of it as a C++ 2011 project, which means that to compile PowerDNS, you’ll need GCC 4.8 (released in March 2013). Gcc 4.8 is not currently the default in Debian stable or RHEL/CentOS 6, but it is available.

It is the default in RHEL7 and in what will become the next Debian stable.  It also ships in Ubuntu 14. We will also be targeting clang 3.5. We have chosen C++ 2011 for a variety of reasons, many of which are described in an earlier blogpost.

NOTE: PowerDNS 4.x products WILL run on older distribution releases of course! However, on older distros, compiling with the system default compiler may not work.

To clarify, the 4.x branch will not fundamentally alter PowerDNS. This should not be compared to BIND 9 to BIND 10, for example (or even 8 to 9). Fundamentally we think the PowerDNS design is sound, it just needs a decent spring cleaning. This will come in especially handy when deploying our DNSSEC validation.

So how long will it take until 4.x is production ready? We’ll let you know once we get there, but we are hoping to finish the cleanup in several months, after which we expect further work to iron out remaining issues. In any case, 3.x will remain supported until gcc 4.8 is widely available on currently shipping distributions.

Thanks, and please again let us know your thoughts about this proposed plan. Although this is what we intend to do, we can be change our mind if there are good reasons to do so!

PowerDNS Recursor 3.7.1 Released

Released February 12th, 2015.

Download page.

This version contains a mix of speedups and improvements, the combined effect of which is vastly improved resilience against traffic spikes and malicious query overloads.

Of further note is the massive community contribution, mostly over Christmas. Especially Ruben Kerkhof, Pieter Lexis, Kees Monshouwer and Aki Tuomi delivered a lot of love. Thanks!

Minor changes:

  • Removal of dead code here and there 04dc6d618
  • Per-qtype response counters are now 64 bit 297bb6acf on 64 bit systems
  • Add IPv6 addresses for b and c.root-servers.net hints efc259542
  • Add IP address to logging about terminated queries 37aa9904d
  • Improve qtype name logging fab3ed345 (Aki Tuomi)
  • Redefine ‘BAD_NETS’ for dont-query based on newer IANA guidance 12cd44ee0 (lochiiconnectivity)
  • Add documentation links to systemd unit eb154adfd (Ruben Kerkhof)

Improvements:

  • Upgrade embedded PolarSSL to 1.3.9: d330a2ea1
  • yahttp upgrade c29097577 c65a57e88 (Aki Tuomi)
  • Replace . in hostnames by – for Carbon so as not to confuse Metronome 46541751e
  • Manpages got a lot of love and are now built from Markdown (Pieter Lexis)
  • Move to PolarSSL base64 488360551 (Kees Monshouwer)
  • The quiet=no query logging is now more informative 461df9d20
  • We can finally bind to 0.0.0.0 and :: and guarantee answers from the correct source b71b60ee7
  • We use per-packet timestamps to drop ancient traffic in case of overload b71b60ee7, non-Linux portability ind63f0d836
  • Builtin webserver can be queried with the API key in the URL again c89f8cd02
  • Ringbuffers are now available via API c89f8cd02
  • Lua 5.3 compatibility 59c6fc3e3 (Kees Monshouwer)
  • No longer leave a stale UNIX domain socket around from rec_control if the recursor was down 524e4f4d8, ticket #2061
  • Running with ‘quiet=no’ would strangely actually prevent debug messages from being logged f48d7b657
  • Webserver now implements CORS for the API ea89a97e8, fixing ticket #1984
  • Houskeeping thread would sometimes run multiple times simultaneously, which worked, but was odd cc59bce67

New features:

  • New root-nx-trust flag makes PowerDNS generalize NXDOMAIN responses from the root-servers 01402d568
  • getregisteredname() for Lua, which turns ‘www.bbc.co.uk’ into ‘bbc.co.uk’ 8cd4851be
  • Lua preoutquery filter 3457a2a0e
  • Lua IP-based filter (ipfilter) before parsing packets 4ea949413
  • iputils class for Lua, to quickly process IP addresses and netmasks in their native format
  • getregisteredname function for Lua, to find the registered domain for a given name
  • Various new ringbuffers: top-servfail-remotes, top-largeanswer-remotes, top-servfail-queries

Speedups:

  • Remove unneeded malloc traffic 93d4a8909 8682c32bc a903b39cf
  • Our nameserver-loop detection carried around a lot of baggage for complex domain names, plus did not differentiate IPv4 and IPv6 well enough 891fbf888
  • Prioritize new queries over nameserver responses, improving latency under query bursts bf3b0cec3
  • Remove escaping in case there was nothing to escape 83b746fd1
  • Our logging infrastructure had a lot of locking d1449e4d0
  • Reduce logging level of certain common messages, which locked up synchronously logging systems 854d44e31
  • Add limit on total wall-clock time spent on a query 9de3e0340
  • Packet cache is now case-insensitive, which increases hitrate 90974597a

Security relevant:

  • Check for PIE, RELRO and stack protector during configure 8d0354b18 (Aki Tuomi)
  • Testing for support of PIE etc was improved in b2053c28c and beyond, fixes #2125 (Ruben Kerkhof)
  • Max query-per-query limit (max-qperq) is now configurable 173d790ea

Bugs fixed:

  • IPv6 outgoing queries had a disproportionate effect on our query load. Fixed in 76f190f2a and beyond.
  • rec_control gave incorrect output on a timeout 12997e9d8
  • When using the webserver AND having an error in the Lua script, recursor could crash during startup 62f0ae629
  • Hugely long version strings would trip up security polling 18b733382 (Kees Monshouwer)
  • The ‘remotes’ ringbuffer was sized incorrectly f8f243b01
  • Cache sizes had an off-by-one scaling problem, with the wrong number of entries allocated per thread f8f243b01
  • Our automatic file descriptor limit raising was attempted after setuid, which made it a lot less effective. Found and fixed by Aki Tuomi a6414fdce
  • Timestamps used for dropping packets were occasionaly wrong 183eb8774 and 4c4765c10 (RC2) with thanks to Winfried for debugging.
  • In RC1, our new DoS protection measures would crash the Recursor if too many root servers were unreachable.6a6fb05ad. Debugging and testing by Fusl.

Various other documentation changes by Christian Hofstaedtler and Ruben Kerkhof. Lots of improvements all over the place by Kees Monshouwer.