Authoritative Server 3.4.3 released

Warning: Version 3.4.3 of the PowerDNS Authoritative Server is a major upgrade if you are coming from 2.9.x. Additionally, if you are coming from any 3.x version (including 3.3.1), there is a mandatory SQL schema upgrade. Please refer to the Upgrade documentation for important information on correct and stable operation, as well as notes on performance and memory use.

Released March 2nd, 2015

Find the downloads on our download page.

Bug fixes:

  • commit ceb49ce: pdns_control: exit 1 on unknown command (Ruben Kerkhof)
  • commit 1406891: evaluate KSK ZSK pairs per algorithm (Kees Monshouwer)
  • commit 3ca050f: always set di.notified_serial in getAllDomains (Kees Monshouwer)
  • commit d9d09e1: pdns_control: don’t open socket in /tmp (Ruben Kerkhof)

New features:

  • commit 2f67952: Limit who can send us AXFR notify queries (Ruben Kerkhof)

Improvements:

  • commit d7bec64: respond REFUSED instead of NOERROR for “unknown zone” situations
  • commit ebeb9d7: Check for Lua 5.3 (Ruben Kerkhof)
  • commit d09931d: Check compiler for relro support instead of linker (Ruben Kerkhof)
  • commit c4b0d0c: Replace PacketHandler with UeberBackend where possible (Christian Hofstaedtler)
  • commit 5a85152: PacketHandler: Share UeberBackend with DNSSECKeeper (Christian Hofstaedtler)
  • commit 97bd444: fix building with GCC 5

Experimental API changes (Christian Hofstaedtler):

From NOERROR to REFUSED

Back in 2011, in the work leading up to the biggest release of the Authoritative Server so far (3.0, with DNSSEC), in an attempt to bring the rcode for a ‘dangling CNAME’ in line with BIND, by accident the rcode for ‘we have no idea about this zone at all’ was also changed to NOERROR. This mostly went unnoticed; we got the occasional question about this behaviour, and always reassured people that this new behaviour was correct. We are aware of other (minority) auths that also do this. We still hold the position that this behaviour is correct, by the way.

However, the DNS landscape is changing. More and more parties are doing their own authoritative DNS implementations, or are buying expensive load balancers on which DNS was an afterthought. Many of these products send bogus replies to any questions that are weird to them (AAAA; EDNS; uppercase names; you name it, some vendor will have broken it). Specifically, it is now common for broken auths to respond with an empty, non-authoritative (AA=0), NOERROR reply when asked for AAAA. This reply is indistinguishable from a PowerDNS auth saying ‘this zone is unknown to me’!

As a result of this, some implementers of recursive servers (notably Google Public DNS, notable not the PowerDNS Recursor) have chosen to treat this reply as NODATA instead of ‘this server is lame’. This means that if one of your PowerDNS auths loses a zone (or a whole database, or any other number of operator errors), Google Public DNS will take your ‘i dunno’ for ‘definitely not’, breaking your zone!

Given all this, while we are confident that our approach since 3.0 is valid, we have decided to change our behaviour, and from now on the PowerDNS Authoritative Server will send REFUSED replies to any questions for unknown zones. This change will also be in Authoritative Server 3.4.3, released today.

Should you currently be affected by this incompatibility between your pre-3.4.3 auth and Google Public DNS or another recursor that misunderstands these replies, then you can use send-root-referral=lean to confuse the resolver into thinking you are lame for a zone. Do note that OARC recommends against this, and indeed recommends REFUSED, which PowerDNS has now switched to.

PowerDNS development plans: 4.x DNSSEC, C++ 2011!

In this post, we’d like to share our current plans for .. PowerDNS 4.x! We shared this first with the PowerDNS-development community, and after we gathered feedback, we’re now announcing it more broadly.

The tl;dr: For the next few months we will be spring cleaning git master, and stable code and releases can be found in the auth-3.4 and rec-3.7 branches. We’ll also be moving to C++ 2011. Please read on for the whole story.

First some background. PowerDNS is a 15 year old software project, and over these 1.5 decades, we have built up some ‘technical debt’, and it is time for a spring cleaning in our code.

Meanwhile, we are broadening what our code does, to include for example smart, DNS-native, load balancing and further denial of service mitigation. And of course, the major work of bringing carrier-grade DNSSEC to the recursor.

Finally, we’ve fallen in love with C++ 2011, and we would like to start taking advantage of this now 4 year old revision of C++.

All this means some important changes. For one, where it used to be the case that our git ‘master’ was usually fit to run in production (and people actually did this), for the coming few months please consider our master branch a ‘heavy development zone’. While we’ll try to keep things working, it might break for hours or even days at a time. Even though there will be somewhat of a wild-west aspect to development, major changes will be implemented as pull requests from separate branches that can be studied by the community.

Meanwhile, PowerDNS 3.x development and maintenance will continue on separate release branches. The latest 3.x releases will remain actively supported until 4.x is more powerful, more stable, and can be compiled on Debian Stable (more about this later). Active support means more than passive maintenance, if there are pressing things that need to happen, they will happen. But the focus for new things will shift to 4.x.

(as an example of what active support means, we are currently gathering the patches for auth-3.4.3, see here)

Things we will be addressing during our spring cleaning include:

  • We treat DNS names as ASCII strings, which we escape and unescape repeatedly. DNS names are not ascii strings, and we keep finding issues related to us treating them like strings.
  • The PowerDNS Authoritative Server distributes queries to multiple backends inefficiently
  • The PowerDNS Recursor cache is both slower and less memory efficient than it could be
  • DNSSEC in the PowerDNS Recursor
  • Move our own atomic, locking and semaphore infrastructure to C++ 2011 native
  • The Lua APIs use an ascii based interface for domain names and IP addresses, and this could be faster
  • One thing we are probably not going to do is change the database format, by the way.

The somewhat bad news about the spring cleaning is that we’ll come out of it as a C++ 2011 project, which means that to compile PowerDNS, you’ll need GCC 4.8 (released in March 2013). Gcc 4.8 is not currently the default in Debian stable or RHEL/CentOS 6, but it is available.

It is the default in RHEL7 and in what will become the next Debian stable.  It also ships in Ubuntu 14. We will also be targeting clang 3.5. We have chosen C++ 2011 for a variety of reasons, many of which are described in an earlier blogpost.

NOTE: PowerDNS 4.x products WILL run on older distribution releases of course! However, on older distros, compiling with the system default compiler may not work.

To clarify, the 4.x branch will not fundamentally alter PowerDNS. This should not be compared to BIND 9 to BIND 10, for example (or even 8 to 9). Fundamentally we think the PowerDNS design is sound, it just needs a decent spring cleaning. This will come in especially handy when deploying our DNSSEC validation.

So how long will it take until 4.x is production ready? We’ll let you know once we get there, but we are hoping to finish the cleanup in several months, after which we expect further work to iron out remaining issues. In any case, 3.x will remain supported until gcc 4.8 is widely available on currently shipping distributions.

Thanks, and please again let us know your thoughts about this proposed plan. Although this is what we intend to do, we can be change our mind if there are good reasons to do so!

PowerDNS Recursor 3.7.1 Released

Released February 12th, 2015.

Download page.

This version contains a mix of speedups and improvements, the combined effect of which is vastly improved resilience against traffic spikes and malicious query overloads.

Of further note is the massive community contribution, mostly over Christmas. Especially Ruben Kerkhof, Pieter Lexis, Kees Monshouwer and Aki Tuomi delivered a lot of love. Thanks!

Minor changes:

  • Removal of dead code here and there 04dc6d618
  • Per-qtype response counters are now 64 bit 297bb6acf on 64 bit systems
  • Add IPv6 addresses for b and c.root-servers.net hints efc259542
  • Add IP address to logging about terminated queries 37aa9904d
  • Improve qtype name logging fab3ed345 (Aki Tuomi)
  • Redefine ‘BAD_NETS’ for dont-query based on newer IANA guidance 12cd44ee0 (lochiiconnectivity)
  • Add documentation links to systemd unit eb154adfd (Ruben Kerkhof)

Improvements:

  • Upgrade embedded PolarSSL to 1.3.9: d330a2ea1
  • yahttp upgrade c29097577 c65a57e88 (Aki Tuomi)
  • Replace . in hostnames by – for Carbon so as not to confuse Metronome 46541751e
  • Manpages got a lot of love and are now built from Markdown (Pieter Lexis)
  • Move to PolarSSL base64 488360551 (Kees Monshouwer)
  • The quiet=no query logging is now more informative 461df9d20
  • We can finally bind to 0.0.0.0 and :: and guarantee answers from the correct source b71b60ee7
  • We use per-packet timestamps to drop ancient traffic in case of overload b71b60ee7, non-Linux portability ind63f0d836
  • Builtin webserver can be queried with the API key in the URL again c89f8cd02
  • Ringbuffers are now available via API c89f8cd02
  • Lua 5.3 compatibility 59c6fc3e3 (Kees Monshouwer)
  • No longer leave a stale UNIX domain socket around from rec_control if the recursor was down 524e4f4d8, ticket #2061
  • Running with ‘quiet=no’ would strangely actually prevent debug messages from being logged f48d7b657
  • Webserver now implements CORS for the API ea89a97e8, fixing ticket #1984
  • Houskeeping thread would sometimes run multiple times simultaneously, which worked, but was odd cc59bce67

New features:

  • New root-nx-trust flag makes PowerDNS generalize NXDOMAIN responses from the root-servers 01402d568
  • getregisteredname() for Lua, which turns ‘www.bbc.co.uk’ into ‘bbc.co.uk’ 8cd4851be
  • Lua preoutquery filter 3457a2a0e
  • Lua IP-based filter (ipfilter) before parsing packets 4ea949413
  • iputils class for Lua, to quickly process IP addresses and netmasks in their native format
  • getregisteredname function for Lua, to find the registered domain for a given name
  • Various new ringbuffers: top-servfail-remotes, top-largeanswer-remotes, top-servfail-queries

Speedups:

  • Remove unneeded malloc traffic 93d4a8909 8682c32bc a903b39cf
  • Our nameserver-loop detection carried around a lot of baggage for complex domain names, plus did not differentiate IPv4 and IPv6 well enough 891fbf888
  • Prioritize new queries over nameserver responses, improving latency under query bursts bf3b0cec3
  • Remove escaping in case there was nothing to escape 83b746fd1
  • Our logging infrastructure had a lot of locking d1449e4d0
  • Reduce logging level of certain common messages, which locked up synchronously logging systems 854d44e31
  • Add limit on total wall-clock time spent on a query 9de3e0340
  • Packet cache is now case-insensitive, which increases hitrate 90974597a

Security relevant:

  • Check for PIE, RELRO and stack protector during configure 8d0354b18 (Aki Tuomi)
  • Testing for support of PIE etc was improved in b2053c28c and beyond, fixes #2125 (Ruben Kerkhof)
  • Max query-per-query limit (max-qperq) is now configurable 173d790ea

Bugs fixed:

  • IPv6 outgoing queries had a disproportionate effect on our query load. Fixed in 76f190f2a and beyond.
  • rec_control gave incorrect output on a timeout 12997e9d8
  • When using the webserver AND having an error in the Lua script, recursor could crash during startup 62f0ae629
  • Hugely long version strings would trip up security polling 18b733382 (Kees Monshouwer)
  • The ‘remotes’ ringbuffer was sized incorrectly f8f243b01
  • Cache sizes had an off-by-one scaling problem, with the wrong number of entries allocated per thread f8f243b01
  • Our automatic file descriptor limit raising was attempted after setuid, which made it a lot less effective. Found and fixed by Aki Tuomi a6414fdce
  • Timestamps used for dropping packets were occasionaly wrong 183eb8774 and 4c4765c10 (RC2) with thanks to Winfried for debugging.
  • In RC1, our new DoS protection measures would crash the Recursor if too many root servers were unreachable.6a6fb05ad. Debugging and testing by Fusl.

Various other documentation changes by Christian Hofstaedtler and Ruben Kerkhof. Lots of improvements all over the place by Kees Monshouwer.

New PowerDNS employee, the importance of testing RCs, skipping 3.7.0, World Hosting Days 2015

Hi everybody,

Some assorted remarks & PowerDNS news:

1) New employee
2) Please test our release candidates
3) 3.7.0 has been skipped, all hail 3.7.1
4) World Hosting days in Germany

New employee
To start with, the great news is that on March 2nd, Pieter Lexis will be joining PowerDNS as a fulltime employee!

Pieter wrote a paper and software on DANE under our mentorship while at the OS3 program at the University of Amsterdam, and later did an amazing job converting our documentation to the splendor you can now find on http://doc.powerdns.com/ Based on this work, we offered Pieter a job and we’re very happy he accepted!

Pieter (not to be confused with existing employee Peter) will focus on helping customers, improving our code & infrastructure, fixing bugs and working on internet standards relevant for DNS.

Release candidates
When we work on a PowerDNS release, once we feel that it is ready to be used, we issue a Release Candidate. This is something you can run in  production, and we expect it to work fine. If you have issues with an RC, we’ll jump on them and resolve them as quickly as is possible. In the 3.7.0 release process this worked well, and because RC1 and RC2 saw wide deployment, many small issues were found before the actual release. 3.7.0 was looking good, and we tagged it for release.

And then PowerDNS user & packager Ralf van der Enden reported that the 3.7.0 we uploaded did exactly nothing on his FreeBSD system. After intense debugging to see if we could save 3.7.0, we found that we indeed had a bug which meant 3.7.0 compiled on FreeBSD, but did nothing. This was fixed.

Today, we are increasing our regression tests to run on FreeBSD as well to prevent a repeat of this problem.

But we’d like to urge our users, especially the ones on less mainstream platforms than Debian, Ubuntu, Fedora and Red Hat, to test our release  candidates. This is one of the best ways you, like Ralf did, can help us deliver quality products!

3.7.0 will be skipped
Because we had uploaded 3.7.0 and had it built for our various platforms, we are not going to slip the FreeBSD fix into 3.7.0 and end up with two different 3.7.0 releases. The next PowerDNS Recursor release will be 3.7.1. This release is imminent, after we complete our FreeBSD regression testing.

World Hosting Days 2015 in Rust
PowerDNS and several of our Certified Consultants will be at World Hosting Days 2015 in Rust, Germany (March 24-26). As always, we enjoy meeting with PowerDNS users. If you or your management will be there and want to talk,  please let us know!

Authoritative Server 3.4.2 Released

Warning: Version 3.4.2 of the PowerDNS Authoritative Server is a major upgrade if you are coming from 2.9.x. Additionally, if you are coming from any 3.x version (including 3.3.1), there is a mandatory SQL schema upgrade. Please refer to the Upgrade documentation for important information on correct and stable operation, as well as notes on performance and memory use.

Released February 3rd, 2015

Find the downloads on our download page.

This is a performance and bugfix update to 3.4.1 and any earlier version. For high traffic setups, including those using DNSSEC, upgrading to 3.4.2 may show tremendous performance increases. Please let us know!

We would like to thank Patrik Wallström of IIS, Kees Monshouwer and Fredrik Eriksson of Loopia for working with us on solving several issues that only became apparent on a 750000 domain (!) DNSSEC installation, the last of which we could eventually trace to memory fragmentation in the secure allocator of our cryptography library. This bug chase, which lasted for over a month, led to numerous other improvements, like better statistical metrics for plotting (actual CPU usage, uptime, key cache size, signatures/s) and the ‘sharding’ of our internal caches to better support multi-CPU operations.

A list of changes since 3.4.1 follows:

Improvements:

Bug fixes:

  • commit 60b2b7c, commit d962fbc: refuse overly long labels in names
  • commit a64fd6a: auth: limit long version strings to 63 characters and catch exceptions in secpoll
  • commit fa52e02: pdnssec: fix ttl check for RRSIG records
  • commit 0678b25: fix up latency reporting for sub-millisecond latencies (would clip to 0)
  • commit d45c1f1: make sure we don’t throw an exception on “pdns_control show” of an unknown variable
  • commit 63c8088: fix startup race condition with carbon thread already trying to broadcast uninitialized data
  • commit 796321c: make qsize-q more robust
  • commit 407867c: Kees Monshouwer discovered we count corrupt packets and EAGAIN situations as validly received packets, skewing the udp questions/answers graphs on auth.
  • commit f06d069: make latency & qsize reporting ‘live’. Plus fix that we only reported the qsize of the first distributor.
  • commit 2f3498e: fix up statbag for carbon protocol and function pointers
  • commit 0f2f999: get priority from table in Lua axfrfilter; fixes ticket #1857
  • commit 96963e2, commit bbcbbbe, commit d5c9c07: various backends: fix records pointing at root
  • commit e94c2c4: remove additional layer of trailing . stripping, which broke MX records to the root in the BIND backend. Should close ticket #1243.
  • commit 8f35ba2: api: use uncached results for getKeys()
  • commit c574336: read ALLOW-AXFR-FROM from the backend with the metadata

Minor changes:

New features:

  • commit 1b97ba0: add signatures metric to auth, so we can plot signatures/second
  • commit 92cef2d: pdns_control: make it posible to notify all zones at once
  • commit f648752: JSON API: provide flush-cache, notify, axfr-receive
  • commit 02653a7: add ‘bench-db’ to do very simple database backend performance benchmark
  • commit a83257a: enable callback based metrics to statbas, and add 5 such metrics: uptime, sys-msec, user-msec, key-cache-size, meta-cache-size, signature-cache-size

Performance improvements:

An introduction to PowerDNS Recursor 3.7.0

Today we released the first Release Candidate (RC) of PowerDNS Recursor 3.7.0. And although we do document (almost) all of our features, documentation is not the same as effectively informing our users of what our software can do.

In this blog, I’ll be highlighting some of the things that are new in 3.7.0, or which arrived recently in the 3.6 series.

Binding to 0.0.0.0 and ::

For a long time, PowerDNS Recursor did not support binding to the ‘any’ addresses, because Unix & POSIX semantics mean that a socket bound to such an any address has no control over which IP address it answers from. So for example, if your server is on 192.168.1.2 and 192.168.1.3, and question comes in on a socket bound to 0.0.0.0, the answer might go out to either the .2 or the .3 address – no matter what address the question arrived on!

Binding to 0.0.0.0 (and ::, of course) is very useful for load balancing purposes, so as of 3.7.0, we use some specific calls to implement ‘sendfromto()’ so we can force the source address to be correct. Just bind to 0.0.0.0 or :: and the Recursor will do the right thing.

Live graphs

The PowerDNS Recursor has supported our HTTP-based RESTful JSON API for a while now, although we have not publicized this sufficiently while we stabilized the API. In 3.7.0, this API was enhanced to enable it to power a rather impressive (even if we say so ourselves) live display of traffic, updating every second. It looks like this:

For now, development of this live website is proceeding faster than the PowerDNS release schedule, which means you can find the HTML for this page (which is independent of PowerDNS itself) on GitHub, where you can also find instructions on how to enable this live display in the stock version of Recursor 3.7.0.

Graphing as a service

We’ve explained this at length in a previous post, but with our built-in Carbon/Graphite support, it is a breeze to get pretty graphs of your own recursor, or, if you have problems to send your statistics our way. By having a live view on your system, we can quickly diagnose issues. Recommended!

Enhanced Lua scripting

For the last few months, the world of DNS has been about one thing only: attacks. Amplification attacks, reflection attacks and ‘infinite recursion queries’. Our vision is that PowerDNS should be as robust as possible by generally being fast and smart, and therefore be resilient against attacks. Sadly, some attacks can be rather nasty, and can’t be fended off with generic solutions.

PowerDNS has extensive support for Lua scripting, and in 3.7.0 we added a number of features intended to serve as the basis of “scripts against the attack of the day”.

Next to the existing hooks (preresolve, which intercepts a packet after parsing, postresolve which can edit a packet after the answer is in, and nxdomain and noerror which are called for those two results), we’ve added two new ones: ipfilter and preoutquery.

‘ipfilter’ hooks in before client packet parsing even starts, and allows you to block a packet before it has used a lot of CPU cycles. This is a great place to (dynamically) stop attacks.

‘preoutquery’ is a new kind of hook. It does not fire based on user (client) traffic, but before a remote nameserver is consulted. Frequently, attack traffic is for loads of random domain names, but in the end only a few (fixed) IP addresses are the target of the attack. If you know these addresses, you can trigger on them in ‘preoutquery’, and drop the query quickly before it used more CPU and network. But even better, once you know the client IP address that is performing the attack on you, you can block it with ipfilter!

Here is a sample script that does just that:

lethalgroup=iputils.newnmgroup()
lethalgroup:add("192.121.121.0/24") -- touch these nameservers and you die

blockset=iputils.newipset() -- which client IP addresses we block

function preoutquery(remoteip, domain, qtype)
--  print("pdns wants to ask "..remoteip:tostring().." about "..domain.." "..qtype.." on behalf of requestor "..getlocaladdress())
    if(lethalgroup:match(remoteip))
    then
--      print("We matched the group "..lethalgroup:tostring().."!", "killing query dead & adding requestor "..getlocaladdress().." to block list")
        blockset[iputils.newca(getlocaladdress())]=1
        return -3,{} --   -3 means 'kill'
    end
    return -1,{}         --   -1 means 'no opinion'
end


local delcount=0

function ipfilter(remoteip)
    delcount=delcount+1

    if((delcount % 10000)==0)
    then
--      print("Clearing blockset!")
        blockset=iputils.newipset()  -- clear it
    end

    if(blockset[remoteip] ~= nil) then
        return 1         -- block!
    end
    return -1                -- no opinion
end

This script blocks an attacker for the next 10000 queries if they ever ‘touched’ a namserver known to be involved in malicious traffic.

The ‘ipset’ used above is fast enough that a 10 million long list of blocked IP addresses doesn’t noticeably affect performance!

Improved ringbuffers

To investigate traffic, ringbuffers of the last x-thousand (by default 10000) queries or errors are available. Some sample output:

# rec_control top-queries
Over last 10000 entries:
4.04% e3191.dscc.akamaiedge.net.|A
4.04% e3191.dscc.akamaiedge.net.|AAAA
1.50% pool.ntp.org.|A
1.44% us-courier.push-apple.com.akadns.net.|A
1.25% twitter.com.|A
1.24% twitter.com.|AAAA
0.42% www.google.com.|A
(...)
77.40% rest

When under attack, typically a lot of random traffic comes in which makes it harder to see what is going on. For this purpose, we’ve added the ‘pub’ filters, which group queries via the Mozilla Public Suffix list:

# rec_control top-pub-queries
Over last 10000 entries:
5.26% google.com.|A
5.21% google.com.|AAAA
4.38% akamaiedge.net.|AAAA
4.08% akamaiedge.net.|A
2.35% gstatic.com.|A

Other interesting rings:

  • top-servfail-queries, listing the top queries leading to servfail responses. (top-pub-servfail-queries grouped)
  • top-largeanswer-remotes, listing top clients receiving large answers (as typically used in reflection attacks)
  • top-remotes, listing clients causing most queries
  • top-servfail-remotes, listing clients causing most servfail responses

Finally

We hope you enjoy these new features, and we’d love to hear your thoughts on if they do the job, or how they could be better!