Hi everybody,
In this post we’d like to update you on what has been achieved in the development of the PowerDNS 4.x products: Authoritative, Recursor and dnsdist. First, I am very proud that we managed to do a “spring cleaning”. As mentioned earlier, over time any software project picks up so called “technical debt“. Things that looked good at the time, shortcuts to quickly get to a new feature, whole features that weren’t fully thought through: it all piles up.
It is very wise for any piece of software to periodically take a step back and do a cleanup, but it rarely happens. Some projects do go for the “grand redesign”, but this frequently does not lead to a product that is actually better in production (“Second system effect“).
We’re grateful for our customers and users that allowed us the time for the cleanup, and happy that the PowerDNS community itself too had the discipline to work on these invisible improvements. It is always more fun to add new features than to break down things that were working to make them better!
So what happened under the hood? Quite a lot of things.
C++ 2011
PowerDNS is mostly written in C++, and since 2011 this monumental language has been available in a new revision. C++ 2011 makes life a lot easier on programmers, which in turn means you can deliver more functionality in the same time, or perhaps the same functionality but then with higher quality.
C++ 2011 has merged a lot of functionality from the equally monumental Boost libraries, and we’ve taken the opportunity to move PowerDNS to the ‘native’ versions of these functionalities: range-for instead of BOOST_FOREACH, std::shared_ptr instead of boost::shared_ptr, std::to_string instead of boost::lexical_cast, for example.
Taking this step required revamping our entire build & regression testing infrastructure because the environments on which we test did not support the recent versions of the C++ compilers and libraries we required.
DNSName
Within PowerDNS hid a sin. DNS names frequently look like “ascii strings”. But they are anything but. DNS names compare case insensitively. Also, there is the issue of the trailing dot. “www.powerdns.com.” and “WwW.PoWeRdNs.com” are the same from a DNS perspective. Life becomes even more complicated when we realize that DNS names are ‘8-bit clean’. You can put any binary string in DNS and it should work. But how do we encode “some name.powerdns.com”? As some\032name.powerdns.com? Some\ name.powerdns.com?
There is only one worthy answer to these questions: we don’t. DNS is internally not stored as ascii but as a series of labels with specified lengths. So “www.powerdns.com” is stored as the value 3, followed by www, followed by the value 8, followed by powerdns, followed by 3, by com and finally the zero value. This is the right way to store DNS names.
To achieve this, we wrote the DNSName class which stores DNS values in this way, and also offers ways to parse DNSNames straight from packets, and to output them in “human friendly” form. Over the course of 4.0 development, DNSName got reimplemented a few times as we learned more, finally taking a shape where we could do canonical ordering very fast. This gave us the benefit of cleaning up a lot of ugly reversal code, and allowing all relevant caches to be purged not just name-by-name, but for whole zones in one go.
Finally, we’ve equipped DNSName with many methods that are useful in a DNS context like isPartOf() and chopOff(), removing lots of redundant code from PowerDNS.
Ridding PowerDNS from “DNS Names as ASCII” was a monumental undertaking that would not have been possible without extensive help from the community, specifically Kees Monshouwer and Aki Tuomi.
DNSResourceRecord
On a related note, for a long time, the PowerDNS Recursor showed its heritage as a spin-off from PowerDNS Authoritative Server. In the Authoritative Server, backends store DNS details as ASCII. So to encode the AAAA record 2001:888:2000:1d::2, we actually have that string “2001:888:2000:1d::2” of 19 characters in the database. This is not the most efficient thing to do, but for databases it is ok. They are good with strings.
However, INTERNALLY, it makes very little sense to drag these ASCII representations around and convert them into binary addresses and back again for processing. This is the sort of technical debt you build up over 15 years.
With great effort, we’ve been able to purge the PowerDNS Recursor of the DNSResourceRecord struct which carried those ASCII strings, and move everything to the DNSRecord class. We’ve checked, no bits of ASCII are hurt when answering questions in the Recursor anymore!
Netmask & Domain trees
Frequently, PowerDNS products need to check an IP address or domain name against a long list of possible matches. Based on the DNSName, ComboAddress and Netmask classes, Aki Tuomi and we have built generic structures that allow for high speed lookups of domain names & IP addresses against domain suffixes and netmasks, using Patricia Tries. These rapid lookups are now used within all three PowerDNS products, and are also available from Lua.
We can now safely disclose that our previous method for testing an IP address against many netmasks consisted of trying each one in order. Sorry for that.
Malloc Tracer
C++ is a wonderful language, we think, especially if you don’t turn it into a circus. A big problem of C++ however is the astounding amount of memory allocations that happen under the hood. And while memory allocators have gotten better over the years, we discovered we were doing hundreds of mallocs per packet in some circumstances. We’ve found that Heaptrack was helpful in getting a statistical overview of where our allocations were coming from, but we got very high per-packet precision using a very simple built-in (optional, turned off by default) malloc tracer. Using this, we’ve been able to reduce the allocation traffic by over 60% so far.
As an example, in the common case, the PowerDNS Recursor will now only issue two small mallocs per packet.
Configuration State
We try and succeed in getting a performance boost out of using multiple CPUs. This is not straightforward. No two threads can alter the same memory simultaneously, or bad things would happen. The easy solution against that is to lock the data. It turns out that when you sprinkle locks all over your code, any performance boost is gone. Performance might in fact go down with more threads.
A common case however is where a piece of memory rarely if ever changes and we can spare the memory to give each thread its own copy to read from. Only if something changed in the ‘master’ should each thread get a new copy. This idea is roughly what is known as Read Copy Update within system programming, and is for example also used by the great Knot DNS server. We created a set of classes called ‘State Holders‘ to bring this technology to PowerDNS as well.
Package Builder, Repositories
Although we loved Jenkins and we are mostly happy with Travis, we have gotten a lot of power from our ‘buildbot’ building engines. We now build PowerDNS for more and more platforms automatically, and push out those packages to our repositories on https://repo.powerdns.com/. These repositories allow you to install the latest and greatest builds of PowerDNS using apt or yum, and get native packages. This makes testing 4.0 really easy.
So what did we do with all those improvements?
Once this better new infrastructure was in place, we’ve implemented many new things:
- RPZ aka Response Policy Zone, as outlined here
- IXFR slaving in the PowerDNS Recursor for RPZ
- DNSSEC processing in Recursor (authoritative has had this for years)
- DNSSEC validation
- EDNS Client Subnet support in PowerDNS Recursor (authoritative has had this for years)
- GEOIP backend supporting custom netmasks, “fields” in answers
- Newly revived ODBC backend for talking to Microsoft SQL Server & Azure
- Lua asynchronous queries for per-IP/per-domain status
- Caches that can now be wiped per whole zone instead of per name
- An astounding amount of dnsdist features (check out the movie & the presentation!)
- And much more
We’ll outline what is in 4.x more completely in an upcoming post, including details on when and how it will be released. For now, it may be good to know you can test these new features via the package builder and repo service as outlined above.
Good luck!
