Category: Uncategorized

PowerDNS Recursor 4.1.2 Released

This release improves the stability and resiliency of the RPZ implementation and fixes several issues related to EDNS Client Subnet.

The full changelog looks like this:

New Features

  • #6344: Add FFI version of gettag().

Improvements

  • #6298, #6303, #6268, #6290: Add the option to set the AXFR timeout for RPZs.
  • #6172: IXFR: correct behavior of dealing with DNS Name with multiple records and speed up IXFR transaction (Leon Xu).
  • #6379: Add RPZ statistics endpoint to the API.

Bug Fixes

  • #6336, #6293, #6237: Retry loading RPZ zones from server when they fail initially.
  • #6300: Fix ECS-based cache entry refresh code.
  • #6320: Fix ECS-specific NS AAAA not being returned from the cache.

The tarball is available on downloads.powerdns.com (signature) and packages for CentOS 6 and 7, Debian Jessie and Stretch, Ubuntu Artful, Trusty and Xenial are available from repo.powerdns.com.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

“The DNS Camel”, or, the rise in DNS complexity

This week was my first IETF visit. Although I’ve been active in several IETF WGs for nearly twenty years, I had never bothered to show up in person. I now realize this was a very big mistake – I thoroughly enjoyed meeting an extremely high concentration of capable and committed people. While RIPE, various NOG/NOFs and DNS-OARC are great venues as well, nothing is quite the circus of activity that an IETF meeting is. Much recommended!

DYv4Tt4V4AADCeT

Before visiting I read up on recent DNS standardization activity, and I noted a ton of stuff was going on. In our development work, I had also been noticing that many of the new DNS features interact in unexpected ways. In fact, there appears to be somewhat of a combinatorial explosion going on in terms of complexity.

As an example, DNAME and DNSSEC are separate features, but it turns out DNAME can only work with DNSSEC with special handling. And every time a new outgoing feature is introduced, like for exampled DNS cookies, new probing is required to detect authoritative servers that get confused by such newfangled stuff.

This led me to propose a last minute talk (video!) to the DNSOP Working Group, which I tentatively called “The DNS Camel, or, how many features can we add to this protocol before it breaks”. This ended up on the agenda as “The DNS Camel” (with no further explanation) which intrigued everyone greatly. I want to thank DNSOP chairs Suzanne and Tim for accommodating my talk which was submitted at the last moment!

Note: My “DNS is too big” story is far from original! Earlier work includes “DNS Complexity” by Paul Vixie in the ACM Queue and RFC 8324 “DNS Privacy, Authorization, Special Uses, Encoding, Characters, Matching, and Root Structure: Time for Another Look” by John Klensin. Randy Bush presented on this subject in 2000 and even has a slide describing DNS as a camel!

Based on a wonderful chart compiled by ISC, I found that DNS is now described by at least 185 RFCs. Some shell-scripting and HTML scraping later, I found that this adds up to 2781 printed pages, comfortably more than two copies of “The C++ Programming Language (4th edition)”. This book is not known for its brevity.

Screenshot from 2018-03-22 21-21-51

Artist impression of DNS complexity over time

In graph form, I summarised the rise of DNS complexity as above. My claim is that this rise is not innocent. As DNS becomes more complex, the number of people that “get it” also goes down. Notably, the advent of DNSSEC caused a number of implementations to drop out (MaraDNS, MyDNS, for example).

Also, with the rise in complexity and the decrease in number of capable contributers, the inevitable result is a drop in quality:

Screenshot from 2018-03-22 21-27-28

Orange = number of people that “get it”. Green is perceived implementation quality. Also lists work in the pipeline.

And in fact, with the advent of DNSSEC this is what we found. For several years, security & stability bugs in popular nameserver implementations were absolutely dominated by DNSSEC and cryptography related issues.

My claim is that we are heading for that territory again.

So how did this happen? We all love DNS and we don’t want to see it harmed in any way. Traditionally, protocol or product evolution is guided by forces pulling and pushing on it.

Screenshot from 2018-03-22 22-12-43

Actual number of RFC pages over time. Grows at around 2 pages/week. Shutdown of DNSEXT is barely visible

Requirements from operators ‘pull’ DNS in the direction of greater complexity. Implementors meanwhile usually push back on such changes because they fear future bugs, and because they usually have enough to do already. Operators, additionally, are weary of complexity: they are the ones on call 24/7 to fix problems. They don’t want their 3AM remedial work to be any harder than it has to be.

Finally, the standardization community may also find things that need fixing. Standardizers work hard to make the internet better (the new IETF motto I think), and they find lots of things that could be improved – either practically or theoretically.

In the DNS world, we have the unique situation that (resolver) operator feedback is largely absent. Only a few operators manifest themselves in the standardization community (Cloudflare, Comcast, Google, Salesforce being notably present). Specifically, almost no resolver operator (access provider) ever speaks at WG meetings or writes on mailing lists. In reality, large scale resolver operators are exceptionally weary of new DNS features and turn off whatever features they can to preserve their night time rest.

Screenshot from 2018-03-22 21-47-35

On the developer front, the DNS world is truly blessed with some of the most gifted programmers in the world. The current crop of resolvers and authoritative servers is truly excellent. DNS may well be the best served protocol in existence today. This high level of skill also has a downside however. DNS developers frequently see immense complexity not as a problem but as a welcome challenge to be overcome. We say yes to things we should say no to. Less gifted developer communities would have to say no automatically since they simply would not be able to implement all that new stuff. We do not have this problem. We’re also too proud to say we find something (too) hard.

Finally, the standardization community has its own issues. A ‘show of hands’ made it clear that almost no one in the WG session was actually on call for DNS issues. Standardizers enjoy complexity but do not personally bear the costs of that complexity. Standardizers are not on 24/7 call as there rarely is a need for an emergency 3AM standardization session!

Notably, a few years ago I was informed by RFC authors that ‘NSEC3’ was easy. We in the implementation community meanwhile were pondering that the ‘3’ in NSEC3 probably stood for the number of people that understood this RRTYPE! I can also report that as of 2018, the major DNSSEC validator implementations still encounter NSEC3 corner cases where it is not clear what the intended behaviour is.

Note that our standardizers, like our developers, are extremely smart people. This however is again a mixed blessing – this talent creates at the very least an acceptance of complexity and a desire to conquer really hard problems, possibly in very clever ways.

The net result of the various forces on DNS not being checked is obvious: more and more complex features.

Orthogonality of features

As noted above, adding a lot of features can lead to a combinatorial explosion. DNSSEC has to know about DNAME. CZNic contributed related the following gem they discovered during the implementation of ‘aggressive NSEC for NXDOMAIN detection’: it collides with trust-anchor signalling. The TA signalling happens in the form of a query to the root that leads to an NXDOMAIN, with associated NSEC records. These NSEC records then shut up further TA signalling, as no TA related names apparently exist! And here two unrelated features now need to know about each other:  aggressive NSEC needs to be disabled for TA signalling.

If even a limited number of features overlap (ie, are not fully orthogonal), soon the whole codebase consists of features interacting with each other.

We’re well on our way there, and this will lead to a reduction in quality, likely followed by a period of stasis where NO innovation is allowed anymore. And this would be bad. DNS is still not private and there is a lot of work to do.

Suggestions

I rounded off my talk with a few simple suggestions:

Screenshot from 2018-03-22 22-02-30

Quickly a 20 person long queue formed at the mic. It turns out that while I may have correctly diagnosed a problem, and that there is wide agreement that we are digging a hole for ourselves, I had not given sufficient thought about any solutions.

IETF GROW WG chair Job Snijders noted that the BGP-related WGs have implemented different constituencies (vendors, operators) that all have to agree. In addition, interoperable implementations are a requirement before a draft can progress to standard. This alone would cut back significantly on the flow of new standards.

Other speakers with experience in hardware and commercial software noted that in their world the commercial vendors provided ample feedback to not make life too difficult, or that such complexity would at least come at huge monetary cost. Since in open source features are free, we do not “benefit” from that feedback.

There was enthusiasm for the idea of going through the “200 DNS RFCs” and deprecating stuff we no longer thought was a good idea. This enthusiasm was more in theory than in practice though as it is known to be soul crushing work.

The concept however of reducing at least the growth in DNS complexity was very well received. And in fact, in subsequent days, there was frequent discussion about the “DNS Camel”:

camel

And in fact, a draft has even been written that simplifies DNS by specifying DNS implementations no longer need to probe for EDNS0 support. The name of the draft? draft-spacek-edns-camel-diet-00!

I’m somewhat frightened of the amount of attention my presentation got, but happy to conclude it apparently struck a nerve that needed to be struck.

Next steps

So what are the next steps? There is a lot to ponder.

I’ve been urged by several very persuasive people to not only rant about the problem but to also contribute to the solution, and I’ve decided these people are right. So please watch this space!

 

Authoritative server 4.1.1 released

Released: 16th of Feburary 2018

This is the second release in the 4.1 train.

This is a bug-fix only release, with fixes to the LDAP and MySQL backends, the pdnsutil tool, and PDNS internals.

Changes since 4.1.1:

Bug Fixes

  • Backport: forbid label compression in alias wire format
  • Include unistd.h for chroot(2) et al. (Florian Obser)
  • Auth: fix out of bounds exception in caa processing, fixes #6089
  • Add the missing include to mplexer.hh for struct timeval
  • Auth: init openssl and libsodium before chrooting in pdnsutil
  • Auth: always bind the results array after executing a mysql statement
  • Ldap: fix getdomaininfo() to set this as di.backend (Grégory Oestreicher)
  • Ldapbackend: fix listing zones incl. axfr (Chris Hofstaedtler)
  • Ixfr: correct behavior of dealing with dns name with multiple records (Leon Xu)

Tarball (sig) is available on the downloads website. Packages for Debian, CentOS and Ubuntu are uploaded to our repositories.

PowerDNS Recursor 4.1.1

This is the second release in the 4.1 train with a fix for the following DNSSEC security advisory:

Note that although this is a security vulnerability, the impact is limited to certain DNSSEC data wrongly being served as authentic. Specifically, attackers could cause PowerDNS to accept denials of existence for domain names that did in fact exist.

This is a release on the stable branch containing a fix for the aforementioned security issue and several bug fixes from the development branch.

The full changelog looks like this:

Improvements

  • #6085: Don’t process records for another class than IN. We don’t use records of another class than IN, but we used to store some of them in the cache which is useless. Just skip them.

Bug Fixes

  • #6215: Correctly handle ancestor delegation NSEC{,3} for children. Fixes the DNSSEC validation issue found in Knot Resolver, where a NSEC{3} ancestor delegation is wrongly use to prove the non-existence of a RR below the delegation. We already had the correct check for the exact owner name, but not for RRs below the delegation. (Security Advisory 2018-01)
  • #6092: Fix the computation of the closest encloser for positive answers. When the positive answer is expanded from a wildcard with NSEC3, the closest encloser is not always parent of the qname, depending on the number of labels in the initial wildcard.
  • #6095: Pass the correct buffer size to arecvfrom(). The incorrect size could possibly cause DNSSEC failures.
  • #6209: Fix to make primeHints threadsafe, otherwise there’s a small chance on startup that the root-server IPs will be incorrect.
  • #6137: Don’t validate signature for “glue” CNAME, since anything else than the initial CNAME can’t be considered authoritative.

The tarball is available on downloads.powerdns.com (signature) and packages for CentOS 6 and 7, Debian Jessie and Stretch, Ubuntu Artful, Trusty and Xenial are available from repo.powerdns.com.

Please send us all feedback and issues you might have via the mailinglist, or in case of a bug, via GitHub.

 

PowerDNS end of year post: Thank you!

Greetings!

2017 has been a great year for PowerDNS and Open-Xchange. In this post, we want to thank everyone that contributed, and highlight some specific things we are happy about.

HackerOne bug bounty program

After some initial problems with over-reporting of non-issues, our experience with HackerOne is awesome right now. We are very happy we have a clean process for receiving and rewarding security bugs. Various PowerDNS security releases this year have originated as HackerOne reports.

Our community

PowerDNS continues to be a vibrant community. Our IRC channel has around 240 members, our mailing lists have 1225 subscribers. Even though we are now tougher in enforcing our ‘support, out in the open‘ policies, we continue to see many user queries being resolved every day, often leading to improvements in PowerDNS.

As in earlier years, 2017 has seen huge contributions from the community, not only in terms of small patches or constructive bug reports, but also in the revamping of whole subsystems. Specifically Kees Monshouwer was so important for Authoritative Server 4.1 that we would not have been able to do it without him. We hope to continue as a healthy community in 2018!

Facebook bug bounty program

Image result

PowerDNS is an active participant in keeping the internet secure. As part of our work we found a potential security problem in an important Facebook product which we reported to the their bug bounty program.  The bug was fixed quickly, and led to an award of $1500, with the option to turn that into a $3000 charitable donation. We have done so and supported Doctors without Borders in their work.

Our Open Source DNS friends

The DNS community is tight, and it has to be: all our software has to interoperate. New standards are developed cooperatively and problems are discussed together. We love the friendly competition that we have with our friends of CZNIC (Knot, Knot Resolver), ISC (BIND), NLNetLabs (NSD, Unbound, libraries) and others.

To a huge extent, DNS is exclusively Open Source software, sometimes repackaged and rebadged by commercial companies that close down that Open Source software again.

PowerDNS is proud to be part of the open DNS community, and we are grateful for the smooth & fun cooperation we experienced in 2017!

 

Open-Xchange

Since 2015, PowerDNS has been part of Open-Xchange, previously mostly known for the OX AppSuite email platform. The famous Dovecot IMAP project also joined Open-Xchange in 2015. The goal of these mergers was to allow us to focus on technology, while getting the legal, sales and marketing support to get our software out there.

In 2017 we have truly started to harvest the fruits of the merger, by simultaneously delivering important software releases as well as satisfying the needs of some very large new deployments.

We are very happy that PowerDNS not only survived the merger, but is now an important part of Open-Xchange, where we contribute to the mission of keeping the internet open.

Our users

Even without or before contributing code, operators can improve PowerDNS through great bug reports. We specifically want to thank Quad9 (a collaboration of Packet Clearing House, IBM and the Global Cyber Alliance) for taking a year long journey with us with dnsdist and Recursor “straight from GitHub”. Deployments sharing their experiences and problems with the PowerDNS community are vital to creating quality reliable software. Thanks!

Mattermost, the Open Source private Slack Alternative

As PowerDNS grows, we could no longer rely solely on IRC as our communication channel with developers, users and customers. Instead of moving to a third party cloud service that admits to datamining communications, we are very happy to host our own Mattermost instance. And because of PowerDNS user & contributor @42Wim, we can continue our IRC habit with matterircd.

4.1 evolution, dnsdist

In 2016 we released the 4.0 versions of the PowerDNS Authoritative Server and Recursor. As you may recall, the 4.0 releases represented a giant cleanup from the decade old frameworks found in 3.x. The 4.0 versions were a step ahead in functionality and sometimes performance, but the true gains of the new fresher codebase have now been realized in the 4.1 releases.

4.1 represents a big overhaul in caching (both Recursor and Authoritative) and DNSSEC processing (mostly Recursor). Both of these overhauls have been tested over the year by large PowerDNS deployments, and the huge amount of feedback has delivered a near flawless “battle tested” 4.1 release.

Specifically xs4all and two huge European incumbent operators have been instrumental in maturing dnsdist and our 4.1-era DNSSEC and EDNS Client Subnet implementations.

On to 2018!

In 2018 we hope to continue to improve our software and the state of the internet. See you there!

 

PowerDNS Authoritative: Lua Records

Hi everyone,

We are happy to share a new development with you, one that we hinted at over a year ago: Lua resource records. In this post, we ask for your help: did we get the feature right? Are we missing important things? The goal is to release Lua records in January 2018, but we can only make that with your testing and feedback! At the end of this post you will find exact instructions how to test the new LUA records.

Note: The fine authors of the Lua programming language insist that it is Lua and not LUA. Lua means ‘moon’ in Portuguese, and it is not an abbreviation. Sadly, it is DNS convention for record types to be all uppercase. Sorry.

While PowerDNS ships with a powerful geographical backend (geoip), there was a demand for more broader solutions that include uptime monitoring, which in addition could run from existing zones.

After several trials, we have settled on “LUA” resource records, which look like this:

 @   IN   LUA   A   "ifportup(443, {'52.48.64.3', '45.55.10.200'})"

When inserted in a zone with LUA records enabled, any lookups for your domain name will now return one of the listed IP addresses that listens on port 443. If one is down, only the other gets returned. If both are down, both get returned.

But if both are up, wouldn’t it be great if we could return the ‘best’ IP address for that client? Say no more:

@    IN   LUA A ( "ifportup(443, {'52.48.64.3', '45.55.10.200'}, "
                  "{selector='closest'})                          ")

This will pick the IP address closest to that of the client, according to the MaxMind database as loaded in the geoip backend. This of course also takes the EDNS Client Subnet option into account if present.

But why stop there? Merely checking if a port is open may not be enough, so how about:

@ IN LUA A ( "ifurlup('https://powerdns.com/' ,                    "
             "{'52.48.64.3', '45.55.10.200'}, {selector='closest', "
             "stringmatch='founded in the late 1990s'})            ")

This will check if the IP addresses listed actually want to serve the powerdns.com website for us, and if the content served lists a string that should be there.

The ‘closest’ selector relies on third party data, and if you are a large access provider, you may have more precise ideas where your users should go. There are various ways of doing that. One way goes like this:

www IN LUA CNAME (";if(netmask('130.161.0.0/16', '213.244.0.0/24')" 
                  "then return 'local.powerdns.com' else          "
                  "return 'generic.powerdns.com' end              ")
local IN LUA A    "ifportup(443, {'192.0.2.1', '192.0.2.2'}       "
generic IN LUA A ("ifportup(443, {'192.0.2.1', '192.0.2.2',       " 
                  "'198.51.100.1'}, {selector='closest'}          ")

Note: the starting semicolon tells the Lua record that this is a multi-statement record that does not directly return record content. More specifically, PowerDNS will prepend “return ” to your statement normally.

Another way which works without CNAMEs, and thus at the apex, goes like this:

@ IN LUA A (";if(netmask('130.161.0.0/16', '213.244.0.0/24')      " 
            "then return ifportup(443, {'192.0.2.1', '192.0.2.2'})"
            "else return ifportup(443, {'192.0.2.1', '192.0.2.2'},"
            "'198.51.100.1'}, {selector='closest'}                ")

Doing dynamic responses at apex level is a common problem of other GSLB solutions.

To steer based on AS numbers, use if(asnum{286,1136}), for example. Countries can be selected based on their two-letter ISO code using if(country{‘BE’,’NL’,’LU’}).

In the examples above we have been typing the same IP addresses a lot. To make this easier, other records can be included to define variables:

config    IN    LUA    LUA (";settings={stringmatch='Programming in Lua'} "
                            "EUips={'192.0.2.1', '192.0.2.2'}             "
                            "USAips={'198.51.100.1'}                      ")

www       IN    LUA    CNAME ( ";if(continent('EU')) then return 'west.powerdns.org' "
                               "else return 'usa.powerdns.org' end" )

usa       IN    LUA    A    ( ";include('config')                              "
                              "return ifurlup('https://www.lua.org/',        "
                              "{USAips, EUips}, settings)                    " )

west      IN    LUA    A    ( ";include('config')                              "
                              "return ifurlup('https://www.lua.org/',        "
                              "{EUips, USAips}, settings)                    " )

This shows off another feature of ifurlup, it knows about IP groups, where it prefers to give an answer from the first set of IP addresses, and if all of those are down, it tries the second set etc etc. In this example, the ‘local’ set of IP addresses is listed first for both regions.

More possibilities

We use LUA records to power our ‘lua.powerdns.org’, ‘v4.powerdns.org’ and ‘v6.powerdns.org’ zones:

$ dig -t aaaa whoami.v6.powerdns.org +short
2a02:a440:b085:1:20d:b9ff:fe3f:8018
$ dig -t txt whoami-ecs.v6.powerdns.org +short @8.8.8.8
"ip: 2a00:1450:4013:c02::10a, netmask: 86.82.68.0/24"
$ dig -t loc latlon.v4.powerdns.org +short
51 37 15.236 N 5 26 31.920 E 0.00m 1m 10000m 10m
$ dig -t txt whoami.lua.powerdns.org +short
"2a02:a440:b085:1:20d:b9ff:fe3f:8018"

These queries deliver, respectively:

  • IPv6 address of your resolver (will not resolve without IPv6)
  • Any EDNS Client Subnet details over IPv6 (also works on v4.powerdns.org)
  • LOC record of where Maxmind thinks your resolver (or ECS address) is
  • A ‘pick your protocol’ equivalent of the v4 or v6 specific whoami queries

The actual records look like this:

whoami.lua     IN LUA TXT  "who:toString()"
whoami-ecs.lua IN LUA TXT  "'ip: '..who:toString()..', netmask: '..(ecswho and ecswho:toString() or 'no ECS')"
latlon.lua     IN LUA LOC  "latlonloc()"
whoami.v6      IN LUA AAAA "who:toString()"
whoami.v4      IN LUA A    "who:toString()"

Further details

Full documentation for this feature can be found here. To test, packages can be found on:

Install the main PowerDNS package, the gsqlite3 (for example) and geoip backends.

For Ubuntu/Debian: After installing the packages, you may need to run ‘apt-get install -f’ to get the dependencies. In addition, to benefit from Maxmind, you may have to install a package with a name like geoip-database-contrib or geoipupdate.

For CentOS/RHEL:

# yum install epel-release yum-plugin-priorities
# tar xf pdns*luarec*bz2

Then cd into the newly created directory and ‘yum install’ the packages mentioned above.

Setting up PowerDNS & Lua

Setup gsqlite3 as described here (or gmysql, gpgsql), then edit the pdns.conf to include:

launch=gsqlite3,geoip
gsqlite3-database=/location/of/powerdns.sqlite
local-address=0.0.0.0
local-ipv6=::
edns-subnet-processing
log-dns-queries
loglevel=9
geoip-database-files=/usr/share/GeoIP/GeoIPCity.dat,/usr/share/GeoIP/GeoIPASNum.dat
enable-lua-record

Most of this is generic to PowerDNS. Specific for our use is loading the geoip backend and its database files, enabling the LUA record, EDNS Client Subnet processing, and some debug logging so you see what is happening. The geoip-database-files path may be different depending on your operating system.

Next up, generate a test zone, and edit it:

$ pdnsutil create-zone geo.example.com ns1.example.com
Creating empty zone 'geo.example.com'
Also adding one NS record
$ pdnsutil edit-zone geo.example.com

This will fire up an editor, and allows you to insert your first LUA record. For fun, try:

geo.example.com 3600 IN LUA TXT "os.date()"

Save, and pdnsutil will ask you if you want to apply this change. Do so, and then query your PowerDNS:

$ dig -t txt geo.example.com @127.0.0.1 +short
"Thu Dec 14 21:49:00 2017"

After this you can try the zonefiles listed above, or paste from the ‘lua.powerdns.org’, ‘v4.powerdns.org’ and ‘v6.powerdns.org’ zones.

If this does not work for you (even after reading the documentation), please find us through our Open Source page. In addition, if it does work for you but you have feedback or features you need, please also let us know through powerdns.ideas@powerdns.com.

Thanks & enjoy!