How PowerDNS is Open Source & a successful business, or, why are we talking about 5G?

What does PowerDNS actually do?

This is a good question, one we can ask about any company. How do they stay alive, what services do they deliver, who do they sell them to?

For Open Source companies, the question is doubly interesting: if your software is so great, and you give it away for free (as in freedom), how do you survive?

In this post I want to explain how PowerDNS (and our parent Open-Xchange) have squared this circle. In many large countries, PowerDNS & Open-Xchange are now the DNS supplier to the largest telecommunications companies.

Below you will also read why we are all of a sudden talking about end-to-end monitoring, “the 5G transition“, DNS over HTTPS and (Network Function) Virtualization (NFV).

Products
Everything starts with products of course, and PowerDNS has four main ones. The Authoritative Server hosts domain names, and it dominates the mid-size market of hosters running up to 10 million domains. While there are other very good open source authoritative nameservers, PowerDNS has an edge because of its wide support for databases, its DNS-aware checking API and lately the new LUA records which deliver DNS based traffic-engineering, failover and load-balancing.

The PowerDNS Recursor meanwhile has picked an interesting niche among resolvers & caches, where again the open source landscape delivers outstandingly good software. The Recursor supports the big important features of course, like DNSSEC and shortly QName minimization, but our focus has been on providing servers that deliver great performance & rock-solid stability for high-capacity operators, while retaining the flexibility to do malware filtering, parental control and security analysis. Of specific note is our support for interoperating with CDNs like Akamai that require EDNS Client Subnet, while retaining top performance.

Our third product, dnsdist, for now appears to be unique – a scriptable high performance, DoS-aware load balancer & distributor of DNS queries.  It protects installations from denial of service attacks, of which even small ones can burn up a lot of CPU. Dnsdist also delivers such modern encrypted variants of DNS as DNS over HTTPs and DNS over TLS. It has a built-in cache that delivers stellar performance even on top of slow backend. Dnsdist is highly flexible and can redirect queries based on almost every aspect of a question. It frequently replaces dedicated load-balancer hardware. Although only a few years old, we were very pleased to learn dnsdist was part of the recent NATO Locked Shields cybersecurity exercise in Estonia.

These first three products are built in close cooperation with our lovely community. A community is far more than people supplying patches. It also consists of users vocally telling us what they need or pointing out that what we do is exactly what they don’t need. It consists of the heroes that test pre-releases and let us know if the quality or the features are where they should be. We are also super happy with users that point out where documentation is missing or wrong. Conversely, we truly enjoy helping our users improve their lives with open source, where we cooperate daily with other open source projects.

Finally there is the part that is not open source, the PowerDNS Platform that delivers the first three products in an integrated, automated, monitored and graphed solution, with a central graphical & scriptable control plane. In addition, with OX Protect, this platform provides for malware filtering & parental control.

What we actually sell
Who would buy a nameserver when there are so many good ones available to download for free? Asking the question almost answers it: operators that do not wish to deploy and assemble the raw goods they can find on the internet. While it is entirely possible to have teams and infrastructure in place to do just that, many modern telecommunications operators have decided to only deploy fully supported units of functionality.

While it is entirely possible to assemble similar functionality to our Platform with open source components, this is a lot of work and operators would have to learn how to scale, monitor and control such a system. There is value in getting this as a preassambled whole – even as we retain our open interfaces for integration into existing monitoring and graphing systems. But beyond that – assembling platforms by hand is a risky business.

This is a variant of the old story that no serious company would run software without a support contract in place. While this was not quite true, what we are seeing today is a step even beyond that. A support contract is a suitable solution if the operator decides to take full ownership of architecting, deploying, testing and running a project. The support is important for the rare cases where things do not go as planned – it is in fact a warranty.

Although we have a number of excellent customers where we provide such support as a service, in almost all cases our engagement these days goes far beyond answering email messages.

Delivering functionality
A large scale enterprise, like a telecommunication service provider, is a complex organization. For every project there are many stakeholders – there are product departments that want specific functionalities and performance for the subscribers. There are legal and compliance departments that make sure vendors have the right certifications and can be held liable for intellectual property violations. Service Level Agreements need to be spelled out in great detail, including penalty clauses. Whenever consumer communications are touched, GDPR compliance is of utmost importance.

Then there are network and infrastructure teams that each have their own requirements for hardware, virtualization specifications and capacity. On top of this, there are always existing software installations with sometimes custom features that need to be retained and migrated.

Of supreme importance is high-level sign-off. Senior management needs to be reassured that this is a vendor worth betting on. Or as a big PowerDNS customer once phrased it “you need to hire more golf players to grow”. We took this message on board. This is also why you will be seeing PowerDNS opine on 5G deployments, on Network Function Virtualization and End-to-End performance monitoring and reporting.

To round this off, a project of any serious size will be run through a procurement department, often at group level, sometimes even in a different company. Navigating an RFP is a skill in itself – especially when third party integrators or vendors are fronting the project.

In short, to deliver a working solution requires coordination among all these departments and the creation of an architecture, a training plan, a support structure, a hardware/software layout, a migration procedure, and all of this needs to be ‘sold’ through the procurement department.

So if a modern telecommunications company wants to deploy a new nameserver constellation, it will require not just the software but all of the above.

Deploying functionality
After a project has been specced up properly and the papers are signed, next up is the actual deployment and migration. When we launched PowerDNS in the late 1990s, it was clearly up to the operator to perform deployment and migrations. This made sense on one level: testing & deploying software (or hardware) is the best way to make sure operators fully understand what they bought and that they can support it themselves.

Conversely however, a vendor deploys and migrates its products all the time. Vendors therefore have developed tooling and procedures to make this happen swiftly. We can’t expect a service provider that does a hardware refresh once every 4 years to have performed many migrations itself with the existing staff – it simply does not happen that often.

These days, most customers ask us to be very or even completely hands-on during testing, rollout and migration. We do however vastly prefer to perform such operations in close cooperation with the intended operators – because it remains true that “doing” is the ultimate form of training.

Collaborative operations
Traditionally, vendors grudgingly provide support in case of proven malfunctions. It is now so hard to open tickets with major network vendors that at least one company we know of sells “opening network vendor tickets” as-a-service – allowing operators to focus on solving problems. This is not how we want our customers to work with us however!

To our large scale operators, we provide collaborative operations services. This means there is no need to ‘escalate’ something so it is an issue. Whenever there is a need for a configuration change, or there is a worry because of a graph that is going in the wrong way, we are there to provide guidance, scripts or hands on help.

What we have managed to do is retain our open source collaborative nature, but deliver it also to the largest of operators, wrapped in solid service level agreements.

Summing it up
“The secret to PowerDNS’s success” is that we are able to take excellent open source software, and deliver it to large scale telecommunications service providers, while continuing to be an open and accessible vendor. And it turns out that everything we provide on top of the raw open source software is worth good money to our customers.

As of 2019, PowerDNS is growing rapidly. And as the rollout of DNS over TLS/HTTPS, 5G transition, (Network Function) Virtualization at service providers continues, it appears we will be an ever larger part of the telecommunication landscape.

If you our your company are interested in working with us for your next DNS project, please do not hesitate to contact us! For more about PowerDNS, please head to https://powerdns.com or to https://open-xchange.com/.

PowerDNS Recursor 4.2.0 Beta 1 Released

The Beta 1 release comes with a lot of bug fixes, improvements and also some new features:

  • Add a new max-cache-bogus-ttl option to cap the TTL of a record that has been validated as Bogus in the query cache, so it is not kept around for days if the initial TTL is high,
  • Add options dont-throttle-names and dont-throttle-netmasks to throttle authoritative servers that do not answer queries or send responses the recursor does not like,
  • Add an option (pdns-distributes-queries) to make the distributor thread use a bounded load-balancing algorithm while distributing queries to worker threads, making sure that no thread is assigned more queries than distribution-load-factor times the average number of queries currently processed by all the workers.

Please see the changelog for details.

This release was made possible by contributions from: Aki Tuomi, Chris Hofstaedtler, Shane Kerr and Sebastian.

The tarball (signature) is available at downloads.powerdns.com and packages for CentOS 6 and 7, Debian Jessie, Stretch and Buster, Ubuntu Trusty, Xenial, Bionic and Cosmic are available from repo.powerdns.com.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

dnsdist 1.4.0-alpha2 with DNS over HTTPS support

We are very happy to announce the second alpha release of the 1.4.0 version of dnsdist. This version keeps up the DNS privacy improvements with the addition of a new major feature, DNS over HTTPS (DoH), and contains very few changes apart from that.

As with the first alpha, your feedback will be much appreciated so we can deliver a stable 1.4.0 final release!

Please see the dnsdist website for the more complete changelog and the current documentation.

Release tarballs are available on the downloads website.

Several packages are also available on our repository.

First alpha release of dnsdist 1.4.0

We are very happy to announce the 1.4.0 alpha 1 release of dnsdist. This version contains a few new features, but is mostly focused on DNS privacy improvements. We are introducing a new, much more scalable way of handling DNS over TCP and DNS over TLS connections. It will be followed quite quickly by a new alpha including experimental DNS over HTTPS support.

In older versions of dnsdist, a TCP worker could only handle one incoming connection at a time, which was not very efficient when dealing with a larger number of mostly inactive connections, as we are beginning to see with DNS over TLS. Starting with this release, TCP workers are now event-based and each one of them can handle a very large number of incoming connections simultaneously.

Your feedback will be much appreciated so we can deliver a stable 1.4.0 final release!

Important changes

We took the opportunity of this new release to clean up a few things that might require updating your existing configuration. First, the number of parameters to the newPacketCache command was getting out of hand, so we switched it to a table-based syntax as we already did with newServer a while ago.

addLuaAction and addLuaResponseAction have been removed. Instead, use addAction with a LuaAction, or addResponseAction with a LuaResponseAction.

Lua constants for DNS response codes and QTypes have been moved from the ‘dnsdist’ prefix to, respectively, the DNSQType and DNSRCode prefixes.

To improve security, all ambient capabilities are now dropped after the startup phase, which might prevent launching the webserver on a privileged port at run-time, or impact some custom Lua code. In addition, systemd’s sandboxing features are now determined at compile-time, resulting in more restrictions on recent distributions. See pull requests 7138 and 6634 for more information.

And finally, if you are compiling dnsdist, note that several ./configure options have been renamed to provide a more consistent experience. Features that depend on an external component have been prefixed with –with while internal features use –enable. This has lead to the following changes:

  • –enable-fstrm to –enable-dnstap
  • –enable-gnutls to –with-gnutls
  • –enable-libsodium to –with-libsodium
  • –enable-libssl to –with-libssl
  • –enable-re2 to –with-re2

New features and improvements

Dynamic blocks and Lua rules can now use the NoRecurse action, thanks to phonedph1.

Richard Gibson added the possibility to inspect and alter trailing data.

Dmitry Alenichev implemented new rules and actions to deal with unexpected EDNS versions, and to optionally accept completely empty (qdcount=0) responses from a backend.

Andrey Domas added the new QNameSetRule rule, along with the DNSNameSet object, to match exact qnames instead of doing suffix matching.

The health check mechanism has been improved with the new checkInterval, checkTimeout and rise parameters, thanks notably to “1848”.

We added a few convenience functions to pseudonymize IP addresses, as several users reported that they needed it to be GDPR-compliant.

We noticed that, on some specific versions of the Linux kernel, the code we used to measure our memory usage could be quite expensive so we switched to an alternative, cheaper method. You might notice that the memory usage reported by this new version does not exactly match the one reported by older versions, but it should be close enough.

Finally the cost of exporting queries and responses using our remote logging solution based on protobuf has been reduced by a huge margin. System calls that used to be cheap before the Spectre and Meltdown mitigations were introduced are now having a very visible impact, and we designed a new way of sending messages to work around that.

Please see the dnsdist website for the more complete changelog and the current documentation.

Release tarballs are available on the downloads website.

Several packages are also available on our repository. Please be aware that we have enabled a few additional features in our packages, like DNS over TLS and DNSTap support, on distributions where the required dependencies were available.

PowerDNS Recursor 4.1.12 Released

This is a maintenance release with improvements for high-performance sites (and a wild bug fix appeared).

The changelog:

  • #7634: Use a bounded load-balancing algo to distribute queries.
  • #7651: Implement a configurable ECS cache limit so responses with an ECS scope more specific than a certain threshold and a TTL smaller than a specific threshold are not inserted into the records cache at all.
  • #7647: Provide CPU usage statistics per thread (worker & distributor).
  • #7495: Correctly interpret an empty AXFR response to an IXFR query.

The tarball (signature) is available at downloads.powerdns.com and packages for CentOS 6 and 7, Debian Jessie and Stretch, Ubuntu Trusty, Xenial and Bionic are available from repo.powerdns.com.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

PowerDNS Authoritative Server 4.1.8 Released

This release contains a bunch of backports, improvements and bug fixes that we wanted in the 4.1.x release channel after the 4.1.7 security release was out of the door.

Please see the changelog for full details:

  • #7604: Correctly interpret an empty AXFR response to an IXFR query,
  • #7610: Fix replying from ANY address for non-standard port,
  • #7609: Fix rectify for ENT records in narrow zones,
  • #7607: Do not compress the root,
  • #7608: Fix dot stripping in setcontent(),
  • #7605: Fix invalid SOA record in MySQL which prevented the authoritative server from starting,
  • #7603: Prevent leak of file descriptor if running out of ports for incoming AXFR,
  • #7602: Fix API search failed with “Commands out of sync; you can’t run this command now”,
  • #7509: Plug mysql_thread_init memory leak,
  • #7567: EL6: fix CXXFLAGS to build with compiler optimizations.

This release was made possible by contributions from: Aki Tuomi, Chris Hofstaedtler and Kees Monshouwer.

The tarball (signature) is available at downloads.powerdns.com and packages for CentOS 6 and 7, Debian Jessie and Stretch, Ubuntu Trusty, Xenial and Bionic are available from repo.powerdns.com.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.

PowerDNS Authoritative Server 4.2.0 Release Candidate 1 Released

This release fixes an issue with security implications that has been recently reported in the HTTP remote backend of the PowerDNS Authoritative Server. Setups that are not using this backend are not impacted by this issue. More information can be found in the corresponding security advisory:

There are some additional smaller improvements and bug fixes in this release.  Please see the changelog:

  • #7576: Insufficient validation in the HTTP remote backend
  • #7546: Fix API search failed with “Commands out of sync; you can’t run this command now”
  • #7219: Fix static lookup when using weighted records on multiple record types.
  • #7516: Report “checkKey“ errors upwards.

This release was made possible by contributions from: Aki Tuomi, Sebastian, Robin Geuze and Baptiste Courtois.

The tarball (signature) is available at downloads.powerdns.com and packages for CentOS 6 and 7, Debian Jessie and Stretch, Ubuntu Bionic, Trusty, Xenial and Cosmic are available from repo.powerdns.com.

Please send us all feedback and issues you might have via the mailing list, or in case of a bug, via GitHub.