Enabling continuous fuzzing of PowerDNS products
We are very happy to announce that PowerDNS recently joined the OSS-Fuzz initiative, enabling continuous fuzzing for critical parts of our products. We are taking this opportunity to share some of our existing security processes, because we are proud of the progress we have made.
For quite some time now, we have been working on making our products safer by identifying and fixing existing security weaknesses, and deploying processes to prevent new ones from being introduced while reducing the impact should one slip through the cracks.
These efforts have led to the discovery of quite a few security issues in the past years, some of them present in the code base for several years. Nowadays, we are finding these issues because we are actively looking for them, using manual review, automated code analysis tools and fuzzing. But more on that later.
Very often, the line between a bug and a possible security issue is very thin, and vendors have been known to try to downplay the possible impact of an issue whenever possible. At PowerDNS, we choose instead to be forthcoming and open about the possible impact of the bugs that are discovered in our products. For one, there is no point in trying to downplay or obfuscate the real impact, as anyone can read the source code of our open-source products and decide that for themselves. But most of all, we believe our users should be as informed as possible regarding the severity of the issues fixed in a release, allowing them to decide for themselves how quickly they need to upgrade.
Now, let’s review the processes we have in place to prevent and find issues in our code base.
Secure development practices
In 4.0, we switched to a new version of the C++ language, C++11, which introduced several new features that allow us to avoid common pitfalls often associated with C and C++: for example, when they are properly used, smart pointers kill whole classes of bugs and security issues, including memory leaks and use-after-free.
At the same time, we also regularly stop using features that have proven brittle. For example, we recently removed our last usage of variable-length arrays (VLA), because we noticed it was very easy to shoot yourself in the foot and smash the stack by not properly checking the maximum size of such an array. We now rely on a specific compiler flag to prevent us from ever introducing them back by mistake, -Werror=vla.
Peer review and continuous testing
Every change to the code base goes through the same process:
- a pull request is opened ;
- our continuous testing tools make sure that it builds properly, and passes both our unit and functional tests ;
- the change is reviewed by at least one person from our team ;
- finally, it is merged into the appropriate branch.
To increase the likelihood of catching bugs during the continuous testing phase, our products are built with various sanitizing options enabled whenever applicable:
- Address Sanitizer: catching out-of-bounds accesses, use-after-free, double-free, invalid free and memory leaks ;
- Undefined Behavior Sanitizer: catching the use of misaligned or null pointers, signed integer overflow, and conversion to, from, or between floating-point types which would overflow the destination.
We also use several static analysis tools to detect mistakes that eluded both the original author and the reviewer:
- our code base is automatically scanned by Coverity ;
- and we regularly run clang’s static analyser and cppcheck.
Independent audit of the code base
Last year, we contracted a third-party security company, Nixu, to perform an independent audit of our code base, mainly focused on dnsdist and the Recursor but also covering a large part of the Authoritative server. Their assessment included a manual code review, supported by threat analysis using the STRIDE method, and the use of various static analysis tools, fuzzing and manual testing with ad-hoc tools built for that project.
A total of 11 findings were reported to us. A few of them could have allowed an authenticated user to escalate privileges, using defects in our API and web management interface. The remaining issues were denial of service, most of them in the DNSSEC handling code in the Recursor.
All of these issues have been fixed shortly after the report was handed to us, and we published the relevant security advisories about them.
Bug Bounty Program
We believe that working with skilled security researchers across the globe is crucial in identifying weaknesses in any technology. Therefore, in addition to our website advertising clear and simple ways of securely report any concern to us, we have also been running a public bug bounty program on the Hacker One platform for more than two years now: https://hackerone.com/powerdns
This program covers all our open-source products and every type of vulnerability, allowing everyone finding a security issue in our products to do the right thing by reporting it to us, while still getting financially rewarded.
To further restrict the impact of a security issue in one of our products, we enable several hardening measures by default.
Some of these try to detect pathological behavior as early as possible, make it very hard to exploit it and almost always turn it into a crash, which is mitigated by supervisors like systemd:
- buffer overflow, format string detection and prevention (stack-protector, FORTIFY_SOURCE=2) ;
- full address space layout randomization via Position-Independent Executable (PIE) ;
- fully read-only relocations (-Wl,-z,relro,-z,now).
Other measures restrict the privileges of the process, thus limiting what an attacker could do even if it could take full control of the process-execution path:
- filesystem protection (chroot; systemd’s PrivateTmp, PrivateDevices, ProtectSystem and ProtectHome) ;
- privileges dropping (uid/gid switching; systemd’s CapabilityBoundingSet, NoNewPrivileges and RestrictAddressFamilies).
We used to run fuzzing sessions manually using AFL and Address Sanitizer after every important change to one of our parsers, since they deal with user-provided input and are therefore the more exposed part of our code. While we discovered a few issues that way, catching them before they could make it into a release, we were not very thrilled with this method, because it required human interaction.
That’s why we are so happy to have joined the OSS-fuzz project, enabling continuous fuzzing of our parsers. Continuous fuzzing means that every change to the code will get automatically tested and that, should an issue be found, a ticket will be automatically generated and we will be notified.
The OSS-fuzz infrastructure will not only run our targets with AFL as we used to do manually, it will also use the other well-known fuzzing engine, libFuzzer. Both of them will be run with Address and Undefined Behavior sanitizers enabled, catching issues that could otherwise be completely undetected.
Security issue handling
Now that we have explained how we prevent, catch and mitigate security issues, a few words about the way we handle them when they occur in a supported version.
As soon as a security concern has been reported to us, be it via our website, in a private e-mail or our program on HackerOne, we immediately acknowledge the reception of the message and start working on reproducing and assessing the severity of the issue. We then begin working on a patch for all supported versions, and communicate a timeline and a draft of the security advisory to the reporter when applicable.
Every security issue is then communicated to our customers in advance, at least a week before the public release, along with patched packages. We also notify various vendors via the distros security mailing-list. After the release, every security issue is the subject of a public security announcement, and these are publicly available on our website: