This is the second in a series of three blog posts we are publishing about recent innovative developments with respect to our DNS proxy and load balancer PowerDNS DNSdist. The previous blog post explained the overall progress of DNSdist with respect to supported DNS encryption mechanisms and implementation use cases.
This post is about the new AF_XDP feature in DNSdist, which can help improve DNSdist performance when handling DNS queries received using the User Datagram Protocol (UDP) - which is a majority of queries today. It also improves performance of UDP queries between DNSdist and backends.
XDP (Xpress Data Path) is a framework for Berkeley Packet Filter (BPF) that enables high performance programmable packet processing in the Linux kernel. AF_XDP, available since the Linux 4.18 kernel, is an address family for Linux sockets that uses XDP to provide highly optimized packet processing to user-space applications. AF_XDP differs from technologies such as the Data Plane Development Kit (DPDK), which are entirely user-space based, because the features are provided by the kernel; however, it can provide similar levels of performance. Because it is implemented in the kernel, XDP works well with other kernel mechanisms such as namespaces, which is important for containerized solutions such as PowerDNS Cloud Control. For a good overview of AF_XDP, see https://www.kernel.org/doc/html/v4.18/networking/af_xdp.html, which also includes links to materials on the underlying BPF and XDP technologies.
So how does XDP provide such a big performance benefit? Well instead of packets transiting the entire kernel TCP/IP networking stack, which involves lots of copying between kernel and user space, the kernel can now pass raw received packets directly to DNSdist using shared in-memory ring buffers, and DNSdist can do the same when sending raw outgoing packets to the kernel. The kernel also requires a BPF program that selects which incoming packets should be sent to DNSdist, otherwise the AF_XDP socket would not receive any traffic. As mentioned, the main performance benefit comes from the lack of copying of packets, which relies on network interface card (NIC) drivers supporting XDP zero-copy mode. Thankfully, the majority of NIC drivers now support this feature, including for example the Intel IGC Gigabit driver and the Mellanox mlx5 driver.
The main benefit of using AF_XDP of course is performance. So what kind of performance improvement can you expect? We wanted to specifically test the benefits of AF_XDP, eliminating any other factors such as backend performance or cache latency. To that end, we devised a performance test where DNSdist was configured to immediately respond to incoming answers with REFUSED. This would allow us to compare the queries per second and CPU usage achieved on the same hardware with and without XDP. The results are summarized in the following graphs:
As can be seen, the results are quite impressive: The first two graphs show that without AF_XDP DNSdist handled ~1 million QPS; with AF_XDP enabled DNSdist handled ~2.5 million QPS. The third graph shows total CPU usage with AF_XDP enabled was ~25% lower than without AF_XDP.
To summarise, AF_XDP support enables us to process approximately 2.5 times the network packets than previously, with a lower CPU usage. Of course this doesn’t translate directly into real-world performance, because DNSdist typically does more than just process packets. However, packet processing is a significant overhead, particularly in scenarios where caching is not enabled or cache-hit-ratios are very low. Even when cache-hit-rate is high, the cost of packet processing is still a fairly dominant factor in performance, because cache hits are typically very efficient. Overall we expect AF_XDP to give a significant boost to performance for those customers who choose to enable it.
As described above, there is some additional configuration required for DNSdist to use AF_XDP; for more details on this see the DNSdist documentation page for the feature, which also includes more details of our performance tests and results.
Please reach out to us if you would like to discuss how DNSdist and AF_XDP support can enhance your DNS installation.
