We get frequent reports from users/customers about various DNS-related attacks they are facing on either their authoritatives or recursors. This post focuses on one kind of attack that involves both.
The attack works like this: given a target domain of example.com, the attacker takes his botnet, and has it fire off high amounts of $RANDOM.example.com queries. These queries go from the infected hosts to their recursors (i.e. normal ISP recursors). The recursors then need to go out to the auths, after all they don’t have any random string in cache.
When this attack starts, there is no packet count amplification – bot sends query to recursor, recursor sends to auth, answer flows back, done. However, if enough of this happens, one or more of the auths for example.com may get overloaded, and start responding more slowly, or not respond at all. The recursors will then either retry or move on to other auths, spreading the attack in the most effective and destructive way over the auths.
These attacks are painful, especially for authoritatives backed by (SQL) databases, like many PowerDNS users are running. Legitimate traffic for existing names gets cached very well inside pdns_server, but even if you put a wildcard in your database, these random queries will cause an SQL query, and those are costly.
Because SQL and random names are a bad fit, we get requests for being able to combine the bindbackend and an SQL backend in one pdns_server process. This works, but does not have the desired effect of offloading the SQL – we query both before sending out a response. So, something else needs to happen. While pondering that question this week, a few ideas came up:
- use IPTables u32 to match queries for the victim domain, and redirect them (I understand this can be done without generating a lot of state)
- teach dnsdist to pick backends based on domain name
- somehow get the recursors to redirect their traffic
I did not try ideas 1 and 2; I trust they will work in practice, and will effectively remove load from the SQL backends, but they still involve handling the whole malicious query load on the same server pipe. Luckily, it turns out idea 3 is feasible.
The idea behind 3 is to convince a recursor it is talking to the wrong machines, by virtue of sending it a new NSset in the AUTHORITY section of a response. Some authoritative servers will include the NSset from the zone in every response, but PowerDNS does not do this – so we need another trick.
Some time ago we added an experimental, internal-use-only feature to the Authoritative Server called lua-prequery, to be used specifically for our Recursor regression tests. While never designed for production usage, we can abuse it to make idea 3 work.
require 'posix'
function endswith(s, send)
return #s >= #send and s:find(send, #s-#send+1, true) and true or false
end
function prequery ( dnspacket )
qname, qtype = dnspacket:getQuestion()
print(os.time(), qname,qtype)
if endswith(qname, '.example.com') and posix.stat('/etc/powerdns/dropit')
then
dnspacket:setRcode(pdns.NXDOMAIN)
ret = {}
ret[1] = {qname='example.com', qtype=pdns.NS, content="ns-nosql.example.com", place=2, ttl=30}
ret[2] = {qname='example.com', qtype=pdns.NS, content="ns-nosql2.example.com", place=2, ttl=30}
dnspacket:addRecords(ret)
return true
end
return false
end
(A careful reader noted that the stat() call, while cached, may not be the most efficient way to enable/disable this thing. Caveat emptor.)
This piece of code, combined with a reference to it in pdns.conf (‘lua-prequery-script=/etc/powerdns/prequery.lua‘), will cause pdns_server to send authoritative NXDOMAINs for any query ending in example.com, and include a new NSset, suggesting the recursor go look ‘over there’.
In our testing, BIND simply ignored the new NSset (we did not investigate why). PowerDNS Recursor believes what you tell it, and will stick to it until the TTL (30 seconds in this example) runs out. Unbound will also believe you, but if none of the machines you redirect it to actually work, it will come back. So, in general we recommend you point the traffic to a set of machines that can give valid replies.
In a lab setting, we found that with both Unbound and PowerDNS Recursor, this approach can move -all- traffic from your normal nameservers to the offload hosts, except for a few packets every TTL seconds. Depending on attack rate and TTL, this easily means offloading >99.9% of traffic, assuming no BIND is involved. In the real world, where some ISPs do use BIND for recursion, you won’t hit 99% or 90% but this approach may still help a lot.
We have not tried this on a real world attack, yet.
What’s next?
If you are under such an attack, and would like to give this a shot, please contact us, we’d love to try this on a real attack!
If you feel like toying around with this (I really want to find out how to make BIND cooperate, but I ran out of time), please get in touch (IRC preferred), I want to talk to you 😊
