Efficient & optional filtering of domains in Recursor 4.0.0
As we gear up for the release of PowerDNS Recursor 4.0.0, we are doing a series of posts describing new cool features which you can try out today. Many deployments are already running with 4.x alphas or snapshots, and now is a great time to familiarize yourself with the new and upcoming possibilities.
In this post, we’ll explain how to efficiently use the PowerDNS Recursor to optionally block certain domains for some or all of your users. This could be to stop users being tracked, to block advertisements or to protect against malware. The simple scripts below scale to millions of domain names and millions of users, all with acceptable startup times (seconds).
It starts with getting a good data set, and the good people from Mozilla have us covered here. The Mozilla Focus project through their partner Disconnect have a list of trackers and advertising servers which we can retrieve like this:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ git clone https://github.com/mozilla/focus.git | |
| $ cd focus | |
| $ ./checkout.sh | |
| $ cd Lists |
(Note, the actual Focus project performs blocking more cleverly than can be achieved purely with DNS. Focus also takes into account the URL of the page containing the advertisement).
To make these lists usable by PowerDNS, we need a little bit of conversion using the most excellent ‘jq’ tool:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ( | |
| echo 'return{' | |
| for a in $(jq '.[].trigger["url-filter"]' disconnect-advertising.json | | |
| cut -f3 -d? | sed 's:\\\\.:.:g' | sed s:\"::) | |
| do | |
| echo \"$a\", | |
| done | |
| echo '}' | |
| ) > blocklist.lua |
This delivers a file which looks like ‘return{“a.com”, “b.com”, “c.com”}’, which is how to rapidly import data into Lua. To also block trackers, rerun this script for ‘disconnect-analytics.json’ and store as ‘trackers.lua’.
Next up, we use a small script ‘adblock.lua’ to tell PowerDNS to use this list to deny the existence of the filtered domains:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| adservers=newDS() | |
| adservers:add(dofile("blocklist.lua")) | |
| function preresolve(dq) | |
| if(not adservers:check(dq.qname) or (dq.qtype ~= pdns.A and dq.qtype ~= pdns.AAAA)) then | |
| return false | |
| end | |
| dq:addRecord(pdns.SOA, | |
| "fake."..dq.qname:toString().." fake."..dq.qname:toString().." 1 7200 900 1209600 86400", | |
| 2) | |
| return true | |
| end |
Finally, put ‘lua-dns-script=adblock.lua’ in recursor.conf to tell PowerDNS to load this list. And for the basic functionality, we are done! A few words about the script. With ‘newDS()’ we defined a new Domain Set. In the second line, we load the list of advertisement related domain names.
Next we define the function preresolve() which gets called by PowerDNS to determine what to do with a domain. If we see that a query name is not part of one of the advertisement domains, or the query is not for an IP(v6) address, we return false and the normal resolution process continues.
Otherwise if it is a domain to be blocked, we insert a SOA record that says “this domain name exists, but the type you queried for doesn’t”.
Making it optional
Not everyone will want their advertisements filtered like this. And some people just want to be tracked online. To make this optional, first we make a list of IP addresses that want their DNS to be filtered:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (echo return{; | |
| for z in {1..10} | |
| do for a in {1..255} | |
| do for b in {1..255} | |
| do echo \"10.$z.$a.$b\", | |
| done ; done; done | |
| echo } ) > filtercustomers.lua |
This delivers a file with around 650000 IP addresses, which we can import in under a second in our Lua script like this:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| adservers=newDS() | |
| adservers:add(dofile("blocklist.lua")) | |
| — optionally: adservers:add(dofile("trackers.lua")) | |
| filterset=newCAS() | |
| filterset:add(dofile("filtercustomers.lua")) | |
| function preresolve(dq) | |
| if(not adservers:check(dq.qname) or (dq.qtype ~= pdns.A and dq.qtype ~= pdns.AAAA)) then | |
| return false | |
| end | |
| dq.variable=true | |
| if(not filterset:check(dq.remoteaddr)) then | |
| return false | |
| end | |
| dq:addRecord(pdns.SOA, | |
| "fake."..dq.qname:toString().." fake."..dq.qname:toString().." 1 7200 900 1209600 86400", | |
| 2) | |
| return true | |
| end |
New line number 13 informs PowerDNS that this domain name will have variable responses depending on who asks. The following new lines check if this user wants to be filtered or not.
This script can easily be expanded to have different lists for users that want advertising or perhaps only trackers to be blocked.
Finally, to reload the script with new data, issue ‘rec_control reload-lua-script’.
Take away
Using the elementary scripts shown above, we can optionally provide very large userbases with optional advertisement or tracker filtering. To learn more about the PowerDNS Recursor dynamic abilities, head to the documentation, where you can also find how to retrieve user and domain status in real time from external servers.
PowerDNS Blog 
Could could a sinkhole be similarly created using Authoritative 3.x?
Yes, but it would be more work. The Lua API in 3.x is documented on the website, and you can find out there how to do it.
In order for the addblock-optional script to work on a recursor behind dnsdist do you have to instruct lua somehow to parse the client subnet out of the edns directive ?
Hi Andy,
The blog is not a great place to discuss the actual code. Can you join our dnsdist mailing list as described on https://www.powerdns.com/mailing-lists.html ? Thanks! We’ll gladly help you there.
Hi Bert,
Running an Alpha 4.x Recursor behind dnsdist, does the “dq.remoteaddr” directive in LUA detect the source IP of the dnsdist server as opposed to that of the client generating the DNS Query? dnsdist is sending the client subnet via EDNS so I am afraid its not being detected properly. Rules work … tested against localhost but remote query is not detected in the filter.
The blog is not a great place to discuss the actual code. Can you join our dnsdist mailing list as described on https://www.powerdns.com/mailing-lists.html ? Thanks! We’ll gladly help you there.
Thank you Bert. My post is up
http://powerdns.13854.n7.nabble.com/Domain-filtering-in-Recursor-4-0-behind-dnsdist-td12152.html
It appears that it hasn’t made it to the list: “This post has NOT been accepted by the mailing list yet.”. Perhaps “nabble” does something wrong it is not our service. Can you post it yourself?
Odd … I was not receiving the subscription confirmation from mailman.powerdns.com. I had to create a new login but its working now.
http://powerdns.13854.n7.nabble.com/Unable-to-filter-Domains-Recursor-4-x-behind-dnsdist-td12173.html
Odd … I was not receiving the subscription confirmation from mailman.powerdns.com. I had to create a new login but its working now.
http://powerdns.13854.n7.nabble.com/Unable-to-filter-Domains-Recursor-4-x-behind-dnsdist-td12173.html
The same things can be done using wildcards with dnsmasq or rpz with BIND. How efficient (performance-wise) does this compare to the two other options I’ve mentioned? Performance is particularly important to me in the setup I have going on. Thanks.
Oops,
I was hoping something simpler for blocking.
A simple entry with 127.0.0.1 in records tables or something like that, in pdns backend-DB.
Thanks Bert, great reason to justify a pdns setup at home 😉
This script never worked for me… have other?
root@pdns03:/etc/powerdns# lua adblock.lua
lua: adblock.lua:1: attempt to call global `newDS’ (a nil value)
stack traceback:
adblock.lua:1: in main chunk
[C]: ?
Whats wrong?
(for future Googlers)
I got the same error. Seems the version of PowerDNS in the Debian repos was a bit old and not compiled with Lua, so I opted to use the official repos provided by PowerDNS and now this (and other Lua extensions) work great.
Thanks for this post, i refine the blocklist generation a little bit, so that it’s just a single bash script. I also added a check to adblock.lua, in case the blocklist doesn’t exist.
All files can be found here: https://git.faked.org/jan/powerdns-adblock/