Efficient & optional filtering of domains in Recursor 4.0.0

Jan 19, 2016

As we gear up for the release of PowerDNS Recursor 4.0.0, we are doing a series of posts describing new cool features which you can try out today. Many deployments are already running with 4.x alphas or snapshots, and now is a great time to familiarize yourself with the new and upcoming possibilities.

In this post, we’ll explain how to efficiently use the PowerDNS Recursor to optionally block certain domains for some or all of your users. This could be to stop users being tracked, to block advertisements or to protect against malware. The simple scripts below scale to millions of domain names and millions of users, all with acceptable startup times (seconds).

It starts with getting a good data set, and the good people from Mozilla have us covered here. The Mozilla Focus project through their partner Disconnect have a list of trackers and advertising servers which we can retrieve like this:



This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters

 




 

$ git clone https://github.com/mozilla/focus.git
$ cd focus
$ ./checkout.sh
$ cd Lists

 

view raw

retrieve-lists.sh

hosted with ❤️ by GitHub

 

(Note, the actual Focus project performs blocking more cleverly than can be achieved purely with DNS. Focus also takes into account the URL of the page containing the advertisement).

To make these lists usable by PowerDNS, we need a little bit of conversion using the most excellent ‘jq’ tool:



This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters

 



 

(
echo 'return{'
for a in $(jq '.[].trigger["url-filter"]' disconnect-advertising.json |
cut -f3 -d? | sed 's:\\\\.:.:g' | sed s:\"::)
do
echo \"$a\",
done
echo '}'
) > blocklist.lua

 

view raw

convert.sh

hosted with ❤️ by GitHub

 

This delivers a file which looks like ‘return{“a.com”, “b.com”, “c.com”}’, which is how to rapidly import data into Lua. To also block trackers, rerun this script for ‘disconnect-analytics.json’ and store as ‘trackers.lua’.

Next up, we use a small script ‘adblock.lua’ to tell PowerDNS to use this list to deny the existence of the filtered domains:



This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters

 




 

adservers=newDS()
adservers:add(dofile("blocklist.lua"))
function preresolve(dq)
if(not adservers:check(dq.qname) or (dq.qtype ~= pdns.A and dq.qtype ~= pdns.AAAA)) then
return false
end
dq:addRecord(pdns.SOA,
"fake."..dq.qname:toString().." fake."..dq.qname:toString().." 1 7200 900 1209600 86400",
2)
return true
end

 

view raw

adblock.lua

hosted with ❤️ by GitHub

 

Finally, put ‘lua-dns-script=adblock.lua’ in recursor.conf to tell PowerDNS to load this list. And for the basic functionality, we are done! A few words about the script. With ‘newDS()’ we defined a new Domain Set.  In the second line, we load the list of advertisement related domain names.

Next we define the function preresolve() which gets called by PowerDNS to determine what to do with a domain. If we see that a query name is not part of one of the advertisement domains, or the query is not for an IP(v6) address, we return false and the normal resolution process continues.

Otherwise if it is a domain to be blocked, we insert a SOA record that says “this domain name exists, but the type you queried for doesn’t”.

Making it optional

Not everyone will want their advertisements filtered like this. And some people just want to be tracked online. To make this optional, first we make a list of IP addresses that want their DNS to be filtered:



This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters

 



 

(echo return{;
for z in {1..10}
do for a in {1..255}
do for b in {1..255}
do echo \"10.$z.$a.$b\",
done ; done; done
echo } ) > filtercustomers.lua

 

view raw

make-650k-ips.sh

hosted with ❤️ by GitHub

 

This delivers a file with around 650000 IP addresses, which we can import in under a second in our Lua script like this:



This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters

 



 

adservers=newDS()
adservers:add(dofile("blocklist.lua"))
optionally: adservers:add(dofile("trackers.lua"))
filterset=newCAS()
filterset:add(dofile("filtercustomers.lua"))
function preresolve(dq)
if(not adservers:check(dq.qname) or (dq.qtype ~= pdns.A and dq.qtype ~= pdns.AAAA)) then
return false
end
dq.variable=true
if(not filterset:check(dq.remoteaddr)) then
return false
end
dq:addRecord(pdns.SOA,
"fake."..dq.qname:toString().." fake."..dq.qname:toString().." 1 7200 900 1209600 86400",
2)
return true
end

 

view raw

adblock-optional.lua

hosted with ❤️ by GitHub

 

New line number 13 informs PowerDNS that this domain name will have variable responses depending on who asks. The following new lines check if this user wants to be filtered or not.

This script can easily be expanded to have different lists for users that want advertising or perhaps only trackers to be blocked.

Finally, to reload the script with new data, issue ‘rec_control reload-lua-script’.

Take away

Using the elementary scripts shown above, we can optionally provide very large userbases with optional advertisement or tracker filtering. To learn more about the PowerDNS Recursor dynamic abilities, head to the documentation, where you can also find how to retrieve user and domain status in real time from external servers.

About the author

Bert Hubert

Bert Hubert

Principal, PowerDNS

Categories

Related Articles

PowerDNS Recursor 5.1.0 Released

We are proud to announce the release of PowerDNS Recursor 5.1.0!

Otto Moerbeek Jul 10, 2024

PowerDNS Recursor 4.9.7 and 5.0.7 Released

Today we have released PowerDNS Recursor 4.9.7 and 5.0.7. These releases are maintenance releases that fix a few bugs. The...

Otto Moerbeek Jul 3, 2024

PowerDNS Recursor 5.1.0-rc1 Released

We are proud to announce the first release candidate of PowerDNS Recursor 5.1.0!

Otto Moerbeek Jun 25, 2024

PowerDNS Recursor 5.1.0-beta1 Released

We are proud to announce the first beta release of PowerDNS Recursor 5.1.0!

Otto Moerbeek Jun 6, 2024