One of the things we do at IDX is help organizations onboard Virtual Network Functions (VNFs). VNF Onboarding is an involved and complex process with many steps. One of the activities in our onboarding process involves the performance testing, verification, and dimensioning of a VNF. In this particular example we discuss testing a Domain Name System (DNS) VNF; more specifically we are baselining open source DNS servers and tools to use as reference point.
Building high performing DNS systems typically involves some kind of load distribution architecture such as load balancers or, more commonly in the case of DNS, Anycast. While we have previously deployed Anycast solutions at IDX, in this case we decided to work on a load balancer based solution.
When we were performing a short review of load balancers capable of working well with UDP based DNS we were led to a cool tool: dnsdist.
dnsdist
dnsdist is a load balancer specifically designed for DNS. It is fast, Denial of Service (DOS) aware, and, most interestingly, embeds the Lua programming language into its configuration.
Some of dnsdist’s features:
- REST API
- Dynamically add rules and actions without reloading/restarting
- Ability to write custom load balancing schemes
- eBPF support
- Filter millions of queries per second
- Tee/Copy/Tap query streams
With dnsdist we can do all kinds of DNS traffic shaping. Here’s a very simple example of sending queries for www.example.com to one server and everything else to another. The below configuration file is Lua.
newServer({address=”10.101.0.146:53″, pool=”www”})
newServer({address=”10.101.0.149:53″ })
addAction({‘www.example.com’}, PoolAction(“www”))
setServerPolicy(firstAvailable)
There is a lot of flexibility in dnsdist and we are only scratching the surface.
It is also high performing. Below a file called www.txt has been prepared, where half the entries are for www.example.com and the other half at $i.example.com.
$ head www.txt
www.example.com A
1.example.com A
www.example.com A
2.example.com A
www.example.com A
3.example.com A
www.example.com A
4.example.com A
www.example.com A
5.example.com A
Running dnsperf on that file against the dnsdist server, we see what kind of performance is possible (mostly limited by packet processing outside of dnsdist). As a note, both of the DNS servers behind dnsdist are authoritative for example.com; we are not doing any resolving.
$# send as many requests as possible in 60 seconds using the www.txt file
$ dnsperf -s dnsdist-server -d www.txt -l 60
DNS Performance Testing Tool
Nominum Version 2.1.0.0
[Status] Command line: dnsperf -s dnsdist-server -d www.txt -l 60
[Status] Sending queries (to 10.101.0.141)
[Status] Started at: Fri Feb 2 16:52:57 2018
[Status] Stopping after 60.000000 seconds
[Status] Testing complete (time limit)
Statistics:
Queries sent: 3203951
Queries completed: 3203951 (100.00%)
Queries lost: 0 (0.00%)
Response codes: NOERROR 3203951 (100.00%)
Average packet size: request 34, response 50
Run time (s): 60.001306
Queries per second: 53398.021036
Average Latency (s): 0.001567 (min 0.000142, max 0.009222)
Latency StdDev (s): 0.000460
As can be seen from the output, we are doing about 53k queries per second with no errors and an average latency of about 1.5ms. Not bad for a simple non-tuned test.
Load Distribution
Because DNS packets tend to be small, using a load balancer to front DNS servers will limit performance to whatever the load balancer is capable of. If the load balancer is capable of supporting a higher number of packets per second (PPS), then the DNS system will perform better. In our limited testing, small Linux virtual machines running dnsdist topped out at about 50k PPS (this is without any tuning whatsoever). Large DNS systems will certainly require the ability to process a higher number of packets than our single Linux host.
This is why Anycast makes so much sense for UDP based DNS as the load balancer is taken out of the path, and thus there is no middlebox to limit PPS and thus performance limitation typically moves up to the number of ECMP routes a network is capable of supporting, network design, number of DNS servers, etc.
However, fronting DNS servers with something like dnsdist is not necessarily directly related to performance as much as it is about flexibility, especially around defending from DOS attacks and shaping DNS traffic, as well as health checking. In fact, instead of being a DNS load balancer, dnsdist is really a “DNS delivery controller” and a cool tool.