Things I wish I'd known about nsupdate and dynamic DNS updates
Why Dynamic DNS updates?
The network at Async has multiple redundant upstream connections, and one of them is a domestic-grade cable link at 120Mbps. And like probably every other domestic cable provider, Virtua gives out dynamically allocated IPs via DHCP: our firewall (somewhat clunkily) bridges through a Humax HGR100R-L2 modem to get its IP address with dhclient. We don't really run any services on the interface that listens on that dynamic link, but when the other links are down it's a life-saver to be able to ssh into it. Except when you don't know what IP address it got!
Live DNS updating is more generally useful, of course: you can use it to automate your zone file maintenance, which in our case comes in handy as we roll out the stoq.link service to more users of our management solution, Stoq. But anyway, we cut our fingers on a few rough corners of nsupdate and BIND9, and I wanted to share what we learned as part of it.
1. nsupdate is simple
I had held off doing this because I expected dynamic DNS updating, the topic of RFC 2136, to be really complicated, but it turns out that using nsupdate is trivial: once authenticated, a trivial 4 commands add a record:
server 192.168.1.4
zone dynamic.foo.com
update create bar 3600 A 10.1.1.65
send
That's all you need to add an A record for bar.dynamic.foo.com, and the change is applied immediately. And with BIND9, updates to the secondaries are kicked off automatically.
In fact, if the authentication and server-side setup had been done properly, this would have taken a few minutes to set up. It didn't, and you get this blog post in return.
2. BIND9 doesn't require keys to be configured for rndc on localhost to work
In order to avoid having to generate a new key just to try nsupdate out, I initially tried to reuse a key I had generated for rndc. And that would have worked, had our BIND9 configuration files actually included it! Instead, nsupdate failed:
; TSIG error with server: tsig indicates error
update failed: NOTAUTH(BADSIG)
The server log confirmed the error:
named[380]: client 192.168.1.5#10903: request has invalid signature: TSIG rndc-key: tsig verify failure (BADSIG)
It turns out rndc was working because BIND9 -- when no controls statement is present in its configuration file -- implicitly configures a control channel which enables loopback connections, and which by default uses/etc/bind/rndc.key as its key. But an nsupdate request that uses that key will definitely not work until the key is properly configured through an include:
include "/etc/bind/rndc.key";
3. Authentication is actually configured per-zone
Even if you are including the key properly, nothing will work until you indicate the zone can be updated with that key. That's simple enough:
zone "dyn.foo.com" {
type master;
file "/var/lib/bind/master.dyn";
allow-update { key foo-key; };
};
There are finer-grained access control mechanisms in BIND9 with the update-policy option. and And there's a weird IP-based ACL that seems plain wrong and is discouraged in BIND9; how is it a good idea for a UDP-based service to restrict access through a source address?
4. apparmor doesn't like it when BIND tries to write to /etc/bind
Once I got authentication working, nsupdate was able to connect and send the request, but ended with a failure in my syslog:
updating zone 'master.dyn/IN': error: journal open failed: unexpected error
The kernel log had a hint as to why:
foo kernel: type=1400 audit(1337027369.590:40): apparmor="DENIED" operation="mknod" parent=1 profile="/usr/sbin/named" name="/etc/bind/master.dyn.jnl" pid=6024 comm="named" requested_mask="c" denied_mask="c" fsuid=105 ouid=105
Now originally my configuration file had specified the zone's file option just like any other regular zone:
file "master.dyn";
That places the master.dyn file in /etc/bind. It turns out that Ubuntu's apparmor is configured to only allow reads to BIND's configuration directory, and dynamic updates require journal files to be written.
To solve this, there is an obvious hack you can do, which is to relax the apparmor constraints. But you shouldn't, as you'll see next.
5. And /etc/bind isn't writeable by the bind user, anyway
Even if you do relax the apparmor restrictions for /etc/bind, you will need to allow the bind user to create files in it, which can be done with something like chgrp bind /etc/bind && chmod g+w /etc/bind to set it group-writeable. But this is the final hint that apparmor was right: we shouldn't be letting bind write into /etc anyway.
It turns out that the right solution is to put the file in the place designated for dynamically updated zones: /var/lib/bind/. That's what the config stanza in the beginning of this blog does, and it's what you should do too:
file "/var/lib/bind/master.dyn";
Note that BIND also has a journal option which allows you to put the journal alone in a separate directory. But since the main zone file gets rewritten by bind, as you'll see below, I don't see how that could fix things, even if some people suggest using it.
6. Once you give the zone to nsupdate, it's no longer yours
I had missed this subtlety initially: a zone managed dynamically should not have its zone file edited directly. That means that you can't really have a zone where both nsupdate-added records and static ones coexist.
The reason for this, in hindsight, is pretty obvious: BIND manages the zone for you, which includes updating the serial (though not without caveats -- see below), and writing changes out to a journal file with a .jnl extension. That journal is used to regularly update the main zone file. Now if you modify the original file, what does the journal apply against? The documentation goes into the update process in more detail.
Yes, you can use rndc freeze to force an update to the zone file, but it will be unordered and you'll lose any include statements. You're better off not relying on editing the zone file manually.
I could also not get BIND to auto-update the serial number with a zone file that had a date-based serial (i.e. 2014112802). I would add a record with nsupdate, and dig would confirm the record was added, but the serial remained unchanged, and consequently our secondaries never picked up the change. We ended up giving up on this, faced with the realization that we were going to have to create a new zone anyway.
We had originally planned on adding a single record to the async.com.br domain, but for the reasons above used a new zone (with a starting serial of 1) and a dyn.async.com.br subdomain.
7. dhclient exit hooks are great, but they race with BIND startup at boot time
The process we use for propagating changes to the dynamic IP is simple: we have a script in /etc/dhcp/dhclient-exit-hooks.d that runs when the lease is renewed. That's how our iptables config is updated, for example. Unfortunately, at boot time, because both dhclient and named are running on the same machine, this doesn't work: the interface comes up before BIND's named is running, and when the hook triggers nsupdate has nobody to talk to. We ended up hacking around this with an rc.local update, but there has to be a better solution.
Know of one? Tell me about it.