Skip to content

Set up dynamic DNS for the lab zone

This playbook runs BIND9 as authoritative DNS for the lab.riupie.com zone plus recursive resolution for the lab, with TSIG-secured dynamic updates so External-DNS can register records from Kubernetes.

Use it to set up (or rebuild) the lab DNS server on general01, before configuring External-DNS in Kubernetes (step 3 produces its TSIG key).

Item Details
Stack BIND9 9.20 (internetsystemsconsortium/bind9:9.20)
Docker Compose
Target general01 (192.168.10.9, Debian 12)
Run from Fedora 44 KVM host
Env lab

The configuration lives in the repo under addons/bind9/; the files shown below are embedded from there, so edit the repo, not this page.

Prerequisites

  • general01 reachable from the host: ssh cloud@192.168.10.9 hostname
  • Docker Engine with the Compose plugin on general01: docker --version && docker compose version
  • Port 53 free on general01: sudo ss -lntup 'sport = :53' prints nothing (see Troubleshooting if not)
  • dig and nsupdate on the Fedora host: sudo dnf install -y bind-utils
  • A checkout of this repo on the host; commands marked On the host run from its root

Steps

1. Prepare the directory on general01

On the host:

ssh cloud@192.168.10.9 'sudo install -d /opt/bind9/config/keys /opt/bind9/zones /opt/bind9/cache'

2. Copy the configuration

On the host:

stage="$(mktemp -d)"
cp -a addons/bind9/. "$stage"/
rm -rf "$stage/config/keys" "$stage/README.md"   # keys are generated on general01 (step 3)
# existing server (rebuild)? also: rm -rf "$stage/zones"

scp -r "$stage" cloud@192.168.10.9:/tmp/bind9-stage
rm -rf "$stage"
ssh cloud@192.168.10.9 'sudo cp -a /tmp/bind9-stage/* /opt/bind9/ && rm -rf /tmp/bind9-stage'

rsync is not installed on a fresh general01, so the files go through a staging directory instead.

Rebuilds

External-DNS writes records into the zone journal (zones/*.jnl) on general01. Copying zones/ onto a live server overwrites zones/lab.riupie.com.zone; on an existing server copy only config/ and docker-compose.yaml (uncomment the rm -rf "$stage/zones" line), repeat step 4, and bump the SOA serial when you edit the zone by hand.

The copied files:

include "/etc/bind/named.conf.options";
include "/etc/bind/named.conf.local";
//The following keys are used for dynamic DNS updates
include "/etc/bind/keys/external-dns.key";
options {
    directory "/var/cache/bind";
    recursion yes;
    allow-query { any; };
    // recursion only for the lab; localnets covers Docker's bridge (dig @localhost on general01)
    allow-recursion { localnets; 192.168.10.0/24; };

    // libvirt dnsmasq on the Fedora KVM host
    forwarders {
        192.168.10.1;
    };

    dnssec-validation auto;

    listen-on { any; };
    listen-on-v6 { none; };
};
logging {
   channel stdout_channel {
       stderr;
       severity info;
       print-category yes;
       print-severity yes;
       print-time yes;
   };
   category default {
       stdout_channel;
   };
   category queries {
       stdout_channel;
   };
   category security {
       stdout_channel;
   };
   category dnssec {
       stdout_channel;
   };
};
zone "lab.riupie.com" {
    type master;
    file "/zones/lab.riupie.com.zone";
    allow-transfer {
        key "externaldns-key";
    };
    update-policy {
        grant externaldns-key zonesub any;
    };
};
$TTL 3600   ; 1 hour
lab.riupie.com.     IN SOA  ns1.lab.riupie.com. admin.lab.riupie.com. (
                2025050307 ; serial
                3600       ; refresh (1 hour)
                1800       ; retry (30 minutes)
                604800     ; expire (1 week)
                86400      ; minimum (1 day)
                )
            NS  ns1.lab.riupie.com.

ns1.lab.riupie.com.     A       192.168.10.9
services:
  bind9:
    image: internetsystemsconsortium/bind9:9.20
    container_name: bind9
    ports:
      - "53:53/udp"
      - "53:53/tcp"
    volumes:
      - ./config:/etc/bind
      - ./zones:/zones
      - ./cache:/var/cache/bind
    restart: unless-stopped
    environment:
      - TZ=UTC
    healthcheck:
      test: ["CMD", "dig", "@127.0.0.1", "lab.riupie.com", "SOA", "+short"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

3. Generate the TSIG key

On general01 (ssh cloud@192.168.10.9). The key is generated inside the BIND image, so general01 needs no BIND packages:

cd /opt/bind9
umask 077   # the key must not be readable by other users while it is being written
docker run --rm --entrypoint tsig-keygen internetsystemsconsortium/bind9:9.20 \
  -a hmac-sha512 externaldns-key | sudo tee config/keys/external-dns.key >/dev/null

Expected content of config/keys/external-dns.key (sudo cat it; it is not world-readable after step 4):

key "externaldns-key" {
    algorithm hmac-sha512;
    secret "<base64>";
};

Key security

This key can rewrite any record in lab.riupie.com. Don’t commit it unencrypted (the repo copy is protected by git-crypt) and don’t paste it into docs or tickets.

4. Set ownership for named

On general01. named runs as uid/gid 53 (bind) inside the container; it must read the config and key, and write the zone journal and its cache:

sudo chown -R 53:53 /opt/bind9/config /opt/bind9/zones /opt/bind9/cache
sudo chmod 640 /opt/bind9/config/keys/external-dns.key

If you skip this, dynamic updates fail with permission denied on lab.riupie.com.zone.jnl.

5. Validate the configuration

On general01:

cd /opt/bind9
docker run --rm -v "$PWD/config:/etc/bind" --entrypoint named-checkconf \
  internetsystemsconsortium/bind9:9.20 /etc/bind/named.conf
docker run --rm -v "$PWD/zones:/zones" --entrypoint named-checkzone \
  internetsystemsconsortium/bind9:9.20 lab.riupie.com /zones/lab.riupie.com.zone

Expected output (named-checkconf prints nothing when the config is valid):

zone lab.riupie.com/IN: loaded serial 2025050307
OK

6. Start BIND9

On general01:

cd /opt/bind9
docker compose up -d
docker compose ps
docker compose logs bind9 | grep -E 'loaded serial|running'

Expected output (STATUS reaches healthy after up to 40 s):

NAME    IMAGE                                  SERVICE   STATUS
bind9   internetsystemsconsortium/bind9:9.20   bind9     Up 1 minute (healthy)
...
zone lab.riupie.com/IN: loaded serial 2025050307
running

7. Hand the TSIG secret to External-DNS

On general01:

sudo awk -F'"' '/secret/ {print $2}' /opt/bind9/config/keys/external-dns.key

Use the printed value as the RFC2136 TSIG secret in the External-DNS configuration, with key name externaldns-key and algorithm hmac-sha512. In the lab it is stored in Infisical as /external-dns/EXTERNAL_DNS_TSIG_KEY and delivered by External Secrets; see Store Secrets with ESO and Infisical.

8. Resolve the lab zone from the Fedora host

On the host. Send only lab.riupie.com queries to general01, and keep every other lookup on the host’s normal resolver. systemd-resolved does this with a per-link routing domain on the lab bridge. The bridge is created by libvirt, not NetworkManager, so a small systemd unit tied to the bridge device reapplies the setting whenever the bridge appears (boot, virsh net-start net-lab).

Find the bridge name (virbr1 on the reference host):

virsh net-info net-lab | awk '/^Bridge/ {print $2}'

Create the unit (replace virbr1 if your bridge differs):

sudo tee /etc/systemd/system/lab-dns@.service >/dev/null <<'EOF'
[Unit]
Description=Route lab.riupie.com DNS to general01 via %i
BindsTo=sys-subsystem-net-devices-%i.device
After=sys-subsystem-net-devices-%i.device systemd-resolved.service
Wants=systemd-resolved.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/resolvectl dns %i 192.168.10.9
ExecStart=/usr/bin/resolvectl domain %i ~lab.riupie.com
ExecStart=/usr/bin/resolvectl default-route %i false
ExecStop=/usr/bin/resolvectl revert %i

[Install]
WantedBy=sys-subsystem-net-devices-%i.device
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now lab-dns@virbr1.service

systemctl enable warns that the unit is added as a dependency to a non-existent unit sys-subsystem-net-devices-virbr1.device. This is expected: device units are created at runtime by udev and have no unit file, but the dependency link is still created and works.

Check it:

resolvectl status virbr1 | grep -E 'Default Route|DNS Servers|DNS Domain'
resolvectl query ns1.lab.riupie.com

Expected output:

     Default Route: no
       DNS Servers: 192.168.10.9
        DNS Domain: ~lab.riupie.com
ns1.lab.riupie.com: 192.168.10.9                -- link: virbr1

Don’t use a global DNS= for this

A drop-in such as /etc/systemd/resolved.conf.d/lab.conf with DNS=192.168.10.9 and Domains=~lab.riupie.com resolves the lab zone, but systemd-resolved also uses a global DNS server as a default route. Every lookup from the host then also goes to general01, and lookups get slow or fail while general01 is down. If you have that file, remove it after enabling the unit:

sudo rm /etc/systemd/resolved.conf.d/lab.conf
sudo systemctl restart systemd-resolved
sudo systemctl restart lab-dns@virbr1.service   # re-apply after the resolved restart

To undo: sudo systemctl disable --now lab-dns@virbr1.service && sudo rm /etc/systemd/system/lab-dns@.service && sudo systemctl daemon-reload.

Verify

1. Authoritative and recursive resolution

On the host:

dig @192.168.10.9 lab.riupie.com SOA +short
dig @192.168.10.9 example.com A +short

Expected output: the SOA record (ns1.lab.riupie.com. admin.lab.riupie.com. <serial> 3600 1800 604800 86400), then one or more IP addresses for example.com.

2. Dynamic update with the TSIG key

On general01 (the key stays on the server):

cd /opt/bind9
docker compose exec -T bind9 nsupdate -k /etc/bind/keys/external-dns.key <<'EOF'
server 127.0.0.1
zone lab.riupie.com.
update add test.lab.riupie.com. 300 A 192.168.10.100
send
EOF
dig @127.0.0.1 test.lab.riupie.com A +short

Expected output: 192.168.10.100. Remove the test record:

docker compose exec -T bind9 nsupdate -k /etc/bind/keys/external-dns.key <<'EOF'
server 127.0.0.1
zone lab.riupie.com.
update delete test.lab.riupie.com. A
send
EOF

Troubleshooting

Symptom Likely cause Fix
docker compose up fails: address already in use on port 53 The systemd-resolved stub listener holds port 53 (sudo ss -lntup 'sport = :53' shows systemd-resolve) Disable the stub listener (below)
nsupdate returns SERVFAIL/REFUSED, log shows permission denied on .jnl zones/ not writable by uid 53 Step 4
nsupdate returns NOTAUTH / tsig verify failure Wrong key file or algorithm mismatch Use the key from step 3; algorithm must be hmac-sha512 on both sides
Recursive queries return REFUSED Client is outside allow-recursion Add the client network to allow-recursion in named.conf.options, redeploy config/ (step 2), docker compose restart
Host can’t resolve *.lab.riupie.com, or every host lookup is slow while general01 is down Step 8 not applied, or a global DNS=192.168.10.9 drop-in in /etc/systemd/resolved.conf.d/ resolvectl status virbr1 must show Default Route: no and DNS Domain: ~lab.riupie.com; remove the global drop-in (step 8 warning)
Container stays unhealthy named failed to load the zone docker compose logs bind9, then re-run step 5

Disabling the systemd-resolved stub listener on general01:

sudo mkdir -p /etc/systemd/resolved.conf.d
printf '[Resolve]\nDNS=127.0.0.1\nDNSStubListener=no\n' | sudo tee /etc/systemd/resolved.conf.d/bind9.conf
sudo systemctl restart systemd-resolved

Rollback

On general01:

cd /opt/bind9 && docker compose down

Clients that use 192.168.10.9 as their resolver (including the Fedora host for lab.riupie.com) lose DNS until it is restarted. /opt/bind9 (zone, journal, key) is left in place; delete it only if you are rebuilding from scratch, because you will need a new TSIG key in External-DNS afterwards.

References