Redundant OpenBSD Firewalls

Part 3 - Network Services

We will now add some additional services to our single firewall. These are basic configurations which work as-is, but you will almost certainly need to modify and expand them to meet your needs.

Time server

I prefer the reference NTP implementation, but it can only send packets to other time servers from port 123. This won’t work for our use case. If we configure the packet filter on fw1 to accept all UDP traffic on port 123, then the time server on fw2 won’t work.

OpenBSD comes with OpenNTPD, which isn’t as full featured, but it does send outbound UDP packets on high ports. This allows the packet filter to track state on those outbound packets, and send the incoming ones back to the originator, making it possible for us to run two time servers.

Fortunately the NTP protocol design makes it super easy to have redundant servers. We need zero additional software support or configuration to make it happen. We need a NTP server listening on

Make /etc/ntpd.conf look something like this:

# configuration file for ntpd

# serve time to our internal network
listen on
listen on

# time servers
server 0.us.pool.ntp.org
server 1.us.pool.ntp.org
server 2.us.pool.ntp.org
server 3.us.pool.ntp.org

And start it up:

fw1# rcctl start ntpd

It may take a few minutes for the daemon to contact all the servers and sync the clock. To check the status of the time service:

fw1$ ntpctl -s all
4/4 peers valid, constraint offset -1s, clock synced, stratum 4

   wt tl st  next  poll          offset       delay      jitter from pool pool.ntp.org
 *  1 10  3 1292s 1529s        -0.217ms    14.496ms     6.526ms from pool pool.ntp.org
    1 10  3 1507s 1525s        -1.301ms    12.377ms     2.488ms from pool pool.ntp.org
    1 10  2 1414s 1503s        22.369ms    46.584ms     4.257ms from pool pool.ntp.org
    1 10  2 1576s 1587s        16.927ms   186.044ms   116.632ms      

If you see “clock synced”, then you are in good shape.

Authoritative name server for internal network

I have some internal hosts that I want to use friendly host names for, but I don’t want to publish these hosts in a public name server. So I need an internal only name server, which my internal caching resolver is aware of. The internal authoritative name server only needs to be queried by our internal caching resolver.

OpenBSD uses nsd as an authoritative name server. I won’t go into full details of how to configure a name server and zone files. For this exercise, the key configuration detail is to only listen on localhost port 5353.

Let’s set up nsd:

fw1# nsd-control-setup
fw1# rcctl enable nsd

Then install a basic configuration file in /var/nsd/etc/nsd.conf:

# /var/nsd/etc/nsd.conf - configuration for authoritative name server

    hide-version: yes
    verbosity: 1
    database: "" # disable database

    # only listen on localhost
    # the only resolver we care that can access this service
    # is the unbound server running on this same box
    interface: ::1@5353

    control-enable: yes
    control-port: 8952
    server-key-file:   /var/nsd/etc/nsd_server.key
    server-cert-file:  /var/nsd/etc/nsd_server.pem
    control-key-file:  /var/nsd/etc/nsd_control.key
    control-cert-file: /var/nsd/etc/nsd_control.pem

    name: kotfu.net
    zonefile: kotfu.net

    name: 13.168.192.in-addr.arpa

That’s it. Start it up and we are done:

fw1# rcctl start nsd

Caching DNS resolver

In theory, we only need the resolver to run on one IP address. In practice, I have seen stupid network configuration user interfaces on wireless routers that require two name server IP addresses. I have also had cases where I needed to move the resolver to a different piece of hardware. If the resolvers have their own IP addresses, the service can be moved without having to have every DHCP client renew their lease. Finally, most clients that query a resolver will wait a few seconds for it to time out before they try the second resolver in their list. We want to avoid that timeout.

Therefore, we will create a new CARP network interface, and assign it two IP address to be used only for name services. When we add our redundant hardware, we’ll just add it to the CARP group for these IP addresses, and be good to go.

We already have carp0, so let’s create carp1 with two IP addresses.

Put the following into /etc/hostname.carp1:

inet vhid 132 carpdev em0 pass vhid132passwd description "name servers"
inet alias

Bring up the interface:

fw1# sh /etc/netstart carp1

Configure unbound and get curl (to be used shortly):

fw1# rcctl enable unbound
fw1# unbound-control-setup
fw1# pkg-add -vv curl

Use curl to get a list of root name servers:

fw1# curl https://www.internic.net/domain/named.root > /var/unbound/etc/root.hints

Create the unbound configuration file /var/unbound/etc/unbound.conf:

# /var/unbound/etc/unbound.conf - caching resolver configuration

    interface: ::1

    access-control: refuse
    access-control: allow
    access-control: allow
    access-control: ::0/0 refuse
    access-control: ::1 allow

    # by default, unbound won't query localhost, let's change that
    do-not-query-localhost: no

    hide-identity: yes
    hide-version: yes

    # root hints
    # curl https://www.internic.net/domain/named.root > /var/unbound/etc/root.hints
    root-hints: /var/unbound/etc/root.hints

    # enable dnssec by specifying a root key file
    auto-trust-anchor-file: /var/unbound/db/root.key
	# allow RFC1918 addresses to be returned
	local-zone: 13.168.192.in-addr.arpa transparent

	# don't use dnssec for locally hosted zones
	domain-insecure: kotfu.net
	domain-insecure: 13.168.192.in-addr.arpa

# enable remote control, but only on localhost
    control-enable: yes
    control-port: 8953
    server-key-file:   /var/unbound/etc/unbound_server.key
    server-cert-file:  /var/unbound/etc/unbound_server.pem
    control-key-file:  /var/unbound/etc/unbound_control.key
    control-cert-file: /var/unbound/etc/unbound_control.pem

# stub out our internal zones
    name: kotfu.net

    name: 13.168.192.in-addr.arpa

Notice that the stub zones are queried from localhost port 5353, which is where we set up nsd to listen. Start it up and test it out:

fw1# rcctl start unbound
fw1# dig @localhost eff.org

Notice that we are also listening on That’s intentional, and we’ll need it later.

DHCP Server

There are two ways to have redundant DHCP servers. One method requires no software support in the servers for any type of failover. Assign two blocks of dynamic IP addresses, configure the first server to hand out addresses in the first block, and configure the second server to hand out addresses in the second block.

The second approach is to use a DHCP server that knows which instance is the master, and knows how to sync leases between two instances. The DHCP server that ships stock with OpenBSD doesn’t have either of these features. ISC has developed a new DHCP server called Kea, which appears to have better support for failover, but it’s fairly new and the version in the OpenBSD packages is not current. We will install the older ISC DHCP implementation from packages:

fw1# pkg_add -vv isc-dhcp-server
fw1# rcctl enable isc_dhcpd
fw1# rcctl set isc_dhcpd flags "-user _isc-dhcp -group _isc-dhcp em0"

Create a configuration file in /etc/dhcpd.conf:

# /etc/dhcpd.conf


# set our options
option domain-name "kotfu.net";
option domain-name-servers,;
option ntp-servers;
default-lease-time 216000; # 1 hr
max-lease-time 648000; # 3 hr

omapi-port 7911;
omapi-key kotfunet;

key kotfunet {
  algorithm hmac-md5;
  secret 1Mujy7dpQxIc6ulsCWG6owvlP9quKg==;

failover peer "failover-partner" {
  address fw1.kotfu.net;
  port 519;
  peer address fw2.kotfu.net;
  peer port 520;
  max-response-delay 60;
  max-unacked-updates 10;
  mclt 3600;
  split 128;
  load balance max seconds 3;

subnet netmask {
  option routers;
  option broadcast-address;
  pool {
    failover peer "failover-partner";

Note the configuration items defining an hmac-md5 key, as well as the definition of the failover partner. Another key difference from a standard dhcpd configuration is the requirement for a pool declaration around your IP ranges.

All the examples I can find for generating a secret key say to use the dnssec-keygen tool, which is part of BIND. But OpenBSD defaults to nsd and unbound, and so the dnssec-keygen tool is not installed. If you are doing dynamic DNS updates, you certainly will need dnssec-keygen. You could install the isc-bind9 package like everyone else says, or you can just generate a random base64 encoded string to use for the shared secret:

fw1$ dd if=/dev/urandom of=/dev/stdout bs=128 count=1 | uuencode -m -

Notice we have used the two IP addresses assigned to our resolver, as well as the NTP server. Your DHCP configuration will likely be more involved than this one (mine is), but it is sufficient for our current example. Finally, we start up the DHCP service:

fw1# rcctl start isc_dhcpd

Test it out with any device on your network to make sure it can dynamically acquire a working network configuration.

Part 2 is now complete. We have a single server which provides:

  • routing
  • packet filtering
  • authoritative name server
  • caching DNS resolver
  • NTP server
  • DHCP server

Along the way, we have done some preparatory work for Part 3, where we add in our second piece of firewall hardware and configure it for redundant routing and packet filtering.

Redundant OpenBSD Firewall Guide