Redundant OpenBSD Firewalls

Part 4 - Redundant Routing and Packet Filter

If you don’t have OpenBSD installed on fw2, best brew a cuppa’ and get that done.

Now you are ready for the most complex part of this project. We need to configure network interfaces on fw2 and bring them online. We also need to configure the packet filter, and have it sync its state table between the two machines.

Methods of synchronizing files

Before we go much further, we need to select a method to synchronize files between our two firewall servers. In order for most of these services to work in a redundant fashion, they need to have certain configuration details set to be the same. There are many ways to accomplish this goal, including:

Manually copy or edit files when you change them
Write a script to copy everything
Use the built in capabilities of certain daemons (like nsd) to synchronize data between services

Since we can’t exclusively use the third option, we’ll use a mix of options 2 and 3. We’ll create a script on fw2 which will use rsync to pull all the relevant configuration files from fw1. This script will be smart enough to copy files, and restart services as necessary. Another benefit of using this script is that it can include logic to modify the files copied from fw1 (you’ll soon see how this can be useful). Whenever we make a change on fw1, we will run a single command on fw2 and it will be in sync.

For this method to work, you’ll need root on fw2 to have an ssh key. On fw2 run:

fw2# ssh-keygen

On fw1, you’ll need to make sure that root logins are permitted via ssh. I disallow root logins using a password:

PermitRootLogin prohibit-password        

Restart sshd if necessary. Take root’s ssh key from fw2, which is located in the /root/.ssh/id_rsa.pub file and put it in the /root/.ssh/authorized_keys file on fw1.

Create a script on fw2, which we will add to as we go. It will copy configs from fw1, modify them if necessary, and restart or reload services. You can call this script whatever you want, and put it wherever you want. I’m going to call it /etc/sync-fw-config, and it will start off like this:

#!/bin/sh
#
# sync configuration items from fw1
#
# this script must be run as root, and it requires that root have
# passwordless login via ssh to fw1

SRC_HOST=fw1

# some env variables for the two firewall IP addresses
FW1=192.168.13.4
FW2=192.168.13.5

# make an rsync function to copy files
RSYNC="rsync -ai --delete-after --exclude=*~"
dosync() {
    echo checking $1
    $RSYNC $SRC_HOST:$1 $1
}

The dosync function will be used later to synchronize specific files and directories to fw2. Now back to the show.

Internal network interface

The hardware I am using for fw2 has three network interfaces: vr0, vr1, and vr2. On fw2 ensure /etc/hostname.vr0 has the following contents:

inet 192.168.13.5 255.255.255.0 192.168.13.255 description "internal interface"

Start it up:

fw2# sh /etc/netstart vr0

Configure CARP interfaces

When we were setting up fw1 we created a CARP interface with the IP address 192.168.13.1 which our internel network hosts will use to get to the internet. Now we are going to add fw2 to that existing CARP group. All of these actions should be performed on fw2.

Set up CARP:

fw2# sysctl net.inet.carp.allow=1
fw2# sysctl net.inet.carp.preempt=1
fw2# echo 'net.inet.carp.allow=1' >> /etc/sysctl.conf
fw2# echo 'net.inet.carp.preempt=1' >> /etc/sysctl.conf

Next we will create a carp0 network interface and add it to the CARP group that we previously created on fw1.

Put the following into /etc/hostname.carp0:

inet 192.168.13.1 255.255.255.0 192.168.13.255 vhid 131 carpdev vr0 pass vhid131passwd advskew 128 description "internal network gateway"

The vhid and password must be exactly the same as they were in carp0 on fw1, or it won’t work. The default advskew is 0. Setting it to 128 reduces the likelyhood that this interface will be chosen as the master, which is what we want. If fw1 is up, we want it to be the master.

Create /etc/hostname.carp1:

inet 192.168.13.2 255.255.255.0 192.168.13.255 vhid 132 carpdev vr0 pass vhid132passwd advskew 128 description "name servers"
inet alias 192.168.13.3 255.255.255.0

With the interfaces defined, apply the configuration by:

fw2# sh /etc/netstart carp0 carp1

To verify that it’s working, you should see something like this:

fw2# ifconfig carp0
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:00:5e:00:01:83
        description: internal network gateway
        index 7 priority 15 llprio 3
        carp: BACKUP carpdev vr0 vhid 131 advbase 1 advskew 128
        groups: carp egress
        status: backup
        inet 192.168.13.1 netmask 0xffffff00 broadcast 192.168.13.255

We are looking for the "status: backup" line. That means that the interface was added to the CARP group, and another host in the group (in our case it’s fw1) is the master. If you see "status: init", then something is wrong with the configuration. If you see "status: master" then the configuration wasn’t added to the already existing CARP group, which means we now have two groups, each with one master. You probably have an error in the "vhid" or "pass" configuration, or you might have it on the wrong physical interface.

The real test however, is to go over to fw1 and shutdown the carp0 interface.

fw1# ifconfig carp0 down

Give CARP a second or two to fail over, then you should see:

fw2# ifconfig carp0
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:00:5e:00:01:83
        description: internal network gateway
        index 10 priority 15 llprio 3
        carp: MASTER carpdev vr0 vhid 131 advbase 1 advskew 128
        groups: carp egress
        status: master
        inet 192.168.13.1 netmask 0xffffff00 broadcast 192.168.13.255

When you do:

fw1# ifconfig carp0 up

The status of the interface on fw2 should change to "backup".

Before we go on, let’s take a moment and realize the awesomeness of what we just did. We created network interfaces with IP addresses that transparently and automatically move between machines.

Configure packet filter

At this point our cable modem is still plugged directly into em1 on fw1. We are still going to configure the packet filter on fw2 so that the software will be ready when we connect the hardware. First we enable packet forwarding:

fw2# sysctl net.inet.ip.forwarding=1
fw2# echo 'net.inet.ip.forwarding=1' >> /etc/sysctl.conf

We want exactly the same packet filter rules on fw2 as we had on fw1. We could just add /etc/pf.conf to our previously created /etc/sync-fw-config script, except that our interface names are different. The good news is that pf.conf allows us to split our configuration into multiple files.

Let’s do that on fw1:

fw1# cd /etc
fw1# mkdir pf
fw1# chmod 700 pf
fw1# tail +7 pf.conf > pf/filter.conf

Then change /etc/pf.conf so it contains:

#
# /etc/pf.conf
#
# remember to set net.inet.ip.forwarding=1

int_if="em0"

include "/etc/pf/filter.conf"

Now we have /etc/pf/filter.conf which can be exactly the same between our two firewalls, with the differences isolated in /etc/pf.conf.

With fw1 modified, let’s proceed to configure fw2.

fw2# cd /etc
fw2# mkdir pf
fw2# chmod 700 pf

Create /etc/pf.conf:

#
# /etc/pf.conf
#
# remember to set net.inet.ip.forwarding=1

int_if="vr0"

include "/etc/pf/filter.conf"

Add a few lines to /etc/sync-fw-config:

#
# pf rules
dosync /etc/pf/
pfctl -f /etc/pf.conf
if [ $? == 0 ]; then
    echo 'pf(ok)'
else
    echo 'pf(fail)'
fi

Sync the config:

fw2# sh /etc/sync-fw-config

Verify it worked by displaying the rules:

fw2# pfctl -s rules

We should see the same set of rules as we have on fw1.

Synchronize packet filter state

As our internal network clients make connections to the Internet throughour firewall, the packet filter tracks the state of those connections inmemory. In order to ensure a seamless transition from one of our firewallsto the other, we need a mechanism for the two machines to share entries in the packet filter state table. This is done through a special type of network interface called pfsync, which is bound to a physical interface. pfsync does not support authentication, and the default behavior is to multicast updates on the local network. Best practice to secure these updates is to either:

Connect the two nodes that will be exchanging updates directly together using a crossover cable, creating a physically isolated network.
Configure pfsync to unicast updates, and then configure IPSEC between the two hosts to secure the traffic.

For this example, we are going to use the first option. Remember back in the requirements when I said we needed 3 network interfaces and a separate patch cable (or crossover cable)? This is why. So if you are doing this on hardware, it’s time to connect this patch to the third network interface on both of your machines, creating a private, two host network segment. Most Gigabit ethernet adapters are smart enough to solve the cross-over issues on their own, you can just use a regular patch cable. If you have a 100MBit interface you’ll need to use the special crossover cable. If using virtual machines, you’ll need to create a new private network segment, with an interface on each of your two virtual machines.

We need some IP addresses for this private connection between our two firewalls, I’m going to choose a totally different address range, because once this is up and running, we are pretty much never going to touch it again.

On fw1 create /etc/hostname.em2:

inet 172.16.1.4 255.255.255.0 172.16.1.255 description "private segment with fw2"

Start it up:

fw1# sh /etc/netstart em2

On fw2 create /etc/hostname.vr2:

inet 172.16.1.5 255.255.255.0 172.16.1.255 description "private segment with fw1"

Start it up:

fw2# sh /etc/netstart vr2

Verify connectivity by pinging these new IP addresses from both firewalls.

Our current packet filter configuration blocks all inbound network traffic (we created an exception for ICMP so that we can ping). We now must configure our packet filter to allow pfsync packets on the sync interfaces. Because the sync interfaces are directly wired to each other, we can safely pass all traffic on those interfaces.

On fw1 add this line to /etc/pf.conf:

pfsync_if="em2"

Then add this line to /etc/pf/filter.conf, just before the line with the carp rule:

pass quick on { $pfsync_if } proto pfsync keep state (no-sync)

Activate this change on fw1:

fw1# pfctl -f /etc/pf.conf

On fw2 add this line to /etc/pf.conf:

pfsync_if="vr2"

Sync the files, and activate the change:

fw2# sh /etc/sync-fw-config

With the physical interfaces up and running, and our packet filter rules in place to allow pfsync traffic, we can actually create the virtual network interfaces to synchronize the state table.

We’ll start on fw2 (you’ll see why in a minute). Create /etc/hostname.pfsync0:

up syncdev vr2

Activate the interface:

fw2# sh /etc/netstart pfsync0

On fw1 create /etc/hostname.pfsync0 to contain:

up syncdev em2

Activate the interface:

fw1# sh /etc/netstart pfsync0

We did fw2 first, so that it’s up and listening. When we bring up the pfsync interface on fw1, the kernel transmits the entire packet filter state table, and we want fw2 to be ready to catch it.

There are a couple of ways you can check that pfsync is working.

fw1# netstat -s | grep -A 16 pfsync

shows you how many pfsync packets have been sent and received. You can run this on either host, and you should be able to watch the send and receive counts go up .

You can also run

fw1# pfctl -s state

which shows you the entire state table from the packet filter. If you have an external host you can ssh into, you can ssh into that host, and search for the host’s IP address in the output on fw1. You should also be able to find that host’s IP address in the output on fw2.

Approach to configuring external interfaces

Right now on fw1 our external interface is up and working, but there is no redundant capability. The solution is not immediately obvious, here’s the rationale for the eventual solution.

When I request an IP address from my ISP via DHCP, they make a hard association between that IP address and the MAC address the DHCP request came from. If I change the MAC address (like I plug it into a different computer), the internet connection won’t work until I power cycle the cable modem. CARP shares an IP address between interfaces, but not a MAC address. We will need to use the same MAC address on the external interfaces of both firewalls, and only one of those interfaces can have the MAC address at a time. So maybe we try and build an external CARP interface so their is only one interface, and use the CARP interface to DHCP the IP address.

The release notes for OpenBSD 6.7 say that dhclient now supports CARP interfaces. I can’t see how that can work. If the CARP interface fails over, how does dhclient on the new master know the lease information for the interface? The dhclient daemons on all the carps would have to share lease information with each other the same way the dhcpd servers do. If they can do it, there doesn’t seem to be any documentation available of how to configure it. I can’t make it work by experimentation. CARP interfaces won’t come up without an IP address, and dhclient won’t do anything on an interface that isn’t up. Surely there are smarter people than me on the internet, but I can’t find one of them who has written how to make this work. dhclient on CARP is out.

In OpenBSD 6.9 dhcpleased was introduced, and it will eventually replace dhclient. dhcpleased was designed to solve the privilege separation and DNS issues with dhclient. Initial features were driven by the use case of a laptop roaming around to a bunch of different Wi-Fi networks. dhcpleased monitors an interface, and when it comes up, it requests an IP address for it. It also makes a request to the new in 6.9 daemon resolvd, which arbitrates requests to modify /etc/resolv.conf. dhcpleased has all the same issues running on a CARP interface as dhclient. dhcpleased on CARP is out.

Hat tip to Richard R. Charron for figuring out a solution using ifstated and a little layer 2 MAC address spoofing. I’ve simplified his approach, while also expanding the capability so we get a working nameserver for each of the interface states. I’ve also modified Richard’s solution to use dhcpleased instead of dhclient.

Configure external interface on fw1

If you aren’t familiar with ifstated, go read the man page now. You’ll need to understand what it is and how it works.

With that under your belt, I’ll explain what we are going to do. We will set up ifstated on both firewalls to watch the carp0 interface, which is on the internal side and has the IP address that our internal network clients use as their gateway to the internet). On fw1, when carp0 is the master, we will have ifstated bring up the external interface and run dhclient on it. At that moment on fw2, the carp0 interface will be the backup. We will configure ifstated on fw2 to bring down the external interface on fw2.

If fw1 goes down, the carp0 interface will switch so that fw2 is the master. We will configure ifstated on fw2 to notice when that has happened, and it will then bring up the external interface, but using the same MAC address as the external interface has on fw1. This bit of spoofing will make it so I don’t have to restart my cable modem. We’ll also configure ifstated to start dhclient on fw2 and renew the DHCP lease.

We also have to tell ifstated to change the default route for us. If fw1 is the master, then dhcpleased will set the default route. But when fw1 is the backup, it’s default route needs to be fw2. For reasons I don’t fully understand, fw1 can not use 192.168.13.1 on the carp0 interface as it’s default route. It must use a non-CARPed IP address. Be careful when modifying your packet filter rules, because each firewall needs to forward and NAT packets from both the CARP interface and the native interface. For this same reason, we will have to modify /etc/resolv.conf to use a nameserver other than 192.168.13.2 and 192.168.13.3. And it will have to change depending on whether carp1 is master or backup.

To summarize, we need to watch carp0 and carp1 for state changes, and they can change independent of each other, leaving us with four possible states for ifstated.

Let’s make it happen. We will begin on fw1. We need to make a note of the MAC address on our external interface, em1. On ethernet networks, a MAC address is set of 6 pairs of hex digits, separated by colons. OpenBSD calls this the lladdr, and you can find it by typing:

fw1# ifconfig em1

My MAC address is 00:0d:b9:45:36:75. Yours will definitely be different, and you’ll need it later in some configuration files.

Now come a batch of changes on fw1. First we are going to change the configuration of em1. Change /etc/hostname.em1 so it reads:

inet autoconf description "external interface" down

We want this interface to be managed by dhcpleased, that’s what the inet autoconf part does. But we also want it to start down when the machine boots. If we ever become the master on the internal CARP interface, then we will have ifstated turn this interface up, and it will request an IP address via DHCP. Get rid of any hardcoded default gateway, this will now change dynamically.

fw1# rm /etc/mygate

Turn off resolvd because we are going to manage /etc/resolv.conf ourselves:

fw1# rcctl disable resolvd

Create /etc/dhcpleased.conf, and tell dhcpleased not to overwrite our DNS servers with whatever we get from the ISP:

#
interface em1 {
  # we don't want resolvd messing with our name server configs
  # we handle this ourselves with ifstated
  ignore dns
}

With these changes, the standard netstart procedure will not be able to bring up this interface. Which is what we want. We are going to teach ifstated how to do it instead.

Note: the downside of this approach is that if you use:

fw1# sh /etc/netstart em1

after the machine has booted, and we happen to be the CARP master, it’s going to take the interface down, and your internet with it. That’s not ideal, but it is what it is. Good news, if you do this the fix is easy:

fw1# ifconfig em1 up

Let’s teach ifstated how to bring up this interface. Create /etc/ifstated.conf with the following contents:

#
# /etc/ifstated.conf

#
# As of OpenBSD 6.7 dhclient can assign an IP address to a CARP
# interface. However, if the mac address of the computer connected to
# the modem changes, my cable provider requires you to restart the
# modem to in order to route packets from the new mac address. This
# prevents us from using carp on the external interface.
#
# We will use ifstated to watch carp0 (the internal interface) and
# make appropriate changes to the mac address on em1 (the external
# interface hooked to the modem) so that failover works without having
# to powercycle the cable modem.
#
# For this to work properly, don't put anything in hostname.em1 except
# a description.
#
# Simultaneously, we need to watch carp1 to see if we are the master
# for name services, and adjust /etc/resolv.conf when the state changes.
#
# The carp interfaces could independently be master or backup, which leaves
# us with four states:
#   master_master, master_backup, backup_master, backup_backup
#
#   master_master, master_backup, backup_master, backup_backup
#

init-state auto
carp0_master="carp0.link.up"
carp0_backup="!carp0.link.up"
carp1_master="carp1.link.up"
carp1_backup="!carp1.link.up"

state auto {
	if $carp0_master && $carp1_master {
		set-state master_master
	}
	if $carp0_master && $carp1_backup {
		set-state master_backup
	}
	if $carp0_backup && $carp1_master {
		set-state backup_master
	}
	if $carp0_backup && $carp1_backup {
		set-state backup_backup
	}
}


state master_master {
	init {
		run "/etc/ifstated/carp0-master"
		run "/etc/ifstated/carp1-master"
	}

	if $carp0_master && $carp1_backup {
		set-state master_backup
	}
	if $carp0_backup && $carp1_master {
		set-state backup_master
	}
	if $carp0_backup && $carp1_backup {
		set-state backup_backup
	}
}


state master_backup {
	init {
		run "/etc/ifstated/carp0-master"
		run "/etc/ifstated/carp1-backup"
	}

	if $carp0_master && $carp1_master {
		set-state master_master
	}
	if $carp0_backup && $carp1_master {
		set-state backup_master
	}
	if $carp0_backup && $carp1_backup {
		set-state backup_backup
	}
}


state backup_master {
	init {
		run "/etc/ifstated/carp0-backup"
		run "/etc/ifstated/carp1-master"
	}

	if $carp0_master && $carp1_master {
		set-state master_master
	}
	if $carp0_master && $carp1_backup {
		set-state master_backup
	}
	if $carp0_backup && $carp1_backup {
		set-state backup_backup
	}
}


state backup_backup {
	init {
		run "/etc/ifstated/carp0-backup"
		run "/etc/ifstated/carp1-backup"
	}

	if $carp0_master && $carp1_master {
		set-state master_master
	}
	if $carp0_master && $carp1_backup {
		set-state master_backup
	}
	if $carp0_backup && $carp1_master {
		set-state backup_master
	}
}

Notice for each state how we run some scripts to make the necessary changes? We need to create those scripts. These scripts have been carefully designed so that we can copy them to the other firewall and make them work. Create a directory to hold the scripts:

fw1# mkdir /etc/ifstated

Now create /etc/ifstated/carp0-master:

#!/bin/sh
#
# run by ifstated
#
# configure the external interface and routes for us to be the gateway
NETDEV=em1
MAC_ADDR=00:0d:b9:45:36:75

# kill any existing dhclient processes
pkill -9 -f dhclient: $NETDEV

# attach the real MAC address to $NETDEV
ifconfig $NETDEV lladdr $MAC_ADDR up

# flush the arp cache and delete the default route
route -qn flush
route delete default

# renew IP lease, which will set the default route
dhclient $NETDEV

See that variable for the mac address? You should put your own mac address in there. Because we don’t know if carp0 was already in the master state or not when this script runs, we have to do whatever we need to in order to make the interface and routes be in the state we need them to be in. That’s why we kill the dhclient process first.

Next comes /etc/ifstated/carp0-backup:

#!/bin/sh
#
# run by ifstated
#
# configure the egress interface for the other firewall to be the gateway
NETDEV=em1
GW=192.168.13.5
DEADBEEF=de:ad:00:00:be:ef

# kill dhclient
pkill -9 -f dhclient: $NETDEV

# remove the ip address from the external interface, give it
# a bogus MAC address, and shut it down
ifconfig $NETDEV delete lladdr $DEADBEEF down

# flush the arp cache and delete the default route
route -qn flush
route delete default

# set the default route to the other firewall
route add default $GW

When this script gets run, fw2 is the master, so we need to shutdown our em1 interface, and also change the default gateway to point to fw2. Notice that when we are in the backup state, we assign em1 to have a valid but bogus MAC address of de:ad:00:00:be:ef. We do that because we are going to tell fw2 to use the real MAC address from the em1 interface on it’s vr1 interface (you’ll see the config for that in a minute), and everything will break if we have two hosts on the same LAN with the same MAC address.

Now we need to fix up the nameservers. Our method will have major clashes with resolvd, so we begin by disabling it:

fw1# rcctl disable resolvd

Here’s the scripts to change which nameservers we use. Create /etc/ifstated/carp1-master:

#!/bin/sh
#
# run by ifstated
#
# change /etc/resolv.conf to point to our own IP address

NS=192.168.13.4

sed -i "/^nameserver /s/.*/nameserver $NS/" /etc/resolv.conf

If you don’t know sed spend a few minutes and read about it. The name server address needs to change depending on whether we are the primary or the backup. When we are the backup the name server should be the other firewall’s IP address. When we are primary, it should be our IP address. That one line sed command finds the relevant line in /etc/resolv.conf and changes the IP address to be our own IP address.

Create /etc/ifstated/carp1-backup:

#!/bin/sh
#
# run by ifstated
#
# change /etc/resolv.conf to point to the other filewall's IP address

NS=192.168.13.5

sed -i "/^nameserver /s/.*/nameserver $NS/" /etc/resolv.conf

When we are the backup on carp1, we should use fw2 as our nameserver, this script makes the appropriate change.

All the scripts in /etc/ifstated need to be executable:

fw1# chmod 744 /etc/ifstated/*

With that config file in place, we’ll shut down the existing external interface, then fire up ifstated, which will bring it up for us.

fw1# ifconfig em1 down
fw1# rcctl enable ifstated
fw1# /etc/rc.d/ifstated start

If all is well, we should be able to look at the em1 interface with ifconfig, and see it up and running with an IP address. There should be a running dhcpleased process as well.

You can test this yourself by typing:

fw1# ifconfig carp0 down
fw1# ifconfig em1

With the carp0 interface down, ifstated should have brought down the em1 interface and given it a MAC address of de:ad:00:00:be:ef. When you bring up the carp0 interface, ifstated will notice, and change the MAC address back to the “real” one, which dhcpleased will notice and request an IP address for it using DHCP.

Configure external interface on fw2

Yikes, that was a lot of work. Now we turn our attention to fw2, where we will perform a similar procedure. The good news is that we can copy most of the complex configs and scripts from fw1. Start with /etc/hostname.vr1:

inet autoconf description "external interface" down

We want it to be managed by dhcpleased, but we also want the interface to be down when the machine first boots, and remain down until we become the master on the internal CARP interface.

We also need to get rid of any hardcoded default gateway, this will now change dynamically.

fw2# rm /etc/mygate

Turn off resolvd because we are going to manage /etc/resolv.conf ourselves:

fw2# rcctl disable resolvd

Create /etc/dhcpleased.conf, and tell dhcpleased not to overwrite our DNS servers with whatever we get from the ISP. With resolvd disabled, we shouldn’t need this, but I am big fan of belt and suspenders:

#
interface vr1 {
  # we don't want resolvd messing with our name server configs
  # we handle this ourselves with ifstated
  ignore dns
}

Add a few lines to /etc/sync-fw-config:

#
# ifstated
dosync /etc/ifstated.conf
dosync /etc/ifstated/
# now go fix up all the scripts
sed -i "/^NETDEV=/s/em1/vr1/" /etc/ifstated/gw-master
sed -i "/^NETDEV=/s/em1/vr1/" /etc/ifstated/gw-backup
sed -i "/^GW=/s/.*/GW=$FW1/" /etc/ifstated/gw-backup
sed -i "/^NS=/s/.*/NS=$FW2/" /etc/ifstated/ns-master
sed -i "/^NS=/s/.*/NS=$FW1/" /etc/ifstated/ns-backup
/etc/rc.d/ifstated restart

We copy the files, and then use a few sed commands to modify the copied scripts so they are suitable for fw2.

Turn it all on:

fw2# ifconfig vr1 down
fw2# rcctl enable ifstated
fw2# sh /etc/sync-fw-config

If you are running this on real hardware, it’s now time to unplug your cable modem from fw1. You need to plug the cable modem into a new switch, and then plug the external interfaces of both fw1 and fw2 into the same switch.

If we’ve done everything correctly, when we bring down carp0 on fw1, fw2 should bring up vr1 with the same MAC address as the em1 interface on fw1.

You can watch /var/log/daemon while you are bring up and down carp0 and carp1 on fw1, and see the state changes take effect.

kotfu.net

Redundant OpenBSD Firewalls

Part 4 - Redundant Routing and Packet Filter

Methods of synchronizing files

Internal network interface

Configure CARP interfaces

Configure packet filter

Synchronize packet filter state

Approach to configuring external interfaces

Configure external interface on fw1

Configure external interface on fw2

Redundant OpenBSD Firewall Guide