Disable WAN Interface on CARP Backup

10-wancarp

#!/usr/local/bin/php
<?php

require_once("config.inc");
require_once("interfaces.inc");
require_once("util.inc");

$subsystem = !empty($argv[1]) ? $argv[1] : '';
$type = !empty($argv[2]) ? $argv[2] : '';

if ($type != 'MASTER' && $type != 'BACKUP') {
    log_error("Carp '$type' event unknown from source '{$subsystem}'");
    exit(1);
}

if (!strstr($subsystem, '@')) {
    log_error("Carp '$type' event triggered from wrong source '{$subsystem}'");
    exit(1);
}

$ifkey = 'wan';

if ($type === "MASTER") {
    log_error("enable interface '$ifkey' due CARP event '$type'");
    $config['interfaces'][$ifkey]['enable'] = '1';
    write_config("enable interface '$ifkey' due CARP event '$type'", false);
    interface_configure(false, $ifkey, false, false);
} else {
    log_error("disable interface '$ifkey' due CARP event '$type'");
    unset($config['interfaces'][$ifkey]['enable']);
    write_config("disable interface '$ifkey' due CARP event '$type'", false);
    interface_configure(false, $ifkey, false, false);
}

@edward-scroop, Yes I have gateway monitoring set for my WAN gateway of both primary and backup. The problem is not with my primary node switching back to master but my backup node switching back to being a backup. This way, backup has internet access for receiving it OPNsense updates and news Annoncements

For clarity, here is my primary configuration for the WAN link when primary is primary and backup is backup :

...and for my backup configuration. Blue arrow point to the fields where MY_CARP_LAN_VIP is specifed.

From your screenshots, the monitor ip is empty and the disable gateway monitoring is checked. That would mean gateway monitoring is disabled.

I think what is happening is as your WAN gateway has a higher priority than the LAN gateway and with no gateway monitoring, the backup has no way to tell the WAN gateway is down and it then doesn't have a reason to swap to the LAN gateway.

To fix it either set the LAN gateway to a priority higher than the WAN gateway, or set a monitor ip of 1.1.1.1 and uncheck the disable gateway monitoring box.

Hi, WAN Gateway has priority 254 and WAN-to-LAN has 255 (so WAN > WAN-to-LAN).

Anyway, I tried your trick and worse, my backup has no internet access when backup. Default route has shown still point default gateway to the WAN IP that connects to nothing when backup.

Interfaces: Diagnostics: Ping to 1.1.1.1 has 100% loss :(

Since fixing the default gateway (with route delete followed by add CARP_LAN_IP) while being backup of a functional primary node, it might have been the missing trick with my setup that is pretty standard when theISP provided only a public DHCP WAN IP.

I'll keep the setup I shared earlier. Thank's for sharing edward-scroop.

The LAN gateway needs a priority higher than 254. The smaller the value, the higher the priority.

The LAN gateway needs a priority higher than 254. The smaller the value, the higher the priority.

It's the case, LAN has priority 255

I meant, the LAN needs a priority of 1-253.

I upgraded to 24.7.6 today, and our syshook.d scripts that call interface_configure() appears to now crash when an undefined function eventually is called (see my stack trace below). See my post on opnsense forums: https://forum.opnsense.org/index.php?topic=20972.msg216770#msg216770 for the customizations I run, but I'd imagine Spali's version is equally as affected. I submitted a crash report, but did not create an issue on the opnsense github.

I believe we need to be using a more well supported method to enable/disable interfaces in these syshook scripts. The 'interface' PHP functions seem to be in heavy development in 24.7, and many functions seem to be considered 'legacy' methods or becoming deprecated. Or, perhaps this is just a bug.

As a workaround, if you don't want to roll-back, you can comment the $config line, write_config, and interface_configure calls and instead use shell_exec("/sbin/ifconfig {$interface['if']} up"); and shell_exec("/sbin/ifconfig {$interface['if']} down"); instead, but this is less reliable and has other undesirable effects. For example, when only using interface up/down commands, the backup device needs it's WAN interface left as enabled - under that condition, in the event of a reboot, you'll want to manually trigger a failover cycle to have the backup device's WAN interface in "down" state, else you'll have both interfaces up and enabled. Again, we need to find the most well supported way to enable/disable interfaces, and go from there.

[22-Oct-2024 13:17:14 America/New_York] PHP Fatal error: Uncaught Error: Call to undefined function system_routing_configure() in /usr/local/etc/inc/interfaces.inc:3777
Stack trace:
#0 /usr/local/etc/inc/interfaces.inc(2498): interfaces_restart_by_device(false, Array, false)
#1 /usr/local/etc/rc.syshook.d/carp/10-wancarp(24): interface_configure(false, 'opt3', false, false)
#2 {main}
thrown in /usr/local/etc/inc/interfaces.inc on line 3777

As a side note, others are having trouble with carp maintenance mode not working at all (not triggering a failover, as one would expect): opnsense/core#7877

Anyone find a fix for this issue yet?

i haven't tried it yet, but does this issue also occur at 24.7.8? @bitcoredotorg perhaps you tried the update?

I just upgraded to 24.7.8 (I was actually on 24.7.7 and it was working fine... as was it in 24.7.6). I run both my firewalls in Proxmox, so I took a backup snapshot before each upgrade, just in case. When the primary node came back up, the only thing I noticed was that it was pinned up in persistent carp maintenance mode.. I enabled and disabled and the backup failed right over to the primary. Only issue I still have is with Spectrum. For some reason, when I use a vlan on my managed switch (Juniper EX3400 POE), the Spectrum routinely fails to DHCP a new address (I have dhcp snooping and damn near everything else disabled in that vlan that could be interfering). For a goof, I grabbed an old gig switch from Netgear and plugged in the Spectrum primary/backup and circuit.. been fine for 4 months now. Fails over Spectrum with no issues.

Anyway... not seeing the problem in 24.7.8.