OpnSense 24.7 - Disable WAN + OPT2 Interfaces during CARP Failover

10-wancarp

#!/usr/local/bin/php
<?php

require_once("config.inc");
require_once("system.inc");
require_once("interfaces.inc");
require_once("interfaces.lib.inc");
require_once("util.inc");

$subsystem = !empty($argv[1]) ? $argv[1] : '';
$type = !empty($argv[2]) ? $argv[2] : '';

// Add more interfaces that need to be disabled/enabled after a CARP event.
//$iface_aliases = array('wan', 'opt2');
//$iface_names = array('wan' => 'igc0', 'opt2' => 'gif0');
$iface_aliases = array('wan', 'wan');
$iface_names = array('wan' => 'igc0');
$dhcp_ifaces = array('lan', 'opt3', 'opt1');
// Optional if you want the default route removed on the backup system
$lan_vip = 'YOUR_LAN_GATEWAY_Virtual_IP';
$remove_backup_route = False;

if ($type != 'MASTER' && $type != 'BACKUP' && $type != 'INIT') {
    log_error("Carp '$type' event unknown from source '{$subsystem}'");
    exit(1);
}

if (!strstr($subsystem, '@')) {
    log_error("Carp '$type' event triggered from wrong source '{$subsystem}'");
    exit(1);
}

if ($type === "MASTER") {
    if ($config['interfaces']['wan']['enable'] == 0) {
      foreach ($iface_aliases as $ifkey) {
          // $iface_name = $iface_names[$ifkey];
          log_error("enable interface '$ifkey' due CARP event '$type'");
          $config['interfaces'][$ifkey]['enable'] = '1';
          legacy_interface_flags($ifkey, 'up');
          interface_configure(false, $ifkey, true, true);
          write_config("enable interface '$ifkey' due CARP event '$type'", false);
          //usleep(200 * 1000);
          //foreach ($dhcp_ifaces as $dhkey) {
          //  $config['dhcpd'][$dhkey]['enable'] = true;
          //}
      }
    } else {
      log_msg("Carp '$type' duplicate event triggered.");
    }
} else if ($type === "BACKUP") {
    if ($config['interfaces']['wan']['enable'] == 1) {
      foreach ($iface_aliases as $ifkey) {
          // $iface_name = $iface_names[$ifkey];
          log_error("disable interface '$ifkey' due CARP event '$type'");
          //foreach ($dhcp_ifaces as $dhkey) {
          //  $config['dhcpd'][$dhkey]['enable'] = false;
          //}
          interface_reset($ifkey);
          unset($config['interfaces'][$ifkey]['enable']);
          interface_configure(false, $ifkey, true, false);
          exec('/sbin/ifconfig ' . escapeshellarg($ifkey) . 'down 2>&1', $ifc, $ret);
          write_config("disable interface '$ifkey' due CARP event '$type'", false);
          if ($remove_backup_route === True) {
            exec('/sbin/route del default >&1', $ifc, $ret);
            exec('/sbin/route add default ' . $lan_vip . ' >&1', $ifc, $ret);
          }
      }
    } else {
      log_msg("Carp '$type' duplicate event triggered.");
    }
}
?>

Updated to add the requested, optional route removal on backup system. NOTE: This change is untested by me as it is not needed for my setup to work properly.

Happy to see that this help some people get around their situation with CARP like I had before.

I'm seeing an issue when failing over to my backup and back to the master again.

The WAN interfaces on the Master go down and the Backup comes up when I do a failover. What's not happening is a triggering of an interface reset. The interfaces get an IP address, but gateway monitoring doesn't trigger and the interfaces operate like they are down. When I refresh the interfaces, they renew IP and start passing traffic again.

Any ideas on this? I'm thinking maybe a follow up action after the interface bring up that refreshes the interface to force a DHCP renew.

This is the same as using the gui to restart routing.
exec("/usr/local/sbin/pluginctl -s routing restart 2>&1");

That last exec (exec("/usr/local/sbin/pluginctl -s routing restart 2>&1");) seems to break the whole script. As-is, the current script does a great job and turns off the Backup interfaces and brings up the primaries. I still have to manually refresh both WAN Interfaces before traffic will pass. Where in the script should this go? I thought here, but I think I was wrong: if ($type === "MASTER") { if ($config['interfaces']['wan']['enable'] == 0) { foreach ($iface_aliases as $ifkey) { // $iface_name = $iface_names[$ifkey]; log_error("enable interface '$ifkey' due CARP event '$type'"); $config['interfaces'][$ifkey]['enable'] = '1'; legacy_interface_flags($ifkey, 'up'); interface_configure(false, $ifkey, true, true); write_config("enable interface '$ifkey' due CARP event '$type'", false); <------------- exec("/usr/local/sbin/pluginctl -s routing restart 2>&1"); //usleep(200 * 1000); //foreach ($dhcp_ifaces as $dhkey) { // $config['dhcpd'][$dhkey]['enable'] = true; //} } Thanks again to all you who built the script.

…

On Wed, Sep 18, 2024 at 8:35 AM Edward Scroop ***@***.***> wrote: ***@***.**** commented on this gist. ------------------------------ This is the same as using the gui to restart routing. exec("/usr/local/sbin/pluginctl -s routing restart 2>&1"); — Reply to this email directly, view it on GitHub <https://gist.github.com/kronenpj/e90258f12f7a40c4f38a23b609b3288b#gistcomment-5196027> or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF3F35CWYQIFPAMCM7U5FB3ZXFXQ3BFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFVEYTEMBVGY3DMNZQU52HE2LHM5SXFJTDOJSWC5DF> . You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub> .

@toddgonzo74 do you have any logs of the errors?
perhaps adding a sleep right before restarting the routing might help?

sleep(2);
exec("/usr/local/sbin/pluginctl -s routing restart 2>&1");

I actually tried by enabling that sleep just below in the script that controls dhcp interfaces. Didn’t help.

…

On Sat, Sep 21, 2024 at 7:25 AM Edward Scroop ***@***.***> wrote: ***@***.**** commented on this gist. ------------------------------ @toddgonzo74 <https://github.com/toddgonzo74> do you have any logs of the errors? perhaps adding a sleep right before restarting the routing might help? sleep(2); exec("/usr/local/sbin/pluginctl -s routing restart 2>&1"); — Reply to this email directly, view it on GitHub <https://gist.github.com/kronenpj/e90258f12f7a40c4f38a23b609b3288b#gistcomment-5199805> or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF3F35B7D2N2ZUJO7CY2HXTZXVJTBBFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFVEYTEMBVGY3DMNZQU52HE2LHM5SXFJTDOJSWC5DF> . You are receiving this email because you were mentioned. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub> .

I fiddled with it a bit this morning and got it working. Failover seamless both ways now.. I might lose a ping when failing over.

Just added this after "write_config("enable interface '$ifkey' due CARP event '$type'", false);"

      sleep(3);
      exec("/usr/local/sbin/pluginctl -s routing restart 2>&1");

Thanks @kronenpj and @edward-scroop and all others.

I have a slightly new issue and am looking for advice. The script works for failing both ways. What isn't working great is my Spectrum internet. The connection keeps going down because of loss to my monitoring endpoint (i've tried 8.8.8.8, 8.8.4.4, 1.1.1.1, etc). When I set the endpoint to my spectrum gateway, the connection stays stable and rarely drops.

How would I force, upon a new DHCP lease, an update of the gateway monitor IP for the Spectrum gateway? The goal would be to replace the current gateway with the new gateway address (if a new gateway is present).

I haven't found that to be a problem - I'm on spectrum as well. I usually receive the same IP with the same default router.

I will mention that on System: Gateways: Configuration: WAN gateway: Edit, I've checked the Upstream Gateway option (help states This will select the above gateway as a default gateway candidate.) and it's worked well for me.