OPNsense: Start/stop WireGuard based on CARP state change (place in /usr/local/etc/rc.syshook.d/carp/)

10-wireguard

#!/usr/local/bin/php
<?php

/*
 * Copyright (C) 2004 Scott Ullrich <[email protected]>
 * All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions are met:
 *
 * 1. Redistributions of source code must retain the above copyright notice,
 *    this list of conditions and the following disclaimer.
 *
 * 2. Redistributions in binary form must reproduce the above copyright
 *    notice, this list of conditions and the following disclaimer in the
 *    documentation and/or other materials provided with the distribution.
 *
 * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
 * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
 * AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
 * AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
 * OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
 * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
 * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
 * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 * POSSIBILITY OF SUCH DAMAGE.
 */

require_once("config.inc");
require_once("interfaces.inc");
require_once("util.inc");

$subsystem = !empty($argv[1]) ? $argv[1] : '';
$type = !empty($argv[2]) ? $argv[2] : '';

if ($type != 'MASTER' && $type != 'BACKUP') {
    log_error("Carp '$type' event unknown from source '{$subsystem}'");
    exit(1);
}

if (!strstr($subsystem, '@')) {
    log_error("Carp '$type' event triggered from wrong source '{$subsystem}'");
    exit(1);
}

if ($type === "MASTER") {
    log_error("Enabling WireGuard due to CARP event '$type'");
    # Checking `isset` avoids a race condition during startup when the
    # WireGuard config stanza seems like it's not yet loaded. Without it, this
    # can create an extra, empty, invalid stanza that breaks WireGuard.
    if (isset($config['OPNsense']['wireguard']['general']['enabled'])) {
      $config['OPNsense']['wireguard']['general']['enabled'] = '1';
    }
    configd_run('wireguard start');
    write_config("Enable WireGuard due to CARP event '$type'", false);
} else {
    log_error("Disabling WireGuard due to CARP event '$type'");
    configd_run('wireguard stop');
    if (isset($config['OPNsense']['wireguard']['general']['enabled'])) {
        $config['OPNsense']['wireguard']['general']['enabled'] = '0';
    }
    write_config("Disable WireGuard due to CARP event '$type'", false);
}

I really have come to the conclusion the answer lies not in the CARP syshook and running the start multiple times with the sleep statement, but, debugging and fixing the actual wireguard start command that actually gets run.

See my new forum post and franco's reply:
https://forum.opnsense.org/index.php?topic=31962.0

Franco talks about:

In a nutshell it just calls

# /usr/local/etc/rc.d/wireguard start

and does whatever the RC system deems appropriate. No clue what's wrong in your cause, but I do know WireGuard doesn't make itself any easier to debug experimental or not.

glad to see, that the sleep and calling configd_run twice seems to help. It seems, that there is a race condition between enabling wireguard, getting things ready (opnsense site) and starting wireguard. I'm not familiar with the opnsense internas and what the $config['OPNsense']['wireguard']['general']['enabled'] = '1'; is starting. If this is just writing "1" to the config.xml nothing is loaded. If then configd_run('wireguard start'); is running, it takes the change of "wireguard enabled" in the config.xml and informs the actual running system. While the system is doing this change, it can't start wireguard, because this first attempt relies on "wireguard disabled". The sleep gives the system (depending on the used hardware) enough time to end all calls on the system and the second configd_run('wireguard start'); finally starts wireguard, because wireguard is enabled in the config.xml.

I don't know when I can take a deeper look, but I would think of:

-> $config['OPNsense']['wireguard']['general']['enabled'] = '1';
-> "reload settings" of opnsense and wait for completion
-> configd_run('wireguard start');

If anyone is interested, I put in this PR opnsense/plugins#3299. I think it checks all the boxes folks are talking about here. Worst case it is rejected, but feel free to test.

Change log_error to log_msg Regards, Steven On 8/02/2023, at 08:44, Bubbagump209 ***@***.***> wrote:  @Bubbgump209 commented on this gist.

…

________________________________ If anyone is interested, I put in this PR opnsense/plugins#3299<opnsense/plugins#3299>. I think it checks all the boxes folks are talking about here. Worst case it is rejected, but feel free to test. — Reply to this email directly, view it on GitHub<https://gist.github.com/18ca7bf14ddae547ae0fdf6f56d72573#gistcomment-4463654> or unsubscribe<https://github.com/notifications/unsubscribe-auth/AQCG7JEKXTGM22OFOZPYYPLWWLM53BFKMF2HI4TJMJ2XIZLTSKBKK5TBNR2WLJDHNFZXJJDOMFWWLK3UNBZGKYLEL52HS4DFQKSXMYLMOVS2I5DSOVS2I3TBNVS3W5DIOJSWCZC7OBQXE5DJMNUXAYLOORPWCY3UNF3GS5DZVRZXKYTKMVRXIX3UPFYGLK2HNFZXIQ3PNVWWK3TUUZ2G64DJMNZZDAVEOR4XAZNEM5UXG5FFOZQWY5LFVEYTCNBVHA4DGMRUU52HE2LHM5SXFJTDOJSWC5DF>. You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

Derp, good catch! Fixed.

It seems the script broke with opnsense/plugins@86c9e5c

The configd run now requires to give the wireguard instance as parameter.

So if you upgrade to 23.7.3 it breaks.

The following works:

$servers = (new \OPNsense\Wireguard\Server())->servers->server->iterateItems();

foreach ($servers as $key => $node) {
    if (!empty((string)$node->enabled)) {
        $backend->configdRun("wireguard start {$key}");
    }
}

// repeat for stop

This is now unnecessary as proper CARP support is now built into OPNsense with WireGuard since OPNsense 23.7.8 released 09 Nov 2023 and further improved in the latest OPNsense firmware.

The WireGuard follow CARP implementation by the OPNsense dev team is excellent and it works really well!