Skip to content

Instantly share code, notes, and snippets.

@cywang117
Last active August 30, 2024 00:01
Show Gist options
  • Save cywang117/83274f86850b9cbf23581cb9c7fa59ea to your computer and use it in GitHub Desktop.
Save cywang117/83274f86850b9cbf23581cb9c7fa59ea to your computer and use it in GitHub Desktop.
balena-tips

Diagnose devices experiencing inconsistent Docker Engine state

DISCLAIMER: This gist is up-to-date as of Supervisor v16.6.1 and OS 6.0.x and may not necessarily apply to future Supervisor or OS versions.

Inspect Supervisor logs

journalctl -u balena-supervisor -u resin-supervisor -xef

If you see any errors with the format (HTTP code XXX), this is not a Supervisor error format. The Supervisor forwards these error messages from Docker Engine.

Purge Docker directory

DISCLAIMER: This gist is up-to-date as of Supervisor v16.6.1 and OS 6.0.x and may not necessarily apply to future Supervisor or OS versions.

systemctl stop balena
systemctl stop balena-supervisor resin-supervisor
rm -rf /var/lib/docker/{aufs,overlay2,containers,image,tmp,containerd}
systemctl start balena
update-balena-supervisor
systemctl start balena-supervisor

WARNING: This nuclear method will remove Docker directory data and will most likely result in data loss. It is intended as a silver bullet of resetting Engine state on devices, and works around any potential Engine bugs rather than exposing their root cause. Please only use it in cases where device recovery is crucial. In other cases, please contact balena support by raising a ticket.

Upgrade the Supervisor using balena Node SDK from browser

DISCLAIMER: This gist is up-to-date as of Supervisor v16.6.1 and OS 6.0.x and may not necessarily apply to future Supervisor or OS versions.

Get target Supervisor release ID

From browser dev console on dashboard:

await sdk.pine.get({
    resource: 'supervisor_release',
    options: {
        $filter: {
            is_for__device_type: {
                $any: {
                    $alias: 'dt',
                    $expr: {
                        dt: {
                            slug: '$DEVICE_SLUG'
                        }
                    }
                }
            },
            supervisor_version: '$VERSION'
        },
    }
})

Notes:

  • Replace $DEVICE_SLUG with the slug of your device type, found by running cat /mnt/boot/device-type.json | jq -r '.slug'.
  • Replace $VERSION with the target Supervisor version, including the v. For example, v16.6.1.

You will get a response in Array form. Note the id field, this is $SV_RELEASE_ID in the next SDK command.

Upgrade device to Supervisor release

await sdk.pine.patch({
    resource: 'device',
    options: {
        $filter: { uuid: { $startswith: '$BALENA_DEVICE_UUID' }}
    },
    body: {
        should_be_managed_by__supervisor_release: $SV_RELEASE_ID
    }
})

Notes:

  • Replace $BALENA_DEVICE_UUID with the full UUID of your device.
  • Replace $SV_RELEASE_ID with the id number queried in the previous command.

You should receive a 200 OK response.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment