Skip to content

Instantly share code, notes, and snippets.

View cywang117's full-sized avatar

Christina Wang cywang117

5 followers · 1 following
View GitHub Profile
@cywang117
cywang117 / 001.md
Last active August 30, 2024 00:01
balena-tips

Diagnose devices experiencing inconsistent Docker Engine state

DISCLAIMER: This gist is up-to-date as of Supervisor v16.6.1 and OS 6.0.x and may not necessarily apply to future Supervisor or OS versions.

Inspect Supervisor logs

journalctl -u balena-supervisor -u resin-supervisor -xef

If you see any errors with the format (HTTP code XXX), this is not a Supervisor error format. The Supervisor forwards these error messages from Docker Engine.

@cywang117
cywang117 / README.md
Created April 27, 2022 00:05
Pinning Supervisor version in the API (this will NOT allow downgrades)

  1. Run update-balena-supervisor or update-resin-supervisor on the device. If that doesn't work...

  2. PATCH the should_be_managed_by__release (or should_be_managed_by__supervisor_release if that doesn't work) field in the device resource (/v6/device) to equal the release ID of the target Supervisor version. To find the release ID of the target Supervisor version: (Replace variables starting with $ with the fields that are relevant to your case, and run this in a dashboard dev console)

await sdk.pine.get({
    resource: 'supervisor_release',
    options: {
        $filter: {
            is_for__device_type: {
@cywang117
cywang117 / delete-device-config-var-overrides.js
Last active May 2, 2024 18:41
For large balena fleets, it may be necessary to run this from the Chrome dev console.
// This script may be run in a Chrome dev console, and will delete all the device config variables for each device in a fleet,
// which the exception of *_SUPERVISOR_DELTA or *_SUPERVISOR_DELTA_VERSION.
await (async () => {
// Make sure FLEET_ID is the ID of your fleet.
const FLEET_ID = 12345;
// Get devices in fleet
const devices = await sdk.models.device.getAllByApplication(FLEET_ID, {
$select: 'id',
});
@cywang117
cywang117 / state-patch-logs-overall-status
Created January 15, 2022 00:47
Supervisor state current state report logs during an instance of device.overall_status fluctuation
Jan 15 00:29:13 balena balena-supervisor[7724]: [event] Event: Update notification {}
Jan 15 00:29:13 balena balena-supervisor[7724]: [api] POST /v1/update 204 - 17.259 ms
Jan 15 00:29:14 balena balena-supervisor[7724]: [info] Applying target state
Jan 15 00:29:15 balena balena-supervisor[7724]: [event] Event: Service kill {"service":{"appId":1868586,"serviceId":1382522,"serviceName":"proxy","commit":"2f4203beb706ee4583cb13617d15f92d","releaseId":2039758}}
Jan 15 00:29
@cywang117
cywang117 / preload-any-supervisor-version-onto-balena-os.md
Last active September 3, 2024 13:31
Preload any Supervisor onto balenaOS image

Replace loopN with loop number listed with partx command

Download & mount .img

balena os download raspberrypi4-64 -o rpi4.img
losetup -fP --show rpi4.img
mount /dev/loopNp6 $MOUNTPOINT

If the mount fails with wrong fs type or similar, this is most likely because the image is a flasher image (i.e. for Intel NUC, for example), and you will need to do the following:

@cywang117
cywang117 / restore-corrupt-supervisor-conf.md
Created January 5, 2022 20:59
Instructions for how to restore a corrupt /etc/resin-supervisor/supervisor.conf

For all commands below, if OS is v2.78.0 or newer, replace resin-supervisor with balena-supervisor. See: https://github.com/balena-os/meta-balena/blob/master/CHANGELOG.md#v2780

Verify issue

  1. cat /etc/resin-supervisor/supervisor.conf should show corrupted file.
  2. Verify no other files are corrupt with grep -v "/var/cache/ldconfig/aux-cache" /resinos.fingerprint | md5sum --quiet -c - See: https://www.balena.io/docs/learn/more/masterclasses/device-debugging/#112-storage-media-corruption
  3. Note last reboot time and when the device was observed to stop working (i.e. Supervisor could not start), notify support agents if possible

Acquire necessary data to regenerate file from API

  1. From the dashboard page, open Chrome / Firefox developer's console with Ctrl/Cmd + Shift + I. Run the following:
@cywang117
cywang117 / hostname-change-logs.txt
Created January 4, 2022 20:45
Jan 04 20:21:12 pi4 echo[7733]: config.json changed
Jan 04 20:21:12 pi4 openvpn[3310]: Tue Jan 4 20:21:12 2022 event_wait : Interrupted system call (code=4)
Jan 04 20:21:12 pi4 openvpn[3310]: Tue Jan 4 20:21:12 2022 Closing TUN/TAP interface
Jan 04 20:21:12 pi4 openvpn[3310]: Tue Jan 4 20:21:12 2022 /sbin/ip addr del dev resin-vpn local 10.240.72.29 peer 52.4.252.97
Jan 04 20:21:12 pi4 avahi-daemon[3307]: Got SIGTERM, quitting.
Jan 04 20:21:12 pi4 avahi-daemon[3307]: Leaving mDNS multicast group on interface veth9ca5cd5.IPv6 with address fe80::bc2f:bbff:fe42:b0e7.
Jan 04 20:21:12 pi4 avahi-daemon[3307]: Leaving mDNS multicast group on interface vethe7a5351.IPv6 with address fe80::4079:f5ff:fe08:b25e.
Jan 04 20:21:12 pi4 avahi-daemon[3307]: Leaving mDNS multicast group on interface veth604b734.IPv6 with address fe80::ac83:5bff:fe42:aaaf.
Jan 04 20:21:12 pi4 avahi-daemon[3307]: Leaving mDNS multicast group on interface veth8a63e57.IPv6 with address fe80::8801:90ff:fe4c:825a.
Jan 04 20:21:12 pi4 avahi-daemon[
@cywang117
cywang117 / gist:9b4a46acdc32ff868b9dec429564873f
Last active September 21, 2021 01:52
How-to: balena delta image preloading
  1. Trigger an update with deltas enabled between release A and release B on a test device.
  2. Find the delta image that gets pulled from balena images -- it should be REGISTRY_URL:DELTA_TAG
    • Alternatively, if running Supervisor v12.9.5 or greater (12.9.6 highly recommended), look in the Supervisor's journal logs.
    • Alternatively, make a request the delta endpoint with curl -H "Bearer: Authorization $(cat /mnt/boot/config.json | jq -r '.deviceApiKey)" "https://delta.balena-cloud.com/api/v3/delta?src=SOURCE_REGISTRY_URL&dest=TARGET_REGISTRY_URL". This should return the delta URL and is the best amongst these 3 alternatives, but from my own investigations and limited knowledge, this works sporadically so is not recommended.
  3. After you have the registry delta URL, on production devices:
    • balena login registry2.balena-cloud.com -u d_$(cat /mnt/boot/config.json | jq -r '.uuid') -p $(cat /mnt/boot/config.json | jq -r '.deviceApiKey')
    • balena pull DELTA_URL
    • `balena image tag DELTA_URL
@cywang117
cywang117 / gist:e6b2db24bd92ca8692bb1be7659e92f3
Created September 10, 2021 18:17
Supervisor reboot logs from reboot trigger until shutdown
root@2f9551d:~# journalctl --since="2021-09-10 18:10" --until="2021-09-10 18:11" --no-pager -xe
-- Logs begin at Fri 2021-09-10 16:31:28 UTC, end at Fri 2021-09-10 18:16:21 UTC. --
Sep 10 18:10:41 2f9551d systemd[1]: Starting SSH Per-Connection Server (52.4.252.97:48986)...
Sep 10 18:10:41 2f9551d systemd[1]: Started SSH Per-Connection Server (52.4.252.97:48986).
Sep 10 18:10:41 2f9551d dropbear[5031]: Failed loading /etc/dropbear/dropbear_dss_host_key
Sep 10 18:10:41 2f9551d dropbear[5031]: Failed loading /etc/dropbear/dropbear_ecdsa_host_key
Sep 10 18:10:41 2f9551d dropbear[5031]: Child connection from ::ffff:52.4.252.97:48986
Sep 10 18:10:42 2f9551d dropbear[5031]: Pubkey auth succeeded for 'root' with key md5 4b:a1:6d:4b:1b:a1:58:0b:0d:bb:82:06:ea:26:16:c8 from ::ffff:52.4.252.97:48986
Sep 10 18:10:44 2f9551d balenad[1191]: [2021-09-10T18:10:44.686Z] Applying target state
Sep 10 18:10:44 2f9551d resin-supervisor[3877]: [2021-09-10T18:10:44.686Z] Applying target state
@cywang117
cywang117 / 06-24-hup-rollback-health.log
Created June 25, 2021 08:26
-- Logs begin at Wed 2021-06-23 07:55:37 UTC, end at Fri 2021-06-25 08:23:04 UTC. --
Jun 24 09:04:12 localhost kernel: Booting Linux on physical CPU 0x0
Jun 24 09:04:12 localhost kernel: Linux version 5.4.70+ (oe-user@oe-host) (gcc version 9.3.0 (GCC)) #1 SMP PREEMPT Tue Jan 5 03:15:22 UTC 2021
Jun 24 09:04:12 localhost kernel: CPU: ARMv7 Processor [413fc082] revision 2 (ARMv7), cr=10c5387d
Jun 24 09:04:12 localhost kernel: CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
Jun 24 09:04:12 localhost kernel: OF: fdt: Machine model: TI AM335x BeagleBone Black
Jun 24 09:04:12 localhost kernel: Memory policy: Data cache writeback
Jun 24 09:04:12 localhost kernel: efi: Getting EFI parameters from FDT:
Jun 24 09:04:12 localhost kernel: efi: UEFI not found.
Jun 24 09:04:12 localhost kernel: cma: Reserved 48 MiB at 0x9c800000