Skip to content

Instantly share code, notes, and snippets.

@gebi

gebi/tubearchivist_notes-restore-from-backup-retry-upgrade-0_4_13-0_5_7.md

Created October 12, 2025 23:23
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save gebi/ad76f79a6173ac5fafafbcf5c558dce1 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save gebi/ad76f79a6173ac5fafafbcf5c558dce1 to your computer and use it in GitHub Desktop.
Download ZIP
Notes about tubearchivist restore from backup after failed upgrade 0.4.13 to 0.5.7 including full upgrade to 0.5.7
Raw
tubearchivist_notes-restore-from-backup-retry-upgrade-0_4_13-0_5_7.md

PLAN

  • upgrade went wrong, no idea why
  • retry => restore from backup

restore from backup and get 0.4.13 working again {{{

make working with snapshots nicer

zfs set snapdir=visible zfast

restore from backup

# rsync -a -x -H --no-inc-recursive --numeric-ids -vh --delete --stats ../.zfs/snapshot/autosnap_2025-09-29_09\:26\:31_daily/tubearchivist/ ./ 2>&1 |tee >../tubearchivist-new/restore_backup_log

Number of files: 20,307 (reg: 19,919, dir: 383, link: 5)
Number of created files: 773 (reg: 737, dir: 36)
Number of deleted files: 1,501 (reg: 1,445, dir: 56)
Number of regular files transferred: 3,307
Total file size: 1.66G bytes
Total transferred file size: 257.89M bytes
Literal data: 257.89M bytes
Matched data: 0 bytes
File list size: 589.78K
File list generation time: 0.269 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 258.71M
Total bytes received: 162.11K

sent 258.71M bytes  received 162.11K bytes  103.55M bytes/sec
total size is 1.66G  speedup is 6.42

get old elasticsearch and tag it (recommend TA actually publishes it in their docker-hub repo)

docker tag 8fc38801642d bbilly1/tubearchivist-es:8.16.0

use the following images in docker-compose

# grep image docker-compose.yml
image: ta-helper
image: bbilly1/tubearchivist:v0.4.13
image: redis/redis-stack-server
image: bbilly1/tubearchivist-es:8.16.0

start up again

# docker compose up --force-recreate --renew-anon-volumes -d
[+] Running 4/5
 ⠇ Network tubearchivist_default  Created                                                                                                  1.8s
 ✔ Container archivist-es         Started                                                                                                  0.9s
 ✔ Container ta-helper            Started                                                                                                  0.9s
 ✔ Container archivist-redis      Started                                                                                                  1.3s
 ✔ Container tubearchivist        Started

WORKS :) }}}

migrate to 0.5.0 {{{

https://github.com/tubearchivist/tubearchivist/releases/tag/v0.5.0

start 0.4.13 again and backup migration file

docker compose up --force-recreate --renew-anon-volumes -d
docker compose exec -it tubearchivist python manage.py ta_config_backup
ls -lah data-cache/backup/
    -rw-r--r-- 1 root root 724 Okt 12 22:41 migration.json
rm -f data-cache/db.sqlite3

check deps

ta:v0.5.0
es: archivist repo does not hold 8.17.2 anymore, use official elasticsearch one
redis, use without version

config changes

diff -Naup ../.zfs/snapshot/autosnap_2025-09-29_09\:26\:31_daily/tubearchivist/docker-compose.yml ./docker-compose.yml
--- ../.zfs/snapshot/autosnap_2025-09-29_09:26:31_daily/tubearchivist/docker-compose.yml        2025-02-09 22:50:03.944790371 +0100
+++ ./docker-compose.yml        2025-10-12 23:05:13.670035399 +0200
@@ -35,7 +35,7 @@ services:
   tubearchivist:
	 container_name: tubearchivist
	 restart: unless-stopped
-    image: bbilly1/tubearchivist
+    image: bbilly1/tubearchivist:v0.5.0
	 ports:
	   - 8000:8000
	 volumes:
@@ -43,10 +43,10 @@ services:
	   - ./data-cache:/cache
	 environment:
	   - ES_URL=http://archivist-es:9200     # needs protocol e.g. http and port
-      - REDIS_HOST=archivist-redis          # don't add protocol
+      - REDIS_CON=redis://archivist-redis:6379
	   - HOST_UID=1000
	   - HOST_GID=1000
-      - TA_HOST=y                           # set your host name
+      - TA_HOST=http://y:8000               # set your host name with protocol and port
	   - TA_USERNAME=tubearchivist           # your initial TA credentials
	   - TA_PASSWORD=xxx                     # your initial TA credentials
	   - ELASTIC_PASSWORD=xxx                                    # set password for Elasticsearch
@@ -63,7 +63,7 @@ services:
	   - archivist-redis
	   - ta-helper
   archivist-redis:
-    image: redis/redis-stack-server
+    image: redis
	 container_name: archivist-redis
	 restart: unless-stopped
	 expose:
@@ -73,7 +73,8 @@ services:
	 depends_on:
	   - archivist-es
   archivist-es:
-    image: bbilly1/tubearchivist-es         # only for amd64, or use official es 8.14.3
+    image: elasticsearch:8.17.2
	 container_name: archivist-es
	 restart: unless-stopped
	 environment:

start

docker compose up --force-recreate --renew-anon-volumes -d

ta start error

tubearchivist    | [5] check for first run after update
tubearchivist    | Traceback (most recent call last):
tubearchivist    |   File "/app/manage.py", line 23, in <module>
tubearchivist    |     main()
tubearchivist    |   File "/app/manage.py", line 19, in main
tubearchivist    |     execute_from_command_line(sys.argv)
tubearchivist    |   File "/root/.local/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
tubearchivist    |     utility.execute()
tubearchivist    |   File "/root/.local/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
tubearchivist    |     self.fetch_command(subcommand).run_from_argv(self.argv)
tubearchivist    |   File "/root/.local/lib/python3.11/site-packages/django/core/management/base.py", line 413, in run_from_argv
tubearchivist    |     self.execute(*args, **cmd_options)
tubearchivist    |   File "/root/.local/lib/python3.11/site-packages/django/core/management/base.py", line 459, in execute
tubearchivist    |     output = self.handle(*args, **options)
tubearchivist    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tubearchivist    |   File "/app/config/management/commands/ta_startup.py", line 49, in handle
tubearchivist    |     self._version_check()
tubearchivist    |   File "/app/config/management/commands/ta_startup.py", line 134, in _version_check
tubearchivist    |     new_version = ReleaseVersion().is_updated()
tubearchivist    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tubearchivist    |   File "/app/appsettings/src/config.py", line 236, in is_updated
tubearchivist    |     message = self.get_update()
tubearchivist    |               ^^^^^^^^^^^^^^^^^
tubearchivist    |   File "/app/appsettings/src/config.py", line 251, in get_update
tubearchivist    |     message = RedisArchivist().get_message_dict(self.NEW_KEY)
tubearchivist    |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tubearchivist    |   File "/app/common/src/ta_redis.py", line 75, in get_message_dict
tubearchivist    |     reply = self.conn.execute_command("GET", self.NAME_SPACE + key)
tubearchivist    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tubearchivist    |   File "/root/.local/lib/python3.11/site-packages/redis/client.py", line 559, in execute_command
tubearchivist    |     return self._execute_command(*args, **options)
tubearchivist    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tubearchivist    |   File "/root/.local/lib/python3.11/site-packages/redis/client.py", line 567, in _execute_command
tubearchivist    |     return conn.retry.call_with_retry(
tubearchivist    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
tubearchivist    |   File "/root/.local/lib/python3.11/site-packages/redis/retry.py", line 62, in call_with_retry
tubearchivist    |     return do()
tubearchivist    |            ^^^^
tubearchivist    |   File "/root/.local/lib/python3.11/site-packages/redis/client.py", line 568, in <lambda>
tubearchivist    |     lambda: self._send_command_parse_response(
tubearchivist    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tubearchivist    |   File "/root/.local/lib/python3.11/site-packages/redis/client.py", line 542, in _send_command_parse_response
tubearchivist    |     return self.parse_response(conn, command_name, **options)
tubearchivist    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tubearchivist    |   File "/root/.local/lib/python3.11/site-packages/redis/client.py", line 584, in parse_response
tubearchivist    |     response = connection.read_response()
tubearchivist    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^
tubearchivist    |   File "/root/.local/lib/python3.11/site-packages/redis/connection.py", line 616, in read_response
tubearchivist    |     raise response
tubearchivist    | redis.exceptions.ResponseError: WRONGTYPE Operation against a key holding the wrong kind of value
tubearchivist exited with code 1

execute this to remove the WRONGTYPE operation error

# docker compose exec -it archivist-redis redis-cli DEL ta:versioncheck:new
(integer) 1

restore backup

# docker compose exec -it tubearchivist python manage.py ta_config_restore
restore users and schedules
	✓ json file found: /cache/backup/migration.json
delete existing users
recreate users
	✓ recreated user with name: tubearchivist
delete existing schedules
recreate schedules
	✓ recreated schedule: version_check: 58 12 * * * (m/h/dM/MY/d) Europe/Vienna
	✓ recreated schedule: download_pending: 0 3 * * */3 (m/h/dM/MY/d) Europe/Vienna
	✓ recreated schedule: update_subscribed: 0 2 * * */3 (m/h/dM/MY/d) Europe/Vienna
	✓ recreated schedule: thumbnail_check: 0 6 * * */3 (m/h/dM/MY/d) Europe/Vienna
	✓ recreated schedule: check_reindex: 0 5 * * */3 (m/h/dM/MY/d) Europe/Vienna
	✓ restore completed. Please restart the container.

WORKS :)

reset redis data

docker compose down
rm -f data-redis/dump.rdb

}}}

upgrade to 0.5.1 {{{

https://github.com/tubearchivist/tubearchivist/releases/tag/v0.5.1

check deps

ta: v0.5.1
es: same as with ta:v0.5.1, needs es 8.17.2, not available in ta-es, thus keep using elasticsearch:8.17.2
redis: just latest

migration

  • Health check endpoint is now at /api/health/

config diff

# diff -Naurp docker-compose_v0.5.0.yml docker-compose_v0.5.1.yml
--- docker-compose_v0.5.0.yml   2025-10-12 23:30:16.535449137 +0200
+++ docker-compose_v0.5.1.yml   2025-10-12 23:29:13.206716486 +0200
@@ -35,7 +35,7 @@ services:
   tubearchivist:
	 container_name: tubearchivist
	 restart: unless-stopped
-    image: bbilly1/tubearchivist:v0.5.0
+    image: bbilly1/tubearchivist:v0.5.1
	 ports:
	   - 8000:8000
	 volumes:
@@ -53,7 +53,7 @@ services:
	   - TZ=Europe/Vienna                    # set your time zone
	  #- DJANGO_DEBUG=True                   # enable debug mode, don't forget to disable!
	 healthcheck:
-      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
+      test: ["CMD", "curl", "-f", "http://localhost:8000/api/health"]
	   interval: 2m
	   timeout: 10s
	   retries: 3

WORKS :) }}}

upgrade to 0.5.2 {{{

https://github.com/tubearchivist/tubearchivist/releases/tag/v0.5.2

check deps

upgrade to ta-es 8.18.0

config diff

diff -Naup docker-compose_v0.5.1.yml docker-compose.yml
--- docker-compose_v0.5.1.yml   2025-10-12 23:29:13.206716486 +0200
+++ docker-compose.yml  2025-10-12 23:41:01.946859091 +0200
@@ -35,7 +35,7 @@ services:
   tubearchivist:
	 container_name: tubearchivist
	 restart: unless-stopped
-    image: bbilly1/tubearchivist:v0.5.1
+    image: bbilly1/tubearchivist:v0.5.2
	 ports:
	   - 8000:8000
	 volumes:
@@ -73,8 +73,7 @@ services:
	 depends_on:
	   - archivist-es
   archivist-es:
-    image: elasticsearch:8.17.2
+    image: bbilly1/tubearchivist-es:8.18.0         # only for amd64, or use official es 8.18.0
	 container_name: archivist-es
	 restart: unless-stopped
	 environment:

WORKS :) }}}

upgrade to 0.5.3 {{{

https://github.com/tubearchivist/tubearchivist/releases/tag/v0.5.3

deps

no updates

upgrade notes

triggers an index rebuild, should be mostly automatic ;)

config diff

# diff -Naurp docker-compose_v0.5.2.yml docker-compose.yml
--- docker-compose_v0.5.2.yml   2025-10-12 23:41:17.035031543 +0200
+++ docker-compose.yml  2025-10-12 23:50:12.085136634 +0200
@@ -35,7 +35,7 @@ services:
   tubearchivist:
	 container_name: tubearchivist
	 restart: unless-stopped
-    image: bbilly1/tubearchivist:v0.5.2
+    image: bbilly1/tubearchivist:v0.5.3
	 ports:
	   - 8000:8000
	 volumes:

mapping changes in logs

tubearchivist    | detected mapping change: published, {'type': 'date', 'format': 'epoch_second||strict_date_optional_time'}
tubearchivist    | snapshot: executing now: {'snapshot_name': 'ta_daily_-6lcifh8vq4qwpb_pljfckw'}

WORKS :) }}}

upgrade to 0.5.4 {{{

https://github.com/tubearchivist/tubearchivist/releases/tag/v0.5.4

deps

no updates

config diff

# diff -Naurp docker-compose_v0.5.3.yml docker-compose.yml
--- docker-compose_v0.5.3.yml   2025-10-12 23:50:22.409254616 +0200
+++ docker-compose.yml  2025-10-13 00:01:57.609262242 +0200
@@ -35,7 +35,7 @@ services:
   tubearchivist:
	 container_name: tubearchivist
	 restart: unless-stopped
-    image: bbilly1/tubearchivist:v0.5.3
+    image: bbilly1/tubearchivist:v0.5.4
	 ports:
	   - 8000:8000
	 volumes:

WORKS :) }}}

upgrade to 0.5.5 {{{

https://github.com/tubearchivist/tubearchivist/releases/tag/v0.5.4

deps

update to ta-es:8.18.2 (which is tag latest, thus we can finally remove the explicite tag alltogether)

upgrade notes

execute cmd to get optimiezd channel behaviour immediately

# docker compose exec -it tubearchivist python manage.py ta_index_channel_tabs
...
	✓ updated 'Foo Bar Blub' tabs: ['videos']
xxx-xxxxxxxxxxxx: get metadata from es
ERROR: [youtube:tab] xxx-xxxxxxxxxxxx: This channel does not have a streams tab
	✓ updated 'FooFoo' tabs: ['videos', 'shorts']
...

config diff

# diff -Naurp docker-compose_v0.5.4.yml docker-compose.yml
--- docker-compose_v0.5.4.yml   2025-10-13 00:02:22.929552352 +0200
+++ docker-compose.yml  2025-10-13 00:10:47.643325230 +0200
@@ -35,7 +35,7 @@ services:
   tubearchivist:
	 container_name: tubearchivist
	 restart: unless-stopped
-    image: bbilly1/tubearchivist:v0.5.4
+    image: bbilly1/tubearchivist:v0.5.5
	 ports:
	   - 8000:8000
	 volumes:
@@ -73,7 +73,7 @@ services:
	 depends_on:
	   - archivist-es
   archivist-es:
-    image: bbilly1/tubearchivist-es:8.18.0         # only for amd64, or use official es 8.14.3
+    image: bbilly1/tubearchivist-es                # only for amd64, or use official es 8.14.3
	 container_name: archivist-es
	 restart: unless-stopped
	 environment:

WORKS :) }}}

upgrade to 0.5.6 {{{

https://github.com/tubearchivist/tubearchivist/releases/tag/v0.5.6

deps

no new dpes

upgrade notes

yt foo, search with new filter options for 360p videos and re-download (after upgraded to ta:latest!)

config diff

# diff -Naurp docker-compose_v0.5.5.yml docker-compose.yml
--- docker-compose_v0.5.5.yml   2025-10-13 00:11:02.475494676 +0200
+++ docker-compose.yml  2025-10-13 00:22:07.711089106 +0200
@@ -35,7 +35,7 @@ services:
   tubearchivist:
	 container_name: tubearchivist
	 restart: unless-stopped
-    image: bbilly1/tubearchivist:v0.5.5
+    image: bbilly1/tubearchivist:v0.5.6
	 ports:
	   - 8000:8000
	 volumes:

WORKS :) }}}

upgrade to 0.5.7 {{{

https://github.com/tubearchivist/tubearchivist/releases/tag/v0.5.7

deps

back on ta:latest, remove pinned tag/version

upgrade notes

execute search and redownload of incorrectly downloaded 360p videos due to yt foo (from 0.5.6 changelog!) (found 48 videos, select / download worked nicely, thx :)!

config diff

# diff -Naurp docker-compose_v0.5.6.yml docker-compose.yml
--- docker-compose_v0.5.6.yml   2025-10-13 00:22:22.291255499 +0200
+++ docker-compose.yml  2025-10-13 00:30:55.896894660 +0200
@@ -35,7 +35,7 @@ services:
   tubearchivist:
     container_name: tubearchivist
     restart: unless-stopped
-    image: bbilly1/tubearchivist:v0.5.6
+    image: bbilly1/tubearchivist
     ports:
       - 8000:8000
     volumes:

WORKS :) }}}

@2q2code
Copy link

2q2code commented Oct 19, 2025

Thank you so much for this walkthrough. My cat will now have a working instance to watch cat tv from. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment