Skip to content

Instantly share code, notes, and snippets.

@KaKi87
Last active December 11, 2024 17:21
Show Gist options
  • Save KaKi87/f620788e9901abbfef4978eb7ad358b4 to your computer and use it in GitHub Desktop.
Save KaKi87/f620788e9901abbfef4978eb7ad358b4 to your computer and use it in GitHub Desktop.
dynapt experimental setup tutorial

dynapt — Experimental setup tutorial

Introduction

Welcome to the experimental setup tutorial for dynapt, the dynamic APT repository.

If you're reading this, you might be frustrated that an app you're using doesn't offer automated updates, by way of an APT repository, a PPA, or a Flatpak.

The first one of those options consists in creating a file server hosting DEB files to be downloaded by the APT package manager.

Which is what dynapt does, but on your local machine.

Additionally, it comes with the ability to wrap AppImage files into DEBs, and will in the future handle bare binaries.

Installation

dynapt is available as DEBs.

Download & install automatically :

wget https://git.kaki87.net/KaKi87/dynapt/releases/download/0.0.0-dev.af3fa0a/dynapt-$(dpkg --print-architecture).deb && \
sudo apt install ./dynapt-$(dpkg --print-architecture).deb

Or manually for amd64 (aka. x86_64) or arm64 (aka. aarch64).

General configuration

Now that dynapt is installed at /opt/dynapt, create a config.json file there : nano /opt/dynapt/config.json

And paste the following :

{
    "port": 3000,
    "concurrency": null,
    "apps": []
}

You may change the port number if 3000 is already used on your device (if you don't know then likely not).

This configuration will by default download & package all apps simultaneously for faster apt update execution. If your device has low power or low bandwith, change concurrency to 1 to only allow one downloading & packaging at a time (or a little more if you'd like).

Then, add dynapt as an APT repository (don't forget to change the port number if you previously did) :

echo "deb [trusted=yes] http://127.0.0.1:3000/ /" | sudo tee /etc/apt/sources.list.d/dynapt.list

And start it :

nohup dynapt &

Finally, make it start automatically at boot using cron :

crontab -l | { cat; echo "@reboot dynapt"; } | crontab -

App-specific configuration

Configuration for already tested apps are documented on a related issue/discussion, which you probably come from :

For the generic information :

  • apps is a JSON array that takes the following item properties :
    • name (string) : the name of the app ;
    • url (string) : direct download to a DEB or AppImage ;
      or
    • github (object) : GitHub release download info ;
      • repo (string) : repository in owner/name format ;
      • filter (string) : optional file name filter when multiple DEB/AppImage files available (e.g. for variant or architecture) ;
    • appimage (boolean or object) : set to true when using AppImage instead of DEB ;
      • bin (string) : optional command name to make available in /usr/bin ;
    • id (string) : auto-generated UUID, not to be provided by users, used to manage caching.

Troubleshooting

Logs

Get your latest log entry from /opt/dynapt/dynapt.log in addition to your APT command output to get help.

Clean cache

Run dynapt clean to remove all previously downloaded & packaged files, then run sudo apt update to re-download them again.

Manually update

To debug download issues without updating APT, run dynapt update.

To update APT only for the dynapt repository, run sudo apt update -o Dir::Etc::sourcelist="sources.list.d/dynapt.list" -o Dir::Etc::sourceparts="-" -o APT::Get::List-Cleanup="0".

List packages

Run dynapt list to check the status of packages handled by dynapt.

Future updates

dynapt will be able to provide its own upgrades to APT as soon as released as stable. Until then, updates will have to be done manually. To get notified, subscribe to this gist or watch the self-hosted repository.

This manual setup will no longer be required once dynapt is stable, and will offer an optional step-by-step wizard instead.

Further support

Support will be provided to commenters here for as long as dynapt will remain experimental, since promotion occurs here, but questions and bug reports and feature requests are ideally welcome on the self-hosted repository, and will only be handled there as soon as dynapt is released as stable.

Proprietary apps

Here are some tested configurations for proprietary apps.

        {
            "name": "Discord",
            "url": "https://discord.com/api/download?platform=linux&format=deb"
        }
        {
            "name": "Free Download Manager",
            "url": "https://files2.freedownloadmanager.org/6/latest/freedownloadmanager.deb"
        }

Differences with deb-get & AM

deb-get is a package manager that fetches DEB files and AM is a package manager that fetches AppImage files.

Their fetching works similarly to dynapt's, but the difference is dynapt isn't a package manager : it interfaces with APT instead, letting you keep using your usual tools, whether that's apt, aptitude, Synaptic, your desktop environment's app store, etc.


Thank you for reading and for using dynapt ❤️

@JoeJoeTV
Copy link

JoeJoeTV commented Dec 7, 2024

Sorry to bother you with so many questions, but is there currently something implemented which can deal with direct links to deb packages that contain their version number?
For example openrgb: https://openrgb.org/releases/release_0.9/openrgb_0.9_amd64_bookworm_b5f46e3.deb
I don't think there is a link that just points to the newest deb file.

@JoeJoeTV
Copy link

JoeJoeTV commented Dec 7, 2024

In addition, I set up a list of apps that I would like to try out with dynapt:

    "apps": [
        {
            "name": "balena-etcher",
            "github": {
                "repo": "balena-io/etcher",
                "filter": "balena-etcher_*amd64*"
            }
        },
        {
            "name": "bat",
            "github": {
                "repo": "sharkdp/bat",
                "filter": "bat_*amd64*"
            }
        },
        {
            "name": "blockbench",
            "github": {
                "repo": "JannisX11/blockbench",
                "filter": "Blockbench_*"
            }
        },
        {
            "name": "discord",
            "url": "https://discord.com/api/download?platform=linux&format=deb"
        },
        {
            "name": "draw.io",
            "github": {
                "repo": "jgraph/drawio-desktop",
                "filter": "drawio-amd64-*"
            }
        },
        {
            "name": "ferdium",
            "github": {
                "repo": "ferdium/ferdium-app",
                "filter": "Ferdium-linux-*-amd64*"
            }
        },
        {
            "name": "heroic",
            "github": {
                "repo": "Heroic-Games-Launcher/HeroicGamesLauncher",
                "filter": "heroic_*_amd64*"
            }
        },
        {
            "name": "imhex",
            "github": {
                "repo": "WerWolv/ImHex",
                "filter": "imhex-*-Ubuntu-24.04-x86_64*"
            }
        },
        {
            "name": "jellyfin-media-player",
            "github": {
                "repo": "jellyfin/jellyfin-media-player",
                "filter": "jellyfin-media-player_*_amd64-noble*"
            }
        },
        {
            "name": "localsend",
            "github": {
                "repo": "localsend/localsend",
                "filter": "LocalSend-*-linux-x86-64*"
            }
        },
        {
            "name": "lsd",
            "github": {
                "repo": "lsd-rs/lsd",
                "filter": "lsd_*_amd64*"
            }
        },
        {
            "name": "openrgb",
            "url": "https://openrgb.org/releases/release_0.9/openrgb_0.9_amd64_bookworm_b5f46e3.deb"
        },
        {
            "name": "rustdesk",
            "github": {
                "repo": "rustdesk/rustdesk",
                "filter": "rustdesk-*-x86_64*"
            }
        },
        {
            "name": "teams-for-linux",
            "github": {
                "repo": "IsmaelMartinez/teams-for-linux",
                "filter": "teams-for-linux_*_amd64*"
            }
        },
        {
            "name": "weylus",
            "github": {
                "repo": "H-M-H/Weylus",
                "filter": "Weylus_*_amd64*"
            }
        },
        {
            "name": "wasistlos",
            "github": {
                "repo": "xeco23/WasIstLos",
                "filter": "wasistlos_*_amd64*"
            }
        },
        {
            "name": "youtube-music",
            "github": {
                "repo": "th-ch/youtube-music",
                "filter": "youtube-music_*_amd64*"
            }
        }
    ]

With that list in the config and dynapt running, I get the following error while performing an apt update:

E: Encountered a section with no Package: header
E: Problem with MergeList /var/lib/apt/lists/127.0.0.1:3000_Packages
E: The package lists or status file could not be parsed or opened.

The output from dynapt after this is the following:

    [7.12.2024, 19:58:50] Listening to localhost:3000
    [7.12.2024, 19:59:06] GET /InRelease
    [7.12.2024, 19:59:06] GET /Release
    [7.12.2024, 19:59:06] · Updating packages
    [7.12.2024, 19:59:06] ├── Fetching 'balena-etcher'
    [7.12.2024, 19:59:06] ├── Fetching 'bat'
    [7.12.2024, 19:59:06] ├── Fetching 'blockbench'
    [7.12.2024, 19:59:06] ├── Fetching 'discord'
    [7.12.2024, 19:59:06] ├── Fetching 'draw.io'
    [7.12.2024, 19:59:06] ├── Fetching 'ferdium'
    [7.12.2024, 19:59:06] ├── Fetching 'heroic'
    [7.12.2024, 19:59:06] ├── Fetching 'imhex'
    [7.12.2024, 19:59:06] ├── Fetching 'jellyfin-media-player'
    [7.12.2024, 19:59:06] ├── Fetching 'localsend'
    [7.12.2024, 19:59:06] ├── Fetching 'lsd'
    [7.12.2024, 19:59:06] ├── Fetching 'openrgb'
    [7.12.2024, 19:59:06] ├── Fetching 'rustdesk'
    [7.12.2024, 19:59:06] ├── Fetching 'teams-for-linux'
    [7.12.2024, 19:59:07] ├── Fetching 'weylus'
    [7.12.2024, 19:59:07] ├── Fetching 'wasistlos'
    [7.12.2024, 19:59:07] ├── Fetching 'youtube-music'
    [7.12.2024, 19:59:07] ├── Fetched 'blockbench'
    [7.12.2024, 19:59:07] ├── Downloading 'blockbench' (file #1)
    [7.12.2024, 19:59:07] ├── Fetched 'wasistlos'
    [7.12.2024, 19:59:07] ├── Fetched 'balena-etcher'
    [7.12.2024, 19:59:07] ├── Fetched 'ferdium'
    [7.12.2024, 19:59:07] ├── Downloading 'wasistlos' (file #1)
    [7.12.2024, 19:59:07] ├── Downloading 'balena-etcher' (file #1)
    [7.12.2024, 19:59:07] ├── Downloading 'ferdium' (file #1)
    [7.12.2024, 19:59:07] ├── Fetched 'jellyfin-media-player'
    [7.12.2024, 19:59:07] ├── Downloading 'jellyfin-media-player' (file #1)
    [7.12.2024, 19:59:07] ├── Fetched 'heroic'
    [7.12.2024, 19:59:07] ├── Fetched 'bat'
    [7.12.2024, 19:59:07] ├── Downloading 'heroic' (file #1)
    [7.12.2024, 19:59:07] ├── Downloading 'bat' (file #1)
    [7.12.2024, 19:59:07] ├── Fetched 'weylus'
    [7.12.2024, 19:59:07] ├── Downloading 'weylus' (file #1)
    [7.12.2024, 19:59:07] ├── Fetched 'localsend'
    [7.12.2024, 19:59:07] ├── Fetched 'teams-for-linux'
    [7.12.2024, 19:59:07] ├── Downloading 'localsend' (file #1)
    [7.12.2024, 19:59:07] ├── Downloading 'teams-for-linux' (file #1)
    [7.12.2024, 19:59:07] ├── Fetched 'rustdesk'
    [7.12.2024, 19:59:07] ├── Fetched 'lsd'
    [7.12.2024, 19:59:07] ├── Fetched 'youtube-music'
    [7.12.2024, 19:59:07] ├── Fetched 'draw.io'
    [7.12.2024, 19:59:07] ├── Downloading 'rustdesk' (file #1)
    [7.12.2024, 19:59:07] ├── Downloading 'lsd' (file #2)
    [7.12.2024, 19:59:07] ├── Downloading 'draw.io' (file #1)
    [7.12.2024, 19:59:07] ├── Downloading 'rustdesk' (file #2)
    [7.12.2024, 19:59:07] ├── Downloading 'lsd' (file #1)
    [7.12.2024, 19:59:07] ├── Downloading 'youtube-music' (file #1)
    [7.12.2024, 19:59:07] ├── Fetched 'imhex'
    [7.12.2024, 19:59:07] ├── Downloading 'imhex' (file #1)
    [7.12.2024, 19:59:07] ├── Fetched 'discord'
    [7.12.2024, 19:59:07] ├── Downloading 'discord' (file #1)
    [7.12.2024, 19:59:07] ├── Fetched 'openrgb'
    [7.12.2024, 19:59:07] ├── Downloading 'openrgb' (file #1)
    [7.12.2024, 19:59:10] ├── Downloaded 'wasistlos' (file #1)
                          │   Generating package
    [7.12.2024, 19:59:10] ├── Package generated for 'wasistlos' (file #1)
    [7.12.2024, 19:59:11] ├── Downloaded 'jellyfin-media-player' (file #1)
                          │   Generating package
    [7.12.2024, 19:59:11] ├── Package generated for 'jellyfin-media-player' (file #1)
    [7.12.2024, 19:59:12] ├── Downloaded 'openrgb' (file #1)
                          │   Generating package
    [7.12.2024, 19:59:12] ├── Package generated for 'openrgb' (file #1)
    [7.12.2024, 19:59:19] ├── Downloaded 'lsd' (file #1)
                          │   Generating package
    [7.12.2024, 19:59:19] ├── Package generated for 'lsd' (file #1)
    [7.12.2024, 19:59:20] ├── Downloaded 'lsd' (file #2)
                          │   Generating package
    [7.12.2024, 19:59:20] ├── Package generated for 'lsd' (file #2)
    [7.12.2024, 19:59:26] ├── Downloaded 'bat' (file #1)
                          │   Generating package
    [7.12.2024, 19:59:26] ├── Package generated for 'bat' (file #1)
    [7.12.2024, 19:59:36] GET /Release
    [7.12.2024, 20:00:08] GET /Release
    [7.12.2024, 20:00:14] ├── Downloaded 'discord' (file #1)
                          │   Generating package
    [7.12.2024, 20:00:14] ├── Downloaded 'weylus' (file #1)
                          │   Generating package
    [7.12.2024, 20:00:14] ├── Package generated for 'discord' (file #1)
    [7.12.2024, 20:00:15] ├── Package generated for 'weylus' (file #1)
    [7.12.2024, 20:00:31] ├── Downloaded 'localsend' (file #1)
                          │   Generating package
    [7.12.2024, 20:00:31] ├── Package generated for 'localsend' (file #1)
    [7.12.2024, 20:00:37] ├── Downloaded 'rustdesk' (file #1)
                          │   Generating package
    [7.12.2024, 20:00:37] ├── Package generated for 'rustdesk' (file #1)
    [7.12.2024, 20:00:38] GET /Release
    [7.12.2024, 20:00:48] ├── Downloaded 'rustdesk' (file #2)
                          │   Generating package
    [7.12.2024, 20:00:48] ├── Package generated for 'rustdesk' (file #2)
    [7.12.2024, 20:01:10] GET /Release
    [7.12.2024, 20:01:40] GET /Release
    [7.12.2024, 20:02:14] GET /Release
    [7.12.2024, 20:02:44] GET /Release
    [7.12.2024, 20:03:14] GET /Packages.xz
    [7.12.2024, 20:03:14] GET /de.xz
    [7.12.2024, 20:03:14] GET /en.xz
    [7.12.2024, 20:03:14] GET /Packages.bz2
    [7.12.2024, 20:03:14] GET /de.bz2
    [7.12.2024, 20:03:14] GET /en.bz2
    [7.12.2024, 20:03:14] GET /Packages.lzma
    [7.12.2024, 20:03:14] GET /de.lzma
    [7.12.2024, 20:03:14] GET /en.lzma
    [7.12.2024, 20:03:14] GET /Packages.gz
    [7.12.2024, 20:03:14] GET /de.gz
    [7.12.2024, 20:03:14] GET /en.gz
    [7.12.2024, 20:03:14] GET /Packages.lz4
    [7.12.2024, 20:03:14] GET /de.lz4
    [7.12.2024, 20:03:14] GET /en.lz4
    [7.12.2024, 20:03:14] GET /Packages.zst
    [7.12.2024, 20:03:14] GET /de.zst
    [7.12.2024, 20:03:14] GET /en.zst
    [7.12.2024, 20:03:14] GET /Packages
    [7.12.2024, 20:03:31] ├── Downloaded 'imhex' (file #1)
                          │   Generating package
    [7.12.2024, 20:03:31] ├── Package generated for 'imhex' (file #1)
    [7.12.2024, 20:03:44] GET /Packages
    [7.12.2024, 20:04:09] ├── Downloaded 'blockbench' (file #1)
                          │   Generating package
    [7.12.2024, 20:04:10] ├── Package generated for 'blockbench' (file #1)
    [7.12.2024, 20:04:14] GET /de
    [7.12.2024, 20:04:14] GET /en
    [7.12.2024, 20:04:15] GET /Packages
    [7.12.2024, 20:04:39] ├── Downloaded 'youtube-music' (file #1)
                          │   Generating package
    [7.12.2024, 20:04:40] ├── Package generated for 'youtube-music' (file #1)
    [7.12.2024, 20:04:45] GET /Packages
    [7.12.2024, 20:04:50] ├── Downloaded 'teams-for-linux' (file #1)
                          │   Generating package
    [7.12.2024, 20:04:50] ├── Downloaded 'ferdium' (file #1)
                          │   Generating package
    [7.12.2024, 20:04:50] ├── Package generated for 'teams-for-linux' (file #1)
    [7.12.2024, 20:04:51] ├── Package generated for 'ferdium' (file #1)
    [7.12.2024, 20:04:52] ├── Downloaded 'draw.io' (file #1)
                          │   Generating package
    [7.12.2024, 20:04:53] ├── Package generated for 'draw.io' (file #1)
    [7.12.2024, 20:05:04] ├── Downloaded 'heroic' (file #1)
                          │   Generating package
    [7.12.2024, 20:05:05] ├── Package generated for 'heroic' (file #1)
    [7.12.2024, 20:05:11] ├── Downloaded 'balena-etcher' (file #1)
                          │   Generating package
    [7.12.2024, 20:05:12] ├── Package generated for 'balena-etcher' (file #1)
    [7.12.2024, 20:05:12] └── Completed in 365669ms

@KaKi87
Copy link
Author

KaKi87 commented Dec 7, 2024

is there currently something implemented which can deal with direct links to deb packages that contain their version number?

There isn't.
The solution would be to scrape the page that contains the link to the latest release (e.g. openrgb.org/releases.html), but in this case it's not very readable (like a table with name and date), so detecting the latest link would be hard.

I don't think there is a link that just points to the newest deb file.

You're right, there isn't.

However, this app :

  • has a PPA ;
  • is available on Flathub ;
  • was last updated almost a year and a half ago.

So, I'd say it doesn't require dynapt, what do you think ?

With that list in the config and dynapt running, I get the following error while performing an apt update

Wow, that's a lot of packages.

As you can see, it took dynapt 6 minutes to generate load the packages, because it's the first time and that's a lot, so APT wasn't gonna wait for that long.

However, if you run it again, it should be way faster because of cache, and so it should work fine.

@JoeJoeTV
Copy link

JoeJoeTV commented Dec 7, 2024

So, I'd say it doesn't require dynapt, what do you think ?

Hmm, well I normally would set it up, so I can directly update when there is a new version, as the software is still developed, but if there is a PPA, I guess that would be the easier option, which I only now saw through your comment.

Wow, that's a lot of packages.

Really? I think I had more installed with deb-get. Many are useful tools that are just not in default repos, or I would like to have a newer version, so that's why I put them in there.

However, if you run it again, it should be way faster because of cache, and so it should work fine.

Yeah, that seems to have worked. So is dynapt only supposed to be used for a vers small amount of packages?

@KaKi87
Copy link
Author

KaKi87 commented Dec 7, 2024

No, I'm not expecting dynapt to have trouble with a higher number of packages to fetch, it's just that I didn't expect it, because despite all the time I'm investing on this project, most apps are available in a PPA, in an APT repository, on Flathub, Homebrew or Nix, and the remaining ones that are literally nowhere happen to be few, but ones maintained by developers who will never put in the effort of setting up automated updates, so here I am. ^^

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment