manics/README.md

Last active May 20, 2026 09:36

Star (0) You must be signed in to star a gist
Fork (0) You must be signed in to fork a gist

Select an option

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/manics/4c3e02884ef335014d395bfa524258ec.js"></script>
Save manics/4c3e02884ef335014d395bfa524258ec to your computer and use it in GitHub Desktop.

Download ZIP

README.md

Raw

Can an LLM control an application running in a VDI?

Run a Linux desktop VDI that can be accessed through a browser. For example

docker run -it --rm -p127.0.0.1:8888:8888 ghcr.io/manics/jupyter-desktop-mate jupyter-lab --IdentityProvider.token=""

This will run a pre-built image from https://github.com/manics/jupyter-desktop-mate/ that provides a noVNC web interface to a Ubuntu MATE Desktop. Jupyter's token authentication is disabled for convenience but could be easily enabled.

Launch an AI tool that includes a browser integration, for instance Google Antigravity. Ensure that browser support is enabled.

Give this prompt to Antigravity:

http://localhost:8888/desktop/ is a Ubuntu MATE desktop running in a Docker container with a noVNC server.

Use the Antigravity browser to open this page, and check a desktop is visible. Launch a terminal by clicking on the terminal icon on the desktop. Write echo Hello World in the terminal and press enter. Check that Hello World is printed in the terminal.

and set it going!

Example recording:

What actually happens:

Antigravity can access the Document object model, but the desktop appears as a HTML Canvas element so it can't directly manipulate anything on the desktop.

Instead it takes multiple screenshots, interprets them, and works out where to click or enter key presses.

Note that although the prompt said to click on the terminal desktop icon it decided to navigate the menu instead.

Raw

vdi_hello_world_1779226127097.webp

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment