Yes, this actually works. The Nest Wifi point can be set up as a standalone Google Assistant / Cast speaker — much like a Nest Audio — without joining a Nest Wifi mesh and without broadcasting any Wi-Fi of its own. It joins your existing Wi-Fi (any router, doesn't need to be Google) like any other Cast device.
A lot of community advice insists this isn't possible, that you need a separate Google Home structure, or that it only fails with a "cannot connect to router" error. That advice is incomplete. The procedure below has been confirmed to work; the key is declining the pairing prompt with the base router, which exposes a "set up as a standalone speaker" branch that the setup flow otherwise hides.
Credit to the original discussion that documented this approach: https://www.reddit.com/r/GoogleWiFi/comments/mb173e/google_nest_wifi_point_as_speaker_only/.