Sometimes you just have to purge. With Xcode, it helps to completely blow away the Derived Data and SPM Cache directories. These directories contain all of the build artifacts produced by Xcode and resolved Swift packages. When a clean action does not completely reset your build it is time to purge.
These 2 functions can be added to your shell so you can simply run xcpurge to send Derived Data and SPM Cache to your Trash. You can recover it if necessary.
You likely are now using zsh with macOS and so you can update ~/.zshrc to define these functions. What I recommend is creating a folder at ~/.shell and create a file named ~/.shell/functions in that folder. Add the functions below to this file. Make sure to also run chmod u+x ~/.shell/functions and then add source $HOME/.shell/functions to ~/.zshrc so your shell has these functions defined. You can then run xcpurge or trash like any other command. The first time you run it you will need to grant permission since i