Unsigned VSTs etc get flagged by the MacOS Gatekeeper when Ableton scans the plugins. You can't do the right-click open trick to bypass an app from an unknown developer (since VSTs aren't applications), so do this instead:
sudo xattr -rd com.apple.quarantine /Library/Audio/Plug-Ins/VST/unsigned.vst
or
sudo xattr -rd com.apple.quarantine /Library/Audio/Plug-Ins/VST3/unsigned.vst
or
sudo xattr -rd com.apple.quarantine /Library/Audio/Plug-Ins/Components/unsigned.component
References: