For what it's worth (and with all the usual disclaimers about potentially making your mac unstable by disabling system services), here's some commands that will manipulate this service and services like it. Note the $UID in the command, that's just a bash shell variable that will resolve to some number. That's your numeric UID. You just run these commands from a Terminal command line. No special privileges needed.
If you want to disable it entirely, the first command stops it from respawning, and the second kills the one that is currently running:
launchctl disable gui/$UID/com.apple.photoanalysisd
launchctl kill -TERM gui/$UID/com.apple.photoanalysisd
(If you kill it without disabling it will die, but a new one will respawn and pick up where the old one left off)
I don't have this problem myself, so I can't try these next two commands. They're relying on good ole UNIX signals. You could theoretically suspend and resume the process like this ("STOP" and "CONT" are stop and continue):
launchctl kill -STOP gui/$UID/com.apple.photoanalysisd
launchctl kill -CONT gui/$UID/com.apple.photoanalysisd
I don't know what launchd does when running processes are suspended for a long time. Will it detect them as dead and kill and restart them? I dunno. But I do know they won't get any CPU time.
I have 0 (ZERO) photos in my library. I don't use photos or even iCloud Drive for photos storage in any way. Even after reinstall Catalina the problema persists. This thread are kept in my favorites forever.
So many thanks to @huksley , the sub reddit and stackexchange.