Why we don’t need to scale up controller of a CRD (add multiple instances of the operator)?
All core controllers in Kubernetes run single instance. If Kubernetes can do it, our operator can do too as long as we write our code carefully. Anything that is going to overload a well-written controller is also very likely to overload etcd itself. Controllers should never be doing heavy work themselves, they just orchestrate and control. Control part is usually quite fast, complicated but not CPU intensive itself. The operator control things but the actual heavy lifting (for example, DB migration by operator) should happen elsewhere in a Job or similar. Some heavy lifting can also be divided up like for example “call a create API and then wait for it to finish”, then return immediately after calling the API with RequeueAfter a delay. So it is not sitting there blocking on something slow.
Credit: @coderanger in kubebuilder slack channel