In order to authenticate to a private GCS bucket, we need to connect the Cloud CDN load balancer through a Backend Service, instead of a Backend Bucket. See Configure private origin authentication for more details.
Kudos to Navya Dwarakanath for her great article https://medium.com/@thetechbytes/private-gcs-bucket-access-through-google-cloud-cdn-430d940ebad9.