Skip to content

Instantly share code, notes, and snippets.

@danehans

danehans/create_fed_sa_notes.md

Last active August 29, 2018 17:58
Show Gist options
  • Save danehans/0a7ccafb6d9c66c11106912895684599 to your computer and use it in GitHub Desktop.
Save danehans/0a7ccafb6d9c66c11106912895684599 to your computer and use it in GitHub Desktop.
Download ZIP
k8s-fedv2-review
Raw
create_fed_sa_notes.md

Following: https://github.com/kubernetes-sigs/federation-v2/blob/master/docs/development.md

$ kubebuilder create resource --group core --version v1alpha1 --kind FederatedServiceAccount
Creating API files for you to edit...
Edit your API schema...
	pkg/apis/core/v1alpha1/federatedserviceaccount_types.go
	pkg/apis/core/v1alpha1/federatedserviceaccount_types_test.go
Creating controller ...
Edit your controller function...
	pkg/controller/federatedserviceaccount/controller.go
	pkg/controller/federatedserviceaccount/controller_test.go
Edit your sample resource instance...
	hack/sample/federatedserviceaccount.yaml
Generating code for new resource...  Regenerate after editing resources files by running `kubebuilder generate clean; kubebuilder generate`.
Next: Install the API, run the controller and create an instance with:
$ GOBIN=${PWD}/bin go install ${PWD#$GOPATH/src/}/cmd/controller-manager
$ bin/controller-manager --kubeconfig ~/.kube/config
$ kubectl apply -f hack/sample/federatedserviceaccount.yaml

$ kubebuilder create resource --group core --version v1alpha1 --kind FederatedServiceAccountPlacement
Creating API files for you to edit...
Edit your API schema...
	pkg/apis/core/v1alpha1/federatedserviceaccountplacement_types.go
	pkg/apis/core/v1alpha1/federatedserviceaccountplacement_types_test.go
Creating controller ...
Edit your controller function...
	pkg/controller/federatedserviceaccountplacement/controller.go
	pkg/controller/federatedserviceaccountplacement/controller_test.go
Edit your sample resource instance...
	hack/sample/federatedserviceaccountplacement.yaml
Generating code for new resource...  Regenerate after editing resources files by running `kubebuilder generate clean; kubebuilder generate`.
Next: Install the API, run the controller and create an instance with:
$ GOBIN=${PWD}/bin go install ${PWD#$GOPATH/src/}/cmd/controller-manager
$ bin/controller-manager --kubeconfig ~/.kube/config
$ kubectl apply -f hack/sample/federatedserviceaccountplacement.yaml

Testing

  • Follow the dev docs.
  • Deploy the example in the user guide.
  • Add fed type config for new type in ./config/.. and deployusing kubectl
  • Add tmpl/place examples in ./example.sample1/.. and deploy using kubectl

e2e fail:

go test -args -kubeconfig=/root/.kube/config -v=4 -test.v     --ginkgo.focus='"FederatedServiceAccount" resources'
=== RUN   TestE2E
Running Suite: Federation e2e suite
===================================
Random Seed: 1534978140
Will run 1 of 12 specs

SSSSSSSSSTEP: Reading cluster configuration
Aug 22 22:49:00.835: INFO: >>> kubeConfig: /root/.kube/config
STEP: Waiting for apiserver to be ready
STEP: apiserver is ready
STEP: Obtaining a list of federated clusters
STEP: Checking readiness of cluster "cluster1"
Aug 22 22:49:00.945: INFO: Cluster cluster1 is Ready
STEP: Checking readiness of cluster "cluster2"
Aug 22 22:49:00.987: INFO: Cluster cluster2 is Ready
STEP: Creating a namespace to execute the test in
STEP: Created test namespace e2e-tests-federated-types-bkjbl
Aug 22 22:49:01.058: INFO: Creating new FederatedServiceAccount in namespace "e2e-tests-federated-types-bkjbl"
Aug 22 22:49:01.066: INFO: Created new FederatedServiceAccount "e2e-tests-federated-types-bkjbl/test-crud-dxs76"
Aug 22 22:49:01.066: INFO: Creating new FederatedServiceAccountPlacement in namespace "e2e-tests-federated-types-bkjbl"
Aug 22 22:49:01.069: INFO: Created new FederatedServiceAccountPlacement "e2e-tests-federated-types-bkjbl/test-crud-dxs76"
Aug 22 22:49:01.069: INFO: Creating new FederatedServiceAccountOverride in namespace "e2e-tests-federated-types-bkjbl"
Aug 22 22:49:01.073: INFO: Created new FederatedServiceAccountOverride "e2e-tests-federated-types-bkjbl/test-crud-dxs76"
Aug 22 22:49:01.073: INFO: [Resource versions for e2e-tests-federated-types-bkjbl/test-crud-dxs76: template "19498", placement "19499", override "19500"]
Aug 22 22:49:13.087: INFO: Waiting for ServiceAccount "e2e-tests-federated-types-bkjbl/test-crud-dxs76" in cluster "cluster1"
Aug 22 22:50:13.099: INFO: Timeout verifying ServiceAccount "e2e-tests-federated-types-bkjbl/test-crud-dxs76" in cluster "cluster1": timed out waiting for the condition

------------------------------
• Failure [72.264 seconds]
Federated types
/root/go/src/github.com/kubernetes-sigs/federation-v2/test/e2e/crud.go:40
  "FederatedServiceAccount" resources
  /root/go/src/github.com/kubernetes-sigs/federation-v2/test/e2e/crud.go:71
    should be created, read, updated and deleted successfully [It]
    /root/go/src/github.com/kubernetes-sigs/federation-v2/test/e2e/crud.go:72

    Aug 22 22:50:13.099: Timeout verifying ServiceAccount "e2e-tests-federated-types-bkjbl/test-crud-dxs76" in cluster "cluster1": timed out waiting for the condition

    /root/go/src/github.com/kubernetes-sigs/federation-v2/test/e2e/framework/logger.go:39
------------------------------
SSS

Summarizing 1 Failure:

[Fail] Federated types "FederatedServiceAccount" resources [It] should be created, read, updated and deleted successfully 
/root/go/src/github.com/kubernetes-sigs/federation-v2/test/e2e/framework/logger.go:39

Ran 1 of 12 Specs in 72.265 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 11 Skipped
--- FAIL: TestE2E (72.27s)
FAIL
exit status 1
FAIL	github.com/kubernetes-sigs/federation-v2/test/e2e	72.331s
Raw
fed_cr_test_err
# go test ./test/integration/ -v -run ^TestIntegration/Parallel-Integration-Test-Group/TestCrud/FederatedClusterRole$
=== RUN TestIntegration
Flag --insecure-port has been deprecated, This flag will be removed in a future version.
Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version.
I0829 17:48:59.127632 21106 serving.go:273] Generated self-signed cert (/tmp/k8s_test_framework_825470231/apiserver.crt, /tmp/k8s_test_framework_825470231/apiserver.key)
I0829 17:48:59.127685 21106 server.go:703] external host was not specified, using 10.138.0.2
W0829 17:48:59.127701 21106 authentication.go:378] AnonymousAuth is not allowed with the AlwaysAllow authorizer. Resetting AnonymousAuth to false. You should use a different authorizer
I0829 17:48:59.128201 21106 server.go:145] Version: v1.11.0
I0829 17:48:59.158981 21106 plugins.go:158] Loaded 6 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I0829 17:48:59.159011 21106 plugins.go:161] Loaded 5 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0829 17:48:59.159814 21106 plugins.go:158] Loaded 6 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I0829 17:48:59.159832 21106 plugins.go:161] Loaded 5 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0829 17:48:59.218144 21106 master.go:234] Using reconciler: lease
W0829 17:49:00.412377 21106 genericapiserver.go:319] Skipping API batch/v2alpha1 because it has no resources.
W0829 17:49:00.974356 21106 genericapiserver.go:319] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0829 17:49:00.984535 21106 genericapiserver.go:319] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0829 17:49:01.014721 21106 genericapiserver.go:319] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0829 17:49:01.888324 21106 genericapiserver.go:319] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources.
[restful] 2018/08/29 17:49:01 log.go:33: [restful/swagger] listing is available at http://10.138.0.2:443/swaggerapi
[restful] 2018/08/29 17:49:01 log.go:33: [restful/swagger] http://10.138.0.2:443/swaggerui/ is mapped to folder /swagger-ui/
[restful] 2018/08/29 17:49:03 log.go:33: [restful/swagger] listing is available at http://10.138.0.2:443/swaggerapi
[restful] 2018/08/29 17:49:03 log.go:33: [restful/swagger] http://10.138.0.2:443/swaggerui/ is mapped to folder /swagger-ui/
I0829 17:49:03.560888 21106 plugins.go:158] Loaded 6 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I0829 17:49:03.560920 21106 plugins.go:161] Loaded 5 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0829 17:49:06.919551 21106 insecure_handler.go:119] Serving insecurely on 127.0.0.1:45633
I0829 17:49:06.922121 21106 crd_finalizer.go:242] Starting CRDFinalizer
I0829 17:49:06.922928 21106 available_controller.go:278] Starting AvailableConditionController
I0829 17:49:06.922957 21106 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0829 17:49:06.922979 21106 controller.go:84] Starting OpenAPI AggregationController
I0829 17:49:06.923017 21106 autoregister_controller.go:136] Starting autoregister controller
I0829 17:49:06.923025 21106 cache.go:32] Waiting for caches to sync for autoregister controller
I0829 17:49:06.923822 21106 apiservice_controller.go:90] Starting APIServiceRegistrationController
I0829 17:49:06.924044 21106 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0829 17:49:06.925253 21106 customresource_discovery_controller.go:199] Starting DiscoveryController
I0829 17:49:06.925288 21106 naming_controller.go:284] Starting NamingConditionController
I0829 17:49:06.925304 21106 establishing_controller.go:73] Starting EstablishingController
I0829 17:49:06.925333 21106 crdregistration_controller.go:112] Starting crd-autoregister controller
I0829 17:49:06.925350 21106 controller_utils.go:1025] Waiting for caches to sync for crd-autoregister controller
W0829 17:49:06.948247 21106 lease.go:223] Resetting endpoints for master service "kubernetes" to [10.138.0.2]
I0829 17:49:07.023086 21106 cache.go:39] Caches are synced for autoregister controller
I0829 17:49:07.023412 21106 cache.go:39] Caches are synced for AvailableConditionController controller
I0829 17:49:07.026129 21106 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0829 17:49:07.026196 21106 controller_utils.go:1032] Caches are synced for crd-autoregister controller
I0829 17:49:07.925054 21106 storage_scheduling.go:91] created PriorityClass system-node-critical with value 2000001000
I0829 17:49:07.927636 21106 storage_scheduling.go:91] created PriorityClass system-cluster-critical with value 2000000000
I0829 17:49:07.927660 21106 storage_scheduling.go:100] all system priority classes are created successfully or already exist.
2018/08/29 17:49:07 Creating CRD clusters.clusterregistry.k8s.io
2018/08/29 17:49:07 Finished installing.
2018/08/29 17:49:07 Creating CRD federatedclusters.core.federation.k8s.io
2018/08/29 17:49:07 Creating CRD federatedclusterroles.core.federation.k8s.io
2018/08/29 17:49:07 Creating CRD federatedclusterroleplacements.core.federation.k8s.io
2018/08/29 17:49:07 Creating CRD federatedconfigmaps.core.federation.k8s.io
2018/08/29 17:49:08 Creating CRD federatedconfigmapoverrides.core.federation.k8s.io
2018/08/29 17:49:08 Creating CRD federatedconfigmapplacements.core.federation.k8s.io
2018/08/29 17:49:08 Creating CRD federateddeployments.core.federation.k8s.io
2018/08/29 17:49:09 Creating CRD federateddeploymentoverrides.core.federation.k8s.io
2018/08/29 17:49:09 Creating CRD federateddeploymentplacements.core.federation.k8s.io
2018/08/29 17:49:10 Creating CRD federatedingresses.core.federation.k8s.io
2018/08/29 17:49:10 Creating CRD federatedingressplacements.core.federation.k8s.io
2018/08/29 17:49:10 Creating CRD federatedjobs.core.federation.k8s.io
2018/08/29 17:49:11 Creating CRD federatedjoboverrides.core.federation.k8s.io
2018/08/29 17:49:11 Creating CRD federatedjobplacements.core.federation.k8s.io
2018/08/29 17:49:12 Creating CRD federatednamespaceplacements.core.federation.k8s.io
2018/08/29 17:49:12 Creating CRD federatedreplicasets.core.federation.k8s.io
2018/08/29 17:49:12 Creating CRD federatedreplicasetoverrides.core.federation.k8s.io
2018/08/29 17:49:13 Creating CRD federatedreplicasetplacements.core.federation.k8s.io
2018/08/29 17:49:13 Creating CRD federatedsecrets.core.federation.k8s.io
2018/08/29 17:49:14 Creating CRD federatedsecretoverrides.core.federation.k8s.io
2018/08/29 17:49:14 Creating CRD federatedsecretplacements.core.federation.k8s.io
2018/08/29 17:49:14 Creating CRD federatedservices.core.federation.k8s.io
2018/08/29 17:49:15 Creating CRD federatedserviceplacements.core.federation.k8s.io
2018/08/29 17:49:15 Creating CRD federatedtypeconfigs.core.federation.k8s.io
2018/08/29 17:49:16 Creating CRD propagatedversions.core.federation.k8s.io
2018/08/29 17:49:16 Creating CRD dnsendpoints.multiclusterdns.federation.k8s.io
2018/08/29 17:49:16 Creating CRD multiclusteringressdnsrecords.multiclusterdns.federation.k8s.io
2018/08/29 17:49:17 Creating CRD multiclusterservicednsrecords.multiclusterdns.federation.k8s.io
2018/08/29 17:49:17 Creating CRD replicaschedulingpreferences.scheduling.federation.k8s.io
2018/08/29 17:49:17 Finished installing.
I0829 17:49:18.118829 21106 controller.go:597] quota admission added evaluator for: {clusterregistry.k8s.io clusters}
I0829 17:49:18.132324 21106 controller.go:597] quota admission added evaluator for: {core.federation.k8s.io federatedclusters}
Flag --insecure-port has been deprecated, This flag will be removed in a future version.
Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version.
I0829 17:49:18.889950 21120 serving.go:273] Generated self-signed cert (/tmp/k8s_test_framework_977594250/apiserver.crt, /tmp/k8s_test_framework_977594250/apiserver.key)
I0829 17:49:18.890019 21120 server.go:703] external host was not specified, using 10.138.0.2
W0829 17:49:18.890032 21120 authentication.go:378] AnonymousAuth is not allowed with the AlwaysAllow authorizer. Resetting AnonymousAuth to false. You should use a different authorizer
I0829 17:49:18.890565 21120 server.go:145] Version: v1.11.0
I0829 17:49:18.893521 21120 plugins.go:158] Loaded 6 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I0829 17:49:18.893555 21120 plugins.go:161] Loaded 5 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0829 17:49:18.894613 21120 plugins.go:158] Loaded 6 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I0829 17:49:18.894674 21120 plugins.go:161] Loaded 5 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0829 17:49:18.939419 21120 master.go:234] Using reconciler: lease
W0829 17:49:20.070367 21120 genericapiserver.go:319] Skipping API batch/v2alpha1 because it has no resources.
W0829 17:49:20.620439 21120 genericapiserver.go:319] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0829 17:49:20.629853 21120 genericapiserver.go:319] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0829 17:49:20.665576 21120 genericapiserver.go:319] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0829 17:49:21.572384 21120 genericapiserver.go:319] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources.
[restful] 2018/08/29 17:49:21 log.go:33: [restful/swagger] listing is available at http://10.138.0.2:443/swaggerapi
[restful] 2018/08/29 17:49:21 log.go:33: [restful/swagger] http://10.138.0.2:443/swaggerui/ is mapped to folder /swagger-ui/
[restful] 2018/08/29 17:49:23 log.go:33: [restful/swagger] listing is available at http://10.138.0.2:443/swaggerapi
[restful] 2018/08/29 17:49:23 log.go:33: [restful/swagger] http://10.138.0.2:443/swaggerui/ is mapped to folder /swagger-ui/
I0829 17:49:23.250054 21120 plugins.go:158] Loaded 6 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I0829 17:49:23.250085 21120 plugins.go:161] Loaded 5 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0829 17:49:26.651345 21120 insecure_handler.go:119] Serving insecurely on 127.0.0.1:34017
I0829 17:49:26.653199 21120 controller.go:84] Starting OpenAPI AggregationController
I0829 17:49:26.653846 21120 autoregister_controller.go:136] Starting autoregister controller
I0829 17:49:26.653868 21120 cache.go:32] Waiting for caches to sync for autoregister controller
I0829 17:49:26.654591 21120 apiservice_controller.go:90] Starting APIServiceRegistrationController
I0829 17:49:26.654781 21120 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0829 17:49:26.655275 21120 available_controller.go:278] Starting AvailableConditionController
I0829 17:49:26.655295 21120 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0829 17:49:26.654711 21120 crd_finalizer.go:242] Starting CRDFinalizer
I0829 17:49:26.656723 21120 crdregistration_controller.go:112] Starting crd-autoregister controller
I0829 17:49:26.656758 21120 controller_utils.go:1025] Waiting for caches to sync for crd-autoregister controller
I0829 17:49:26.656780 21120 customresource_discovery_controller.go:199] Starting DiscoveryController
I0829 17:49:26.656813 21120 naming_controller.go:284] Starting NamingConditionController
I0829 17:49:26.656863 21120 establishing_controller.go:73] Starting EstablishingController
W0829 17:49:26.683197 21120 lease.go:223] Resetting endpoints for master service "kubernetes" to [10.138.0.2]
I0829 17:49:26.754054 21120 cache.go:39] Caches are synced for autoregister controller
I0829 17:49:26.755848 21120 cache.go:39] Caches are synced for AvailableConditionController controller
I0829 17:49:26.755955 21120 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0829 17:49:26.766865 21120 controller_utils.go:1032] Caches are synced for crd-autoregister controller
I0829 17:49:27.658259 21120 storage_scheduling.go:91] created PriorityClass system-node-critical with value 2000001000
I0829 17:49:27.662304 21120 storage_scheduling.go:91] created PriorityClass system-cluster-critical with value 2000000000
I0829 17:49:27.662328 21120 storage_scheduling.go:100] all system priority classes are created successfully or already exist.
=== RUN TestIntegration/Parallel-Integration-Test-Group
=== RUN TestIntegration/Parallel-Integration-Test-Group/TestCrud
=== PAUSE TestIntegration/Parallel-Integration-Test-Group/TestCrud
=== CONT TestIntegration/Parallel-Integration-Test-Group/TestCrud
=== RUN TestIntegration/Parallel-Integration-Test-Group/TestCrud/FederatedClusterRole
I0829 17:49:27.739114 21106 controller.go:597] quota admission added evaluator for: {core.federation.k8s.io federatedclusterroles}
I0829 17:49:27.739565 21106 controller.go:597] quota admission added evaluator for: { namespaces}
I0829 17:49:27.747122 21106 controller.go:597] quota admission added evaluator for: {core.federation.k8s.io federatedclusterroleplacements}
E0829 17:49:28.808613 21088 reflector.go:205] github.com/kubernetes-sigs/federation-v2/pkg/controller/util/federated_informer.go:430: Failed to list <nil>: the server could not find the requested resource (get clusterroles.rbac)
E0829 17:49:28.819205 21088 reflector.go:205] github.com/kubernetes-sigs/federation-v2/pkg/controller/util/federated_informer.go:430: Failed to list <nil>: the server could not find the requested resource (get clusterroles.rbac)
I0829 17:49:28.862616 21120 controller.go:597] quota admission added evaluator for: { namespaces}
I0829 17:49:28.864858 21106 controller.go:597] quota admission added evaluator for: {core.federation.k8s.io propagatedversions}
E0829 17:49:29.811492 21088 reflector.go:205] github.com/kubernetes-sigs/federation-v2/pkg/controller/util/federated_informer.go:430: Failed to list <nil>: the server could not find the requested resource (get clusterroles.rbac)
E0829 17:49:29.820193 21088 reflector.go:205] github.com/kubernetes-sigs/federation-v2/pkg/controller/util/federated_informer.go:430: Failed to list <nil>: the server could not find the requested resource (get clusterroles.rbac)
Raw
federated_istio.md

Port each resource from top-level istio.yaml and crds.yaml to Federated.

Start with the istio crds. CRDs is not a federated deployment type yet, so deploy manually to each cluster. TODO: Add Federated CRD support.

fedv2 did not deploy the statsd-prom cm to cluster2:

# kubectl get cm -n istio-system --context cluster2
NAME                             DATA      AGE
istio-galley-configuration       1         18h
istio-grafana-custom-resources   2         1h
root@dev:~/go/src/github.com/kubernetes-sigs/federation-v2/example/istio# kubectl get cm -n istio-system --context cluster1
NAME                             DATA      AGE
istio-galley-configuration       1         18h
istio-grafana-custom-resources   2         1h
istio-statsd-prom-bridge         1         1h
root@dev:~/go/src/github.com/kubernetes-sigs/federation-v2/example/istio# cat federatedconfigmap-statsdprom-template.yaml 
# Source: istio/charts/mixer/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: istio-statsd-prom-bridge
  namespace: istio-system
  labels:
    app: istio-statsd-prom-bridge
    chart: mixer-1.0.0
    release: RELEASE-NAME
    heritage: Tiller
    istio: mixer
data:
  mapping.conf: |-

root@dev:~/go/src/github.com/kubernetes-sigs/federation-v2/example/istio# cat federatedconfigmap-statsdprom-placement.yaml 
apiVersion: v1
kind: List
items:
- apiVersion: core.federation.k8s.io/v1alpha1
  kind: FederatedConfigMapPlacement
  metadata:
    name: istio-statsd-prom-bridge
    namespace: istio-system
  spec:
    clusternames:
    - cluster1
    - cluster2
Raw
fedsa_e2e_test_fail
$ go test -args -kubeconfig=/root/.kube/config -v=4 -test.v --ginkgo.focus='"FederatedServiceAccount" resources'
=== RUN TestE2E
Running Suite: Federation e2e suite
===================================
Random Seed: 1535063524
Will run 1 of 12 specs
SSSSSSSSSSSTEP: Reading cluster configuration
Aug 23 22:32:04.977: INFO: >>> kubeConfig: /root/.kube/config
STEP: Waiting for apiserver to be ready
STEP: apiserver is ready
STEP: Obtaining a list of federated clusters
STEP: Checking readiness of cluster "cluster1"
Aug 23 22:32:05.039: INFO: Cluster cluster1 is Ready
STEP: Checking readiness of cluster "cluster2"
Aug 23 22:32:05.046: INFO: Cluster cluster2 is Ready
STEP: Creating a namespace to execute the test in
STEP: Created test namespace e2e-tests-federated-types-5zqkd
Aug 23 22:32:05.110: INFO: Creating new FederatedServiceAccount in namespace "e2e-tests-federated-types-5zqkd"
Aug 23 22:32:05.270: INFO: Created new FederatedServiceAccount "e2e-tests-federated-types-5zqkd/test-crud-zmrz7"
Aug 23 22:32:05.270: INFO: Creating new FederatedServiceAccountPlacement in namespace "e2e-tests-federated-types-5zqkd"
Aug 23 22:32:05.291: INFO: Created new FederatedServiceAccountPlacement "e2e-tests-federated-types-5zqkd/test-crud-zmrz7"
Aug 23 22:32:05.291: INFO: [Resource versions for e2e-tests-federated-types-5zqkd/test-crud-zmrz7: template "7472", placement "7473"]
Aug 23 22:32:07.303: INFO: Waiting for ServiceAccount "e2e-tests-federated-types-5zqkd/test-crud-zmrz7" in cluster "cluster2"
Aug 23 22:33:37.317: INFO: Timeout verifying ServiceAccount "e2e-tests-federated-types-5zqkd/test-crud-zmrz7" in cluster "cluster2": timed out waiting for the condition
------------------------------
• Failure [92.341 seconds]
Federated types
/root/go/src/github.com/kubernetes-sigs/federation-v2/test/e2e/crud.go:40
"FederatedServiceAccount" resources
/root/go/src/github.com/kubernetes-sigs/federation-v2/test/e2e/crud.go:71
should be created, read, updated and deleted successfully [It]
/root/go/src/github.com/kubernetes-sigs/federation-v2/test/e2e/crud.go:72
Aug 23 22:33:37.317: Timeout verifying ServiceAccount "e2e-tests-federated-types-5zqkd/test-crud-zmrz7" in cluster "cluster2": timed out waiting for the condition
/root/go/src/github.com/kubernetes-sigs/federation-v2/test/e2e/framework/logger.go:39
------------------------------
S
Summarizing 1 Failure:
[Fail] Federated types "FederatedServiceAccount" resources [It] should be created, read, updated and deleted successfully
/root/go/src/github.com/kubernetes-sigs/federation-v2/test/e2e/framework/logger.go:39
Ran 1 of 12 Specs in 92.342 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 11 Skipped
--- FAIL: TestE2E (92.34s)
FAIL
exit status 1
FAIL github.com/kubernetes-sigs/federation-v2/test/e2e 92.440s
# Timeout error due to sa getting created on cluster 2, but the sa is created as expected:
$ kubectl get sa -n e2e-tests-federated-types-5zqkd --context cluster1
NAME SECRETS AGE
default 1 6m
test-crud-zmrz7 1 6m
$ kubectl get sa -n e2e-tests-federated-types-5zqkd --context cluster2
NAME SECRETS AGE
default 1 4m
test-crud-zmrz7 1 4m
Raw
install_steps.md

I had to use the minikube "none" driver. Therefore, I had to create 2 different gcp vm's, 1 for each k8s cluster.

Install deps:


curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.11.0/bin/linux/amd64/kubectl
chmod +x kubectl 
sudo mv ./kubectl /usr/local/bin/kubectl
curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
chmod +x minikube
sudo cp minikube /usr/local/bin/ && rm minikube
url -LO https://storage.googleapis.com/kubernetes-release/release/v1.11.0/bin/linux/amd64/kubeadm
chmod +x kubeadm 
sudo mv ./kubeadm /usr/local/bin/kubeadm

Install Docker:

apt-get remove docker docker-engine docker.io
apt-get update && apt-get install     apt-transport-https     ca-certificates     curl     software-properties-common -y
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88
sudo add-apt-repository    "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
apt-get update && apt-cache madison docker-ce
apt-get install docker-ce=17.03.2~ce-0~ubuntu-xenial

Verify Docker:

docker run hello-world

For some reason, minikube is requiring crictl:

VERSION="v1.11.1"   
wget https://github.com/kubernetes-incubator/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz
sudo tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin
rm -f crictl-$VERSION-linux-amd64.tar.gz

Start a minikube cluster:

minikube start -p cluster1 --vm-driver none --kubernetes-version v1.11.0

Verify minikube:

kubectl run hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080
# Wait for the pod to run, then:
kubectl get po -o yaml | grep hostIP
kubectl expose deployment hello-minikube --type=NodePort
kubectl get svc
# Test $hostIP:svc-node-port
curl -I http://10.138.0.3:32330

Repeat the above steps for your 2nd gcp vm instance. Then scp kubeconfig and associated certs between instances. I create dir's for each cluster name:

cd .minikube/
mkdir -p cluster1 cluster2
# On cluster1-vm
mv ca.crt client.crt client.key cluster1
# On cluster2-vm
mv ca.crt client.crt client.key cluster2

Exit your gcloud instance to scp certs/key between instances. This will allow kubectl from each vm to access either k8s cluster by setting the context:

# copy certs/key from cluster1-vm > local machine > cluster2-vm
gcloud compute scp root@cluster1-vm:/root/.minikube/cluster1/ca.crt ca.crt
gcloud compute scp root@cluster1-vm:/root/.minikube/cluster1/client.crt client.crt
gcloud compute scp root@cluster1-vm:/root/.minikube/cluster1/client.key client.key
gcloud compute scp ca.crt root@cluster2-vm:/root/.minikube/cluster1/ca.crt
gcloud compute scp client.crt root@cluster2-vm:/root/.minikube/cluster1/client.crt
gcloud compute scp client.key root@cluster2-vm:/root/.minikube/cluster1/client.key

# copy certs/key from cluster2-vm > local machine > cluster1-vm
gcloud compute scp root@cluster2-vm:/root/.minikube/cluster2/ca.crt ca.crt
gcloud compute scp root@cluster2-vm:/root/.minikube/cluster2/client.crt client.crt
gcloud compute scp root@cluster2-vm:/root/.minikube/cluster2/client.key client.key
gcloud compute scp ca.crt root@cluster1-vm:/root/.minikube/cluster2/ca.crt
gcloud compute scp client.crt root@cluster1-vm:/root/.minikube/cluster2/client.crt
gcloud compute scp client.key root@cluster1-vm:/root/.minikube/cluster2/client.key

Update the kubeconf on each gcp vm for the 2nd cluster. Here is an example:

cat .kube/config 
apiVersion: v1
clusters:
- cluster:
    certificate-authority: /root/.minikube/cluster1/ca.crt
    server: https://10.138.0.2:8443
  name: cluster1
- cluster:
    certificate-authority: /root/.minikube/cluster2/ca.crt
    server: https://10.138.0.3:8443
  name: cluster2
contexts:
- context:
    cluster: cluster1
    user: cluster1
  name: cluster1
- context:
    cluster: cluster2
    user: cluster2
  name: cluster2
current-context: cluster2
kind: Config
preferences: {}
users:
- name: cluster1
  user:
    client-certificate: /root/.minikube/cluster1/client.crt
    client-key: /root/.minikube/cluster1/client.key
- name: cluster2
  user:
    client-certificate: /root/.minikube/cluster2/client.crt
    client-key: /root/.minikube/cluster2/client.key

Verify the kubeconfig from each vm:

kubectl config get-contexts
kubectl config use-context cluster1
kubectl cluster-info
kubectl config use-context cluster2
kubectl cluster-info

Create a sample deployment and service for each cluster:

kubectl run hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080
kubectl expose deployment hello-minikube --type=NodePort

Start a minikube cluster:

minikube start -p cluster1 --vm-driver none --kubernetes-version v1.11.0

Verify minikube:

kubectl run hello-minikube --image=k8s.gcr.io/echoserver:1.4 --port=8080
# Wait for the pod to run, then:
kubectl get po -o yaml | grep hostIP
kubectl expose deployment hello-minikube --type=NodePort
kubectl get svc
# Test $hostIP:svc-node-port
curl -I http://10.138.0.3:32330

Repeat for 2nd cluster.

Raw
integration_test_fail
$ go test -v -run ^TestIntegration/Parallel-Integration-Test-Group/TestCrud/FederatedConfigMap$
=== RUN TestIntegration
Flag --insecure-port has been deprecated, This flag will be removed in a future version.
Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version.
I0822 17:05:29.152180 5375 server.go:135] Version: v0.0.0-master+$Format:%h$
I0822 17:05:29.915739 5375 serving.go:275] Generated self-signed cert (/tmp/k8s_test_framework_252190941/apiserver.crt, /tmp/k8s_test_framework_252190941/apiserver.key)
I0822 17:05:29.915804 5375 server.go:721] external host was not specified, using 10.138.0.2
W0822 17:05:29.915816 5375 authentication.go:377] AnonymousAuth is not allowed with the AllowAll authorizer. Resetting AnonymousAuth to false. You should use a different authorizer
W0822 17:05:29.940505 5375 admission.go:68] PersistentVolumeLabel admission controller is deprecated. Please remove this controller from your configuration files and scripts.
I0822 17:05:29.942443 5375 plugins.go:149] Loaded 9 admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultTolerationSeconds,PersistentVolumeLabel,DefaultStorageClass,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota.
W0822 17:05:29.943100 5375 admission.go:68] PersistentVolumeLabel admission controller is deprecated. Please remove this controller from your configuration files and scripts.
I0822 17:05:29.943629 5375 plugins.go:149] Loaded 9 admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultTolerationSeconds,PersistentVolumeLabel,DefaultStorageClass,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota.
I0822 17:05:29.968008 5375 master.go:228] Using reconciler: master-count
W0822 17:05:30.222815 5375 genericapiserver.go:342] Skipping API batch/v2alpha1 because it has no resources.
W0822 17:05:30.245903 5375 genericapiserver.go:342] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0822 17:05:30.249215 5375 genericapiserver.go:342] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0822 17:05:30.263488 5375 genericapiserver.go:342] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources.
[restful] 2018/08/22 17:05:30 log.go:33: [restful/swagger] listing is available at http://10.138.0.2:443/swaggerapi
[restful] 2018/08/22 17:05:30 log.go:33: [restful/swagger] http://10.138.0.2:443/swaggerui/ is mapped to folder /swagger-ui/
[restful] 2018/08/22 17:05:31 log.go:33: [restful/swagger] listing is available at http://10.138.0.2:443/swaggerapi
[restful] 2018/08/22 17:05:31 log.go:33: [restful/swagger] http://10.138.0.2:443/swaggerui/ is mapped to folder /swagger-ui/
W0822 17:05:31.893567 5375 admission.go:68] PersistentVolumeLabel admission controller is deprecated. Please remove this controller from your configuration files and scripts.
I0822 17:05:31.894174 5375 plugins.go:149] Loaded 9 admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultTolerationSeconds,PersistentVolumeLabel,DefaultStorageClass,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota.
I0822 17:05:35.409999 5375 insecure_handler.go:121] Serving insecurely on 127.0.0.1:35251
I0822 17:05:35.411448 5375 controller.go:84] Starting OpenAPI AggregationController
I0822 17:05:35.413643 5375 available_controller.go:262] Starting AvailableConditionController
I0822 17:05:35.413661 5375 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0822 17:05:35.452666 5375 crd_finalizer.go:242] Starting CRDFinalizer
I0822 17:05:35.452715 5375 apiservice_controller.go:90] Starting APIServiceRegistrationController
I0822 17:05:35.452722 5375 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0822 17:05:35.461305 5375 crdregistration_controller.go:110] Starting crd-autoregister controller
I0822 17:05:35.461443 5375 controller_utils.go:1019] Waiting for caches to sync for crd-autoregister controller
I0822 17:05:35.462928 5375 customresource_discovery_controller.go:174] Starting DiscoveryController
I0822 17:05:35.462965 5375 naming_controller.go:276] Starting NamingConditionController
I0822 17:05:35.517059 5375 cache.go:39] Caches are synced for AvailableConditionController controller
I0822 17:05:35.552906 5375 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0822 17:05:35.561701 5375 controller_utils.go:1026] Caches are synced for crd-autoregister controller
I0822 17:05:35.561810 5375 autoregister_controller.go:136] Starting autoregister controller
I0822 17:05:35.561839 5375 cache.go:32] Waiting for caches to sync for autoregister controller
I0822 17:05:35.597774 5375 controller.go:537] quota admission added evaluator for: { endpoints}
I0822 17:05:35.665732 5375 cache.go:39] Caches are synced for autoregister controller
2018/08/22 17:05:35 Creating CRD clusters.clusterregistry.k8s.io
2018/08/22 17:05:35 Finished installing.
2018/08/22 17:05:35 Creating CRD federatedclusters.core.federation.k8s.io
2018/08/22 17:05:35 Creating CRD federatedconfigmaps.core.federation.k8s.io
2018/08/22 17:05:35 Creating CRD federatedconfigmapoverrides.core.federation.k8s.io
2018/08/22 17:05:35 Creating CRD federatedconfigmapplacements.core.federation.k8s.io
2018/08/22 17:05:35 Creating CRD federateddeployments.core.federation.k8s.io
2018/08/22 17:05:36 Creating CRD federateddeploymentoverrides.core.federation.k8s.io
2018/08/22 17:05:36 Creating CRD federateddeploymentplacements.core.federation.k8s.io
2018/08/22 17:05:37 Creating CRD federatedingresses.core.federation.k8s.io
2018/08/22 17:05:37 Creating CRD federatedingressplacements.core.federation.k8s.io
2018/08/22 17:05:37 Creating CRD federatedjobs.core.federation.k8s.io
2018/08/22 17:05:38 Creating CRD federatedjoboverrides.core.federation.k8s.io
2018/08/22 17:05:38 Creating CRD federatedjobplacements.core.federation.k8s.io
2018/08/22 17:05:39 Creating CRD federatednamespaceplacements.core.federation.k8s.io
2018/08/22 17:05:39 Creating CRD federatedreplicasets.core.federation.k8s.io
2018/08/22 17:05:39 Creating CRD federatedreplicasetoverrides.core.federation.k8s.io
2018/08/22 17:05:40 Creating CRD federatedreplicasetplacements.core.federation.k8s.io
2018/08/22 17:05:40 Creating CRD federatedsecrets.core.federation.k8s.io
2018/08/22 17:05:41 Creating CRD federatedsecretoverrides.core.federation.k8s.io
2018/08/22 17:05:41 Creating CRD federatedsecretplacements.core.federation.k8s.io
2018/08/22 17:05:41 Creating CRD federatedservices.core.federation.k8s.io
2018/08/22 17:05:42 Creating CRD federatedserviceplacements.core.federation.k8s.io
2018/08/22 17:05:42 Creating CRD federatedtypeconfigs.core.federation.k8s.io
2018/08/22 17:05:43 Creating CRD propagatedversions.core.federation.k8s.io
2018/08/22 17:05:43 Creating CRD dnsendpoints.multiclusterdns.federation.k8s.io
2018/08/22 17:05:43 Creating CRD multiclusterservicednsrecords.multiclusterdns.federation.k8s.io
2018/08/22 17:05:44 Creating CRD replicaschedulingpreferences.scheduling.federation.k8s.io
2018/08/22 17:05:44 Finished installing.
I0822 17:05:44.736447 5375 controller.go:537] quota admission added evaluator for: {clusterregistry.k8s.io clusters}
I0822 17:05:44.749571 5375 controller.go:537] quota admission added evaluator for: {core.federation.k8s.io federatedclusters}
Flag --insecure-port has been deprecated, This flag will be removed in a future version.
Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version.
I0822 17:05:44.988677 5490 server.go:135] Version: v0.0.0-master+$Format:%h$
I0822 17:05:45.730479 5490 serving.go:275] Generated self-signed cert (/tmp/k8s_test_framework_295397016/apiserver.crt, /tmp/k8s_test_framework_295397016/apiserver.key)
I0822 17:05:45.730511 5490 server.go:721] external host was not specified, using 10.138.0.2
W0822 17:05:45.730527 5490 authentication.go:377] AnonymousAuth is not allowed with the AllowAll authorizer. Resetting AnonymousAuth to false. You should use a different authorizer
W0822 17:05:45.733137 5490 admission.go:68] PersistentVolumeLabel admission controller is deprecated. Please remove this controller from your configuration files and scripts.
I0822 17:05:45.733676 5490 plugins.go:149] Loaded 9 admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultTolerationSeconds,PersistentVolumeLabel,DefaultStorageClass,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota.
W0822 17:05:45.734180 5490 admission.go:68] PersistentVolumeLabel admission controller is deprecated. Please remove this controller from your configuration files and scripts.
I0822 17:05:45.734578 5490 plugins.go:149] Loaded 9 admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultTolerationSeconds,PersistentVolumeLabel,DefaultStorageClass,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota.
I0822 17:05:45.738062 5490 master.go:228] Using reconciler: master-count
W0822 17:05:45.856980 5490 genericapiserver.go:342] Skipping API batch/v2alpha1 because it has no resources.
W0822 17:05:45.886095 5490 genericapiserver.go:342] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0822 17:05:45.891150 5490 genericapiserver.go:342] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0822 17:05:45.902427 5490 genericapiserver.go:342] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources.
[restful] 2018/08/22 17:05:45 log.go:33: [restful/swagger] listing is available at http://10.138.0.2:443/swaggerapi
[restful] 2018/08/22 17:05:45 log.go:33: [restful/swagger] http://10.138.0.2:443/swaggerui/ is mapped to folder /swagger-ui/
[restful] 2018/08/22 17:05:47 log.go:33: [restful/swagger] listing is available at http://10.138.0.2:443/swaggerapi
[restful] 2018/08/22 17:05:47 log.go:33: [restful/swagger] http://10.138.0.2:443/swaggerui/ is mapped to folder /swagger-ui/
W0822 17:05:47.842345 5490 admission.go:68] PersistentVolumeLabel admission controller is deprecated. Please remove this controller from your configuration files and scripts.
I0822 17:05:47.842842 5490 plugins.go:149] Loaded 9 admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultTolerationSeconds,PersistentVolumeLabel,DefaultStorageClass,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota.
I0822 17:05:51.681485 5490 insecure_handler.go:121] Serving insecurely on 127.0.0.1:36613
I0822 17:05:51.682880 5490 crd_finalizer.go:242] Starting CRDFinalizer
I0822 17:05:51.683057 5490 crdregistration_controller.go:110] Starting crd-autoregister controller
I0822 17:05:51.683197 5490 controller_utils.go:1019] Waiting for caches to sync for crd-autoregister controller
I0822 17:05:51.683300 5490 customresource_discovery_controller.go:174] Starting DiscoveryController
I0822 17:05:51.683393 5490 naming_controller.go:276] Starting NamingConditionController
I0822 17:05:51.684474 5490 apiservice_controller.go:90] Starting APIServiceRegistrationController
I0822 17:05:51.684623 5490 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0822 17:05:51.684719 5490 available_controller.go:262] Starting AvailableConditionController
I0822 17:05:51.684775 5490 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0822 17:05:51.687564 5490 controller.go:84] Starting OpenAPI AggregationController
I0822 17:05:51.725725 5490 controller.go:537] quota admission added evaluator for: { endpoints}
I0822 17:05:51.783416 5490 controller_utils.go:1026] Caches are synced for crd-autoregister controller
I0822 17:05:51.783590 5490 autoregister_controller.go:136] Starting autoregister controller
I0822 17:05:51.783621 5490 cache.go:32] Waiting for caches to sync for autoregister controller
I0822 17:05:51.785050 5490 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0822 17:05:51.785050 5490 cache.go:39] Caches are synced for AvailableConditionController controller
I0822 17:05:51.883885 5490 cache.go:39] Caches are synced for autoregister controller
=== RUN TestIntegration/Parallel-Integration-Test-Group
=== RUN TestIntegration/Parallel-Integration-Test-Group/TestCrud
=== PAUSE TestIntegration/Parallel-Integration-Test-Group/TestCrud
=== CONT TestIntegration/Parallel-Integration-Test-Group/TestCrud
=== RUN TestIntegration/Parallel-Integration-Test-Group/TestCrud/FederatedConfigMap
E0822 17:05:52.128340 5329 federated_informer.go:212] Cluster &{{FederatedCluster core.federation.k8s.io/v1alpha1} {test-cluster-2rsr7 federation-system-24drm /apis/core.federation.k8s.io/v1alpha1/namespaces/federation-system-24drm/federatedclusters/test-cluster-2rsr7 9b81797a-a62d-11e8-a883-42010a8a0002 94 1 2018-08-22 17:05:44 +0000 UTC <nil> <nil> map[] map[] [] nil [] } {{test-cluster-2rsr7} &LocalObjectReference{Name:test-cluster-2rsr7-credentialsjhsnr,}} {[] }} not added. Not of correct type, or cluster not ready.
E0822 17:05:52.131634 5329 federated_informer.go:212] Cluster &{{FederatedCluster core.federation.k8s.io/v1alpha1} {test-cluster-dwvqt federation-system-24drm /apis/core.federation.k8s.io/v1alpha1/namespaces/federation-system-24drm/federatedclusters/test-cluster-dwvqt 9fd58ddd-a62d-11e8-a883-42010a8a0002 128 1 2018-08-22 17:05:52 +0000 UTC <nil> <nil> map[] map[] [] nil [] } {{test-cluster-dwvqt} &LocalObjectReference{Name:test-cluster-dwvqt-credentialsml7zr,}} {[] }} not added. Not of correct type, or cluster not ready.
E0822 17:05:52.130368 5329 federated_informer.go:212] Cluster &{{FederatedCluster core.federation.k8s.io/v1alpha1} {test-cluster-2rsr7 federation-system-24drm /apis/core.federation.k8s.io/v1alpha1/namespaces/federation-system-24drm/federatedclusters/test-cluster-2rsr7 9b81797a-a62d-11e8-a883-42010a8a0002 94 1 2018-08-22 17:05:44 +0000 UTC <nil> <nil> map[] map[] [] nil [] } {{test-cluster-2rsr7} &LocalObjectReference{Name:test-cluster-2rsr7-credentialsjhsnr,}} {[] }} not added. Not of correct type, or cluster not ready.
E0822 17:05:52.131729 5329 federated_informer.go:212] Cluster &{{FederatedCluster core.federation.k8s.io/v1alpha1} {test-cluster-dwvqt federation-system-24drm /apis/core.federation.k8s.io/v1alpha1/namespaces/federation-system-24drm/federatedclusters/test-cluster-dwvqt 9fd58ddd-a62d-11e8-a883-42010a8a0002 128 1 2018-08-22 17:05:52 +0000 UTC <nil> <nil> map[] map[] [] nil [] } {{test-cluster-dwvqt} &LocalObjectReference{Name:test-cluster-dwvqt-credentialsml7zr,}} {[] }} not added. Not of correct type, or cluster not ready.
I0822 17:05:52.165539 5375 controller.go:537] quota admission added evaluator for: {core.federation.k8s.io federatedconfigmaps}
I0822 17:05:52.169717 5375 controller.go:537] quota admission added evaluator for: {core.federation.k8s.io federatedconfigmapplacements}
I0822 17:05:52.173192 5375 controller.go:537] quota admission added evaluator for: {core.federation.k8s.io federatedconfigmapoverrides}
I0822 17:05:53.126040 5375 controller.go:537] quota admission added evaluator for: { namespaces}
I0822 17:06:22.681737 5375 autoregister_controller.go:160] Shutting down autoregister controller
I0822 17:06:22.681810 5375 controller.go:90] Shutting down OpenAPI AggregationController
I0822 17:06:22.681901 5375 customresource_discovery_controller.go:185] Shutting down DiscoveryController
I0822 17:06:22.681918 5375 naming_controller.go:287] Shutting down NamingConditionController
I0822 17:06:22.681932 5375 crdregistration_controller.go:139] Shutting down crd-autoregister controller
I0822 17:06:22.681945 5375 crd_finalizer.go:254] Shutting down CRDFinalizer
I0822 17:06:22.681958 5375 apiservice_controller.go:102] Shutting down APIServiceRegistrationController
I0822 17:06:22.682003 5375 available_controller.go:274] Shutting down AvailableConditionController
I0822 17:06:22.682024 5375 serve.go:136] Stopped listening on 127.0.0.1:35251
I0822 17:06:22.805438 5490 autoregister_controller.go:160] Shutting down autoregister controller
I0822 17:06:22.805474 5490 available_controller.go:274] Shutting down AvailableConditionController
I0822 17:06:22.805491 5490 apiservice_controller.go:102] Shutting down APIServiceRegistrationController
I0822 17:06:22.805509 5490 naming_controller.go:287] Shutting down NamingConditionController
I0822 17:06:22.805522 5490 crdregistration_controller.go:139] Shutting down crd-autoregister controller
I0822 17:06:22.805537 5490 customresource_discovery_controller.go:185] Shutting down DiscoveryController
I0822 17:06:22.805550 5490 crd_finalizer.go:254] Shutting down CRDFinalizer
I0822 17:06:22.805727 5490 controller.go:90] Shutting down OpenAPI AggregationController
I0822 17:06:22.806005 5490 serve.go:136] Stopped listening on 127.0.0.1:36613
--- FAIL: TestIntegration (58.36s)
logger.go:52: Starting a federation of 2 clusters...
logger.go:52: Creating Cluster Registry CRD
logger.go:52: Creating Federation CRDs
logger.go:52: Added cluster test-cluster-2rsr7 to the federation
logger.go:52: Added cluster test-cluster-dwvqt to the federation
logger.go:52: Starting cluster controller
logger.go:48: Federation started.
--- FAIL: TestIntegration/Parallel-Integration-Test-Group (0.00s)
--- FAIL: TestIntegration/Parallel-Integration-Test-Group/TestCrud (30.46s)
--- FAIL: TestIntegration/Parallel-Integration-Test-Group/TestCrud/FederatedConfigMap (30.46s)
logger.go:52: Creating new FederatedConfigMap in namespace "crud-federatedconfigmap-f64zc"
logger.go:52: Created new FederatedConfigMap "crud-federatedconfigmap-f64zc/test-crud-77zw4"
logger.go:52: Creating new FederatedConfigMapPlacement in namespace "crud-federatedconfigmap-f64zc"
logger.go:52: Created new FederatedConfigMapPlacement "crud-federatedconfigmap-f64zc/test-crud-77zw4"
logger.go:52: Creating new FederatedConfigMapOverride in namespace "crud-federatedconfigmap-f64zc"
logger.go:52: Created new FederatedConfigMapOverride "crud-federatedconfigmap-f64zc/test-crud-77zw4"
logger.go:48: Resource versions for crud-federatedconfigmap-f64zc/test-crud-77zw4: template "130", placement "131", override "133"
logger.go:44: Error waiting for propagated version for ConfigMap "crud-federatedconfigmap-f64zc/test-crud-77zw4": timed out waiting for the condition
FAIL
exit status 1
FAIL github.com/kubernetes-sigs/federation-v2/test/integration 58.375s
root@dev:~/go/src/github.com/kubernetes-sigs/federation-v2/test/integration# echo $TEST_ASSET_PATH
/usr/local/kubebuilder/bin
root@dev:~/go/src/github.com/kubernetes-sigs/federation-v2/test/integration# go test -v -run ^TestIntegration/Parallel-Integration-Test-Group/TestCrud/FederatedSecret$
=== RUN TestIntegration
Flag --insecure-port has been deprecated, This flag will be removed in a future version.
Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version.
I0822 17:08:55.766398 6530 server.go:135] Version: v0.0.0-master+$Format:%h$
I0822 17:08:56.173220 6530 serving.go:275] Generated self-signed cert (/tmp/k8s_test_framework_004573235/apiserver.crt, /tmp/k8s_test_framework_004573235/apiserver.key)
I0822 17:08:56.173250 6530 server.go:721] external host was not specified, using 10.138.0.2
W0822 17:08:56.173265 6530 authentication.go:377] AnonymousAuth is not allowed with the AllowAll authorizer. Resetting AnonymousAuth to false. You should use a different authorizer
W0822 17:08:56.175962 6530 admission.go:68] PersistentVolumeLabel admission controller is deprecated. Please remove this controller from your configuration files and scripts.
I0822 17:08:56.176499 6530 plugins.go:149] Loaded 9 admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultTolerationSeconds,PersistentVolumeLabel,DefaultStorageClass,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota.
W0822 17:08:56.176957 6530 admission.go:68] PersistentVolumeLabel admission controller is deprecated. Please remove this controller from your configuration files and scripts.
I0822 17:08:56.177340 6530 plugins.go:149] Loaded 9 admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultTolerationSeconds,PersistentVolumeLabel,DefaultStorageClass,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota.
I0822 17:08:56.184242 6530 master.go:228] Using reconciler: master-count
W0822 17:08:56.287992 6530 genericapiserver.go:342] Skipping API batch/v2alpha1 because it has no resources.
W0822 17:08:56.310012 6530 genericapiserver.go:342] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0822 17:08:56.311377 6530 genericapiserver.go:342] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0822 17:08:56.337474 6530 genericapiserver.go:342] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources.
[restful] 2018/08/22 17:08:56 log.go:33: [restful/swagger] listing is available at http://10.138.0.2:443/swaggerapi
[restful] 2018/08/22 17:08:56 log.go:33: [restful/swagger] http://10.138.0.2:443/swaggerui/ is mapped to folder /swagger-ui/
[restful] 2018/08/22 17:08:57 log.go:33: [restful/swagger] listing is available at http://10.138.0.2:443/swaggerapi
[restful] 2018/08/22 17:08:57 log.go:33: [restful/swagger] http://10.138.0.2:443/swaggerui/ is mapped to folder /swagger-ui/
W0822 17:08:57.935910 6530 admission.go:68] PersistentVolumeLabel admission controller is deprecated. Please remove this controller from your configuration files and scripts.
I0822 17:08:57.936416 6530 plugins.go:149] Loaded 9 admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultTolerationSeconds,PersistentVolumeLabel,DefaultStorageClass,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota.
I0822 17:09:01.228669 6530 insecure_handler.go:121] Serving insecurely on 127.0.0.1:42099
I0822 17:09:01.230758 6530 crd_finalizer.go:242] Starting CRDFinalizer
I0822 17:09:01.230942 6530 apiservice_controller.go:90] Starting APIServiceRegistrationController
I0822 17:09:01.230955 6530 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0822 17:09:01.230974 6530 available_controller.go:262] Starting AvailableConditionController
I0822 17:09:01.230980 6530 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0822 17:09:01.230998 6530 controller.go:84] Starting OpenAPI AggregationController
I0822 17:09:01.231328 6530 customresource_discovery_controller.go:174] Starting DiscoveryController
I0822 17:09:01.231371 6530 naming_controller.go:276] Starting NamingConditionController
I0822 17:09:01.232141 6530 crdregistration_controller.go:110] Starting crd-autoregister controller
I0822 17:09:01.232160 6530 controller_utils.go:1019] Waiting for caches to sync for crd-autoregister controller
I0822 17:09:01.255023 6530 controller.go:537] quota admission added evaluator for: { endpoints}
I0822 17:09:01.331088 6530 cache.go:39] Caches are synced for AvailableConditionController controller
I0822 17:09:01.331088 6530 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0822 17:09:01.332359 6530 controller_utils.go:1026] Caches are synced for crd-autoregister controller
I0822 17:09:01.332451 6530 autoregister_controller.go:136] Starting autoregister controller
I0822 17:09:01.332462 6530 cache.go:32] Waiting for caches to sync for autoregister controller
I0822 17:09:01.432620 6530 cache.go:39] Caches are synced for autoregister controller
2018/08/22 17:09:01 Creating CRD clusters.clusterregistry.k8s.io
2018/08/22 17:09:01 Finished installing.
2018/08/22 17:09:01 Creating CRD federatedclusters.core.federation.k8s.io
2018/08/22 17:09:01 Creating CRD federatedconfigmaps.core.federation.k8s.io
2018/08/22 17:09:01 Creating CRD federatedconfigmapoverrides.core.federation.k8s.io
E0822 17:09:01.543430 6530 autoregister_controller.go:190] v1alpha1.core.federation.k8s.io failed with : apiservices.apiregistration.k8s.io "v1alpha1.core.federation.k8s.io" already exists
2018/08/22 17:09:01 Creating CRD federatedconfigmapplacements.core.federation.k8s.io
2018/08/22 17:09:01 Creating CRD federateddeployments.core.federation.k8s.io
2018/08/22 17:09:02 Creating CRD federateddeploymentoverrides.core.federation.k8s.io
2018/08/22 17:09:02 Creating CRD federateddeploymentplacements.core.federation.k8s.io
2018/08/22 17:09:02 Creating CRD federatedingresses.core.federation.k8s.io
2018/08/22 17:09:03 Creating CRD federatedingressplacements.core.federation.k8s.io
2018/08/22 17:09:03 Creating CRD federatedjobs.core.federation.k8s.io
2018/08/22 17:09:04 Creating CRD federatedjoboverrides.core.federation.k8s.io
2018/08/22 17:09:04 Creating CRD federatedjobplacements.core.federation.k8s.io
2018/08/22 17:09:04 Creating CRD federatednamespaceplacements.core.federation.k8s.io
2018/08/22 17:09:05 Creating CRD federatedreplicasets.core.federation.k8s.io
2018/08/22 17:09:05 Creating CRD federatedreplicasetoverrides.core.federation.k8s.io
2018/08/22 17:09:06 Creating CRD federatedreplicasetplacements.core.federation.k8s.io
2018/08/22 17:09:06 Creating CRD federatedsecrets.core.federation.k8s.io
2018/08/22 17:09:06 Creating CRD federatedsecretoverrides.core.federation.k8s.io
2018/08/22 17:09:07 Creating CRD federatedsecretplacements.core.federation.k8s.io
2018/08/22 17:09:07 Creating CRD federatedservices.core.federation.k8s.io
2018/08/22 17:09:08 Creating CRD federatedserviceplacements.core.federation.k8s.io
2018/08/22 17:09:08 Creating CRD federatedtypeconfigs.core.federation.k8s.io
2018/08/22 17:09:08 Creating CRD propagatedversions.core.federation.k8s.io
2018/08/22 17:09:09 Creating CRD dnsendpoints.multiclusterdns.federation.k8s.io
E0822 17:09:09.504614 6530 autoregister_controller.go:190] v1alpha1.multiclusterdns.federation.k8s.io failed with : apiservices.apiregistration.k8s.io "v1alpha1.multiclusterdns.federation.k8s.io" already exists
2018/08/22 17:09:09 Creating CRD multiclusterservicednsrecords.multiclusterdns.federation.k8s.io
2018/08/22 17:09:10 Creating CRD replicaschedulingpreferences.scheduling.federation.k8s.io
2018/08/22 17:09:10 Finished installing.
I0822 17:09:10.370457 6530 controller.go:537] quota admission added evaluator for: {clusterregistry.k8s.io clusters}
I0822 17:09:10.383750 6530 controller.go:537] quota admission added evaluator for: {core.federation.k8s.io federatedclusters}
Flag --insecure-port has been deprecated, This flag will be removed in a future version.
Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version.
I0822 17:09:10.526311 6543 server.go:135] Version: v0.0.0-master+$Format:%h$
I0822 17:09:10.735450 6543 serving.go:275] Generated self-signed cert (/tmp/k8s_test_framework_969878006/apiserver.crt, /tmp/k8s_test_framework_969878006/apiserver.key)
I0822 17:09:10.735480 6543 server.go:721] external host was not specified, using 10.138.0.2
W0822 17:09:10.735495 6543 authentication.go:377] AnonymousAuth is not allowed with the AllowAll authorizer. Resetting AnonymousAuth to false. You should use a different authorizer
W0822 17:09:10.737925 6543 admission.go:68] PersistentVolumeLabel admission controller is deprecated. Please remove this controller from your configuration files and scripts.
I0822 17:09:10.738432 6543 plugins.go:149] Loaded 9 admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultTolerationSeconds,PersistentVolumeLabel,DefaultStorageClass,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota.
W0822 17:09:10.738815 6543 admission.go:68] PersistentVolumeLabel admission controller is deprecated. Please remove this controller from your configuration files and scripts.
I0822 17:09:10.739238 6543 plugins.go:149] Loaded 9 admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultTolerationSeconds,PersistentVolumeLabel,DefaultStorageClass,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota.
I0822 17:09:10.742710 6543 master.go:228] Using reconciler: master-count
W0822 17:09:10.846552 6543 genericapiserver.go:342] Skipping API batch/v2alpha1 because it has no resources.
W0822 17:09:10.876275 6543 genericapiserver.go:342] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0822 17:09:10.877706 6543 genericapiserver.go:342] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0822 17:09:10.911640 6543 genericapiserver.go:342] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources.
[restful] 2018/08/22 17:09:10 log.go:33: [restful/swagger] listing is available at http://10.138.0.2:443/swaggerapi
[restful] 2018/08/22 17:09:10 log.go:33: [restful/swagger] http://10.138.0.2:443/swaggerui/ is mapped to folder /swagger-ui/
[restful] 2018/08/22 17:09:12 log.go:33: [restful/swagger] listing is available at http://10.138.0.2:443/swaggerapi
[restful] 2018/08/22 17:09:12 log.go:33: [restful/swagger] http://10.138.0.2:443/swaggerui/ is mapped to folder /swagger-ui/
W0822 17:09:12.518714 6543 admission.go:68] PersistentVolumeLabel admission controller is deprecated. Please remove this controller from your configuration files and scripts.
I0822 17:09:12.519156 6543 plugins.go:149] Loaded 9 admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultTolerationSeconds,PersistentVolumeLabel,DefaultStorageClass,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota.
I0822 17:09:15.796099 6543 insecure_handler.go:121] Serving insecurely on 127.0.0.1:40167
I0822 17:09:15.797498 6543 crd_finalizer.go:242] Starting CRDFinalizer
I0822 17:09:15.798188 6543 apiservice_controller.go:90] Starting APIServiceRegistrationController
I0822 17:09:15.798373 6543 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0822 17:09:15.798594 6543 customresource_discovery_controller.go:174] Starting DiscoveryController
I0822 17:09:15.798753 6543 naming_controller.go:276] Starting NamingConditionController
I0822 17:09:15.799978 6543 available_controller.go:262] Starting AvailableConditionController
I0822 17:09:15.799998 6543 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0822 17:09:15.800977 6543 controller.go:84] Starting OpenAPI AggregationController
I0822 17:09:15.801962 6543 crdregistration_controller.go:110] Starting crd-autoregister controller
I0822 17:09:15.801983 6543 controller_utils.go:1019] Waiting for caches to sync for crd-autoregister controller
I0822 17:09:15.831037 6543 controller.go:537] quota admission added evaluator for: { endpoints}
I0822 17:09:15.898610 6543 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0822 17:09:15.900160 6543 cache.go:39] Caches are synced for AvailableConditionController controller
I0822 17:09:15.902130 6543 controller_utils.go:1026] Caches are synced for crd-autoregister controller
I0822 17:09:15.902250 6543 autoregister_controller.go:136] Starting autoregister controller
I0822 17:09:15.902265 6543 cache.go:32] Waiting for caches to sync for autoregister controller
I0822 17:09:16.002418 6543 cache.go:39] Caches are synced for autoregister controller
=== RUN TestIntegration/Parallel-Integration-Test-Group
=== RUN TestIntegration/Parallel-Integration-Test-Group/TestCrud
=== PAUSE TestIntegration/Parallel-Integration-Test-Group/TestCrud
=== CONT TestIntegration/Parallel-Integration-Test-Group/TestCrud
=== RUN TestIntegration/Parallel-Integration-Test-Group/TestCrud/FederatedSecret
E0822 17:09:16.188318 6507 federated_informer.go:212] Cluster &{{FederatedCluster core.federation.k8s.io/v1alpha1} {test-cluster-67vs2 federation-system-trqsj /apis/core.federation.k8s.io/v1alpha1/namespaces/federation-system-trqsj/federatedclusters/test-cluster-67vs2 1612c587-a62e-11e8-be00-42010a8a0002 94 1 2018-08-22 17:09:10 +0000 UTC <nil> <nil> map[] map[] [] nil [] } {{test-cluster-67vs2} &LocalObjectReference{Name:test-cluster-67vs2-credentialspk8cd,}} {[] }} not added. Not of correct type, or cluster not ready.
E0822 17:09:16.188843 6507 federated_informer.go:212] Cluster &{{FederatedCluster core.federation.k8s.io/v1alpha1} {test-cluster-48wtw federation-system-trqsj /apis/core.federation.k8s.io/v1alpha1/namespaces/federation-system-trqsj/federatedclusters/test-cluster-48wtw 197fcd44-a62e-11e8-be00-42010a8a0002 128 1 2018-08-22 17:09:16 +0000 UTC <nil> <nil> map[] map[] [] nil [] } {{test-cluster-48wtw} &LocalObjectReference{Name:test-cluster-48wtw-credentials767rs,}} {[] }} not added. Not of correct type, or cluster not ready.
E0822 17:09:16.189827 6507 federated_informer.go:212] Cluster &{{FederatedCluster core.federation.k8s.io/v1alpha1} {test-cluster-67vs2 federation-system-trqsj /apis/core.federation.k8s.io/v1alpha1/namespaces/federation-system-trqsj/federatedclusters/test-cluster-67vs2 1612c587-a62e-11e8-be00-42010a8a0002 94 1 2018-08-22 17:09:10 +0000 UTC <nil> <nil> map[] map[] [] nil [] } {{test-cluster-67vs2} &LocalObjectReference{Name:test-cluster-67vs2-credentialspk8cd,}} {[] }} not added. Not of correct type, or cluster not ready.
E0822 17:09:16.189962 6507 federated_informer.go:212] Cluster &{{FederatedCluster core.federation.k8s.io/v1alpha1} {test-cluster-48wtw federation-system-trqsj /apis/core.federation.k8s.io/v1alpha1/namespaces/federation-system-trqsj/federatedclusters/test-cluster-48wtw 197fcd44-a62e-11e8-be00-42010a8a0002 128 1 2018-08-22 17:09:16 +0000 UTC <nil> <nil> map[] map[] [] nil [] } {{test-cluster-48wtw} &LocalObjectReference{Name:test-cluster-48wtw-credentials767rs,}} {[] }} not added. Not of correct type, or cluster not ready.
I0822 17:09:16.284697 6530 controller.go:537] quota admission added evaluator for: {core.federation.k8s.io federatedsecrets}
I0822 17:09:16.288853 6530 controller.go:537] quota admission added evaluator for: {core.federation.k8s.io federatedsecretplacements}
I0822 17:09:16.291675 6530 controller.go:537] quota admission added evaluator for: {core.federation.k8s.io federatedsecretoverrides}
E0822 17:09:16.295598 6507 controller.go:462] Failed to ensure finalizers for FederatedSecret "crud-federatedsecret-6hmjm/test-crud-6p2kq": failed to add finalizers map[federation.kubernetes.io/delete-from-underlying-clusters:{} orphan:{}] to object crud-federatedsecret-6hmjm/test-crud-6p2kq: Operation cannot be fulfilled on federatedsecrets.core.federation.k8s.io "test-crud-6p2kq": the object has been modified; please apply your changes to the latest version and try again
I0822 17:09:17.180361 6530 controller.go:537] quota admission added evaluator for: { namespaces}
I0822 17:09:46.596939 6530 autoregister_controller.go:160] Shutting down autoregister controller
I0822 17:09:46.596977 6530 crdregistration_controller.go:139] Shutting down crd-autoregister controller
I0822 17:09:46.596995 6530 naming_controller.go:287] Shutting down NamingConditionController
I0822 17:09:46.597009 6530 customresource_discovery_controller.go:185] Shutting down DiscoveryController
I0822 17:09:46.597019 6530 apiservice_controller.go:102] Shutting down APIServiceRegistrationController
I0822 17:09:46.597030 6530 available_controller.go:274] Shutting down AvailableConditionController
I0822 17:09:46.597043 6530 crd_finalizer.go:254] Shutting down CRDFinalizer
I0822 17:09:46.597396 6530 controller.go:90] Shutting down OpenAPI AggregationController
I0822 17:09:46.597950 6530 serve.go:136] Stopped listening on 127.0.0.1:42099
I0822 17:09:46.724046 6543 autoregister_controller.go:160] Shutting down autoregister controller
I0822 17:09:46.724080 6543 crdregistration_controller.go:139] Shutting down crd-autoregister controller
I0822 17:09:46.724097 6543 available_controller.go:274] Shutting down AvailableConditionController
I0822 17:09:46.724111 6543 naming_controller.go:287] Shutting down NamingConditionController
I0822 17:09:46.724124 6543 customresource_discovery_controller.go:185] Shutting down DiscoveryController
I0822 17:09:46.724136 6543 apiservice_controller.go:102] Shutting down APIServiceRegistrationController
I0822 17:09:46.724151 6543 crd_finalizer.go:254] Shutting down CRDFinalizer
I0822 17:09:46.724312 6543 controller.go:90] Shutting down OpenAPI AggregationController
I0822 17:09:46.724585 6543 serve.go:136] Stopped listening on 127.0.0.1:40167
--- FAIL: TestIntegration (51.72s)
logger.go:52: Starting a federation of 2 clusters...
logger.go:52: Creating Cluster Registry CRD
logger.go:52: Creating Federation CRDs
logger.go:52: Added cluster test-cluster-67vs2 to the federation
logger.go:52: Added cluster test-cluster-48wtw to the federation
logger.go:52: Starting cluster controller
logger.go:48: Federation started.
--- FAIL: TestIntegration/Parallel-Integration-Test-Group (0.00s)
--- FAIL: TestIntegration/Parallel-Integration-Test-Group/TestCrud (30.32s)
--- FAIL: TestIntegration/Parallel-Integration-Test-Group/TestCrud/FederatedSecret (30.32s)
logger.go:52: Creating new FederatedSecret in namespace "crud-federatedsecret-6hmjm"
logger.go:52: Created new FederatedSecret "crud-federatedsecret-6hmjm/test-crud-6p2kq"
logger.go:52: Creating new FederatedSecretPlacement in namespace "crud-federatedsecret-6hmjm"
logger.go:52: Created new FederatedSecretPlacement "crud-federatedsecret-6hmjm/test-crud-6p2kq"
logger.go:52: Creating new FederatedSecretOverride in namespace "crud-federatedsecret-6hmjm"
logger.go:52: Created new FederatedSecretOverride "crud-federatedsecret-6hmjm/test-crud-6p2kq"
logger.go:48: Resource versions for crud-federatedsecret-6hmjm/test-crud-6p2kq: template "130", placement "131", override "133"
logger.go:44: Error waiting for propagated version for Secret "crud-federatedsecret-6hmjm/test-crud-6p2kq": timed out waiting for the condition
FAIL
exit status 1
FAIL github.com/kubernetes-sigs/federation-v2/test/integration 51.737s
Raw
kubefedv2_join_logs
root@dev:~/go/src/github.com/kubernetes-sigs/federation-v2# ./bin/kubefed2 join cluster1 --cluster-context cluster1 \
> --host-cluster-context cluster1 --add-to-registry --v=2
I0824 19:10:35.339837 29837 join.go:131] Args and flags: name cluster1, host: cluster1, host-system-namespace: federation-system, cluster-namespace: kube-multicluster-public, kubeconfig: , cluster-context: cluster1, secret-name: , limited-scope: %!s(bool=false), dry-run: false
I0824 19:10:35.487010 29837 join.go:185] Performing preflight checks.
I0824 19:10:35.500006 29837 join.go:273] Registering cluster: cluster1 with the cluster registry.
I0824 19:10:35.579901 29837 join.go:282] Registered cluster: cluster1 with the cluster registry.
I0824 19:10:35.579930 29837 join.go:206] Creating federation-system namespace in joining cluster
I0824 19:10:35.651358 29837 join.go:214] Created federation-system namespace in joining cluster
I0824 19:10:35.651393 29837 join.go:217] Creating cluster credentials secret
I0824 19:10:35.651407 29837 join.go:408] Creating service account in joining cluster: cluster1
I0824 19:10:35.657466 29837 join.go:418] Created service account: cluster1-cluster1 in joining cluster: cluster1
I0824 19:10:35.657491 29837 join.go:434] Creating cluster role binding for service account: cluster1-cluster1 in joining cluster: cluster1
I0824 19:10:35.665156 29837 join.go:444] Created cluster role binding for service account: cluster1-cluster1 in joining cluster: cluster1
I0824 19:10:35.665196 29837 join.go:448] Creating secret in host cluster: cluster1
I0824 19:10:36.672500 29837 join.go:690] Using secret named: cluster1-cluster1-token-g7pwn
I0824 19:10:36.675681 29837 join.go:722] Created secret in host cluster named: cluster1-5rvdl
I0824 19:10:36.675711 29837 join.go:457] Created secret in host cluster: cluster1
I0824 19:10:36.675721 29837 join.go:227] Cluster credentials secret created
I0824 19:10:36.675728 29837 join.go:229] Creating federated cluster resource
I0824 19:10:36.688468 29837 join.go:238] Created federated cluster resource
root@dev:~/go/src/github.com/kubernetes-sigs/federation-v2# ./bin/kubefed2 join cluster2 --cluster-context cluster2 \
> --host-cluster-context cluster1 --add-to-registry --v=2^C
root@dev:~/go/src/github.com/kubernetes-sigs/federation-v2# ./bin/kubefed2 join cluster2 --cluster-context cluster2 \
> --host-cluster-context cluster1 --add-to-registry --v=2
I0824 19:12:21.565718 30343 join.go:131] Args and flags: name cluster2, host: cluster1, host-system-namespace: federation-system, cluster-namespace: kube-multicluster-public, kubeconfig: , cluster-context: cluster2, secret-name: , limited-scope: %!s(bool=false), dry-run: false
I0824 19:12:21.675917 30343 join.go:185] Performing preflight checks.
I0824 19:12:21.691735 30343 join.go:273] Registering cluster: cluster2 with the cluster registry.
I0824 19:12:21.708059 30343 join.go:282] Registered cluster: cluster2 with the cluster registry.
I0824 19:12:21.708086 30343 join.go:206] Creating federation-system namespace in joining cluster
I0824 19:12:21.712051 30343 join.go:214] Created federation-system namespace in joining cluster
I0824 19:12:21.712077 30343 join.go:217] Creating cluster credentials secret
I0824 19:12:21.712087 30343 join.go:408] Creating service account in joining cluster: cluster2
I0824 19:12:21.718664 30343 join.go:418] Created service account: cluster2-cluster1 in joining cluster: cluster2
I0824 19:12:21.718688 30343 join.go:434] Creating cluster role binding for service account: cluster2-cluster1 in joining cluster: cluster2
I0824 19:12:21.725355 30343 join.go:444] Created cluster role binding for service account: cluster2-cluster1 in joining cluster: cluster2
I0824 19:12:21.725385 30343 join.go:448] Creating secret in host cluster: cluster1
I0824 19:12:22.759454 30343 join.go:690] Using secret named: cluster2-cluster1-token-dgmbz
I0824 19:12:22.766731 30343 join.go:722] Created secret in host cluster named: cluster2-xzgvr
I0824 19:12:22.766769 30343 join.go:457] Created secret in host cluster: cluster1
I0824 19:12:22.766780 30343 join.go:227] Cluster credentials secret created
I0824 19:12:22.766788 30343 join.go:229] Creating federated cluster resource
I0824 19:12:22.796003 30343 join.go:238] Created federated cluster resource
Raw
notes.md

Introduction

This document captures pertinent information from my initial review on 8/18/18 of the k8s fedv2 project.

Partial Join Failure

If all actions of a join DO NOT suceed, the join should be considered in a FILED state. Any actions that suceeded should be undone. For example:

$ kubefed2 join cluster1 --cluster-context cluster1 \
> --add-to-registry --v=2
I0817 18:10:58.480026   15537 join.go:129] Args and flags: name cluster1, host: , host-system-namespace: federation-system, kubeconfig: , cluster-context: cluster1, secret-name: , dry-run: false
I0817 18:10:58.648580   15537 join.go:183] Performing preflight checks.
I0817 18:10:58.689124   15537 join.go:271] Registering cluster: cluster1 with the cluster registry.
I0817 18:10:58.810255   15537 join.go:280] Registered cluster: cluster1 with the cluster registry.
I0817 18:10:58.810286   15537 join.go:204] Creating federation-system namespace in joining cluster
I0817 18:10:58.814134   15537 join.go:212] Created federation-system namespace in joining cluster
I0817 18:10:58.814157   15537 join.go:215] Creating cluster credentials secret
I0817 18:10:58.814166   15537 join.go:406] Creating service account in joining cluster: cluster1
I0817 18:10:58.862792   15537 join.go:411] Error creating service account:  in joining cluster: cluster1 due to: ServiceAccount "cluster1-" is invalid: metadata.name: Invalid value: "cluster1-": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
I0817 18:10:58.862833   15537 join.go:221] Could not create cluster credentials secret: ServiceAccount "cluster1-" is invalid: metadata.name: Invalid value: "cluster1-": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
F0817 18:10:58.862846   15537 join.go:105] error: ServiceAccount "cluster1-" is invalid: metadata.name: Invalid value: "cluster1-": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')

By not undoing succcessful actions, it leaves the join in a broken state and the successful actions must be manually undone.

This join appears to work, but I see wierd log msgs:

# kubefed2 join cluster1 --cluster-context cluster1 \
>     --host-cluster-context cluster1 --add-to-registry --v=2
I0820 22:09:51.537079   11532 join.go:129] Args and flags: name cluster1, host: cluster1, host-system-namespace: federation-system, kubeconfig: , cluster-context: cluster1, secret-name: , dry-run: false
I0820 22:09:51.541981   11532 join.go:183] Performing preflight checks.
I0820 22:09:51.556052   11532 join.go:271] Registering cluster: cluster1 with the cluster registry.
I0820 22:09:51.586687   11532 join.go:280] Registered cluster: cluster1 with the cluster registry.
I0820 22:09:51.586716   11532 join.go:204] Creating federation-system namespace in joining cluster
I0820 22:09:51.590435   11532 join.go:212] Created federation-system namespace in joining cluster
I0820 22:09:51.590459   11532 join.go:215] Creating cluster credentials secret
I0820 22:09:51.590464   11532 join.go:406] Creating service account in joining cluster: cluster1
I0820 22:09:51.595837   11532 join.go:416] Created service account: cluster1-cluster1 in joining cluster: cluster1
I0820 22:09:51.595875   11532 join.go:418] Creating cluster role binding for service account: cluster1-cluster1 in joining cluster: cluster1
I0820 22:09:51.607615   11532 join.go:428] Created cluster role binding for service account: cluster1-cluster1 in joining cluster: cluster1
I0820 22:09:51.607639   11532 join.go:431] Creating secret in host cluster: cluster1
I0820 22:09:52.616612   11532 join.go:569] Using secret named: cluster1-cluster1-token-4lwp4
I0820 22:09:52.620976   11532 join.go:601] Created secret in host cluster named: cluster1-2mwbp
I0820 22:09:52.621007   11532 join.go:440] Created secret in host cluster: cluster1
I0820 22:09:52.621024   11532 join.go:225] Cluster credentials secret created
I0820 22:09:52.621031   11532 join.go:227] Creating federated cluster resource
I0820 22:09:52.632568   11532 join.go:236] Created federated cluster resource

From the container log:

E0820 22:09:52.647983       1 federated_informer.go:209] Cluster &{{ } {cluster1  federation-system /apis/core.federation.k8s.io/v1alpha1/namespaces/federation-system/federatedclusters/cluster1 c34276b5-a4c5-11e8-b24e-42010a8a0002 17505 1 2018-08-20 22:09:52 +0000 UTC <nil> <nil> map[] map[] [] nil [] } {{cluster1} &LocalObjectReference{Name:cluster1-2mwbp,}} {[]  }} not added.  Not of correct type, or cluster not ready.

Even before doing a join, I see this msg throughout the logs b/c install-latest.yaml (fed crds) does not include an ingress:

E0820 22:09:52.443028       1 reflector.go:205] github.com/kubernetes-sigs/federation-v2/pkg/controller/sync/controller.go:304: Failed to list <nil>: the server could not find the requested resource (get federatedingresss.core.federation.k8s.io)

Need to add a federatedns to install-latest.yaml

Federated config types used in example deploy:

# kubectl get federatedtypeconfig
NAME                   AGE
configmaps             21h
deployments.apps       21h
ingresses.extensions   21h
jobs.batch             21h
namespaces             21h
replicasets.apps       21h
secrets                21h
services               21h

crd's deployed by example:

# kubectl get crd | grep feder
federatedclusters.core.federation.k8s.io                          2018-08-20T19:36:46Z
federatedconfigmapoverrides.core.federation.k8s.io                2018-08-20T19:36:46Z
federatedconfigmapplacements.core.federation.k8s.io               2018-08-20T19:36:46Z
federatedconfigmaps.core.federation.k8s.io                        2018-08-20T19:36:46Z
federateddeploymentoverrides.core.federation.k8s.io               2018-08-20T19:36:46Z
federateddeploymentplacements.core.federation.k8s.io              2018-08-20T19:36:46Z
federateddeployments.core.federation.k8s.io                       2018-08-20T19:36:46Z
federatedjoboverrides.core.federation.k8s.io                      2018-08-20T19:36:46Z
federatedjobplacements.core.federation.k8s.io                     2018-08-20T19:36:46Z
federatedjobs.core.federation.k8s.io                              2018-08-20T19:36:46Z
federatednamespaceplacements.core.federation.k8s.io               2018-08-20T19:36:46Z
federatedreplicasetoverrides.core.federation.k8s.io               2018-08-20T19:36:46Z
federatedreplicasetplacements.core.federation.k8s.io              2018-08-20T19:36:46Z
federatedreplicasets.core.federation.k8s.io                       2018-08-20T19:36:46Z
federatedsecretoverrides.core.federation.k8s.io                   2018-08-20T19:36:46Z
federatedsecretplacements.core.federation.k8s.io                  2018-08-20T19:36:46Z
federatedsecrets.core.federation.k8s.io                           2018-08-20T19:36:46Z
federatedserviceplacements.core.federation.k8s.io                 2018-08-20T19:36:46Z
federatedservices.core.federation.k8s.io                          2018-08-20T19:36:46Z
federatedtypeconfigs.core.federation.k8s.io                       2018-08-20T19:36:46Z
multiclusterservicednsrecords.multiclusterdns.federation.k8s.io   2018-08-20T19:36:46Z
propagatedversions.core.federation.k8s.io                         2018-08-20T19:36:46Z
replicaschedulingpreferences.scheduling.federation.k8s.io         2018-08-20T19:36:46Z
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment