Created
June 21, 2018 20:08
-
-
Save nrb/3e1b986deaa1f3d4e0a6c190be23940b to your computer and use it in GitHub Desktop.
restic backup hang
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ./_output/bin/linux/amd64/ark restic repo init nginx-example | |
| ./_output/bin/linux/amd64/ark restic repo change-key nginx-example | |
| kubectl -n nginx-example annotate pod/nginx-deployment-99997d74d-j2qvh backup.ark.heptio.com/backup-volumes=nginx-logs | |
| ./_output/bin/linux/amd64/ark backup create nginx --include-namespaces nginx-example | |
| # backup created, but hasn't been updated. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| % ./_output/bin/linux/amd64/ark backup get nginx -o yaml | |
| apiVersion: ark.heptio.com/v1 | |
| kind: Backup | |
| metadata: | |
| creationTimestamp: 2018-06-21T19:58:37Z | |
| name: nginx | |
| namespace: heptio-ark | |
| resourceVersion: "2040" | |
| selfLink: /apis/ark.heptio.com/v1/namespaces/heptio-ark/backups/nginx | |
| uid: 7cca3773-758d-11e8-b147-42010a960092 | |
| spec: | |
| excludedNamespaces: null | |
| excludedResources: null | |
| hooks: | |
| resources: null | |
| includeClusterResources: null | |
| includedNamespaces: | |
| - nginx-example | |
| includedResources: null | |
| labelSelector: null | |
| snapshotVolumes: null | |
| ttl: 720h0m0s | |
| status: | |
| expiration: 2018-07-21T19:58:37Z | |
| phase: InProgress | |
| validationErrors: null | |
| version: 1 | |
| volumeBackups: null |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| apiVersion: v1 | |
| items: | |
| - apiVersion: ark.heptio.com/v1 | |
| kind: PodVolumeBackup | |
| metadata: | |
| clusterName: "" | |
| creationTimestamp: 2018-06-21T19:58:45Z | |
| generateName: nginx- | |
| labels: | |
| ark.heptio.com/backup-name: nginx | |
| ark.heptio.com/backup-uid: 7cca3773-758d-11e8-b147-42010a960092 | |
| name: nginx-7n8kf | |
| namespace: heptio-ark | |
| ownerReferences: | |
| - apiVersion: ark.heptio.com/v1 | |
| controller: true | |
| kind: Backup | |
| name: nginx | |
| uid: 7cca3773-758d-11e8-b147-42010a960092 | |
| resourceVersion: "2052" | |
| selfLink: /apis/ark.heptio.com/v1/namespaces/heptio-ark/podvolumebackups/nginx-7n8kf | |
| uid: 816c15fa-758d-11e8-b147-42010a960092 | |
| spec: | |
| node: gke-cluster-1-default-pool-0b8556a0-bcfx | |
| pod: | |
| kind: Pod | |
| name: nginx-deployment-99997d74d-j2qvh | |
| namespace: nginx-example | |
| uid: 41e09172-758b-11e8-b147-42010a960092 | |
| repoIdentifier: gs:<REDACTED>_restic:/nginx-example | |
| tags: | |
| backup: nginx | |
| backup-uid: 7cca3773-758d-11e8-b147-42010a960092 | |
| ns: nginx-example | |
| pod: nginx-deployment-99997d74d-j2qvh | |
| pod-uid: 41e09172-758b-11e8-b147-42010a960092 | |
| volume: nginx-logs | |
| volume: nginx-logs | |
| status: | |
| message: "" | |
| path: "" | |
| phase: "" | |
| snapshotID: "" | |
| kind: List | |
| metadata: | |
| resourceVersion: "" | |
| selfLink: "" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| x1c in /home/nrb/go/src/github.com/heptio/ark (git) skriss-rotate-restic-key U | |
| % GOOGLE_APPLICATION_CREDENTIALS=credentials-ark-gke ./_output/bin/linux/amd64/ark server --log-level debug | |
| INFO[0000] setting log-level to DEBUG | |
| INFO[0000] Starting Ark server v0.9.0-alpha.2-25-ge12e2b5-dirty logSource="pkg/cmd/server/server.go:100" | |
| INFO[0000] Ensuring namespace exists for backups logSource="pkg/cmd/server/server.go:259" namespace=heptio-ark | |
| INFO[0000] Namespace already exists logSource="pkg/cmd/server/server.go:271" namespace=heptio-ark | |
| INFO[0000] Retrieving Ark configuration logSource="pkg/cmd/server/server.go:276" | |
| INFO[0000] Successfully retrieved Ark configuration logSource="pkg/cmd/server/server.go:294" | |
| INFO[0000] Using default resource priorities logSource="pkg/cmd/server/server.go:344" priorities="[namespaces persistentvolumes persistentvolumeclaims secrets configmaps serviceaccounts limitranges pods]" | |
| INFO[0000] Configuring cloud provider for backup service logSource="pkg/cmd/server/server.go:393" | |
| DEBU[0000] starting plugin args="[./_output/bin/linux/amd64/ark run-plugin cloudprovider gcp]" logSource="pkg/plugin/logrus_adapter.go:74" path=./_output/bin/linux/amd64/ark | |
| DEBU[0000] waiting for RPC address logSource="pkg/plugin/logrus_adapter.go:74" path=./_output/bin/linux/amd64/ark | |
| DEBU[0000] plugin address address=/tmp/plugin973202925 logSource="pkg/plugin/logrus_adapter.go:74" network=unix pluginName=ark | |
| INFO[0000] Configuring cloud provider for snapshot service logSource="pkg/cmd/server/server.go:410" | |
| INFO[0001] Starting controllers logSource="pkg/cmd/server/server.go:499" | |
| INFO[0001] Caching cloud backups every 30m0s logSource="pkg/cmd/server/server.go:505" | |
| DEBU[0001] refreshing all cached backup lists from object storage logSource="pkg/cloudprovider/backup_cache.go:69" | |
| INFO[0001] Running backup sync controller logSource="pkg/controller/backup_sync_controller.go:70" | |
| INFO[0001] Syncing backups from object storage logSource="pkg/controller/backup_sync_controller.go:78" | |
| DEBU[0001] Bucket is not in cache - doing a live lookup bucket=<REDACTED> logSource="pkg/cloudprovider/backup_cache.go:89" | |
| INFO[0001] Starting metric server at address [:8085] logSource="pkg/cmd/server/server.go:516" | |
| INFO[0001] Got backups from object storage backupCount=0 logSource="pkg/controller/backup_sync_controller.go:84" | |
| INFO[0003] Starting BackupController logSource="pkg/controller/backup_controller.go:153" | |
| INFO[0003] Waiting for caches to sync logSource="pkg/controller/backup_controller.go:156" | |
| INFO[0003] Starting ScheduleController logSource="pkg/controller/schedule_controller.go:135" | |
| INFO[0003] Waiting for caches to sync logSource="pkg/controller/schedule_controller.go:138" | |
| INFO[0003] Starting controller controller=gc-controller logSource="pkg/controller/generic_controller.go:77" | |
| INFO[0003] Waiting for caches to sync controller=gc-controller logSource="pkg/controller/generic_controller.go:80" | |
| INFO[0003] Starting controller controller=restic-repository logSource="pkg/controller/generic_controller.go:77" | |
| INFO[0003] Starting DownloadRequestController logSource="pkg/controller/download_request_controller.go:122" | |
| INFO[0003] Waiting for caches to sync logSource="pkg/controller/download_request_controller.go:125" | |
| INFO[0003] Waiting for caches to sync controller=restic-repository logSource="pkg/controller/generic_controller.go:80" | |
| INFO[0003] Starting RestoreController logSource="pkg/controller/restore_controller.go:154" | |
| INFO[0003] Waiting for caches to sync logSource="pkg/controller/restore_controller.go:157" | |
| INFO[0003] Starting controller controller=restic-key-change logSource="pkg/controller/generic_controller.go:77" | |
| INFO[0003] Waiting for caches to sync controller=restic-key-change logSource="pkg/controller/generic_controller.go:80" | |
| INFO[0003] Starting controller controller=backup-deletion logSource="pkg/controller/generic_controller.go:77" | |
| INFO[0003] Waiting for caches to sync controller=backup-deletion logSource="pkg/controller/generic_controller.go:80" | |
| INFO[0003] Caches are synced controller=restic-repository logSource="pkg/controller/generic_controller.go:84" | |
| DEBU[0003] resticRepositoryController.enqueueAllRepositories controller=restic-repository logSource="pkg/controller/restic_repository_controller.go:88" | |
| INFO[0003] Caches are synced controller=restic-key-change logSource="pkg/controller/generic_controller.go:84" | |
| INFO[0003] Caches are synced logSource="pkg/controller/backup_controller.go:160" | |
| INFO[0003] Server started successfully logSource="pkg/cmd/server/server.go:719" | |
| INFO[0003] Caches are synced logSource="pkg/controller/schedule_controller.go:142" | |
| INFO[0003] Caches are synced controller=gc-controller logSource="pkg/controller/generic_controller.go:84" | |
| DEBU[0003] gcController.enqueueAllBackups logSource="pkg/controller/gc_controller.go:88" | |
| INFO[0003] Caches are synced logSource="pkg/controller/download_request_controller.go:129" | |
| INFO[0003] Caches are synced logSource="pkg/controller/restore_controller.go:161" | |
| INFO[0003] Caches are synced controller=backup-deletion logSource="pkg/controller/generic_controller.go:84" | |
| INFO[0003] Checking for expired DeleteBackupRequests controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:310" | |
| INFO[0003] Done checking for expired DeleteBackupRequests controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:338" | |
| DEBU[0186] Running processQueueItem controller=restic-repository key=heptio-ark/nginx-example logSource="pkg/controller/restic_repository_controller.go:103" | |
| INFO[0186] Initializing restic repository controller=restic-repository logSource="pkg/controller/restic_repository_controller.go:138" name=nginx-example namespace=heptio-ark | |
| DEBU[0187] Ran restic command command="restic check --repo=gs:<REDACTED>_restic:/nginx-example --password-file=/tmp/343676998" logSource="pkg/restic/repository_manager.go:369" repository=nginx-example stderr="Fatal: unable to open config file: service.Objects.Get: googleapi: Error 404: No such object: <REDACTED>_restic/nginx-example/config, notFound\nIs there a repository at the following location?\ngs:<REDACTED>_restic:/nginx-example\n" stdout="using temporary cache in /tmp/restic-check-cache-251735651\n" | |
| DEBU[0191] Ran restic command command="restic init --repo=gs:<REDACTED>_restic:/nginx-example --password-file=/tmp/058945261" logSource="pkg/restic/repository_manager.go:369" repository=nginx-example stderr= stdout="created restic repository 59b1a739fd at gs:<REDACTED>_restic:/nginx-example\n\nPlease note that knowledge of your password is required to access\nthe repository. Losing your password means that your data is\nirrecoverably lost.\n" | |
| DEBU[0475] Running processQueueItem controller=restic-key-change key=nginx-example/ark-restic-credentials logSource="pkg/controller/restic_key_change_controller.go:111" | |
| INFO[0475] Processing repository key change controller=restic-key-change logSource="pkg/controller/restic_key_change_controller.go:142" name=ark-restic-credentials namespace=nginx-example | |
| DEBU[0475] Changing key logSource="pkg/restic/repository_manager.go:218" repo=nginx-example | |
| DEBU[0475] Found key in secret at new-key logSource="pkg/restic/repository_manager.go:239" repo=nginx-example | |
| DEBU[0477] Key in new-key is not valid logSource="pkg/restic/repository_manager.go:257" repo=nginx-example | |
| DEBU[0482] Ran restic command command="restic key passwd --repo=gs:<REDACTED>_restic:/nginx-example --password-file=/tmp/182908839 --new-password-file=/tmp/389628264" logSource="pkg/restic/repository_manager.go:369" repository=nginx-example stderr= stdout="saved new key as <Key of nrb@x1c, created on 2018-06-21 15:52:51.020864706 -0400 EDT m=+4.023001500>\n" | |
| DEBU[0482] Ran restic key passwd command successfully logSource="pkg/restic/repository_manager.go:261" repo=nginx-example | |
| DEBU[0482] Patched secret to move new-key into ark-restic-credentials and ark-restic-credentials into old-key logSource="pkg/restic/repository_manager.go:271" repo=nginx-example | |
| DEBU[0482] Found key in secret at old-key logSource="pkg/restic/repository_manager.go:280" repo=nginx-example | |
| DEBU[0483] Patched secret to remove old-key logSource="pkg/restic/repository_manager.go:305" repo=nginx-example | |
| DEBU[0483] Running processQueueItem controller=restic-key-change key=nginx-example/ark-restic-credentials logSource="pkg/controller/restic_key_change_controller.go:111" | |
| DEBU[0827] Running processBackup key=heptio-ark/nginx logSource="pkg/controller/backup_controller.go:210" | |
| DEBU[0827] Backup has not expired yet, skipping backup=heptio-ark/nginx expiration="0001-01-01 00:00:00 +0000 UTC" logSource="pkg/controller/gc_controller.go:129" | |
| DEBU[0827] Getting backup key=heptio-ark/nginx logSource="pkg/controller/backup_controller.go:216" | |
| DEBU[0827] Cloning backup key=heptio-ark/nginx logSource="pkg/controller/backup_controller.go:237" | |
| DEBU[0828] Backup has not expired yet, skipping backup=heptio-ark/nginx expiration="2018-07-21 15:58:37 -0400 EDT" logSource="pkg/controller/gc_controller.go:129" | |
| DEBU[0828] Running backup key=heptio-ark/nginx logSource="pkg/controller/backup_controller.go:274" | |
| INFO[0828] Starting backup backup=heptio-ark/nginx logSource="pkg/controller/backup_controller.go:339" | |
| DEBU[0828] starting plugin args="[./_output/bin/linux/amd64/ark run-plugin backupitemaction pv]" logSource="pkg/plugin/logrus_adapter.go:74" path=./_output/bin/linux/amd64/ark | |
| DEBU[0828] waiting for RPC address logSource="pkg/plugin/logrus_adapter.go:74" path=./_output/bin/linux/amd64/ark | |
| DEBU[0828] plugin address address=/tmp/plugin130920220 logSource="pkg/plugin/logrus_adapter.go:74" network=unix pluginName=ark | |
| DEBU[0828] starting plugin args="[./_output/bin/linux/amd64/ark run-plugin backupitemaction pod]" logSource="pkg/plugin/logrus_adapter.go:74" path=./_output/bin/linux/amd64/ark | |
| DEBU[0828] waiting for RPC address logSource="pkg/plugin/logrus_adapter.go:74" path=./_output/bin/linux/amd64/ark | |
| DEBU[0828] plugin address address=/tmp/plugin130920220 logSource="pkg/plugin/logrus_adapter.go:74" network=unix pluginName=ark | |
| DEBU[0828] starting plugin args="[./_output/bin/linux/amd64/ark run-plugin backupitemaction pod]" logSource="pkg/plugin/logrus_adapter.go:74" path=./_output/bin/linux/amd64/ark | |
| DEBU[0828] waiting for RPC address logSource="pkg/plugin/logrus_adapter.go:74" path=./_output/bin/linux/amd64/ark | |
| DEBU[0828] plugin address address=/tmp/plugin716157810 logSource="pkg/plugin/logrus_adapter.go:74" network=unix pluginName=ark | |
| DEBU[0828] starting plugin args="[./_output/bin/linux/amd64/ark run-plugin backupitemaction serviceaccount]" logSource="pkg/plugin/logrus_adapter.go:74" path=./_output/bin/linux/amd64/ark | |
| DEBU[0828] waiting for RPC address logSource="pkg/plugin/logrus_adapter.go:74" path=./_output/bin/linux/amd64/ark | |
| DEBU[0828] plugin address address=/tmp/plugin332854551 logSource="pkg/plugin/logrus_adapter.go:74" network=unix pluginName=ark |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment