Skip to content

Instantly share code, notes, and snippets.

@mabhi

mabhi/gist:b191eabed93a0b60c75442d4927cc6ae

Last active July 25, 2024 13:34
Show Gist options
  • Save mabhi/b191eabed93a0b60c75442d4927cc6ae to your computer and use it in GitHub Desktop.
Save mabhi/b191eabed93a0b60c75442d4927cc6ae to your computer and use it in GitHub Desktop.
Download ZIP
Capture Pod logs on job fail during DR E2E test run
Raw
gistfile1.txt
restore_test.go:319: Job k10-restore-k10restore failed with message: "Job has reached the specified backoff limit", reason: "BackoffLimitExceeded"
restore_test.go:354: k10-restore-k10restore-nf5zw logs:
{"File":"kasten.io/k10/kio/kube/client.go","Function":"kasten.io/k10/kio/kube.LoadConfig","Line":21,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","hostname":"k10-restore-k10restore-nf5zw","level":"debug","msg":"Attempting to use InCluster config","time":"2024-07-25T13:03:27.258Z"}
{"File":"kasten.io/k10/kio/tools/restorectl/restore.go","Function":"kasten.io/k10/kio/tools/restorectl.runRestoreCommand.func1","Line":65,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"Waiting for restore to start","time":"2024-07-25T13:03:27.265Z"}
{"File":"kasten.io/k10/kio/dr/utils.go","Function":"kasten.io/k10/kio/dr.RestoreStatusForProgress","Line":171,"Progress":"Restore started","cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"K10 Disaster Recovery in progress","time":"2024-07-25T13:03:27.265Z"}
{"File":"kasten.io/k10/kio/dr/utils.go","Function":"kasten.io/k10/kio/dr.RestoreStatusForProgress","Line":171,"Progress":"Initialized DR using quick mode workflow","cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"K10 Disaster Recovery in progress","time":"2024-07-25T13:03:27.281Z"}
{"File":"kasten.io/k10/kio/dr/utils.go","Function":"kasten.io/k10/kio/dr.RestoreStatusForProgress","Line":171,"Progress":"Fetching data from storage","cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"K10 Disaster Recovery in progress","time":"2024-07-25T13:03:27.291Z"}
{"File":"kasten.io/k10/kio/kanister/override.go","Function":"kasten.io/k10/kio/kanister.getPodOverrideFieldValue","Line":175,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","configMap":"pod-spec-override","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"PodOverride ConfigMap is not provided. Only K10 default settings will be used","time":"2024-07-25T13:03:27.302Z"}
{"File":"kasten.io/k10/kio/tools/restorectl/restore.go","Function":"kasten.io/k10/kio/tools/restorectl.runRestoreCommand.func1","Line":68,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"Waiting for restore to complete","progress":"Fetching data from storage","time":"2024-07-25T13:03:27.365Z"}
{"File":"kasten.io/k10/kio/tools/restorectl/restore.go","Function":"kasten.io/k10/kio/tools/restorectl.runRestoreCommand.func1","Line":68,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"Waiting for restore to complete","progress":"Fetching data from storage","time":"2024-07-25T13:03:27.566Z"}
{"File":"kasten.io/k10/kio/kanister/utils.go","Function":"kasten.io/k10/kio/kanister.GetGoDebugEnvironmentVariablesFromFeatureflag","Line":280,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","flag":"K10DebugDatamoverServicePodGoMemLimit","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"checking feature flag","time":"2024-07-25T13:03:27.683Z"}
{"File":"kasten.io/k10/kio/kanister/utils.go","Function":"kasten.io/k10/kio/kanister.GetGoDebugEnvironmentVariablesFromFeatureflag","Line":283,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","flag":"K10DebugDatamoverServicePodGoMemLimit","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"skipping, no flag found","time":"2024-07-25T13:03:27.683Z"}
{"File":"kasten.io/k10/kio/kanister/utils.go","Function":"kasten.io/k10/kio/kanister.GetGoDebugEnvironmentVariablesFromFeatureflag","Line":280,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","flag":"K10DebugDatamoverServicePodGoMaxProcs","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"checking feature flag","time":"2024-07-25T13:03:27.683Z"}
{"File":"kasten.io/k10/kio/kanister/utils.go","Function":"kasten.io/k10/kio/kanister.GetGoDebugEnvironmentVariablesFromFeatureflag","Line":283,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","flag":"K10DebugDatamoverServicePodGoMaxProcs","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"skipping, no flag found","time":"2024-07-25T13:03:27.683Z"}
{"File":"kasten.io/k10/kio/kanister/utils.go","Function":"kasten.io/k10/kio/kanister.GetGoDebugEnvironmentVariablesFromFeatureflag","Line":280,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","flag":"K10DebugDatamoverServicePodGoGc","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"checking feature flag","time":"2024-07-25T13:03:27.683Z"}
{"File":"kasten.io/k10/kio/kanister/utils.go","Function":"kasten.io/k10/kio/kanister.GetGoDebugEnvironmentVariablesFromFeatureflag","Line":283,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","flag":"K10DebugDatamoverServicePodGoGc","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"skipping, no flag found","time":"2024-07-25T13:03:27.683Z"}
{"File":"kasten.io/k10/kio/kanister/utils.go","Function":"kasten.io/k10/kio/kanister.GetGoDebugEnvironmentVariablesFromFeatureflag","Line":280,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","flag":"K10DebugDatamoverServicePodGoTraceback","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"checking feature flag","time":"2024-07-25T13:03:27.683Z"}
{"File":"kasten.io/k10/kio/kanister/utils.go","Function":"kasten.io/k10/kio/kanister.GetGoDebugEnvironmentVariablesFromFeatureflag","Line":283,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","flag":"K10DebugDatamoverServicePodGoTraceback","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"skipping, no flag found","time":"2024-07-25T13:03:27.684Z"}
{"File":"kasten.io/k10/kio/kanister/utils.go","Function":"kasten.io/k10/kio/kanister.GetGoDebugEnvironmentVariablesFromFeatureflag","Line":280,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","flag":"K10DebugDatamoverServicePodGoDebug","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"checking feature flag","time":"2024-07-25T13:03:27.684Z"}
{"File":"kasten.io/k10/kio/kanister/utils.go","Function":"kasten.io/k10/kio/kanister.GetGoDebugEnvironmentVariablesFromFeatureflag","Line":283,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","flag":"K10DebugDatamoverServicePodGoDebug","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"skipping, no flag found","time":"2024-07-25T13:03:27.684Z"}
{"File":"kasten.io/k10/kio/tools/restorectl/restore.go","Function":"kasten.io/k10/kio/tools/restorectl.runRestoreCommand.func1","Line":68,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"Waiting for restore to complete","progress":"Fetching data from storage","time":"2024-07-25T13:03:27.967Z"}
{"File":"kasten.io/k10/kio/tools/restorectl/restore.go","Function":"kasten.io/k10/kio/tools/restorectl.runRestoreCommand.func1","Line":68,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"Waiting for restore to complete","progress":"Fetching data from storage","time":"2024-07-25T13:03:28.767Z"}
{"Command":"kopia --log-level=error --config-file=/tmp/kopia-repository.config --log-dir=/tmp/kopia-log --password=\u003c****\u003e repository connect --no-check-for-updates --cache-directory=/tmp/kopia-cache --content-cache-size-limit-mb=0 --metadata-cache-size-limit-mb=500 --override-hostname=data-mover-server-pod --override-username=k10-admin s3 --bucket=kasten-io-tests --access-key=\u003c****\u003e --secret-access-key=\u003c****\u003e --prefix=k10/24b70a84-079a-40a1-b057-d19627ca1ad5/migration/24b70a84-079a-40a1-b057-d19627ca1ad5/k10/repo/ --region=us-west-2","File":"kasten.io/k10/kio/kopia/kopia.go","Function":"kasten.io/k10/kio/kopia.stringSliceCommand","Line":134,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"kopia command","time":"2024-07-25T13:03:30.273Z"}
{"File":"kasten.io/k10/kio/tools/restorectl/restore.go","Function":"kasten.io/k10/kio/tools/restorectl.runRestoreCommand.func1","Line":68,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"Waiting for restore to complete","progress":"Fetching data from storage","time":"2024-07-25T13:03:30.368Z"}
{"Command":"kopia --log-level=info --config-file=/tmp/kopia-repository.config --log-dir=/tmp/kopia-log repository status --json","File":"pkg/kopia/command/repository.go","Function":"github.com/kanisterio/kanister/pkg/kopia/command.RepositoryStatusCommand","Line":183,"hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"Kopia Command","time":"2024-07-25T13:03:31.485736386Z"}
{"File":"kasten.io/k10/kio/kopia/utils.go","Function":"kasten.io/k10/kio/kopia.execWithLogging","Line":40,"PodExec":"data-mover-svc-8vnts","callerFile":"kasten.io/k10/kio/kopia/repository.go","callerFunction":"kasten.io/k10/kio/kopia.isRetentionAlreadySet","callerLine":1010,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","hostname":"k10-restore-k10restore-nf5zw","level":"debug","msg":"executing set retention period command","time":"2024-07-25T13:03:31.485Z"}
{"File":"kasten.io/k10/kio/kopia/utils.go","Function":"kasten.io/k10/kio/kopia.execWithLogging","Line":50,"PodExec":"data-mover-svc-8vnts","Stderr":"","Stdout":"{\"configFile\":\"/tmp/kopia-repository.config\",\"uniqueIDHex\":\"886c62c05fe678684654fbca51866c0cc3cba473eddd5a618f724859610b21ff\",\"clientOptions\":{\"hostname\":\"data-mover-server-pod\",\"username\":\"k10-admin\",\"description\":\"Repository in S3: s3.amazonaws.com kasten-io-tests\",\"enableActions\":false,\"formatBlobCacheDuration\":900000000000},\"storage\":{\"type\":\"s3\",\"config\":{\"bucket\":\"kasten-io-tests\",\"prefix\":\"k10/24b70a84-079a-40a1-b057-d19627ca1ad5/migration/24b70a84-079a-40a1-b057-d19627ca1ad5/k10/repo/\",\"endpoint\":\"s3.amazonaws.com\",\"accessKeyID\":\"AKIAQREAKNKDIZ5MIFHL\",\"secretAccessKey\":\"****************************************\",\"sessionToken\":\"\",\"region\":\"us-west-2\"}},\"contentFormat\":{\"hash\":\"BLAKE2B-256-128\",\"encryption\":\"AES256-GCM-HMAC-SHA256\",\"version\":3,\"maxPackSize\":20971520,\"indexVersion\":2,\"epochParameters\":{\"Enabled\":true,\"EpochRefreshFrequency\":1200000000000,\"FullCheckpointFrequency\":7,\"CleanupSafetyMargin\":14400000000000,\"MinEpochDuration\":86400000000000,\"EpochAdvanceOnCountThreshold\":20,\"EpochAdvanceOnTotalSizeBytesThreshold\":10485760,\"DeleteParallelism\":4},\"enablePasswordChange\":true},\"objectFormat\":{\"splitter\":\"DYNAMIC-4M-BUZHASH\"},\"blobRetention\":{}}\n","callerFile":"kasten.io/k10/kio/kopia/repository.go","callerFunction":"kasten.io/k10/kio/kopia.isRetentionAlreadySet","callerLine":1010,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","hostname":"k10-restore-k10restore-nf5zw","level":"debug","msg":"set retention period command output","time":"2024-07-25T13:03:32.298Z"}
{"Container":"container","File":"pkg/format/format.go","Function":"github.com/kanisterio/kanister/pkg/format.LogWithCtx","Line":90,"LogKind":"datapath","Out":"{\"configFile\":\"/tmp/kopia-repository.config\",\"uniqueIDHex\":\"886c62c05fe678684654fbca51866c0cc3cba473eddd5a618f724859610b21ff\",\"clientOptions\":{\"hostname\":\"data-mover-server-pod\",\"username\":\"k10-admin\",\"description\":\"Repository in S3: s3.amazonaws.com kasten-io-tests\",\"enableActions\":false,\"formatBlobCacheDuration\":900000000000},\"storage\":{\"type\":\"s3\",\"config\":{\"bucket\":\"kasten-io-tests\",\"prefix\":\"k10/24b70a84-079a-40a1-b057-d19627ca1ad5/migration/24b70a84-079a-40a1-b057-d19627ca1ad5/k10/repo/\",\"endpoint\":\"s3.amazonaws.com\",\"accessKeyID\":\"AKIAQREAKNKDIZ5MIFHL\",\"secretAccessKey\":\"****************************************\",\"sessionToken\":\"\",\"region\":\"us-west-2\"}},\"contentFormat\":{\"hash\":\"BLAKE2B-256-128\",\"encryption\":\"AES256-GCM-HMAC-SHA256\",\"version\":3,\"maxPackSize\":20971520,\"indexVersion\":2,\"epochParameters\":{\"Enabled\":true,\"EpochRefreshFrequency\":1200000000000,\"FullCheckpointFrequency\":7,\"CleanupSafetyMargin\":14400000000000,\"MinEpochDuration\":86400000000000,\"EpochAdvanceOnCountThreshold\":20,\"EpochAdvanceOnTotalSizeBytesThreshold\":10485760,\"DeleteParallelism\":4},\"enablePasswordChange\":true},\"objectFormat\":{\"splitter\":\"DYNAMIC-4M-BUZHASH\"},\"blobRetention\":{}}","Pod":"data-mover-svc-8vnts","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"Pod Update","time":"2024-07-25T13:03:32.298899257Z"}
{"Command":"kopia --log-level=error --config-file=/tmp/kopia-repository.config --log-dir=/tmp/kopia-log maintenance info --json","File":"pkg/kopia/command/maintenance.go","Function":"github.com/kanisterio/kanister/pkg/kopia/command.MaintenanceInfo","Line":30,"hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"Kopia Command","time":"2024-07-25T13:03:32.299685328Z"}
{"Container":"container","File":"pkg/format/format.go","Function":"github.com/kanisterio/kanister/pkg/format.LogWithCtx","Line":90,"LogKind":"datapath","Out":"{\"owner\":\"k10-admin@maintenance-owner\",\"quick\":{\"enabled\":false,\"interval\":3600000000000},\"full\":{\"enabled\":true,\"interval\":86400000000000},\"logRetention\":{\"maxTotalSize\":1073741824,\"maxCount\":10000,\"maxAge\":2592000000000000},\"extendObjectLocks\":false,\"schedule\":{\"nextFullMaintenance\":\"0001-01-01T00:00:00Z\",\"nextQuickMaintenance\":\"0001-01-01T00:00:00Z\",\"runs\":null}}","Pod":"data-mover-svc-8vnts","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"Pod Update","time":"2024-07-25T13:03:33.236331657Z"}
{"Command":"kopia --log-level=error --config-file=/tmp/kopia-repository.config --log-dir=/tmp/kopia-log --password=\u003c****\u003e policy show --global --json","File":"pkg/kopia/command/policy_show_global.go","Function":"github.com/kanisterio/kanister/pkg/kopia/command.PolicyShowGlobal","Line":30,"hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"Kopia Command","time":"2024-07-25T13:03:33.236757249Z"}
{"File":"kasten.io/k10/kio/kopia/utils.go","Function":"kasten.io/k10/kio/kopia.execWithLogging","Line":40,"PodExec":"data-mover-svc-8vnts","callerFile":"kasten.io/k10/kio/kopia/repository.go","callerFunction":"kasten.io/k10/kio/kopia.checkGlobalPolicySettings","callerLine":1127,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","hostname":"k10-restore-k10restore-nf5zw","level":"debug","msg":"executing show global policy command","time":"2024-07-25T13:03:33.236Z"}
{"File":"kasten.io/k10/kio/tools/restorectl/restore.go","Function":"kasten.io/k10/kio/tools/restorectl.runRestoreCommand.func1","Line":68,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"Waiting for restore to complete","progress":"Fetching data from storage","time":"2024-07-25T13:03:33.569Z"}
{"File":"kasten.io/k10/kio/kopia/utils.go","Function":"kasten.io/k10/kio/kopia.execWithLogging","Line":50,"PodExec":"data-mover-svc-8vnts","Stderr":"","Stdout":"{\"retention\":{\"keepLatest\":2147483647,\"keepHourly\":0,\"keepDaily\":0,\"keepWeekly\":0,\"keepMonthly\":0,\"keepAnnual\":0,\"ignoreIdenticalSnapshots\":false},\"files\":{\"ignoreDotFiles\":[\".kopiaignore\"]},\"errorHandling\":{\"ignoreFileErrors\":false,\"ignoreDirectoryErrors\":false,\"ignoreUnknownTypes\":true},\"scheduling\":{\"runMissed\":true},\"compression\":{\"compressorName\":\"s2-default\"},\"actions\":{},\"osSnapshots\":{\"volumeShadowCopy\":{\"enable\":0}},\"logging\":{\"directories\":{\"snapshotted\":5,\"ignored\":5},\"entries\":{\"snapshotted\":0,\"ignored\":5,\"cacheHit\":0,\"cacheMiss\":0}},\"upload\":{\"maxParallelSnapshots\":1,\"parallelUploadAboveSize\":2147483648}}\n","callerFile":"kasten.io/k10/kio/kopia/repository.go","callerFunction":"kasten.io/k10/kio/kopia.checkGlobalPolicySettings","callerLine":1127,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","hostname":"k10-restore-k10restore-nf5zw","level":"debug","msg":"show global policy command output","time":"2024-07-25T13:03:33.992Z"}
{"Container":"container","File":"pkg/format/format.go","Function":"github.com/kanisterio/kanister/pkg/format.LogWithCtx","Line":90,"LogKind":"datapath","Out":"{\"retention\":{\"keepLatest\":2147483647,\"keepHourly\":0,\"keepDaily\":0,\"keepWeekly\":0,\"keepMonthly\":0,\"keepAnnual\":0,\"ignoreIdenticalSnapshots\":false},\"files\":{\"ignoreDotFiles\":[\".kopiaignore\"]},\"errorHandling\":{\"ignoreFileErrors\":false,\"ignoreDirectoryErrors\":false,\"ignoreUnknownTypes\":true},\"scheduling\":{\"runMissed\":true},\"compression\":{\"compressorName\":\"s2-default\"},\"actions\":{},\"osSnapshots\":{\"volumeShadowCopy\":{\"enable\":0}},\"logging\":{\"directories\":{\"snapshotted\":5,\"ignored\":5},\"entries\":{\"snapshotted\":0,\"ignored\":5,\"cacheHit\":0,\"cacheMiss\":0}},\"upload\":{\"maxParallelSnapshots\":1,\"parallelUploadAboveSize\":2147483648}}","Pod":"data-mover-svc-8vnts","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"Pod Update","time":"2024-07-25T13:03:33.992469207Z"}
{"Command":"kopia --log-level=error --config-file=/tmp/kopia-repository.config --log-dir=/tmp/kopia-log server start --address=https://10.244.1.64:51515 --tls-cert-file=/etc/certs/cert/tls.crt --tls-key-file=/etc/certs/key/tls.key --server-username=k10-admin@data-mover-server-pod --server-password=\u003c****\u003e --server-control-username=k10-admin@data-mover-server-pod --server-control-password=\u003c****\u003e \u003e /dev/null 2\u003e\u00261 \u0026","File":"pkg/kopia/command/server.go","Function":"github.com/kanisterio/kanister/pkg/kopia/command.ServerStart","Line":76,"hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"Kopia Command","time":"2024-07-25T13:03:34.002803986Z"}
{"Command":"kopia --log-level=error --config-file=/tmp/kopia-repository.config --log-dir=/tmp/kopia-log server status --address=https://10.244.1.64:51515 --server-cert-fingerprint=\u003c****\u003e --server-username=k10-admin@data-mover-server-pod --server-password=\u003c****\u003e","File":"pkg/kopia/command/server.go","Function":"github.com/kanisterio/kanister/pkg/kopia/command.ServerStatus","Line":117,"hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"Kopia Command","time":"2024-07-25T13:03:34.137854276Z"}
{"Container":"container","File":"pkg/format/format.go","Function":"github.com/kanisterio/kanister/pkg/format.LogWithCtx","Line":90,"LogKind":"datapath","Out":"\u001b[31mERROR\u001b[0m unable to list sources: error running http request: Get \"https://10.244.1.64:51515/api/v1/control/sources\": dial tcp 10.244.1.64:51515: connect: connection refused","Pod":"data-mover-svc-8vnts","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"Pod Update","time":"2024-07-25T13:03:34.290642573Z"}
{"Container":"container","File":"pkg/format/format.go","Function":"github.com/kanisterio/kanister/pkg/format.LogWithCtx","Line":90,"LogKind":"datapath","Out":"\u001b[31mERROR\u001b[0m unable to list sources: error running http request: Get \"https://10.244.1.64:51515/api/v1/control/sources\": dial tcp 10.244.1.64:51515: connect: connection refused","Pod":"data-mover-svc-8vnts","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"Pod Update","time":"2024-07-25T13:03:34.569096553Z"}
{"Container":"container","File":"pkg/format/format.go","Function":"github.com/kanisterio/kanister/pkg/format.LogWithCtx","Line":90,"LogKind":"datapath","Out":"\u001b[31mERROR\u001b[0m unable to list sources: error running http request: Get \"https://10.244.1.64:51515/api/v1/control/sources\": dial tcp 10.244.1.64:51515: connect: connection refused","Pod":"data-mover-svc-8vnts","hostname":"k10-restore-k10restore-nf5zw","level":"info","msg":"Pod Update","time":"2024-07-25T13:03:34.931808774Z"}
{"File":"kasten.io/k10/kio/kopiaclient/repository.go","Function":"kasten.io/k10/kio/kopiaclient.ConnectToAPIServer.func1","Line":98,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","error":"unable to establish session for purpose=: error establishing session: unable to initialize session: rpc error: code = PermissionDenied desc = access denied for k10-admin@backup-k10-host: EOF","hostname":"k10-restore-k10restore-nf5zw","level":"debug","msg":"Connecting to the Kopia API Server","time":"2024-07-25T13:03:35.967Z"}
{"File":"kasten.io/k10/kio/dr/utils.go","Function":"kasten.io/k10/kio/dr.RestoreStatusForError","Line":162,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","error":{"message":"Failed connecting to the Kopia API Server","function":"kasten.io/k10/kio/kopiaclient.ConnectToAPIServer","linenumber":103,"file":"kasten.io/k10/kio/kopiaclient/repository.go:103","cause":{"message":"unable to establish session for purpose=: error establishing session: unable to initialize session: rpc error: code = PermissionDenied desc = access denied for k10-admin@backup-k10-host: EOF"}},"hostname":"k10-restore-k10restore-nf5zw","level":"error","msg":"Failed to connect to Kopia API Server","time":"2024-07-25T13:03:35.968Z"}
{"File":"kasten.io/k10/kio/exec/phases/phase/dr_restore.go","Function":"kasten.io/k10/kio/exec/phases/phase.RunK10DRRestore","Line":100,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","error":{"message":"Failed to connect to Kopia API Server","function":"kasten.io/k10/kio/dr.Import","linenumber":62,"file":"kasten.io/k10/kio/dr/import.go:62","cause":{"message":"Failed connecting to the Kopia API Server","function":"kasten.io/k10/kio/kopiaclient.ConnectToAPIServer","linenumber":103,"file":"kasten.io/k10/kio/kopiaclient/repository.go:103","cause":{"message":"unable to establish session for purpose=: error establishing session: unable to initialize session: rpc error: code = PermissionDenied desc = access denied for k10-admin@backup-k10-host: EOF"}}},"hostname":"k10-restore-k10restore-nf5zw","level":"error","msg":"Failed to perform Disaster Recovery using quick mode workflow","time":"2024-07-25T13:03:36.024Z"}
Error: {"message":"Failed to perform K10 Disaster Recovery","function":"kasten.io/k10/kio/tools/restorectl.runRestoreCommand.func1","linenumber":71,"file":"kasten.io/k10/kio/tools/restorectl/restore.go:71","fields":[{"name":"error","value":"Failed to connect to Kopia API Server"}]}
Usage:
restorectl restore [flags]
Flags:
-q, --enableQuickDisasterRecovery string This flag should be enabled if quick K10 Disaster Recovery
needs to be performed. The flag should be used if
kastenDisasterRecovery.quickMode was enabled for K10 helm chart
-h, --help help for restore
-i, --snapshotID restorectl validate Specify one of the snapshot IDs obtained from the restorectl validate command, latest is selected by default.
Global Flags:
-c, --clusterid string Specify the cluster ID captured during backup (required)
-n, --namespace string Specify the namespace where K10 is currently deployed (required)
-t, --point-in-time string Specify an optional point in time (RFC3339) at which to evaluate restore data
-p, --profile string Specify the profile that was used during backup (required)
-s, --skipResource string Specify if restore of policies,profiles,secrets needs to be skipped.
{"File":"kasten.io/k10/kio/tools/restorectl/root.go","Function":"kasten.io/k10/kio/tools/restorectl.Execute","Line":26,"cluster_name":"24b70a84-079a-40a1-b057-d19627ca1ad5","error":{"message":"Failed to perform K10 Disaster Recovery","function":"kasten.io/k10/kio/tools/restorectl.runRestoreCommand.func1","linenumber":71,"file":"kasten.io/k10/kio/tools/restorectl/restore.go:71","fields":[{"name":"error","value":"Failed to connect to Kopia API Server"}]},"hostname":"k10-restore-k10restore-nf5zw","level":"error","msg":"Failed","time":"2024-07-25T13:03:39.972Z"}
restore_test.go:206:
error:
got non-nil error
got:
e`{"message":"Job failed","function":"kasten.io/k10/test/dr3.waitForJobCompletion","linenumber":321,"file":"kasten.io/k10/test/dr3/restore_test.go:321"}`
stack:
/repo/go/src/kasten.io/k10/test/dr3/restore_test.go:206
c.Assert(err, qt.IsNil)
/repo/go/src/kasten.io/k10/test/dr3/restore_test.go:144
testRestoreK10(c, ctx, s.tp, helmSetFlags)
/usr/local/go/src/reflect/value.go:596
call(frametype, fn, stackArgs, uint32(frametype.Size()), uint32(abid.retOffset), uint32(frameSize), &regArgs)
/usr/local/go/src/reflect/value.go:380
return v.call("Call", in)
/cachevol/mod/github.com/frankban/[email protected]/qtsuite/suite.go:105
m.Func.Call(args)
--- FAIL: TestK10DR3RestoreSuite/TestRestoreK10 (80.80s)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment