Skip to content

Instantly share code, notes, and snippets.

@RobertKielty

RobertKielty/gist:07c90d71d7c8561303554534d3006fdd

Last active September 28, 2020 10:40
Show Gist options
  • Save RobertKielty/07c90d71d7c8561303554534d3006fdd to your computer and use it in GitHub Desktop.
Save RobertKielty/07c90d71d7c8561303554534d3006fdd to your computer and use it in GitHub Desktop.
Download ZIP
CI Signal Report for 28-Sep
Raw
gistfile1.md

Table of Contents

  1. Resolved
    1. SIG Release
      1. #87376 - Failing Generated Jobs (sig-release-generated)
      2. #88408 - [Flaky Test] build-packages-debs (ci-release-build-packages-debs)
    2. SIG Scalability
      1. #90981 - [Failing test] kubemark-500 failing due to no NetworkProgrammingLatency samples
    3. SIG Testing
      1. #91107 - kubeadm-kinder-master (ci-kubernetes-e2e-kubeadm-kinder-master)
      2. #89572 - skew-cluster-latest-kubectl-stable1-gce (ci-kubernetes-e2e-gce-master-new-gci-kubectl-skew )
      3. #89052 - [Flaking Test] verify-1.18 (ci-kubernetes-verify-beta)
      4. #89150 - [Flaky Test] integration-master (ci-kubernetes-integration-master)
    4. SIG cluster-lifecycle
      1. #92511 - capg-conformance-v1alpha3-k8s-master (ci-cluster-api-provider-gcp-make-conformance-v1alpha3-k8s-ci-artifacts)
      2. #89485 - capa-conformance-stable-k8s-master && capg-conformance-stable-k8s-master
    5. SIG Scheduling
      1. #91985 - [sig-scheduling] SchedulerPreemption [Serial] PreemptionExecutionPath runs ReplicaSets to verify preemption running path [Conformance]
      2. #90686 - [Flaky Test] capg-conformance-v1alpha3-k8s-master (ci-cluster-api-provider-gcp-make-conformance-v1alpha3-k8s-ci-artifacts)
      3. #92502 - [Failing test] Conformance - GCE- master (ci-kubernetes-gce-conformance-latest)
      4. #92612 - [sig-scheduling] SchedulerPredicates [Serial] validates resource limits of pods that are allowed to run [Conformance]
    6. SIG Architecture
      1. #92386 - [Flaky Test] conformance-ga-only (ci-kubernetes-conformance-kind-ga-only)
    7. SIG CLI
      1. #89659 - [Flaky Test] gce-master-scale-correctness (ci-kubernetes-e2e-gce-scale-correctness)
      2. #89177 - gce-master-scale-correctness (ci-kubernetes-e2e-gce-scale-correctness)
    8. SIG Network
      1. #89782 - gce-cos-master-alpha-features (ci-kubernetes-e2e-gci-gce-alpha-features)
    9. SIG Windows
      1. #89134 - aks-engine-azure-1-18-windows (ci-kubernetes-e2e-aks-engine-azure-1-18-windows )
    10. SIG Instrumentation
      1. #93688 - [Flaky Test] [sig-instrumentation] MetricsGrabber should grab all metrics from a ControllerManager (ci-kubernetes-e2e-ubuntu-gce-containerd)
    11. SIG Storage
      1. #91242 - Storage tests are failing due to quay.io being down
    12. SIG Node
      1. #91579 - [failing test] "Pod Container Status should never report success for a pending container" hangs and times out KIND jobs
  2. Observing (observe test flake before marking resolved)
    1. SIG Network
      1. #93536 - [Flaky Test] [sig-network] Loadbalancing: L7 GCE [Slow] [Feature:Ingress] should support multiple TLS certs
      2. #91774 - [Flaky test] [sig-network] Networking Granular Checks: Services should update nodePort: udp
      3. #93542 - gce-master-scale-correctness (ci-kubernetes-e2e-gce-scale-correctness)
      4. #90955 - [Flaky Test] gce-master-scale-correctness (ci-kubernetes-e2e-gce-scale-correctness)
      5. #90830 - [Flaky Test] gce-cos-master-slow (ci-kubernetes-e2e-gci-gce-slow)
      6. #92965 - kind-ipv6-master-parallel (ci-kubernetes-kind-ipv6-e2e-parallel)
    2. SIG Node
      1. #94370 - [Flaky Test] [k8s.io] ResourceMetricsAPI [NodeFeature:ResourceMetrics] when querying /resource/metrics should report resource usage through the resouce metrics api
      2. #94391 - kind-1.19-parallel (ci-kubernetes-kind-e2e-parallel-1-19)
      3. #94392 - kind-1.18-parallel (ci-kubernetes-kind-e2e-parallel-1-18)
      4. #94394 - kind-master-parallel (ci-kubernetes-kind-e2e-parallel)
      5. #89847 - [Failing Job] pull-kubernetes-node-e2e-containerd is timing out
      6. #91292 - [Flaky Test] node-kubelet-master (ci-kubernetes-node-kubelet)
    3. SIG Scalability
      1. [#93217 - [flaky] SchedulingThroughput - SchedulingThroughput error: scheduler throughput: actual throughput 81.200000 lower than threshold 90.000000]](#org85121c5)
      2. #87468 - pr:pull-kubernetes-e2e-gce-100-performance flaked 32 times in the past week
      3. #92780 - [Flaky Test] gce-cos-master-scalability-100 (ci-kubernetes-e2e-gci-gce-scalability)
    4. SIG Cloud-Provider
      1. #86181 - [Flaky Test] [sig-storage] PersistentVolumes GCEPD should test that deleting the Namespace of a PVC and Pod causes the successful detach of Persistent Disk
    5. SIG Storage
      1. #94221 - [Flaky Test] [sig-storage] In-tree Volumes [Driver: gluster] subPath should support file as subpath [LinuxOnly]
      2. #93780 - [Flaky Test] [sig-storage] TestVolumeBinding experiencing pod scheduling timeout (ci-kubernetes-integration-master)
      3. #93749 - [Flaky Test] [sig-storage] CSI mock volume CSI attach test using mock driver should not require VolumeAttach for drivers without attachment
      4. #93321 - gce-cos-master-alpha-features (ci-kubernetes-e2e-gci-gce-alpha-features )
    6. SIG CLI
      1. #92468 - [Flaky Test] [sig-cli] Kubectl client Kubectl logs should be able to retrieve and filter logs [Conformance]
    7. SIG Testing
      1. #94184 - [Flaky Test] integration-{master,1.19,1.18,1.17} (ci-kubernetes-integration-*)
      2. #94197 - [Flaky Test] verify-1.19 (ci-kubernetes-verify-beta)
      3. #93584 - gce-cos-master-scalability-100 (ci-kubernetes-e2e-gci-gce-scalability)
      4. #93334 - [Flaky Test] some tests affected by pod scheduling timeouts
    8. SIG Release
      1. #93561 - [SIG Release, SIG Testing] "Extract" phase is failing for multiple jobs using the ci/latest-fast version marker
    9. SIG cluster-lifecycle
      1. #90761 - [Flaky Test] kubeadm-kinder-upgrade-1-18-1-19 (ci-kubernetes-e2e-kubeadm-kinder-upgrade-1-18-1-19)
      2. #93309 - capg-conformance-v1alpha3-k8s-master (ci-cluster-api-provider-gcp-make-conformance-v1alpha3-k8s-ci-artifacts)
      3. #92360 - [Flaky Test] integration-master (ci-kubernetes-integration-master)
      4. #93223 - kubeadm-kinder-master (ci-kubernetes-e2e-kubeadm-kinder-master)
  3. Under Investigation (prioritized)
    1. SIG Scheduling
      1. #94230 - [Flaky Test] [sig-scheduling] k8s.io/kubernetes/test/integration/scheduler.TestPostFilterPlugin
      2. #93782 - [Flaky Test] [sig-scheduling] LimitRange should create a LimitRange with defaults and ensure pod has those defaults applied. [Conformance]
    2. SIG Scalability
      1. #93716 - [Flaky Test] [sig-scalability] there should be no high-latency requests (ci-kubernetes-e2e-gci-gce-scalability)
    3. SIG Cloud-Provider
      1. #91739 - Conformance - OpenStack (ci-cloud-provider-openstack-acceptance-test-e2e-conformance)
    4. SIG
      1. #1918 - kinder: investigate flakes related to /kinder/cluster-settings.yaml
    5. SIG Testing
      1. #87807 - [Flaky Test] verify-master (ci-kubernetes-verify-master)
    6. SIG Windows
      1. #88974 - gce-windows-1909-master (ci-kubernetes-e2e-windows-gce-1909)
    7. SIG Network
      1. #91236 - Flaky test: [sig-network] Services should be able to preserve UDP traffic when server pod cycles for a NodePort service
      2. #94998 - [Flaky Test][sig-network] EndpointSlice should create Endpoints and EndpointSlices for Pods matching a Service
      3. #93740 - [Flaky Test][sig-network] Loadbalancing: L7 GCE [Slow] [Feature:Ingress] should conform to Ingress spec
      4. #92306 - [Flaky Test] gce-ubuntu-master-containerd (ci-kubernetes-e2e-ubuntu-gce-containerd)
      5. #94256 - [Flaky Test] [sig-network] ESIPP [Slow] should work for type=LoadBalancer
    8. SIG Node
      1. #94753 - [Flaky Test] [sig-node] NoExecuteTaintManager Multiple Pods [Serial] evicts pods with minTolerationSeconds [Disruptive] [Conformance]
      2. #94223 - [Flaky Test] [k8s.io] MirrorPodWithGracePeriod when create a mirror pod mirror pod termination should satisfy grace period when static pod is deleted [NodeConformance]
      3. #75355 - [Flaky test] [k8s.io] Pods should support pod readiness gates [NodeFeature:PodReadinessGate]
    9. SIG Release
      1. #92756 - [Flaky Test] verify-1.18 (ci-kubernetes-verify-beta)
      2. #95026 - [Flaky Test] sig-release-master-blocking/ci-kubernetes-build
    10. SIG CLI
      1. #93651 - [Flaky Test] runkubectlapplytests (ci-kubernetes-integration-master)
  4. New (no response yet)
    1. SIG Testing
      1. #95064 - [Flaky Test] //staging/src/k8s.io/apiserver/pkg/storage/tests/godefaulttest:run1of2 - TestWatchBookmarksWithCorrectResourceVersion
    2. SIG
      1. #92155 - pr:pull-kubernetes-e2e-gce-ubuntu-containerd flaked 49 times in the past week
    3. SIG Storage
      1. #94224 - [Flaky Test][sig-storage] Subpath Container restart should verify that container can restart successfully after configmaps modified
    4. SIG Node
      1. #95017 - [Flaky Test] [sig-node] Summary API [NodeConformance] when querying /stats/summary should report resource usage through the stats api
      2. #94931 - [Flaky Test][k8s.io] [sig-node] NoExecuteTaintManager Multiple Pods [Serial] evicts pods with minTolerationSeconds [Disruptive] [Conformance]
      3. #95000 - [Flaky Test] MirrorPodWithGracePeriod when create a mirror pod mirror pod termination should satisfy grace period when static pod is deleted [NodeConformance] [ubuntu]
    5. SIG Network
      1. #94997 - [Flaky Test] [sig-network] Services should only allow access from service loadbalancer source ranges [Slow]
    6. SIG Cloud-Provider
      1. #95004 - [Flaky Test] [sig-cloud-provider-gcp] Reboot each node by dropping all inbound packets for a while and ensure they function afterwards
  5. Failures in Master-Blocking
  6. Failures in Master-Informing

(*github.RateLimits)(0xc00052a1b0)(github.RateLimits{Core:github.Rate{Limit:5000, Remaining:4994, Reset:github.Timestamp{2020-09-28 11:10:19 +0100 IST}}, Search:github.Rate{Limit:30, Remaining:30, Reset:github.Timestamp{2020-09-28 10:55:34 +0100 IST}}})

Resolved

SIG Release

#87376 - Failing Generated Jobs (sig-release-generated)

#88408 - [Flaky Test] build-packages-debs (ci-release-build-packages-debs)

SIG Scalability

#90981 - [Failing test] kubemark-500 failing due to no NetworkProgrammingLatency samples

SIG Testing

#91107 - kubeadm-kinder-master (ci-kubernetes-e2e-kubeadm-kinder-master)

#89572 - skew-cluster-latest-kubectl-stable1-gce (ci-kubernetes-e2e-gce-master-new-gci-kubectl-skew )

#89052 - [Flaking Test] verify-1.18 (ci-kubernetes-verify-beta)

#89150 - [Flaky Test] integration-master (ci-kubernetes-integration-master)

SIG cluster-lifecycle

#92511 - capg-conformance-v1alpha3-k8s-master (ci-cluster-api-provider-gcp-make-conformance-v1alpha3-k8s-ci-artifacts)

#89485 - capa-conformance-stable-k8s-master && capg-conformance-stable-k8s-master

SIG Scheduling

#91985 - [sig-scheduling] SchedulerPreemption [Serial] PreemptionExecutionPath runs ReplicaSets to verify preemption running path [Conformance]

#90686 - [Flaky Test] capg-conformance-v1alpha3-k8s-master (ci-cluster-api-provider-gcp-make-conformance-v1alpha3-k8s-ci-artifacts)

#92502 - [Failing test] Conformance - GCE- master (ci-kubernetes-gce-conformance-latest)

#92612 - [sig-scheduling] SchedulerPredicates [Serial] validates resource limits of pods that are allowed to run [Conformance]

SIG Architecture

#92386 - [Flaky Test] conformance-ga-only (ci-kubernetes-conformance-kind-ga-only)

SIG CLI

#89659 - [Flaky Test] gce-master-scale-correctness (ci-kubernetes-e2e-gce-scale-correctness)

#89177 - gce-master-scale-correctness (ci-kubernetes-e2e-gce-scale-correctness)

SIG Network

#89782 - gce-cos-master-alpha-features (ci-kubernetes-e2e-gci-gce-alpha-features)

SIG Windows

#89134 - aks-engine-azure-1-18-windows (ci-kubernetes-e2e-aks-engine-azure-1-18-windows )

SIG Instrumentation

#93688 - [Flaky Test] [sig-instrumentation] MetricsGrabber should grab all metrics from a ControllerManager (ci-kubernetes-e2e-ubuntu-gce-containerd)

SIG Storage

#91242 - Storage tests are failing due to quay.io being down

SIG Node

#91579 - [failing test] "Pod Container Status should never report success for a pending container" hangs and times out KIND jobs

Observing (observe test flake before marking resolved)

SIG Network

#93536 - [Flaky Test] [sig-network] Loadbalancing: L7 GCE [Slow] [Feature:Ingress] should support multiple TLS certs

#91774 - [Flaky test] [sig-network] Networking Granular Checks: Services should update nodePort: udp

#93542 - gce-master-scale-correctness (ci-kubernetes-e2e-gce-scale-correctness)

#90955 - [Flaky Test] gce-master-scale-correctness (ci-kubernetes-e2e-gce-scale-correctness)

#90830 - [Flaky Test] gce-cos-master-slow (ci-kubernetes-e2e-gci-gce-slow)

#92965 - kind-ipv6-master-parallel (ci-kubernetes-kind-ipv6-e2e-parallel)

SIG Node

#94370 - [Flaky Test] [k8s.io] ResourceMetricsAPI [NodeFeature:ResourceMetrics] when querying /resource/metrics should report resource usage through the resouce metrics api

#94391 - kind-1.19-parallel (ci-kubernetes-kind-e2e-parallel-1-19)

#94392 - kind-1.18-parallel (ci-kubernetes-kind-e2e-parallel-1-18)

#94394 - kind-master-parallel (ci-kubernetes-kind-e2e-parallel)

#89847 - [Failing Job] pull-kubernetes-node-e2e-containerd is timing out

#91292 - [Flaky Test] node-kubelet-master (ci-kubernetes-node-kubelet)

SIG Scalability

#93217 - [flaky] SchedulingThroughput - SchedulingThroughput error: scheduler throughput: actual throughput 81.200000 lower than threshold 90.000000]

#87468 - pr:pull-kubernetes-e2e-gce-100-performance flaked 32 times in the past week

#92780 - [Flaky Test] gce-cos-master-scalability-100 (ci-kubernetes-e2e-gci-gce-scalability)

SIG Cloud-Provider

#86181 - [Flaky Test] [sig-storage] PersistentVolumes GCEPD should test that deleting the Namespace of a PVC and Pod causes the successful detach of Persistent Disk

SIG Storage

#94221 - [Flaky Test] [sig-storage] In-tree Volumes [Driver: gluster] subPath should support file as subpath [LinuxOnly]

#93780 - [Flaky Test] [sig-storage] TestVolumeBinding experiencing pod scheduling timeout (ci-kubernetes-integration-master)

#93749 - [Flaky Test] [sig-storage] CSI mock volume CSI attach test using mock driver should not require VolumeAttach for drivers without attachment

#93321 - gce-cos-master-alpha-features (ci-kubernetes-e2e-gci-gce-alpha-features )

SIG CLI

#92468 - [Flaky Test] [sig-cli] Kubectl client Kubectl logs should be able to retrieve and filter logs [Conformance]

SIG Testing

#94184 - [Flaky Test] integration-{master,1.19,1.18,1.17} (ci-kubernetes-integration-*)

#94197 - [Flaky Test] verify-1.19 (ci-kubernetes-verify-beta)

#93584 - gce-cos-master-scalability-100 (ci-kubernetes-e2e-gci-gce-scalability)

#93334 - [Flaky Test] some tests affected by pod scheduling timeouts

SIG Release

#93561 - [SIG Release, SIG Testing] "Extract" phase is failing for multiple jobs using the ci/latest-fast version marker

SIG cluster-lifecycle

#90761 - [Flaky Test] kubeadm-kinder-upgrade-1-18-1-19 (ci-kubernetes-e2e-kubeadm-kinder-upgrade-1-18-1-19)

#93309 - capg-conformance-v1alpha3-k8s-master (ci-cluster-api-provider-gcp-make-conformance-v1alpha3-k8s-ci-artifacts)

#92360 - [Flaky Test] integration-master (ci-kubernetes-integration-master)

#93223 - kubeadm-kinder-master (ci-kubernetes-e2e-kubeadm-kinder-master)

Under Investigation (prioritized)

SIG Scheduling

#94230 - [Flaky Test] [sig-scheduling] k8s.io/kubernetes/test/integration/scheduler.TestPostFilterPlugin

#93782 - [Flaky Test] [sig-scheduling] LimitRange should create a LimitRange with defaults and ensure pod has those defaults applied. [Conformance]

SIG Scalability

#93716 - [Flaky Test] [sig-scalability] there should be no high-latency requests (ci-kubernetes-e2e-gci-gce-scalability)

SIG Cloud-Provider

#91739 - Conformance - OpenStack (ci-cloud-provider-openstack-acceptance-test-e2e-conformance)

SIG

#1918 - kinder: investigate flakes related to /kinder/cluster-settings.yaml

SIG Testing

#87807 - [Flaky Test] verify-master (ci-kubernetes-verify-master)

SIG Windows

#88974 - gce-windows-1909-master (ci-kubernetes-e2e-windows-gce-1909)

SIG Network

#91236 - Flaky test: [sig-network] Services should be able to preserve UDP traffic when server pod cycles for a NodePort service

#94998 - [Flaky Test][sig-network] EndpointSlice should create Endpoints and EndpointSlices for Pods matching a Service

#93740 - [Flaky Test][sig-network] Loadbalancing: L7 GCE [Slow] [Feature:Ingress] should conform to Ingress spec

#92306 - [Flaky Test] gce-ubuntu-master-containerd (ci-kubernetes-e2e-ubuntu-gce-containerd)

#94256 - [Flaky Test] [sig-network] ESIPP [Slow] should work for type=LoadBalancer

SIG Node

#94753 - [Flaky Test] [sig-node] NoExecuteTaintManager Multiple Pods [Serial] evicts pods with minTolerationSeconds [Disruptive] [Conformance]

#94223 - [Flaky Test] [k8s.io] MirrorPodWithGracePeriod when create a mirror pod mirror pod termination should satisfy grace period when static pod is deleted [NodeConformance]

#75355 - [Flaky test] [k8s.io] Pods should support pod readiness gates [NodeFeature:PodReadinessGate]

SIG Release

#92756 - [Flaky Test] verify-1.18 (ci-kubernetes-verify-beta)

#95026 - [Flaky Test] sig-release-master-blocking/ci-kubernetes-build

SIG CLI

#93651 - [Flaky Test] runkubectlapplytests (ci-kubernetes-integration-master)

New (no response yet)

SIG Testing

#95064 - [Flaky Test] //staging/src/k8s.io/apiserver/pkg/storage/tests/godefaulttest:run1of2 - TestWatchBookmarksWithCorrectResourceVersion

SIG

#92155 - pr:pull-kubernetes-e2e-gce-ubuntu-containerd flaked 49 times in the past week

SIG Storage

#94224 - [Flaky Test][sig-storage] Subpath Container restart should verify that container can restart successfully after configmaps modified

SIG Node

#95017 - [Flaky Test] [sig-node] Summary API [NodeConformance] when querying /stats/summary should report resource usage through the stats api

#94931 - [Flaky Test][k8s.io] [sig-node] NoExecuteTaintManager Multiple Pods [Serial] evicts pods with minTolerationSeconds [Disruptive] [Conformance]

#95000 - [Flaky Test] MirrorPodWithGracePeriod when create a mirror pod mirror pod termination should satisfy grace period when static pod is deleted [NodeConformance] [ubuntu]

SIG Network

#94997 - [Flaky Test] [sig-network] Services should only allow access from service loadbalancer source ranges [Slow]

SIG Cloud-Provider

#95004 - [Flaky Test] [sig-cloud-provider-gcp] Reboot each node by dropping all inbound packets for a while and ensure they function afterwards

Failures in Master-Blocking

  • 19 jobs total
  • 13 are passing
  • 6 are flaking
  • 0 are failing
  • 0 are stale

Failures in Master-Informing

  • 22 jobs total
  • 3 are passing
  • 8 are flaking
  • 6 are failing
  • 5 are stale
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment