Skip to content

Instantly share code, notes, and snippets.

@Harsh-2002

Harsh-2002/admin_extended.log

Created May 16, 2026 10:55
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save Harsh-2002/0d5602edb7f3a75655a65a3bb6d02c28 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save Harsh-2002/0d5602edb7f3a75655a65a3bb6d02c28 to your computer and use it in GitHub Desktop.
Download ZIP
Noema QA extreme/phase-11 20260516-075835Z — log artefacts
Raw
admin_extended.log
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-7.4.4, pluggy-1.4.0 -- /usr/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.12.3', 'Platform': 'Linux-6.8.0-52-generic-x86_64-with-glibc2.39', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.4.0'}, 'Plugins': {'timeout': '2.4.0', 'metadata': '3.1.1', 'xdist': '3.8.0', 'anyio': '4.13.0', 'rerunfailures': '16.1', 'json-report': '1.5.0', 'html': '4.2.0'}}
rootdir: /home/av/noema/qa
configfile: pytest.ini
plugins: timeout-2.4.0, metadata-3.1.1, xdist-3.8.0, anyio-4.13.0, rerunfailures-16.1, json-report-1.5.0, html-4.2.0
timeout: 120.0s
timeout method: thread
timeout func_only: False
created: 4/4 workers
4 workers [12 items]
scheduling tests via LoadScheduling
tests/admin_extended/test_inspect.py::test_inspect_returns_placement[tls-cluster]
tests/admin_extended/test_profile.py::test_memory_profile_starts[plain-cluster]
tests/admin_extended/test_inspect.py::test_inspect_returns_placement[plain-cluster]
tests/admin_extended/test_profile.py::test_memory_profile_starts[tls-cluster]
[gw3] [ 8%] PASSED tests/admin_extended/test_profile.py::test_memory_profile_starts[plain-cluster]
tests/admin_extended/test_profile.py::test_profile_download[plain-cluster]
[gw2] [ 16%] PASSED tests/admin_extended/test_inspect.py::test_inspect_returns_placement[plain-cluster]
tests/admin_extended/test_profile.py::test_cpu_profile_starts[plain-cluster]
[gw2] [ 25%] PASSED tests/admin_extended/test_profile.py::test_cpu_profile_starts[plain-cluster]
tests/admin_extended/test_profile.py::test_cpu_profile_starts[standalone]
[gw2] [ 33%] PASSED tests/admin_extended/test_profile.py::test_cpu_profile_starts[standalone]
tests/admin_extended/test_profile.py::test_memory_profile_starts[standalone]
[gw2] [ 41%] PASSED tests/admin_extended/test_profile.py::test_memory_profile_starts[standalone]
tests/admin_extended/test_profile.py::test_profile_download[standalone]
[gw3] [ 50%] PASSED tests/admin_extended/test_profile.py::test_profile_download[plain-cluster]
tests/admin_extended/test_inspect.py::test_inspect_returns_placement[standalone]
[gw3] [ 58%] SKIPPED tests/admin_extended/test_inspect.py::test_inspect_returns_placement[standalone]
[gw2] [ 66%] PASSED tests/admin_extended/test_profile.py::test_profile_download[standalone]
[gw1] [ 75%] RERUN tests/admin_extended/test_profile.py::test_memory_profile_starts[tls-cluster]
tests/admin_extended/test_profile.py::test_memory_profile_starts[tls-cluster]
[gw1] [ 75%] FAILED tests/admin_extended/test_profile.py::test_memory_profile_starts[tls-cluster]
tests/admin_extended/test_profile.py::test_profile_download[tls-cluster]
[gw1] [ 83%] RERUN tests/admin_extended/test_profile.py::test_profile_download[tls-cluster]
tests/admin_extended/test_profile.py::test_profile_download[tls-cluster]
[gw1] [ 83%] FAILED tests/admin_extended/test_profile.py::test_profile_download[tls-cluster]
[gw0] node down: Not properly terminated
[gw0] [ 91%] RERUN tests/admin_extended/test_inspect.py::test_inspect_returns_placement[tls-cluster]
replacing crashed worker gw0
tests/admin_extended/test_inspect.py::test_inspect_returns_placement[tls-cluster]
[gw4] node down: Not properly terminated
[gw4] [ 91%] FAILED tests/admin_extended/test_inspect.py::test_inspect_returns_placement[tls-cluster]
replacing crashed worker gw4
tests/admin_extended/test_profile.py::test_cpu_profile_starts[tls-cluster]
[gw5] [100%] RERUN tests/admin_extended/test_profile.py::test_cpu_profile_starts[tls-cluster]
tests/admin_extended/test_profile.py::test_cpu_profile_starts[tls-cluster]
[gw5] [100%] FAILED tests/admin_extended/test_profile.py::test_cpu_profile_starts[tls-cluster]
=================================== FAILURES ===================================
___________________ test_memory_profile_starts[tls-cluster] ____________________
[gw1] linux -- Python 3.12.3 /usr/bin/python3
../../.local/lib/python3.12/site-packages/httpx/_transports/default.py:101: in map_httpcore_exceptions
yield
../../.local/lib/python3.12/site-packages/httpx/_transports/default.py:250: in handle_request
resp = self._pool.handle_request(req)
../../.local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py:256: in handle_request
raise exc from None
../../.local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py:236: in handle_request
response = connection.handle_request(
../../.local/lib/python3.12/site-packages/httpcore/_sync/connection.py:101: in handle_request
raise exc
../../.local/lib/python3.12/site-packages/httpcore/_sync/connection.py:78: in handle_request
stream = self._connect(request)
../../.local/lib/python3.12/site-packages/httpcore/_sync/connection.py:124: in _connect
stream = self._network_backend.connect_tcp(**kwargs)
../../.local/lib/python3.12/site-packages/httpcore/_backends/sync.py:207: in connect_tcp
with map_exceptions(exc_map):
/usr/lib/python3.12/contextlib.py:158: in __exit__
self.gen.throw(value)
../../.local/lib/python3.12/site-packages/httpcore/_exceptions.py:14: in map_exceptions
raise to_exc(exc) from exc
E httpcore.ConnectTimeout: timed out
The above exception was the direct cause of the following exception:
tests/admin_extended/test_profile.py:25: in test_memory_profile_starts
r = admin.client.post("/api/v1/profile", json={"type": "memory", "duration_secs": 5})
fixtures/admin_clients.py:73: in post
return self._retry_on_leader("POST", path, json=json, **kw)
fixtures/admin_clients.py:52: in _retry_on_leader
r = self._client.request(method, url, **kwargs)
../../.local/lib/python3.12/site-packages/httpx/_client.py:825: in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
../../.local/lib/python3.12/site-packages/httpx/_client.py:914: in send
response = self._send_handling_auth(
../../.local/lib/python3.12/site-packages/httpx/_client.py:942: in _send_handling_auth
response = self._send_handling_redirects(
../../.local/lib/python3.12/site-packages/httpx/_client.py:979: in _send_handling_redirects
response = self._send_single_request(request)
../../.local/lib/python3.12/site-packages/httpx/_client.py:1014: in _send_single_request
response = transport.handle_request(request)
../../.local/lib/python3.12/site-packages/httpx/_transports/default.py:249: in handle_request
with map_httpcore_exceptions():
/usr/lib/python3.12/contextlib.py:158: in __exit__
self.gen.throw(value)
../../.local/lib/python3.12/site-packages/httpx/_transports/default.py:118: in map_httpcore_exceptions
raise mapped_exc(message) from exc
E httpx.ConnectTimeout: timed out
______________________ test_profile_download[tls-cluster] ______________________
[gw1] linux -- Python 3.12.3 /usr/bin/python3
../../.local/lib/python3.12/site-packages/httpx/_transports/default.py:101: in map_httpcore_exceptions
yield
../../.local/lib/python3.12/site-packages/httpx/_transports/default.py:250: in handle_request
resp = self._pool.handle_request(req)
../../.local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py:256: in handle_request
raise exc from None
../../.local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py:236: in handle_request
response = connection.handle_request(
../../.local/lib/python3.12/site-packages/httpcore/_sync/connection.py:101: in handle_request
raise exc
../../.local/lib/python3.12/site-packages/httpcore/_sync/connection.py:78: in handle_request
stream = self._connect(request)
../../.local/lib/python3.12/site-packages/httpcore/_sync/connection.py:124: in _connect
stream = self._network_backend.connect_tcp(**kwargs)
../../.local/lib/python3.12/site-packages/httpcore/_backends/sync.py:207: in connect_tcp
with map_exceptions(exc_map):
/usr/lib/python3.12/contextlib.py:158: in __exit__
self.gen.throw(value)
../../.local/lib/python3.12/site-packages/httpcore/_exceptions.py:14: in map_exceptions
raise to_exc(exc) from exc
E httpcore.ConnectTimeout: timed out
The above exception was the direct cause of the following exception:
tests/admin_extended/test_profile.py:35: in test_profile_download
r = admin.client.post("/api/v1/profile", json={"type": "cpu", "duration_secs": 3})
fixtures/admin_clients.py:73: in post
return self._retry_on_leader("POST", path, json=json, **kw)
fixtures/admin_clients.py:52: in _retry_on_leader
r = self._client.request(method, url, **kwargs)
../../.local/lib/python3.12/site-packages/httpx/_client.py:825: in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
../../.local/lib/python3.12/site-packages/httpx/_client.py:914: in send
response = self._send_handling_auth(
../../.local/lib/python3.12/site-packages/httpx/_client.py:942: in _send_handling_auth
response = self._send_handling_redirects(
../../.local/lib/python3.12/site-packages/httpx/_client.py:979: in _send_handling_redirects
response = self._send_single_request(request)
../../.local/lib/python3.12/site-packages/httpx/_client.py:1014: in _send_single_request
response = transport.handle_request(request)
../../.local/lib/python3.12/site-packages/httpx/_transports/default.py:249: in handle_request
with map_httpcore_exceptions():
/usr/lib/python3.12/contextlib.py:158: in __exit__
self.gen.throw(value)
../../.local/lib/python3.12/site-packages/httpx/_transports/default.py:118: in map_httpcore_exceptions
raise mapped_exc(message) from exc
E httpx.ConnectTimeout: timed out
_____________________ tests/admin_extended/test_inspect.py _____________________
[gw4] linux -- Python 3.12.3 /usr/bin/python3
worker 'gw4' crashed while running 'tests/admin_extended/test_inspect.py::test_inspect_returns_placement[tls-cluster]'
_____________________ test_cpu_profile_starts[tls-cluster] _____________________
[gw5] linux -- Python 3.12.3 /usr/bin/python3
../../.local/lib/python3.12/site-packages/httpx/_transports/default.py:101: in map_httpcore_exceptions
yield
../../.local/lib/python3.12/site-packages/httpx/_transports/default.py:250: in handle_request
resp = self._pool.handle_request(req)
../../.local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py:256: in handle_request
raise exc from None
../../.local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py:236: in handle_request
response = connection.handle_request(
../../.local/lib/python3.12/site-packages/httpcore/_sync/connection.py:101: in handle_request
raise exc
../../.local/lib/python3.12/site-packages/httpcore/_sync/connection.py:78: in handle_request
stream = self._connect(request)
../../.local/lib/python3.12/site-packages/httpcore/_sync/connection.py:124: in _connect
stream = self._network_backend.connect_tcp(**kwargs)
../../.local/lib/python3.12/site-packages/httpcore/_backends/sync.py:207: in connect_tcp
with map_exceptions(exc_map):
/usr/lib/python3.12/contextlib.py:158: in __exit__
self.gen.throw(value)
../../.local/lib/python3.12/site-packages/httpcore/_exceptions.py:14: in map_exceptions
raise to_exc(exc) from exc
E httpcore.ConnectTimeout: timed out
The above exception was the direct cause of the following exception:
tests/admin_extended/test_profile.py:16: in test_cpu_profile_starts
r = admin.client.post("/api/v1/profile", json={"type": "cpu", "duration_secs": 5})
fixtures/admin_clients.py:73: in post
return self._retry_on_leader("POST", path, json=json, **kw)
fixtures/admin_clients.py:52: in _retry_on_leader
r = self._client.request(method, url, **kwargs)
../../.local/lib/python3.12/site-packages/httpx/_client.py:825: in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
../../.local/lib/python3.12/site-packages/httpx/_client.py:914: in send
response = self._send_handling_auth(
../../.local/lib/python3.12/site-packages/httpx/_client.py:942: in _send_handling_auth
response = self._send_handling_redirects(
../../.local/lib/python3.12/site-packages/httpx/_client.py:979: in _send_handling_redirects
response = self._send_single_request(request)
../../.local/lib/python3.12/site-packages/httpx/_client.py:1014: in _send_single_request
response = transport.handle_request(request)
../../.local/lib/python3.12/site-packages/httpx/_transports/default.py:249: in handle_request
with map_httpcore_exceptions():
/usr/lib/python3.12/contextlib.py:158: in __exit__
self.gen.throw(value)
../../.local/lib/python3.12/site-packages/httpx/_transports/default.py:118: in map_httpcore_exceptions
raise mapped_exc(message) from exc
E httpx.ConnectTimeout: timed out
- generated xml file: /tmp/noema-qa-extreme-20260516-075835Z/pytest-admin_extended.junit.xml -
--------------------------------- JSON report ----------------------------------
report saved to: /tmp/noema-qa-extreme-20260516-075835Z/pytest-admin_extended.json
- Generated html report: file:///tmp/noema-qa-extreme-20260516-075835Z/pytest-admin_extended.html -
=========================== short test summary info ============================
SKIPPED [1] tests/admin_extended/test_inspect.py: cluster_only test skipped on standalone
FAILED tests/admin_extended/test_profile.py::test_memory_profile_starts[tls-cluster]
FAILED tests/admin_extended/test_profile.py::test_profile_download[tls-cluster]
FAILED tests/admin_extended/test_inspect.py::test_inspect_returns_placement[tls-cluster]
FAILED tests/admin_extended/test_profile.py::test_cpu_profile_starts[tls-cluster]
========= 4 failed, 7 passed, 1 skipped, 4 rerun in 278.19s (0:04:38) ==========
Raw
bringup.log
[bring-up] running preflight
✓ docker 29.3.1
✓ docker compose 5.1.1
✓ dbr network dbr ipam subnets: ['103.165.99.0/24', '10.0.0.0/20', '203.153.55.0/24']
✓ ip collisions no collisions on 10.0.12.65–.73
✓ gh auth github.com
✓ release v0.1.95 v0.1.95 reachable
✓ python packages all required python packages importable
PREFLIGHT OK
[bring-up] binary present: /home/av/noema/qa/binary/noema
[bring-up] issuing certs for tls-cluster
[bring-up] master keys ready
[bring-up] membership files ready
[bring-up] data/conf dirs ready
[bring-up] compose rendered → /home/av/noema/qa/docker/docker-compose.yaml
[bring-up] docker compose up -d --build
Image noema-qa:bookworm Building
Image noema-qa:bookworm Building
Image noema-qa:bookworm Building
Image noema-qa:bookworm Building
Image noema-qa:bookworm Building
Image noema-qa:bookworm Building
Image noema-qa:bookworm Building
Image noema-qa:bookworm Building
Image noema-qa:bookworm Building
#1 [internal] load local bake definitions
#1 reading from stdin 4.28kB done
#1 DONE 0.0s
#2 [noema-tls-3 internal] load build definition from Dockerfile.noema
#2 transferring dockerfile: 1.61kB done
#2 DONE 0.0s
#3 [noema-solo-1 internal] load metadata for docker.io/library/debian:bookworm-slim
#3 DONE 0.5s
#4 [noema-solo-1 internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.0s
#5 [noema-tls-4 1/5] FROM docker.io/library/debian:bookworm-slim@sha256:67b30a61dc87758f0caf819646104f29ecbda97d920aaf5edc834128ac8493d3
#5 DONE 0.0s
#6 [noema-tls-4 internal] load build context
#6 transferring context: 35B done
#6 DONE 0.0s
#7 [noema-tls-3 2/5] RUN set -eux; apt-get update; DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends tini ca-certificates iputils-ping iproute2 iptables net-tools curl jq gosu ; rm -rf /var/lib/apt/lists/*
#7 CACHED
#8 [noema-tls-3 3/5] RUN mkdir -p /data /conf /usr/local/bin
#8 CACHED
#9 [noema-tls-3 4/5] COPY entrypoint.sh /entrypoint.sh
#9 CACHED
#10 [noema-tls-4 5/5] RUN chmod +x /entrypoint.sh
#10 CACHED
#11 [noema-tls-2] exporting to image
#11 exporting layers done
#11 writing image sha256:074bd94f054f27e727372ad44dc94c68099e5cf2dc311fe1264ef317aed7dd5e done
#11 naming to docker.io/library/noema-qa:bookworm done
#11 DONE 0.0s
#12 [noema-tls-3] exporting to image
#12 exporting layers done
#12 writing image sha256:bab7e8be7ed4ae35e35bd84b5ff0de401c9616a88bbc5d14f3fcd4edc62d40f8 done
#12 naming to docker.io/library/noema-qa:bookworm done
#12 DONE 0.0s
#13 [noema-tls-4] exporting to image
#13 exporting layers done
#13 writing image sha256:3fceaf04160c5be331e4c2f8299d2319efbb028a46f5e3942466b3577f5c32c6 done
#13 naming to docker.io/library/noema-qa:bookworm done
#13 DONE 0.0s
#14 [noema-plain-4] exporting to image
#14 exporting layers done
#14 writing image sha256:ac572f66040f67853a540e1943639c3594153f1c97182e0b94fd40bc0f24efe2 done
#14 naming to docker.io/library/noema-qa:bookworm done
#14 DONE 0.0s
#15 [noema-plain-1] exporting to image
#15 exporting layers done
#15 writing image sha256:43632aeb06c11fe17c2df6fcaeddaf35094ced43dcba6ac867d021ea555a8d10 done
#15 naming to docker.io/library/noema-qa:bookworm done
#15 DONE 0.0s
#16 [noema-tls-1] exporting to image
#16 exporting layers done
#16 writing image sha256:e784eea215243cad264eca33900dabbe905641546fd8e9015b0c92328e08e9d3 done
#16 naming to docker.io/library/noema-qa:bookworm done
#16 DONE 0.0s
#17 [noema-plain-3] exporting to image
#17 exporting layers done
#17 writing image sha256:89558423813b2db0e8719576cdb77a9e9c423cde87a3abfefa0f4535c9f68a04 done
#17 naming to docker.io/library/noema-qa:bookworm done
#17 DONE 0.0s
#18 [noema-plain-2] exporting to image
#18 exporting layers done
#18 writing image sha256:c3ec231421e4818674dbc4bd9beae36c6e6af7c1683b564fd0aa525216639547 done
#18 naming to docker.io/library/noema-qa:bookworm done
#18 DONE 0.0s
#19 [noema-solo-1] exporting to image
#19 exporting layers done
#19 writing image sha256:cfe393469435879116cd71c1b790048f90a780c9ff3acb393774de78564b4c3a done
#19 naming to docker.io/library/noema-qa:bookworm done
#19 DONE 0.0s
#20 [noema-solo-1] resolving provenance for metadata file
#20 DONE 0.0s
#21 [noema-plain-1] resolving provenance for metadata file
#21 DONE 0.0s
#22 [noema-tls-2] resolving provenance for metadata file
#22 DONE 0.0s
#23 [noema-plain-4] resolving provenance for metadata file
#23 DONE 0.0s
#24 [noema-tls-4] resolving provenance for metadata file
#24 DONE 0.0s
#25 [noema-plain-2] resolving provenance for metadata file
#25 DONE 0.0s
#26 [noema-tls-1] resolving provenance for metadata file
#26 DONE 0.0s
#27 [noema-plain-3] resolving provenance for metadata file
#27 DONE 0.0s
#28 [noema-tls-3] resolving provenance for metadata file
#28 DONE 0.0s
Image noema-qa:bookworm Built
Image noema-qa:bookworm Built
Image noema-qa:bookworm Built
Image noema-qa:bookworm Built
Image noema-qa:bookworm Built
Image noema-qa:bookworm Built
Image noema-qa:bookworm Built
Image noema-qa:bookworm Built
Image noema-qa:bookworm Built
Container noema-plain-1 Recreate
Container noema-tls-3 Recreate
Container noema-tls-2 Recreate
Container noema-plain-3 Recreate
Container noema-tls-1 Recreate
Container noema-plain-2 Recreate
Container noema-plain-4 Recreate
Container noema-solo-1 Recreate
Container noema-tls-4 Recreate
Container noema-tls-4 Recreated
Container noema-tls-1 Recreated
Container noema-tls-3 Recreated
Container noema-solo-1 Recreated
Container noema-plain-4 Recreated
Container noema-plain-1 Recreated
Container noema-tls-2 Recreated
Container noema-plain-2 Recreated
Container noema-plain-3 Recreated
Container noema-plain-1 Starting
Container noema-tls-1 Starting
Container noema-solo-1 Starting
Container noema-tls-3 Starting
Container noema-tls-2 Starting
Container noema-tls-4 Starting
Container noema-plain-4 Starting
Container noema-plain-2 Starting
Container noema-plain-3 Starting
Container noema-tls-2 Started
Container noema-tls-3 Started
Container noema-tls-1 Started
Container noema-plain-2 Started
Container noema-plain-4 Started
Container noema-plain-1 Started
Container noema-solo-1 Started
Container noema-plain-3 Started
Container noema-tls-4 Started
[bring-up] ✓ noema-tls-1 http://10.0.12.65:9001
[bring-up] ✓ noema-tls-2 http://10.0.12.66:9001
[bring-up] ✗ noema-tls-3 http://10.0.12.67:9001
[bring-up] ✓ noema-tls-4 http://10.0.12.68:9001
[bring-up] ✓ noema-plain-1 http://10.0.12.69:9001
[bring-up] ✓ noema-plain-2 http://10.0.12.70:9001
[bring-up] ✗ noema-plain-3 http://10.0.12.71:9001
[bring-up] ✓ noema-plain-4 http://10.0.12.72:9001
[bring-up] ✓ noema-solo-1 http://10.0.12.73:9001
[bring-up] FAILED to reach health on: ['noema-tls-3', 'noema-plain-3']
Raw
combined.json
{
"run_id": "20260516T105557Z",
"results_dir": "/tmp/noema-qa-extreme-20260516-075835Z",
"noema_version": "v0.1.95",
"harness_version": "qa-1.0.0",
"overall_pass": true,
"pytest": {},
"warp": {
"summary": {
"total": 0,
"passed": 0,
"failed": 0,
"known_bugs": 0
},
"results": []
},
"chaos": {
"total": 0,
"recovered": 0,
"data_loss": 0
}
}
Raw
integrity-chaos.log
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-7.4.4, pluggy-1.4.0 -- /usr/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.12.3', 'Platform': 'Linux-6.8.0-52-generic-x86_64-with-glibc2.39', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.4.0'}, 'Plugins': {'timeout': '2.4.0', 'metadata': '3.1.1', 'xdist': '3.8.0', 'anyio': '4.13.0', 'rerunfailures': '16.1', 'json-report': '1.5.0', 'html': '4.2.0'}}
rootdir: /home/av/noema/qa
configfile: pytest.ini
plugins: timeout-2.4.0, metadata-3.1.1, xdist-3.8.0, anyio-4.13.0, rerunfailures-16.1, json-report-1.5.0, html-4.2.0
timeout: 120.0s
timeout method: thread
timeout func_only: False
collecting ... collected 63 items / 39 deselected / 24 selected
tests/integrity/test_bitflip.py::test_single_bitflip_in_data_shard_still_reads[tls-cluster] SKIPPED [ 4%]
tests/integrity/test_bitflip.py::test_bitflip_in_meta_file[tls-cluster] SKIPPED [ 8%]
tests/integrity/test_bitflip.py::test_multiple_bitflips_within_repair_threshold[tls-cluster] SKIPPED [ 12%]
tests/integrity/test_deep_corruption.py::test_truncate_shard_to_half[tls-cluster] SKIPPED [ 16%]
tests/integrity/test_deep_corruption.py::test_zero_fill_shard[tls-cluster] SKIPPED [ 20%]
tests/integrity/test_deep_corruption.py::test_multiple_simultaneous_corruptions[tls-cluster] PASSED [ 25%]
tests/integrity/test_deep_corruption.py::test_corrupt_wal_file[tls-cluster] RERUN [ 29%]
tests/integrity/test_deep_corruption.py::test_corrupt_wal_file[tls-cluster] FAILED [ 29%]
tests/integrity/test_master_key_rotation.py::test_master_key_rotation_breaks_reads[tls-cluster] SKIPPED [ 33%]
tests/integrity/test_bitflip.py::test_single_bitflip_in_data_shard_still_reads[plain-cluster] RERUN [ 37%]
tests/integrity/test_bitflip.py::test_single_bitflip_in_data_shard_still_reads[plain-cluster] FAILED [ 37%]
tests/integrity/test_bitflip.py::test_bitflip_in_meta_file[plain-cluster] SKIPPED [ 41%]
tests/integrity/test_bitflip.py::test_multiple_bitflips_within_repair_threshold[plain-cluster] SKIPPED [ 45%]
tests/integrity/test_deep_corruption.py::test_truncate_shard_to_half[plain-cluster] SKIPPED [ 50%]
tests/integrity/test_deep_corruption.py::test_zero_fill_shard[plain-cluster] SKIPPED [ 54%]
tests/integrity/test_deep_corruption.py::test_multiple_simultaneous_corruptions[plain-cluster] PASSED [ 58%]
tests/integrity/test_deep_corruption.py::test_corrupt_wal_file[plain-cluster] +++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
~~~~~~~~~~~~~~~ Stack of Thread-1 (run_server) (134090525464256) ~~~~~~~~~~~~~~~
File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run()
File "/usr/lib/python3.12/threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "/home/av/.local/lib/python3.12/site-packages/pytest_rerunfailures.py", line 466, in run_server
conn, _ = self.sock.accept()
File "/usr/lib/python3.12/socket.py", line 295, in accept
fd, addr = self._accept()
~~~~~~~~~~~~~~~~~~~~ Stack of MainThread (134090576081024) ~~~~~~~~~~~~~~~~~~~~~
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/usr/lib/python3/dist-packages/pytest/__main__.py", line 5, in <module>
raise SystemExit(pytest.console_main())
File "/usr/lib/python3/dist-packages/_pytest/config/__init__.py", line 192, in console_main
code = main()
File "/usr/lib/python3/dist-packages/_pytest/config/__init__.py", line 169, in main
ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 501, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 119, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 102, in _multicall
res = hook_impl.function(*args)
File "/usr/lib/python3/dist-packages/_pytest/main.py", line 318, in pytest_cmdline_main
return wrap_session(config, _main)
File "/usr/lib/python3/dist-packages/_pytest/main.py", line 271, in wrap_session
session.exitstatus = doit(config, session) or 0
File "/usr/lib/python3/dist-packages/_pytest/main.py", line 325, in _main
config.hook.pytest_runtestloop(session=session)
File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 501, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 119, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 102, in _multicall
res = hook_impl.function(*args)
File "/usr/lib/python3/dist-packages/_pytest/main.py", line 350, in pytest_runtestloop
item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 501, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 119, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 102, in _multicall
res = hook_impl.function(*args)
File "/home/av/.local/lib/python3.12/site-packages/pytest_rerunfailures.py", line 593, in pytest_runtest_protocol
reports = runtestprotocol(item, nextitem=nextitem, log=False)
File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 133, in runtestprotocol
reports.append(call_and_report(item, "call", log))
File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 222, in call_and_report
call = call_runtest_hook(item, when, **kwds)
File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 261, in call_runtest_hook
return CallInfo.from_call(
File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 341, in from_call
result: Optional[TResult] = func()
File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 262, in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 501, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 119, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 102, in _multicall
res = hook_impl.function(*args)
File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 169, in pytest_runtest_call
item.runtest()
File "/usr/lib/python3/dist-packages/_pytest/python.py", line 1792, in runtest
self.ihook.pytest_pyfunc_call(pyfuncitem=self)
File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 501, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 119, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 102, in _multicall
res = hook_impl.function(*args)
File "/usr/lib/python3/dist-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
result = testfunction(**testargs)
File "/home/av/noema/qa/tests/integrity/test_deep_corruption.py", line 98, in test_corrupt_wal_file
assert s3.client.get_object(Bucket=bucket, Key="pre-wal")["Body"].read() == b"x"
File "/usr/lib/python3/dist-packages/botocore/client.py", line 553, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/lib/python3/dist-packages/botocore/client.py", line 989, in _make_api_call
http, parsed_response = self._make_request(
File "/usr/lib/python3/dist-packages/botocore/client.py", line 1015, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/usr/lib/python3/dist-packages/botocore/endpoint.py", line 119, in make_request
return self._send_request(request_dict, operation_model)
File "/usr/lib/python3/dist-packages/botocore/endpoint.py", line 218, in _send_request
success_response, exception = self._get_response(
File "/usr/lib/python3/dist-packages/botocore/endpoint.py", line 241, in _get_response
success_response, exception = self._do_get_response(
File "/usr/lib/python3/dist-packages/botocore/endpoint.py", line 281, in _do_get_response
http_response = self._send(request)
File "/usr/lib/python3/dist-packages/botocore/endpoint.py", line 377, in _send
return self.http_session.send(request)
File "/usr/lib/python3/dist-packages/botocore/httpsession.py", line 464, in send
urllib_response = conn.urlopen(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 791, in urlopen
response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 537, in _make_request
response = conn.getresponse()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 461, in getresponse
httplib_response = super().getresponse()
File "/usr/lib/python3.12/http/client.py", line 1448, in getresponse
response.begin()
File "/usr/lib/python3.12/http/client.py", line 336, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.12/http/client.py", line 297, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.12/socket.py", line 707, in readinto
return self._sock.recv_into(b)
+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
Raw
integrity-fast.log
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-7.4.4, pluggy-1.4.0 -- /usr/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.12.3', 'Platform': 'Linux-6.8.0-52-generic-x86_64-with-glibc2.39', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.4.0'}, 'Plugins': {'timeout': '2.4.0', 'metadata': '3.1.1', 'xdist': '3.8.0', 'anyio': '4.13.0', 'rerunfailures': '16.1', 'json-report': '1.5.0', 'html': '4.2.0'}}
rootdir: /home/av/noema/qa
configfile: pytest.ini
plugins: timeout-2.4.0, metadata-3.1.1, xdist-3.8.0, anyio-4.13.0, rerunfailures-16.1, json-report-1.5.0, html-4.2.0
timeout: 120.0s
timeout method: thread
timeout func_only: False
created: 4/4 workers
4 workers [39 items]
scheduling tests via LoadScheduling
tests/integrity/test_compression_edges.py::test_roundtrip_size[tls-cluster-4096]
tests/integrity/test_compression_edges.py::test_roundtrip_size[tls-cluster-65535]
tests/integrity/test_compression_edges.py::test_roundtrip_size[tls-cluster-1]
tests/integrity/test_compression_edges.py::test_roundtrip_size[tls-cluster-16]
[gw2] [ 2%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[tls-cluster-4096]
tests/integrity/test_compression_edges.py::test_roundtrip_size[tls-cluster-4097]
[gw1] [ 5%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[tls-cluster-16]
tests/integrity/test_compression_edges.py::test_roundtrip_size[tls-cluster-4095]
[gw0] [ 7%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[tls-cluster-1]
tests/integrity/test_compression_edges.py::test_roundtrip_size[tls-cluster-2]
[gw3] [ 10%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[tls-cluster-65535]
tests/integrity/test_compression_edges.py::test_roundtrip_size[tls-cluster-65536]
[gw2] [ 12%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[tls-cluster-4097]
tests/integrity/test_compression_edges.py::test_roundtrip_size[tls-cluster-65537]
[gw0] [ 15%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[tls-cluster-2]
tests/integrity/test_compression_edges.py::test_etag_is_plaintext_md5[tls-cluster]
[gw1] [ 17%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[tls-cluster-4095]
tests/integrity/test_compression_edges.py::test_roundtrip_size[tls-cluster-1048576]
[gw3] [ 20%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[tls-cluster-65536]
tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-2]
[gw3] [ 23%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-2]
tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-16]
[gw0] [ 25%] PASSED tests/integrity/test_compression_edges.py::test_etag_is_plaintext_md5[tls-cluster]
tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-1]
[gw2] [ 28%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[tls-cluster-65537]
tests/integrity/test_compression_edges.py::test_roundtrip_size[tls-cluster-1048575]
[gw1] [ 30%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[tls-cluster-1048576]
tests/integrity/test_compression_edges.py::test_roundtrip_size[tls-cluster-1048577]
[gw2] [ 33%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[tls-cluster-1048575]
tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-4097]
[gw3] [ 35%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-16]
tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-4095]
[gw1] [ 38%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[tls-cluster-1048577]
tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-65535]
[gw0] [ 41%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-1]
tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-4096]
[gw3] [ 43%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-4095]
tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-65537]
[gw0] [ 46%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-4096]
tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-1048576]
[gw2] [ 48%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-4097]
tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-65536]
[gw1] [ 51%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-65535]
tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-1048575]
[gw3] [ 53%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-65537]
tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-1048577]
[gw2] [ 56%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-65536]
tests/integrity/test_compression_edges.py::test_roundtrip_size[standalone-1]
[gw2] [ 58%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[standalone-1]
tests/integrity/test_compression_edges.py::test_roundtrip_size[standalone-4095]
[gw2] [ 61%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[standalone-4095]
tests/integrity/test_compression_edges.py::test_roundtrip_size[standalone-4096]
[gw2] [ 64%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[standalone-4096]
tests/integrity/test_compression_edges.py::test_roundtrip_size[standalone-4097]
[gw2] [ 66%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[standalone-4097]
tests/integrity/test_compression_edges.py::test_roundtrip_size[standalone-65535]
[gw2] [ 69%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[standalone-65535]
tests/integrity/test_compression_edges.py::test_roundtrip_size[standalone-65536]
[gw2] [ 71%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[standalone-65536]
tests/integrity/test_compression_edges.py::test_roundtrip_size[standalone-65537]
[gw2] [ 74%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[standalone-65537]
tests/integrity/test_compression_edges.py::test_roundtrip_size[standalone-1048575]
[gw2] [ 76%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[standalone-1048575]
tests/integrity/test_compression_edges.py::test_roundtrip_size[standalone-1048576]
[gw2] [ 79%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[standalone-1048576]
tests/integrity/test_compression_edges.py::test_roundtrip_size[standalone-1048577]
[gw2] [ 82%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[standalone-1048577]
tests/integrity/test_compression_edges.py::test_etag_is_plaintext_md5[standalone]
[gw2] [ 84%] PASSED tests/integrity/test_compression_edges.py::test_etag_is_plaintext_md5[standalone]
[gw0] [ 87%] RERUN tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-1048576]
tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-1048576]
[gw1] [ 89%] RERUN tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-1048575]
tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-1048575]
[gw3] [ 92%] RERUN tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-1048577]
tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-1048577]
[gw0] [ 92%] FAILED tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-1048576]
tests/integrity/test_compression_edges.py::test_etag_is_plaintext_md5[plain-cluster]
[gw1] [ 92%] FAILED tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-1048575]
tests/integrity/test_compression_edges.py::test_roundtrip_size[standalone-2]
[gw0] [ 94%] PASSED tests/integrity/test_compression_edges.py::test_etag_is_plaintext_md5[plain-cluster]
[gw1] [ 97%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[standalone-2]
[gw3] [ 97%] FAILED tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-1048577]
tests/integrity/test_compression_edges.py::test_roundtrip_size[standalone-16]
[gw3] [100%] PASSED tests/integrity/test_compression_edges.py::test_roundtrip_size[standalone-16]
=================================== FAILURES ===================================
__________________ test_roundtrip_size[plain-cluster-1048576] __________________
[gw0] linux -- Python 3.12.3 /usr/bin/python3
tests/integrity/test_compression_edges.py:27: in test_roundtrip_size
s3.client.put_object(Bucket=bucket, Key=f"sz-{size}", Body=p.body)
/usr/lib/python3/dist-packages/botocore/client.py:553: in _api_call
return self._make_api_call(operation_name, kwargs)
/usr/lib/python3/dist-packages/botocore/client.py:1009: in _make_api_call
raise error_class(parsed_response, operation_name)
E botocore.exceptions.ClientError: An error occurred (SlowDown) when calling the PutObject operation (reached max retries: 1): Please reduce your request rate.
__________________ test_roundtrip_size[plain-cluster-1048575] __________________
[gw1] linux -- Python 3.12.3 /usr/bin/python3
tests/integrity/test_compression_edges.py:27: in test_roundtrip_size
s3.client.put_object(Bucket=bucket, Key=f"sz-{size}", Body=p.body)
/usr/lib/python3/dist-packages/botocore/client.py:553: in _api_call
return self._make_api_call(operation_name, kwargs)
/usr/lib/python3/dist-packages/botocore/client.py:1009: in _make_api_call
raise error_class(parsed_response, operation_name)
E botocore.exceptions.ClientError: An error occurred (SlowDown) when calling the PutObject operation (reached max retries: 1): Please reduce your request rate.
__________________ test_roundtrip_size[plain-cluster-1048577] __________________
[gw3] linux -- Python 3.12.3 /usr/bin/python3
tests/integrity/test_compression_edges.py:27: in test_roundtrip_size
s3.client.put_object(Bucket=bucket, Key=f"sz-{size}", Body=p.body)
/usr/lib/python3/dist-packages/botocore/client.py:553: in _api_call
return self._make_api_call(operation_name, kwargs)
/usr/lib/python3/dist-packages/botocore/client.py:1009: in _make_api_call
raise error_class(parsed_response, operation_name)
E botocore.exceptions.ClientError: An error occurred (SlowDown) when calling the PutObject operation (reached max retries: 1): Please reduce your request rate.
- generated xml file: /tmp/noema-qa-extreme-20260516-075835Z/pytest-integrity-fast.junit.xml -
--------------------------------- JSON report ----------------------------------
report saved to: /tmp/noema-qa-extreme-20260516-075835Z/pytest-integrity-fast.json
- Generated html report: file:///tmp/noema-qa-extreme-20260516-075835Z/pytest-integrity-fast.html -
=========================== short test summary info ============================
FAILED tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-1048576]
FAILED tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-1048575]
FAILED tests/integrity/test_compression_edges.py::test_roundtrip_size[plain-cluster-1048577]
=============== 3 failed, 36 passed, 3 rerun in 97.81s (0:01:37) ===============
Raw
operational-chaos.log
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-7.4.4, pluggy-1.4.0 -- /usr/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.12.3', 'Platform': 'Linux-6.8.0-52-generic-x86_64-with-glibc2.39', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.4.0'}, 'Plugins': {'timeout': '2.4.0', 'metadata': '3.1.1', 'xdist': '3.8.0', 'anyio': '4.13.0', 'rerunfailures': '16.1', 'json-report': '1.5.0', 'html': '4.2.0'}}
rootdir: /home/av/noema/qa
configfile: pytest.ini
plugins: timeout-2.4.0, metadata-3.1.1, xdist-3.8.0, anyio-4.13.0, rerunfailures-16.1, json-report-1.5.0, html-4.2.0
timeout: 120.0s
timeout method: thread
timeout func_only: False
collecting ... collected 78 items / 24 deselected / 54 selected
tests/operational/test_capacity_management.py::test_filesystem_readonly_simulation[tls-cluster] PASSED [ 1%]
tests/operational/test_capacity_management.py::test_drive_full_during_multipart[tls-cluster] PASSED [ 3%]
tests/operational/test_catastrophic_recovery.py::test_kill_all_then_restore[tls-cluster] RERUN [ 5%]
tests/operational/test_catastrophic_recovery.py::test_kill_all_then_restore[tls-cluster] ERROR [ 5%]
tests/operational/test_catastrophic_recovery.py::test_writes_during_quorum_loss_recover[tls-cluster] RERUN [ 7%]
tests/operational/test_catastrophic_recovery.py::test_writes_during_quorum_loss_recover[tls-cluster] ERROR [ 7%]
tests/operational/test_cluster_capacity_exhaustion.py::test_one_drive_full_per_node_writes_continue[tls-cluster] RERUN [ 9%]
tests/operational/test_cluster_capacity_exhaustion.py::test_one_drive_full_per_node_writes_continue[tls-cluster] ERROR [ 9%]
tests/operational/test_cluster_capacity_exhaustion.py::test_capacity_alert_under_pressure[tls-cluster] RERUN [ 11%]
tests/operational/test_cluster_capacity_exhaustion.py::test_capacity_alert_under_pressure[tls-cluster] PASSED [ 11%]
tests/operational/test_dr_consistency.py::test_snapshot_then_writes_then_restore_loses_post_snapshot_data[tls-cluster] +++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
~~~~~~~~~~~~~~~ Stack of Thread-1 (run_server) (128963215316672) ~~~~~~~~~~~~~~~
File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run()
File "/usr/lib/python3.12/threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "/home/av/.local/lib/python3.12/site-packages/pytest_rerunfailures.py", line 466, in run_server
conn, _ = self.sock.accept()
File "/usr/lib/python3.12/socket.py", line 295, in accept
fd, addr = self._accept()
~~~~~~~~~~~~~~~~~~~~ Stack of MainThread (128963266465920) ~~~~~~~~~~~~~~~~~~~~~
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/usr/lib/python3/dist-packages/pytest/__main__.py", line 5, in <module>
raise SystemExit(pytest.console_main())
File "/usr/lib/python3/dist-packages/_pytest/config/__init__.py", line 192, in console_main
code = main()
File "/usr/lib/python3/dist-packages/_pytest/config/__init__.py", line 169, in main
ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 501, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 119, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 102, in _multicall
res = hook_impl.function(*args)
File "/usr/lib/python3/dist-packages/_pytest/main.py", line 318, in pytest_cmdline_main
return wrap_session(config, _main)
File "/usr/lib/python3/dist-packages/_pytest/main.py", line 271, in wrap_session
session.exitstatus = doit(config, session) or 0
File "/usr/lib/python3/dist-packages/_pytest/main.py", line 325, in _main
config.hook.pytest_runtestloop(session=session)
File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 501, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 119, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 102, in _multicall
res = hook_impl.function(*args)
File "/usr/lib/python3/dist-packages/_pytest/main.py", line 350, in pytest_runtestloop
item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 501, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 119, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 102, in _multicall
res = hook_impl.function(*args)
File "/home/av/.local/lib/python3.12/site-packages/pytest_rerunfailures.py", line 593, in pytest_runtest_protocol
reports = runtestprotocol(item, nextitem=nextitem, log=False)
File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 133, in runtestprotocol
reports.append(call_and_report(item, "call", log))
File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 222, in call_and_report
call = call_runtest_hook(item, when, **kwds)
File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 261, in call_runtest_hook
return CallInfo.from_call(
File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 341, in from_call
result: Optional[TResult] = func()
File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 262, in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 501, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 119, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 102, in _multicall
res = hook_impl.function(*args)
File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 169, in pytest_runtest_call
item.runtest()
File "/usr/lib/python3/dist-packages/_pytest/python.py", line 1792, in runtest
self.ihook.pytest_pyfunc_call(pyfuncitem=self)
File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 501, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 119, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 102, in _multicall
res = hook_impl.function(*args)
File "/usr/lib/python3/dist-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
result = testfunction(**testargs)
File "/home/av/noema/qa/tests/operational/test_dr_consistency.py", line 75, in test_snapshot_then_writes_then_restore_loses_post_snapshot_data
_full_dr_cycle(s3.deployment, mutate_after_snapshot=write_more)
File "/home/av/noema/qa/tests/operational/test_dr_consistency.py", line 52, in _full_dr_cycle
archive = _snapshot(deployment)
File "/home/av/noema/qa/tests/operational/test_dr_consistency.py", line 26, in _snapshot
shutil.copytree(n.data_dir, archive / n.hostname / "data")
File "/usr/lib/python3.12/shutil.py", line 600, in copytree
return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
File "/usr/lib/python3.12/shutil.py", line 536, in _copytree
copytree(srcobj, dstname, symlinks, ignore, copy_function,
File "/usr/lib/python3.12/shutil.py", line 600, in copytree
return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
File "/usr/lib/python3.12/shutil.py", line 536, in _copytree
copytree(srcobj, dstname, symlinks, ignore, copy_function,
File "/usr/lib/python3.12/shutil.py", line 600, in copytree
return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
File "/usr/lib/python3.12/shutil.py", line 536, in _copytree
copytree(srcobj, dstname, symlinks, ignore, copy_function,
File "/usr/lib/python3.12/shutil.py", line 600, in copytree
return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
File "/usr/lib/python3.12/shutil.py", line 536, in _copytree
copytree(srcobj, dstname, symlinks, ignore, copy_function,
File "/usr/lib/python3.12/shutil.py", line 600, in copytree
return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
File "/usr/lib/python3.12/shutil.py", line 536, in _copytree
copytree(srcobj, dstname, symlinks, ignore, copy_function,
File "/usr/lib/python3.12/shutil.py", line 600, in copytree
return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
File "/usr/lib/python3.12/shutil.py", line 540, in _copytree
copy_function(srcobj, dstname)
File "/usr/lib/python3.12/shutil.py", line 475, in copy2
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/usr/lib/python3.12/shutil.py", line 273, in copyfile
_fastcopy_sendfile(fsrc, fdst)
File "/usr/lib/python3.12/shutil.py", line 150, in _fastcopy_sendfile
sent = os.sendfile(outfd, infd, offset, blocksize)
+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
Raw
operational-fast.log
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-7.4.4, pluggy-1.4.0 -- /usr/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.12.3', 'Platform': 'Linux-6.8.0-52-generic-x86_64-with-glibc2.39', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.4.0'}, 'Plugins': {'timeout': '2.4.0', 'metadata': '3.1.1', 'xdist': '3.8.0', 'anyio': '4.13.0', 'rerunfailures': '16.1', 'json-report': '1.5.0', 'html': '4.2.0'}}
rootdir: /home/av/noema/qa
configfile: pytest.ini
plugins: timeout-2.4.0, metadata-3.1.1, xdist-3.8.0, anyio-4.13.0, rerunfailures-16.1, json-report-1.5.0, html-4.2.0
timeout: 120.0s
timeout method: thread
timeout func_only: False
created: 4/4 workers
4 workers [24 items]
scheduling tests via LoadScheduling
tests/operational/test_objectlock_compliance.py::test_compliance_delete_rejected[tls-cluster]
tests/operational/test_config_reload.py::test_compression_level_roundtrip[tls-cluster]
tests/operational/test_config_reload.py::test_unknown_subsystem[tls-cluster]
tests/operational/test_config_reload.py::test_healing_max_concurrent[tls-cluster]
[gw2] [ 4%] PASSED tests/operational/test_config_reload.py::test_unknown_subsystem[tls-cluster]
tests/operational/test_config_reload.py::test_help_endpoint[tls-cluster]
[gw2] [ 8%] PASSED tests/operational/test_config_reload.py::test_help_endpoint[tls-cluster]
[gw0] [ 12%] PASSED tests/operational/test_config_reload.py::test_compression_level_roundtrip[tls-cluster]
tests/operational/test_config_reload.py::test_compression_level_roundtrip[plain-cluster]
tests/operational/test_config_reload.py::test_compression_enabled_toggle[tls-cluster]
[gw1] [ 16%] PASSED tests/operational/test_config_reload.py::test_healing_max_concurrent[tls-cluster]
tests/operational/test_config_reload.py::test_invalid_value_rejected[tls-cluster]
[gw2] [ 20%] PASSED tests/operational/test_config_reload.py::test_compression_level_roundtrip[plain-cluster]
tests/operational/test_config_reload.py::test_compression_enabled_toggle[plain-cluster]
[gw1] [ 25%] PASSED tests/operational/test_config_reload.py::test_invalid_value_rejected[tls-cluster]
[gw0] [ 29%] PASSED tests/operational/test_config_reload.py::test_compression_enabled_toggle[tls-cluster]
tests/operational/test_config_reload.py::test_invalid_value_rejected[plain-cluster]
tests/operational/test_config_reload.py::test_healing_max_concurrent[plain-cluster]
[gw2] [ 33%] PASSED tests/operational/test_config_reload.py::test_compression_enabled_toggle[plain-cluster]
tests/operational/test_config_reload.py::test_unknown_subsystem[plain-cluster]
[gw1] [ 37%] PASSED tests/operational/test_config_reload.py::test_invalid_value_rejected[plain-cluster]
tests/operational/test_config_reload.py::test_help_endpoint[plain-cluster]
[gw2] [ 41%] PASSED tests/operational/test_config_reload.py::test_unknown_subsystem[plain-cluster]
[gw1] [ 45%] PASSED tests/operational/test_config_reload.py::test_help_endpoint[plain-cluster]
tests/operational/test_objectlock_compliance.py::test_governance_bypass_works_compliance_does_not[plain-cluster]
tests/operational/test_config_reload.py::test_compression_level_roundtrip[standalone]
[gw0] [ 50%] PASSED tests/operational/test_config_reload.py::test_healing_max_concurrent[plain-cluster]
tests/operational/test_objectlock_compliance.py::test_compliance_delete_rejected[plain-cluster]
[gw1] [ 54%] PASSED tests/operational/test_config_reload.py::test_compression_level_roundtrip[standalone]
tests/operational/test_config_reload.py::test_healing_max_concurrent[standalone]
[gw1] [ 58%] PASSED tests/operational/test_config_reload.py::test_healing_max_concurrent[standalone]
tests/operational/test_config_reload.py::test_unknown_subsystem[standalone]
[gw1] [ 62%] PASSED tests/operational/test_config_reload.py::test_unknown_subsystem[standalone]
tests/operational/test_config_reload.py::test_help_endpoint[standalone]
[gw1] [ 66%] PASSED tests/operational/test_config_reload.py::test_help_endpoint[standalone]
tests/operational/test_objectlock_compliance.py::test_compliance_delete_rejected[standalone]
[gw3] [ 70%] PASSED tests/operational/test_objectlock_compliance.py::test_compliance_delete_rejected[tls-cluster]
tests/operational/test_objectlock_compliance.py::test_governance_bypass_works_compliance_does_not[tls-cluster]
[gw1] [ 75%] PASSED tests/operational/test_objectlock_compliance.py::test_compliance_delete_rejected[standalone]
tests/operational/test_objectlock_compliance.py::test_governance_bypass_works_compliance_does_not[standalone]
[gw3] [ 79%] PASSED tests/operational/test_objectlock_compliance.py::test_governance_bypass_works_compliance_does_not[tls-cluster]
[gw1] [ 83%] PASSED tests/operational/test_objectlock_compliance.py::test_governance_bypass_works_compliance_does_not[standalone]
[gw2] [ 87%] PASSED tests/operational/test_objectlock_compliance.py::test_governance_bypass_works_compliance_does_not[plain-cluster]
tests/operational/test_config_reload.py::test_compression_enabled_toggle[standalone]
[gw2] [ 91%] PASSED tests/operational/test_config_reload.py::test_compression_enabled_toggle[standalone]
[gw0] [ 95%] PASSED tests/operational/test_objectlock_compliance.py::test_compliance_delete_rejected[plain-cluster]
tests/operational/test_config_reload.py::test_invalid_value_rejected[standalone]
[gw0] [100%] PASSED tests/operational/test_config_reload.py::test_invalid_value_rejected[standalone]
- generated xml file: /tmp/noema-qa-extreme-20260516-075835Z/pytest-operational-fast.junit.xml -
--------------------------------- JSON report ----------------------------------
report saved to: /tmp/noema-qa-extreme-20260516-075835Z/pytest-operational-fast.json
- Generated html report: file:///tmp/noema-qa-extreme-20260516-075835Z/pytest-operational-fast.html -
============================== 24 passed in 9.84s ==============================
Raw
protocol.log
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-7.4.4, pluggy-1.4.0 -- /usr/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.12.3', 'Platform': 'Linux-6.8.0-52-generic-x86_64-with-glibc2.39', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.4.0'}, 'Plugins': {'timeout': '2.4.0', 'metadata': '3.1.1', 'xdist': '3.8.0', 'anyio': '4.13.0', 'rerunfailures': '16.1', 'json-report': '1.5.0', 'html': '4.2.0'}}
rootdir: /home/av/noema/qa
configfile: pytest.ini
plugins: timeout-2.4.0, metadata-3.1.1, xdist-3.8.0, anyio-4.13.0, rerunfailures-16.1, json-report-1.5.0, html-4.2.0
timeout: 120.0s
timeout method: thread
timeout func_only: False
created: 4/4 workers
4 workers [24 items]
scheduling tests via LoadScheduling
tests/protocol/test_http_edges.py::test_request_with_huge_header[tls-cluster]
tests/protocol/test_http_edges.py::test_put_with_content_length_too_large[tls-cluster]
tests/protocol/test_request_storms.py::test_concurrent_head_requests[tls-cluster]
tests/protocol/test_http_edges.py::test_empty_key_rejected[tls-cluster]
[gw0] node down: Not properly terminated
[gw0] [ 4%] RERUN tests/protocol/test_http_edges.py::test_put_with_content_length_too_large[tls-cluster]
replacing crashed worker gw0
[gw2] node down: Not properly terminated
[gw2] [ 8%] RERUN tests/protocol/test_http_edges.py::test_empty_key_rejected[tls-cluster]
replacing crashed worker gw2
[gw1] node down: Not properly terminated
[gw1] [ 12%] RERUN tests/protocol/test_http_edges.py::test_request_with_huge_header[tls-cluster]
replacing crashed worker gw1
tests/protocol/test_http_edges.py::test_request_with_huge_header[tls-cluster]
tests/protocol/test_http_edges.py::test_put_with_content_length_too_large[standalone]
tests/protocol/test_http_edges.py::test_request_with_huge_header[standalone]
[gw3] [ 16%] RERUN tests/protocol/test_request_storms.py::test_concurrent_head_requests[tls-cluster]
tests/protocol/test_request_storms.py::test_concurrent_head_requests[tls-cluster]
[gw6] [ 20%] PASSED tests/protocol/test_http_edges.py::test_request_with_huge_header[standalone]
tests/protocol/test_http_edges.py::test_trailing_slash_on_bucket[standalone]
[gw6] [ 25%] PASSED tests/protocol/test_http_edges.py::test_trailing_slash_on_bucket[standalone]
tests/protocol/test_http_edges.py::test_empty_key_rejected[standalone]
[gw6] [ 29%] PASSED tests/protocol/test_http_edges.py::test_empty_key_rejected[standalone]
tests/protocol/test_request_storms.py::test_admin_health_under_load[standalone]
[gw5] [ 33%] RERUN tests/protocol/test_http_edges.py::test_put_with_content_length_too_large[standalone]
tests/protocol/test_http_edges.py::test_put_with_content_length_too_large[standalone]
[gw5] [ 33%] FAILED tests/protocol/test_http_edges.py::test_put_with_content_length_too_large[standalone]
tests/protocol/test_http_edges.py::test_put_no_content_length[standalone]
[gw5] [ 37%] PASSED tests/protocol/test_http_edges.py::test_put_no_content_length[standalone]
tests/protocol/test_request_storms.py::test_many_small_puts_burst[standalone]
[gw6] [ 41%] PASSED tests/protocol/test_request_storms.py::test_admin_health_under_load[standalone]
tests/protocol/test_request_storms.py::test_concurrent_head_requests[standalone]
[gw5] [ 45%] PASSED tests/protocol/test_request_storms.py::test_many_small_puts_burst[standalone]
tests/protocol/test_http_edges.py::test_put_no_content_length[tls-cluster]
[gw6] [ 50%] PASSED tests/protocol/test_request_storms.py::test_concurrent_head_requests[standalone]
tests/protocol/test_request_storms.py::test_admin_health_under_load[tls-cluster]
[gw4] node down: Not properly terminated
[gw4] [ 50%] FAILED tests/protocol/test_http_edges.py::test_request_with_huge_header[tls-cluster]
replacing crashed worker gw4
tests/protocol/test_request_storms.py::test_concurrent_head_requests[plain-cluster]
[gw3] [ 50%] ERROR tests/protocol/test_request_storms.py::test_concurrent_head_requests[tls-cluster]
tests/protocol/test_request_storms.py::test_many_small_puts_burst[tls-cluster]
[gw7] [ 54%] PASSED tests/protocol/test_request_storms.py::test_concurrent_head_requests[plain-cluster]
tests/protocol/test_request_storms.py::test_many_small_puts_burst[plain-cluster]
[gw6] [ 58%] RERUN tests/protocol/test_request_storms.py::test_admin_health_under_load[tls-cluster]
tests/protocol/test_request_storms.py::test_admin_health_under_load[tls-cluster]
[gw6] [ 58%] FAILED tests/protocol/test_request_storms.py::test_admin_health_under_load[tls-cluster]
[gw5] node down: Not properly terminated
[gw5] [ 62%] RERUN tests/protocol/test_http_edges.py::test_put_no_content_length[tls-cluster]
replacing crashed worker gw5
tests/protocol/test_http_edges.py::test_put_no_content_length[tls-cluster]
[gw3] [ 66%] RERUN tests/protocol/test_request_storms.py::test_many_small_puts_burst[tls-cluster]
tests/protocol/test_request_storms.py::test_many_small_puts_burst[tls-cluster]
[gw7] [ 70%] PASSED tests/protocol/test_request_storms.py::test_many_small_puts_burst[plain-cluster]
[gw8] node down: Not properly terminated
[gw8] [ 70%] FAILED tests/protocol/test_http_edges.py::test_put_no_content_length[tls-cluster]
replacing crashed worker gw8
[gw3] [ 70%] ERROR tests/protocol/test_request_storms.py::test_many_small_puts_burst[tls-cluster]
tests/protocol/test_http_edges.py::test_empty_key_rejected[tls-cluster]
[gw3] node down: Not properly terminated
[gw3] [ 70%] FAILED tests/protocol/test_http_edges.py::test_empty_key_rejected[tls-cluster]
replacing crashed worker gw3
tests/protocol/test_http_edges.py::test_trailing_slash_on_bucket[tls-cluster]
tests/protocol/test_http_edges.py::test_put_no_content_length[plain-cluster]
[gw10] [ 75%] PASSED tests/protocol/test_http_edges.py::test_put_no_content_length[plain-cluster]
tests/protocol/test_http_edges.py::test_request_with_huge_header[plain-cluster]
[gw10] [ 79%] PASSED tests/protocol/test_http_edges.py::test_request_with_huge_header[plain-cluster]
tests/protocol/test_http_edges.py::test_trailing_slash_on_bucket[plain-cluster]
[gw10] [ 83%] PASSED tests/protocol/test_http_edges.py::test_trailing_slash_on_bucket[plain-cluster]
tests/protocol/test_http_edges.py::test_empty_key_rejected[plain-cluster]
[gw10] [ 87%] PASSED tests/protocol/test_http_edges.py::test_empty_key_rejected[plain-cluster]
tests/protocol/test_request_storms.py::test_admin_health_under_load[plain-cluster]
[gw10] [ 91%] PASSED tests/protocol/test_request_storms.py::test_admin_health_under_load[plain-cluster]
[gw9] node down: Not properly terminated
[gw9] [ 95%] RERUN tests/protocol/test_http_edges.py::test_trailing_slash_on_bucket[tls-cluster]
replacing crashed worker gw9
tests/protocol/test_http_edges.py::test_trailing_slash_on_bucket[tls-cluster]
[gw11] node down: Not properly terminated
[gw11] [ 95%] FAILED tests/protocol/test_http_edges.py::test_trailing_slash_on_bucket[tls-cluster]
replacing crashed worker gw11
tests/protocol/test_http_edges.py::test_put_with_content_length_too_large[plain-cluster]
[gw12] [100%] RERUN tests/protocol/test_http_edges.py::test_put_with_content_length_too_large[plain-cluster]
tests/protocol/test_http_edges.py::test_put_with_content_length_too_large[plain-cluster]
[gw12] [100%] FAILED tests/protocol/test_http_edges.py::test_put_with_content_length_too_large[plain-cluster]
tests/protocol/test_http_edges.py::test_put_with_content_length_too_large[tls-cluster]
[gw12] node down: Not properly terminated
[gw12] [100%] FAILED tests/protocol/test_http_edges.py::test_put_with_content_length_too_large[tls-cluster]
replacing crashed worker gw12
==================================== ERRORS ====================================
_________ ERROR at setup of test_concurrent_head_requests[tls-cluster] _________
[gw3] linux -- Python 3.12.3 /usr/bin/python3
/usr/lib/python3/dist-packages/urllib3/connection.py:203: in _new_conn
sock = connection.create_connection(
/usr/lib/python3/dist-packages/urllib3/util/connection.py:85: in create_connection
raise err
/usr/lib/python3/dist-packages/urllib3/util/connection.py:73: in create_connection
sock.connect(sa)
E TimeoutError: timed out
The above exception was the direct cause of the following exception:
/usr/lib/python3/dist-packages/botocore/httpsession.py:464: in send
urllib_response = conn.urlopen(
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:845: in urlopen
retries = retries.increment(
/usr/lib/python3/dist-packages/urllib3/util/retry.py:447: in increment
raise reraise(type(error), error, _stacktrace)
/usr/lib/python3/dist-packages/urllib3/util/util.py:39: in reraise
raise value
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:791: in urlopen
response = self._make_request(
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:492: in _make_request
raise new_e
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:468: in _make_request
self._validate_conn(conn)
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:1097: in _validate_conn
conn.connect()
/usr/lib/python3/dist-packages/urllib3/connection.py:611: in connect
self.sock = sock = self._new_conn()
/usr/lib/python3/dist-packages/urllib3/connection.py:212: in _new_conn
raise ConnectTimeoutError(
E urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPSConnection object at 0x7e1cc9131a60>, 'Connection to 10.0.12.65 timed out. (connect timeout=60)')
During handling of the above exception, another exception occurred:
fixtures/buckets.py:73: in bucket
s3.client.create_bucket(Bucket=name)
/usr/lib/python3/dist-packages/botocore/client.py:553: in _api_call
return self._make_api_call(operation_name, kwargs)
/usr/lib/python3/dist-packages/botocore/client.py:989: in _make_api_call
http, parsed_response = self._make_request(
/usr/lib/python3/dist-packages/botocore/client.py:1015: in _make_request
return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3/dist-packages/botocore/endpoint.py:119: in make_request
return self._send_request(request_dict, operation_model)
/usr/lib/python3/dist-packages/botocore/endpoint.py:231: in _send_request
raise exception
/usr/lib/python3/dist-packages/botocore/endpoint.py:281: in _do_get_response
http_response = self._send(request)
/usr/lib/python3/dist-packages/botocore/endpoint.py:377: in _send
return self.http_session.send(request)
/usr/lib/python3/dist-packages/botocore/httpsession.py:499: in send
raise ConnectTimeoutError(endpoint_url=request.url, error=e)
E botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://10.0.12.65:9000/qa-test-concurrent-head-requests-46dc3678"
__________ ERROR at setup of test_many_small_puts_burst[tls-cluster] ___________
[gw3] linux -- Python 3.12.3 /usr/bin/python3
/usr/lib/python3/dist-packages/urllib3/connection.py:203: in _new_conn
sock = connection.create_connection(
/usr/lib/python3/dist-packages/urllib3/util/connection.py:85: in create_connection
raise err
/usr/lib/python3/dist-packages/urllib3/util/connection.py:73: in create_connection
sock.connect(sa)
E TimeoutError: timed out
The above exception was the direct cause of the following exception:
/usr/lib/python3/dist-packages/botocore/httpsession.py:464: in send
urllib_response = conn.urlopen(
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:845: in urlopen
retries = retries.increment(
/usr/lib/python3/dist-packages/urllib3/util/retry.py:447: in increment
raise reraise(type(error), error, _stacktrace)
/usr/lib/python3/dist-packages/urllib3/util/util.py:39: in reraise
raise value
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:791: in urlopen
response = self._make_request(
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:492: in _make_request
raise new_e
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:468: in _make_request
self._validate_conn(conn)
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:1097: in _validate_conn
conn.connect()
/usr/lib/python3/dist-packages/urllib3/connection.py:611: in connect
self.sock = sock = self._new_conn()
/usr/lib/python3/dist-packages/urllib3/connection.py:212: in _new_conn
raise ConnectTimeoutError(
E urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPSConnection object at 0x7e1cc91f7290>, 'Connection to 10.0.12.65 timed out. (connect timeout=60)')
During handling of the above exception, another exception occurred:
fixtures/buckets.py:73: in bucket
s3.client.create_bucket(Bucket=name)
/usr/lib/python3/dist-packages/botocore/client.py:553: in _api_call
return self._make_api_call(operation_name, kwargs)
/usr/lib/python3/dist-packages/botocore/client.py:989: in _make_api_call
http, parsed_response = self._make_request(
/usr/lib/python3/dist-packages/botocore/client.py:1015: in _make_request
return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3/dist-packages/botocore/endpoint.py:119: in make_request
return self._send_request(request_dict, operation_model)
/usr/lib/python3/dist-packages/botocore/endpoint.py:231: in _send_request
raise exception
/usr/lib/python3/dist-packages/botocore/endpoint.py:281: in _do_get_response
http_response = self._send(request)
/usr/lib/python3/dist-packages/botocore/endpoint.py:377: in _send
return self.http_session.send(request)
/usr/lib/python3/dist-packages/botocore/httpsession.py:499: in send
raise ConnectTimeoutError(endpoint_url=request.url, error=e)
E botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://10.0.12.65:9000/qa-test-many-small-puts-bursttls-b9aed4a1"
=================================== FAILURES ===================================
______________ test_put_with_content_length_too_large[standalone] ______________
[gw5] linux -- Python 3.12.3 /usr/bin/python3
tests/protocol/test_http_edges.py:34: in test_put_with_content_length_too_large
assert r.status_code != 200, "accepted body shorter than Content-Length"
E AssertionError: accepted body shorter than Content-Length
E assert 200 != 200
E + where 200 = <Response [200]>.status_code
______________________ tests/protocol/test_http_edges.py _______________________
[gw4] linux -- Python 3.12.3 /usr/bin/python3
worker 'gw4' crashed while running 'tests/protocol/test_http_edges.py::test_request_with_huge_header[tls-cluster]'
__________________ test_admin_health_under_load[tls-cluster] ___________________
[gw6] linux -- Python 3.12.3 /usr/bin/python3
tests/protocol/test_request_storms.py:30: in test_admin_health_under_load
assert success >= 190, f"only {success}/200 health requests returned 200"
E AssertionError: only 0/200 health requests returned 200
E assert 0 >= 190
______________________ tests/protocol/test_http_edges.py _______________________
[gw8] linux -- Python 3.12.3 /usr/bin/python3
worker 'gw8' crashed while running 'tests/protocol/test_http_edges.py::test_put_no_content_length[tls-cluster]'
______________________ tests/protocol/test_http_edges.py _______________________
[gw3] linux -- Python 3.12.3 /usr/bin/python3
worker 'gw3' crashed while running 'tests/protocol/test_http_edges.py::test_empty_key_rejected[tls-cluster]'
______________________ tests/protocol/test_http_edges.py _______________________
[gw11] linux -- Python 3.12.3 /usr/bin/python3
worker 'gw11' crashed while running 'tests/protocol/test_http_edges.py::test_trailing_slash_on_bucket[tls-cluster]'
____________ test_put_with_content_length_too_large[plain-cluster] _____________
[gw12] linux -- Python 3.12.3 /usr/bin/python3
tests/protocol/test_http_edges.py:34: in test_put_with_content_length_too_large
assert r.status_code != 200, "accepted body shorter than Content-Length"
E AssertionError: accepted body shorter than Content-Length
E assert 200 != 200
E + where 200 = <Response [200]>.status_code
______________________ tests/protocol/test_http_edges.py _______________________
[gw12] linux -- Python 3.12.3 /usr/bin/python3
worker 'gw12' crashed while running 'tests/protocol/test_http_edges.py::test_put_with_content_length_too_large[tls-cluster]'
- generated xml file: /tmp/noema-qa-extreme-20260516-075835Z/pytest-protocol.junit.xml -
--------------------------------- JSON report ----------------------------------
report saved to: /tmp/noema-qa-extreme-20260516-075835Z/pytest-protocol.json
- Generated html report: file:///tmp/noema-qa-extreme-20260516-075835Z/pytest-protocol.html -
=========================== short test summary info ============================
ERROR tests/protocol/test_request_storms.py::test_concurrent_head_requests[tls-cluster]
ERROR tests/protocol/test_request_storms.py::test_many_small_puts_burst[tls-cluster]
FAILED tests/protocol/test_http_edges.py::test_put_with_content_length_too_large[standalone]
FAILED tests/protocol/test_http_edges.py::test_request_with_huge_header[tls-cluster]
FAILED tests/protocol/test_request_storms.py::test_admin_health_under_load[tls-cluster]
FAILED tests/protocol/test_http_edges.py::test_put_no_content_length[tls-cluster]
FAILED tests/protocol/test_http_edges.py::test_empty_key_rejected[tls-cluster]
FAILED tests/protocol/test_http_edges.py::test_trailing_slash_on_bucket[tls-cluster]
FAILED tests/protocol/test_http_edges.py::test_put_with_content_length_too_large[plain-cluster]
FAILED tests/protocol/test_http_edges.py::test_put_with_content_length_too_large[tls-cluster]
========= 8 failed, 14 passed, 2 errors, 10 rerun in 982.24s (0:16:22) =========
Raw
recover.log
[bring-up] ✓ noema-plain-2 http://10.0.12.70:9001
[bring-up] ✗ noema-plain-3 http://10.0.12.71:9001
[bring-up] ✓ noema-plain-4 http://10.0.12.72:9001
[bring-up] ✓ noema-solo-1 http://10.0.12.73:9001
[bring-up] FAILED to reach health on: ['noema-tls-2', 'noema-tls-3', 'noema-plain-3']
Raw
replication.log
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-7.4.4, pluggy-1.4.0 -- /usr/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.12.3', 'Platform': 'Linux-6.8.0-52-generic-x86_64-with-glibc2.39', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.4.0'}, 'Plugins': {'timeout': '2.4.0', 'metadata': '3.1.1', 'xdist': '3.8.0', 'anyio': '4.13.0', 'rerunfailures': '16.1', 'json-report': '1.5.0', 'html': '4.2.0'}}
rootdir: /home/av/noema/qa
configfile: pytest.ini
plugins: timeout-2.4.0, metadata-3.1.1, xdist-3.8.0, anyio-4.13.0, rerunfailures-16.1, json-report-1.5.0, html-4.2.0
timeout: 120.0s
timeout method: thread
timeout func_only: False
collecting ... collected 9 items
tests/replication/test_actual_delivery.py::test_replication_actually_delivers_bytes[tls-cluster] RERUN [ 11%]
tests/replication/test_actual_delivery.py::test_replication_actually_delivers_bytes[tls-cluster] ERROR [ 11%]
tests/replication/test_cross_cluster.py::test_add_remove_replication_site[tls-cluster] SKIPPED [ 22%]
tests/replication/test_cross_cluster.py::test_replication_status_endpoint[tls-cluster] SKIPPED [ 33%]
tests/replication/test_actual_delivery.py::test_replication_actually_delivers_bytes[plain-cluster] SKIPPED [ 44%]
tests/replication/test_cross_cluster.py::test_add_remove_replication_site[plain-cluster] PASSED [ 55%]
tests/replication/test_cross_cluster.py::test_replication_status_endpoint[plain-cluster] PASSED [ 66%]
tests/replication/test_actual_delivery.py::test_replication_actually_delivers_bytes[standalone] SKIPPED [ 77%]
tests/replication/test_cross_cluster.py::test_add_remove_replication_site[standalone] SKIPPED [ 88%]
tests/replication/test_cross_cluster.py::test_replication_status_endpoint[standalone] SKIPPED [100%]
==================================== ERRORS ====================================
___ ERROR at setup of test_replication_actually_delivers_bytes[tls-cluster] ____
/usr/lib/python3/dist-packages/urllib3/connection.py:203: in _new_conn
sock = connection.create_connection(
/usr/lib/python3/dist-packages/urllib3/util/connection.py:85: in create_connection
raise err
/usr/lib/python3/dist-packages/urllib3/util/connection.py:73: in create_connection
sock.connect(sa)
E TimeoutError: timed out
The above exception was the direct cause of the following exception:
/usr/lib/python3/dist-packages/botocore/httpsession.py:464: in send
urllib_response = conn.urlopen(
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:845: in urlopen
retries = retries.increment(
/usr/lib/python3/dist-packages/urllib3/util/retry.py:447: in increment
raise reraise(type(error), error, _stacktrace)
/usr/lib/python3/dist-packages/urllib3/util/util.py:39: in reraise
raise value
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:791: in urlopen
response = self._make_request(
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:492: in _make_request
raise new_e
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:468: in _make_request
self._validate_conn(conn)
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:1097: in _validate_conn
conn.connect()
/usr/lib/python3/dist-packages/urllib3/connection.py:611: in connect
self.sock = sock = self._new_conn()
/usr/lib/python3/dist-packages/urllib3/connection.py:212: in _new_conn
raise ConnectTimeoutError(
E urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPSConnection object at 0x7ed598155310>, 'Connection to 10.0.12.65 timed out. (connect timeout=60)')
During handling of the above exception, another exception occurred:
fixtures/buckets.py:73: in bucket
s3.client.create_bucket(Bucket=name)
/usr/lib/python3/dist-packages/botocore/client.py:553: in _api_call
return self._make_api_call(operation_name, kwargs)
/usr/lib/python3/dist-packages/botocore/client.py:989: in _make_api_call
http, parsed_response = self._make_request(
/usr/lib/python3/dist-packages/botocore/client.py:1015: in _make_request
return self._endpoint.make_request(operation_model, request_dict)
/usr/lib/python3/dist-packages/botocore/endpoint.py:119: in make_request
return self._send_request(request_dict, operation_model)
/usr/lib/python3/dist-packages/botocore/endpoint.py:231: in _send_request
raise exception
/usr/lib/python3/dist-packages/botocore/endpoint.py:281: in _do_get_response
http_response = self._send(request)
/usr/lib/python3/dist-packages/botocore/endpoint.py:377: in _send
return self.http_session.send(request)
/usr/lib/python3/dist-packages/botocore/httpsession.py:499: in send
raise ConnectTimeoutError(endpoint_url=request.url, error=e)
E botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://10.0.12.65:9000/qa-test-replication-actually-deli-d84032aa"
- generated xml file: /tmp/noema-qa-extreme-20260516-075835Z/pytest-replication.junit.xml -
--------------------------------- JSON report ----------------------------------
report saved to: /tmp/noema-qa-extreme-20260516-075835Z/pytest-replication.json
- Generated html report: file:///tmp/noema-qa-extreme-20260516-075835Z/pytest-replication.html -
=========================== short test summary info ============================
SKIPPED [1] tests/replication/test_cross_cluster.py:45: test runs only on plain-cluster
SKIPPED [1] tests/replication/test_cross_cluster.py:57: Skipped
SKIPPED [1] tests/replication/test_actual_delivery.py:89: replication delivery not observed in 30 s — may need bucket replication config (PutBucketReplication) on top of site config
SKIPPED [1] tests/replication/test_actual_delivery.py: cluster_only test skipped on standalone
SKIPPED [2] tests/replication/test_cross_cluster.py: cluster_only test skipped on standalone
ERROR tests/replication/test_actual_delivery.py::test_replication_actually_delivers_bytes[tls-cluster]
========== 2 passed, 6 skipped, 1 error, 1 rerun in 279.93s (0:04:39) ==========
Raw
run.log
2026-05-16T07:58:35Z | ===== EXTREME PHASE: 0 — pre-flight bring-up =====
2026-05-16T08:07:42Z | ===== EXTREME PHASE: A — security & IAM enforcement =====
2026-05-16T08:07:42Z | pytest -m 'security' -n 4 tests/security
2026-05-16T08:08:09Z | ===== EXTREME PHASE: B — data integrity (incl. bit-flip) =====
2026-05-16T08:08:09Z | pytest -m 'integrity and not chaos' -n 4 tests/integrity
2026-05-16T08:09:49Z | pytest -m 'integrity and chaos' -n 0 tests/integrity
2026-05-16T08:22:49Z | ===== EXTREME PHASE: C — scale (special keys, many objects) =====
2026-05-16T08:22:49Z | pytest -m 'scale and not slow' -n 4 tests/scale
2026-05-16T08:26:36Z | pytest -m 'scale and slow' -n 0 tests/scale
2026-05-16T09:03:41Z | ===== EXTREME PHASE: D — operational (config reload, restart, COMPLIANCE) =====
2026-05-16T09:03:41Z | pytest -m 'operational and not chaos' -n 4 tests/operational
2026-05-16T09:03:52Z | pytest -m 'operational and chaos' -n 0 tests/operational
2026-05-16T09:20:34Z | ===== EXTREME PHASE: E — cross-cluster replication =====
2026-05-16T09:20:34Z | pytest -m 'replication and not chaos' -n 0 tests/replication
2026-05-16T09:25:16Z | ===== EXTREME PHASE: F — S3 protocol edge cases =====
2026-05-16T09:25:16Z | pytest -m 'protocol or admin_extended' -n 4 tests/protocol tests/admin_extended
2026-05-16T09:41:40Z | ===== EXTREME PHASE: G — admin extended =====
2026-05-16T09:41:40Z | pytest -m 'admin' -n 4 tests/admin_extended
2026-05-16T09:46:20Z | ===== EXTREME PHASE: H — time / clock invariants =====
2026-05-16T09:46:20Z | pytest -m 'time' -n 4 tests/time
2026-05-16T09:46:24Z | ===== EXTREME PHASE: I — 1-hour memory/FD soak (plain cluster only) =====
2026-05-16T10:46:30Z | ===== EXTREME PHASE: post-extreme bring-up (recover from any leftover chaos) =====
2026-05-16T10:55:57Z | ===== EXTREME PHASE: aggregate extreme results =====
run_id: 20260516T105557Z
overall_pass: True
results_dir: /tmp/noema-qa-extreme-20260516-075835Z
warp: total=0 passed=0 failed=0 known_bugs=0
chaos: total=0 recovered=0 data_loss=0
2026-05-16T10:55:58Z | ===== EXTREME PHASE: bundle logs into gist =====
Raw
scale-fast.log
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-7.4.4, pluggy-1.4.0 -- /usr/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.12.3', 'Platform': 'Linux-6.8.0-52-generic-x86_64-with-glibc2.39', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.4.0'}, 'Plugins': {'timeout': '2.4.0', 'metadata': '3.1.1', 'xdist': '3.8.0', 'anyio': '4.13.0', 'rerunfailures': '16.1', 'json-report': '1.5.0', 'html': '4.2.0'}}
rootdir: /home/av/noema/qa
configfile: pytest.ini
plugins: timeout-2.4.0, metadata-3.1.1, xdist-3.8.0, anyio-4.13.0, rerunfailures-16.1, json-report-1.5.0, html-4.2.0
timeout: 120.0s
timeout method: thread
timeout func_only: False
created: 4/4 workers
4 workers [42 items]
scheduling tests via LoadScheduling
tests/scale/test_special_keys.py::test_roundtrip_special_keys[tls-cluster-with+plus+chars]
tests/scale/test_multi_bucket.py::test_100_buckets_list_pagination[tls-cluster]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[tls-cluster-with=equals&and-amp]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[tls-cluster-ascii-plain]
[gw3] [ 2%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[tls-cluster-with=equals&and-amp]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[tls-cluster-\u4e2d\u6587\u6587\u4ef6\u540d.bin]
[gw1] [ 4%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[tls-cluster-ascii-plain]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[tls-cluster-with spaces and dots.txt]
[gw2] [ 7%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[tls-cluster-with+plus+chars]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[tls-cluster-with%percent]
[gw1] [ 9%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[tls-cluster-with spaces and dots.txt]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[tls-cluster-a/b/c/deep]
[gw3] [ 11%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[tls-cluster-\u4e2d\u6587\u6587\u4ef6\u540d.bin]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[tls-cluster-emoji-\U0001f680-\U0001f389.bin]
[gw2] [ 14%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[tls-cluster-with%percent]
tests/scale/test_special_keys.py::test_long_key_1025_rejected_or_works[tls-cluster]
[gw1] [ 16%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[tls-cluster-a/b/c/deep]
tests/scale/test_special_keys.py::test_long_key_1023[tls-cluster]
[gw3] [ 19%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[tls-cluster-emoji-\U0001f680-\U0001f389.bin]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[tls-cluster-with-tab\there]
[gw2] [ 21%] PASSED tests/scale/test_special_keys.py::test_long_key_1025_rejected_or_works[tls-cluster]
tests/scale/test_multi_bucket.py::test_100_buckets_list_pagination[plain-cluster]
[gw3] [ 23%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[tls-cluster-with-tab\there]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[tls-cluster-unicode-na\xefve-caf\xe9]
[gw1] [ 26%] PASSED tests/scale/test_special_keys.py::test_long_key_1023[tls-cluster]
tests/scale/test_multi_bucket.py::test_concurrent_create_same_bucket[plain-cluster]
[gw3] [ 28%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[tls-cluster-unicode-na\xefve-caf\xe9]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[plain-cluster-with%percent]
[gw0] [ 30%] PASSED tests/scale/test_multi_bucket.py::test_100_buckets_list_pagination[tls-cluster]
tests/scale/test_multi_bucket.py::test_concurrent_create_same_bucket[tls-cluster]
[gw0] [ 33%] PASSED tests/scale/test_multi_bucket.py::test_concurrent_create_same_bucket[tls-cluster]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[plain-cluster-\u4e2d\u6587\u6587\u4ef6\u540d.bin]
[gw0] [ 35%] RERUN tests/scale/test_special_keys.py::test_roundtrip_special_keys[plain-cluster-\u4e2d\u6587\u6587\u4ef6\u540d.bin]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[plain-cluster-\u4e2d\u6587\u6587\u4ef6\u540d.bin]
[gw3] [ 38%] RERUN tests/scale/test_special_keys.py::test_roundtrip_special_keys[plain-cluster-with%percent]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[plain-cluster-with%percent]
[gw1] [ 40%] RERUN tests/scale/test_multi_bucket.py::test_concurrent_create_same_bucket[plain-cluster]
tests/scale/test_multi_bucket.py::test_concurrent_create_same_bucket[plain-cluster]
[gw1] [ 40%] PASSED tests/scale/test_multi_bucket.py::test_concurrent_create_same_bucket[plain-cluster]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[plain-cluster-ascii-plain]
[gw3] [ 40%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[plain-cluster-with%percent]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[plain-cluster-with=equals&and-amp]
[gw0] [ 40%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[plain-cluster-\u4e2d\u6587\u6587\u4ef6\u540d.bin]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[plain-cluster-emoji-\U0001f680-\U0001f389.bin]
[gw0] [ 42%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[plain-cluster-emoji-\U0001f680-\U0001f389.bin]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[plain-cluster-a/b/c/deep]
[gw1] [ 45%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[plain-cluster-ascii-plain]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[plain-cluster-with-tab\there]
[gw3] [ 47%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[plain-cluster-with=equals&and-amp]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[plain-cluster-unicode-na\xefve-caf\xe9]
[gw3] [ 50%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[plain-cluster-unicode-na\xefve-caf\xe9]
tests/scale/test_multi_bucket.py::test_100_buckets_list_pagination[standalone]
[gw0] [ 52%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[plain-cluster-a/b/c/deep]
tests/scale/test_special_keys.py::test_long_key_1023[plain-cluster]
[gw1] [ 54%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[plain-cluster-with-tab\there]
tests/scale/test_special_keys.py::test_long_key_1025_rejected_or_works[plain-cluster]
[gw1] [ 57%] PASSED tests/scale/test_special_keys.py::test_long_key_1025_rejected_or_works[plain-cluster]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[standalone-with spaces and dots.txt]
[gw0] [ 59%] PASSED tests/scale/test_special_keys.py::test_long_key_1023[plain-cluster]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[standalone-ascii-plain]
[gw1] [ 61%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[standalone-with spaces and dots.txt]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[standalone-with+plus+chars]
[gw1] [ 64%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[standalone-with+plus+chars]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[standalone-with=equals&and-amp]
[gw0] [ 66%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[standalone-ascii-plain]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[standalone-with%percent]
[gw1] [ 69%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[standalone-with=equals&and-amp]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[standalone-\u4e2d\u6587\u6587\u4ef6\u540d.bin]
[gw0] [ 71%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[standalone-with%percent]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[standalone-emoji-\U0001f680-\U0001f389.bin]
[gw1] [ 73%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[standalone-\u4e2d\u6587\u6587\u4ef6\u540d.bin]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[standalone-with-tab\there]
[gw0] [ 76%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[standalone-emoji-\U0001f680-\U0001f389.bin]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[standalone-unicode-na\xefve-caf\xe9]
[gw2] [ 78%] RERUN tests/scale/test_multi_bucket.py::test_100_buckets_list_pagination[plain-cluster]
tests/scale/test_multi_bucket.py::test_100_buckets_list_pagination[plain-cluster]
[gw1] [ 80%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[standalone-with-tab\there]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[standalone-a/b/c/deep]
[gw0] [ 83%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[standalone-unicode-na\xefve-caf\xe9]
tests/scale/test_special_keys.py::test_long_key_1023[standalone]
[gw1] [ 85%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[standalone-a/b/c/deep]
tests/scale/test_special_keys.py::test_long_key_1025_rejected_or_works[standalone]
[gw0] [ 88%] PASSED tests/scale/test_special_keys.py::test_long_key_1023[standalone]
[gw1] [ 90%] PASSED tests/scale/test_special_keys.py::test_long_key_1025_rejected_or_works[standalone]
[gw3] [ 92%] PASSED tests/scale/test_multi_bucket.py::test_100_buckets_list_pagination[standalone]
tests/scale/test_multi_bucket.py::test_concurrent_create_same_bucket[standalone]
[gw3] [ 95%] PASSED tests/scale/test_multi_bucket.py::test_concurrent_create_same_bucket[standalone]
[gw2] [ 95%] PASSED tests/scale/test_multi_bucket.py::test_100_buckets_list_pagination[plain-cluster]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[plain-cluster-with spaces and dots.txt]
[gw2] [ 97%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[plain-cluster-with spaces and dots.txt]
tests/scale/test_special_keys.py::test_roundtrip_special_keys[plain-cluster-with+plus+chars]
[gw2] [100%] PASSED tests/scale/test_special_keys.py::test_roundtrip_special_keys[plain-cluster-with+plus+chars]
- generated xml file: /tmp/noema-qa-extreme-20260516-075835Z/pytest-scale-fast.junit.xml -
--------------------------------- JSON report ----------------------------------
report saved to: /tmp/noema-qa-extreme-20260516-075835Z/pytest-scale-fast.json
- Generated html report: file:///tmp/noema-qa-extreme-20260516-075835Z/pytest-scale-fast.html -
=================== 42 passed, 4 rerun in 225.44s (0:03:45) ====================
Raw
scale-slow.log
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-7.4.4, pluggy-1.4.0 -- /usr/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.12.3', 'Platform': 'Linux-6.8.0-52-generic-x86_64-with-glibc2.39', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.4.0'}, 'Plugins': {'timeout': '2.4.0', 'metadata': '3.1.1', 'xdist': '3.8.0', 'anyio': '4.13.0', 'rerunfailures': '16.1', 'json-report': '1.5.0', 'html': '4.2.0'}}
rootdir: /home/av/noema/qa
configfile: pytest.ini
plugins: timeout-2.4.0, metadata-3.1.1, xdist-3.8.0, anyio-4.13.0, rerunfailures-16.1, json-report-1.5.0, html-4.2.0
timeout: 120.0s
timeout method: thread
timeout func_only: False
collecting ... collected 54 items / 42 deselected / 12 selected
tests/scale/test_many_objects.py::test_500_objects_list_pagination[tls-cluster] PASSED [ 8%]
tests/scale/test_many_objects.py::test_500_objects_prefix_query[tls-cluster] PASSED [ 16%]
tests/scale/test_many_objects.py::test_10k_objects_listing[tls-cluster] +++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Captured stdout ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
put 1000/10000 in 626.3s
put 2000/10000 in 1306.7s
~~~~~~~~~~~~~~~ Stack of Thread-1 (run_server) (128575504340672) ~~~~~~~~~~~~~~~
File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run()
File "/usr/lib/python3.12/threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "/home/av/.local/lib/python3.12/site-packages/pytest_rerunfailures.py", line 466, in run_server
conn, _ = self.sock.accept()
File "/usr/lib/python3.12/socket.py", line 295, in accept
fd, addr = self._accept()
~~~~~~~~~~~~~~~~~~~~ Stack of MainThread (128575555793024) ~~~~~~~~~~~~~~~~~~~~~
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/usr/lib/python3/dist-packages/pytest/__main__.py", line 5, in <module>
raise SystemExit(pytest.console_main())
File "/usr/lib/python3/dist-packages/_pytest/config/__init__.py", line 192, in console_main
code = main()
File "/usr/lib/python3/dist-packages/_pytest/config/__init__.py", line 169, in main
ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 501, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 119, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 102, in _multicall
res = hook_impl.function(*args)
File "/usr/lib/python3/dist-packages/_pytest/main.py", line 318, in pytest_cmdline_main
return wrap_session(config, _main)
File "/usr/lib/python3/dist-packages/_pytest/main.py", line 271, in wrap_session
session.exitstatus = doit(config, session) or 0
File "/usr/lib/python3/dist-packages/_pytest/main.py", line 325, in _main
config.hook.pytest_runtestloop(session=session)
File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 501, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 119, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 102, in _multicall
res = hook_impl.function(*args)
File "/usr/lib/python3/dist-packages/_pytest/main.py", line 350, in pytest_runtestloop
item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 501, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 119, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 102, in _multicall
res = hook_impl.function(*args)
File "/home/av/.local/lib/python3.12/site-packages/pytest_rerunfailures.py", line 593, in pytest_runtest_protocol
reports = runtestprotocol(item, nextitem=nextitem, log=False)
File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 133, in runtestprotocol
reports.append(call_and_report(item, "call", log))
File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 222, in call_and_report
call = call_runtest_hook(item, when, **kwds)
File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 261, in call_runtest_hook
return CallInfo.from_call(
File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 341, in from_call
result: Optional[TResult] = func()
File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 262, in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 501, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 119, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 102, in _multicall
res = hook_impl.function(*args)
File "/usr/lib/python3/dist-packages/_pytest/runner.py", line 169, in pytest_runtest_call
item.runtest()
File "/usr/lib/python3/dist-packages/_pytest/python.py", line 1792, in runtest
self.ihook.pytest_pyfunc_call(pyfuncitem=self)
File "/usr/lib/python3/dist-packages/pluggy/_hooks.py", line 501, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_manager.py", line 119, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/usr/lib/python3/dist-packages/pluggy/_callers.py", line 102, in _multicall
res = hook_impl.function(*args)
File "/usr/lib/python3/dist-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
result = testfunction(**testargs)
File "/home/av/noema/qa/tests/scale/test_many_objects.py", line 50, in test_10k_objects_listing
s3.client.put_object(Bucket=bucket, Key=f"big/{i:05d}", Body=b"")
File "/usr/lib/python3/dist-packages/botocore/client.py", line 553, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/lib/python3/dist-packages/botocore/client.py", line 989, in _make_api_call
http, parsed_response = self._make_request(
File "/usr/lib/python3/dist-packages/botocore/client.py", line 1015, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/usr/lib/python3/dist-packages/botocore/endpoint.py", line 119, in make_request
return self._send_request(request_dict, operation_model)
File "/usr/lib/python3/dist-packages/botocore/endpoint.py", line 199, in _send_request
success_response, exception = self._get_response(
File "/usr/lib/python3/dist-packages/botocore/endpoint.py", line 241, in _get_response
success_response, exception = self._do_get_response(
File "/usr/lib/python3/dist-packages/botocore/endpoint.py", line 281, in _do_get_response
http_response = self._send(request)
File "/usr/lib/python3/dist-packages/botocore/endpoint.py", line 377, in _send
return self.http_session.send(request)
File "/usr/lib/python3/dist-packages/botocore/httpsession.py", line 464, in send
urllib_response = conn.urlopen(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 791, in urlopen
response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 537, in _make_request
response = conn.getresponse()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 461, in getresponse
httplib_response = super().getresponse()
File "/usr/lib/python3.12/http/client.py", line 1448, in getresponse
response.begin()
File "/usr/lib/python3.12/http/client.py", line 336, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.12/http/client.py", line 297, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.12/socket.py", line 707, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.12/ssl.py", line 1252, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib/python3.12/ssl.py", line 1104, in read
return self._sslobj.read(len, buffer)
+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
Raw
security.log
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-7.4.4, pluggy-1.4.0 -- /usr/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.12.3', 'Platform': 'Linux-6.8.0-52-generic-x86_64-with-glibc2.39', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.4.0'}, 'Plugins': {'timeout': '2.4.0', 'metadata': '3.1.1', 'xdist': '3.8.0', 'anyio': '4.13.0', 'rerunfailures': '16.1', 'json-report': '1.5.0', 'html': '4.2.0'}}
rootdir: /home/av/noema/qa
configfile: pytest.ini
plugins: timeout-2.4.0, metadata-3.1.1, xdist-3.8.0, anyio-4.13.0, rerunfailures-16.1, json-report-1.5.0, html-4.2.0
timeout: 120.0s
timeout method: thread
timeout func_only: False
created: 4/4 workers
4 workers [39 items]
scheduling tests via LoadScheduling
tests/security/test_iam_enforcement.py::test_disabled_user_cannot_authenticate[tls-cluster]
tests/security/test_iam_enforcement.py::test_readonly_policy_denies_put[tls-cluster]
tests/security/test_anonymous_access.py::test_anon_get_with_public_policy[tls-cluster]
tests/security/test_iam_enforcement.py::test_explicit_deny_overrides_allow[tls-cluster]
[gw0] [ 2%] SKIPPED tests/security/test_anonymous_access.py::test_anon_get_with_public_policy[tls-cluster]
tests/security/test_anonymous_access.py::test_anon_get_without_policy_denied[tls-cluster]
[gw1] [ 5%] PASSED tests/security/test_iam_enforcement.py::test_readonly_policy_denies_put[tls-cluster]
tests/security/test_iam_enforcement.py::test_writeonly_policy_denies_get[tls-cluster]
[gw3] [ 7%] PASSED tests/security/test_iam_enforcement.py::test_disabled_user_cannot_authenticate[tls-cluster]
tests/security/test_macaroon_caveats.py::test_macaroon_mint_basic[tls-cluster]
[gw3] [ 10%] PASSED tests/security/test_macaroon_caveats.py::test_macaroon_mint_basic[tls-cluster]
tests/security/test_macaroon_caveats.py::test_macaroon_mint_max_depth_too_high[tls-cluster]
[gw3] [ 12%] PASSED tests/security/test_macaroon_caveats.py::test_macaroon_mint_max_depth_too_high[tls-cluster]
tests/security/test_anonymous_access.py::test_anon_get_with_public_policy[plain-cluster]
[gw2] [ 15%] PASSED tests/security/test_iam_enforcement.py::test_explicit_deny_overrides_allow[tls-cluster]
tests/security/test_iam_enforcement.py::test_user_with_no_policy_denied_by_default[tls-cluster]
[gw3] [ 17%] SKIPPED tests/security/test_anonymous_access.py::test_anon_get_with_public_policy[plain-cluster]
tests/security/test_anonymous_access.py::test_anon_get_without_policy_denied[plain-cluster]
[gw0] [ 20%] PASSED tests/security/test_anonymous_access.py::test_anon_get_without_policy_denied[tls-cluster]
tests/security/test_macaroon_caveats.py::test_macaroon_mint_with_prefix[tls-cluster]
[gw0] [ 23%] PASSED tests/security/test_macaroon_caveats.py::test_macaroon_mint_with_prefix[tls-cluster]
tests/security/test_macaroon_caveats.py::test_macaroon_mint_with_short_ttl[tls-cluster]
[gw0] [ 25%] PASSED tests/security/test_macaroon_caveats.py::test_macaroon_mint_with_short_ttl[tls-cluster]
tests/security/test_iam_enforcement.py::test_explicit_deny_overrides_allow[plain-cluster]
[gw2] [ 28%] PASSED tests/security/test_iam_enforcement.py::test_user_with_no_policy_denied_by_default[tls-cluster]
tests/security/test_iam_enforcement.py::test_writeonly_policy_denies_get[plain-cluster]
[gw3] [ 30%] PASSED tests/security/test_anonymous_access.py::test_anon_get_without_policy_denied[plain-cluster]
tests/security/test_iam_enforcement.py::test_readonly_policy_denies_put[plain-cluster]
[gw1] [ 33%] PASSED tests/security/test_iam_enforcement.py::test_writeonly_policy_denies_get[tls-cluster]
tests/security/test_macaroon_caveats.py::test_macaroon_mint_with_agent_binding[tls-cluster]
[gw1] [ 35%] PASSED tests/security/test_macaroon_caveats.py::test_macaroon_mint_with_agent_binding[tls-cluster]
tests/security/test_macaroon_caveats.py::test_macaroon_mint_empty_ops_rejected[tls-cluster]
[gw1] [ 38%] PASSED tests/security/test_macaroon_caveats.py::test_macaroon_mint_empty_ops_rejected[tls-cluster]
tests/security/test_macaroon_caveats.py::test_macaroon_mint_with_prefix[plain-cluster]
[gw1] [ 41%] PASSED tests/security/test_macaroon_caveats.py::test_macaroon_mint_with_prefix[plain-cluster]
tests/security/test_macaroon_caveats.py::test_macaroon_mint_with_short_ttl[plain-cluster]
[gw1] [ 43%] PASSED tests/security/test_macaroon_caveats.py::test_macaroon_mint_with_short_ttl[plain-cluster]
tests/security/test_macaroon_caveats.py::test_macaroon_mint_with_agent_binding[plain-cluster]
[gw1] [ 46%] PASSED tests/security/test_macaroon_caveats.py::test_macaroon_mint_with_agent_binding[plain-cluster]
tests/security/test_macaroon_caveats.py::test_macaroon_mint_empty_ops_rejected[plain-cluster]
[gw1] [ 48%] PASSED tests/security/test_macaroon_caveats.py::test_macaroon_mint_empty_ops_rejected[plain-cluster]
tests/security/test_macaroon_caveats.py::test_macaroon_mint_max_depth_too_high[plain-cluster]
[gw1] [ 51%] PASSED tests/security/test_macaroon_caveats.py::test_macaroon_mint_max_depth_too_high[plain-cluster]
tests/security/test_anonymous_access.py::test_anon_get_with_public_policy[standalone]
[gw1] [ 53%] SKIPPED tests/security/test_anonymous_access.py::test_anon_get_with_public_policy[standalone]
tests/security/test_anonymous_access.py::test_anon_get_without_policy_denied[standalone]
[gw1] [ 56%] PASSED tests/security/test_anonymous_access.py::test_anon_get_without_policy_denied[standalone]
tests/security/test_iam_enforcement.py::test_readonly_policy_denies_put[standalone]
[gw3] [ 58%] PASSED tests/security/test_iam_enforcement.py::test_readonly_policy_denies_put[plain-cluster]
tests/security/test_macaroon_caveats.py::test_macaroon_mint_basic[plain-cluster]
[gw3] [ 61%] PASSED tests/security/test_macaroon_caveats.py::test_macaroon_mint_basic[plain-cluster]
tests/security/test_iam_enforcement.py::test_explicit_deny_overrides_allow[standalone]
[gw1] [ 64%] PASSED tests/security/test_iam_enforcement.py::test_readonly_policy_denies_put[standalone]
tests/security/test_iam_enforcement.py::test_writeonly_policy_denies_get[standalone]
[gw0] [ 66%] PASSED tests/security/test_iam_enforcement.py::test_explicit_deny_overrides_allow[plain-cluster]
tests/security/test_iam_enforcement.py::test_user_with_no_policy_denied_by_default[plain-cluster]
[gw2] [ 69%] PASSED tests/security/test_iam_enforcement.py::test_writeonly_policy_denies_get[plain-cluster]
tests/security/test_iam_enforcement.py::test_disabled_user_cannot_authenticate[plain-cluster]
[gw2] [ 71%] PASSED tests/security/test_iam_enforcement.py::test_disabled_user_cannot_authenticate[plain-cluster]
tests/security/test_macaroon_caveats.py::test_macaroon_mint_with_prefix[standalone]
[gw2] [ 74%] SKIPPED tests/security/test_macaroon_caveats.py::test_macaroon_mint_with_prefix[standalone]
tests/security/test_macaroon_caveats.py::test_macaroon_mint_with_short_ttl[standalone]
[gw3] [ 76%] PASSED tests/security/test_iam_enforcement.py::test_explicit_deny_overrides_allow[standalone]
tests/security/test_iam_enforcement.py::test_user_with_no_policy_denied_by_default[standalone]
[gw2] [ 79%] SKIPPED tests/security/test_macaroon_caveats.py::test_macaroon_mint_with_short_ttl[standalone]
tests/security/test_macaroon_caveats.py::test_macaroon_mint_with_agent_binding[standalone]
[gw0] [ 82%] PASSED tests/security/test_iam_enforcement.py::test_user_with_no_policy_denied_by_default[plain-cluster]
tests/security/test_macaroon_caveats.py::test_macaroon_mint_basic[standalone]
[gw0] [ 84%] SKIPPED tests/security/test_macaroon_caveats.py::test_macaroon_mint_basic[standalone]
[gw1] [ 87%] PASSED tests/security/test_iam_enforcement.py::test_writeonly_policy_denies_get[standalone]
[gw2] [ 89%] SKIPPED tests/security/test_macaroon_caveats.py::test_macaroon_mint_with_agent_binding[standalone]
tests/security/test_iam_enforcement.py::test_disabled_user_cannot_authenticate[standalone]
tests/security/test_macaroon_caveats.py::test_macaroon_mint_max_depth_too_high[standalone]
[gw2] [ 92%] SKIPPED tests/security/test_macaroon_caveats.py::test_macaroon_mint_max_depth_too_high[standalone]
[gw3] [ 94%] PASSED tests/security/test_iam_enforcement.py::test_user_with_no_policy_denied_by_default[standalone]
tests/security/test_macaroon_caveats.py::test_macaroon_mint_empty_ops_rejected[standalone]
[gw3] [ 97%] SKIPPED tests/security/test_macaroon_caveats.py::test_macaroon_mint_empty_ops_rejected[standalone]
[gw1] [100%] PASSED tests/security/test_iam_enforcement.py::test_disabled_user_cannot_authenticate[standalone]
- generated xml file: /tmp/noema-qa-extreme-20260516-075835Z/pytest-security.junit.xml -
--------------------------------- JSON report ----------------------------------
report saved to: /tmp/noema-qa-extreme-20260516-075835Z/pytest-security.json
- Generated html report: file:///tmp/noema-qa-extreme-20260516-075835Z/pytest-security.html -
=========================== short test summary info ============================
SKIPPED [3] tests/security/test_anonymous_access.py:38: noema doesn't honor public-read bucket policy for unauth requests
SKIPPED [1] tests/security/test_macaroon_caveats.py:35: Skipped
SKIPPED [1] tests/security/test_macaroon_caveats.py:42: Skipped
SKIPPED [1] tests/security/test_macaroon_caveats.py:28: macaroon mint not configured (no master key file)
SKIPPED [1] tests/security/test_macaroon_caveats.py:52: Skipped
SKIPPED [1] tests/security/test_macaroon_caveats.py:69: Skipped
SKIPPED [1] tests/security/test_macaroon_caveats.py:60: Skipped
======================== 30 passed, 9 skipped in 25.58s ========================
Raw
soak.log
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-7.4.4, pluggy-1.4.0 -- /usr/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.12.3', 'Platform': 'Linux-6.8.0-52-generic-x86_64-with-glibc2.39', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.4.0'}, 'Plugins': {'timeout': '2.4.0', 'metadata': '3.1.1', 'xdist': '3.8.0', 'anyio': '4.13.0', 'rerunfailures': '16.1', 'json-report': '1.5.0', 'html': '4.2.0'}}
rootdir: /home/av/noema/qa
configfile: pytest.ini
plugins: timeout-2.4.0, metadata-3.1.1, xdist-3.8.0, anyio-4.13.0, rerunfailures-16.1, json-report-1.5.0, html-4.2.0
timeout: 120.0s
timeout method: thread
timeout func_only: False
collecting ... collected 1 item
tests/soak/test_memory_fd_soak.py::test_one_hour_soak[plain-cluster] PASSED [100%]
- generated xml file: /tmp/noema-qa-extreme-20260516-075835Z/pytest-soak.junit.xml -
--------------------------------- JSON report ----------------------------------
report saved to: /tmp/noema-qa-extreme-20260516-075835Z/pytest-soak.json
======================== 1 passed in 3604.41s (1:00:04) ========================
Raw
summary.txt
run_id: 20260516T105557Z
overall_pass: True
results_dir: /tmp/noema-qa-extreme-20260516-075835Z
warp: total=0 passed=0 failed=0 known_bugs=0
chaos: total=0 recovered=0 data_loss=0
Raw
time.log
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-7.4.4, pluggy-1.4.0 -- /usr/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.12.3', 'Platform': 'Linux-6.8.0-52-generic-x86_64-with-glibc2.39', 'Packages': {'pytest': '7.4.4', 'pluggy': '1.4.0'}, 'Plugins': {'timeout': '2.4.0', 'metadata': '3.1.1', 'xdist': '3.8.0', 'anyio': '4.13.0', 'rerunfailures': '16.1', 'json-report': '1.5.0', 'html': '4.2.0'}}
rootdir: /home/av/noema/qa
configfile: pytest.ini
plugins: timeout-2.4.0, metadata-3.1.1, xdist-3.8.0, anyio-4.13.0, rerunfailures-16.1, json-report-1.5.0, html-4.2.0
timeout: 120.0s
timeout method: thread
timeout func_only: False
created: 4/4 workers
4 workers [0 items]
scheduling tests via LoadScheduling
- generated xml file: /tmp/noema-qa-extreme-20260516-075835Z/pytest-time.junit.xml -
--------------------------------- JSON report ----------------------------------
report saved to: /tmp/noema-qa-extreme-20260516-075835Z/pytest-time.json
- Generated html report: file:///tmp/noema-qa-extreme-20260516-075835Z/pytest-time.html -
============================ no tests ran in 2.06s =============================
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment