Skip to content

Instantly share code, notes, and snippets.

@acmerfight
Created September 7, 2026 08:10
Show Gist options
  • Select an option

  • Save acmerfight/bfe4a686c2689a0a524c0c7c532efee4 to your computer and use it in GitHub Desktop.

Select an option

Save acmerfight/bfe4a686c2689a0a524c0c7c532efee4 to your computer and use it in GitHub Desktop.
GPT-6 Astra low: Codex experimental notes/history vs remote compaction — reproducible single-run comparison

GPT-6 Astra / low:Codex 新上下文机制与远端压缩的一次小型对照

结论:两组代码检查都是 63/63。一个被要求不写进笔记的历史编号,新机制通过历史工具查回;旧组没恢复,也没有编造。

这是一次能力演示,不是统计 benchmark,不证明新机制全面胜出。实验由 Codex 辅助搭建、执行和整理;包含可离线核对的代码、逐项评分与线上复跑脚本。

1. 环境与策略

  • 实验日期:2026-09-07。
  • 实际执行客户端:codex-cli 0.153.4,macOS。
  • 模型:两组均为 gpt-6-astra;app-server 返回的 reasoningEffort 均为 low
  • 通过既有 ChatGPT 登录与默认官方 provider 运行,没有自定义模型目录。
  • 顺序执行:新机制在前、旧机制在后;每组只有一次。
  • 公共设置:workspace-write、approvalPolicy=never,相同任务和继续指令;没有修改真实项目。
  • 新组:features.context_management.experimental_mode=true
  • 旧组:features.context_management.experimental_mode=falsefeatures.token_budget.enabled=false
  • 使用既有用户配置及指令,不是完全隔离的标准化环境。复跑时工具、模型目录、缓存、服务端更新等可能不同。

术语:“新机制”指笔记+可检索历史+不生成摘要的窗口重置;“旧机制”指这里实际触发的远端压缩路径。旧压缩输出可能含加密压缩项,不能简单等同于一段普通文字摘要。

2. 做了什么

起始文件只有一个错误的 retry_delay(attempt, status, retry_after=None) 函数。

初始用户消息给出以下任务:

  1. 保持函数签名,只用 Python 标准库。
  2. attempt 从 0 开始;0–3 可以重试,>=4 返回 None;负数一律先抛 ValueError。
  3. 只允许 HTTP 429、502、503、504 重试。
  4. 默认间隔为 min(3 * 2**attempt, 17)
  5. 只有 429 使用 Retry-After;严格校验非负数和 ASCII 数字字符串,排除 bool、NaN、无穷大、负数、十进制字符串和 Unicode 数字;合法值封顶 17,零合法,无效值回到默认退避。
  6. 禁止 sleep、随机性、新依赖和签名变化。

另外给出一条纯合成历史信息:事件 ORCHID-682,关联编号 trace-7ac91e,服务 eu-west-queue-b。要求模型不要把关联编号复制进笔记或阶段 1 回答,可以保存原消息指针。编号与代码实现无关,专门探测压缩/换窗口后的历史恢复能力。

两阶段流程

  • 阶段 1:读取代码、分析修复、使用可用的机制准备交接;不改代码,不把需求写成本地文件。明确提醒模型维护交接信息,因此不是测试它能否自发记笔记。
  • 控制器对同一 task 调用一次 thread/compact/start。新组走无摘要重置,旧组走远端压缩。这里是手动强制触发,不是自然填满长上下文
  • 阶段 2:只要求继续原任务、测试并报告历史编号,不再次给出原始约束或答案。
  • 外部 Python 评估器执行 63 个功能/接口检查,再对历史编号做 1 个精确匹配检查。评估器在工作目录之外。63 项不是 63 个独立编程任务。
  • 两组各自编写了 8 个 unittest 测试方法,均由控制实验的程序再次运行确认;这些测试另计,不混入 63 项评分。

完整提示词、初始代码和 JSON-RPC 流程均在 run.py

3. 实测数据

项目 新机制 旧远端压缩
独立代码与接口检查 63/63 63/63
历史编号精确恢复 1/1 0/1
模型自建测试,独立复跑 8/8 8/8
准备阶段 50.34 秒 35.77 秒
窗口重置/压缩 1.80 秒 27.17 秒
恢复、实现、测试、回答 67.26 秒 60.30 秒
三阶段合计(不含启动) 119.40 秒 123.24 秒

新组最终回答包含正确编号 trace-7ac91e。旧组表示编号未保存在 checkpoint,原消息历史读取不可用,无法恢复;没有猜测。

原始累计 usage 的选择字段保存在 comparison.json:新组 inputTokens 226156,其中 cachedInputTokens 211840;旧组 inputTokens 176081,其中 cachedInputTokens 142720。这些是多次请求累计值,不是单次窗口长度。没有独立核算压缩及笔记服务全部计费,不能据此推断费用优劣。

4. 如何确认实际用了不同策略

从本地 rollout 提取的记录(comparison.json):

  • 新组:替换记录的 message 为空、无 compaction response ID;新上下文包含 5 个 message 项。换窗口后实际出现 notes.read_filehistory.read_item 调用,然后回答正确历史编号。
  • 旧组:有 compaction response ID;替换内容包含 1 个 message 项与 1 个 compaction 项,说明使用了远端压缩,而不是新组的无摘要重置。
  • message 为空本身不能区分策略,因为加密压缩路径的文本字段也可以为空。

笔记正文及一些工具返回内容是加密的,没有独立解密。因此严谨表述是“要求模型不把编号写进笔记,并观察到重置后的历史读取及正确回答”,而不是声称已逐字审计服务端笔记、证明答案只能来自某一个渠道。

**对照中的辅助记忆差异:**旧组自主调用了 code-mode 的 store/load 保存、核对 checkpoint。它没有原生 notes/history 调用。所以这是实验开关开/关后的真实行为对照,不是禁绝一切辅助记忆的纯摘要算法比较。其阶段 1 和阶段 2 原样回答收录在 artifacts.json

5. 能说明什么,不能说明什么

能说明:在这次 Astra low 试验中,新机制顺利交接代码任务,并展示了通过历史读取补充交接信息的能力;旧机制在代码正确性上没有落后。

不能说明:

  • 新机制在所有任务中更准确、更快、更便宜。
  • 开启后 GPT 不会忘记要求,或能同时看到全部历史。
  • 多次跨窗口、数小时工程任务、自然达到上下文上限时也能稳定保持同样效果。
  • 模型在没有提示的情况下会及时记笔记,或会意识到所有遗漏的约束并主动查找。
  • 一次 1/1 与 0/1 是可泛化的恢复率。

此前还有一次 medium 新机制测试;用户随后要求使用 low,medium 的旧组被中断。medium 不进入这张对照表。

6. 文件与复现

  • artifacts.json:两组实际代码、模型自建测试、两阶段回答、逐项评分。
  • comparison.json:按白名单提取的模型、推理级别、耗时、累计用量、调用序列和压缩记录。
  • evaluate.py:原实验使用的独立评估器。
  • verify_saved_results.py:将已发布代码放到临时目录,离线复跑测试并核对评分;不调用 API。
  • run.py:原 low 试验控制脚本,可尝试线上复跑。依赖已登录且支持实验功能的 Codex CLI、Python 3、可用的 notes/history 后端;使用现有用户配置,会消费模型用量、创建测试任务和合成笔记。

将 gist 文件下载到一个新的空目录后:

# 离线核对已发布的结果,不消费模型用量
python3 verify_saved_results.py

# 可选:重新进行线上实验,结果可能变化
python3 run.py new old
python3 evaluate.py

线上脚本会在当前脚本目录创建 new/workold/work 及本地日志。不要放进真实项目目录运行;保留工作目录隔离。脚本没有更改用户全局配置,试验设置通过 thread/start 覆盖。新旧版本接口或模型可用性变化可能需要调整脚本。

隐私处理:不发布完整 rollout、thread/start 返回值、账户信息、登录配置、本地绝对路径、会话标识、加密笔记载荷或无关工具日志。comparison.json 是提取数据,不冒充完整原始网络记录。线上复跑产生的本地原始日志也不应直接公开。

7. 参考

代码检查的仓库快照是 694b6319d3(2026-09-07);实际执行二进制是 0.153.4,两者不能当作同一构建。实际行为判断以本次运行记录为准。

{
"new": {
"implementation": "import math\n\n\ndef retry_delay(attempt, status, retry_after=None):\n \"\"\"Return delay seconds, or None when retry should stop.\"\"\"\n if attempt < 0:\n raise ValueError(\"attempt must be nonnegative\")\n if attempt >= 4 or status not in (429, 502, 503, 504):\n return None\n\n if status == 429 and not isinstance(retry_after, bool):\n if isinstance(retry_after, int) and retry_after >= 0:\n return min(retry_after, 17)\n if isinstance(retry_after, float):\n if math.isfinite(retry_after) and retry_after >= 0:\n return min(retry_after, 17)\n elif isinstance(retry_after, str):\n value = retry_after.strip()\n if value and all(\"0\" <= digit <= \"9\" for digit in value):\n # Bound conversion even for strings exceeding Python's int limit.\n value = value.lstrip(\"0\") or \"0\"\n if len(value) > 2:\n return 17\n return min(int(value), 17)\n\n return min(3 * 2 ** attempt, 17)\n",
"model_tests": "import inspect\nimport unittest\n\nfrom retry_policy import retry_delay\n\n\nclass RetryDelayTests(unittest.TestCase):\n def test_public_signature(self):\n self.assertEqual(str(inspect.signature(retry_delay)),\n \"(attempt, status, retry_after=None)\")\n\n def test_default_backoff_for_retryable_statuses(self):\n for status in (429, 502, 503, 504):\n for attempt, expected in enumerate((3, 6, 12, 17)):\n with self.subTest(status=status, attempt=attempt):\n self.assertEqual(retry_delay(attempt, status), expected)\n\n def test_only_allowed_http_statuses_retry(self):\n for status in range(100, 600):\n with self.subTest(status=status):\n expected = 3 if status in (429, 502, 503, 504) else None\n self.assertEqual(retry_delay(0, status, 3), expected)\n\n def test_exhausted_attempts_ignore_override(self):\n for attempt in (4, 5, 100, 10 ** 100):\n for status in (200, 429, 502, 503, 504):\n with self.subTest(attempt=attempt, status=status):\n self.assertIsNone(retry_delay(attempt, status, 0))\n\n def test_negative_attempts_raise_for_every_status(self):\n for attempt in (-1, -10):\n for status in (200, 400, 429, 500, 502, 503, 504):\n with self.subTest(attempt=attempt, status=status):\n with self.assertRaises(ValueError):\n retry_delay(attempt, status, 0)\n\n def test_valid_retry_after(self):\n cases = [(0, 0), (0.0, 0), (2, 2), (2.5, 2.5), (17, 17),\n (18, 17), (17.5, 17), (1e308, 17), (10 ** 1000, 17),\n (\"0\", 0), (\" 004 \\t\\n\", 4), (\"17\", 17), (\"18\", 17),\n (\"999\", 17), (\"9\" * 5000, 17), (\"0\" * 5000, 0),\n (\"0\" * 5000 + \"12\", 12)]\n for value, expected in cases:\n for attempt in range(4):\n with self.subTest(value=repr(value)[:80], attempt=attempt):\n self.assertEqual(retry_delay(attempt, 429, value), expected)\n\n def test_invalid_retry_after_uses_default(self):\n values = [None, True, False, -1, -0.5, float(\"nan\"), float(\"inf\"),\n float(\"-inf\"), \"\", \" \\t\\n\", \"1.5\", \"-1\", \"+1\", \"1e2\",\n \"1_000\", \"1 2\", \"12\", \"١٢\", \"²\", \"12٣\", b\"12\", [],\n {}, object(), complex(1, 0)]\n for value in values:\n for attempt, expected in enumerate((3, 6, 12, 17)):\n with self.subTest(value=value, attempt=attempt):\n self.assertEqual(retry_delay(attempt, 429, value), expected)\n\n def test_other_retryable_statuses_ignore_retry_after(self):\n for status in (502, 503, 504):\n for value in (0, 2.5, \"10\", 100, object()):\n for attempt, expected in enumerate((3, 6, 12, 17)):\n with self.subTest(status=status, value=value, attempt=attempt):\n self.assertEqual(retry_delay(attempt, status, value), expected)\n\n\nif __name__ == \"__main__\":\n unittest.main()\n",
"phase1_response": "I’ll inspect the code, identify the correction, and save and verify a native checkpoint for phase 2 without changing files.\n\nREADY",
"phase2_response": "I’ll recover the original requirements and incident ID, then update the retry policy and run focused local tests.\nRepaired `retry_policy.py` with the required attempt limits, status allowlist, backoff, and `retry_after` validation. Preserved the public signature and used only the standard library.\n\nAdded `test_retry_policy.py`. Ran `python3 -m unittest -v test_retry_policy`: **8 tests passed**, including invalid inputs and oversized numeric values.\n\nHistorical incident ORCHID-682 correlation ID: **`trace-7ac91e`**.",
"evaluation": {
"passed": 64,
"total": 64,
"checks": [
{
"name": "nonretry_200",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "nonretry_400",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "nonretry_401",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "nonretry_404",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "nonretry_408",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "nonretry_500",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "nonretry_501",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "nonretry_505",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "backoff_429_0",
"pass": true,
"expected": 3,
"actual": 3
},
{
"name": "backoff_429_1",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "backoff_429_2",
"pass": true,
"expected": 12,
"actual": 12
},
{
"name": "backoff_429_3",
"pass": true,
"expected": 17,
"actual": 17
},
{
"name": "backoff_429_4",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "backoff_429_9",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "backoff_502_0",
"pass": true,
"expected": 3,
"actual": 3
},
{
"name": "backoff_502_1",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "backoff_502_2",
"pass": true,
"expected": 12,
"actual": 12
},
{
"name": "backoff_502_3",
"pass": true,
"expected": 17,
"actual": 17
},
{
"name": "backoff_502_4",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "backoff_502_9",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "backoff_503_0",
"pass": true,
"expected": 3,
"actual": 3
},
{
"name": "backoff_503_1",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "backoff_503_2",
"pass": true,
"expected": 12,
"actual": 12
},
{
"name": "backoff_503_3",
"pass": true,
"expected": 17,
"actual": 17
},
{
"name": "backoff_503_4",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "backoff_503_9",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "backoff_504_0",
"pass": true,
"expected": 3,
"actual": 3
},
{
"name": "backoff_504_1",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "backoff_504_2",
"pass": true,
"expected": 12,
"actual": 12
},
{
"name": "backoff_504_3",
"pass": true,
"expected": 17,
"actual": 17
},
{
"name": "backoff_504_4",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "backoff_504_9",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "valid_header_0",
"pass": true,
"expected": 0,
"actual": 0
},
{
"name": "valid_header_1",
"pass": true,
"expected": 1,
"actual": 1
},
{
"name": "valid_header_2.5",
"pass": true,
"expected": 2.5,
"actual": 2.5
},
{
"name": "valid_header_999",
"pass": true,
"expected": 17,
"actual": 17
},
{
"name": "valid_header_' 07 '",
"pass": true,
"expected": 7,
"actual": 7
},
{
"name": "valid_header_'0'",
"pass": true,
"expected": 0,
"actual": 0
},
{
"name": "valid_header_'003'",
"pass": true,
"expected": 3,
"actual": 3
},
{
"name": "valid_header_'999'",
"pass": true,
"expected": 17,
"actual": 17
},
{
"name": "invalid_header_True",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_False",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_-1",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_-0.5",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_'-1'",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_'1.5'",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_''",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_' '",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_'12'",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_'١٢'",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_'+2'",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_nan",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_inf",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_-inf",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_[]",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_{}",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "ignore_header_502",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "ignore_header_503",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "ignore_header_504",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "negative_200",
"pass": true,
"expected": "ValueError",
"actual": "ValueError"
},
{
"name": "negative_429",
"pass": true,
"expected": "ValueError",
"actual": "ValueError"
},
{
"name": "negative_503",
"pass": true,
"expected": "ValueError",
"actual": "ValueError"
},
{
"name": "signature",
"pass": true,
"actual": "(attempt, status, retry_after=None)"
},
{
"name": "historical_trace",
"pass": true
}
]
}
},
"old": {
"implementation": "import math\n\n\ndef retry_delay(attempt, status, retry_after=None):\n \"\"\"Return delay seconds, or None when retry should stop.\"\"\"\n if attempt < 0:\n raise ValueError(\"attempt must be nonnegative\")\n if attempt >= 4 or status not in (429, 502, 503, 504):\n return None\n\n if status == 429:\n if isinstance(retry_after, int) and not isinstance(retry_after, bool):\n if retry_after >= 0:\n return min(retry_after, 17)\n elif isinstance(retry_after, float):\n if math.isfinite(retry_after) and retry_after >= 0:\n return min(retry_after, 17)\n elif isinstance(retry_after, str):\n value = retry_after.strip()\n if value and value.isascii() and value.isdigit():\n # Bound conversion even for strings exceeding Python's digit limit.\n value = value.lstrip(\"0\") or \"0\"\n return 17 if len(value) > 2 else min(int(value), 17)\n\n return min(3 * 2 ** attempt, 17)\n",
"model_tests": "import inspect\nimport unittest\nfrom decimal import Decimal\nfrom fractions import Fraction\n\nfrom retry_policy import retry_delay\n\n\nclass RetryDelayTests(unittest.TestCase):\n def test_signature(self):\n self.assertEqual(str(inspect.signature(retry_delay)),\n \"(attempt, status, retry_after=None)\")\n\n def test_default_backoff(self):\n for status in (429, 502, 503, 504):\n for attempt, expected in enumerate((3, 6, 12, 17)):\n with self.subTest(status=status, attempt=attempt):\n self.assertEqual(retry_delay(attempt, status), expected)\n\n def test_exhausted_attempts(self):\n for attempt in (4, 5, 100, 10 ** 100):\n for status in (200, 429, 502, 503, 504):\n with self.subTest(attempt=attempt, status=status):\n self.assertIsNone(retry_delay(attempt, status, 0))\n\n def test_negative_attempts_always_raise(self):\n for attempt in (-1, -10):\n for status in (200, 400, 429, 500, 502, 503, 504):\n with self.subTest(attempt=attempt, status=status):\n with self.assertRaises(ValueError):\n retry_delay(attempt, status, 0)\n\n def test_other_statuses_do_not_retry(self):\n for status in range(100, 600):\n if status not in (429, 502, 503, 504):\n with self.subTest(status=status):\n self.assertIsNone(retry_delay(0, status, 0))\n\n def test_valid_retry_after(self):\n cases = [(0, 0), (0.0, 0), (-0.0, 0), (2, 2), (2.5, 2.5),\n (17, 17), (18, 17), (18.5, 17), (10 ** 1000, 17),\n (\"0\", 0), (\" 007\\t\", 7), (\"17\", 17), (\"99\", 17),\n (\"9\" * 5000, 17), (\"0\" * 5000, 0),\n (\"0\" * 5000 + \"12\", 12)]\n for index, (value, expected) in enumerate(cases):\n for attempt in range(4):\n with self.subTest(case=index, attempt=attempt):\n self.assertEqual(retry_delay(attempt, 429, value), expected)\n\n def test_invalid_retry_after_falls_back(self):\n values = [None, True, False, -1, -0.1, float(\"nan\"),\n float(\"inf\"), float(\"-inf\"), \"\", \" \\t\", \"1.5\", \"-1\",\n \"+1\", \"1e2\", \"NaN\", \"inf\", \"١\", \"12\", \"²\", \"1 2\",\n b\"1\", [], {}, object(), Decimal(\"1\"), Fraction(1, 2)]\n for index, value in enumerate(values):\n for attempt, expected in enumerate((3, 6, 12, 17)):\n with self.subTest(case=index, attempt=attempt):\n self.assertEqual(retry_delay(attempt, 429, value), expected)\n\n def test_retry_after_ignored_for_other_retryable_statuses(self):\n for status in (502, 503, 504):\n for value in (0, 1.5, \"2\", 100, object()):\n for attempt, expected in enumerate((3, 6, 12, 17)):\n with self.subTest(status=status, attempt=attempt, value=value):\n self.assertEqual(retry_delay(attempt, status, value), expected)\n\n\nif __name__ == \"__main__\":\n unittest.main()\n",
"phase1_response": "I’ll inspect the code and available native checkpoint tools, then outline the correction without changing files.\n\nInspected `retry_policy.py` and planned the correction. Code remains unchanged. Saved and verified checkpoint `retry_policy_phase1_checkpoint` in native session memory.\n\nREADY",
"phase2_response": "I’ll recover the checkpoint, implement the retry rules, and run boundary and input-validation tests. I’ll report the incident ID only if I can recover it from the original message.\nImplemented the repair with the exact signature and standard library only.\n\nValidation: `python3 -B -m unittest -v test_retry_policy` — all 8 tests passed.\n\nThe correlation ID was omitted from the checkpoint, and original-message history retrieval is unavailable. I can’t recover it without guessing.",
"evaluation": {
"passed": 63,
"total": 64,
"checks": [
{
"name": "nonretry_200",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "nonretry_400",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "nonretry_401",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "nonretry_404",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "nonretry_408",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "nonretry_500",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "nonretry_501",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "nonretry_505",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "backoff_429_0",
"pass": true,
"expected": 3,
"actual": 3
},
{
"name": "backoff_429_1",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "backoff_429_2",
"pass": true,
"expected": 12,
"actual": 12
},
{
"name": "backoff_429_3",
"pass": true,
"expected": 17,
"actual": 17
},
{
"name": "backoff_429_4",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "backoff_429_9",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "backoff_502_0",
"pass": true,
"expected": 3,
"actual": 3
},
{
"name": "backoff_502_1",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "backoff_502_2",
"pass": true,
"expected": 12,
"actual": 12
},
{
"name": "backoff_502_3",
"pass": true,
"expected": 17,
"actual": 17
},
{
"name": "backoff_502_4",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "backoff_502_9",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "backoff_503_0",
"pass": true,
"expected": 3,
"actual": 3
},
{
"name": "backoff_503_1",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "backoff_503_2",
"pass": true,
"expected": 12,
"actual": 12
},
{
"name": "backoff_503_3",
"pass": true,
"expected": 17,
"actual": 17
},
{
"name": "backoff_503_4",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "backoff_503_9",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "backoff_504_0",
"pass": true,
"expected": 3,
"actual": 3
},
{
"name": "backoff_504_1",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "backoff_504_2",
"pass": true,
"expected": 12,
"actual": 12
},
{
"name": "backoff_504_3",
"pass": true,
"expected": 17,
"actual": 17
},
{
"name": "backoff_504_4",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "backoff_504_9",
"pass": true,
"expected": null,
"actual": null
},
{
"name": "valid_header_0",
"pass": true,
"expected": 0,
"actual": 0
},
{
"name": "valid_header_1",
"pass": true,
"expected": 1,
"actual": 1
},
{
"name": "valid_header_2.5",
"pass": true,
"expected": 2.5,
"actual": 2.5
},
{
"name": "valid_header_999",
"pass": true,
"expected": 17,
"actual": 17
},
{
"name": "valid_header_' 07 '",
"pass": true,
"expected": 7,
"actual": 7
},
{
"name": "valid_header_'0'",
"pass": true,
"expected": 0,
"actual": 0
},
{
"name": "valid_header_'003'",
"pass": true,
"expected": 3,
"actual": 3
},
{
"name": "valid_header_'999'",
"pass": true,
"expected": 17,
"actual": 17
},
{
"name": "invalid_header_True",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_False",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_-1",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_-0.5",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_'-1'",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_'1.5'",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_''",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_' '",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_'12'",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_'١٢'",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_'+2'",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_nan",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_inf",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_-inf",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_[]",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "invalid_header_{}",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "ignore_header_502",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "ignore_header_503",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "ignore_header_504",
"pass": true,
"expected": 6,
"actual": 6
},
{
"name": "negative_200",
"pass": true,
"expected": "ValueError",
"actual": "ValueError"
},
{
"name": "negative_429",
"pass": true,
"expected": "ValueError",
"actual": "ValueError"
},
{
"name": "negative_503",
"pass": true,
"expected": "ValueError",
"actual": "ValueError"
},
{
"name": "signature",
"pass": true,
"actual": "(attempt, status, retry_after=None)"
},
{
"name": "historical_trace",
"pass": false
}
]
}
}
}
{
"new": {
"model": "gpt-6-astra",
"effort": "low",
"score": "64/64",
"phases": [
{
"seconds": 50.34,
"status": "completed"
},
{
"seconds": 1.8,
"status": "completed"
},
{
"seconds": 67.26,
"status": "completed"
}
],
"usage": {
"totalTokens": 228667,
"inputTokens": 226156,
"cachedInputTokens": 211840,
"cacheWriteInputTokens": 0,
"outputTokens": 2511,
"reasoningOutputTokens": 11
},
"compactions": [
{
"empty_summary": true,
"compaction_response_id_present": false,
"replacement_items": 5,
"replacement_types": {
"message": 5
}
}
],
"tool_calls": [
{
"window": 0,
"namespace": null,
"name": "exec"
},
{
"window": 0,
"namespace": null,
"name": "exec"
},
{
"window": 0,
"namespace": "history",
"name": "list_items"
},
{
"window": 0,
"namespace": "notes",
"name": "write_file"
},
{
"window": 0,
"namespace": "notes",
"name": "read_file"
},
{
"window": 1,
"namespace": "notes",
"name": "read_file"
},
{
"window": 1,
"namespace": "history",
"name": "read_item"
},
{
"window": 1,
"namespace": null,
"name": "exec"
},
{
"window": 1,
"namespace": null,
"name": "exec"
}
],
"commands": [
{
"command": "/bin/zsh -lc pwd",
"exitCode": 0
},
{
"command": "/bin/zsh -lc \"rg --files -g 'retry_policy.py' -g 'AGENTS.md' -g '*retry*'\"",
"exitCode": 0
},
{
"command": "/bin/zsh -lc 'cat retry_policy.py'",
"exitCode": 0
},
{
"command": "/bin/zsh -lc \"pwd && rg --files -g '*.py' -g 'AGENTS.md' -g 'pyproject.toml' -g 'pytest.ini' -g 'setup.cfg' && cat retry_policy.py\"",
"exitCode": 0
},
{
"command": "/bin/zsh -lc 'PYTHONDONTWRITEBYTECODE=1 python3 -m unittest -v test_retry_policy'",
"exitCode": 0
}
]
},
"old": {
"model": "gpt-6-astra",
"effort": "low",
"score": "63/64",
"phases": [
{
"seconds": 35.77,
"status": "completed"
},
{
"seconds": 27.17,
"status": "completed"
},
{
"seconds": 60.3,
"status": "completed"
}
],
"usage": {
"totalTokens": 178375,
"inputTokens": 176081,
"cachedInputTokens": 142720,
"cacheWriteInputTokens": 0,
"outputTokens": 2294,
"reasoningOutputTokens": 35
},
"compactions": [
{
"empty_summary": true,
"compaction_response_id_present": true,
"replacement_items": 2,
"replacement_types": {
"message": 1,
"compaction": 1
}
}
],
"tool_calls": [
{
"window": 0,
"namespace": null,
"name": "exec"
},
{
"window": 0,
"namespace": null,
"name": "exec"
},
{
"window": 0,
"namespace": null,
"name": "exec"
},
{
"window": 0,
"namespace": null,
"name": "exec"
},
{
"window": 1,
"namespace": null,
"name": "exec"
},
{
"window": 1,
"namespace": null,
"name": "exec"
}
],
"commands": [
{
"command": "/bin/zsh -lc \"pwd; rg --files -g '\"'!'\"\\\\.git' -g 'AGENTS.md' -g '*retry*' -g '*test*'\"",
"exitCode": 0
},
{
"command": "/bin/zsh -lc 'cat retry_policy.py'",
"exitCode": 0
},
{
"command": "/bin/zsh -lc 'cat retry_policy.py'",
"exitCode": 0
},
{
"command": "/bin/zsh -lc 'python3 -B -m unittest -v test_retry_policy'",
"exitCode": 0
}
]
}
}
import pathlib,json,importlib.util,hashlib,math,inspect
ROOT=pathlib.Path(__file__).resolve().parent
for mode in ['new','old']:
p=ROOT/mode/'work'/'retry_policy.py'
if not (ROOT/mode/'result.txt').exists():continue
spec=importlib.util.spec_from_file_location('retry_'+mode,p);m=importlib.util.module_from_spec(spec);spec.loader.exec_module(m);f=m.retry_delay
cases=[]
def add(name,args,expect):cases.append((name,args,expect))
for status in [200,400,401,404,408,500,501,505]:add('nonretry_'+str(status),(0,status,8),None)
for status in [429,502,503,504]:
for attempt,expect in [(0,3),(1,6),(2,12),(3,17),(4,None),(9,None)]:add(f'backoff_{status}_{attempt}',(attempt,status),expect)
for value,expect in [(0,0),(1,1),(2.5,2.5),(999,17),(' 07 ',7),('0',0),('003',3),('999',17)]:add('valid_header_'+repr(value),(1,429,value),expect)
for value in [True,False,-1,-0.5,'-1','1.5','', ' ', '12','١٢','+2',float('nan'),float('inf'),float('-inf'),[],{}]:add('invalid_header_'+repr(value),(1,429,value),6)
for status in [502,503,504]:add('ignore_header_'+str(status),(1,status,0),6)
for status in [200,429,503]:add('negative_'+str(status),(-1,status),'ValueError')
checks=[]
for name,args,expected in cases:
try:value=f(*args);ok=value==expected
except Exception as e:value=type(e).__name__;ok=value==expected
checks.append({'name':name,'pass':ok,'expected':expected,'actual':value})
sig=str(inspect.signature(f));checks.append({'name':'signature','pass':sig=='(attempt, status, retry_after=None)','actual':sig})
result=(ROOT/mode/'result.txt').read_text();checks.append({'name':'historical_trace','pass':'trace-7ac91e' in result})
summary={'passed':sum(c['pass'] for c in checks),'total':len(checks),'checks':checks}
(ROOT/mode/'evaluation.json').write_text(json.dumps(summary,indent=2,default=str));print(mode,summary['passed'],'/',summary['total']);print('failures', [c for c in checks if not c['pass']])
import asyncio,json,pathlib,time,sys
ROOT=pathlib.Path(__file__).resolve().parent
CODE='''def retry_delay(attempt, status, retry_after=None):
"""Return delay seconds, or None when retry should stop."""
if status >= 400:
return min(2 ** attempt, 30)
return None
'''
PROMPT='''You are participating in an authorized isolated coding experiment. Work only in this directory. Do not delegate, browse the web, inspect other directories, or use other tasks' history. Native notes/history for this task are allowed. Never save prompts, requirements, or progress to local files; native notes are allowed. Do not create AGENTS.md.
Task: repair retry_delay in retry_policy.py, preserving its exact public signature and using Python standard library only.
Requirements supplied only in this conversation:
- attempt is zero-based; attempts 0,1,2,3 may retry, attempt >=4 must return None.
- negative attempts must raise ValueError, even when the HTTP status is not retryable.
- only 429,502,503,504 are retryable. All other statuses return None.
- default backoff is 3 * 2**attempt seconds, capped at 17.
- retry_after is used ONLY for status 429. Accept a nonnegative finite int or float excluding bool, or a stripped nonempty string of ASCII digits. Cap accepted values at 17. All invalid inputs fall back to default backoff. Do not accept decimal strings, negative strings, unicode digits, NaN, or infinity.
- zero retry_after is valid. Never sleep, use randomness, change the public signature, or add dependencies.
Synthetic historical incident (not present in files): ticket ORCHID-682 had correlation ID trace-7ac91e; the failed service was eu-west-queue-b. That trace ID is incident metadata, not the retry algorithm. To test history retrieval, do NOT copy the trace ID into your final planning response or any notes; you may keep a pointer to the original message. A later question may request it.
PHASE 1 ONLY: inspect the existing code, analyze the required correction, and prepare to continue after context compaction using whatever native mechanism is available. Save a concise checkpoint if native notes are available, and verify it was saved. Do not change code yet. Do not call new_context yourself; the controller will compact exactly once between phases. End with READY or a concrete tool failure.
'''
FOLLOW='''PHASE 2: Continue the original coding task after the context transition. Recover the original requirements using the available context/notes/history. Implement the repair, create and run useful tests, and finish. Only local implementation and test files may now be written. In the final response report actual test results and the correlation ID of the historical incident from phase 1. Do not guess it. Do not inspect outside this work directory or other tasks. Do not delegate or perform another context reset.'''
class RPC:
async def start(self,mode):
self.mode=mode;self.i=0;self.pending={};self.q=asyncio.Queue();self.log=(ROOT/mode/'events.jsonl').open('w');self.err=(ROOT/mode/'stderr.log').open('w')
self.proc=await asyncio.create_subprocess_exec('codex','app-server','--stdio',stdin=asyncio.subprocess.PIPE,stdout=asyncio.subprocess.PIPE,stderr=self.err)
self.reader=asyncio.create_task(self.read())
await self.call('initialize',{'clientInfo':{'name':'context_effectiveness_probe','version':'1.0'},'capabilities':{'experimentalApi':True}})
await self.send({'method':'initialized','params':{}})
async def send(self,d):
self.proc.stdin.write((json.dumps(d)+'\n').encode());await self.proc.stdin.drain()
async def read(self):
while line:=await self.proc.stdout.readline():
try:d=json.loads(line)
except:continue
self.log.write(json.dumps({'at':time.time(),'rpc':d})+'\n');self.log.flush()
if 'id' in d and 'method' not in d:
fut=self.pending.pop(d['id'],None)
if fut and not fut.done():fut.set_result(d)
elif 'method' in d and 'id' in d:
await self.send({'id':d['id'],'error':{'code':-32601,'message':'Interactive requests not supported in this isolated test'}})
else: await self.q.put(d)
async def call(self,method,params):
self.i+=1;f=asyncio.get_running_loop().create_future();self.pending[self.i]=f
await self.send({'id':self.i,'method':method,'params':params});d=await asyncio.wait_for(f,120)
if 'error' in d:raise RuntimeError(d['error'])
return d['result']
async def finish_turn(self):
messages=[]
while True:
d=await asyncio.wait_for(self.q.get(),360)
m=d.get('method','');p=d.get('params',{})
if m=='item/completed':
item=p.get('item',{})
if item.get('type')=='agentMessage':messages.append(item.get('text',''))
if m=='turn/completed':
if p.get('turn',{}).get('status')!='completed':raise RuntimeError(p)
return '\n'.join(messages)
async def close(self):
self.proc.stdin.close()
try:await asyncio.wait_for(self.proc.wait(),8)
except: self.proc.terminate();await self.proc.wait()
self.reader.cancel();self.log.close();self.err.close()
async def run(mode):
path=ROOT/mode;work=path/'work';work.mkdir(parents=True,exist_ok=True);(work/'retry_policy.py').write_text(CODE)
rpc=RPC();started=time.time()
try:
await rpc.start(mode)
cfg={'features.context_management.experimental_mode':mode=='new','model_reasoning_effort':'low'}
if mode=='old':cfg['features.token_budget.enabled']=False
r=await rpc.call('thread/start',{'model':'gpt-6-astra','cwd':str(work),'approvalPolicy':'never','sandbox':'workspace-write','config':cfg})
tid=r['thread']['id'];(path/'thread.json').write_text(json.dumps(r,indent=2));print(mode,'started',tid,flush=True)
await rpc.call('turn/start',{'threadId':tid,'input':[{'type':'text','text':PROMPT}]})
plan=await rpc.finish_turn();(path/'plan.txt').write_text(plan);print(mode,'phase 1 complete',flush=True)
t=time.time();await rpc.call('thread/compact/start',{'threadId':tid});await rpc.finish_turn();compact=time.time()-t;print(mode,'compacted',round(compact,1),'seconds',flush=True)
await rpc.call('turn/start',{'threadId':tid,'input':[{'type':'text','text':FOLLOW}]})
result=await rpc.finish_turn();(path/'result.txt').write_text(result)
(path/'timing.json').write_text(json.dumps({'total_seconds':time.time()-started,'compact_seconds':compact}))
print(mode,'completed',flush=True)
except Exception as e:
(path/'failure.txt').write_text(repr(e));print(mode,'ERROR',repr(e),flush=True)
finally:await rpc.close()
async def main():
for mode in sys.argv[1:] or ['new','old']:await run(mode)
asyncio.run(main())
"""Offline verification of the published code and scores; no API calls."""
import json,pathlib,tempfile,subprocess,sys,shutil
root=pathlib.Path(__file__).resolve().parent
artifacts=json.loads((root/'artifacts.json').read_text())
with tempfile.TemporaryDirectory(prefix='codex-context-check-') as temp:
temp=pathlib.Path(temp)
shutil.copy2(root/'evaluate.py',temp/'evaluate.py')
for mode,a in artifacts.items():
work=temp/mode/'work';work.mkdir(parents=True)
(work/'retry_policy.py').write_text(a['implementation'])
(work/'test_retry_policy.py').write_text(a['model_tests'])
(temp/mode/'result.txt').write_text(a['phase2_response'])
print('Model-authored tests:',mode,flush=True)
subprocess.run([sys.executable,'-B','-m','unittest','-v','test_retry_policy'],cwd=work,check=True)
subprocess.run([sys.executable,'-B',str(temp/'evaluate.py')],check=True)
for mode,a in artifacts.items():
actual=json.loads((temp/mode/'evaluation.json').read_text())
assert actual==a['evaluation'],f'Published score mismatch: {mode}'
print('All published scores reproduced from saved artifacts.')
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment