sometime I get following error with same sample same code sometime I don't. What do you think the issue is?
INFO [resume] attempt 1 for zz_4l[1] using root://eos.cms.rcac.purdue.edu/ run_stage1.py:435
INFO rm command executed for: /work/projects/hmm/shar1172/hmm_ntuples/copperheadV1clean/Run3_nanoAODv12_FilterJets_July08_tightPassLepVeto_DefaultjetPt25GeV_JVMFilterJets/stage1_output/2022postEE/f1_0/zz_4l/1 run_stage1.py:449
Preprocessing 100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 94/94 [ 0:00:02 < 0:00:00 | 41.1 file/s ]
WARNING [resume] attempt 1 failed for zz_4l[1] (AttributeError: 'NoneType' object has no attribute '__traceback__') run_stage1.py:495
INFO Saved error info to /work/projects/hmm/shar1172/hmm_ntuples/copperheadV1clean/Run3_nanoAODv12_FilterJets_July08_tightPassLepVeto_DefaultjetPt25GeV_JVMFilterJets/stage1_output/2022postEE/_status/error_zz_4l_1_20260812-124524.txt run_stage1.py:514
ERROR [resume] write failed after 1 attempts for zz_4l[1] run_stage1.py:538
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ /work/users/shar1172/copperheadV2_develop/run_stage1.py:453 in <module> │
│ │
│ 450 │ │ │ │ │ │ │ eos_mkdirs(save_path) │
│ 451 │ │ │ │ │ │ │ │
│ 452 │ │ │ │ │ │ │ # rebuild the events/out collections for this attempt │
│ ❱ 453 │ │ │ │ │ │ │ processed_event_count = dataset_loop(coffea_processor, alt_s │
│ 454 │ │ │ │ │ │ │ │
│ 455 │ │ │ │ │ │ │ logger.info(f"Expected events: {ExpectedEvents_from_prestag │
│ 456 │ │ │ │ │ │ │ logger.info(f"Processed events: {processed_event_count}") │
│ │
│ /work/users/shar1172/copperheadV2_develop/run_stage1.py:187 in dataset_loop │
│ │
│ 184 │ │ } │
│ 185 │ } │
│ 186 │ │
│ ❱ 187 │ result = runner(fileset, processor_instance=adapter) │
│ 188 │ total_processed = int(result["__n_processed__"]) │
│ 189 │ │
│ 190 │ return total_processed │
│ │
│ /cvmfs/cms-af.opensciencegrid.org/paf/pixi/copperheadV2/.pixi/envs/default/lib/python3.11/site-packages/coffea/processor/executor.py:1632 in __call__ │
│ │
│ 1629 │ │ │ uproot_options = {} │
│ 1630 │ │ if iteritems_options is None: │
│ 1631 │ │ │ iteritems_options = {} │
│ ❱ 1632 │ │ wrapped_out = self.run( │
│ 1633 │ │ │ fileset=fileset, │
│ 1634 │ │ │ processor_instance=processor_instance, │
│ 1635 │ │ │ treename=treename, │
│ │
│ /cvmfs/cms-af.opensciencegrid.org/paf/pixi/copperheadV2/.pixi/envs/default/lib/python3.11/site-packages/coffea/processor/executor.py:1867 in run │
│ │
│ 1864 │ │ │ closure, │
│ 1865 │ │ ) │
│ 1866 │ │ │
│ ❱ 1867 │ │ wrapped_out, e = executor(chunks, closure, None) │
│ 1868 │ │ if wrapped_out is None: │
│ 1869 │ │ │ raise ValueError( │
│ 1870 │ │ │ │ "No chunks returned results, verify ``processor`` instance structure.\n" │
│ │
│ /cvmfs/cms-af.opensciencegrid.org/paf/pixi/copperheadV2/.pixi/envs/default/lib/python3.11/site-packages/coffea/processor/executor.py:807 in __call__ │
│ │
│ 804 │ │ │ │ if self.status: │
│ 805 │ │ │ │ │ from ._dask import progress │
│ 806 │ │ │ │ │ │
│ ❱ 807 │ │ │ │ │ progress(work, description=f"[green]{self.desc}", unit=self.unit) │
│ 808 │ │ │ │ return ( │
│ 809 │ │ │ │ │ accumulate( │
│ 810 │ │ │ │ │ │ [ │
│ │
│ /cvmfs/cms-af.opensciencegrid.org/paf/pixi/copperheadV2/.pixi/envs/default/lib/python3.11/site-packages/coffea/processor/_dask.py:84 in progress │
│ │
│ 81 │ │ futures = futures_of(futures) │
│ 82 │ │ if not isinstance(futures, (set, list)): │
│ 83 │ │ │ futures = [futures] │
│ ❱ 84 │ │ RichProgressBar(futures, complete=complete, **kwargs) │
│ 85 │
│ │
│ /cvmfs/cms-af.opensciencegrid.org/paf/pixi/copperheadV2/.pixi/envs/default/lib/python3.11/site-packages/coffea/processor/_dask.py:39 in __init__ │
│ │
│ 36 │ │ self.task = self.pbar.add_task(description, total=len(keys), unit=unit) │
│ 37 │ │ │
│ 38 │ │ self._loop_runner = LoopRunner(loop=None) │
│ ❱ 39 │ │ self._loop_runner.run_sync(self.listen) │
│ 40 │ │
│ 41 │ @property │
│ 42 │ def loop(self) -> IOLoop | None: │
│ │
│ /cvmfs/cms-af.opensciencegrid.org/paf/pixi/copperheadV2/.pixi/envs/default/lib/python3.11/site-packages/distributed/utils.py:627 in run_sync │
│ │
│ 624 │ │ else: │
│ 625 │ │ │ self.start() │
│ 626 │ │ │ try: │
│ ❱ 627 │ │ │ │ return sync(self.loop, func, *args, **kwargs) │
│ 628 │ │ │ finally: │
│ 629 │ │ │ │ self.stop() │
│ 630 │
│ │
│ /cvmfs/cms-af.opensciencegrid.org/paf/pixi/copperheadV2/.pixi/envs/default/lib/python3.11/site-packages/distributed/utils.py:439 in sync │
│ │
│ 436 │ │ │ wait(10) │
│ 437 │ │
│ 438 │ if error is not None: │
│ ❱ 439 │ │ raise error │
│ 440 │ else: │
│ 441 │ │ return result │
│ 442 │
│ │
│ /cvmfs/cms-af.opensciencegrid.org/paf/pixi/copperheadV2/.pixi/envs/default/lib/python3.11/site-packages/distributed/utils.py:413 in f │
│ │
│ 410 │ │ │ if timeout is not None: │
│ 411 │ │ │ │ awaitable = wait_for(awaitable, timeout) │
│ 412 │ │ │ future = asyncio.ensure_future(awaitable) │
│ ❱ 413 │ │ │ result = yield future │
│ 414 │ │ except Exception as exception: │
│ 415 │ │ │ error = exception │
│ 416 │ │ finally: │
│ │
│ /cvmfs/cms-af.opensciencegrid.org/paf/pixi/copperheadV2/.pixi/envs/default/lib/python3.11/site-packages/tornado/gen.py:783 in run │
│ │
│ 780 │ │ │ │ self.future = None │
│ 781 │ │ │ │ try: │
│ 782 │ │ │ │ │ try: │
│ ❱ 783 │ │ │ │ │ │ value = future.result() │
│ 784 │ │ │ │ │ except Exception as e: │
│ 785 │ │ │ │ │ │ # Save the exception for later. It's important that │
│ 786 │ │ │ │ │ │ # gen.throw() not be called inside this try/except block │
│ │
│ /cvmfs/cms-af.opensciencegrid.org/paf/pixi/copperheadV2/.pixi/envs/default/lib/python3.11/site-packages/distributed/diagnostics/progressbar.py:103 in listen │
│ │
│ 100 │ │ │ self._draw_bar(**response) │
│ 101 │ │ │ if response["status"] in ("error", "finished"): │
│ 102 │ │ │ │ await self.comm.close() │
│ ❱ 103 │ │ │ │ self._draw_stop(**response) │
│ 104 │ │ │ │ break │
│ 105 │ │ │
│ 106 │ │ logger.debug("Progressbar disconnected from scheduler") │
│ │
│ /cvmfs/cms-af.opensciencegrid.org/paf/pixi/copperheadV2/.pixi/envs/default/lib/python3.11/site-packages/coffea/processor/_dask.py:61 in _draw_stop │
│ │
│ 58 │ │ │ rtc = Traceback.from_exception( │
│ 59 │ │ │ │ type(exception), │
│ 60 │ │ │ │ exception, │
│ ❱ 61 │ │ │ │ exception.__traceback__, │
│ 62 │ │ │ ) │
│ 63 │ │ │ self.pbar.console.print(rtc) │
│ 64 │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'NoneType' object has no attribute '__traceback__'
INFO Finished file_idx 1 in 58.612 s. run_stage1.py:543Response from claude:
Found it — this is a real bug in coffea itself, and it's masking the actual underlying error rather than being the real cause.
Look at coffea/processor/_dask.py:52-63:
def _draw_stop(self, remaining, all, status, exception=None, **kwargs):
if status == "error":
_, exception, _ = clean_exception(exception)
rtc = Traceback.from_exception(
type(exception),
exception,
exception.__traceback__, # <- crashes if exception is None
)clean_exception() can legitimately return exception=None — this happens when a task fails without a real Python exception attached, which is exactly what happens when a Dask worker itself dies (OOM-killed, evicted by Kubernetes, or otherwise lost) rather than the task code raising a normal exception. There's no exception object to report because nothing in Python actually raised anything — the worker just vanished. Coffea's progress-bar code doesn't guard against that case, so instead of showing you the real failure (a lost/killed worker), it crashes with this confusing AttributeError and swallows the actual cause.
Why it's intermittent on the same sample/code: this is consistent with worker loss, not a deterministic bug — timing-dependent on cluster load/memory pressure at that exact moment. It also lines up with the same Dask Gateway adaptive-scaling instability we found earlier this session (desired worker count oscillating wildly), which is exactly the kind of thing that gets workers killed mid-task. Important side effect: Dask's own per-task retry mechanism (default-task-retries=5, set at the top of run_stage1.py) is normally supposed to transparently retry a task when its worker is lost — but this coffea progress-bar bug crashes before that retry can complete, turning what should be a silent, automatic recovery into a hard failure for the whole chunk.