Kohaku-Blueleaf KohakuBlueleaf
KohakuBlueleaf
/ test.py
Created
May 4, 2025 14:44
A minimal reproduction of the "CUDA error: an illegal memory access was encountered" error in nccl + pytorch for ddp model training
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import os | |
| import torch | |
| import torch.nn as nn | |
| import torch.optim as optim | |
| import torch.distributed as dist | |
| import torch.multiprocessing as mp | |
| import torchvision.models as models | |
| from torch.utils.data import Dataset | |
| from torch.nn.parallel import DistributedDataParallel as DDP | |
| from torch.utils.data.distributed import DistributedSampler |
KohakuBlueleaf
/ repro_mib_bugs
Created
June 30, 2026 08:26
A simple script to find and reproduce the "different are in MIG group" in Intel FloorSet training partition.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| """Minimal, self-contained reproduction of the FloorSet-Lite MIB inconsistency. | |
| MIB (Multiple-Instantiation Block) requires every member of a group to share IDENTICAL (w,h) | |
| dimensions -- which forces an identical area. This script proves, straight from IntelLabs' published | |
| data (no project code on the train path), that the TRAIN set assigns members of one MIB group | |
| DIFFERENT target areas, so the MIB constraint is unsatisfiable by construction. It then runs the exact | |
| same check on the VAL set, which is clean -- demonstrating the check is correct and the bug is in the | |
| train data, not the check. | |
| Raw layout tensor layout (FloorSet-Lite): ``c[0][i]`` is ``[n,6]`` with column 0 = target area and |
OlderNewer