Issue: JuliaLang/julia#56840 Fix branch:
cyhan/csl-sysimage(based on master)
When building Julia on Windows MSYS2 MINGW64, linking sys.dll (basecompiler.dll) fails:
ld.exe: dllcrt2.o: undefined reference to `_initterm_e'
The sys.dll link command in sysimage.mk:22 (before the fix):
$(CXX) $(LDFLAGS) -shared $(fPIC) -L$(build_private_libdir) -L$(build_libdir) -L$(build_shlibdir) -o $@ ...Files participating in the link fall into two injection paths:
| Injection Path | File | Source | Can -L override? |
|---|---|---|---|
| GCC spec hardcoded absolute path | dllcrt2.o |
System compiler (/mingw64/.../lib/dllcrt2.o) |
❌ |
| GCC spec hardcoded absolute path | crtbegin.o |
System compiler (/mingw64/.../gcc/16.1.0/crtbegin.o) |
❌ |
| GCC spec hardcoded absolute path | crtend.o |
System compiler (/mingw64/.../gcc/16.1.0/crtend.o) |
❌ |
-L + implicit -l |
libmsvcrt.a |
build_private_libdir (CSL, GCC 15 CRT) |
✅ |
-L + implicit -l |
libmingw32.a |
build_private_libdir (CSL, GCC 15 CRT) |
✅ |
-L + explicit -lssp |
libssp.dll.a |
build_private_libdir (CSL) |
✅ |
The user's build machine uses GCC 16. Its mingw-w64 CRT's dllcrt2.o calls both _initterm and _initterm_e. CSL (CompilerSupportLibraries jll) ships GCC 15's mingw-w64 CRT, whose libmsvcrt.a provides only _initterm, not _initterm_e.
┌───────────────────────────────────────────────────────────┐
│ GCC spec injection (absolute path, -L cannot override) │
│ │
│ dllcrt2.o → System GCC 16 CRT │
│ U _initterm │
│ U _initterm_e ← needed │
├───────────────────────────────────────────────────────────┤
│ -L$(build_private_libdir) is first in search order │
│ │
│ libmsvcrt.a → CSL (GCC 15 CRT) │
│ T _initterm ← provided │
│ (no _initterm_e) ← missing ❌ │
└───────────────────────────────────────────────────────────┘
The link command in sysimage.mk traces back to the 2013 Makefile:
2013-11-22 "static compiling prototype"
Introduced sys.dll link command:
-L$(BUILD)/$(JL_PRIVATE_LIBDIR) -L$(BUILD)/$(JL_LIBDIR) -o $@ $< \
-lgrisu -lrandom -lgmp -lpcre -lmpfr $(LIBBLAS) $(LIBLAPACK) ...
Purpose of -L$(build_private_libdir): find Julia's own build of
third-party libraries
2013-11-29 "lookup all ccall/cglobal values at runtime"
Transitioned to runtime loading; removed all explicit -l libraries
(-lgrisu, -lgmp, etc.)
Added: -lssp (Windows)
But -L$(build_private_libdir) was forgotten, becoming a dead parameter
2014-03-29 "assume libssp-0.dll is needed. fix #6153"
Changed -lssp to unconditional linking on Windows
2024+ CSL began placing mingw-w64 CRT import libraries in private_libdir
The dead parameter suddenly had a side effect → #56840
After November 29, 2013, the sys.dll link command no longer needed any file from build_private_libdir:
-ljulia-internal/-ljulia: located in$(build_libdir), not$(build_private_libdir)-lssp: present in GCC's default search paths (the system GCC shipslibssp.dll.a)- Implicit
-lmingw32 -lmsvcrt -lgcc -lgcc_s: resolved from GCC's default search paths dllcrt2.o/crtbegin.o/crtend.o: injected by GCC spec via absolute paths, never affected by-L
This parameter had no function for 12 consecutive years.
| sysimage.mk (GCC) | linking.jl (LLD) | |
|---|---|---|
| When | Julia build time | User runtime (pkgimage compilation) |
| Linker | Build machine's GCC | Julia's bundled LLD |
| CRT should come from | System compiler (must match spec-injected dllcrt2.o) | CSL fixed version (reproducible, CI-tested) |
| File retrieval | GCC spec + -L search |
_find_static() + explicit -L |
linking.jl (LLD, pkgimage linking):
_find_static("dllcrt2.o") → private_libdir/dllcrt2.o ← CSL
-lmsvcrt → private_libdir/libmsvcrt.a ← CSL
-lgcc_s → private_libdir/libgcc_s.a ← CSL
crtbegin.o → private_libdir/crtbegin.o ← CSL
The LLD path does not go through GCC spec; all CRT objects are explicitly fetched from private_libdir. They must be version-consistent → all provided by CSL. This is the only valid reason CSL includes these files, and why they must remain.
At build time, GCC already injects the system versions of dllcrt2.o, crtbegin.o, and crtend.o via its spec. If the implicit -l libraries (libmsvcrt, libgcc, etc.) are also resolved by the system GCC, they will be version-consistent with the injected startup objects. Any extra -L may introduce inconsistent versions.
sysimage.mk: Remove -L$(build_private_libdir) from the link command.
- -L$(build_private_libdir) -L$(build_libdir) -L$(build_shlibdir)
+ -L$(build_libdir) -L$(build_shlibdir)After the fix:
sysimage.mk (GCC) linking.jl (LLD)
───────────────── ─────────────────
dllcrt2.o System (spec-injected) CSL (_find_static)
crtbegin/crtend System (spec-injected) CSL (_find_static)
libmsvcrt.a System (GCC default paths) CSL (-L private_libdir)
libmingw32.a System (GCC default paths) CSL (-L private_libdir)
libgcc.a System (GCC default paths) CSL (-L private_libdir)
libgcc_s.a System (GCC default paths) CSL (-L private_libdir)
libssp.dll.a System (GCC default paths) CSL (-L private_libdir)
libstdc++.dll — CSL
───────────────── ─────────────────
All system → consistent ✅ All CSL → consistent ✅
deps/csl.mk: CSL continues to provide all CRT files to$(build_private_libdir)for use bylinking.jlbase/linking.jl: LLD pkgimage link path is completely unaffectedMake.inc: No changes needed- Cross-compilation: Unaffected — when cross-compiling, the GCC spec injects dllcrt2.o from the cross-compiler's sysroot, and implicit
-llibraries are resolved by the same cross-compiler, so versions are naturally consistent
| Date | Commit | Description |
|---|---|---|
| 2013-11-22 | 0a0ec3147a |
Introduced sys.dll linking; -L$(build_private_libdir) used to find third-party libraries |
| 2013-11-29 | 67a11cb760 |
Removed explicit -l library references; -L left behind |
| 2014-03-29 | a23effcd6b |
Made -lssp unconditional on Windows |
| 2024–2025 | multiple | CSL began placing mingw-w64 CRT files in private_libdir |
| 2026-06 | d374574f |
Removed -L$(build_private_libdir), fixes #56840 |