mirror of
https://github.com/bryanthaboi/gen1recomp.git
synced 2026-08-12 08:21:02 +02:00
thibautbus
fdb3184c71
subprocess.run(..., capture_output=True, text=True) with no explicit encoding falls back to locale.getpreferredencoding(False) -- the OS default codepage. On Windows that's a legacy single-byte codepage (e.g. cp1252), never UTF-8. When the LuaJIT dump contains a byte with no mapping in that codepage, subprocess's internal _readerthread crashes with an uncaught UnicodeDecodeError in a background thread; the thread dies silently and the caller gets back stdout=None instead of a string, crashing one line later with AttributeError: 'NoneType' object has no attribute 'splitlines'. Concretely, the Yellow-side imported dataset contains: "_ColosseumHeightText" -> "...6’8” tall!" The right double quotation mark (U+201D) encodes in UTF-8 as E2 80 9D; 0x9D has no defined character in cp1252, so decoding as cp1252 fails outright. Verified against the real imported dataset: the Red/Blue-only dump has zero bytes outside cp1252's defined range; the Yellow dump has exactly one, at this row. UTF-8 is the actual encoding these dumps are produced in -- the driver Lua sources are read/written as UTF-8 throughout this file, and LuaJIT writes those source strings' bytes back out verbatim -- so passing encoding="utf-8" explicitly at the three affected call sites (run_loader, check_data_dump, dump_dataset) is a no-op on platforms whose default codepage is already UTF-8 (Linux/macOS) and a correctness fix on Windows. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>