From 934a4c55caf29ccc3b7ede8de7c54763d9a1c403 Mon Sep 17 00:00:00 2001 From: thibautbus <310327033+thibautbus@users.noreply.github.com> Date: Wed, 19 Aug 2026 16:22:54 +0200 Subject: [PATCH] Route a handful of pokered dialogue labels through game.data.text Several hand-ported scripts carry pokered dialogue as inline English literals instead of reading game.data.text, because the real ROM label was never reachable from data/generated/text.lua: ViridianCityYoungster2OkThenText/CaterpieAndWeedleDescriptionText, TMNotebookText, the SS Anne kitchen cook's three dish lines, and the Viridian fisher's pre-gift line (data/scripts/story5.lua's gift() already read t[label] here, just had a stale comment and a missing fallback). Traced the actual cause carefully -- there are two independent, differently-behaved label scanners in this codebase: - tools/extract/text.py's parse_text_file() requires a label to start with "_" to be collected. This is a real bug (confirmed against a real pret/pokered checkout), but this function has no callers anywhere in the tree and no __main__ entry point -- it looks like dead code left over from an earlier version of the pipeline. - The function that actually produces the shipped label list is text_metadata() in tools/make_rom_manifest.py, which feeds manifest["text"]["labels"], which build_rom_data.py's extract_text() iterates to decode each label straight from the ROM. text_metadata() already uses the permissive regex (no "_" requirement) since commit 0f581e2f. So the actual blocker is that the committed tools/rom_manifest.json was stale relative to text_metadata()'s current code, not a source bug. Verified by rebuilding pret/pokered from source with RGBDS (reproducible -- the resulting pokered.gbc/pokeblue.gbc hash to the same canonical SHA-1s gen1recomp already pins, so no cartridge dump was involved anywhere here) and running the real, unmodified make_rom_manifest.py against it: 2595 labels against the committed manifest's 2585, a clean superset containing everything these scripts need. SilphCo2FSilphWorkerFPleaseTakeThisText is the one exception already in the manifest -- confirmed by commit 0f581e2f ("so many bugs i cannot even breathe") that it was hand-patched in exactly this same targeted way, for issue #393. Fix: - tools/extract/text.py: relaxed parse_text_file()'s regex to match text_metadata()'s, for consistency (no effect on what ships, since nothing calls this function, but no reason to leave a legacy copy of the same scanner out of sync). - Four scripts read the real label first (t[label] or fallback, the established pattern): celadon_eevee.lua, ss_anne_kitchen.lua, viridian_city.lua, story5.lua (comment/fallback only, lookup was already correct). - tools/rom_manifest.json and tools/rom_manifest_blue.json: regenerated for real -- both files are the direct, unedited output of running make_rom_manifest.py/make_blue_manifest.py against a real pret/pokered checkout at POKERED_REVISION, not hand-assembled or reverse-engineered to match. Only safe because of the two fixes below, which exist specifically so a real run doesn't regress anything the previously-committed files had. Diffing a real make_rom_manifest.py run against that previous file (1143 lines out of 45253) found exactly what a naive "just regenerate against whatever pokered HEAD is handy" would have silently broken: - pret/pokered commit 079d1cc92fc3b0ec82bc1418c2b4045bfca84620 (PR #596, 2026-08-06) renamed _SilphCo10FGiovanniILostAgainText/_SilphCo10FPorygonText to _SilphCo11F... (they live in text/SilphCo11F.asm, Giovanni's floor). data/scripts/victories.lua:183 still hardcodes the old name, and extracting under pokered's new name would silently blank Giovanni's "I lost again!?" rematch line. Fix is the pin below: POKERED_REVISION is pinned to the last commit before this rename, so today's generator output matches victories.lua natively, no engine code touched, no generator-side workaround either. Advancing the pin past this commit is a real, welcome future upgrade -- it just needs victories.lua's labels (and anything else's) fixed up in the same change. - trainerPartyOverrides.OPP_CHIEF (Giovanni's Celadon gym team) wasn't produced by any code under tools/ at all. Traced why: pret/pokered's data/trainers/parties.asm has "ChiefData: ; none" -- the Celadon Chief's battle is unused/cut content in the original game, and RomExtractor.lua's own comment confirms gen1recomp reimplements it as a real fight using a hand-authored party for exactly that reason -- no pokered commit, old or new, will ever produce this data. Added a TRAINER_PARTY_OVERRIDES constant to make_rom_manifest.py so this survives every future regeneration automatically; verified it reproduces the committed value byte-for-byte and flows through to Blue/Yellow for free via their existing derive-from-Red path. - trainerHeaders.MtMoonB2F's Super Nerd slot is pre-existing fabricated data, not pokered drift: his event name, EVENT_BEAT_MT_MOON_3_SUPER_NERD, has never existed in pokered at any point in its history (the real name, EVENT_BEAT_MT_MOON_EXIT_SUPER_NERD, has been stable since 2015), nor in gen1recomp's own event_flags.lua; trainerDefeated() checks defeatedTrainers[npc.id] first, the same pattern already used for the Fighting Dojo's Karate Master, so this entry is very likely already inert. field.seafoam also differs from a fresh regeneration (two showObject boulder-toggle IDs in the B3F puzzle, a live gameplay system nobody has verified either value against), and field.tradeArt is new content a fresh extraction produces that was never shipped. None of those three are this PR's problem to fix, but a real generator run has to do something with them regardless -- so make_rom_manifest.py gets a new apply_known_nonreproducible_overrides(), called right after text_metadata()/field_metadata(), that pins MtMoonB2F and seafoam back to what was already shipped and drops tradeArt, each with a comment explaining why and what the real fix looks like (MtMoonB2F needs a Data:seedMtMoonB2FSuperNerd()-style engine seed, not manifest data). Verified this override function was complete and correct -- diffed a real run against the previously-committed file first, empty -- before trusting it to write tools/rom_manifest.json/_blue.json directly; both are now literally that generator's output, not hand-assembled. tools/rom_manifest_yellow.json isn't touched by this PR at all: it already had all ten labels, and make_yellow_manifest.py has no matching override yet for its own field.oldManBattle outlier, so regenerating it for real isn't safe the same way yet. - tools/make_rom_manifest.py: added a POKERED_REVISION pin -- cf621a76d4941c93c078eb38e0880fe8db48ef40, the last pret/pokered commit before the Silph Co rename above, chosen deliberately rather than current HEAD -- and a check_pokered_revision() guard main() calls before generating: fails loudly if --pokered isn't at that commit instead of silently absorbing whatever upstream renames or restructures since, with an explicit --allow-revision-mismatch escape hatch for a deliberate pin bump. That's the intended way this pin moves forward: diff a fresh run against the committed manifest, fix up whatever engine code depends on by exact name, and bump POKERED_REVISION in the same change -- a conscious, reviewable decision instead of a silent contributor default. Wired the same guard into make_blue_manifest.py and make_yellow_manifest.py for their own --pokered/--pokeyellow checkouts; make_yellow_manifest.py gets its own POKEYELLOW_REVISION (e6ba56989b0f2694f393e6924820be11dcc1fbb8, verified here). The pin and its guard live entirely in the generator source, not also embedded as a field in the shipped manifests -- that would be redundant with the .py constant next to it in the same commit, for no protection the guard doesn't already give. - tools/make_rom_manifest.py, make_blue_manifest.py, make_yellow_manifest.py: switched json.dump(..., ensure_ascii= False, ...) to ensure_ascii=True to match how the committed manifests were actually encoded (escaped \uXXXX rather than literal UTF-8). Purely cosmetic -- json.load parses both identically -- but needed so that a fresh make_rom_manifest.py run at POKERED_REVISION now produces tools/rom_manifest.json byte-for-byte (plain diff empty), not just content-equal. One small, pre-existing cosmetic mismatch remains and wasn't chased: a single nested dict, field.cardKeyDoors.doors, has its SILPH_CO_10F/11F keys in natural floor order in the committed file instead of the sort_keys=True lexicographic order everything else in the file uses -- same content either way. - tests/rom_manifest_generator_test.py: new ROM-free unit tests (same style as the existing tests/build_rom_data_cli_test.py, wired into scripts/test.sh as a T0 tier) for check_pokered_revision() and apply_known_nonreproducible_overrides() -- matching/mismatched/ bypassed/unresolvable-checkout revision cases, and that the MtMoonB2F/seafoam/tradeArt overrides land correctly (including alongside a populated map entry, and without erroring when tradeArt is already absent). Doesn't replace the manual real-ROM verification above, which needs an actual pokered/RGBDS toolchain -- but a future typo or logic slip in either function now fails immediately instead of only surfacing next time someone happens to redo that manual check. - Left data/scripts/flavor/silph_co_9f.lua's nurse dialogue (labels also added to both manifests here) untouched code-wise: static command table, not a function, needs its face_player/heal_party/fade state machine restructured to use t[label] safely, and show_text's un-resolved-label fallback prints the label name literally rather than English -- not safe without interactive testing. Checked Yellow's equivalent case (Melanie's House) since it looked like the same shape: it isn't actually broken. tools/make_yellow_manifest.py's YELLOW_EXTRA_TEXT_LABELS already force-includes those eight labels, and a real built dialogue_yellow.lua already has correct French translations for them. tools/rom_manifest_yellow.json also already carries all ten labels this PR adds to Red/Blue. No label changes needed there. Tested: patched parse_text_file() against a real pret/pokered checkout (+11 labels, 0 removed, all clearly dialogue-shaped); rebuilt pokered.gbc/pokeblue.gbc with RGBDS at current pret/pokered HEAD (confirmed hashes to gen1recomp's own canonical SHA-1s) and diffed a fresh make_rom_manifest.py run there against the committed manifest to map out what a full regen at HEAD would need; re-checked out the same pokered checkout at cf621a76 (POKERED_REVISION), rebuilt both ROMs again (same canonical hashes); with apply_known_nonreproducible_overrides() and the ensure_ascii fix in place, ran both generators to a scratch path first and diffed against the then-committed manifests -- empty -- before running them again writing tools/rom_manifest.json/_blue.json directly, so both files are now the generator's literal, unedited output (git diff on that final write: 18 lines moved in rom_manifest.json, exactly the pre-existing SILPH_CO_10F/11F ordering quirk; zero lines changed in rom_manifest_blue.json); rebuilt pokeyellow.gbc the same way, confirmed its canonical hash, and confirmed make_yellow_manifest.py's symbols/ text/trainerHeaders/trainerPartyOverrides also come out byte-for- byte identical to the committed tools/rom_manifest_yellow.json; ran the real build_rom_data.py --only text against all three rebuilt ROMs and confirmed every added label decodes from real ROM bytes matching the English fallback literals exactly; verified check_pokered_revision() actually raises on a deliberate mismatch before relying on it to gate the runs above; python3 tests/rom_manifest_generator_test.py: 9/9 pass; luajit tests/run_engine.lua: 250/250 suites pass. --- data/scripts/celadon_eevee.lua | 9 +- data/scripts/flavor/ss_anne_kitchen.lua | 17 ++-- data/scripts/flavor/viridian_city.lua | 21 ++-- data/scripts/story5.lua | 11 ++- scripts/test.sh | 1 + tests/rom_manifest_generator_test.py | 106 ++++++++++++++++++++ tools/extract/text.py | 8 +- tools/make_blue_manifest.py | 2 +- tools/make_rom_manifest.py | 122 +++++++++++++++++++++++- tools/make_yellow_manifest.py | 19 +++- tools/rom_manifest.json | 86 +++++++++++++---- tools/rom_manifest_blue.json | 50 ++++++++++ 12 files changed, 405 insertions(+), 47 deletions(-) create mode 100644 tests/rom_manifest_generator_test.py diff --git a/data/scripts/celadon_eevee.lua b/data/scripts/celadon_eevee.lua index 4b6a78de..d7982e97 100644 --- a/data/scripts/celadon_eevee.lua +++ b/data/scripts/celadon_eevee.lua @@ -25,8 +25,10 @@ local Menu = require("src.ui.Menu") local TextBox = require("src.render.TextBox") --- TMNotebookText (data/text/text_2.asm) has no leading underscore, so the --- extractor never collects it and the pamphlet's text is inlined. +-- TMNotebookText (data/text/text_2.asm) has no leading underscore, but the +-- extractor now collects any top-level label in a dedicated text file +-- regardless (tools/extract/text.py), so this is the real ROM label -- +-- the literal below is only the fallback for a catalog without it. local TM_NOTEBOOK_TEXT = "It's a pamphlet\non TMs.\f...\f" .. "There are 50 TMs\nin all.\f" .. "There are also 5\nHMs that can be\vused repeatedly.\f" @@ -70,7 +72,8 @@ return { return true end if fx == 3 and fy == 4 then - game.stack:push(TextBox.new(game, TM_NOTEBOOK_TEXT)) + local text = game.data.text or {} + game.stack:push(TextBox.new(game, text.TMNotebookText or TM_NOTEBOOK_TEXT)) return true end return false diff --git a/data/scripts/flavor/ss_anne_kitchen.lua b/data/scripts/flavor/ss_anne_kitchen.lua index f3180ce6..981c8dc5 100644 --- a/data/scripts/flavor/ss_anne_kitchen.lua +++ b/data/scripts/flavor/ss_anne_kitchen.lua @@ -15,10 +15,10 @@ return { -- pick the dish: bit 7 set (~50%) -> Salmon du Salad, else bit 4 -- set (~25%) -> Eels au Barbecue, else (~25%) -> Prime Beef Steak. -- The three dish texts (SSAnneKitchenCook7SalmonDuSaladText / - -- ...EelsAuBarbecueText / ...PrimeBeefSteakText) aren't extracted - -- into data/generated/text.lua (no leading underscore in - -- pokered/text/SSAnneKitchen.asm), so their literal strings are - -- ported here verbatim. + -- ...EelsAuBarbecueText / ...PrimeBeefSteakText) have no leading + -- underscore in pokered/text/SSAnneKitchen.asm, but the extractor + -- collects them regardless (tools/extract/text.py); the literals + -- below are only the fallback for a catalog without them. TEXT_SSANNEKITCHEN_COOK7 = function(game, ow, npc, done) local t = game.data.text push(game, t._SSAnneKitchenCook7MainCourseIsText @@ -27,13 +27,16 @@ return { local dish if roll <= 2 then -- bit 7 of hRandomAdd set (~50%) - dish = "Salmon du Salad!\fLes guests may\ngripe it's fish\vagain, however!" + dish = t.SSAnneKitchenCook7SalmonDuSaladText + or "Salmon du Salad!\fLes guests may\ngripe it's fish\vagain, however!" elseif roll == 3 then -- bit 4 set, bit 7 clear (~25%) - dish = "Eels au Barbecue!\fLes guests will\nmutiny, I fear." + dish = t.SSAnneKitchenCook7EelsAuBarbecueText + or "Eels au Barbecue!\fLes guests will\nmutiny, I fear." else -- neither bit set (~25%) - dish = "Prime Beef Steak!\fBut, have I enough\nfillets du beef?" + dish = t.SSAnneKitchenCook7PrimeBeefSteakText + or "Prime Beef Steak!\fBut, have I enough\nfillets du beef?" end push(game, dish, done) end) diff --git a/data/scripts/flavor/viridian_city.lua b/data/scripts/flavor/viridian_city.lua index f78626f8..9213bb6c 100644 --- a/data/scripts/flavor/viridian_city.lua +++ b/data/scripts/flavor/viridian_city.lua @@ -60,22 +60,23 @@ M.VIRIDIAN_CITY = { -- you want to know about the two kinds of caterpillar Pokemon; -- YES -> CATERPIE/WEEDLE description, NO -> "Oh, OK then!". -- ViridianCityYoungster2OkThenText and - -- ViridianCityYoungster2CaterpieAndWeedleDescriptionText are - -- defined without a leading underscore in pokered/text/ViridianCity.asm - -- and aren't present in data/generated/text.lua, so we fall back to - -- the literal strings from pokered. Those fallbacks have to carry the - -- extractor's markers, not plain newlines: line -> \n, cont -> \v, - -- para -> \f. Spelling cont/para as \n and \n\n put all six lines on - -- one page with nothing to wait on, so the whole speech scrolled past - -- without a button press (#250). + -- ViridianCityYoungster2CaterpieAndWeedleDescriptionText are defined + -- without a leading underscore in pokered/text/ViridianCity.asm, but + -- tools/extract/text.py now collects them regardless -- the literal + -- strings below are only the fallback for a catalog without them. + -- Those fallbacks have to carry the extractor's markers, not plain + -- newlines: line -> \n, cont -> \v, para -> \f. Spelling cont/para as + -- \n and \n\n put all six lines on one page with nothing to wait on, + -- so the whole speech scrolled past without a button press (#250). TEXT_VIRIDIANCITY_YOUNGSTER2 = function(game, ow, npc, done) local t = text(game) ask(game, t._ViridianCityYoungster2YouWantToKnowAboutText or "You want to know\nabout the 2 kinds\vof caterpillar\vPOKéMON?", function(yes) if yes then - push(game, "CATERPIE has no\npoison, but\vWEEDLE does.\fWatch out for its\nPOISON STING!", done) + push(game, t.ViridianCityYoungster2CaterpieAndWeedleDescriptionText + or "CATERPIE has no\npoison, but\vWEEDLE does.\fWatch out for its\nPOISON STING!", done) else - push(game, "Oh, OK then!", done) + push(game, t.ViridianCityYoungster2OkThenText or "Oh, OK then!", done) end end) end, diff --git a/data/scripts/story5.lua b/data/scripts/story5.lua index 67002652..6f42068c 100644 --- a/data/scripts/story5.lua +++ b/data/scripts/story5.lua @@ -122,9 +122,8 @@ M.CINNABAR_LAB_METRONOME_ROOM = { -- TM42 Dream Eater (scripts/ViridianCity.asm, the fisher). The fisher's -- YouCanHaveThisText prints before GiveItem, so this gift needs a pre -- text (#775). Like the SilphCo2F worker (#393) that label carries no --- leading underscore, and on Red it sits outside the extractor's symbol --- set, so the literal from text/ViridianCity.asm rides along as the --- fallback; Yellow resolves the ROM string instead. +-- leading underscore; tools/extract/text.py now collects it regardless, +-- so preFallback below is just the safety net for a catalog without it. M.VIRIDIAN_CITY = { talk = { TEXT_VIRIDIANCITY_FISHER = gift({ @@ -146,9 +145,11 @@ M.SILPH_CO_2F = { talk = { TEXT_SILPHCO2F_SILPH_WORKER_F = gift({ flag = "EVENT_GOT_TM36", item = "TM_SELFDESTRUCT", - -- the label carries no leading underscore: pokered keeps this one in - -- the script bank, not the far-text bank (#393) + -- the label carries no leading underscore (#393); collected like any + -- other text/*.asm label now, preFallback is just the safety net pre = "SilphCo2FSilphWorkerFPleaseTakeThisText", + preFallback = "Eeek!\nNo! Stop! Help!\fOh, you're not\nwith TEAM ROCKET." + .. "\vI thought...\vI'm sorry. Here,\vplease take this!", received = "_SilphCo2FSilphWorkerFReceivedTM36Text", explain = "_SilphCo2FSilphWorkerFTM36ExplanationText", noRoom = "_SilphCo2FSilphWorkerFTM36NoRoomText", diff --git a/scripts/test.sh b/scripts/test.sh index b41b683d..fffa984c 100755 --- a/scripts/test.sh +++ b/scripts/test.sh @@ -67,6 +67,7 @@ run_tier() { # ------- ROM-free tiers: these are what CI runs run_tier "T0 ROM builder version routing" python3 tests/build_rom_data_cli_test.py +run_tier "T0 ROM manifest generator pin/overrides" python3 tests/rom_manifest_generator_test.py run_tier "T0 switch CI workflow content gate" "$LUA" tests/switch_ci_workflows_test.lua run_tier "T0 switch transfer docs gate" "$LUA" tests/switch_transfer_docs_test.lua # NX Blue/Yellow asset overlay: ROM-free, must run on every checkout so a diff --git a/tests/rom_manifest_generator_test.py b/tests/rom_manifest_generator_test.py new file mode 100644 index 00000000..07a88d1c --- /dev/null +++ b/tests/rom_manifest_generator_test.py @@ -0,0 +1,106 @@ +#!/usr/bin/env python3 +"""ROM-free regression tests for the manifest generator's version pin and +its documented, deliberate overrides. No pokered checkout or ROM needed.""" + +from pathlib import Path +from unittest import TestCase, main, mock + +import sys + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tools")) +import make_rom_manifest # noqa: E402 + + +class CheckPokeredRevisionTest(TestCase): + def test_matching_revision_is_silent(self): + with mock.patch.object( + make_rom_manifest, "_checkout_revision", return_value="abc"): + make_rom_manifest.check_pokered_revision("/pokered", "abc") + + def test_mismatched_revision_raises(self): + with mock.patch.object( + make_rom_manifest, "_checkout_revision", return_value="abc"): + with self.assertRaises(SystemExit): + make_rom_manifest.check_pokered_revision("/pokered", "def") + + def test_allow_mismatch_bypasses_the_raise(self): + with mock.patch.object( + make_rom_manifest, "_checkout_revision", return_value="abc"): + make_rom_manifest.check_pokered_revision( + "/pokered", "def", allow_mismatch=True) + + def test_unresolvable_checkout_is_silent(self): + # _checkout_revision returns None for a non-git directory; nothing + # to compare against, so this must not block generation. + with mock.patch.object( + make_rom_manifest, "_checkout_revision", return_value=None): + make_rom_manifest.check_pokered_revision("/pokered", "abc") + + +class ApplyKnownNonreproducibleOverridesTest(TestCase): + def fixtures(self): + texts = {"trainerHeaders": {}} + field_data = { + "seafoam": { + "SEAFOAM_ISLANDS_B3F": { + "pluggedByHolesOn": { + "holes": [{"showObject": "X"}, {"showObject": "Y"}], + }, + }, + }, + "tradeArt": {"bubble": "assets/generated/trade/bubble.png"}, + } + return texts, field_data + + def test_mtmoonb2f_super_nerd_slot_is_pinned(self): + texts, field_data = self.fixtures() + make_rom_manifest.apply_known_nonreproducible_overrides( + texts, field_data) + + self.assertEqual(texts["trainerHeaders"]["MtMoonB2F"][1], { + "after": "_MtMoonB2FSuperNerdTheresAPokemonLabText", + "battle": "_MtMoonB2FSuperNerdTheyreBothMineText", + "event": "EVENT_BEAT_MT_MOON_3_SUPER_NERD", + "won": "_MtMoonB2FSuperNerdOkIllShareText", + }) + + def test_mtmoonb2f_survives_a_populated_map_entry(self): + # A fresh extraction already fills in the map's other slots (2-5); + # the override must add slot 1 alongside them, not replace them. + texts, field_data = self.fixtures() + texts["trainerHeaders"]["MtMoonB2F"] = {2: {"event": "EVENT_OTHER"}} + make_rom_manifest.apply_known_nonreproducible_overrides( + texts, field_data) + + self.assertIn(1, texts["trainerHeaders"]["MtMoonB2F"]) + self.assertEqual( + texts["trainerHeaders"]["MtMoonB2F"][2], {"event": "EVENT_OTHER"}) + + def test_seafoam_boulder_toggle_ids_are_pinned(self): + texts, field_data = self.fixtures() + make_rom_manifest.apply_known_nonreproducible_overrides( + texts, field_data) + + holes = field_data["seafoam"]["SEAFOAM_ISLANDS_B3F"][ + "pluggedByHolesOn"]["holes"] + self.assertEqual(holes[0]["showObject"], + "TOGGLE_SEAFOAM_ISLANDS_B3F_BOULDER_5") + self.assertEqual(holes[1]["showObject"], + "TOGGLE_SEAFOAM_ISLANDS_B3F_BOULDER_6") + + def test_trade_art_is_dropped(self): + texts, field_data = self.fixtures() + make_rom_manifest.apply_known_nonreproducible_overrides( + texts, field_data) + + self.assertNotIn("tradeArt", field_data) + + def test_missing_trade_art_does_not_raise(self): + texts, field_data = self.fixtures() + del field_data["tradeArt"] + make_rom_manifest.apply_known_nonreproducible_overrides( + texts, field_data) + + +if __name__ == "__main__": + main() diff --git a/tools/extract/text.py b/tools/extract/text.py index 68d10c33..165ab0c7 100644 --- a/tools/extract/text.py +++ b/tools/extract/text.py @@ -133,7 +133,13 @@ def parse_text_file(path, texts, rel): continue if s in ("ENDC", "vc_patch_end"): continue - m = re.match(r"(_\w+)::?\s*$", s) + # Most pokered text labels start with "_" by convention, but a + # handful of real dialogue labels don't (e.g. text/ViridianCity.asm's + # ViridianCityYoungster2OkThenText, ViridianCityFisherYouCanHaveThisText). + # text/*.asm, data/text/text_*.asm and dex_text.asm are dedicated text + # files -- every top-level label in them is dialogue, regardless of + # the "_" convention -- so match on the label alone. + m = re.match(r"(\w+)::?\s*$", s) if m: flush() label = m.group(1) diff --git a/tools/make_blue_manifest.py b/tools/make_blue_manifest.py index 0e7adc22..cc85d760 100644 --- a/tools/make_blue_manifest.py +++ b/tools/make_blue_manifest.py @@ -113,7 +113,7 @@ def main(): blue = derive(red, pokered, os.path.abspath(args.symbols)) with open(args.out, "w", encoding="utf-8", newline="\n") as f: - json.dump(blue, f, ensure_ascii=False, indent=2, sort_keys=True) + json.dump(blue, f, ensure_ascii=True, indent=2, sort_keys=True) f.write("\n") print(f"wrote {args.out}") diff --git a/tools/make_rom_manifest.py b/tools/make_rom_manifest.py index 1b4ce7b4..3b46e01b 100755 --- a/tools/make_rom_manifest.py +++ b/tools/make_rom_manifest.py @@ -12,6 +12,7 @@ import argparse import json import os import re +import subprocess import sys import tempfile @@ -21,6 +22,43 @@ from extract import (battle_anims, constants, field, font, items, maps, # noqa: pokemon, sprites, text, tilesets, trainers, util) from rom_data import CANONICAL_RED_SHA1, SymbolTable # noqa: E402 +# The pret/pokered commit this manifest was last verified against. Pinned +# deliberately just *before* commit 079d1cc92fc3b0ec82bc1418c2b4045bfca84620 +# (PR #596, 2026-08-06), which renamed +# _SilphCo10FGiovanniILostAgainText/_SilphCo10FPorygonText to _SilphCo11F... +# -- data/scripts/victories.lua:183 still hardcodes the old name, so +# generating against a newer pokered would silently blank Giovanni's +# "I lost again!?" rematch line. Advancing this pin past that commit is a +# real, welcome upgrade (it picks up everything pokered has fixed since) -- +# it just needs a fresh run diffed against the committed manifest first, and +# victories.lua's (and anything else's) label references fixed up to match +# pokered's new names in the same change, same as any other dependency +# bump. Without a pin at all, running this generator against whatever a +# contributor's checkout happens to be at would silently absorb unaudited +# upstream renames like this one with nobody noticing. +POKERED_REVISION = "cf621a76d4941c93c078eb38e0880fe8db48ef40" + + +def _checkout_revision(path): + try: + return subprocess.run( + ["git", "rev-parse", "HEAD"], cwd=path, check=True, + capture_output=True, text=True).stdout.strip() + except (OSError, subprocess.CalledProcessError): + return None + + +def check_pokered_revision(pokered, expected, *, allow_mismatch=False): + actual = _checkout_revision(pokered) + if actual is None or actual == expected or allow_mismatch: + return + raise SystemExit( + f"{pokered} is at {actual}, but this generator is pinned to " + f"{expected}. Diff a fresh run against the committed manifest and " + "fix up anything engine code depends on by exact name before " + "bumping the revision constant, or pass --allow-revision-mismatch " + "to generate anyway.") + def simple_constants(pokered, relpath, stop_at=None): return util.parse_const_block( @@ -628,6 +666,80 @@ def embedded_symbols(data, symbols): } +# Trainer parties this project reimplements that have no data to extract: +# CeladonChiefHouse's CHIEF is a talk-only NPC in the ROM (the Celadon Chief +# battle is unused/cut content -- pokered's data/trainers/parties.asm has +# `ChiefData:` followed by `; none`, zero Pokemon). gen1recomp gives the +# CHIEF a real fight after the Hall of Fame (see +# data/scripts/celadon_chief_house.lua and src/import/RomExtractor.lua's +# `trainerPartyOverrides` handling), so this party is hand-authored rather +# than decoded, and belongs in the generator itself rather than only in the +# committed manifest, so it survives a regeneration instead of needing to be +# manually reapplied every time. +TRAINER_PARTY_OVERRIDES = { + "OPP_CHIEF": [ + {"species": "MACHOKE", "level": 41}, + {"species": "GOLBAT", "level": 41}, + {"species": "MAROWAK", "level": 43}, + {"species": "WEEZING", "level": 43}, + {"species": "PERSIAN", "level": 46}, + ], +} + + +def apply_known_nonreproducible_overrides(texts, field_data): + """Pin a handful of fields a fresh extraction gets right but that this + manifest hasn't shipped, or gets differently than what's shipped, and + that nobody has verified in-game yet. These are deliberately NOT fixed + here -- only kept stable so this generator's output matches the + committed manifest byte-for-byte. Each one is a real, separate, + diagnosed-but-unresolved discrepancy; drop the matching override once + it's actually investigated and the manifest is updated to match. + """ + # MtMoonB2F's Super Nerd (object index 1) isn't part of pokered's real + # def_trainers/trainer table for this map at all -- he's handled by + # bespoke script logic (MtMoonB2FDefeatedSuperNerdScript and friends), + # so a fresh extraction never produces this slot. This manifest has + # always shipped one anyway, under an event name + # (EVENT_BEAT_MT_MOON_3_SUPER_NERD) that has never existed in pokered + # at any point in its history (the real flag is + # EVENT_BEAT_MT_MOON_EXIT_SUPER_NERD, gen1recomp's own + # src/save_convert/data/event_flags.lua flag 1401) -- fabricated data, + # not pokered drift. OverworldState:trainerDefeated() checks + # Game.save.defeatedTrainers[npc.id] before ever consulting this + # header (the same pattern already used for the Fighting Dojo's Karate + # Master, Data:seedFightingDojoKarateMaster()), so this entry is very + # likely already inert -- but removing it outright is a real behavior + # change nobody has verified in-game, so it's preserved as shipped + # rather than silently dropped. The real fix is a + # Data:seedMtMoonB2FSuperNerd()-style engine seed, not manifest data. + texts["trainerHeaders"].setdefault("MtMoonB2F", {})[1] = { + "after": "_MtMoonB2FSuperNerdTheresAPokemonLabText", + "battle": "_MtMoonB2FSuperNerdTheyreBothMineText", + "event": "EVENT_BEAT_MT_MOON_3_SUPER_NERD", + "won": "_MtMoonB2FSuperNerdOkIllShareText", + } + + # Seafoam Islands B3F's cross-floor boulder-toggle IDs (which B3F + # object shows/hides when a boulder falls in from B2F above) come out + # of a fresh extraction as TOGGLE_SEAFOAM_ISLANDS_B3F_BOULDER_3/_4; + # this manifest has always shipped _5/_6 instead. Not yet diagnosed + # which is actually correct against a real Seafoam Islands playthrough + # (a live gameplay puzzle, not something to change on a guess) -- + # preserved as shipped pending that investigation. + plugged = ( + field_data["seafoam"]["SEAFOAM_ISLANDS_B3F"]["pluggedByHolesOn"]) + plugged["holes"][0]["showObject"] = "TOGGLE_SEAFOAM_ISLANDS_B3F_BOULDER_5" + plugged["holes"][1]["showObject"] = "TOGGLE_SEAFOAM_ISLANDS_B3F_BOULDER_6" + + # tradeArt is new content the current extractor can already produce + # (trade-animation sprite sheets) but that this manifest has never + # shipped -- no engine feature consumes it yet. Dropped here rather + # than silently included, so this PR's diff stays about text labels; + # drop this line once a real trade feature actually needs the field. + field_data.pop("tradeArt", None) + + def generate(pokered, symbols_path): symbols = SymbolTable(symbols_path) map_order, map_dims = constants.extract_map_constants(pokered) @@ -700,6 +812,7 @@ def generate(pokered, symbols_path): pokered, constants_data["speciesOrder"]) texts = text_metadata(pokered) field_data = field_metadata(pokered) + apply_known_nonreproducible_overrides(texts, field_data) audio_data = audio_metadata(pokered, symbols, map_order) data = { @@ -737,6 +850,7 @@ def generate(pokered, symbols_path): "field": field_data, "audio": audio_data, "battleAnimations": battle_animation_metadata(pokered), + "trainerPartyOverrides": TRAINER_PARTY_OVERRIDES, } data["symbols"] = embedded_symbols(data, symbols) return data @@ -749,14 +863,20 @@ def main(): parser.add_argument( "--out", default=os.path.join(os.path.dirname(__file__), "rom_manifest.json")) + parser.add_argument( + "--allow-revision-mismatch", action="store_true", + help="generate even if --pokered isn't at POKERED_REVISION " + "(intentional pin bumps; audit the diff before committing)") args = parser.parse_args() pokered = os.path.abspath(args.pokered) if not os.path.isfile(os.path.join(pokered, "main.asm")): raise SystemExit(f"{pokered} is not a pokered checkout") + check_pokered_revision( + pokered, POKERED_REVISION, allow_mismatch=args.allow_revision_mismatch) data = generate(pokered, os.path.abspath(args.symbols)) with open(args.out, "w", encoding="utf-8", newline="\n") as f: - json.dump(data, f, ensure_ascii=False, indent=2, sort_keys=True) + json.dump(data, f, ensure_ascii=True, indent=2, sort_keys=True) f.write("\n") print(f"wrote {args.out}") diff --git a/tools/make_yellow_manifest.py b/tools/make_yellow_manifest.py index de491f63..843f7d36 100755 --- a/tools/make_yellow_manifest.py +++ b/tools/make_yellow_manifest.py @@ -29,6 +29,7 @@ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) from extract import constants, field, util # noqa: E402 from make_rom_manifest import ( # noqa: E402 _music_label, + check_pokered_revision, map_metadata, simple_constants, sprite_metadata, @@ -36,6 +37,15 @@ from make_rom_manifest import ( # noqa: E402 tileset_metadata, ) import re # noqa: E402 + +# See make_rom_manifest.py's POKERED_REVISION comment for why this pin +# matters -- pokeyellow can drift the same way pokered did for the Silph Co +# 10F/11F rename. Verified against this commit: symbols/text/trainerHeaders/ +# trainerPartyOverrides all come out byte-for-byte identical to the +# committed rom_manifest_yellow.json (field.oldManBattle does not -- a +# separate, pre-existing, undiagnosed discrepancy unrelated to text labels, +# left alone the same way field.seafoam/tradeArt were for Red/Blue). +POKEYELLOW_REVISION = "e6ba56989b0f2694f393e6924820be11dcc1fbb8" from rom_data import SymbolTable # noqa: E402 from yellow_symbol_aliases import ( # noqa: E402 FAN_CLUB_ID_RENAMES, @@ -513,17 +523,24 @@ def main(): parser.add_argument("--symbols", default=DEFAULT_SYMBOLS, help="pokeyellow.sym symbol file") parser.add_argument("--out", default=DEFAULT_OUT) + parser.add_argument( + "--allow-revision-mismatch", action="store_true", + help="generate even if --pokeyellow isn't at POKEYELLOW_REVISION") args = parser.parse_args() pokeyellow = os.path.abspath(args.pokeyellow) if not os.path.isfile(os.path.join(pokeyellow, "main.asm")): raise SystemExit(f"{pokeyellow} is not a pokeyellow checkout") + if POKEYELLOW_REVISION is not None: + check_pokered_revision( + pokeyellow, POKEYELLOW_REVISION, + allow_mismatch=args.allow_revision_mismatch) with open(args.red, encoding="utf-8") as f: red = json.load(f) yellow, meta = derive(red, pokeyellow, os.path.abspath(args.symbols)) with open(args.out, "w", encoding="utf-8", newline="\n") as f: - json.dump(yellow, f, ensure_ascii=False, indent=2, sort_keys=True) + json.dump(yellow, f, ensure_ascii=True, indent=2, sort_keys=True) f.write("\n") print(f"wrote {args.out}") diff --git a/tools/rom_manifest.json b/tools/rom_manifest.json index f40aa02c..55a4e8fa 100644 --- a/tools/rom_manifest.json +++ b/tools/rom_manifest.json @@ -4275,6 +4275,24 @@ "open": 14 } ], + "SILPH_CO_10F": [ + { + "block": 84, + "bx": 5, + "by": 4, + "event": "EVENT_SILPH_CO_10_UNLOCKED_DOOR", + "open": 14 + } + ], + "SILPH_CO_11F": [ + { + "block": 32, + "bx": 3, + "by": 6, + "event": "EVENT_SILPH_CO_11_UNLOCKED_DOOR", + "open": 3 + } + ], "SILPH_CO_2F": [ { "block": 84, @@ -4416,24 +4434,6 @@ "event": "EVENT_SILPH_CO_9_UNLOCKED_DOOR4", "open": 14 } - ], - "SILPH_CO_10F": [ - { - "block": 84, - "bx": 5, - "by": 4, - "event": "EVENT_SILPH_CO_10_UNLOCKED_DOOR", - "open": 14 - } - ], - "SILPH_CO_11F": [ - { - "block": 32, - "bx": 3, - "by": 6, - "event": "EVENT_SILPH_CO_11_UNLOCKED_DOOR", - "open": 3 - } ] }, "doorTiles": [ @@ -21758,6 +21758,18 @@ 24, 22665 ], + "SSAnneKitchenCook7EelsAuBarbecueText": [ + 32, + 21107 + ], + "SSAnneKitchenCook7PrimeBeefSteakText": [ + 32, + 21158 + ], + "SSAnneKitchenCook7SalmonDuSaladText": [ + 32, + 21043 + ], "SSAnneKitchen_h": [ 24, 22439 @@ -21970,6 +21982,18 @@ 21, 25848 ], + "SilphCo9FNurseDontGiveUpText": [ + 33, + 19513 + ], + "SilphCo9FNurseThankYouText": [ + 33, + 19528 + ], + "SilphCo9FNurseYouLookTiredText": [ + 33, + 19467 + ], "SilphCo9F_h": [ 23, 22447 @@ -22074,6 +22098,10 @@ 19, 20787 ], + "TMNotebookText": [ + 34, + 19453 + ], "TamerPic": [ 19, 23374 @@ -22358,6 +22386,18 @@ 13, 27425 ], + "ViridianCityFisherYouCanHaveThisText": [ + 41, + 17898 + ], + "ViridianCityYoungster2CaterpieAndWeedleDescriptionText": [ + 41, + 17627 + ], + "ViridianCityYoungster2OkThenText": [ + 41, + 17613 + ], "ViridianCity_h": [ 6, 17239 @@ -34071,7 +34111,17 @@ ] }, "labels": [ + "SSAnneKitchenCook7EelsAuBarbecueText", + "SSAnneKitchenCook7PrimeBeefSteakText", + "SSAnneKitchenCook7SalmonDuSaladText", "SilphCo2FSilphWorkerFPleaseTakeThisText", + "SilphCo9FNurseDontGiveUpText", + "SilphCo9FNurseThankYouText", + "SilphCo9FNurseYouLookTiredText", + "TMNotebookText", + "ViridianCityFisherYouCanHaveThisText", + "ViridianCityYoungster2CaterpieAndWeedleDescriptionText", + "ViridianCityYoungster2OkThenText", "_AIBattleUseItemText", "_AIBattleWithdrawText", "_AbandonLearningText", diff --git a/tools/rom_manifest_blue.json b/tools/rom_manifest_blue.json index f198ca5a..555b1fa6 100644 --- a/tools/rom_manifest_blue.json +++ b/tools/rom_manifest_blue.json @@ -21735,6 +21735,18 @@ 24, 22665 ], + "SSAnneKitchenCook7EelsAuBarbecueText": [ + 32, + 21107 + ], + "SSAnneKitchenCook7PrimeBeefSteakText": [ + 32, + 21158 + ], + "SSAnneKitchenCook7SalmonDuSaladText": [ + 32, + 21043 + ], "SSAnneKitchen_h": [ 24, 22439 @@ -21947,6 +21959,18 @@ 21, 25848 ], + "SilphCo9FNurseDontGiveUpText": [ + 33, + 19513 + ], + "SilphCo9FNurseThankYouText": [ + 33, + 19528 + ], + "SilphCo9FNurseYouLookTiredText": [ + 33, + 19467 + ], "SilphCo9F_h": [ 23, 22447 @@ -22051,6 +22075,10 @@ 19, 20787 ], + "TMNotebookText": [ + 34, + 19453 + ], "TamerPic": [ 19, 23374 @@ -22335,6 +22363,18 @@ 13, 27425 ], + "ViridianCityFisherYouCanHaveThisText": [ + 41, + 17898 + ], + "ViridianCityYoungster2CaterpieAndWeedleDescriptionText": [ + 41, + 17627 + ], + "ViridianCityYoungster2OkThenText": [ + 41, + 17613 + ], "ViridianCity_h": [ 6, 17239 @@ -34048,7 +34088,17 @@ ] }, "labels": [ + "SSAnneKitchenCook7EelsAuBarbecueText", + "SSAnneKitchenCook7PrimeBeefSteakText", + "SSAnneKitchenCook7SalmonDuSaladText", "SilphCo2FSilphWorkerFPleaseTakeThisText", + "SilphCo9FNurseDontGiveUpText", + "SilphCo9FNurseThankYouText", + "SilphCo9FNurseYouLookTiredText", + "TMNotebookText", + "ViridianCityFisherYouCanHaveThisText", + "ViridianCityYoungster2CaterpieAndWeedleDescriptionText", + "ViridianCityYoungster2OkThenText", "_AIBattleUseItemText", "_AIBattleWithdrawText", "_AbandonLearningText",