DramaticShapeVoxelMod/assets/docs/buidling_to_voxel/sprite_to_voxel_methodology.md

Sprite → Voxel Methodology

Translating Game Boy ¾-view overworld sprites into 3D voxel models

Worked example: Red's house exterior — PALLET_TOWN blocks 56/57/60/61, 64×48 px (reds_house_voxel.html). The same pipeline applies to any GB building sprite.

---

Core principle

A Game Boy overworld sprite is a fake-3D projection: it packs several different 3D facings into one flat image. Roof tops are drawn as if seen from above, walls as if seen from the front, and sloped surfaces as diagonal silhouettes. Voxelization is therefore not one operation — it is (1) classifying each region of the sprite by which 3D surface it depicts, then (2) applying the matching geometric operation per region.

Two invariants govern everything:

1. Every visible voxel color comes from a sprite pixel wherever the lookup lands inside the silhouette. Synthesis is allowed only for geometry the sprite implies but never paints (undersides, depth extensions, interiors), and synthesized cells must continue the sprite's own periodic patterns and 4-shade palette.
2. The sprite is ground truth, not the tile documentation. The tile-grid doc described 1px ground flanks; the actual sprite has 5px insets on the ground floor and a 3-level wedding-cake silhouette. Always extract from the real pixels and verify any doc claims against the extracted mask.

---

Stage 1 — Extraction

Sample the sprite down to its native resolution (uploads are usually integer upscales; sample the center of each scale×scale block, never bilinear). Quantize to the palette — for GB art expect exactly #ffffff / #aaaaaa / #555555 / #000000. Anything else in the histogram means the sampling grid is misaligned.

Compute the silhouette with a flood fill from the image border that spreads only through light pixels (luminance > 130). This threshold is load-bearing: the black outline (#000) and the dark shading (#555) together form the boundary. A naive "not black" threshold lets the fill eat the #555 shaded flanks and corrupts the silhouette — that bug produced a 398-vs-148 pixel discrepancy on the first pass here.

Output: a JSON of the palette plus one string per row (. = outside the outline, digit = palette index). Dump the mask as ASCII and read it — this is where the building's real structure becomes visible, and it is the input to Stage 2. Tooling: sprite_extract.py.

Stage 2 — Band classification

Segment the sprite rows into horizontal bands and assign each a facing. The cues generalize:

Cue in the mask / pixels	Reads as	3D treatment
Top band, plain fill with full-width course lines	Roof top seen from above	Lay horizontal (flat)
Silhouette tapering at a constant rate (here 2 px per row) with slat/stripe texture	Sloped roof surface; the taper rate is the slope	Stepped diagonal surface, 1 down per (taper rate) out
Band containing window/door frames	Vertical facade	Straight extrusion
Full-width band with a black underline sitting above an inset band	Ledge / awning overhang	Extrusion + protrusion
Dark #555 runs beside a facade under a taper	Shadow on the wall beneath an eave	Leave as wall — the geometry above produces the shadow's meaning
Scattered light shapes on a dark field, bracketed by TWO full-width black rims, shallow band below the lower rim	The inside of an open container seen from above, with contents lying in it	Hollow tray: walls to the rims, floor slab, air between — never an extrusion
Ellipse drawn wider than tall (e.g. 9x5)	A horizontal circle seen from above — a mouth, a lid, a pot rim	Cut face of a round hull; the aspect ratio is the proof of the top view
Arcs above/below a round object's straight flanks, lowest point at the centre column, often a 1px #555 halo outside	The SAME circles seen curving — ground contact and mouth back-edge, i.e. depth, not narrowing	Strip them from the revolve; run the last body row's disc to the floor
A side band shearing sideways as it descends (¾-view)	The projection sliding a receding wall, not the wall's position	Un-project: the wall goes where the plan says

The band table for Red's house, which Blue's house shares verbatim:

Sprite rows	Content	Treatment
0–15, x16–47	Coursed grey panel, black rims, highlight course	Flat horizontal top
0–15, flanks	White/dark slats, 2 px-per-row taper	Sloped ends, 2:1
16–23	Gable wall, 3 windows, slat flank pixels	Extrude; roof solid overwrites the corners
24–31	Awning slab, double black underline	Extrude + 2-voxel ledge front/back
32–47	Ground floor siding, door (own inner pane), 2 windows	Extrude; recess panes

Interpretation rules settled during this build — do not relearn them: a top-facing band must end up level everywhere (no synthesized skirts rising through it); a tapering silhouette on a top band means sloped surfaces in elevation, not chamfered corners in plan; and the slope's eave tips should land where the sprite's taper pixels stop (here: just above the awning band).

Stage 3 — Geometry construction

Vertical bands extrude straight back to depth D (26 here). Front and back layers take the sprite pixel; interior layers take a de-outlined color — if the pixel is black, walk inward up to 3 px for the first non-black color. Without this, the side faces of the model are solid outline-black slabs. With it, sides read as material with black front/back corner edges, which is the correct GB-cartoon look. The back face becomes a mirror of the front for free; that is sprite-pure and acceptable.

Ledges replicate the band's front pixels ±2 voxels in z past the walls. The sprite's own black-underline rows become the visible dark underside.

Recesses delete the front voxel of every non-black pixel inside a window/door rect. Frames stay proud; the identically-colored voxel behind becomes the pane, one voxel deep. Rects with nested frames (the door's inner window) produce layered relief automatically.

The flat top lays the top-facing rows horizontal. The band is shallower than the house (16 rows vs 34+ of depth), so extend it by cycling a mid-row band whose period matches the course rhythm (rows 5–8 here, period 4) — this continues both course lines and slat columns seamlessly. Map the outermost sprite rows to the front/back rims so the black-line/fascia trim survives.

Sloped ends are driven by an elevation profile T(x): flat at YTOP over the plateau, dropping 1 voxel per (taper rate = 2) columns outward to the eave tips. Build a solid of constant vertical thickness (4) following T(x), spanning the full roof depth including overhangs. Build order matters: walls first, then the roof solid overwrites wall voxels it intersects, then trim any wall voxel above T(x) so nothing pokes through the surface. The wall strip left exposed beneath the slope shows the sprite's own #555 shadow pixels — the sprite encodes this geometry.

Outline pass: cells on the roof's outer boundary get black (top layer), grey (second layer), black (below) — reproducing the sprite's black-grey-black fascia — and all interior undersides are dark.

Parameters used here, to tune per building: depth D=26; roof overhang 4 in x and z beyond the walls; slab thickness 4; YTOP=35, derived so the eave tips (after the 10-step drop) land one voxel above the awning band, matching where the sprite's taper ends.

Stage 4 — Color sourcing off the sprite

For any roof cell, map z to a sprite row (roofSy), then look up (x, row). If that lands inside the silhouette, use the pixel. If it lands outside (overhang extensions), continue the sprite's periodic texture: the slat rhythm is period 3 with a per-side phase (left flank white at x % 3 == 1, right at x % 3 == 2 — derive the phases from the actual pixels, and mind negative-modulo semantics in JS). Plain top areas fall back to the mid grey. Ground plane and path are presentation-only and the single place non-palette colors are permitted.

Stage 5 — Verification (non-negotiable)

Every bug in this build was caught by one of these, none by eyeballing alone:

1. Dual implementation parity. Build the identical algorithm in a reference implementation (Python) and in the shipping runtime (JS in the HTML). Diff total voxel count and post-cull shell count — they must match exactly (final build: 58,356 / 14,169).
2. Numeric asserts on intent. Flatness: the set of y-layers above the walls must be exactly the slab layers. Slope: the top-surface profile at mid-depth must read tips → 1-per-2 steps → flat plateau, mirrored. Zero wall voxels with y > T(x). Full wall coverage by the roof footprint.
3. Isometric preview. A ~60-line painter's-algorithm render (sort by (x+z, y), draw top/left/right faces of shell voxels) catches texture and layering mistakes cheaply before touching the runtime.
4. Hidden-face culling. Drop voxels whose 6 neighbors all exist before instancing; render the shell as one InstancedMesh.

Tooling: voxel_build_verify.py (builds, asserts, renders previews).

Repeat checklist

1. Obtain the sprite; sample to native resolution via block centers.
2. Extract palette + silhouette (light-only flood fill, threshold 130); review the ASCII mask — rendered large, not hand-counted.
3. Name the real object first (including whether it is hollow, round or thin — see "Beyond the house"), then segment rows into bands using the Stage-2 cues; write the band table as prose, one line per row range with where each band lands, before writing any geometry code. The correct reading makes the row arithmetic land exactly.
4. Measure taper rates from the mask; derive T(x), YTOP, overhangs, D.
5. Build: extrude verticals (de-outlined interiors) → ledges → recesses → flat top (mid-row cycling) → sloped solids (overwrite, then trim) → outline pass → ground presentation.
6. Verify: parity counts, profile/flatness/poke asserts, iso preview.
7. Ship: embed palette + row strings in the HTML; the builder runs client-side and doubles as the reference implementation of the algorithm.

Applying this in the mod

Because tilesets are shared, the band table can be keyed by tile id rather than by sprite: each id gets a treatment record (extrude / lay-flat / slope, plus ledge, recess, warp flags) in the spirit of mods/DRAMATIC_SHAPE/data/voxel_heights.lua. Pinning blocks 56/57/60/61 with the profile above voxelizes Blue's house identically for free and propagates to the Fuchsia City, Route 25, and Saffron City instances of the same art. The door's lower-left tile (27) is the warp/walkable tile — keep its recessed front face aligned with the collision cell so the 3D doorway matches REDS_HOUSE_1F's warp.

What shipping it settled

The mod implements this as mods/DRAMATIC_SHAPE/lib/Buildings.lua, driven by a buildings list in the profile. Three things changed from the sketch above, each for a reason worth keeping:

Key the band table by the building's tile GRID, not by tile id. A tile id is not a band: tile 23 is the house's awning course and the top of its roof and the eave course that ends Oak's lab's much taller roof. Matching the exact grid (../buildings/ catalogues one per building, with every map that places it) is unambiguous, still shares one entry across all seven placements of Red's house, and cost nothing to verify — a scan of all 222 maps returns exactly the catalogued placements.

Measure everything measurable. Only the band table needs a human to read the drawing. The silhouette, the taper rate, the eave height and every window and doorway come off the pixels: a pane is a non-black region the drawing seals behind its own black frame, and T(x) = YTOP - topRow(x) falls straight out of the mask — which is also what makes YTOP stop being hand-tuned. The eave tips landing one voxel above the awning, tuned by hand here, then happens by itself.

Depth is the plot, and the drawn row → depth mapping has a direction. In a diorama D is free; on a map it is the building's footprint (48px for Red's house, 64 for the lab), which is why the roof band has to be cycled so far. And the drawing looks at the roof from the north: its top rows are the FAR edge and its bottom rows the eave over the facade. Getting that backwards is invisible in the counts and in a symmetric preview — it shows up in game as a fascia along the wrong rim.

One colour note: the outline pass's grey fascia band (GREY at t-1) is right for the raw GB palette but comes out white once the atlas is recoloured, turning every sloped end into a black-and-white zip. The drawing's own eave is black / #555 / black, and using that reads correctly under every palette.

Beyond the house: the forms later objects added

The house is all solid masses — every band either lies flat or extrudes. Later objects forced the taxonomy open, and each addition came from the same root move: name the real 3D form first, then ask which surfaces the drawing shows. The recurring failure at every step was the extruded picture — and it has a second-order form that survives re-segmentation. The Bike Shop's toolbox was re-read from "a prop" into "a cabinet with a pump beside it": named parts, correct plot, de-outlined sides, and still wrong, because the region read as a cabinet front was the inside of an open box seen from above. Naming the parts is not enough; every REGION must answer "what surface of the real object is this?" The reliable arbiter is arithmetic: the correct reading makes the drawn row counts land exactly (the toolbox: 1 back-wall rim + 6 interior rows + 1 front rim = 8 = the one-tile plot depth). Forcing rows to fit means the reading is wrong.

Hollow forms. An open container is the one shape whose model must contain AIR, which no band table or extrusion can produce. The tray treatment builds four walls to the drawn rims, lays the top-view band on the floor of the cavity (its contents — a wrench — come along free, since they are just pixels of that band), and leaves the space between empty. Two rules only containers hit: the pane-recess pass must never run on a one-voxel wall (it deletes the front voxel to expose the one behind, and there is nothing behind — the wall becomes a hole), and the hollowness needs its own verification assert, because a later change that refills the cavity leaves every count looking plausible.

Round forms. A drawn ellipse wider than tall is a horizontal circle seen from above — that one aspect-ratio measurement settles the whole reading. Straight flanks give diameter and height at once (round in plan, so drawn width IS depth — the one depth never authored). The arcs above and below the straight run are the same top and base circles seen curving: ground contact and mouth edge, not narrowing — revolving them puts the object on a stem. The hull's chord representation stores one z-interval per column/row, so a taper is expressible (re-cut the chords, squeeze the art into the narrowed span so the rim outline survives) but a hollow ring needs a second chord. Voxel resolution bounds taste: on an 11-wide object a one-step taper reads as damage and two steps as a cone; pick the step count and derive the amount.

Thin forms. A line drawing cannot be thick. The air inside a bicycle's frame is what makes it read as a bicycle; extrude each stroke 5 voxels and the side faces of neighbouring strokes close every gap off-axis — six bikes become one dark mass. Standee thickness is a vocabulary (PINNED_DEPTH: 1 for paper, 2 for plates and side-on vehicles, 5 for silhouettes, 10 for objects with a body), and when a standee looks wrong the first move is to dump the detector's mask — if the mask is a clean object, thickness is the problem, not segmentation.

Authored masks. When a drawing shares its tiles and shades with what it is painted into, nothing automatic can separate them; the profile carries a pixel mask instead. A person becomes a figures card (flat, leaning with the camera, standing on its feet — because GB character art is face-on iconography); an object becomes a mounted slab (fixed in the world, holding the wall's plane, keeping its drawn elevation — because a side-on drawing is a plane parallel to the wall). And when the backdrop is a regular pattern, the mask should be MEASURED, not hand-drawn: composite the plain backdrop tile over the same grid and flood from the border through pixels that still match it — what the flood cannot reach is the object, sprite-pure and exact.

Verification, extended. Isometric previews miss what only the game shows: shoot in-game at both the ¾ rung and the low rung (front-face holes and proportion errors are invisible from above), crop and NEAREST-upscale before judging, and remember the flat rung renders no model at all. Two cheap renders beat argument: the front-most voxel per (x, y) laid beside the composited drawing catches anchoring and texel leaks instantly, and the same render with sunk voxels flagged turns the recess pass into something you look at. When shared builder code moves, a saved count baseline diffed after every edit (mind the line endings) is what proves a generalization is an identity for every model that already shipped.