radiance · Radiance self-hosting compiler

compiler/ kernel/ lib/ examples/ std/ arch/ char/ collections/ lang/ alloc/ ast/ gen/ il/ module/ parser/ resolver/ scanner/ alloc.rad 7.1 KiB ast.rad 26.7 KiB gen.rad 513 B il.rad 19.5 KiB lower.rad 308.1 KiB module.rad 14.9 KiB package.rad 1.3 KiB parser.rad 89.5 KiB resolver.rad 439.6 KiB scanner.rad 18.0 KiB sexpr.rad 6.4 KiB strings.rad 2.2 KiB types.rad 1.6 KiB sys/ arch.rad 68 B char.rad 855 B collections.rad 39 B fmt.rad 8.3 KiB intrinsics.rad 467 B io.rad 1.7 KiB lang.rad 276 B mem.rad 2.3 KiB sys.rad 179 B testing.rad 2.4 KiB tests.rad 15.7 KiB vec.rad 3.2 KiB std.rad 281 B scripts/ seed/ sublime/ test/ vim/ .gitignore 336 B .gitsigners 112 B CONTRIBUTING 2.1 KiB LICENSE 1.1 KiB Makefile 9.2 KiB README 2.5 KiB STYLE 2.5 KiB std.lib 1.5 KiB std.lib.test 662 B
lib/std/lang/lower.rad 308.1 KiB raw
//! AST to IL lowering pass.
//!
//! This module converts the typed AST produced by the resolver into a linear
//! SSA-based intermediate language (IL). The IL is suitable for further
//! optimization and code generation.
//!
//! # Design Overview
//!
//! The lowering process works in two main phases:
//!
//! 1. Module-level pass: Iterates over top-level declarations, lowering
//!    each function independently while accumulating global data items (strings,
//!    constants, static arrays) into a shared data section.
//!
//! 2. Function-level pass: For each function, constructs an SSA-form control
//!    flow graph (CFG) with basic blocks connected by jumps and branches. Uses
//!    a simplified SSA construction algorithm where variables are tracked per-block
//!    and block parameters are inserted lazily when a variable is used before
//!    being defined in a block.
//!
//! # Memory Model
//!
//! All allocations use an arena allocator passed through the `Lowerer` context.
//! The IL is entirely stack-based at runtime -- there's no heap allocation during
//! program execution. Aggregate values (records, slices, optionals) are passed
//! by pointer on the stack.
//!
//! # SSA Construction
//!
//! SSA form is built incrementally as the AST is walked. When a variable is
//! defined (via `defVar`), its current value is recorded in the current block.
//! When a variable is used (via `useVar`), the algorithm either:
//! - Returns the value if defined in this block
//! - Recurses to predecessors if the block is sealed (all predecessors known)
//! - Inserts a block parameter if the variable value must come from multiple paths
//!
//! Block "sealing" indicates that all predecessor edges are known, enabling
//! the SSA construction to resolve cross-block variable references.
//!
//! # SSA Variable API
//!
//! 1. `newVar` creates a logical variable that can be defined differently in each block.
//! 2. `defVar` defines the variable's value in the current block.
//! 3. `useVar` reads the variable; if called in a block with multiple
//!             predecessors that defined it differently, the SSA algorithm
//!             automatically creates a block parameter.
//!
//! # Expression Lowering
//!
//! Expressions produce IL values which can be:
//!
//! - Imm(i64): immediate/constant values
//! - Reg(u32): SSA register references
//! - Sym(name): symbol references (for function pointers, data addresses)
//! - Undef: for unused values
//!
//! For aggregate types (records, arrays, slices, optionals), the "value" is
//! actually a pointer to stack-allocated memory containing the aggregate.
//!
//! # Block ordering invariant
//!
//! The register allocator processes blocks in forward index order and uses a
//! single global assignment array. This means a value's definition block must
//! have a lower index than any block that uses the value (except through
//! back-edges, where block parameters handle the merge). The lowerer maintains
//! this by creating blocks in control-flow order. For `for` loops, the step
//! block is created lazily (after the loop body) so that its index is higher
//! than all body blocks -- see [`lowerForLoop`] and [`getOrCreateContinueBlock`].
//!
//! A more robust alternative would be per-block register maps with edge-copy
//! resolution (as in QBE's `rega.c`), which is block-order-independent. That
//! would eliminate this invariant at the cost of a more complex allocator.
//!
//! # Notes on constant data lowering
//!
//! The lowerer flattens constant AST expressions directly into IL data values.
//! Primitive constants use [`resolver::ConstValue`] as an intermediate form.
//! Record padding is explicit: `undef * N;` for N padding bytes.
//!
//!   AST Node (ArrayLit, RecordLit, literals, etc.)
//!       ↓ [`lowerConstData`]
//!   resolver::ConstValue (Bool, Char, String, Int)
//!       ↓ [`constValueToDataItem`]
//!   il::DataItem (Val, Sym, Str, Undef)
//!       ↓
//!   il::DataValue
//!       ↓
//!   il::Data
//!
use std::fmt;
use std::io;
use std::lang::alloc;
use std::lang::types;
use std::mem;
use std::lang::ast;
use std::lang::il;
use std::lang::module;
use std::lang::resolver;

// TODO: Search for all `_ as i32` to ensure that casts from u32 to i32 don't
// happen, since they are potentially truncating values.

// TODO: Support constant union lowering.
// TODO: Void unions should be passed by value.

////////////////////
// Error Handling //
////////////////////

/// Lowering errors are typically unrecoverable since they indicate bugs in
/// the resolver or malformed AST that should have been caught earlier.
export union LowerError: Copy {
    /// A node's symbol was not set before lowering.
    MissingSymbol(*ast::Node),
    /// A node's type was not set before lowering.
    MissingType(*ast::Node),
    /// A node's constant value was not set before lowering.
    MissingConst(*ast::Node),
    /// An optional type was expected.
    ExpectedOptional,
    /// A record type was expected.
    ExpectedRecord,
    /// An array was expected.
    ExpectedArray,
    /// A block was expected.
    ExpectedBlock(*ast::Node),
    /// An identifier was expected.
    ExpectedIdentifier,
    /// A function parameter was expected.
    ExpectedFunctionParam,
    /// A function type was expected.
    ExpectedFunction,
    /// Trying to lower loop construct outside of loop.
    OutsideOfLoop,
    /// Invalid variable use.
    InvalidUse,
    /// Unexpected node value.
    UnexpectedNodeValue(*ast::Node),
    /// Unexpected type.
    UnexpectedType(resolver::Type),

    /// Missing control flow target.
    MissingTarget,
    /// Missing metadata that should have been set by resolver.
    MissingMetadata,
    /// Expected a slice or array type for operation.
    ExpectedSliceOrArray,
    /// Expected a variant symbol.
    ExpectedVariant,
    /// Expected a call expression.
    ExpectedCall,
    /// Field not found in record or invalid field access.
    FieldNotFound,
    /// Assignment to an immutable variable.
    ImmutableAssignment,
    /// Nil used in non-optional context.
    NilInNonOptional,
    /// Invalid argument count for builtin.
    InvalidArgCount,
    /// Feature or pattern not supported by the lowerer.
    Unsupported,
    /// Unknown intrinsic function.
    UnknownIntrinsic,
    /// Allocation failure.
    AllocationFailed,
}

/// Print a LowerError for debugging.
export fn printError(err: LowerError) {
    match err {
        case LowerError::MissingSymbol(_) => io::print("MissingSymbol"),
        case LowerError::MissingType(_) => io::print("MissingType"),
        case LowerError::MissingConst(_) => io::print("MissingConst"),
        case LowerError::ExpectedOptional => io::print("ExpectedOptional"),
        case LowerError::ExpectedRecord => io::print("ExpectedRecord"),
        case LowerError::ExpectedArray => io::print("ExpectedArray"),
        case LowerError::ExpectedBlock(_) => io::print("ExpectedBlock"),
        case LowerError::ExpectedIdentifier => io::print("ExpectedIdentifier"),
        case LowerError::ExpectedFunctionParam => io::print("ExpectedFunctionParam"),
        case LowerError::ExpectedFunction => io::print("ExpectedFunction"),
        case LowerError::OutsideOfLoop => io::print("OutsideOfLoop"),
        case LowerError::InvalidUse => io::print("InvalidUse"),
        case LowerError::UnexpectedNodeValue(_) => io::print("UnexpectedNodeValue"),
        case LowerError::UnexpectedType(_) => io::print("UnexpectedType"),
        case LowerError::MissingTarget => io::print("MissingTarget"),
        case LowerError::MissingMetadata => io::print("MissingMetadata"),
        case LowerError::ExpectedSliceOrArray => io::print("ExpectedSliceOrArray"),
        case LowerError::ExpectedVariant => io::print("ExpectedVariant"),
        case LowerError::ExpectedCall => io::print("ExpectedCall"),
        case LowerError::FieldNotFound => io::print("FieldNotFound"),
        case LowerError::ImmutableAssignment => io::print("ImmutableAssignment"),
        case LowerError::NilInNonOptional => io::print("NilInNonOptional"),
        case LowerError::InvalidArgCount => io::print("InvalidArgCount"),
        case LowerError::Unsupported => io::print("Unsupported"),
        case LowerError::UnknownIntrinsic => io::print("UnknownIntrinsic"),
        case LowerError::AllocationFailed => io::print("AllocationFailed"),
    }
}

///////////////
// Constants //
///////////////

/// Maximum nesting depth of loops.
constant MAX_LOOP_DEPTH: u32 = 16;
/// Maximum number of `catch` clauses per `try`.
constant MAX_CATCH_CLAUSES: u32 = 32;

// Slice Layout
//
// A slice is a fat pointer consisting of a data pointer, a length and a capacity.
// `{ ptr: u32, len: u32, cap: u32 }`.

/// Slice data pointer offset.
constant SLICE_PTR_OFFSET: i32 = 0;
/// Offset of slice length in slice data structure.
constant SLICE_LEN_OFFSET: i32 = resolver::PTR_SIZE as i32;
/// Offset of slice capacity in slice data structure.
constant SLICE_CAP_OFFSET: i32 = resolver::PTR_SIZE as i32 + 4;

// Trait Object Layout
//
// A trait object is a fat pointer consisting of a data pointer and a
// v-table pointer. `{ data: *T, vtable: *VTable }`.

/// Trait object data pointer offset.
constant TRAIT_OBJ_DATA_OFFSET: i32 = 0;
/// Trait object v-table pointer offset.
constant TRAIT_OBJ_VTABLE_OFFSET: i32 = resolver::PTR_SIZE as i32;

// Tagged Value Layout (optionals, tagged unions)
//
// Optionals and unions use 1-byte tags. Results use 8-byte tags.
//
// `{ tag: u8, [padding], payload: T }`
//
// Optionals use `tag: 0` for `nil` and `tag: 1` otherwise.
// When `T` is a pointer, the entire optional is stored as a single pointer.
//
// Tagged unions have a payload the size of the maximum variant size.

/// Offset of tag in tagged value data structure.
constant TVAL_TAG_OFFSET: i32 = 0;
/// Offset of value in result data structure (8-byte tag).
constant RESULT_VAL_OFFSET: i32 = resolver::PTR_SIZE as i32;

//////////////////////////
// Core Data Structures //
//////////////////////////

/// Options controlling the lowering pass.
export record LowerOptions: Copy {
    /// Whether to emit source location info.
    debug: bool,
    /// Whether to lower `@test` functions.
    buildTest: bool,
}

/// Role of a lowered function in the final program.
export union FnRole: Copy {
    /// Regular function.
    Normal,
    /// Program entry function marked with `@default`.
    Default,
}

/// One function ready for a caller to consume before lowering continues.
export record LoweredFunction: Copy {
    /// Function allocated in the current function arena.
    function: *unsafe il::Fn,
    /// Entry-point role of the source declaration.
    role: FnRole,
}

/// Position within one module's declarations and instance methods.
export record ModuleCursor {
    /// Top-level declaration nodes in source order.
    declarations: *[*ast::Node],
    /// Next declaration to process.
    next: u32,
    /// Whether default functions in this module are program entry points.
    isRoot: bool,
    /// Instance whose methods are currently being lowered.
    instanceState: InstanceCursor,
    /// Whether the instance cursor contains an active declaration.
    instanceActive: bool,
}

/// Persistent v-table names and position within an instance declaration.
export record InstanceCursor: Copy {
    /// Trait whose method slots define the v-table.
    traitInfo: *unsafe resolver::TraitType,
    /// Trait source name.
    traitName: *[u8],
    /// Concrete type source name.
    typeName: *[u8],
    /// Method declarations in source order.
    methods: *[*ast::Node],
    /// Next method to process.
    next: u32,
    /// Persistent qualified function names, indexed by trait method slot.
    methodNames: [*[u8]; ast::MAX_TRAIT_METHODS],
    /// Slots initialized by concrete methods.
    methodNameSet: [bool; ast::MAX_TRAIT_METHODS],
}

/// Module-level lowering context. Shared across all function lowerings.
/// Holds global state like the data section (strings, constants) and provides
/// access to the resolver for type queries.
export record Lowerer: 'arena + 'phase where 'arena: 'phase {
    /// Arena for persistent lowering state, including data and symbol names.
    arena: *unsafe mut alloc::Arena,
    /// Resolver for type information. Used to query types, symbols, and
    /// compile-time constant values during lowering.
    resolver: &'phase resolver::Resolver 'arena,
    /// Module graph for cross-module symbol resolution.
    moduleGraph: ?&'phase module::ModuleGraph,
    /// Package name for qualified symbol names.
    pkgName: *[u8],
    /// Current module being lowered.
    currentMod: ?u16,
    /// Global data items (string literals, constants, static arrays).
    /// These become the data sections in the final binary.
    data: *mut [il::Data],
    /// First data entry owned by the package currently being lowered.
    packageDataStart: u32,
    /// Functions retained for accumulated output.
    functions: *unsafe mut [*unsafe il::Fn],
    /// Function and data symbols with their emitted names.
    symbolNames: *mut [SymbolNameEntry],
    /// Global error type tag table. Maps nominal types to unique tags.
    errTags: *mut [ErrTagEntry],
    /// Next error tag to assign (starts at 1; 0 = success).
    errTagCounter: u32,
    /// Lowering options.
    options: LowerOptions,
}

/// Entry mapping a function or data symbol to its emitted name.
record SymbolNameEntry: Copy {
    /// Stable resolver symbol identity.
    sym: *unsafe resolver::Symbol,
    /// Qualified name in the emitted program.
    qualName: *[u8],
}

/// Entry in the global error tag table.
record ErrTagEntry: Copy {
    /// The type of this error, identified by its interned pointer.
    ty: resolver::Type,
    /// The globally unique tag assigned to this error type (non-zero).
    tag: u32,
}

/// Compute the maximum size of any error type in a throw list.
unsafe fn maxErrSize(throwList: *[*resolver::Type]) -> u32 {
    let mut maxSize: u32 = 0;
    for ty in throwList {
        let size = resolver::getTypeLayout(*ty).size;
        if size > maxSize {
            set maxSize = size;
        }
    }
    return maxSize;
}

/// Get or assign a globally unique error tag for the given error type.
/// Tag `0` is reserved for success; error tags start at `1`.
unsafe fn getOrAssignErrorTag 'arena 'phase (self: &mut Lowerer 'arena 'phase, errType: resolver::Type) -> u32 where 'arena: 'phase {
    for entry in &self.errTags[..] {
        if resolver::erasedTypesEqual(entry.ty, errType) {
            return entry.tag;
        }
    }
    let tag = self.errTagCounter;

    set self.errTagCounter += 1;
    self.errTags.append(ErrTagEntry { ty: errType, tag }, alloc::arenaAllocator(self.arena));

    return tag;
}

/// Retain a lowered function for accumulated IL output.
unsafe fn emitFunction 'arena 'phase (self: &mut Lowerer 'arena 'phase, func: *unsafe il::Fn) where 'arena: 'phase {
    self.functions.append(func, alloc::arenaAllocator(self.arena));
}

/// Get the function role for a top-level function declaration.
fn lowerFnRole(isRoot: bool, attrs: ?ast::Attributes) -> FnRole {
    if isRoot and checkAttr(attrs, ast::Attribute::Default) {
        return FnRole::Default;
    }
    return FnRole::Normal;
}

/// Builder for accumulating data values during constant lowering.
record DataValueBuilder {
    /// Allocator used to grow the value storage.
    allocator: alloc::Allocator,
    /// Owned values under construction.
    values: *mut [il::DataValue],
    /// Whether all pushed values can be represented by zero-filled memory.
    zeroInit: bool,
}

/// Result of lowering constant data.
record ConstDataResult: Copy {
    /// Immutable initializer values.
    values: *[il::DataValue],
    /// Whether the values can be represented by zero-filled memory.
    zeroInit: bool,
}

/// Create a new builder.
fn dataBuilder(allocator: alloc::Allocator) -> DataValueBuilder {
    return DataValueBuilder { allocator, values: &mut [], zeroInit: true };
}

/// Return whether a data item can be omitted from a zero-filled image.
fn dataIsZero(item: il::DataItem) -> bool {
    match item {
        case il::DataItem::Undef => return true,
        case il::DataItem::Val { val, .. } => return val == 0,
        else => return false,
    }
}

/// Return whether all data values can be omitted from a zero-filled image.
fn dataValuesAreZeroInit(values: *[il::DataValue]) -> bool {
    for value in values {
        if value.count > 0 and not dataIsZero(value.item) {
            return false;
        }
    }
    return true;
}

/// Append a data value to the builder.
fn dataBuilderPush(b: &mut DataValueBuilder, value: il::DataValue) {
    b.values.append(value, b.allocator);

    if value.count > 0 and not dataIsZero(value.item) {
        set b.zeroInit = false;
    }
}

/// Return the accumulated values.
fn dataBuilderFinish(b: DataValueBuilder) -> ConstDataResult {
    let case DataValueBuilder { values, zeroInit, .. } = b
        else panic "expected data builder";
    return ConstDataResult { values, zeroInit };
}

///////////////////////////
// SSA Variable Tracking //
///////////////////////////

// The SSA construction algorithm tracks variable definitions per-block.
// Each source-level variable gets a [`Var`] handle, and each block maintains
// a mapping from [`Var`] to current SSA value. When control flow merges,
// block parameters are inserted to merge values from different control flow paths.

/// A variable handle. Represents a source-level variable during lowering.
/// The same [`Var`] can have different SSA values in different blocks.
export record Var: Copy(u32);

/// Metadata for a source-level variable, stored once per function.
///
/// Each variable declaration in the source creates one [`VarData`] entry in the
/// function's `variables` array, indexed by `id`. This contains static
/// properties that don't change across basic blocks.
///
/// Per-block SSA values are tracked separately in [`BlockData::vars`] as [`?il::Val`],
/// where `nil` means "not yet assigned in this block". Together they implement
/// SSA construction.
///
record VarData: Copy {
    /// Variable name, used by [`lookupVarByName`] to resolve identifiers.
    /// Nil for anonymous variables (e.g., internal loop counters).
    name: ?*[u8],
    /// IL type of this variable. Set at declaration time and used when
    /// generating loads, stores, and type-checking assignments.
    type: il::Type,
    /// Whether this variable was declared with `mut`. Controls whether [`defVar`]
    /// is allowed after the initial definition.
    mutable: bool,
    /// Whether this variable's address has been taken (e.g. via `&mut x`).
    /// When true, the SSA value is a pointer to a stack slot and reads/writes
    /// must go through memory instead of using the cached SSA value directly.
    addressTaken: bool,
}

/// Links a function parameter to its corresponding variable for the entry block.
/// After creating the entry block, we iterate through these to define initial values.
record FnParamBinding: Copy {
    /// The variable that receives this parameter's value.
    var: Var,
    /// SSA register containing the parameter value from the caller.
    reg: il::Reg,
}

////////////////////////////////
// Basic Block Representation //
////////////////////////////////

// During lowering, we build a CFG of basic blocks. Each block accumulates
// instructions until terminated by a jump, branch, return, or unreachable.
// Blocks can be created before they're filled (forward references for jumps).

/// A handle to a basic block within the current function.
/// Block handles are stable, they don't change as more blocks are added.
export record BlockId: Copy(u32);

/// Internal block state during construction.
///
/// The key invariants:
///
/// - A block is "open" if it has no terminator; instructions can be added.
/// - A block is "sealed" when all predecessor edges are known.
/// - Sealing resolves the predecessor arguments for block parameters.
///
/// This differs from the final [`il::Block`] which is immutable and fully formed.
record BlockData: Copy {
    /// Block label for debugging and IL printing.
    label: *[u8],
    /// Block parameters for merging values at control flow joins. These
    /// receive values from predecessor edges when control flow merges.
    params: *unsafe mut [il::Param],
    /// Variable ids in parameter order. Before sealing, these are the variables
    /// whose predecessor arguments must be resolved.
    paramVars: *unsafe mut [u32],
    /// Instructions accumulated so far. The last instruction should eventually
    /// be a terminator.
    instrs: *unsafe mut [il::Instr],
    /// Debug source locations, one per instruction. Only populated when
    /// debug info is enabled.
    locs: *unsafe mut [il::SrcLoc],
    /// Predecessor block ids. Used for SSA construction to propagate values
    /// from predecessors when a variable is used before being defined locally.
    preds: *unsafe mut [u32],
    /// The current SSA value of each variable in this block. Indexed by variable
    /// id. A `nil` means the variable wasn't assigned in this block. Updated by
    /// [`defVar`], queried by [`useVarInBlock`].
    vars: *unsafe mut [?il::Val],
    /// Sealing state. Once sealed, all predecessors are known and we can resolve
    /// variable uses that need to pull values from predecessors.
    sealState: Sealed,
    /// Loop nesting depth when this block was created.
    loopDepth: u32,
}

/// Block sealing state for SSA construction.
///
/// A block is "unsealed" while its predecessors are still being discovered.
/// Before sealing, `BlockData.paramVars` records each variable that needs a
/// block parameter. Once all predecessors are known, the block is sealed and
/// those parameters are resolved via [`resolveBlockArgs`].
union Sealed: Copy {
    /// Block is unsealed; predecessors may still be added.
    No,
    /// Block is sealed; all predecessors are known.
    Yes,
}

///////////////////////////////////
// Loop and Control Flow Context //
///////////////////////////////////

/// Context for break/continue statements within a loop.
/// Each nested loop pushes a new context onto the loop stack.
export record LoopCtx: Copy {
    /// Where `break` should transfer control (the loop's exit block).
    breakTarget: BlockId,
    /// Where `continue` should transfer control.
    continueTarget: ?BlockId,
}

/// Logical operator.
union LogicalOp: Copy { And, Or }

/// Iterator state for for-loop lowering.
union ForIter: Copy {
    /// Range iterator: `for i in 0..n`.
    Range {
        valVar: Var,
        indexVar: ?Var,
        endVal: il::Val,
        valType: il::Type,
        unsigned: bool,
    },
    /// Collection iterator: `for elem in slice`.
    Collection {
        valVar: ?Var,
        idxVar: Var,
        dataReg: il::Reg,
        lengthVal: il::Val,
        elemType: *resolver::Type,
    },
}

//////////////////////////////
// Pattern Matching Support //
//////////////////////////////

// Match expressions are lowered by evaluating the subject once, then emitting
// a chain of comparison-and-branch sequences for each arm. The algorithm
// handles several subject types specially:
//
// - Optional pointers: compared against `null`.
// - Optional aggregates: tag checked then payload extracted.
// - Unions: tag compared against variant indices.

/// Cached information about a match subject. Computed once and reused across
/// all arms to avoid redundant lowering and type queries.
record MatchSubject: Copy {
    /// The lowered subject value.
    val: il::Val,
    /// Source-level type from the resolver.
    type: resolver::Type,
    /// IL-level type for code generation.
    ilType: il::Type,
    /// The type that binding arms should use. For optionals, this is the
    /// inner type; for regular values, it's the same as `type`.
    bindType: resolver::Type,
    /// Classification of how the subject should be compared and destructured.
    kind: MatchSubjectKind,
    /// How bindings are created: by value, or by reference.
    by: resolver::MatchBy,
}

/// Classifies a match subject by how it should be compared and destructured.
union MatchSubjectKind: Copy {
    /// Regular value: direct equality comparison.
    Regular,
    /// Optional with null pointer optimization: `?*T`, `?*[T]`.
    OptionalPtr,
    /// Optional aggregate `?T`: tagged union with payload.
    OptionalAggregate,
    /// Union type: tag compared against variant indices.
    Union(resolver::UnionType),
}

/// Determine the kind of a match subject from its type.
unsafe fn matchSubjectKind(type: resolver::Type) -> MatchSubjectKind {
    if resolver::isOptionalPointer(type) {
        return MatchSubjectKind::OptionalPtr;
    }
    if resolver::isOptionalAggregate(type) {
        return MatchSubjectKind::OptionalAggregate;
    }
    if let info = unionInfoFromType(type) {
        return MatchSubjectKind::Union(info);
    }
    return MatchSubjectKind::Regular;
}

//////////////////////////
// Field Access Support //
//////////////////////////

/// Result of resolving a place expression to a memory location.
record FieldRef: Copy {
    /// Base pointer register (points to the container).
    base: il::Reg,
    /// Byte offset of the value within the container.
    offset: i32,
    /// Type of the value held at that location.
    fieldType: resolver::Type,
}

/// Result of computing an element pointer for array/slice subscript operations.
/// Used by [`lowerElemPtr`] to return both the element address register and
/// the element type for subsequent load or address-of operations.
record ElemPtrResult: Copy {
    /// Register holding the computed element address.
    elemReg: il::Reg,
    /// Source-level type of the element.
    elemType: resolver::Type,
}

/// Result of resolving a slice range to a data pointer and element count.
record SliceRangeResult: Copy {
    dataReg: il::Reg,
    count: il::Val,
}

/////////////////////////////
// Function Lowering State //
/////////////////////////////

/// Per-function lowering state. Created fresh for each function and contains
/// all the mutable state needed during function body lowering.
record FnLowerer: 'arena + 'phase + 'function where 'arena: 'phase, 'phase: 'function {
    /// Reference to the module-level lowerer.
    low: &'function mut Lowerer 'arena 'phase,
    /// Arena for allocations owned by this function.
    arena: &'function mut alloc::Arena,
    /// Type signature of the function being lowered.
    fnType: *resolver::FnType,
    /// Number of SSA variable slots required for each block.
    localCount: u32,
    /// Function name, used as prefix for generated data symbols.
    fnName: *[u8],

    // ~ SSA variable tracking ~ //

    /// Metadata (name, type, mutability) for each variable. Indexed by variable
    /// id. Doesn't change after declaration. For the SSA value of a variable in
    /// a specific block, see [`BlockData::vars`].
    vars: *unsafe mut [VarData],
    /// Parameter-to-variable bindings, initialized in the entry block.
    params: [FnParamBinding; resolver::MAX_FN_PARAMS],
    /// Number of initialized parameter bindings.
    paramsLen: u32,

    // ~ Basic block management ~ //

    /// Block storage array, indexed by block id.
    blockData: *unsafe mut [BlockData],
    /// The entry block for this function.
    entryBlock: ?BlockId,
    /// The block currently receiving new instructions.
    currentBlock: ?BlockId,

    // ~ Loop management ~ //

    /// Loop contexts indexed by nesting depth.
    /// The active prefix contains `loopDepth` entries.
    loopStack: [LoopCtx; MAX_LOOP_DEPTH],
    /// Current nesting depth (index into loopStack).
    loopDepth: u32,

    // ~ Counters ~ //

    /// Counter for generating unique block labels like `then#0`, `loop#1`, etc.
    labelCounter: u32,
    /// Counter for generating unique data names within this function.
    /// Each literal gets a name like `fnName$literal$N`.
    dataCounter: u32,
    /// Counter for generating SSA register numbers.
    regCounter: u32,
    /// When the function returns an aggregate type, the caller passes a hidden
    /// pointer as the first parameter. The callee writes the return value into
    /// this buffer and returns the pointer.
    returnReg: ?il::Reg,
    /// Whether the function is a leaf.
    isLeaf: bool,

    // ~ Debug info ~ //

    /// Current debug source location, set when processing AST nodes.
    srcLoc: il::SrcLoc,
}

/////////////////////////////////
// Module Lowering Entry Point //
/////////////////////////////////

/// Lower a complete module AST to an IL program.
///
/// This is the main entry point for the lowering pass. It:
/// 1. Counts functions to preallocate the output array.
/// 2. Iterates over top-level declarations, lowering each.
/// 3. Returns the complete IL program with functions and data section.
///
/// The resolver must have already processed the AST -- we rely on its type
/// annotations, symbol table, and constant evaluations.
export unsafe fn lower 'arena (
    res: &resolver::Resolver 'arena,
    root: *ast::Node,
    pkgName: *[u8],
    arena: &mut alloc::Arena
) -> il::Program throws (LowerError) {
    let functionArena = (&mut *arena) as *unsafe mut alloc::Arena;
    let resolved: 'phase = &*res where 'arena: 'phase in {
        let mut low = Lowerer 'arena 'phase {
            arena: (&mut *arena) as *unsafe mut alloc::Arena,
            resolver: resolved,
            moduleGraph: nil,
            pkgName,
            currentMod: nil,
            data: &mut [],
            packageDataStart: 0,
            functions: &mut [],
            symbolNames: &mut [],
            errTags: &mut [],
            errTagCounter: 1,
            options: LowerOptions { debug: false, buildTest: false },
        };
        try lowerDecls(&mut low, root, true, functionArena);

        return finalize(low);
    }
}

/////////////////////////////////
// Multi-Module Lowering API   //
/////////////////////////////////

/// Create a lowerer for multi-module compilation.
/// The resolver, graph, and persistent arena must outlive the returned lowerer.
export unsafe fn lowerer 'arena 'phase (
    res: &'phase resolver::Resolver 'arena,
    graph: &'phase module::ModuleGraph,
    pkgName: *[u8],
    arena: *unsafe mut alloc::Arena,
    options: LowerOptions
) -> Lowerer 'arena 'phase where 'arena: 'phase {
    return Lowerer 'arena 'phase {
        arena,
        resolver: res,
        moduleGraph: graph,
        pkgName,
        currentMod: nil,
        data: &mut [],
        packageDataStart: 0,
        functions: &mut [],
        symbolNames: &mut [],
        errTags: &mut [],
        errTagCounter: 1,
        options,
    };
}

/// Lower a module's AST into the lowerer.
/// Call this for each module in the package, then use `finalize` to get the program.
export unsafe fn lowerModule 'arena 'phase (
    low: &mut Lowerer 'arena 'phase,
    moduleId: u16,
    root: *ast::Node,
    isRoot: bool,
    functionArena: &mut alloc::Arena
) throws (LowerError) where 'arena: 'phase {
    set low.currentMod = moduleId;
    try lowerDecls(low, root, isRoot, functionArena);
}

/// Start lowering the declarations of one module.
export unsafe fn moduleCursor(root: *ast::Node, isRoot: bool) -> ModuleCursor throws (LowerError) {
    let case ast::NodeValue::Block(block) = root.value else {
        throw LowerError::ExpectedBlock(root);
    };
    return ModuleCursor { declarations: block.statements, next: 0, isRoot, instanceState: undefined, instanceActive: false };
}

/// Lower declarations until one function is ready or the module is complete.
/// The caller must finish using a function before reclaiming its arena storage.
export unsafe fn lowerNext 'arena 'phase (
    low: &mut Lowerer 'arena 'phase,
    cursor: &mut ModuleCursor,
    functionArena: &mut alloc::Arena
) -> ?LoweredFunction throws (LowerError) where 'arena: 'phase {
    loop {
        if cursor.instanceActive {
            if let function = try lowerNextInstance(low, &mut cursor.instanceState, functionArena) {
                return LoweredFunction { function, role: FnRole::Normal };
            }
            set cursor.instanceActive = false;
        }
        if cursor.next >= cursor.declarations.len {
            return nil;
        }
        let node = cursor.declarations[cursor.next];
        set cursor.next += 1;
        match node.value {
            case ast::NodeValue::FnDecl(decl) => {
                if let function = try lowerFnDecl(low, node, decl, functionArena) {
                    return LoweredFunction { function, role: lowerFnRole(cursor.isRoot, decl.attrs) };
                }
            }
            case ast::NodeValue::ConstDecl(decl) =>
                try lowerDataDecl(low, node, decl.value, true),
            case ast::NodeValue::StaticDecl(decl) =>
                try lowerDataDecl(low, node, decl.value, false),
            case ast::NodeValue::InstanceDecl { traitName, targetType, methods, .. } => {
                set cursor.instanceState = try instanceCursor(low, traitName, targetType, methods);
                set cursor.instanceActive = true;
            }
            case ast::NodeValue::MethodDecl { name, receiverName, sig, body, .. } => {
                if let function = try lowerMethodDecl(
                    low, node, name, receiverName, sig, body, functionArena
                ) {
                    return LoweredFunction { function, role: FnRole::Normal };
                }
            }
            else => {},
        }
    }
}

/// Lower all top-level declarations in a block.
unsafe fn lowerDecls 'arena 'phase (
    low: &mut Lowerer 'arena 'phase,
    root: *ast::Node,
    isRoot: bool,
    functionArena: &mut alloc::Arena
) throws (LowerError) where 'arena: 'phase {
    let mut cursor = try moduleCursor(root, isRoot);
    while let result = try lowerNext(low, &mut cursor, functionArena) {
        emitFunction(low, result.function);
    }
}

/// Consume the lowerer and publish its global data as an immutable array.
export fn finalize 'arena 'phase (low: Lowerer 'arena 'phase) -> il::Program where 'arena: 'phase {
    let case Lowerer 'arena 'phase { data, functions, .. } = low
        else panic "expected lowerer";
    return il::Program { data, fns: functions };
}

/////////////////////////////////
// Qualified Name Construction //
/////////////////////////////////

/// Get module path segments for the current or specified module.
/// Returns empty slice if no module graph or module not found.
unsafe fn getModulePath 'arena 'phase (self: &mut Lowerer 'arena 'phase, modId: ?u16) -> *[*[u8]] where 'arena: 'phase {
    let graph = self.moduleGraph else {
        return &[];
    };
    let mut id = modId;
    if id == nil {
        set id = self.currentMod;
    }
    let actualId = id else {
        return &[];
    };
    let entry = module::get(graph, actualId) else {
        return &[];
    };
    return module::moduleQualifiedPath(entry);
}

/// Build a qualified name string for a symbol.
/// If `modId` is nil, uses current module.
unsafe fn qualifyName 'arena 'phase (self: &mut Lowerer 'arena 'phase, modId: ?u16, name: *[u8]) -> *[u8] where 'arena: 'phase {
    let path = getModulePath(self, modId);
    if path.len == 0 {
        return name;
    }
    return il::formatQualifiedName(self.arena, path, name);
}

/// Register the emitted name for a function or data symbol.
/// Calls and address expressions use this name across package boundaries.
unsafe fn registerSymbolName 'arena 'phase (self: &mut Lowerer 'arena 'phase, sym: *unsafe resolver::Symbol, qualName: *[u8]) where 'arena: 'phase {
    self.symbolNames.append(SymbolNameEntry { sym, qualName }, alloc::arenaAllocator(self.arena));
}

/// Look up the emitted name for a function or data symbol.
/// Return `nil` if its declaration has not been lowered.
// TODO: This is kind of dubious as an optimization, if it depends on the order
// in which modules are lowered.
// TODO: Use a hash table here?
unsafe fn lookupSymbolName 'arena 'phase (self: &Lowerer 'arena 'phase, sym: *unsafe resolver::Symbol) -> ?*[u8] where 'arena: 'phase {
    for entry in &self.symbolNames[..] {
        if entry.sym == sym {
            return entry.qualName;
        }
    }
    return nil;
}

/// Set the package context for lowering.
/// Called before lowering each package.
export unsafe fn setPackage 'arena 'phase (self: &mut Lowerer 'arena 'phase, pkgName: *[u8]) where 'arena: 'phase {
    set self.pkgName = pkgName;
    set self.currentMod = nil;
    set self.packageDataStart = self.data.len;
}

/// Create a new function lowerer for a given function type and name.
unsafe fn fnLowerer 'arena 'phase 'function (
    self: &'function mut Lowerer 'arena 'phase,
    node: *ast::Node,
    fnType: *resolver::FnType,
    qualName: *[u8],
    functionArena: &'function mut alloc::Arena
) -> FnLowerer 'arena 'phase 'function where 'arena: 'phase, 'phase: 'function {
    let localCount = resolver::nodeData(self.resolver, node).localCount;
    let mut fnLow = FnLowerer 'arena 'phase 'function {
        low: self,
        arena: functionArena,
        fnType,
        localCount,
        fnName: qualName,
        vars: &mut [],
        params: [FnParamBinding { var: Var(0), reg: il::Reg { n: 0 } }; resolver::MAX_FN_PARAMS],
        paramsLen: 0,
        blockData: &mut [],
        entryBlock: nil,
        currentBlock: nil,
        loopStack: [LoopCtx { breakTarget: BlockId(0), continueTarget: nil }; MAX_LOOP_DEPTH],
        loopDepth: 0,
        labelCounter: 0,
        dataCounter: 0,
        regCounter: 0,
        returnReg: nil,
        isLeaf: true,
        srcLoc: undefined,
    };
    if fnLow.low.options.debug {
        let modId = fnLow.low.currentMod else {
            panic "fnLowerer: debug enabled but no current module";
        };
        set fnLow.srcLoc = il::SrcLoc {
            moduleId: modId,
            offset: node.span.offset,
        };
    }
    return fnLow;
}

/// Lower a function declaration.
///
/// This sets up the per-function lowering state, processes parameters,
/// then lowers the function body into a CFG of basic blocks.
///
/// For throwing functions, the return type is a result aggregate
/// rather than the declared return type.
unsafe fn lowerFnDecl 'arena 'phase (
    self: &mut Lowerer 'arena 'phase,
    node: *ast::Node,
    decl: ast::FnDecl,
    functionArena: &mut alloc::Arena
) -> ?*unsafe il::Fn throws (LowerError) where 'arena: 'phase {
    if not shouldLowerFn(&decl, self.options.buildTest) {
        return nil;
    }
    let case ast::NodeValue::Ident(name) = decl.name.value else {
        throw LowerError::ExpectedIdentifier;
    };
    let data = resolver::nodeData(self.resolver, node);
    let case resolver::Type::Fn(fnType) = data.ty else {
        throw LowerError::ExpectedFunction;
    };
    let isExtern = checkAttr(decl.attrs, ast::Attribute::Extern);

    // Build qualified function name for multi-module compilation.
    let qualName = qualifyName(self, nil, name);

    // Register function symbol for cross-package call resolution.
    if let sym = data.sym {
        registerSymbolName(self, sym, qualName);
    }
    let parent: 'function = &mut *self, arena = &mut *functionArena where 'phase: 'function in {
        let mut fnLow = fnLowerer(parent, node, fnType, qualName, arena);

        // If the function returns an aggregate or is throwing, prepend a hidden
        // return parameter. The caller allocates the buffer and passes it
        // as the first argument; the callee writes the return value into it.
        if requiresReturnParam(fnType) and not isExtern {
            set fnLow.returnReg = nextReg(&mut fnLow);
        }
        let lowParams = try lowerParams(&mut fnLow, *fnType, decl.sig.params, nil);
        let func = try! alloc::allocRaw(fnLow.arena, @sizeOf(il::Fn), @alignOf(il::Fn)) as *unsafe mut il::Fn;

        // Throwing functions return a result aggregate (word-sized pointer).
        // TODO: The resolver should set an appropriate type that takes into account
        //       the throws list. It shouldn't set the return type to the "success"
        //       value only.
        let returnType = il::Type::W64 if fnType.throwList.len > 0
            else ilType(fnLow.low, *fnType.returnType);
        set *func = il::Fn {
            name: qualName,
            params: lowParams,
            returnType,
            isExtern,
            isLeaf: true,
            blocks: &[],
        };
        let body = decl.body else {
            // Extern functions have no body.
            assert isExtern;
            return func;
        };
        set func.blocks = try lowerFnBody(&mut fnLow, body);
        set func.isLeaf = fnLow.isLeaf;

        return func;
    }
}

/// Build a qualified name of the form "Type::method".
unsafe fn instanceMethodName 'arena 'phase (self: &mut Lowerer 'arena 'phase, modId: ?u16, typeName: *[u8], methodName: *[u8]) -> *[u8] where 'arena: 'phase {
    let sepLen: u32 = 2; // "::"
    let totalLen = typeName.len + sepLen + methodName.len;
    let buf = try! alloc::allocSlice(self.arena, 1, 1, totalLen) as *mut [u8];
    let mut pos: u32 = 0;

    set pos += try! mem::copy(&mut buf[pos..], typeName);
    set pos += try! mem::copy(&mut buf[pos..], "::");
    set pos += try! mem::copy(&mut buf[pos..], methodName);
    assert pos == totalLen;

    return qualifyName(self, modId, &buf[..totalLen]);
}

/// Build a v-table data name of the form "vtable::Type::Trait".
unsafe fn vtableName 'arena 'phase (self: &mut Lowerer 'arena 'phase, modId: ?u16, typeName: *[u8], traitName: *[u8]) -> *[u8] where 'arena: 'phase {
    let prefix = "vtable::";
    let sepLen: u32 = 2; // "::"
    let totalLen = prefix.len + typeName.len + sepLen + traitName.len;
    let buf = try! alloc::allocSlice(self.arena, 1, 1, totalLen) as *mut [u8];
    let mut pos: u32 = 0;

    set pos += try! mem::copy(&mut buf[pos..], prefix);
    set pos += try! mem::copy(&mut buf[pos..], typeName);
    set pos += try! mem::copy(&mut buf[pos..], "::");
    set pos += try! mem::copy(&mut buf[pos..], traitName);
    assert pos == totalLen;

    return qualifyName(self, modId, &buf[..totalLen]);
}

/// Lower an instance declaration (`instance Trait for Type { ... }`).
///
/// Each method in the instance block is lowered as a standalone function
/// with a qualified name of the form `Type::method`. A read-only v-table
/// data record is emitted containing pointers to these functions, ordered
/// by the trait's method indices. The v-table is later referenced when
/// constructing trait objects for dynamic dispatch.
unsafe fn instanceCursor 'arena 'phase (
    self: &mut Lowerer 'arena 'phase,
    traitNameNode: *ast::Node,
    targetTypeNode: *ast::Node,
    methods: *[*ast::Node]
) -> InstanceCursor throws (LowerError) where 'arena: 'phase {
    // Look up the trait and type from the resolver.
    let traitSym = resolver::nodeData(self.resolver, traitNameNode).sym
        else throw LowerError::MissingSymbol(traitNameNode);
    let case resolver::SymbolData::Trait(traitInfo) = traitSym.data
        else throw LowerError::MissingMetadata;
    let typeSym = resolver::nodeData(self.resolver, targetTypeNode).sym
        else throw LowerError::MissingSymbol(targetTypeNode);

    // Lower each instance method as a regular function.
    // Collect qualified names for the v-table. Empty entries are filled
    // later from inherited supertrait methods.
    return InstanceCursor {
        traitInfo, traitName: traitSym.name, typeName: typeSym.name, methods, next: 0,
        methodNames: undefined, methodNameSet: [false; ast::MAX_TRAIT_METHODS],
    };
}

/// Lower the next concrete method, or finish the instance's v-table.
unsafe fn lowerNextInstance 'arena 'phase (
    self: &mut Lowerer 'arena 'phase,
    state: &mut InstanceCursor,
    functionArena: &mut alloc::Arena
) -> ?*unsafe il::Fn throws (LowerError) where 'arena: 'phase {
    let traitInfo = state.traitInfo;
    let typeName = state.typeName;
    let tName = state.traitName;
    while state.next < state.methods.len {
        let methodNode = state.methods[state.next];
        set state.next += 1;
        let case ast::NodeValue::MethodDecl {
            name, receiverName, sig, body, ..
        } = methodNode.value else continue;

        let case ast::NodeValue::Ident(mName) = name.value else {
            throw LowerError::ExpectedIdentifier;
        };
        let qualName = instanceMethodName(self, nil, typeName, mName);
        let func = try lowerMethod(self, methodNode, qualName, receiverName, sig, body, functionArena)
            else continue;

        let method = resolver::findTraitMethod(traitInfo, mName)
            else panic "lowerInstanceDecl: method not found in trait";

        set state.methodNames[method.index] = qualName;
        set state.methodNameSet[method.index] = true;
        return func;
    }

    // Fill inherited method slots from supertraits.
    // These methods were already lowered as part of the supertrait instance
    // declarations and use the same `Type::method` qualified name.
    for method, i in traitInfo.methods {
        if not state.methodNameSet[i] {
            set state.methodNames[i] = instanceMethodName(self, nil, typeName, method.name);
        }
    }

    // Create v-table in data section, used for dynamic dispatch.
    let vName = vtableName(self, nil, typeName, tName);
    let values = try! alloc::allocSlice(
        self.arena, @sizeOf(il::DataValue), @alignOf(il::DataValue), traitInfo.methods.len as u32
    ) as *mut [il::DataValue];

    for i in 0..traitInfo.methods.len {
        set values[i] = il::DataValue {
            item: il::DataItem::Fn(state.methodNames[i]),
            count: 1,
        };
    }
    self.data.append(il::Data {
        name: vName,
        size: traitInfo.methods.len as u32 * resolver::PTR_SIZE,
        alignment: resolver::PTR_SIZE,
        readOnly: true,
        isZeroInit: false,
        values: &values[..traitInfo.methods.len as u32],
    }, alloc::arenaAllocator(self.arena));
    return nil;
}

/// Lower a method node into an IL function with the given qualified name.
/// Shared by both instance methods and standalone methods.
unsafe fn lowerMethod 'arena 'phase (
    self: &mut Lowerer 'arena 'phase,
    node: *ast::Node,
    qualName: *[u8],
    receiverName: *ast::Node,
    sig: ast::FnSig,
    body: *ast::Node,
    functionArena: &mut alloc::Arena,
) -> ?*unsafe il::Fn throws (LowerError) where 'arena: 'phase {
    let data = resolver::nodeData(self.resolver, node);
    let case resolver::Type::Fn(fnType) = data.ty else {
        throw LowerError::ExpectedFunction;
    };
    let sym = data.sym else throw LowerError::MissingSymbol(node);
    registerSymbolName(self, sym, qualName);

    let parent: 'function = &mut *self, arena = &mut *functionArena where 'phase: 'function in {
        let mut fnLow = fnLowerer(parent, node, fnType, qualName, arena);
        if requiresReturnParam(fnType) {
            set fnLow.returnReg = nextReg(&mut fnLow);
        }
        let lowParams = try lowerParams(&mut fnLow, *fnType, sig.params, receiverName);
        let func = try! alloc::allocRaw(fnLow.arena, @sizeOf(il::Fn), @alignOf(il::Fn)) as *unsafe mut il::Fn;

        let returnType = il::Type::W64 if fnType.throwList.len > 0
            else ilType(fnLow.low, *fnType.returnType);
        set *func = il::Fn {
            name: qualName,
            params: lowParams,
            returnType,
            isExtern: false,
            isLeaf: true,
            blocks: &[],
        };
        set func.blocks = try lowerFnBody(&mut fnLow, body);
        set func.isLeaf = fnLow.isLeaf;

        return func;
    }
}

/// Lower a standalone method declaration.
/// Produces a function with qualified name `Type::method`.
unsafe fn lowerMethodDecl 'arena 'phase (
    self: &mut Lowerer 'arena 'phase,
    node: *ast::Node,
    name: *ast::Node,
    receiverName: *ast::Node,
    sig: ast::FnSig,
    body: *ast::Node,
    functionArena: &mut alloc::Arena,
) -> ?*unsafe il::Fn throws (LowerError) where 'arena: 'phase {
    let sym = resolver::nodeData(self.resolver, node).sym
        else throw LowerError::MissingSymbol(node);
    let case ast::NodeValue::Ident(mName) = name.value
        else throw LowerError::ExpectedIdentifier;
    let me = resolver::findMethodBySymbol(self.resolver, sym)
        else throw LowerError::MissingMetadata;
    let qualName = instanceMethodName(self, nil, me.concreteTypeName, mName);

    return try lowerMethod(self, node, qualName, receiverName, sig, body, functionArena);
}

/// Check if a function should be lowered.
fn shouldLowerFn(decl: &ast::FnDecl, buildTest: bool) -> bool {
    if checkAttr(decl.attrs, ast::Attribute::Test) {
        return buildTest;
    }
    return true;
}

/// Check if a specific attribute is present in the attribute set.
fn checkAttr(attrs: ?ast::Attributes, attr: ast::Attribute) -> bool {
    if let a = attrs {
        return ast::attributesContains(&a, attr);
    }
    return false;
}

/// Create a label with a numeric suffix, eg. `@base0`.
/// This ensures unique labels like `@then0`, `@then1`, etc.
unsafe fn labelWithSuffix 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, base: *[u8], suffix: u32) -> *[u8] throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let mut digits: [u8; fmt::U32_STR_LEN] = undefined;
    let start = fmt::formatU32(suffix, &mut digits[..]);
    let totalLen = base.len + digits.len - start;
    let buf = try! alloc::allocSlice(self.arena, 1, 1, totalLen) as *mut [u8];

    try! mem::copy(&mut buf[..base.len], base);
    try! mem::copy(&mut buf[base.len..totalLen], &digits[start..]);

    return &buf[..totalLen];
}

/// Generate a unique label by appending the global counter to the base.
unsafe fn nextLabel 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, base: *[u8]) -> *[u8] throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let idx = self.labelCounter;
    set self.labelCounter += 1;

    return try labelWithSuffix(self, base, idx);
}

///////////////////////////////
// Data Section Construction //
///////////////////////////////

// Functions for building the data section from constant/static
// declarations and inline literals.

/// Convert a constant integer payload to a signed 64-bit value.
fn constIntToI64(intVal: resolver::ConstInt) -> i64 {
    return (0 - intVal.magnitude as i64) if intVal.negative else intVal.magnitude as i64;
}

/// Convert a scalar constant value to an i64.
/// String constants are not handled here and should be checked before calling.
/// Panics if a non-scalar value is passed.
fn constToScalar(val: resolver::ConstValue) -> i64 {
    match val {
        case resolver::ConstValue::Bool(b) => return 1 if b else 0,
        case resolver::ConstValue::Char(c) => return c as i64,
        case resolver::ConstValue::Int(i) => return constIntToI64(i),
        else => panic,
    }
}

/// Convert a constant value to an IL value.
unsafe fn constValueToVal 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, val: resolver::ConstValue, node: *ast::Node) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    if let case resolver::ConstValue::String(s) = val {
        return try lowerStringLit(self, node, s);
    }
    return il::Val::Imm(constToScalar(val));
}

/// Convert a resolver constant value to an IL data initializer item.
unsafe fn constValueToDataItem 'arena 'phase (self: &mut Lowerer 'arena 'phase, val: resolver::ConstValue, typ: resolver::Type) -> il::DataItem where 'arena: 'phase {
    if let case resolver::ConstValue::String(s) = val {
        return il::DataItem::Str(s);
    }
    // Bool and char are byte-sized; integer uses the declared type.
    let mut irTyp = il::Type::W8;
    if let case resolver::ConstValue::Int(_) = val {
        set irTyp = ilType(self, typ);
    }
    return il::DataItem::Val { typ: irTyp, val: constToScalar(val) };
}

/// Lower scalar-like constant nodes into data values, including fallback handling
/// for void-variant tags and slice string initializers.
unsafe fn lowerConstScalarDataInto 'arena 'phase (
    self: &mut Lowerer 'arena 'phase,
    node: *ast::Node,
    ty: resolver::Type,
    dataPrefix: *[u8],
    b: &mut DataValueBuilder
) throws (LowerError) where 'arena: 'phase {
    let val = resolver::constValueEntry(self.resolver, node) else {
        if let idx = voidVariantIndex(self.resolver, node) {
            dataBuilderPush(b, il::DataValue {
                item: il::DataItem::Val { typ: il::Type::W8, val: idx },
                count: 1
            });
            return;
        }
        throw LowerError::MissingConst(node);
    };

    if let case resolver::ConstValue::String(s) = val {
        if let case resolver::Type::Slice { .. } = ty {
            let strSym = try getOrCreateStringData(self, s, dataPrefix);
            dataSliceHeader(b, strSym, s.len);
            return;
        }
    }
    dataBuilderPush(b, il::DataValue {
        item: constValueToDataItem(self, val, ty),
        count: 1
    });
}

/// Lower a constant or static declaration to the data section.
unsafe fn lowerDataDecl 'arena 'phase (
    self: &mut Lowerer 'arena 'phase,
    node: *ast::Node,
    value: *ast::Node,
    readOnly: bool
) throws (LowerError) where 'arena: 'phase {
    let data = resolver::nodeData(self.resolver, node);
    let sym = data.sym else {
        throw LowerError::MissingSymbol(node);
    };
    if data.ty == resolver::Type::Unknown {
        throw LowerError::MissingType(node);
    }
    let layout = resolver::getTypeLayout(data.ty);
    let qualName = lookupSymbolName(self, sym) else qualifyName(self, nil, sym.name);
    let mut b = dataBuilder(alloc::arenaAllocator(self.arena));
    try lowerConstDataInto(self, value, data.ty, layout.size, qualName, &mut b);
    let result = dataBuilderFinish(b);

    self.data.append(il::Data {
        name: qualName,
        size: layout.size,
        alignment: layout.alignment,
        readOnly,
        isZeroInit: not readOnly and result.zeroInit,
        values: result.values,
    }, alloc::arenaAllocator(self.arena));
}

/// Emit the in-memory representation of a slice header: `{ ptr, len, cap }`.
fn dataSliceHeader(b: &mut DataValueBuilder, dataSym: *[u8], len: u32) {
    dataBuilderPush(b, il::DataValue {
        item: il::DataItem::Sym(dataSym),
        count: 1
    });
    dataBuilderPush(b, il::DataValue {
        item: il::DataItem::Val {
            typ: il::Type::W32,
            val: len as i64
        },
        count: 1
    });
    dataBuilderPush(b, il::DataValue {
        item: il::DataItem::Val {
            typ: il::Type::W32,
            val: len as i64
        },
        count: 1
    });
}

/// Lower a compile-time `&[...]` expression to a concrete slice header.
unsafe fn lowerConstAddressSliceInto 'arena 'phase (
    self: &mut Lowerer 'arena 'phase,
    addr: ast::AddressOf,
    ty: resolver::Type,
    dataPrefix: *[u8],
    b: &mut DataValueBuilder
) throws (LowerError) where 'arena: 'phase {
    let case resolver::Type::Slice { mutable, .. } = ty
        else throw LowerError::ExpectedSliceOrArray;
    let targetTy = resolver::typeFor(self.resolver, addr.target)
        else throw LowerError::MissingType(addr.target);
    let case resolver::Type::Array(arrInfo) = targetTy
        else throw LowerError::ExpectedArray;

    let mut nested = dataBuilder(alloc::arenaAllocator(self.arena));
    let layout = resolver::getTypeLayout(targetTy);
    try lowerConstDataInto(self, addr.target, targetTy, layout.size, dataPrefix, &mut nested);

    let backing = dataBuilderFinish(nested);
    let readOnly = not mutable;
    let mut dataName: *[u8] = undefined;
    if readOnly {
        if let found = findConstData(self, backing.values, layout.alignment) {
            set dataName = found;
        } else {
            set dataName = try pushDeclData(self, layout.size, layout.alignment, readOnly, backing.values, dataPrefix);
        }
    } else {
        set dataName = try pushDeclData(self, layout.size, layout.alignment, readOnly, backing.values, dataPrefix);
    }
    dataSliceHeader(b, dataName, arrInfo.length);
}

/// Lower a constant expression payload into a builder without slot padding.
/// Compute the type layout when nil or undefined data needs a byte count.
unsafe fn lowerConstDataPayloadInto 'arena 'phase (
    self: &mut Lowerer 'arena 'phase,
    node: *ast::Node,
    ty: resolver::Type,
    dataPrefix: *[u8],
    b: &mut DataValueBuilder
) throws (LowerError) where 'arena: 'phase {
    // Optional coercions define the tag and payload layout of constant values.
    if let case resolver::Type::Optional(inner) = ty {
        if let coercion = resolver::coercionFor(self.resolver, node) {
            if let case resolver::Coercion::OptionalLift(target) = coercion {
                if resolver::typesEqual(target, ty) {
                    if resolver::isNullableType(*inner) {
                        try lowerConstDataPayloadInto(self, node, *inner, dataPrefix, b);
                    } else {
                        let layout = resolver::getTypeLayout(ty);
                        let offset = resolver::getOptionalValOffset(*inner);
                        dataBuilderPush(b, il::DataValue {
                            item: il::DataItem::Val { typ: il::Type::W8, val: 1 }, count: 1,
                        });
                        if offset > 1 {
                            dataBuilderPush(b, il::DataValue { item: il::DataItem::Undef, count: offset - 1 });
                        }
                        try lowerConstDataInto(self, node, *inner, layout.size - offset, dataPrefix, b);
                    }
                    return;
                }
            }
        }
    }
    // Function pointer references in constant data.
    if let case resolver::Type::Fn(_) = ty {
        let sym = resolver::nodeData(self.resolver, node).sym
            else throw LowerError::MissingSymbol(node);
        let modId = resolver::moduleIdForSymbol(self.resolver, sym);
        let qualName = qualifyName(self, modId, sym.name);
        dataBuilderPush(b, il::DataValue {
            item: il::DataItem::Fn(qualName), count: 1,
        });
        return;
    }
    // In constant data, a void variant of a mixed union still occupies the
    // full tagged-union slot. Emitting only the tag corrupts following fields.
    if let sym = resolver::nodeData(self.resolver, node).sym {
        if let case resolver::SymbolData::Variant { type: resolver::Type::Void, .. } = sym.data {
            if let case resolver::Type::Nominal(resolver::NominalType::Union(_)) = ty {
                try lowerConstUnionVariantInto(self, node, sym, ty, &mut [], dataPrefix, b);
                return;
            }
        }
    }
    match node.value {
        case ast::NodeValue::Nil => {
            let case resolver::Type::Optional(_) = ty else throw LowerError::NilInNonOptional;
            let layout = resolver::getTypeLayout(ty);
            dataBuilderPush(b, il::DataValue {
                item: il::DataItem::Val { typ: il::Type::W8, val: 0 }, count: layout.size,
            });
        },
        case ast::NodeValue::Undef => {
            let layout = resolver::getTypeLayout(ty);
            dataBuilderPush(b, il::DataValue {
                item: il::DataItem::Undef,
                count: layout.size
            });
        }
        case ast::NodeValue::ArrayLit(elems) =>
            try lowerConstArrayLitInto(self, elems, ty, dataPrefix, b),
        case ast::NodeValue::ArrayRepeatLit(repeat) =>
            try lowerConstArrayRepeatInto(self, repeat, ty, dataPrefix, b),
        case ast::NodeValue::RecordLit(recLit) =>
            try lowerConstRecordLitInto(self, node, recLit, ty, dataPrefix, b),
        case ast::NodeValue::Call(call) => {
            let calleeSym = resolver::nodeData(self.resolver, call.callee).sym
                else throw LowerError::MissingSymbol(call.callee);
            match calleeSym.data {
                case resolver::SymbolData::Variant { .. } =>
                    try lowerConstUnionVariantInto(self, node, calleeSym, ty, call.args, dataPrefix, b),
                case resolver::SymbolData::Type(resolver::NominalType::Record(recInfo)) => {
                    try lowerConstRecordCtorInto(self, call.args, recInfo, dataPrefix, b);
                }
                else => throw LowerError::MissingConst(node)
            }
        }
        case ast::NodeValue::AddressOf(addr) => {
            try lowerConstAddressSliceInto(self, addr, ty, dataPrefix, b);
        }
        case ast::NodeValue::Ident(_) => {
            // Identifier referencing a constant.
            let sym = resolver::nodeData(self.resolver, node).sym
                else throw LowerError::MissingSymbol(node);
            let case ast::NodeValue::ConstDecl(decl) = sym.node.value
                else throw LowerError::MissingConst(node);

            try lowerConstDataPayloadInto(self, decl.value, ty, dataPrefix, b);
        },
        case ast::NodeValue::ScopeAccess(_) => {
            let sym = resolver::nodeData(self.resolver, node).sym
                else throw LowerError::MissingSymbol(node);
            if let case ast::NodeValue::ConstDecl(decl) = sym.node.value {
                try lowerConstDataPayloadInto(self, decl.value, ty, dataPrefix, b);
            } else {
                try lowerConstScalarDataInto(self, node, ty, dataPrefix, b);
            }
        }
        else => {
            // Scalar values: integers, bools, strings, void union variants, etc.
            try lowerConstScalarDataInto(self, node, ty, dataPrefix, b);
        }
    }
}

/// Lower a constant expression into a builder, padding to the given slot size.
unsafe fn lowerConstDataInto 'arena 'phase (
    self: &mut Lowerer 'arena 'phase,
    node: *ast::Node,
    ty: resolver::Type,
    slotSize: u32,
    dataPrefix: *[u8],
    b: &mut DataValueBuilder
) throws (LowerError) where 'arena: 'phase {
    let layout = resolver::getTypeLayout(ty);
    try lowerConstDataPayloadInto(self, node, ty, dataPrefix, b);
    // Pad to fill the enclosing slot.
    let padding = slotSize - layout.size;
    if padding > 0 {
        dataBuilderPush(b, il::DataValue { item: il::DataItem::Undef, count: padding });
    }
}

/// Flatten a constant array literal `[a, b, c]` into a builder.
/// Each element payload fills its type size; no extra slot padding is needed.
unsafe fn lowerConstArrayLitInto 'arena 'phase (
    self: &mut Lowerer 'arena 'phase,
    elems: *[*ast::Node],
    ty: resolver::Type,
    dataPrefix: *[u8],
    b: &mut DataValueBuilder
) throws (LowerError) where 'arena: 'phase {
    let case resolver::Type::Array(arrInfo) = ty
        else throw LowerError::ExpectedArray;
    let elemTy = *arrInfo.item;

    for elem in elems {
        try lowerConstDataPayloadInto(self, elem, elemTy, dataPrefix, b);
    }
}

/// Build data values for a constant array repeat literal `[item; count]`.
/// Repeat element payloads without extra slot padding. Undefined data uses
/// the element layout to compute the total byte count.
unsafe fn lowerConstArrayRepeatInto 'arena 'phase (
    self: &mut Lowerer 'arena 'phase,
    repeat: ast::ArrayRepeatLit,
    ty: resolver::Type,
    dataPrefix: *[u8],
    b: &mut DataValueBuilder
) throws (LowerError) where 'arena: 'phase {
    let case resolver::Type::Array(arrInfo) = ty
        else throw LowerError::ExpectedArray;
    let length = arrInfo.length;
    let elemTy = *arrInfo.item;

    if let case resolver::Type::Optional(_) = elemTy {
        for _ in 0..length {
            try lowerConstDataPayloadInto(self, repeat.item, elemTy, dataPrefix, b);
        }
        return;
    }

    if let case ast::NodeValue::Undef = repeat.item.value {
        let elemLayout = resolver::getTypeLayout(elemTy);
        dataBuilderPush(b, il::DataValue {
            item: il::DataItem::Undef,
            count: elemLayout.size * length
        });
    } else if let val = resolver::constValueEntry(self.resolver, repeat.item) {
        if let case resolver::ConstValue::String(_) = val {
            // A string used as a slice is represented by a three-word slice
            // header, not by the bytes of the string itself.
            for _ in 0..length {
                try lowerConstDataPayloadInto(self, repeat.item, elemTy, dataPrefix, b);
            }
        } else {
            dataBuilderPush(b, il::DataValue {
                item: constValueToDataItem(self, val, elemTy),
                count: length
            });
        }
    } else {
        for _ in 0..length {
            try lowerConstDataPayloadInto(self, repeat.item, elemTy, dataPrefix, b);
        }
    }
}

/// Build data values for a constant record literal.
/// Each field is lowered with a slot size that includes trailing padding.
unsafe fn lowerConstRecordLitInto 'arena 'phase (
    self: &mut Lowerer 'arena 'phase,
    node: *ast::Node,
    recLit: ast::RecordLit,
    ty: resolver::Type,
    dataPrefix: *[u8],
    b: &mut DataValueBuilder
) throws (LowerError) where 'arena: 'phase {
    match ty {
        case resolver::Type::Nominal(resolver::NominalType::Record(recInfo)) => {
            try lowerConstRecordCtorInto(self, recLit.fields, recInfo, dataPrefix, b);
        }
        case resolver::Type::Nominal(resolver::NominalType::Union(_)) => {
            let typeName = recLit.typeName else {
                throw LowerError::ExpectedVariant;
            };
            let sym = resolver::nodeData(self.resolver, typeName).sym else {
                throw LowerError::MissingSymbol(typeName);
            };
            try lowerConstUnionVariantInto(self, node, sym, ty, recLit.fields, dataPrefix, b);
        }
        else => throw LowerError::ExpectedRecord,
    }
}

/// Build data values for record constants.
unsafe fn lowerConstRecordCtorInto 'arena 'phase (
    self: &mut Lowerer 'arena 'phase,
    args: *[*ast::Node],
    recInfo: resolver::RecordType,
    dataPrefix: *[u8],
    b: &mut DataValueBuilder
) throws (LowerError) where 'arena: 'phase {
    let layout = *recInfo.layout;
    for argNode, i in args {
        let mut valueNode = argNode;
        if let case ast::NodeValue::RecordLitField(fieldLit) = argNode.value {
            set valueNode = fieldLit.value;
        }
        let fieldInfo = recInfo.fields[i];
        let fieldOffset = fieldInfo.offset as u32;

        // Slot extends to the next field's offset,
        // or record size for the last field.
        let slotEnd = recInfo.fields[i + 1].offset as u32 if i + 1 < recInfo.fields.len else layout.size;
        let slotSize = slotEnd - fieldOffset;

        try lowerConstDataInto(self, valueNode, fieldInfo.fieldType, slotSize, dataPrefix, b);
    }
}

/// Build data values for a constant union variant value from payload fields/args.
unsafe fn lowerConstUnionVariantInto 'arena 'phase (
    self: &mut Lowerer 'arena 'phase,
    node: *ast::Node,
    variantSym: *unsafe mut resolver::Symbol,
    ty: resolver::Type,
    payloadArgs: *[*ast::Node],
    dataPrefix: *[u8],
    b: &mut DataValueBuilder
) throws (LowerError) where 'arena: 'phase {
    let case resolver::SymbolData::Variant { ordinal, index, .. } = variantSym.data
        else throw LowerError::UnexpectedNodeValue(node);

    let unionInfo = unionInfoFromType(ty) else {
        throw LowerError::MissingMetadata;
    };
    let payloadType = unionInfo.variants[ordinal].valueType;
    let unionLayout = resolver::getTypeLayout(ty);
    let payloadSlotSize = unionLayout.size - unionInfo.valOffset;

    // Tag byte.
    dataBuilderPush(b, il::DataValue {
        item: il::DataItem::Val {
            typ: il::Type::W8,
            val: index as i64
        },
        count: 1
    });
    // Padding between tag and payload.
    if unionInfo.valOffset > 1 {
        dataBuilderPush(b, il::DataValue {
            item: il::DataItem::Undef,
            count: unionInfo.valOffset - 1
        });
    }
    if payloadType == resolver::Type::Void {
        if payloadSlotSize > 0 {
            dataBuilderPush(b, il::DataValue {
                item: il::DataItem::Undef,
                count: payloadSlotSize
            });
        }
        return;
    }

    let case resolver::Type::Nominal(resolver::NominalType::Record(payloadRec)) = payloadType else {
        throw LowerError::ExpectedRecord;
    };
    let payloadLayout = *payloadRec.layout;
    try lowerConstRecordCtorInto(self, payloadArgs, payloadRec, dataPrefix, b);

    // Unused bytes in the union payload slot for smaller variants.
    if payloadSlotSize > payloadLayout.size {
        dataBuilderPush(b, il::DataValue {
            item: il::DataItem::Undef,
            count: payloadSlotSize - payloadLayout.size
        });
    }
}

/// Find an existing string data entry with matching content.
// TODO: Optimize with hash table or remove?
fn findStringData 'arena 'phase (self: &Lowerer 'arena 'phase, s: *[u8]) -> ?*[u8] where 'arena: 'phase {
    for d in &self.data[self.packageDataStart..] {
        if d.values.len == 1 {
            if let case il::DataItem::Str(existing) = d.values[0].item {
                if mem::eq(existing, s) {
                    return d.name;
                }
            }
        }
    }
    return nil;
}

/// Compose a segmented symbol name from a list of path segments.
/// Example: `["func", "nominal", "VALUE"]` -> `func$nominal$VALUE`.
unsafe fn buildSegmentedName 'arena 'phase (
    self: &mut Lowerer 'arena 'phase,
    segments: &[*[u8]]
) -> *[u8] throws (LowerError) where 'arena: 'phase {
    assert segments.len > 0;

    let mut totalLen: u32 = 0;
    for segment in segments {
        set totalLen += segment.len;
    }
    if segments.len > 1 {
        set totalLen += segments.len - 1;
    }
    let buf = try! alloc::allocSlice(self.arena, 1, 1, totalLen) as *mut [u8];
    let mut pos: u32 = 0;

    for segment, i in segments {
        set pos += try! mem::copy(&mut buf[pos..], segment);
        if i + 1 <> segments.len {
            set buf[pos] = '$';
            set pos += 1;
        }
    }
    assert pos == totalLen;

    return &buf[..totalLen];
}

/// Generate a unique name for declaration-local backing data entries.
unsafe fn nextDeclDataName 'arena 'phase (
    self: &mut Lowerer 'arena 'phase,
    prefix: *[u8],
    count: u32,
    namespace: *[u8]
) -> *[u8] throws (LowerError) where 'arena: 'phase {
    let mut digits: [u8; fmt::U32_STR_LEN] = undefined;
    let start = fmt::formatU32(count, &mut digits[..]);
    let suffix = try! alloc::allocSlice(self.arena, 1, 1, digits.len - start) as *mut [u8];
    try! mem::copy(suffix, &digits[start..]);
    let segments = [prefix, namespace, suffix];

    return try buildSegmentedName(self, &segments[..]);
}

/// Append a data entry using a function-local literal namespace (`prefix$literal$N`).
unsafe fn pushDeclData 'arena 'phase (
    self: &mut Lowerer 'arena 'phase,
    size: u32,
    alignment: u32,
    readOnly: bool,
    values: *[il::DataValue],
    dataPrefix: *[u8]
) -> *[u8] throws (LowerError) where 'arena: 'phase {
    let dataCount = self.data.len;
    let name = try nextDeclDataName(self, dataPrefix, dataCount, "literal");
    self.data.append(il::Data {
        name,
        size,
        alignment,
        readOnly,
        isZeroInit: not readOnly and dataValuesAreZeroInit(values),
        values,
    }, alloc::arenaAllocator(self.arena));

    return name;
}

/// Find or create read-only string data and return its symbol name.
unsafe fn getOrCreateStringData 'arena 'phase (
    self: &mut Lowerer 'arena 'phase,
    s: *[u8],
    dataPrefix: *[u8]
) -> *[u8] throws (LowerError) where 'arena: 'phase {
    if let existing = findStringData(self, s) {
        return existing;
    }
    let values = try! alloc::allocSlice(
        self.arena, @sizeOf(il::DataValue), @alignOf(il::DataValue), 1
    ) as *mut [il::DataValue];

    set values[0] = il::DataValue {
        item: il::DataItem::Str(s),
        count: 1
    };
    return try pushDeclData(self, s.len, 1, true, &values[..1], dataPrefix);
}

/// Compare two data items for structural equality.
/// Unlike raw byte comparison, this correctly ignores padding bytes in unions.
fn dataItemEq(a: il::DataItem, b: il::DataItem) -> bool {
    match a {
        case il::DataItem::Val { typ: aTyp, val: aVal } =>
            if let case il::DataItem::Val { typ: bTyp, val: bVal } = b {
                return aTyp == bTyp and aVal == bVal;
            } else {
                return false;
            },
        case il::DataItem::Sym(aPtr) =>
            if let case il::DataItem::Sym(bPtr) = b {
                return mem::eq(aPtr, bPtr);
            } else {
                return false;
            },
        case il::DataItem::Fn(aName) =>
            if let case il::DataItem::Fn(bName) = b {
                return mem::eq(aName, bName);
            } else {
                return false;
            },
        case il::DataItem::Str(aStr) =>
            if let case il::DataItem::Str(bStr) = b {
                return mem::eq(aStr, bStr);
            } else {
                return false;
            },
        case il::DataItem::Undef =>
            if let case il::DataItem::Undef = b {
                return true;
            } else {
                return false;
            },
    }
}

/// Compare two data value slices for structural equality.
fn dataValuesEq(a: &[il::DataValue], b: &[il::DataValue]) -> bool {
    if a.len <> b.len {
        return false;
    }
    for i in 0..a.len {
        if a[i].count <> b[i].count {
            return false;
        }
        if not dataItemEq(a[i].item, b[i].item) {
            return false;
        }
    }
    return true;
}

/// Find an existing read-only slice data entry with matching values.
// TODO: Optimize with hash table or remove?
fn findSliceData 'arena 'phase (self: &Lowerer 'arena 'phase, values: *[il::DataValue], alignment: u32) -> ?*[u8] where 'arena: 'phase {
    for d in &self.data[self.packageDataStart..] {
        if d.alignment == alignment and d.readOnly and dataValuesEq(d.values, values) {
            return d.name;
        }
    }
    return nil;
}

/// Find existing constant data entry with matching content.
/// Handles both string data and slice data.
fn findConstData 'arena 'phase (self: &Lowerer 'arena 'phase, values: *[il::DataValue], alignment: u32) -> ?*[u8] where 'arena: 'phase {
    // Fast path for strings.
    if values.len == 1 and alignment == 1 {
        if let case il::DataItem::Str(s) = values[0].item {
            return findStringData(self, s);
        }
    }
    // General case: byte comparison of data values.
    return findSliceData(self, values, alignment);
}

/// Lower constant data to a slice value.
/// Creates or reuses a data section entry, then builds a slice header on the stack.
unsafe fn lowerConstDataAsSlice 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    result: &ConstDataResult,
    alignment: u32,
    readOnly: bool,
    elemTy: *resolver::Type,
    mutable: bool,
    length: u32
) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let values = result.values;
    let elemLayout = resolver::getTypeLayout(*elemTy);
    let size = elemLayout.size * length;
    let mut dataName: *[u8] = undefined;
    let mut found: ?*[u8] = nil;
    if readOnly {
        set found = findConstData(self.low, values, alignment);
    }
    if let name = found {
        set dataName = name;
    } else {
        set dataName = try nextDataName(self);
        self.low.data.append(il::Data {
            name: dataName,
            size,
            alignment,
            readOnly,
            isZeroInit: not readOnly and result.zeroInit,
            values,
        }, alloc::arenaAllocator(self.low.arena));
    }

    // Get data address.
    let ptrReg = nextReg(self);
    emit(self, il::Instr::Copy { dst: ptrReg, val: il::Val::DataSym(dataName) });

    return try buildSliceValue(
        self, elemTy, mutable, il::Val::Reg(ptrReg), il::Val::Imm(length as i64), il::Val::Imm(length as i64)
    );
}

/// Generate a unique data name for inline literals, eg. `fnName$literal$N`.
unsafe fn nextDataName 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function) -> *[u8] throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let counter = self.dataCounter;
    set self.dataCounter += 1;
    let fnName = self.fnName;
    return try nextDeclDataName(self.low, fnName, counter, "literal");
}

/// Assign a unique function-local data symbol name.
unsafe fn registerLocalDataDeclName 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let sym = resolver::nodeData(self.low.resolver, node).sym
        else throw LowerError::MissingSymbol(node);

    let prefix = self.fnName;
    let segments = [prefix, "nominal", sym.name];
    let name = try buildSegmentedName(self.low, &segments[..]);

    let qualified = qualifyName(self.low, nil, name);
    registerSymbolName(self.low, sym, qualified);
}

/// Get the next available SSA register.
fn nextReg 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function) -> il::Reg where 'arena: 'phase, 'phase: 'function {
    let reg = il::Reg { n: self.regCounter };
    set self.regCounter += 1;
    return reg;
}

/// Look up the resolved type of an AST node, or throw `MissingType`.
unsafe fn typeOf 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node) -> resolver::Type throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let ty = resolver::typeFor(self.low.resolver, node)
        else throw LowerError::MissingType(node);
    return ty;
}

/// Look up the symbol for an AST node, or throw `MissingSymbol`.
unsafe fn symOf 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node) -> *unsafe mut resolver::Symbol throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let sym = resolver::nodeData(self.low.resolver, node).sym
        else throw LowerError::MissingSymbol(node);
    return sym;
}

/// Remove the last block parameter and its associated variable.
/// Used when detecting a trivial phi that can be eliminated.
unsafe fn removeLastBlockParam 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, block: BlockId) where 'arena: 'phase, 'phase: 'function {
    let blk = getBlockMut(self, block);
    if blk.params.len > 0 {
        // TODO: Use `pop`?
        set blk.params = @sliceOf(blk.params.ptr, blk.params.len - 1, blk.params.cap);
    }
    if blk.paramVars.len > 0 {
        // TODO: Use `pop`?
        set blk.paramVars = @sliceOf(blk.paramVars.ptr, blk.paramVars.len - 1, blk.paramVars.cap);
    }
}

/// Rewrite cached SSA values for a variable across all blocks, and also
/// rewrite any terminator arguments that reference the provisional register.
/// The latter is necessary because recursive SSA resolution may have already
/// patched terminator arguments with the provisional value before it was
/// found to be trivial.
unsafe fn rewriteCachedVarValue 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, v: Var, from: il::Val, to: il::Val) where 'arena: 'phase, 'phase: 'function {
    for i in 0..self.blockData.len {
        let blk = getBlockMut(self, BlockId(i));
        if blk.vars[*v] == from {
            set blk.vars[*v] = to;
        }
        if blk.instrs.len > 0 {
            let ix = blk.instrs.len - 1;
            match &mut blk.instrs[ix] {
                case il::Instr::Jmp { args, .. } =>
                    rewriteValInSlice(*args, from, to),
                case il::Instr::Br { thenArgs, elseArgs, .. } => {
                    rewriteValInSlice(*thenArgs, from, to);
                    rewriteValInSlice(*elseArgs, from, to);
                }
                case il::Instr::Switch { defaultArgs, cases, .. } => {
                    rewriteValInSlice(*defaultArgs, from, to);
                    for j in 0..cases.len {
                        rewriteValInSlice(cases[j].args, from, to);
                    }
                }
                else => {}
            }
        }
    }
}

/// Replace all occurrences of `from` with `to` in an args slice.
unsafe fn rewriteValInSlice(args: *unsafe mut [il::Val], from: il::Val, to: il::Val) {
    for i in 0..args.len {
        if args[i] == from {
            set args[i] = to;
        }
    }
}

////////////////////////////
// Basic Block Management //
////////////////////////////

// Basic blocks are the fundamental unit of the CFG. Each block contains a
// sequence of instructions ending in a terminator (jump, branch, return).
//
// The block management API supports forward references -- you can create a
// block before switching to it and emitting instructions.
// This is essential for control flow where we need to reference target blocks
// before we've built them (e.g., the "else" block when building "then").
//
// Sealing is a key concept for SSA construction: a block is sealed once all
// its predecessor edges are known. Before sealing, we can't resolve variable
// uses that require looking up values from predecessors.

/// Create a new basic block with the given label base.
///
/// The block is initially unsealed (predecessors may be added later) and empty.
/// Returns a [`BlockId`] that can be used for jumps and branches. The block must
/// be switched to via [`switchToBlock`] before instructions can be emitted.
unsafe fn createBlock 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, labelBase: *[u8]) -> BlockId throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let label = try nextLabel(self, labelBase);
    let id = BlockId(self.blockData.len);
    let varCount = self.localCount;
    let vars = try! alloc::allocRawSlice(self.arena, @sizeOf(?il::Val), @alignOf(?il::Val), varCount) as *unsafe mut [?il::Val];

    for i in 0..varCount {
        set vars[i] = nil;
    }
    self.blockData.append(BlockData {
        label,
        params: &mut [],
        paramVars: &mut [],
        instrs: &mut [],
        locs: &mut [],
        preds: &mut [],
        vars,
        sealState: Sealed::No,
        loopDepth: self.loopDepth,
    }, alloc::arenaAllocator(self.arena));

    return id;
}

/// Create a new block with a single parameter.
unsafe fn createBlockWithParam 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    labelBase: *[u8],
    param: il::Param
) -> BlockId throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let block = try createBlock(self, labelBase);
    let blk = getBlockMut(self, block);
    blk.params.append(param, alloc::arenaAllocator(self.arena));

    return block;
}

/// Switch to building a different block.
/// All subsequent `emit` calls will add instructions to this block.
fn switchToBlock 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, block: BlockId) where 'arena: 'phase, 'phase: 'function {
    set self.currentBlock = block;
}

/// Seal a block, indicating all predecessor edges are now known.
///
/// Sealing enables SSA construction to resolve variable uses by looking up
/// values from predecessors and inserting block parameters as needed. It
/// does not prevent instructions from being added to the block.
unsafe fn sealBlock 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, block: BlockId) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let blk = getBlockMut(self, block);
    let case Sealed::No = blk.sealState else {
        return; // Already sealed.
    };
    // Keep the current parameter list. Resolution can add more parameters.
    let paramVars = blk.paramVars;
    set blk.sealState = Sealed::Yes;

    // Complete each parameter that was created before sealing.
    for varId, paramIdx in paramVars {
        try resolveBlockArgs(self, block, Var(varId), paramIdx);
    }
}

/// Seal a block and switch to it.
unsafe fn switchToAndSeal 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, block: BlockId) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    try sealBlock(self, block);
    switchToBlock(self, block);
}

/// Get block data by block id.
unsafe fn getBlock 'arena 'phase 'function (self: &FnLowerer 'arena 'phase 'function, block: BlockId) -> *unsafe BlockData where 'arena: 'phase, 'phase: 'function {
    return &self.blockData[*block];
}

/// Get mutable block data by block id.
unsafe fn getBlockMut 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, block: BlockId) -> *unsafe mut BlockData where 'arena: 'phase, 'phase: 'function {
    return &mut self.blockData[*block];
}

/// Get the current block being built.
fn currentBlock 'arena 'phase 'function (self: &FnLowerer 'arena 'phase 'function) -> BlockId where 'arena: 'phase, 'phase: 'function {
    let block = self.currentBlock else {
        panic "currentBlock: no current block";
    };
    return block;
}

//////////////////////////
// Instruction Emission //
//////////////////////////

/// Emit an instruction to the current block.
unsafe fn emit 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, instr: il::Instr) where 'arena: 'phase, 'phase: 'function {
    let blk = self.currentBlock else panic;
    let mut block = getBlockMut(self, blk);

    // Track whether this function is a leaf.
    if il::isCall(instr) {
        set self.isLeaf = false;
    }
    // Record source location alongside instruction when enabled.
    if self.low.options.debug {
        block.locs.append(self.srcLoc, alloc::arenaAllocator(self.arena));
    }
    block.instrs.append(instr, alloc::arenaAllocator(self.arena));
}

/// Emit an unconditional jump to `target`.
unsafe fn emitJmp 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, target: BlockId) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    emit(self, il::Instr::Jmp { target: *target, args: &mut [] });
    addPredecessor(self, target, currentBlock(self));
}

/// Emit an unconditional jump to `target` with a single argument.
unsafe fn emitJmpWithArg 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, target: BlockId, arg: il::Val) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let args = try allocVal(self, arg);
    emit(self, il::Instr::Jmp { target: *target, args });
    addPredecessor(self, target, currentBlock(self));
}

/// Emit an unconditional jump to `target` and switch to it.
unsafe fn switchAndJumpTo 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, target: BlockId) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    try emitJmp(self, target);
    switchToBlock(self, target);
}

/// Emit a conditional branch based on `cond`.
unsafe fn emitBr 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, cond: il::Reg, thenBlock: BlockId, elseBlock: BlockId) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    assert thenBlock <> elseBlock;
    emit(self, il::Instr::Br {
        op: il::CmpOp::Ne,
        typ: il::Type::W32,
        a: il::Val::Reg(cond),
        b: il::Val::Imm(0),
        thenTarget: *thenBlock,
        thenArgs: &mut [],
        elseTarget: *elseBlock,
        elseArgs: &mut [],
    });
    addPredecessor(self, thenBlock, currentBlock(self));
    addPredecessor(self, elseBlock, currentBlock(self));
}

/// Emit a compare-and-branch instruction with the given comparison op.
unsafe fn emitBrCmp 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    op: il::CmpOp,
    typ: il::Type,
    a: il::Val,
    b: il::Val,
    thenBlock: BlockId,
    elseBlock: BlockId
) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    assert thenBlock <> elseBlock;
    emit(self, il::Instr::Br {
        op, typ, a, b,
        thenTarget: *thenBlock, thenArgs: &mut [],
        elseTarget: *elseBlock, elseArgs: &mut [],
    });
    addPredecessor(self, thenBlock, currentBlock(self));
    addPredecessor(self, elseBlock, currentBlock(self));
}

/// Emit a guard that traps with `ebreak` when a comparison is false.
unsafe fn emitTrapUnlessCmp 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    op: il::CmpOp,
    typ: il::Type,
    a: il::Val,
    b: il::Val
) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let passBlock = try createBlock(self, "guard#pass");
    let trapBlock = try createBlock(self, "guard#trap");

    try emitBrCmp(self, op, typ, a, b, passBlock, trapBlock);
    try switchToAndSeal(self, trapBlock);

    emit(self, il::Instr::Ebreak);
    emit(self, il::Instr::Unreachable);

    try switchToAndSeal(self, passBlock);
}

/// Return whether two IL values are known `u32` immediates satisfying `a <= b`.
fn isLtEq(a: il::Val, b: il::Val) -> bool {
    if a == b {
        return true;
    }
    if a == il::Val::Imm(0) {
        return true;
    }
    let case il::Val::Imm(aa) = a else {
        return false;
    };
    let case il::Val::Imm(bb) = b else {
        return false;
    };
    if aa < 0 or bb < 0 {
        return false;
    }
    return aa <= bb;
}

/// Emit an `ebreak` when `a < b` holds.
unsafe fn emitTrapIfLt 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    typ: il::Type,
    a: il::Val,
    b: il::Val
) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let trapBlock = try createBlock(self, "guard#trap");
    let passBlock = try createBlock(self, "guard#pass");

    try emitBrCmp(self, il::CmpOp::Ult, typ, a, b, trapBlock, passBlock);
    try switchToAndSeal(self, trapBlock);

    emit(self, il::Instr::Ebreak);
    emit(self, il::Instr::Unreachable);

    try switchToAndSeal(self, passBlock);
}

/// Emit a conditional branch. Uses fused compare-and-branch for simple scalar
/// comparisons, falls back to separate comparison plus branch otherwise.
unsafe fn emitCondBranch 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    cond: *ast::Node,
    thenBlock: BlockId,
    elseBlock: BlockId
) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    // Try fused compare-and-branch for simple scalar comparisons.
    if let case ast::NodeValue::BinOp(binop) = cond.value {
        let leftTy = try typeOf(self, binop.left);
        let rightTy = try typeOf(self, binop.right);
        if not isAggregateType(leftTy) and not isAggregateType(rightTy) {
            let operandTy = scalarComparisonType(leftTy, rightTy);
            let unsigned = isUnsignedType(operandTy);
            if let op = cmpOpFrom(binop.op, unsigned) {
                let a = try lowerExpr(self, binop.left);
                let b = try lowerExpr(self, binop.right);
                let typ = ilType(self.low, operandTy);

                // Swap operands if needed.
                match binop.op {
                    case ast::BinaryOp::Gt => // `a > b` = `b < a`
                        try emitBrCmp(self, op, typ, b, a, thenBlock, elseBlock),
                    case ast::BinaryOp::Lte => // `a <= b` = `!(b < a)`
                        try emitBrCmp(self, op, typ, b, a, elseBlock, thenBlock),
                    case ast::BinaryOp::Gte => // `a >= b` = `!(a < b)`
                        try emitBrCmp(self, op, typ, a, b, elseBlock, thenBlock),
                    else =>
                        try emitBrCmp(self, op, typ, a, b, thenBlock, elseBlock),
                }
                return;
            }
        }
    }
    // Fallback: evaluate condition and emit boolean branch.
    let condVal = try lowerExpr(self, cond);
    let condReg = emitValToReg(self, condVal);

    try emitBr(self, condReg, thenBlock, elseBlock);
}

/// Emit a 32-bit store instruction at the given offset.
unsafe fn emitStoreW32At 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, src: il::Val, dst: il::Reg, offset: i32) where 'arena: 'phase, 'phase: 'function {
    emit(self, il::Instr::Store { typ: il::Type::W32, src, dst, offset });
}

/// Emit a 32-bit load instruction at the given offset.
unsafe fn emitLoadW32At 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, dst: il::Reg, src: il::Reg, offset: i32) where 'arena: 'phase, 'phase: 'function {
    emit(self, il::Instr::Load { typ: il::Type::W32, dst, src, offset });
}

/// Emit an 8-bit store instruction at the given offset.
unsafe fn emitStoreW8At 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, src: il::Val, dst: il::Reg, offset: i32) where 'arena: 'phase, 'phase: 'function {
    emit(self, il::Instr::Store { typ: il::Type::W8, src, dst, offset });
}

/// Emit an 8-bit load instruction at the given offset.
unsafe fn emitLoadW8At 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, dst: il::Reg, src: il::Reg, offset: i32) where 'arena: 'phase, 'phase: 'function {
    emit(self, il::Instr::Load { typ: il::Type::W8, dst, src, offset });
}

/// Emit a 64-bit store instruction at the given offset.
unsafe fn emitStoreW64At 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, src: il::Val, dst: il::Reg, offset: i32) where 'arena: 'phase, 'phase: 'function {
    emit(self, il::Instr::Store { typ: il::Type::W64, src, dst, offset });
}

/// Emit a 64-bit load instruction at the given offset.
unsafe fn emitLoadW64At 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, dst: il::Reg, src: il::Reg, offset: i32) where 'arena: 'phase, 'phase: 'function {
    emit(self, il::Instr::Load { typ: il::Type::W64, dst, src, offset });
}

/// Load a tag from memory at `src` plus `offset` with the given IL type.
unsafe fn loadTag 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, src: il::Reg, offset: i32, tagType: il::Type) -> il::Val where 'arena: 'phase, 'phase: 'function {
    let dst = nextReg(self);
    emit(self, il::Instr::Load { typ: tagType, dst, src, offset });
    return il::Val::Reg(dst);
}

/// Load the data pointer from a slice value.
unsafe fn loadSlicePtr 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, sliceReg: il::Reg) -> il::Reg where 'arena: 'phase, 'phase: 'function {
    let ptrReg = nextReg(self);
    emitLoadW64At(self, ptrReg, sliceReg, SLICE_PTR_OFFSET);
    return ptrReg;
}

/// Load the length from a slice value.
unsafe fn loadSliceLen 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, sliceReg: il::Reg) -> il::Val where 'arena: 'phase, 'phase: 'function {
    let lenReg = nextReg(self);
    emitLoadW32At(self, lenReg, sliceReg, SLICE_LEN_OFFSET);
    return il::Val::Reg(lenReg);
}

/// Load the capacity from a slice value.
unsafe fn loadSliceCap 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, sliceReg: il::Reg) -> il::Val where 'arena: 'phase, 'phase: 'function {
    let capReg = nextReg(self);
    emitLoadW32At(self, capReg, sliceReg, SLICE_CAP_OFFSET);
    return il::Val::Reg(capReg);
}

/// Emit a load instruction for a scalar value at `src` plus `offset`.
/// For reading values that may be aggregates, use `emitRead` instead.
unsafe fn emitLoad 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, src: il::Reg, offset: i32, typ: resolver::Type) -> il::Val where 'arena: 'phase, 'phase: 'function {
    let dst = nextReg(self);
    let ilTyp = ilType(self.low, typ);

    if isSignedType(typ) {
        emit(self, il::Instr::Sload { typ: ilTyp, dst, src, offset });
    } else {
        emit(self, il::Instr::Load { typ: ilTyp, dst, src, offset });
    }
    return il::Val::Reg(dst);
}

/// Read a value from memory at `src` plus `offset`. Aggregates are represented
/// as pointers, so we return the address directly. Scalars are loaded via [`emitLoad`].
unsafe fn emitRead 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, src: il::Reg, offset: i32, typ: resolver::Type) -> il::Val where 'arena: 'phase, 'phase: 'function {
    if isAggregateType(typ) {
        let ptr = emitPtrOffset(self, src, offset);
        return il::Val::Reg(ptr);
    }
    return emitLoad(self, src, offset, typ);
}

/// Emit a copy instruction that loads a data symbol's address into a register.
unsafe fn emitDataAddr 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, sym: *unsafe resolver::Symbol) -> il::Reg where 'arena: 'phase, 'phase: 'function {
    let dst = nextReg(self);
    let modId = resolver::moduleIdForSymbol(self.low.resolver, sym);
    let qualName = lookupSymbolName(self.low, sym) else qualifyName(self.low, modId, sym.name);

    emit(self, il::Instr::Copy { dst, val: il::Val::DataSym(qualName) });

    return dst;
}

/// Emit a copy instruction that loads a function's address into a register.
unsafe fn emitFnAddr 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, sym: *unsafe resolver::Symbol) -> il::Reg where 'arena: 'phase, 'phase: 'function {
    let dst = nextReg(self);
    let modId = resolver::moduleIdForSymbol(self.low.resolver, sym);
    let qualName = qualifyName(self.low, modId, sym.name);

    emit(self, il::Instr::Copy { dst, val: il::Val::FnAddr(qualName) });

    return dst;
}

/// Emit pattern tests for a single pattern.
unsafe fn emitPatternMatch 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    subject: &MatchSubject,
    pattern: *ast::Node,
    matchBlock: BlockId,
    fallthrough: BlockId
) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    // Wildcards always match; array patterns are tested element-by-element
    // during binding, so they also unconditionally enter the match block.
    if isWildcardPattern(pattern) {
        try emitJmp(self, matchBlock);
        return;
    }
    if let case ast::NodeValue::ArrayLit(_) = pattern.value {
        try emitJmp(self, matchBlock);
        return;
    }
    if let case resolver::Type::Nominal(resolver::NominalType::Record(_)) = subject.type {
        match pattern.value {
            case ast::NodeValue::RecordLit(_), ast::NodeValue::Call(_) => {
                try emitJmp(self, matchBlock);
                return;
            }
            else => {
            },
        }
    }
    let isNil = pattern.value == ast::NodeValue::Nil;

    match subject.kind {
        case MatchSubjectKind::OptionalPtr if isNil => {
            // Null pointer optimization: branch on the data pointer being null.
            let nilReg = try optionalNilReg(self, subject.val, subject.type);
            try emitBrCmp(self, il::CmpOp::Eq, il::Type::W64, il::Val::Reg(nilReg), il::Val::Imm(0), matchBlock, fallthrough);
        }
        case MatchSubjectKind::OptionalAggregate => {
            let base = emitValToReg(self, subject.val);

            if isNil { // Optional aggregate: `nil` means tag is zero.
                let tagReg = tvalTagReg(self, base);
                try emitBr(self, tagReg, fallthrough, matchBlock);
            } else {
                // `lowerExpr` applies the resolver's optional-lift coercion,
                // materializing the tagged representation for value patterns.
                let pattVal = try lowerExpr(self, pattern);
                let pattReg = emitValToReg(self, pattVal);
                let eq = try lowerOptionalEq(self, subject.bindType, base, pattReg, 0);
                let eqReg = emitValToReg(self, eq);

                try emitBr(self, eqReg, matchBlock, fallthrough);
            }
        }
        case MatchSubjectKind::Union(unionInfo) => {
            assert not isNil;

            let case resolver::NodeExtra::UnionVariant { tag: variantTag, .. } =
                resolver::nodeData(self.low.resolver, pattern).extra
            else {
                throw LowerError::ExpectedVariant;
            };
            // Void unions are passed by value (the tag itself).
            // Non-void unions are passed by reference (need to load tag).
            // When matching by reference, always load from the pointer.
            if unionInfo.isAllVoid {
                let mut tagVal = subject.val;
                match subject.by {
                    case resolver::MatchBy::Ref, resolver::MatchBy::MutRef => {
                        let base = emitValToReg(self, subject.val);
                        set tagVal = loadTag(self, base, 0, il::Type::W8);
                    }
                    case resolver::MatchBy::Value => {}
                }
                try emitBrCmp(self, il::CmpOp::Eq, il::Type::W8, tagVal, il::Val::Imm(variantTag as i64), matchBlock, fallthrough);
            } else {
                let base = emitValToReg(self, subject.val);
                let tagReg = tvalTagReg(self, base);

                try emitBrCmp(self, il::CmpOp::Eq, il::Type::W8, il::Val::Reg(tagReg), il::Val::Imm(variantTag as i64), matchBlock, fallthrough);
            }
        }
        else => { // Value comparison.
            assert not isNil;
            let pattVal = try lowerExpr(self, pattern);
            if isAggregateType(subject.type) {
                // Aggregate types need structural comparison rather than
                // scalar compare.
                let subjectReg = emitValToReg(self, subject.val);
                let pattReg = emitValToReg(self, pattVal);
                let eq = try lowerAggregateEq(self, subject.type, subjectReg, pattReg, 0);
                let eqReg = emitValToReg(self, eq);

                try emitBr(self, eqReg, matchBlock, fallthrough);
            } else {
                try emitBrCmp(self, il::CmpOp::Eq, subject.ilType, subject.val, pattVal, matchBlock, fallthrough);
            }
        }
    }
}

/// Emit branches for multiple patterns. The first pattern that matches
/// causes a jump to the match block. If no patterns match, we jump to the
/// fallthrough block.
unsafe fn emitPatternMatches 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    subject: &MatchSubject,
    patterns: &[*ast::Node],
    matchBlock: BlockId,
    fallthrough: BlockId
) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    assert patterns.len > 0;

    for i in 0..(patterns.len - 1) {
        let pattern = patterns[i];
        let nextArm = try createBlock(self, "arm");
        try emitPatternMatch(self, subject, pattern, matchBlock, nextArm);

        // Seal the intermediate arm block: all predecessor edges are known
        // This ensures SSA construction can resolve variable uses through
        // single-predecessor optimization instead of creating unresolved block
        // parameters.
        try switchToAndSeal(self, nextArm);
    }
    // Handle last pattern: go to fallthrough block on failure.
    let last = patterns[patterns.len - 1];
    try emitPatternMatch(self, subject, last, matchBlock, fallthrough);
}

/// Emit a match binding pattern.
/// Binding patterns always match for regular values, but for optionals they
/// check for the presence of a value. Jumps to `valuePresent` on success,
/// `valueAbsent` on failure.
unsafe fn emitBindingTest 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    subject: &MatchSubject,
    valuePresent: BlockId,
    valueAbsent: BlockId
) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    match subject.kind {
        case MatchSubjectKind::OptionalPtr, MatchSubjectKind::OptionalAggregate => {
            let nilReg = try optionalNilReg(self, subject.val, subject.type);
            try emitBr(self, nilReg, valuePresent, valueAbsent);
        }
        else => {
            // Regular values always match binding patterns unconditionally.
            try emitJmp(self, valuePresent);
        }
    }
}

/// Emit a jump to target if the current block hasn't terminated, then seal the target block.
unsafe fn emitJmpAndSeal 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, target: BlockId) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    if not blockHasTerminator(self) {
        try emitJmp(self, target);
    }
    try sealBlock(self, target);
}

/// Check if the current block already has a terminator instruction.
unsafe fn blockHasTerminator 'arena 'phase 'function (self: &FnLowerer 'arena 'phase 'function) -> bool where 'arena: 'phase, 'phase: 'function {
    let blk = getBlock(self, currentBlock(self));
    if blk.instrs.len == 0 {
        return false;
    }
    match blk.instrs[blk.instrs.len - 1] {
        case il::Instr::Ret { .. },
             il::Instr::Jmp { .. },
             il::Instr::Br { .. },
             il::Instr::Switch { .. },
             il::Instr::Unreachable =>
            return true,
        else =>
            return false,
    }
}

/// Emit a jump to merge block if the current block hasn't terminated.
///
/// This is used after lowering branches of an if-else, for example. If the
/// branch diverges, the block already has a terminator and no jump is needed.
///
/// This handles cases like:
///
///     if cond {
///         return 1;   // @then diverges, no jump to merge.
///     } else {
///         return 0;   // @else diverges, no jump to merge.
///     }
///
/// In the above example, the merge block stays `nil`, and no code is generated
/// after the `if`. The merge block is created on first use.
unsafe fn emitMergeIfUnterminated 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, mergeBlock: &mut ?BlockId) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    if not blockHasTerminator(self) {
        if *mergeBlock == nil {
            set *mergeBlock = try createBlock(self, "merge");
        }
        let target = *mergeBlock else { throw LowerError::MissingTarget; };
        try emitJmp(self, target);
    }
}

//////////////////////////////////
// Control Flow Edge Management //
//////////////////////////////////

/// Add a predecessor edge from `pred` to `target`.
/// Must be called before the target block is sealed. Duplicates are ignored.
unsafe fn addPredecessor 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, target: BlockId, pred: BlockId) where 'arena: 'phase, 'phase: 'function {
    let blk = getBlockMut(self, target);
    assert blk.sealState <> Sealed::Yes, "addPredecessor: adding predecessor to sealed block";
    let preds = &mut blk.preds;
    for i in 0..preds.len {
        if preds[i] == *pred { // Avoid duplicate predecessor entries.
            return;
        }
    }
    preds.append(*pred, alloc::arenaAllocator(self.arena));
}

/// Finalize all blocks and return the block array.
unsafe fn finalizeBlocks 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function) -> *unsafe [il::Block] throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let blockCount = self.blockData.len;
    let blocks = try! alloc::allocRawSlice(
        self.arena, @sizeOf(il::Block), @alignOf(il::Block), blockCount
    ) as *unsafe mut [il::Block];

    for i in 0..self.blockData.len {
        let data = &self.blockData[i];

        set blocks[i] = il::Block {
            label: data.label,
            params: &data.params[..],
            instrs: data.instrs,
            locs: &data.locs[..],
            preds: &data.preds[..],
            loopDepth: data.loopDepth,
        };
    }
    return &blocks[..self.blockData.len];
}

/////////////////////
// Loop Management //
/////////////////////

/// Enter a loop context for break/continue handling.
/// `continueBlock` is `nil` when the continue target is created lazily.
fn enterLoop 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, breakBlock: BlockId, continueBlock: ?BlockId) where 'arena: 'phase, 'phase: 'function {
    assert self.loopDepth < self.loopStack.len, "enterLoop: loop depth overflow";
    let slot = &mut self.loopStack[self.loopDepth];

    set slot.breakTarget = breakBlock;
    set slot.continueTarget = continueBlock;
    set self.loopDepth += 1;
}

/// Exit the current loop context.
fn exitLoop 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function) where 'arena: 'phase, 'phase: 'function {
    assert self.loopDepth <> 0, "exitLoop: loopDepth is zero";
    set self.loopDepth -= 1;
}

/// Copy the current loop targets.
fn currentLoop 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function) -> ?LoopCtx where 'arena: 'phase, 'phase: 'function {
    if self.loopDepth == 0 {
        return nil;
    }
    return self.loopStack[self.loopDepth - 1];
}

/// Get or lazily create the continue target block for the current loop.
unsafe fn getOrCreateContinueBlock 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function) -> BlockId throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let ctx = currentLoop(self) else {
        throw LowerError::OutsideOfLoop;
    };
    if let block = ctx.continueTarget {
        return block;
    }
    let block = try createBlock(self, "step");
    set self.loopStack[self.loopDepth - 1].continueTarget = block;
    return block;
}

/// Allocate a slice of values in the lowering arena.
unsafe fn allocVals 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, len: u32) -> *unsafe mut [il::Val] throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    return try! alloc::allocRawSlice(self.arena, @sizeOf(il::Val), @alignOf(il::Val), len) as *unsafe mut [il::Val];
}

/// Allocate a single-value slice in the lowering arena.
unsafe fn allocVal 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, val: il::Val) -> *unsafe mut [il::Val] throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let args = try allocVals(self, 1);
    set args[0] = val;
    return args;
}

////////////////////////
// SSA Var Management //
////////////////////////

// This section implements SSA construction following "Simple and Efficient
// Construction of Static Single Assignment Form" (Braun et al., 2013). The IL
// uses block parameters (equivalent to phi nodes) that receive values via
// terminator arguments. Block args are resolved eagerly at seal time via
// [`resolveBlockArgs`], matching Braun's on-the-fly approach.
//
// In SSA form, each variable definition creates a unique value. When control
// flow diverges and merges (like after an if-else), a variable might have
// different definitions from different paths.
//
// # What "Value" means
//
// An [`il::Val`] is a compile-time representation of *where* a runtime value
// lives, not the runtime value itself. Values can live in registers, as symbol
// references, or as immediate values, ie. static constants.
//
// When we "find the value" of a variable, we're answering: "Which SSA register
// (or constant) represents this variable at this program point?"
//
// Each source-level variable gets a `Var` handle on declaration. When a
// variable is defined via [`defVar`], its SSA value is recorded in the current
// block's variable mapping. When a variable is used with [`useVar`] or
// [`useVarInBlock`], we either return the local definition or recursively look
// up the value from predecessor blocks. When multiple predecessors define
// different values for a given variable, we insert a block parameter
// (equivalent to a phi node).
//
// The algorithm used by [`useVarInBlock`] handles three cases:
//
// 1. **Local definition exists**: If the variable was assigned in this block,
//    return that value immediately (fast path).
//
// 2. **Sealed block with single predecessor**: If all incoming edges are known
//    and there's exactly one predecessor, recurse to that predecessor. No merge
//    is needed since there's only one path. The result is cached.
//
// 3. **Unsealed block or multiple predecessors**: Create a block parameter to
//    receive the merged value. If sealed, look up each predecessor's value and
//    patch its terminator via [`resolveBlockArgs`]. If unsealed, `paramVars`
//    records the variable in parameter order. [`sealBlock`] resolves these
//    parameters after all incoming edges are known.
//
// Consider this code:
//
//     let mut x = 1;
//     if cond {
//         x = 2;
//     } else {
//         x = f();   // Result in register %r.
//     }
//     print(x);      // Which value?
//
// The Control Flow Graph (CFG) looks like:
//
//         [entry]
//           _|_
//          /   \
//     [then]   [else]
//     x = 2    x = %r
//         \    /
//        [merge]
//         use(x)
//
// At the `print(x)` point, the compiler doesn't know which branch ran (that's
// a runtime decision), but it needs to emit code that works for either case.
// This is where [`useVarInBlock`] is called.
//
// The generated IL looks like this:
//
//     @then
//       jmp @merge(2);           // pass immediate `2`
//     @else
//       jmp @merge(%r);          // pass register `%r`
//     @merge(w32 %m)             // `%m` receives whichever value arrives at runtime
//       call w32 $print(%m);     // `print` is called with runtime value in `%m`
//
// A block is "sealed" when all predecessor edges are known. This is crucial
// because until sealed, we can't know how many paths merge into the block,
// and thus can't create the right number of block parameter arguments.
//
// If a block isn't sealed but we need a variable's value, we still create
// a block parameter, but defer filling in the terminator arguments. When the
// block is later sealed via [`sealBlock`], all incomplete block params are resolved.

/// Declare a new source-level variable and define its initial value.
/// If called before any block exists (e.g., for parameters), the definition is skipped.
unsafe fn newVar 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    name: ?*[u8],
    type: il::Type,
    mutable: bool,
    val: il::Val
) -> Var where 'arena: 'phase, 'phase: 'function {
    let id = self.vars.len;
    self.vars.append(
        VarData { name, type, mutable, addressTaken: false },
        alloc::arenaAllocator(self.arena)
    );

    let v = Var(id);
    if self.currentBlock <> nil {
        defVar(self, v, val);
    }
    return v;
}

/// Define (write) a variable. Record the SSA value of a variable in the
/// current block. Called when a variable is assigned or initialized (`let`
/// bindings, assignments, loop updates). When [`useVar`] is later called,
/// it will retrieve this value.
unsafe fn defVar 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, v: Var, val: il::Val) where 'arena: 'phase, 'phase: 'function {
    assert *v < self.vars.len;
    set getBlockMut(self, currentBlock(self)).vars[*v] = val;
}

/// Use (read) the current value of a variable in the current block.
/// May insert block parameters if the value must come from predecessors.
unsafe fn useVar 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, v: Var) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    return try useVarInBlock(self, currentBlock(self), v);
}

/// Resolve which SSA definition of a variable reaches a use point in a given block.
///
/// Given a variable and a block where it's used, this function finds the
/// correct [`il::Val`] that holds the variable's value at that program point.
/// When control flow merges from multiple predecessors with different
/// definitions, it creates a block parameter to unify them.
unsafe fn useVarInBlock 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, block: BlockId, v: Var) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    assert *v < self.vars.len;

    let blk = getBlockMut(self, block);
    if let val = blk.vars[*v] {
        return val;
    }
    // Entry block cannot have block parameters. If variable isn't defined
    // in entry, we return undefined.
    if block == self.entryBlock {
        return il::Val::Undef;
    }
    if blk.sealState == Sealed::Yes {
        if blk.preds.len == 0 {
            // Variable used in sealed block with no predecessors.
            throw LowerError::InvalidUse;
        }
        // Single predecessor means no merge needed, variable is implicitly
        // available without a block parameter.
        if blk.preds.len == 1 {
            let pred = BlockId(blk.preds[0]);
            if *pred <> *block {
                let val = try useVarInBlock(self, pred, v);
                set blk.vars[*v] = val; // Cache.
                return val;
            }
        }
    }
    // Multiple predecessors or unsealed block: need a block parameter to merge
    // the control flow paths.
    return try createBlockParam(self, block, v);
}

/// Look up a variable by name in the current scope.
/// Searches from most recently declared to first, enabling shadowing.
unsafe fn lookupVarByName 'arena 'phase 'function (self: &FnLowerer 'arena 'phase 'function, name: *[u8]) -> ?Var where 'arena: 'phase, 'phase: 'function {
    let mut id = self.vars.len;
    while id > 0 {
        set id -= 1;
        if let varName = self.vars[id].name {
            // Names are interned strings, so pointer comparison suffices.
            if varName == name {
                return Var(id);
            }
        }
    }
    return nil;
}

/// Look up a local variable bound to an identifier node.
unsafe fn lookupLocalVar 'arena 'phase 'function (self: &FnLowerer 'arena 'phase 'function, node: *ast::Node) -> ?Var where 'arena: 'phase, 'phase: 'function {
    let case ast::NodeValue::Ident(name) = node.value else {
        return nil;
    };
    return lookupVarByName(self, name);
}

/// Save current lexical variable scope depth.
unsafe fn enterVarScope 'arena 'phase 'function (self: &FnLowerer 'arena 'phase 'function) -> u32 where 'arena: 'phase, 'phase: 'function {
    return self.vars.len;
}

/// Restore lexical variable scope depth.
unsafe fn exitVarScope 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, savedVarsLen: u32) where 'arena: 'phase, 'phase: 'function {
    set self.vars = @sliceOf(self.vars.ptr, savedVarsLen, self.vars.cap);
}

/// Get the metadata for a variable.
unsafe fn getVar 'arena 'phase 'function (self: &FnLowerer 'arena 'phase 'function, v: Var) -> *unsafe VarData where 'arena: 'phase, 'phase: 'function {
    assert *v < self.vars.len;
    return &self.vars[*v];
}

/// Create a block parameter to merge a variable's value from multiple
/// control flow paths.
///
/// Called when [`useVarInBlock`] cannot find a local definition and the block is
/// unsealed or has multiple predecessors. For example, `x` can be used in `@end`
/// but defined differently in `@then` and `@else`:
///
///     @then
///       jmp @end(1);            // x = 1
///     @else
///       jmp @end(2);            // x = 2
///     @end(w32 %1)              // x = %1, merged from predecessors
///       ret %1;
///
/// Create a register `%1` as a block parameter. In a sealed block, patch each
/// predecessor's jump to pass its value of `x`. Otherwise, defer until sealing.
unsafe fn createBlockParam 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, block: BlockId, v: Var) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    // Entry block must not have block parameters.
    assert block <> self.entryBlock, "createBlockParam: entry block must not have block parameters";
    // Allocate a register to hold the merged value.
    let reg = nextReg(self);
    let type = getVar(self, v).type;

    // Create block parameter and add it to the block.
    let param = il::Param { value: reg, type };
    let blk = getBlockMut(self, block);
    let paramIdx = blk.paramVars.len;
    blk.params.append(param, alloc::arenaAllocator(self.arena));
    blk.paramVars.append(*v, alloc::arenaAllocator(self.arena)); // Associate variable with parameter.

    // Record that this variable's value in this block is now the parameter register.
    // This must happen before the predecessor loop to handle self-referential loops.
    set blk.vars[*v] = il::Val::Reg(reg);

    if blk.sealState == Sealed::Yes {
        // Block sealed: check for trivial phi before committing. If all
        // predecessors provide the same value, we can remove the param we
        // just created and use that value directly.
        if let trivial = try getTrivialPhiVal(self, block, v) {
            let provisional = il::Val::Reg(reg);
            removeLastBlockParam(self, block);
            rewriteCachedVarValue(self, v, provisional, trivial);
            set getBlockMut(self, block).vars[*v] = trivial;
            return trivial;
        }
        // Non-trivial phi: patch predecessors to pass their values.
        try resolveBlockArgs(self, block, v, paramIdx);
    }
    return il::Val::Reg(reg);
}

/// Complete the block parameter at `paramIdx`. Look up the variable's value in
/// each predecessor and patch its terminator with the edge argument.
///
/// This is the block-parameter equivalent of adding operands to a phi-function in
/// traditional SSA. Where a phi-function merges values at the join point:
///
///     x3 = phi(x1, x2)
///
/// This representation avoids the need for phi nodes to reference their
/// predecessor blocks explicitly, since the control flow edges already encode
/// that information.
unsafe fn resolveBlockArgs 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, block: BlockId, v: Var, paramIdx: u32) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let blk = getBlock(self, block);

    // For each predecessor, recursively look up the variable's reaching definition
    // in that block, then patch the predecessor's terminator to pass that value
    // as an argument to this block's parameter.
    for predId in blk.preds {
        let pred = BlockId(predId);
        // This may recursively trigger more block arg resolution if the
        // predecessor also needs to look up the variable from its predecessors.
        let val = try useVarInBlock(self, pred, v);
        assert val <> il::Val::Undef, "createBlockParam: predecessor provides undef value for block parameter";
        patchTerminatorArg(self, pred, *block, paramIdx, val);
    }
}

/// Check if a block parameter is trivial, i.e. all predecessors provide
/// the same value. Returns the trivial value if so.
unsafe fn getTrivialPhiVal 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, block: BlockId, v: Var) -> ?il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let blk = getBlock(self, block);
    // Get the block parameter register.
    let paramReg = blk.vars[*v];
    // Check if all predecessors provide the same value.
    let mut sameVal: ?il::Val = nil;

    for predId in blk.preds {
        let pred = BlockId(predId);
        let val = try useVarInBlock(self, pred, v);

        // Check if this is a self-reference.
        // This happens in cycles where the loop back-edge passes the phi to
        // itself. We skip self-references when checking for trivial phis.
        if let reg = paramReg {
            if val == reg {
                // Self-reference, skip this predecessor.
            } else if let sv = sameVal {
                if val <> sv {
                    // Multiple different values, not trivial.
                    return nil;
                }
            } else {
                set sameVal = val;
            }
        } else { // No param reg set yet, can't be trivial.
            return nil;
        }
    }
    return sameVal;
}

/// Patch a single terminator argument for a specific edge. This is used during
/// SSA construction to pass variable values along control flow edges.
unsafe fn patchTerminatorArg 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    from: BlockId,         // The predecessor block containing the terminator to patch.
    target: u32,           // The index of the target block we're passing the value to.
    paramIdx: u32,         // The index of the block parameter to set.
    val: il::Val           // The value to pass as the argument.
) where 'arena: 'phase, 'phase: 'function {
    let data = getBlockMut(self, from);
    let ix = data.instrs.len - 1; // The terminator is always the last instruction.

    // TODO: We shouldn't need to use a mutable subscript here, given that the
    // fields are already mutable.
    match &mut data.instrs[ix] {
        case il::Instr::Jmp { args, .. } => {
            set *args = growArgs(self, *args, paramIdx + 1);
            set args[paramIdx] = val;
        }
        case il::Instr::Br { thenTarget, thenArgs, elseTarget, elseArgs, .. } => {
            // Nb. both branches could target the same block (e.g. `if cond { x } else { x }`).
            if *thenTarget == target {
                set *thenArgs = growArgs(self, *thenArgs, paramIdx + 1);
                set thenArgs[paramIdx] = val;
            }
            if *elseTarget == target {
                set *elseArgs = growArgs(self, *elseArgs, paramIdx + 1);
                set elseArgs[paramIdx] = val;
            }
        }
        case il::Instr::Switch { defaultTarget, defaultArgs, cases, .. } => {
            if *defaultTarget == target {
                set *defaultArgs = growArgs(self, *defaultArgs, paramIdx + 1);
                set defaultArgs[paramIdx] = val;
            }
            let idx = paramIdx + 1;
            for ci in 0..cases.len {
                if cases[ci].target == target {
                    set cases[ci].args = growArgs(self, cases[ci].args, idx);
                    set cases[ci].args[paramIdx] = val;
                }
            }
        }
        else => {
            // Other terminators (e.g. `Ret`, `Unreachable`) don't have successor blocks.
        }
    }
}

/// Grow an args array to hold at least the given capacity.
unsafe fn growArgs 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, args: *unsafe mut [il::Val], capacity: u32) -> *unsafe mut [il::Val] where 'arena: 'phase, 'phase: 'function {
    if args.len >= capacity {
        return args;
    }
    let newArgs = try! alloc::allocRawSlice(
        self.arena, @sizeOf(il::Val), @alignOf(il::Val), capacity
    ) as *unsafe mut [il::Val];

    for arg, i in args {
        set newArgs[i] = arg;
    }
    for i in args.len..capacity {
        set newArgs[i] = il::Val::Undef;
    }
    return newArgs;
}

/// Extract the parameter name from an [`FnParam`] AST node value.
fn paramName(value: *ast::NodeValue) -> *[u8] throws (LowerError) {
    let case ast::NodeValue::FnParam(param) = *value else {
        throw LowerError::ExpectedFunctionParam;
    };
    let case ast::NodeValue::Ident(name) = param.name.value else {
        throw LowerError::ExpectedIdentifier;
    };
    return name;
}

/// Lower function parameters. Declares variables for each parameter.
/// When a receiver name is passed, we're handling a trait method.
unsafe fn lowerParams 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    fnType: resolver::FnType,
    astParams: *[*ast::Node],
    receiverName: ?*ast::Node
) -> *unsafe [il::Param] throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let offset: u32 = 1 if self.returnReg <> nil else 0;
    let totalLen = fnType.paramTypes.len as u32 + offset;
    if totalLen == 0 {
        return &[];
    }
    assert fnType.paramTypes.len as u32 <= resolver::MAX_FN_PARAMS;

    let params = try! alloc::allocRawSlice(
        self.arena, @sizeOf(il::Param), @alignOf(il::Param), totalLen
    ) as *unsafe mut [il::Param];

    if let reg = self.returnReg {
        set params[0] = il::Param { value: reg, type: il::Type::W64 };
    }
    for i in 0..fnType.paramTypes.len as u32 {
        let type = ilType(self.low, *fnType.paramTypes[i]);
        let reg = nextReg(self);

        set params[i + offset] = il::Param { value: reg, type };

        // Declare the parameter variable. For the receiver, the name comes
        // from the receiver node.
        // For all other parameters, the name comes from the AST params.
        let mut name: *[u8] = undefined;
        if let recNode = receiverName {
            if i == 0 {
                let case ast::NodeValue::Ident(recName) = recNode.value else {
                    throw LowerError::ExpectedIdentifier;
                };
                set name = recName;
            } else {
                set name = try paramName(&astParams[i - 1].value);
            }
        } else {
            set name = try paramName(&astParams[i].value);
        }
        let v = newVar(self, name, type, false, il::Val::Undef);

        set self.params[i] = FnParamBinding { var: v, reg };
        set self.paramsLen += 1;
    }
    return params;
}

/// Resolve match subject.
unsafe fn lowerMatchSubject 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, subject: *ast::Node) -> MatchSubject throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let mut val = try lowerExpr(self, subject);
    let subjectType = try typeOf(self, subject);
    let unwrapped = resolver::unwrapMatchSubject(subjectType);

    // When matching an aggregate by value, copy it to a fresh stack slot so
    // that bindings are independent of the original memory.  Without this,
    // the lowerer returns a pointer into the source and mutations to the
    // source silently corrupt the bound variables.
    if unwrapped.by == resolver::MatchBy::Value and isAggregateType(unwrapped.effectiveTy) {
        set val = try emitStackVal(self, unwrapped.effectiveTy, val);
    }

    let mut bindType = unwrapped.effectiveTy;
    if let case resolver::Type::Optional(inner) = unwrapped.effectiveTy {
        set bindType = *inner;
    }
    let ilType = ilType(self.low, unwrapped.effectiveTy);
    let kind = matchSubjectKind(unwrapped.effectiveTy);

    return MatchSubject { val, type: unwrapped.effectiveTy, ilType, bindType, kind, by: unwrapped.by };
}

/// Check whether a case pattern is an unconditional wildcard.
fn isWildcardPattern(pattern: *ast::Node) -> bool {
    match pattern.value {
        case ast::NodeValue::Placeholder => return true,
        else => return false,
    }
}

/// Check whether an AST node is the `undefined` literal.
fn isUndef(node: *ast::Node) -> bool {
    match node.value {
        case ast::NodeValue::Undef => return true,
        else => return false,
    }
}

/// Load the tag byte from a tagged value aggregate (optionals and unions).
unsafe fn tvalTagReg 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, base: il::Reg) -> il::Reg where 'arena: 'phase, 'phase: 'function {
    let tagReg = nextReg(self);
    emitLoadW8At(self, tagReg, base, TVAL_TAG_OFFSET);
    return tagReg;
}

/// Load the tag word from a result aggregate.
unsafe fn resultTagReg 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, base: il::Reg) -> il::Reg where 'arena: 'phase, 'phase: 'function {
    let tagReg = nextReg(self);
    emitLoadW64At(self, tagReg, base, TVAL_TAG_OFFSET);
    return tagReg;
}

/// Get the register to compare against `0` for optional `nil` checking.
/// For null-ptr-optimized types, loads the data pointer, or returns it
/// directly for scalar pointers. For aggregates, returns the tag register.
unsafe fn optionalNilReg 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, val: il::Val, typ: resolver::Type) -> il::Reg throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let reg = emitValToReg(self, val);

    match typ {
        case resolver::Type::Optional(resolver::Type::Slice { .. }) => {
            let ptrReg = nextReg(self);
            emitLoadW64At(self, ptrReg, reg, SLICE_PTR_OFFSET);
            return ptrReg;
        }
        case resolver::Type::Optional(resolver::Type::Pointer { .. }) => return reg,
        case resolver::Type::Optional(_) => return tvalTagReg(self, reg),
        else => return reg,
    }
}

/// Lower an optional nil check (`opt == nil` or `opt <> nil`).
unsafe fn lowerNilCheck 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, opt: *ast::Node, isEq: bool) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let optTy = try typeOf(self, opt);
    // Handle `nil == nil` or `nil <> nil`.
    if optTy == resolver::Type::Nil {
        return il::Val::Imm(1) if isEq else il::Val::Imm(0);
    }
    let val = try lowerExpr(self, opt);
    let cmpReg = try optionalNilReg(self, val, optTy);

    // Null-pointer-optimized types compare a 64-bit pointer against zero.
    // Aggregate optionals compare an 8-bit tag byte against zero.
    let cmpType = il::Type::W64 if resolver::isOptionalPointer(optTy) else il::Type::W8;

    let op = il::BinOp::Eq if isEq else il::BinOp::Ne;
    return emitTypedBinOp(self, op, cmpType, il::Val::Reg(cmpReg), il::Val::Imm(0));
}

/// Load the payload value from a tagged value aggregate at the given offset.
unsafe fn tvalPayloadVal 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, base: il::Reg, payload: resolver::Type, valOffset: i32) -> il::Val where 'arena: 'phase, 'phase: 'function {
    if payload == resolver::Type::Void {
        return il::Val::Undef;
    }
    return emitRead(self, base, valOffset, payload);
}

/// Compute the address of the payload in a tagged value aggregate.
unsafe fn tvalPayloadAddr 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, base: il::Reg, valOffset: i32) -> il::Val where 'arena: 'phase, 'phase: 'function {
    return il::Val::Reg(emitPtrOffset(self, base, valOffset));
}

/// Bind a variable to a tagged value's payload.
unsafe fn bindPayloadVariable 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    name: *[u8],
    subjectVal: il::Val,
    bindType: resolver::Type,
    matchBy: resolver::MatchBy,
    valOffset: i32,
    mutable: bool
) -> Var throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let base = emitValToReg(self, subjectVal);
    let mut payload: il::Val = undefined;

    match matchBy {
        case resolver::MatchBy::Value =>
            set payload = tvalPayloadVal(self, base, bindType, valOffset),
        case resolver::MatchBy::Ref, resolver::MatchBy::MutRef =>
            set payload = tvalPayloadAddr(self, base, valOffset),
    };
    return newVar(self, name, ilType(self.low, bindType), mutable, payload);
}

/// Bind an identifier from a matched subject.
unsafe fn bindMatchVariable 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    subject: &MatchSubject,
    binding: *ast::Node,
    mutable: bool
) -> ?Var throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    // Only bind if the pattern is an identifier.
    let case ast::NodeValue::Ident(name) = binding.value else {
        return nil;
    };
    // For optional aggregates, extract the payload from the tagged value.
    // The tag check already passed, so we know the payload is valid.
    if let case MatchSubjectKind::OptionalAggregate = subject.kind {
        let valOffset = resolver::getOptionalValOffset(subject.bindType) as i32;
        return try bindPayloadVariable(self, name, subject.val, subject.bindType, subject.by, valOffset, mutable);
    }
    // Declare the variable in the current block's scope.
    return newVar(self, name, ilType(self.low, subject.bindType), mutable, subject.val);
}

/// Bind variables from inside case patterns (union variants, records, slices).
/// `failBlock` is passed when nested patterns may require additional tests
/// that branch on mismatch (e.g. nested union variant tests).
unsafe fn bindPatternVariables 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, subject: &MatchSubject, patterns: &[*ast::Node], failBlock: BlockId) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    for pattern in patterns {

        // Handle simple variant patterns like `Variant(x)`.
        if let arg = resolver::variantPatternBinding(self.low.resolver, pattern) {
            let case MatchSubjectKind::Union(unionInfo) = subject.kind
                else panic "bindPatternVariables: expected union subject";
            let valOffset = unionInfo.valOffset as i32;

            // Get the actual field type from the variant's record info.
            // This preserves the original data layout type (e.g. `*T`) even when
            // the resolver resolved the pattern against a dereferenced type (`T`).
            let variantExtra = resolver::nodeData(self.low.resolver, pattern).extra;
            let case resolver::NodeExtra::UnionVariant { ordinal, .. } = variantExtra
                else panic "bindPatternVariables: expected variant extra";
            let payloadType = unionInfo.variants[ordinal].valueType;
            let payloadRec = resolver::getRecord(payloadType)
                else panic "bindPatternVariables: expected record payload";
            let fieldType = payloadRec.fields[0].fieldType;

            match arg.value {
                case ast::NodeValue::Ident(name) => {
                    try bindPayloadVariable(self, name, subject.val, fieldType, subject.by, valOffset, false);
                }
                case ast::NodeValue::Placeholder => {}
                else => {
                    // Nested pattern inside a variant call, e.g. `Variant(Inner { x, y })`.
                    let base = emitValToReg(self, subject.val);
                    let payloadBase = emitPtrOffset(self, base, valOffset);
                    let fieldInfo = resolver::RecordField {
                        name: nil,
                        fieldType,
                        offset: 0,
                    };
                    try bindFieldVariable(self, arg, payloadBase, fieldInfo, subject.by, failBlock);
                }
            }
        }
        match pattern.value {
            case ast::NodeValue::Call(call) => {
                if let body = resolver::getRecord(subject.type) {
                    let base = emitValToReg(self, subject.val);
                    for binding, i in call.args {
                        try bindFieldVariable(self, binding, base, body.fields[i], subject.by, failBlock);
                    }
                }
            }
            // Compound variant patterns like `Variant { a, b }`.
            case ast::NodeValue::RecordLit(lit) =>
                try bindRecordPatternFields(self, subject, pattern, lit, failBlock),
            // Array patterns like `[a, b, c]`.
            case ast::NodeValue::ArrayLit(items) =>
                try bindArrayPatternElements(self, subject, items, failBlock),
            // Literals, wildcards, identifiers: no bindings needed.
            else => {},
        }
    }
}

/// Bind variables from an array literal pattern (e.g., `[a, 1, c]`).
/// Each element is either bound as a variable, skipped (placeholder), or
/// tested against the subject element, branching to `failBlock` on mismatch.
unsafe fn bindArrayPatternElements 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    subject: &MatchSubject,
    items: *[*ast::Node],
    failBlock: BlockId
) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let case resolver::Type::Array(arrInfo) = subject.type
        else throw LowerError::ExpectedSliceOrArray;

    let elemTy = *arrInfo.item;
    let elemLayout = resolver::getTypeLayout(elemTy);
    let stride = elemLayout.size as i32;
    let base = emitValToReg(self, subject.val);

    for elem, i in items {
        let fieldInfo = resolver::RecordField {
            name: nil,
            fieldType: elemTy,
            offset: (i as i32) * stride,
        };
        try bindFieldVariable(self, elem, base, fieldInfo, subject.by, failBlock);
    }
}

/// Bind fields from a record pattern.
unsafe fn bindRecordPatternFields 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, subject: &MatchSubject, pattern: *ast::Node, lit: ast::RecordLit, failBlock: BlockId) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    // No fields to bind (e.g., `{ .. }`).
    if lit.fields.len == 0 {
        return;
    }
    // Optional value patterns were already compared structurally by
    // `emitPatternMatch`; unlike union record patterns, they have no payload
    // bindings to extract here.
    if let case MatchSubjectKind::OptionalAggregate = subject.kind {
        return;
    }
    if let recInfo = resolver::getRecord(subject.type) {
        let base = emitValToReg(self, subject.val);
        try bindNestedRecordFields(self, base, lit, recInfo, subject.by, failBlock);
        return;
    }
    // Get the union type info from the subject.
    let case MatchSubjectKind::Union(unionInfo) = subject.kind
        else panic "bindRecordPatternFields: expected union subject";

    // Get the variant index from the pattern node.
    let case resolver::NodeExtra::UnionVariant { ordinal: variantOrdinal, .. } =
        resolver::nodeData(self.low.resolver, pattern).extra
    else throw LowerError::MissingMetadata;

    // Get the record type from the variant's payload type.
    let payloadType = unionInfo.variants[variantOrdinal].valueType;
    let recInfo = resolver::getRecord(payloadType)
        else throw LowerError::ExpectedRecord;

    // Get the payload base pointer which points to the record within the tagged union.
    let base = emitValToReg(self, subject.val);
    let valOffset = unionInfo.valOffset as i32;
    let payloadBase = emitPtrOffset(self, base, valOffset);

    try bindNestedRecordFields(self, payloadBase, lit, recInfo, subject.by, failBlock);
}

/// Bind a single record field to a pattern variable, with support for nested
/// pattern tests that branch to `failBlock` on mismatch.
unsafe fn bindFieldVariable 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    binding: *ast::Node,
    base: il::Reg,
    fieldInfo: resolver::RecordField,
    matchBy: resolver::MatchBy,
    failBlock: BlockId
) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    match binding.value {
        case ast::NodeValue::Ident(name) => {
            let val = emitRead(self, base, fieldInfo.offset, fieldInfo.fieldType)
                if matchBy == resolver::MatchBy::Value
                else il::Val::Reg(emitPtrOffset(self, base, fieldInfo.offset));
            newVar(self, name, ilType(self.low, fieldInfo.fieldType), false, val);
        }
        case ast::NodeValue::Placeholder => {}
        case ast::NodeValue::RecordLit(lit) => {
            // Check if this record literal is a union variant pattern.
            if let keyNode = resolver::patternVariantKeyNode(binding) {
                if let case resolver::NodeExtra::UnionVariant { .. } = resolver::nodeData(self.low.resolver, keyNode).extra {
                    try emitNestedFieldTest(self, binding, base, fieldInfo, matchBy, failBlock);
                    return;
                }
            }
            // Plain nested record destructuring pattern.
            // Auto-deref: if the field is a pointer, load it first.
            let mut derefType = fieldInfo.fieldType;
            let mut nestedBase = emitPtrOffset(self, base, fieldInfo.offset);
            if let case resolver::Type::Pointer { target, .. } = fieldInfo.fieldType {
                let ptrReg = nextReg(self);
                emitLoadW64At(self, ptrReg, nestedBase, 0);
                set nestedBase = ptrReg;
                set derefType = *target;
            }
            let recInfo = resolver::getRecord(derefType)
                else throw LowerError::ExpectedRecord;

            try bindNestedRecordFields(self, nestedBase, lit, recInfo, matchBy, failBlock);
        }
        else => {
            // Nested pattern requiring a test (union variant scope access, literal, etc).
            try emitNestedFieldTest(self, binding, base, fieldInfo, matchBy, failBlock);
        }
    }
}

/// Emit a nested pattern test for a record field value, branching to
/// `failBlock` if the pattern does not match. On success, continues in
/// a fresh block and binds any nested variables.
unsafe fn emitNestedFieldTest 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    pattern: *ast::Node,
    base: il::Reg,
    fieldInfo: resolver::RecordField,
    matchBy: resolver::MatchBy,
    failBlock: BlockId
) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let mut fieldType = fieldInfo.fieldType;
    let fieldPtr = emitPtrOffset(self, base, fieldInfo.offset);

    // Auto-deref: when the field is a pointer and the pattern destructures
    // the pointed-to value, load the pointer and use the target type.
    // The loaded pointer becomes the base address for the nested subject.
    let mut derefBase: ?il::Reg = nil;
    if let case resolver::Type::Pointer { target, .. } = fieldType {
        if resolver::isDestructuringPattern(pattern) {
            let ptrReg = nextReg(self);
            emitLoadW64At(self, ptrReg, fieldPtr, 0);
            set derefBase = ptrReg;
            set fieldType = *target;
        }
    }
    // Build a MatchSubject for the nested field.
    let ilTy = ilType(self.low, fieldType);
    let kind = matchSubjectKind(fieldType);

    // Determine the subject value.
    let mut val: il::Val = undefined;
    if let reg = derefBase {
        // Auto-deref: the loaded pointer is the address of the target value.
        set val = il::Val::Reg(reg);
    } else if isAggregateType(fieldType) {
        // Aggregate: use the pointer.
        set val = il::Val::Reg(fieldPtr);
    } else {
        // Scalar: load the value.
        set val = emitRead(self, base, fieldInfo.offset, fieldType);
    }
    let nestedSubject = MatchSubject {
        val,
        type: fieldType,
        ilType: ilTy,
        bindType: fieldType,
        kind,
        by: matchBy,
    };

    // Emit the pattern test: on success jump to `continueBlock`, on fail to `failBlock`.
    let continueBlock = try createBlock(self, "nest");
    try emitPatternMatch(self, &nestedSubject, pattern, continueBlock, failBlock);
    try switchToAndSeal(self, continueBlock);

    // After the test succeeds, bind any nested variables.
    let patterns = [pattern];
    try bindPatternVariables(self, &nestedSubject, &patterns[..], failBlock);
}

/// Bind variables from a nested record literal pattern.
unsafe fn bindNestedRecordFields 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    base: il::Reg,
    lit: ast::RecordLit,
    recInfo: resolver::RecordType,
    matchBy: resolver::MatchBy,
    failBlock: BlockId
) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    for fieldNode in lit.fields {
        let case ast::NodeValue::RecordLitField(field) = fieldNode.value else {
            throw LowerError::UnexpectedNodeValue(fieldNode);
        };
        let fieldIdx = resolver::recordFieldIndexFor(self.low.resolver, fieldNode)
            else throw LowerError::MissingMetadata;
        if fieldIdx >= recInfo.fields.len {
            throw LowerError::MissingMetadata;
        }
        let fieldInfo = recInfo.fields[fieldIdx];

        try bindFieldVariable(self, field.value, base, fieldInfo, matchBy, failBlock);
    }
}

/// Lower function body to a list of basic blocks.
unsafe fn lowerFnBody 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, body: *ast::Node) -> *unsafe [il::Block] throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    // Create and switch to entry block.
    let entry = try createBlock(self, "entry");
    set self.entryBlock = entry;
    switchToBlock(self, entry);

    /// Bind parameter registers to variables in the entry block.
    for i in 0..self.paramsLen {
        let def = self.params[i];
        defVar(self, def.var, il::Val::Reg(def.reg));
    }
    /// Lower function body.
    try lowerBlock(self, body);

    // Add implicit return if body doesn't diverge.
    if not blockHasTerminator(self) {
        if *self.fnType.returnType == resolver::Type::Never {
            emit(self, il::Instr::Unreachable);
        } else if self.fnType.throwList.len > 0 {
            if *self.fnType.returnType == resolver::Type::Void {
                // Implicit `void` return in throwing function: wrap in result success.
                let val = try buildResult(self, 0, nil, resolver::Type::Void);
                try emitRetVal(self, val);
            } else {
                // Non-void throwing function without explicit return should
                // not happen.
                panic "lowerFnBody: missing return in non-void function";
            }
        } else {
            emit(self, il::Instr::Ret { val: nil });
        }
    }
    return try finalizeBlocks(self);
}

/// Lower a scalar match as a switch instruction.
unsafe fn lowerMatchSwitch 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, prongs: *[*ast::Node], subject: &MatchSubject, mergeBlock: &mut ?BlockId) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let mut blocks: [BlockId; 32] = undefined;
    let mut cases: *unsafe mut [il::SwitchCase] = &mut [];
    let mut defaultIdx: u32 = 0;
    let entry = currentBlock(self);

    for p, i in prongs {
        let case ast::NodeValue::MatchProng(prong) = p.value
            else throw LowerError::UnexpectedNodeValue(p);

        match prong.arm {
            case ast::ProngArm::Binding(_), ast::ProngArm::Else => {
                set blocks[i] = try createBlock(self, "default");
                set defaultIdx = i;
            }
            case ast::ProngArm::Case(pats) => {
                set blocks[i] = try createBlock(self, "case");
                for pat in pats {
                    let cv = resolver::constValueEntry(self.low.resolver, pat)
                        else throw LowerError::MissingConst(pat);

                    cases.append(il::SwitchCase {
                        value: constToScalar(cv),
                        target: *blocks[i],
                        args: &mut []
                    }, alloc::arenaAllocator(self.arena));
                }
            }
        }
        addPredecessor(self, blocks[i], entry);
    }
    emit(self, il::Instr::Switch {
        val: subject.val,
        defaultTarget: *blocks[defaultIdx],
        defaultArgs: &mut [],
        cases: &mut cases[..]
    });

    for p, i in prongs {
        let case ast::NodeValue::MatchProng(prong) = p.value
            else throw LowerError::UnexpectedNodeValue(p);

        try switchToAndSeal(self, blocks[i]);
        try lowerNode(self, prong.body);
        try emitMergeIfUnterminated(self, mergeBlock);
    }
    if let blk = *mergeBlock {
        try switchToAndSeal(self, blk);
    }
}

/// Lower a match statement.
///
/// Processes prongs sequentially, generating comparison code and branches for each.
/// Prongs are processed in source order, so earlier prongs take precedence.
///
/// Guards are handled by emitting an additional branch after pattern matching
/// but before the body.
///
/// Example:
///
///   match x {
///       case 0 => return 0,
///       case 1 => return 1,
///       else => return 2,
///   }
///
/// Generates:
///
///   arm#0:
///       br.eq w32 %x 0 case#0 arm#1;    // if `x == 0`, jump to case#0, else arm#1
///   case#0:
///       ret 0;                          // `case 0` body
///   arm#1:
///       br.eq w32 %x 1 case#1 arm#2;    // if `x == 1`, jump to case#1, else arm#2
///   case#1:
///       ret 1;                          // `case 1` body
///   arm#2:
///       jmp else#0;                     // fallthrough to `else`
///   else#0:
///       ret 2;                          // `else` body
///
/// Example: Binding with guard
///
///   match x {
///       y if y > 0 => return y,
///       else => return 0,
///   }
///
/// Generates:
///
///   arm#0:
///       jmp guard#0;                    // catch-all binding, jump to guard
///   case#0(w32 %y):
///       ret %y;                         // guarded case body, receives bound var
///   guard#0:
///       sgt w32 %cmp %x 0;              // evaluate guard `y > 0`
///       br.ne w32 %cmp 0 case#0 arm#1;  // if `true`, jump to case body
///   arm#1:
///       jmp else#0;                     // guard failed, fallthrough to `else`
///   else#0:
///       ret 0;                          // `else` body
///
unsafe fn lowerMatch 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node, m: ast::Match) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    assert m.prongs.len > 0;

    let prongs = m.prongs;
    // Lower the subject expression once; reused across all arms.
    let subject = try lowerMatchSubject(self, m.subject);
    // Merge block created lazily if any arm needs it (i.e., doesn't diverge).
    let mut mergeBlock: ?BlockId = nil;

    // Use `switch` instruction for matches with constant patterns.
    if resolver::isMatchConst(self.low.resolver, node) {
        try lowerMatchSwitch(self, prongs, &subject, &mut mergeBlock);
        return;
    }
    // Fallback: chained branches.
    let firstArm = try createBlock(self, "arm");
    try emitJmp(self, firstArm);
    try switchToAndSeal(self, firstArm);

    for prongNode, i in prongs {
        let prongScope = enterVarScope(self);
        let case ast::NodeValue::MatchProng(prong) = prongNode.value
            else panic "lowerMatch: expected match prong";

        let isLastArm = i + 1 == prongs.len;
        let hasGuard = prong.guard <> nil;
        let catchAll = resolver::isProngCatchAll(self.low.resolver, prongNode);

        // Entry block: guard block if present, otherwise the body block.
        // The guard block must be created before the body block so that
        // block indices are in reverse post-order (RPO), which the register
        // allocator requires.
        let mut entryBlock: BlockId = undefined;
        if hasGuard {
            set entryBlock = try createBlock(self, "guard");
        }
        // Body block: where the case body lives.
        let mut bodyLabel = "case";
        if prong.arm == ast::ProngArm::Else {
            set bodyLabel = "else";
        }
        let mut bodyBlock = try createBlock(self, bodyLabel);
        if not hasGuard {
            set entryBlock = bodyBlock;
        }
        // Fallthrough block: jumped to when pattern or guard fails.
        let nextArm = try createBlock(self, "arm");

        // Emit pattern test: branch to entry block on match, next arm on fail.
        match prong.arm {
            case ast::ProngArm::Binding(_) if not catchAll =>
                try emitBindingTest(self, &subject, entryBlock, nextArm),
            case ast::ProngArm::Case(patterns) if not catchAll =>
                try emitPatternMatches(self, &subject, patterns, entryBlock, nextArm),
            else =>
                try emitJmp(self, entryBlock),
        }
        // Switch to entry block, where any variable bindings need to be created.
        try switchToAndSeal(self, entryBlock);

        // Bind pattern variables after successful match. Note that the guard
        // has not been evaluated yet. Nested patterns may emit additional
        // tests that branch to `nextArm` on failure, switching the current
        // block.
        match prong.arm {
            case ast::ProngArm::Binding(pat) =>
                try bindMatchVariable(self, &subject, pat, false),
            case ast::ProngArm::Case(patterns) =>
                try bindPatternVariables(self, &subject, patterns, nextArm),
            else => {},
        }

        // Evaluate guard if present; can still fail to next arm.
        if let g = prong.guard {
            try emitCondBranch(self, g, bodyBlock, nextArm);
        } else if *currentBlock(self) <> *bodyBlock {
            // Nested tests changed the current block. Create a new body block
            // after the nest blocks to maintain RPO ordering, and jump to it.
            set bodyBlock = try createBlock(self, bodyLabel);
            try emitJmp(self, bodyBlock);
        }
        // Lower prong body and jump to merge if unterminated.
        try switchToAndSeal(self, bodyBlock);
        try lowerNode(self, prong.body);
        try emitMergeIfUnterminated(self, &mut mergeBlock);
        exitVarScope(self, prongScope);

        // Switch to next arm, unless last arm without guard.
        if not isLastArm or hasGuard {
            try switchToAndSeal(self, nextArm);
            if isLastArm {
                // Last arm with guard: guard failure jumps to merge.
                try emitMergeIfUnterminated(self, &mut mergeBlock);
            }
        }
    }
    // Continue in merge block if we have one, ie. if at least one arm doesn't
    // diverge.
    if let blk = mergeBlock {
        try switchToAndSeal(self, blk);
    }
}

/// Lower an `if let` statement.
unsafe fn lowerIfLet 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, cond: ast::IfLet) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let savedVarsLen = enterVarScope(self);
    let subject = try lowerMatchSubject(self, cond.pattern.scrutinee);
    let mut thenBlock: BlockId = undefined;
    if cond.pattern.guard == nil {
        set thenBlock = try createBlock(self, "then");
    }
    let elseBlock = try createBlock(self, "else");
    let mut mergeBlock: ?BlockId = nil;

    // Pattern match: jump to @then on success, @else on failure.
    try lowerPatternMatch(self, &subject, &cond.pattern, &mut thenBlock, "then", elseBlock);

    // Lower then branch.
    try lowerNode(self, cond.thenBranch);
    try emitMergeIfUnterminated(self, &mut mergeBlock);
    // Pattern bindings are visible only in the success branch. Restore the
    // outer variable scope before lowering `else`, where a same-named outer
    // variable may be referenced.
    exitVarScope(self, savedVarsLen);

    // Lower else branch.
    try switchToAndSeal(self, elseBlock);
    if let elseBranch = cond.elseBranch {
        try lowerNode(self, elseBranch);
    }
    try emitMergeIfUnterminated(self, &mut mergeBlock);

    if let blk = mergeBlock {
        try switchToAndSeal(self, blk);
    }
}

/// Emit pattern match branch with optional guard, and bind variables.
/// Used by `if-let`, `let-else`, and `while-let` lowering.
///
/// When a guard is present, the guard block is created before `successBlock`
/// to ensure block indices are in RPO.
unsafe fn lowerPatternMatch 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    subject: &MatchSubject,
    pat: &ast::PatternMatch,
    successBlock: &mut BlockId,
    successLabel: *[u8],
    failBlock: BlockId
) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    // If guard present, pattern match jumps to @guard, then guard evaluation
    // jumps to `successBlock` or `failBlock`. Otherwise, jump directly to
    // `successBlock`.
    let mut targetBlock: BlockId = undefined;
    if pat.guard <> nil {
        set targetBlock = try createBlock(self, "guard");
        set *successBlock = try createBlock(self, successLabel);
    } else {
        set targetBlock = *successBlock;
    }
    match pat.kind {
        case ast::PatternKind::Case => {
            let patterns = [pat.pattern];
            // Jump to `targetBlock` if the pattern matches, `failBlock` otherwise.
            try emitPatternMatches(self, subject, &patterns[..], targetBlock, failBlock);
            try switchToAndSeal(self, targetBlock);
            // Bind any variables inside the pattern. Nested patterns may
            // emit additional tests that branch to `failBlock`, switching
            // the current block.
            try bindPatternVariables(self, subject, &patterns[..], failBlock);
        }
        case ast::PatternKind::Binding => {
            // Jump to `targetBlock` if there is a value present, `failBlock` otherwise.
            try emitBindingTest(self, subject, targetBlock, failBlock);
            try switchToAndSeal(self, targetBlock);
            // Bind the matched value to the pattern variable.
            try bindMatchVariable(self, subject, pat.pattern, pat.mutable);
        }
    }
    // Handle guard: on success jump to `successBlock`, on failure jump to `failBlock`.
    if let g = pat.guard {
        try emitCondBranch(self, g, *successBlock, failBlock);
        try switchToAndSeal(self, *successBlock);
    } else if *currentBlock(self) <> *targetBlock {
        // Nested tests changed the current block. Create a new success block
        // after the nest blocks to maintain RPO ordering, and jump to it.
        set *successBlock = try createBlock(self, successLabel);

        try emitJmp(self, *successBlock);
        try switchToAndSeal(self, *successBlock);
    }
}

/// Lower a `let-else` statement.
unsafe fn lowerLetElse 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, letElse: ast::LetElse) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let subject = try lowerMatchSubject(self, letElse.pattern.scrutinee);
    let mut successBlock: BlockId = undefined;
    if letElse.pattern.guard == nil {
        set successBlock = try createBlock(self, "success");
    }
    // The else branch executes when the pattern fails to match.
    let elseBlock = try createBlock(self, "else");

    // Evaluate the pattern and jump to @success or @else.
    try lowerPatternMatch(
        self,
        &subject,
        &letElse.pattern,
        &mut successBlock,
        "success",
        elseBlock,
    );
    let mut bindingVar: ?Var = nil;
    if let case ast::PatternKind::Binding = letElse.pattern.kind {
        set bindingVar = lookupLocalVar(self, letElse.pattern.pattern);
    }
    let mergeBlock = try createBlock(self, "merge");
    try emitJmp(self, mergeBlock);

    try switchToAndSeal(self, elseBlock);
    if try typeOf(self, letElse.elseBranch) == resolver::Type::Never {
        try lowerNode(self, letElse.elseBranch);
    } else {
        let fallback = try lowerExpr(self, letElse.elseBranch);
        if let variable = bindingVar {
            defVar(self, variable, fallback);
        }
        try emitJmp(self, mergeBlock);
    }

    // Continue at @merge after a successful match or value-producing fallback.
    try switchToAndSeal(self, mergeBlock);
}

/// Lower a `while let` loop as a match-driven loop.
unsafe fn lowerWhileLet 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, w: ast::WhileLet) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let savedVarsLen = enterVarScope(self);
    // Create control flow blocks: loop header, body (created lazily when
    // there's a guard), and exit.
    let whileBlock = try createBlock(self, "while");
    let mut bodyBlock: BlockId = undefined;
    if w.pattern.guard == nil {
        set bodyBlock = try createBlock(self, "body");
    }
    let endBlock = try createBlock(self, "merge");

    // Enter loop context and jump to loop header.
    enterLoop(self, endBlock, whileBlock);
    try switchAndJumpTo(self, whileBlock);
    let subject = try lowerMatchSubject(self, w.pattern.scrutinee);

    // Evaluate pattern and jump to loop body or loop end.
    try lowerPatternMatch(self, &subject, &w.pattern, &mut bodyBlock, "body", endBlock);

    // Lower loop body, jump back to loop header, and exit loop context.
    try lowerBlock(self, w.body);
    try emitJmpAndSeal(self, whileBlock);

    exitLoop(self);
    try switchToAndSeal(self, endBlock);
    exitVarScope(self, savedVarsLen);
}

///////////////////
// Node Lowering //
///////////////////

/// Lower an AST node.
unsafe fn lowerNode 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    if self.low.options.debug {
        set self.srcLoc.offset = node.span.offset;
    }
    match node.value {
        case ast::NodeValue::RegionBlock { bindings, body, .. } => {
            let savedVarsLen = enterVarScope(self);
            let values = try allocVals(self, bindings.len);
            for bindingNode, i in bindings {
                let case ast::NodeValue::RegionBinding(binding) = bindingNode.value
                    else throw LowerError::ExpectedIdentifier;
                set values[i] = try lowerExpr(self, binding.value);
                if blockHasTerminator(self) {
                    exitVarScope(self, savedVarsLen);
                    return;
                }
            }
            for bindingNode, i in bindings {
                let case ast::NodeValue::RegionBinding(binding) = bindingNode.value
                    else throw LowerError::ExpectedIdentifier;
                try bindLetValue(self, bindingNode, ast::borrowBinding(binding), values[i]);
            }
            try lowerBlock(self, body);
            exitVarScope(self, savedVarsLen);
        }
        case ast::NodeValue::Block(_) => {
            try lowerBlock(self, node);
        }
        case ast::NodeValue::Return { value } => {
            try lowerReturnStmt(self, node, value);
        }
        case ast::NodeValue::Throw { expr } => {
            try lowerThrowStmt(self, expr);
        }
        case ast::NodeValue::Let(l) => {
            try lowerLet(self, node, l);
        }
        case ast::NodeValue::ConstDecl(decl) => {
            // Local constants lower to data declarations and emit no runtime code.
            try registerLocalDataDeclName(self, node);
            try lowerDataDecl(self.low, node, decl.value, true);
        }
        case ast::NodeValue::StaticDecl(decl) => {
            // Local statics lower to data declarations and emit no runtime code.
            try registerLocalDataDeclName(self, node);
            try lowerDataDecl(self.low, node, decl.value, false);
        }
        case ast::NodeValue::If(i) => {
            try lowerIf(self, i);
        }
        case ast::NodeValue::IfLet(i) => {
            try lowerIfLet(self, i);
        }
        case ast::NodeValue::Assign(a) => {
            try lowerAssign(self, node, a);
        }
        case ast::NodeValue::Loop { body } => {
            try lowerLoop(self, body);
        }
        case ast::NodeValue::While(w) => {
            try lowerWhile(self, w);
        }
        case ast::NodeValue::WhileLet(w) => {
            try lowerWhileLet(self, w);
        }
        case ast::NodeValue::For(f) => {
            try lowerFor(self, node, f);
        }
        case ast::NodeValue::Break => {
            try lowerBreak(self);
        }
        case ast::NodeValue::Continue => {
            try lowerContinue(self);
        }
        case ast::NodeValue::Match(m) => {
            try lowerMatch(self, node, m);
        }
        case ast::NodeValue::LetElse(letElse) => {
            try lowerLetElse(self, letElse);
        }
        case ast::NodeValue::ExprStmt(expr) => {
            let _ = try lowerExpr(self, expr);
        }
        case ast::NodeValue::Panic { .. } => {
            emit(self, il::Instr::Unreachable);
        }
        case ast::NodeValue::Assert { condition, .. } => {
            // Lower `assert <cond>` as: `if not cond { unreachable; }`.
            let thenBlock = try createBlock(self, "assert.fail");
            let endBlock = try createBlock(self, "assert.ok");

            // Branch: if condition is `true`, go to `endBlock`; if `false`, go to `thenBlock`.
            try emitCondBranch(self, condition, endBlock, thenBlock);
            try sealBlock(self, thenBlock);

            // Emit `unreachable` in the failure block.
            switchToBlock(self, thenBlock);
            emit(self, il::Instr::Unreachable);

            // Continue after the assert.
            try switchToAndSeal(self, endBlock);
        }
        else => {
            // Treat as expression statement, discard result.
            let _ = try lowerExpr(self, node);
        }
    }
}

/// Lower a code block.
unsafe fn lowerBlock 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let case ast::NodeValue::Block(blk) = node.value else {
        throw LowerError::ExpectedBlock(node);
    };
    let savedVarsLen = enterVarScope(self);
    for stmt in blk.statements {
        try lowerNode(self, stmt);

        // If the statement diverges, further statements are unreachable.
        if blockHasTerminator(self) {
            exitVarScope(self, savedVarsLen);
            return;
        }
    }
    exitVarScope(self, savedVarsLen);
}

///////////////////////////////////////
// Record and Aggregate Type Helpers //
///////////////////////////////////////

/// Extract the nominal record info from a resolver type.
unsafe fn recordInfoFromType(typ: resolver::Type) -> ?resolver::RecordType {
    let case resolver::Type::Nominal(resolver::NominalType::Record(recInfo)) = typ
        else return nil;

    return recInfo;
}

/// Extract the nominal union info from a resolver type.
unsafe fn unionInfoFromType(typ: resolver::Type) -> ?resolver::UnionType {
    let case resolver::Type::Nominal(resolver::NominalType::Union(unionInfo)) = typ
        else return nil;

    return unionInfo;
}

/// Return the effective type of a node after any coercion applied by
/// the resolver. `lowerExpr` already materializes the coercion in the
/// IL value, so the lowerer must use the post-coercion type when
/// choosing how to compare or store that value.
unsafe fn effectiveType 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node) -> resolver::Type throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let ty = try typeOf(self, node);
    if let coerce = resolver::coercionFor(self.low.resolver, node) {
        if let case resolver::Coercion::OptionalLift(optTy) = coerce {
            return optTy;
        }
    }
    return ty;
}

/// Check if a resolver type lowers to an aggregate in memory.
unsafe fn isAggregateType(typ: resolver::Type) -> bool {
    match typ {
        case resolver::Type::Slice { .. },
             resolver::Type::TraitObject { .. },
             resolver::Type::Session(_) => return true,
        case resolver::Type::Optional(resolver::Type::Pointer { .. }) => {
            // Optional pointers are scalar due to NPO.
            return false;
        }
        case resolver::Type::Optional(_) => {
            // All other optionals, including optional slices, are aggregates.
            return true;
        }
        case resolver::Type::Nominal(_) => {
            // Void unions are small enough to pass by value.
            return not resolver::isVoidUnion(typ);
        }
        case resolver::Type::Array(_),
             resolver::Type::Nil => return true,
        else => return false,
    }
}

/// Check if a resolver type is a small aggregate that can be
/// passed or returned by value in a register.
unsafe fn isSmallAggregate(typ: resolver::Type) -> bool {
    match typ {
        case resolver::Type::Nominal(_) => {
            if resolver::isVoidUnion(typ) {
                return false;
            }
            let layout = resolver::getTypeLayout(typ);
            return layout.size <= resolver::PTR_SIZE;
        }
        else => return false,
    }
}

/// Whether a function needs a hidden return parameter.
///
/// This is the case for throwing functions, which return a result aggregate,
/// and for functions returning large aggregates that cannot be passed in
/// registers.
unsafe fn requiresReturnParam(fnType: *resolver::FnType) -> bool {
    return fnType.throwList.len > 0
        or (isAggregateType(*fnType.returnType)
        and not isSmallAggregate(*fnType.returnType));
}

/// Check if a node is a void union variant literal (e.g. `Color::Red`).
/// If so, returns the variant's tag index. This enables optimized comparisons
/// that only check the tag instead of doing full aggregate comparison.
unsafe fn voidVariantIndex 'arena (res: &resolver::Resolver 'arena, node: *ast::Node) -> ?i64 {
    let data = resolver::nodeData(res, node);
    // Optional equality checks both the presence tag and the union value.
    if let case resolver::Coercion::OptionalLift(_) = data.coercion {
        return nil;
    }
    let sym = data.sym else {
        return nil;
    };
    let case resolver::SymbolData::Variant { type: payloadType, index, .. } = sym.data else {
        return nil;
    };
    // Only void variants can use tag-only comparison.
    if payloadType <> resolver::Type::Void {
        return nil;
    }
    return index as i64;
}

/// Reserve stack storage for a value of the given type.
unsafe fn emitReserve 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, typ: resolver::Type) -> il::Reg throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let layout = resolver::getTypeLayout(typ);
    return emitReserveLayout(self, layout);
}

/// Reserve stack storage with an explicit layout.
unsafe fn emitReserveLayout 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, layout: resolver::Layout) -> il::Reg where 'arena: 'phase, 'phase: 'function {
    let dst = nextReg(self);

    emit(self, il::Instr::Reserve {
        dst,
        size: il::Val::Imm(layout.size as i64),
        alignment: layout.alignment,
    });
    return dst;
}

/// Store a value into an address.
unsafe fn emitStore 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, base: il::Reg, offset: i32, typ: resolver::Type, src: il::Val) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    // `undefined` values need no store.
    if let case il::Val::Undef = src {
        return;
    }
    if isAggregateType(typ) {
        let dst = emitPtrOffset(self, base, offset);
        let src = emitValToReg(self, src);
        let layout = resolver::getTypeLayout(typ);

        emit(self, il::Instr::Blit { dst, src, size: il::Val::Imm(layout.size as i64) });
    } else {
        emit(self, il::Instr::Store {
            typ: ilType(self.low, typ),
            src,
            dst: base,
            offset,
        });
    }
}

/// Allocate stack space for a value and store it. Returns a pointer to the value.
unsafe fn emitStackVal 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, typ: resolver::Type, val: il::Val) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let ptr = try emitReserve(self, typ);
    try emitStore(self, ptr, 0, typ, val);
    return il::Val::Reg(ptr);
}

/// Generic helper to build any tagged aggregate.
/// Reserves space based on the provided layout, stores the tag, and optionally
/// stores the payload value at `valOffset`.
unsafe fn buildTagged 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    layout: resolver::Layout,
    tag: i64,
    payload: ?il::Val,
    payloadType: resolver::Type,
    tagSize: u32,
    valOffset: i32
) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let dst = nextReg(self);
    emit(self, il::Instr::Reserve {
        dst,
        size: il::Val::Imm(layout.size as i64),
        alignment: layout.alignment,
    });
    if tagSize == 1 {
        emitStoreW8At(self, il::Val::Imm(tag), dst, TVAL_TAG_OFFSET);
    } else {
        emitStoreW64At(self, il::Val::Imm(tag), dst, TVAL_TAG_OFFSET);
    }

    if let val = payload {
        if payloadType <> resolver::Type::Void {
            try emitStore(self, dst, valOffset, payloadType, val);
        }
    }
    return il::Val::Reg(dst);
}

/// Wrap a value in an optional type.
///
/// For optional pointers (`?*T`), the value is returned as-is since pointers
/// use zero to represent `nil`. For other optionals, builds a tagged aggregate.
/// with the tag set to `1`, and the value as payload.
unsafe fn wrapInOptional 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, val: il::Val, optType: resolver::Type) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let case resolver::Type::Optional(inner) = optType else {
        throw LowerError::ExpectedOptional;
    };
    // Null-pointer-optimized (NPO) types are used as-is -- valid values are never null.
    if resolver::isNullableType(*inner) {
        return val;
    }
    let layout = resolver::getTypeLayout(optType);
    let valOffset = resolver::getOptionalValOffset(*inner) as i32;

    return try buildTagged(self, layout, 1, val, *inner, 1, valOffset);
}

/// Build a `nil` value for an optional type.
///
/// For optional pointers (`?*T`), returns an immediate `0` (null pointer).
/// For other optionals, builds a tagged aggregate with tag set to `0` (absent).
unsafe fn buildNilOptional 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, optType: resolver::Type) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let case resolver::Type::Optional(inner) = optType
        else throw LowerError::ExpectedOptional;
    if let case resolver::Type::Pointer { .. } = *inner {
        return il::Val::Imm(0);
    }
    if let case resolver::Type::Slice { item, mutable, .. } = *inner {
        return try buildSliceValue(
            self, item, mutable, il::Val::Imm(0), il::Val::Imm(0), il::Val::Imm(0)
        );
    }
    let valOffset = resolver::getOptionalValOffset(*inner) as i32;
    return try buildTagged(self, resolver::getTypeLayout(optType), 0, nil, *inner, 1, valOffset);
}

/// Build a result value for throwing functions.
unsafe fn buildResult 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    tag: i64,
    payload: ?il::Val,
    payloadType: resolver::Type
) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let successType = *self.fnType.returnType;
    let layout = resolver::getResultLayout(
        successType, self.fnType.throwList
    );
    return try buildTagged(self, layout, tag, payload, payloadType, resolver::PTR_SIZE as i32, RESULT_VAL_OFFSET);
}

/// Build a slice aggregate from a data pointer, length and capacity.
unsafe fn buildSliceValue 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    elemTy: *resolver::Type,
    mutable: bool,
    ptrVal: il::Val,
    lenVal: il::Val,
    capVal: il::Val
) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let sliceType = resolver::Type::Slice {
        class: types::PointerClass::Unsafe,
        item: elemTy,
        mutable,
    };
    let dst = try emitReserve(self, sliceType);
    let ptrTy = resolver::Type::Pointer {
        class: types::PointerClass::Unsafe,
        target: elemTy,
        mutable,
    };

    try emitStore(self, dst, SLICE_PTR_OFFSET, ptrTy, ptrVal);
    try emitStore(self, dst, SLICE_LEN_OFFSET, resolver::Type::U32, lenVal);
    try emitStore(self, dst, SLICE_CAP_OFFSET, resolver::Type::U32, capVal);

    return il::Val::Reg(dst);
}

/// Build a trait object fat pointer from a data pointer and a v-table.
unsafe fn buildTraitObject 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    dataVal: il::Val,
    traitInfo: *unsafe resolver::TraitType,
    inst: &resolver::InstanceEntry
) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let vName = vtableName(self.low, inst.moduleId, inst.concreteTypeName, traitInfo.name);

    // Reserve space for the trait object on the stack.
    let slot = emitReserveLayout(self, resolver::Layout {
        size: resolver::PTR_SIZE * 2,
        alignment: resolver::PTR_SIZE,
    });

    // Store data pointer.
    emit(self, il::Instr::Store {
        typ: il::Type::W64,
        src: dataVal,
        dst: slot,
        offset: TRAIT_OBJ_DATA_OFFSET,
    });

    // Store v-table address.
    emit(self, il::Instr::Store {
        typ: il::Type::W64,
        src: il::Val::DataSym(vName),
        dst: slot,
        offset: TRAIT_OBJ_VTABLE_OFFSET,
    });
    return il::Val::Reg(slot);
}

/// Compute a field pointer by adding a byte offset to a base address.
unsafe fn emitPtrOffset 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, base: il::Reg, offset: i32) -> il::Reg where 'arena: 'phase, 'phase: 'function {
    if offset == 0 {
        return base;
    }
    let dst = nextReg(self);

    emit(self, il::Instr::BinOp {
        op: il::BinOp::Add,
        typ: il::Type::W64,
        dst,
        a: il::Val::Reg(base),
        b: il::Val::Imm(offset as i64),
    });
    return dst;
}

/// Emit an element address computation for array/slice indexing.
/// Computes: `base + idx * stride`.
unsafe fn emitElem 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, stride: u32, base: il::Reg, idx: il::Val) -> il::Reg where 'arena: 'phase, 'phase: 'function {
    // If index is zero, return base directly.
    if idx == il::Val::Imm(0) {
        return base;
    }
    // If stride is `1`, skip the multiply.
    if stride == 1 {
        let dst = nextReg(self);

        emit(self, il::Instr::BinOp {
            op: il::BinOp::Add,
            typ: il::Type::W64,
            dst,
            a: il::Val::Reg(base),
            b: idx
        });
        return dst;
    }
    // Compute `offset = idx * stride`.
    let offset = nextReg(self);

    emit(self, il::Instr::BinOp {
        op: il::BinOp::Mul,
        typ: il::Type::W64,
        dst: offset,
        a: idx,
        b: il::Val::Imm(stride as i64)
    });
    // Compute `dst = base + offset`.
    let dst = nextReg(self);
    emit(self, il::Instr::BinOp {
        op: il::BinOp::Add,
        typ: il::Type::W64,
        dst,
        a: il::Val::Reg(base),
        b: il::Val::Reg(offset)
    });
    return dst;
}

/// Emit a typed binary operation, returning the result as a value.
unsafe fn emitTypedBinOp 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, op: il::BinOp, typ: il::Type, a: il::Val, b: il::Val) -> il::Val where 'arena: 'phase, 'phase: 'function {
    let dst = nextReg(self);
    emit(self, il::Instr::BinOp { op, typ, dst, a, b });
    return il::Val::Reg(dst);
}

/// Emit a tag comparison for void variant equality/inequality.
unsafe fn emitTagCmp 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, op: ast::BinaryOp, val: il::Val, tagIdx: i64, valType: resolver::Type) -> il::Val
    throws (LowerError) where 'arena: 'phase, 'phase: 'function
{
    let reg = emitValToReg(self, val);

    // For all-void unions, the value *is* the tag, not a pointer.
    let mut tag: il::Val = undefined;
    if resolver::isVoidUnion(valType) {
        set tag = il::Val::Reg(reg);
    } else {
        set tag = loadTag(self, reg, TVAL_TAG_OFFSET, il::Type::W8);
    }
    let binOp = il::BinOp::Eq if op == ast::BinaryOp::Eq else il::BinOp::Ne;
    return emitTypedBinOp(self, binOp, il::Type::W8, tag, il::Val::Imm(tagIdx));
}

/// Logical "and" between two values. Returns the result in a register.
unsafe fn emitLogicalAnd 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, left: ?il::Val, right: il::Val) -> il::Val where 'arena: 'phase, 'phase: 'function {
    let prev = left else {
        return right;
    };
    return emitTypedBinOp(self, il::BinOp::And, il::Type::W32, prev, right);
}

//////////////////////////
// Aggregate Comparison //
//////////////////////////

/// Emit an equality test for values at an offset of the given base registers.
unsafe fn emitEqAtOffset 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    left: il::Reg,
    right: il::Reg,
    offset: i32,
    fieldType: resolver::Type
) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    // For aggregate types, pass offset through and compare recursively.
    if isAggregateType(fieldType) {
        return try lowerAggregateEq(self, fieldType, left, right, offset);
    }
    // For scalar types, load and compare directly.
    let a = emitLoad(self, left, offset, fieldType);
    let b = emitLoad(self, right, offset, fieldType);
    let dst = nextReg(self);
    emit(self, il::Instr::BinOp { op: il::BinOp::Eq, typ: ilType(self.low, fieldType), dst, a, b });

    return il::Val::Reg(dst);
}

/// Compare two record values for equality.
unsafe fn lowerRecordEq 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    recInfo: resolver::RecordType,
    a: il::Reg,
    b: il::Reg,
    offset: i32
) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let mut result: ?il::Val = nil;

    for field in recInfo.fields {
        let cmp = try emitEqAtOffset(self, a, b, offset + field.offset, field.fieldType);

        set result = emitLogicalAnd(self, result, cmp);
    }
    if let r = result {
        return r;
    }
    return il::Val::Imm(1);
}

/// Compare two slice values for equality.
unsafe fn lowerSliceEq 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    elemTy: *resolver::Type,
    mutable: bool,
    a: il::Reg,
    b: il::Reg,
    offset: i32
) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let ptrTy = resolver::Type::Pointer {
        class: types::PointerClass::Unsafe,
        target: elemTy,
        mutable,
    };
    let ptrEq = try emitEqAtOffset(self, a, b, offset + SLICE_PTR_OFFSET, ptrTy);
    let lenEq = try emitEqAtOffset(self, a, b, offset + SLICE_LEN_OFFSET, resolver::Type::U32);

    return emitTypedBinOp(self, il::BinOp::And, il::Type::W32, ptrEq, lenEq);
}

/// Compare two optional aggregate values for equality.
///
/// Two optionals are equal when their tags match and either both are `nil` or
/// their payloads are equal.
///
/// For inner types that are safe to compare even when uninitialised, we use a
/// branchless formulation: `tagEq AND (tagNil OR payloadEq)`
///
/// For inner types that may contain uninitialized data when `nil` (unions,
/// nested optionals), the payload comparison is guarded behind a branch
/// so that `nil` payloads are never inspected.
unsafe fn lowerOptionalEq 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    inner: resolver::Type,
    a: il::Reg,
    b: il::Reg,
    offset: i32
) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let valOffset = resolver::getOptionalValOffset(inner) as i32;

    // Load tags.
    let tagA = loadTag(self, a, offset + TVAL_TAG_OFFSET, il::Type::W8);
    let tagB = loadTag(self, b, offset + TVAL_TAG_OFFSET, il::Type::W8);

    // For simple inner types (no unions/nested optionals), use branchless comparison.
    // Unions and nested optionals may contain uninitialized payload bytes
    // when nil, so they need a guarded comparison.
    let isUnion = unionInfoFromType(inner) <> nil;
    let mut isOptional = false;
    if let case resolver::Type::Optional(_) = inner {
        set isOptional = true;
    }
    if not isUnion and not isOptional {
        let tagEq = emitTypedBinOp(self, il::BinOp::Eq, il::Type::W8, tagA, tagB);
        let tagNil = emitTypedBinOp(self, il::BinOp::Eq, il::Type::W8, tagA, il::Val::Imm(0));
        let payloadEq = try emitEqAtOffset(self, a, b, offset + valOffset, inner);

        return emitTypedBinOp(self, il::BinOp::And, il::Type::W32, tagEq,
            emitTypedBinOp(self, il::BinOp::Or, il::Type::W32, tagNil, payloadEq));
    }

    // For complex inner types, use branching comparison to avoid inspecting
    // uninitialized payload bytes.
    let resultReg = nextReg(self);
    let mergeBlock = try createBlockWithParam(self, "opteq#merge", il::Param {
        value: resultReg, type: il::Type::W8
    });
    let nilCheck = try createBlock(self, "opteq#nil");
    let payloadCmp = try createBlock(self, "opteq#payload");

    let falseArgs = try allocVal(self, il::Val::Imm(0));
    let trueArgs = try allocVal(self, il::Val::Imm(1));

    // Check if tags differ.
    emit(self, il::Instr::Br {
        op: il::CmpOp::Eq, typ: il::Type::W8, a: tagA, b: tagB,
        thenTarget: *nilCheck, thenArgs: &mut [],
        elseTarget: *mergeBlock, elseArgs: falseArgs,
    });
    addPredecessor(self, nilCheck, currentBlock(self));
    addPredecessor(self, mergeBlock, currentBlock(self));

    // Check if both are `nil`.
    try switchToAndSeal(self, nilCheck);
    emit(self, il::Instr::Br {
        op: il::CmpOp::Ne, typ: il::Type::W8, a: tagA, b: il::Val::Imm(0),
        thenTarget: *payloadCmp, thenArgs: &mut [],
        elseTarget: *mergeBlock, elseArgs: trueArgs,
    });
    addPredecessor(self, payloadCmp, currentBlock(self));
    addPredecessor(self, mergeBlock, currentBlock(self));

    // Both are non-`nil`, compare payloads.
    try switchToAndSeal(self, payloadCmp);
    let payloadEq = try emitEqAtOffset(self, a, b, offset + valOffset, inner);
    try emitJmpWithArg(self, mergeBlock, payloadEq);
    try switchToAndSeal(self, mergeBlock);

    return il::Val::Reg(resultReg);
}

/// Compare two union values for equality.
///
/// Two unions are equal iff their tags match and, for non-void variants,
/// their payloads are also equal. The comparison proceeds as follows.
///
/// First, compare the tags. If they differ, the unions are not equal, so
/// jump to the merge block with `false`. If they match, jump to the tag
/// block to determine which variant we're dealing with.
///
/// The tag block uses a switch on the tag value to dispatch to the appropriate
/// comparison block. Void variants jump directly to merge with `true`.
/// Non-void variants each have their own payload block that compares the
/// payload and jumps to the merge block with the result.
///
/// The merge block collects results from all paths via a block parameter
/// and returns the final equality result.
///
/// For all-void unions, we skip the control flow entirely and just compare
/// the tags directly.
///
/// TODO: Could be optimized to branchless when all non-void variants share
/// the same payload type: `tagEq AND (isVoidVariant OR payloadEq)`.
unsafe fn lowerUnionEq 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    unionInfo: resolver::UnionType,
    a: il::Reg,
    b: il::Reg,
    offset: i32
) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    // Compare tags.
    let tagA = loadTag(self, a, offset + TVAL_TAG_OFFSET, il::Type::W8);
    let tagB = loadTag(self, b, offset + TVAL_TAG_OFFSET, il::Type::W8);

    // Fast path: all-void union just needs tag comparison.
    if unionInfo.isAllVoid {
        return emitTypedBinOp(self, il::BinOp::Eq, il::Type::W8, tagA, tagB);
    }
    // Holds the equality result.
    let resultReg = nextReg(self);

    // Where control flow continues after equality check is done. Receives
    // the result as a parameter.
    let mergeBlock = try createBlockWithParam(self, "eq#merge", il::Param {
        value: resultReg, type: il::Type::W8
    });
    // Where we switch on the tag to compare payloads.
    let tagBlock = try createBlock(self, "eq#tag");

    // Compare tags: if they differ, jump to merge with `false`; otherwise check payloads.
    let falseArgs = try allocVal(self, il::Val::Imm(0));

    assert tagBlock <> mergeBlock;

    // TODO: Use the helper once the compiler supports more than eight function params.
    emit(self, il::Instr::Br {
        op: il::CmpOp::Eq, typ: il::Type::W8, a: tagA, b: tagB,
        thenTarget: *tagBlock, thenArgs: &mut [],
        elseTarget: *mergeBlock, elseArgs: falseArgs,
    });
    addPredecessor(self, tagBlock, currentBlock(self));
    addPredecessor(self, mergeBlock, currentBlock(self));

    // Create comparison blocks for each non-void variant and build switch cases.
    // Void variants jump directly to merge with `true`.
    let trueArgs = try allocVal(self, il::Val::Imm(1));
    let cases = try! alloc::allocRawSlice(
        self.arena, @sizeOf(il::SwitchCase), @alignOf(il::SwitchCase), unionInfo.variants.len as u32
    ) as *unsafe mut [il::SwitchCase];

    let mut caseBlocks: [?BlockId; resolver::MAX_UNION_VARIANTS] = undefined;
    for variant, i in unionInfo.variants {
        if variant.valueType == resolver::Type::Void {
            set cases[i] = il::SwitchCase {
                value: i as i64,
                target: *mergeBlock,
                args: trueArgs
            };
            set caseBlocks[i] = nil;
        } else {
            let payloadBlock = try createBlock(self, "eq#payload");
            set cases[i] = il::SwitchCase {
                value: i as i64,
                target: *payloadBlock,
                args: &mut []
            };
            set caseBlocks[i] = payloadBlock;
        }
    }

    // Emit switch in @tag block. Default arm is unreachable since we cover all variants.
    let unreachableBlock = try createBlock(self, "eq#unreachable");
    try switchToAndSeal(self, tagBlock);
    emit(self, il::Instr::Switch {
        val: tagA,
        defaultTarget: *unreachableBlock,
        defaultArgs: &mut [],
        cases
    });

    // Add predecessor edges for switch targets.
    addPredecessor(self, unreachableBlock, tagBlock);
    for i in 0..unionInfo.variants.len {
        if let caseBlock = caseBlocks[i] {
            addPredecessor(self, caseBlock, tagBlock);
        } else {
            addPredecessor(self, mergeBlock, tagBlock);
        }
    }
    let valOffset = unionInfo.valOffset as i32;

    // Emit payload comparison blocks for non-void variants.
    for variant, i in unionInfo.variants {
        if let caseBlock = caseBlocks[i] {
            try switchToAndSeal(self, caseBlock);
            let payloadEq = try emitEqAtOffset(
                self, a, b, offset + valOffset, variant.valueType
            );
            try emitJmpWithArg(self, mergeBlock, payloadEq);
        }
    }
    // Emit unreachable block.
    try switchToAndSeal(self, unreachableBlock);
    emit(self, il::Instr::Unreachable);

    try switchToAndSeal(self, mergeBlock);
    return il::Val::Reg(resultReg);
}

/// Compare two array values for equality, element by element.
unsafe fn lowerArrayEq 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    arr: resolver::ArrayType,
    a: il::Reg,
    b: il::Reg,
    offset: i32
) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let elemLayout = resolver::getTypeLayout(*arr.item);
    let stride = elemLayout.size as i32;
    let mut result: ?il::Val = nil;

    for i in 0..arr.length {
        let elemOffset = offset + (i as i32) * stride;
        let cmp = try emitEqAtOffset(self, a, b, elemOffset, *arr.item);
        set result = emitLogicalAnd(self, result, cmp);
    }
    if let r = result {
        return r;
    }
    // Empty arrays are always equal.
    return il::Val::Imm(1);
}

/// Compare two aggregate values for equality.
unsafe fn lowerAggregateEq 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    typ: resolver::Type,
    a: il::Reg,
    b: il::Reg,
    offset: i32
) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    match typ {
        case resolver::Type::Slice { item, mutable, .. } =>
            return try lowerSliceEq(self, item, mutable, a, b, offset),
        case resolver::Type::Optional(inner) => {
            if let case resolver::Type::Slice { item, mutable, .. } = *inner {
                // Optional slices use null pointer optimization.
                return try lowerSliceEq(self, item, mutable, a, b, offset);
            }
            return try lowerOptionalEq(self, *inner, a, b, offset);
        }
        case resolver::Type::Array(arr) =>
            return try lowerArrayEq(self, arr, a, b, offset),
        case resolver::Type::Nominal(resolver::NominalType::Record(recInfo)) =>
            return try lowerRecordEq(self, recInfo, a, b, offset),
        case resolver::Type::Nominal(resolver::NominalType::Union(unionInfo)) =>
            return try lowerUnionEq(self, unionInfo, a, b, offset),
        else => {
            let recInfo = recordInfoFromType(typ) else {
                throw LowerError::ExpectedRecord;
            };
            return try lowerRecordEq(self, recInfo, a, b, offset);
        }
    }
}

/// Lower a record literal expression. Handles both plain records and union variant
/// record literals like `Union::Variant { field: value }`.
unsafe fn lowerRecordLit 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node, lit: ast::RecordLit) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let typ = try typeOf(self, node);
    match typ {
        case resolver::Type::Nominal(resolver::NominalType::Record(recInfo)) => {
            let dst = try emitReserve(self, typ);
            try lowerRecordFields(self, dst, &recInfo, lit.fields, 0);

            return il::Val::Reg(dst);
        }
        case resolver::Type::Nominal(resolver::NominalType::Union(_)) => {
            let typeName = lit.typeName else {
                throw LowerError::ExpectedVariant;
            };
            let sym = try symOf(self, typeName);
            let case resolver::SymbolData::Variant { type: payloadType, index, .. } = sym.data else {
                throw LowerError::ExpectedVariant;
            };
            let recInfo = recordInfoFromType(payloadType) else {
                throw LowerError::ExpectedRecord;
            };
            let unionInfo = unionInfoFromType(typ) else {
                throw LowerError::MissingMetadata;
            };
            let valOffset = unionInfo.valOffset as i32;
            let dst = try emitReserve(self, typ);

            emitStoreW8At(self, il::Val::Imm(index as i64), dst, TVAL_TAG_OFFSET);
            try lowerRecordFields(self, dst, &recInfo, lit.fields, valOffset);

            return il::Val::Reg(dst);
        }
        else => throw LowerError::UnexpectedType(typ),
    }
}

/// Lower fields of a record literal into a destination register.
/// The `offset` is added to each field's offset when storing.
unsafe fn lowerRecordFields 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    dst: il::Reg,
    recInfo: &resolver::RecordType,
    fields: *[*ast::Node],
    offset: i32
) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    for fieldNode, i in fields {
        let case ast::NodeValue::RecordLitField(field) = fieldNode.value else {
            throw LowerError::UnexpectedNodeValue(fieldNode);
        };
        let mut fieldIdx: u32 = i;
        if recInfo.labeled {
            let idx = resolver::recordFieldIndexFor(self.low.resolver, fieldNode) else {
                throw LowerError::MissingMetadata;
            };
            set fieldIdx = idx;
        }
        // Skip `undefined` fields, they need no initialization.
        // Emitting a blit from an uninitialised reserve produces a
        // phantom SSA source value that the backend cannot handle.
        if not isUndef(field.value) {
            let fieldTy = recInfo.fields[fieldIdx].fieldType;
            let fieldVal = try lowerExpr(self, field.value);
            try emitStore(self, dst, offset + recInfo.fields[fieldIdx].offset, fieldTy, fieldVal);
        }
    }
}

/// Lower an unlabeled record constructor call.
unsafe fn lowerRecordCtor 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, nominal: *unsafe resolver::NominalType, args: *[*ast::Node]) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let case resolver::NominalType::Record(recInfo) = *nominal else {
        throw LowerError::ExpectedRecord;
    };
    let typ = resolver::Type::Nominal(nominal);
    let dst = try emitReserve(self, typ);

    for argNode, i in args {
        // Skip `undefined` arguments.
        if not isUndef(argNode) {
            let fieldTy = recInfo.fields[i].fieldType;
            let argVal = try lowerExpr(self, argNode);
            try emitStore(self, dst, recInfo.fields[i].offset, fieldTy, argVal);
        }
    }
    return il::Val::Reg(dst);
}

/// Lower an array literal expression like `[1, 2, 3]`.
unsafe fn lowerArrayLit 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node, elements: *[*ast::Node]) -> il::Val
    throws (LowerError) where 'arena: 'phase, 'phase: 'function
{
    let typ = try typeOf(self, node);
    let case resolver::Type::Array(arrInfo) = typ else {
        throw LowerError::ExpectedArray;
    };
    let elemTy = *arrInfo.item;
    let elemLayout = resolver::getTypeLayout(elemTy);
    let dst = try emitReserve(self, typ);

    for elemNode, i in elements {
        let elemVal = try lowerExpr(self, elemNode);
        let offset = i * elemLayout.size;

        try emitStore(self, dst, offset as i32, elemTy, elemVal);
    }
    return il::Val::Reg(dst);
}

/// Lower an array repeat literal expression like `[42; 3]`.
/// Unrolls the initialization at compile time.
// TODO: Beyond a certain length, lower this to a loop.
unsafe fn lowerArrayRepeatLit 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node, repeat: ast::ArrayRepeatLit) -> il::Val
    throws (LowerError) where 'arena: 'phase, 'phase: 'function
{
    let typ = try typeOf(self, node);
    let case resolver::Type::Array(arrInfo) = typ else {
        throw LowerError::ExpectedArray;
    };
    let elemTy = *arrInfo.item;
    let length = arrInfo.length;
    let elemLayout = resolver::getTypeLayout(elemTy);
    let dst = try emitReserve(self, typ);

    // Evaluate the repeated item once.
    let repeatVal = try lowerExpr(self, repeat.item);

    // Unroll: store at each offset.
    for i in 0..length {
        let offset = i * elemLayout.size;
        try emitStore(self, dst, offset as i32, elemTy, repeatVal);
    }
    return il::Val::Reg(dst);
}

/// Lower a union constructor call like `Union::Variant(...)`.
unsafe fn lowerUnionCtor 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node, sym: *unsafe mut resolver::Symbol, call: ast::Call) -> il::Val
    throws (LowerError) where 'arena: 'phase, 'phase: 'function
{
    let unionTy = try typeOf(self, node);
    let case resolver::SymbolData::Variant { type: payloadType, index, .. } = sym.data else {
        throw LowerError::ExpectedVariant;
    };
    let unionInfo = unionInfoFromType(unionTy) else {
        throw LowerError::MissingMetadata;
    };
    let valOffset = unionInfo.valOffset as i32;
    let mut payloadVal: ?il::Val = nil;
    if payloadType <> resolver::Type::Void {
        let case resolver::Type::Nominal(payloadNominal) = payloadType else {
            throw LowerError::MissingMetadata;
        };
        set payloadVal = try lowerRecordCtor(self, payloadNominal, call.args);
    }
    return try buildTagged(self, resolver::getTypeLayout(unionTy), index as i64, payloadVal, payloadType, 1, valOffset);
}

/// Lower a field access into a pointer to the field.
unsafe fn lowerFieldRef 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, access: ast::Access) -> FieldRef throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let parentTy = try typeOf(self, access.parent);
    let subjectTy = resolver::autoDeref(parentTy);
    let fieldIdx = resolver::recordFieldIndexFor(self.low.resolver, access.child) else {
        throw LowerError::MissingMetadata;
    };
    let fieldInfo = resolver::getRecordField(subjectTy, fieldIdx) else {
        throw LowerError::FieldNotFound;
    };
    let baseVal = try lowerExpr(self, access.parent);
    let baseReg = emitValToReg(self, baseVal);

    return FieldRef {
        base: baseReg,
        offset: fieldInfo.offset,
        fieldType: fieldInfo.fieldType,
    };
}

/// Lower a field access expression.
unsafe fn lowerFieldAccess 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, access: ast::Access) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let fieldRef = try lowerFieldRef(self, access);
    return emitRead(self, fieldRef.base, fieldRef.offset, fieldRef.fieldType);
}

/// Compute data pointer and element count for a range into a container.
/// Used by both slice range expressions (`&a[start..end]`) and slice
/// assignments (`a[start..end] = value`).
unsafe fn resolveSliceRangePtr 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    container: *ast::Node,
    range: ast::Range,
    info: resolver::SliceRangeInfo
) -> SliceRangeResult throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let baseVal = try lowerExpr(self, container);
    let baseReg = emitValToReg(self, baseVal);

    // Extract data pointer and container length.
    let mut dataReg = baseReg;
    let mut containerLen: il::Val = undefined;
    if let cap = info.capacity { // Slice from array.
        set containerLen = il::Val::Imm(cap as i64);
    } else { // Slice from slice.
        set dataReg = loadSlicePtr(self, baseReg);
        set containerLen = loadSliceLen(self, baseReg);
    }

    // Compute range bounds.
    let mut startVal: il::Val = il::Val::Imm(0);
    if let start = range.start {
        set startVal = try lowerExpr(self, start);
    }
    let mut endVal = containerLen;
    if let end = range.end {
        set endVal = try lowerExpr(self, end);
    }

    // Runtime slice bounds checks for dynamic range expressions.
    if not isLtEq(startVal, containerLen) {
        try emitTrapIfLt(self, il::Type::W32, containerLen, startVal);
    }
    if not isLtEq(endVal, containerLen) {
        try emitTrapIfLt(self, il::Type::W32, containerLen, endVal);
    }
    if startVal <> il::Val::Imm(0) and endVal <> containerLen and not isLtEq(startVal, endVal) {
        try emitTrapIfLt(self, il::Type::W32, endVal, startVal);
    }

    // If the start value is known to be zero, the count is just the end
    // value. Otherwise, we have to compute it.
    let mut count = endVal;

    // Only compute range offset and count if the start value is not
    // statically known to be zero.
    if startVal <> il::Val::Imm(0) {
        // Offset the data pointer by the start value.
        set dataReg = emitElem(
            self, resolver::getTypeLayout(*info.itemType).size, dataReg, startVal
        );
        // Compute the count as `end - start`.
        let lenReg = nextReg(self);
        emit(self, il::Instr::BinOp {
            op: il::BinOp::Sub,
            typ: il::Type::W32,
            dst: lenReg,
            a: endVal,
            b: startVal,
        });
        set count = il::Val::Reg(lenReg);
    }
    return SliceRangeResult { dataReg, count };
}

/// Lower a slice range expression into a slice header value.
unsafe fn lowerSliceRange 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    container: *ast::Node,
    range: ast::Range,
    sliceNode: *ast::Node
) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let info = resolver::sliceRangeInfoFor(self.low.resolver, sliceNode) else {
        throw LowerError::MissingMetadata;
    };
    let r = try resolveSliceRangePtr(self, container, range, info);
    return try buildSliceValue(
        self, info.itemType, info.mutable, il::Val::Reg(r.dataReg), r.count, r.count
    );
}

/// Lower an address-of (`&x`) expression.
unsafe fn lowerAddressOf 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node, addr: ast::AddressOf) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    // Handle subscript: `&ary[i]` or `&ary[start..end]`.
    if let case ast::NodeValue::Subscript { container, index } = addr.target.value {
        if let case ast::NodeValue::Range(range) = index.value {
            return try lowerSliceRange(self, container, range, node);
        }
        let result = try lowerElemPtr(self, container, index);

        return il::Val::Reg(result.elemReg);
    }
    // Handle field address: `&x.field`.
    if let case ast::NodeValue::FieldAccess(access) = addr.target.value {
        let fieldRef = try lowerFieldRef(self, access);
        let ptr = emitPtrOffset(self, fieldRef.base, fieldRef.offset);

        return il::Val::Reg(ptr);
    }
    // A qualified constant or static uses its resolved owner's data symbol.
    if let case ast::NodeValue::ScopeAccess(_) = addr.target.value {
        let sym = resolver::nodeData(self.low.resolver, addr.target).sym else {
            throw LowerError::MissingSymbol(addr.target);
        };
        return il::Val::Reg(emitDataAddr(self, sym));
    }
    // Handle variable address: `&x`
    if let case ast::NodeValue::Ident(_) = addr.target.value {
        if let v = lookupLocalVar(self, addr.target) {
            let val = try useVar(self, v);
            let typ = try typeOf(self, addr.target);
            // For aggregates, the value is already a pointer.
            if isAggregateType(typ) {
                return val;
            }
            // For scalars, if we've already materialized a stack slot for this
            // variable, the SSA value is that slot pointer.
            if self.vars[*v].addressTaken {
                // Already address-taken; return existing stack pointer.
                return val;
            }
            // Materialize a stack slot using the declaration's resolved
            // layout so `align(N)` on locals is honored.
            let layout = resolver::getLayout(self.low.resolver, addr.target, typ);
            let slot = emitReserveLayout(self, layout);
            try emitStore(self, slot, 0, typ, val);
            let stackVal = il::Val::Reg(slot);

            set self.vars[*v].addressTaken = true;
            defVar(self, v, stackVal);

            return stackVal;
        }
        // Fall back to symbol lookup for constants/statics.
        if let sym = resolver::nodeData(self.low.resolver, addr.target).sym {
            return il::Val::Reg(emitDataAddr(self, sym));
        } else {
            throw LowerError::MissingSymbol(node);
        }
    }
    // Handle dereference address: `&(*ptr) = ptr`.
    if let case ast::NodeValue::Deref(target) = addr.target.value {
        return try lowerExpr(self, target);
    }
    // Array literals become slices when their address is taken.
    match addr.target.value {
        case ast::NodeValue::ArrayLit(_),
             ast::NodeValue::ArrayRepeatLit(_) =>
        {
            return try lowerArrayLiteralSlice(self, node, addr.target);
        }
        else => {}
    }
    throw LowerError::UnexpectedNodeValue(addr.target);
}

/// Lower an addressed array literal as a slice.
unsafe fn lowerArrayLiteralSlice 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    sliceNode: *ast::Node,
    arrayNode: *ast::Node
) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let sliceTy = try typeOf(self, sliceNode);
    let case resolver::Type::Slice { item, mutable, .. } = sliceTy else {
        throw LowerError::UnexpectedType(sliceTy);
    };
    let arrayTy = try typeOf(self, arrayNode);
    let case resolver::Type::Array(arrayInfo) = arrayTy else {
        throw LowerError::ExpectedArray;
    };
    let length = arrayInfo.length;
    if length == 0 {
        return try buildSliceValue(
            self, item, mutable, il::Val::Imm(0), il::Val::Imm(0), il::Val::Imm(0)
        );
    }
    if resolver::isConstExpr(self.low.resolver, arrayNode) {
        let fnName = self.fnName;
        let mut b = dataBuilder(alloc::arenaAllocator(self.low.arena));
        match arrayNode.value {
            case ast::NodeValue::ArrayLit(elements) =>
                try lowerConstArrayLitInto(self.low, elements, arrayTy, fnName, &mut b),
            case ast::NodeValue::ArrayRepeatLit(repeat) =>
                try lowerConstArrayRepeatInto(self.low, repeat, arrayTy, fnName, &mut b),
            else => throw LowerError::UnexpectedNodeValue(arrayNode),
        }
        let result = dataBuilderFinish(b);
        let alignment = resolver::getTypeLayout(*item).alignment;
        return try lowerConstDataAsSlice(
            self, &result, alignment, not mutable,
            item, mutable, length
        );
    }
    let data = try lowerExpr(self, arrayNode);
    let count = il::Val::Imm(length as i64);
    return try buildSliceValue(self, item, mutable, data, count, count);
}

/// Lower the common element pointer computation for subscript operations.
/// Handles both arrays and slices by resolving the container type, extracting
/// the data pointer (for slices), and emitting an [`il::Instr::Elem`] to compute
/// the element address.
unsafe fn lowerElemPtr 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function, container: *ast::Node, index: *ast::Node
) -> ElemPtrResult throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let containerTy = try typeOf(self, container);
    let subjectTy = resolver::autoDeref(containerTy);
    let baseVal = try lowerExpr(self, container);
    let indexVal = try lowerExpr(self, index);
    let baseReg = emitValToReg(self, baseVal);

    let mut dataReg = baseReg;
    let mut elemType: resolver::Type = undefined;

    match subjectTy {
        case resolver::Type::Slice { item, .. } => {
            set elemType = *item;
            let sliceLen = loadSliceLen(self, baseReg);
            // Runtime safety check: index must be strictly less than slice length.
            try emitTrapUnlessCmp(self, il::CmpOp::Ult, il::Type::W32, indexVal, sliceLen);

            set dataReg = loadSlicePtr(self, baseReg);
        }
        case resolver::Type::Array(arrInfo) => {
            set elemType = *arrInfo.item;
            // Runtime safety check: index must be strictly less than array length.
            // Skip when the index is a compile-time constant, since we check
            // that in the resolver.
            if not resolver::isConstExpr(self.low.resolver, index) {
                let arrLen = il::Val::Imm(arrInfo.length as i64);
                try emitTrapUnlessCmp(self, il::CmpOp::Ult, il::Type::W32, indexVal, arrLen);
            }
        }
        else => throw LowerError::ExpectedSliceOrArray,
    }
    let elemLayout = resolver::getTypeLayout(elemType);
    let elemReg = emitElem(self, elemLayout.size, dataReg, indexVal);

    return ElemPtrResult { elemReg, elemType };
}

/// Lower a dereference expression.
/// Handles both pointer deref (`*ptr`) and record deref (`*r` on single-field
/// unlabeled record). Both read at offset 0 using the resolver-assigned type.
unsafe fn lowerDeref 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node, target: *ast::Node) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let type = try typeOf(self, node);
    let ptrVal = try lowerExpr(self, target);
    let ptrReg = emitValToReg(self, ptrVal);

    let value = emitRead(self, ptrReg, 0, type);
    if let case resolver::Type::Cell { .. } = try typeOf(self, target); isAggregateType(type) {
        return try emitStackVal(self, type, value);
    }
    return value;
}

/// Lower a subscript expression.
unsafe fn lowerSubscript 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node, container: *ast::Node, index: *ast::Node) -> il::Val
    throws (LowerError) where 'arena: 'phase, 'phase: 'function
{
    if let case ast::NodeValue::Range(_) = index.value {
        panic "lowerSubscript: range subscript must use address-of (&)";
    }
    let result = try lowerElemPtr(self, container, index);

    return emitRead(self, result.elemReg, 0, result.elemType);
}

/// Lower a let binding.
unsafe fn lowerLet 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node, l: ast::Let) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    // Evaluate value.
    let val = try lowerExpr(self, l.value);
    if blockHasTerminator(self) {
        return;
    }
    try bindLetValue(self, node, l, val);
}

/// Bind an evaluated initializer in the current variable scope.
unsafe fn bindLetValue 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node, l: ast::Let, val: il::Val)
    throws (LowerError) where 'arena: 'phase, 'phase: 'function
{
    // Handle placeholder pattern: `let _ = expr;`
    if let case ast::NodeValue::Placeholder = l.ident.value {
        return;
    }
    let case ast::NodeValue::Ident(name) = l.ident.value else {
        throw LowerError::ExpectedIdentifier;
    };
    let typ = try typeOf(self, l.ident);
    let ilType = ilType(self.low, typ);
    let mut varVal = val;

    // Aggregates with persistent storage need a local copy to avoid aliasing.
    // Temporaries such as literals or call results can be adopted directly.
    // This is because aggregates are represented as memory addresses
    // internally, even though they have value semantics, so without an explicit
    // copy, only the address is is written. Function calls on the other hand
    // reserve their own local stack space, so copying would be redundant.
    // Void variant literals (e.g. `Option::None`) use scope access syntax and
    // are flagged as place expressions, but they are freshly constructed
    // temporaries with no persistent storage.
    if isAggregateType(typ) and
        ast::isPlaceExpr(l.value) and
        not resolver::isCellDeref(self.low.resolver, l.value) and
        voidVariantIndex(self.low.resolver, l.value) == nil {
        set varVal = try emitStackVal(self, typ, val);
    }

    // If the resolver determined that this variable's address is taken
    // anywhere in the function, allocate a stack slot immediately so the
    // SSA value is always a pointer. This avoids mixing integer and pointer
    // values in loop phis when `&var` or `&mut var` appears inside a loop.
    if not isAggregateType(typ) {
        if let sym = resolver::nodeData(self.low.resolver, node).sym {
            if let case resolver::SymbolData::Value { addressTaken, .. } = sym.data; addressTaken {
                let layout = resolver::getLayout(self.low.resolver, node, typ);
                let slot = emitReserveLayout(self, layout);
                try emitStore(self, slot, 0, typ, varVal);

                let v = newVar(self, name, ilType, l.mutable, il::Val::Reg(slot));
                set self.vars[*v].addressTaken = true;

                return;
            }
        }
    }
    let _ = newVar(self, name, ilType, l.mutable, varVal);
}

/// Lower an if statement: `if <cond> { <then> } else { <else> }`.
///
/// With else branch:
///
///     @entry -> (true)  @then ---> @merge <--.
///         |                                   )
///         `---> (false) @else ---------------'
///
/// Without else branch:
///
///     @entry -> (true)  @then ---> @end <--.
///         |                                 )
///         `---- (false) -------------------'
///
unsafe fn lowerIf 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, i: ast::If) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let thenBlock = try createBlock(self, "then");

    if let elseNode = i.elseBranch { // If-else case.
        let elseBlock = try createBlock(self, "else");
        try emitCondBranch(self, i.condition, thenBlock, elseBlock);

        // Both @then and @else have exactly one predecessor (@entry), so we can
        // seal them immediately.
        try sealBlock(self, thenBlock);
        try sealBlock(self, elseBlock);

        // The merge block is created lazily by [`emitMergeIfUnterminated`]. We
        // only need it if at least one branch doesn't diverge (i.e., needs to
        // continue execution after the `if`). If both branches diverge (eg.
        // both `return`), no merge block is created and control flow
        // doesn't continue past the `if` statement.
        let mut mergeBlock: ?BlockId = nil;

        // Lower the @then block: switch to it, emit its code, then jump to
        // merge if the block doesn't diverge.
        switchToBlock(self, thenBlock);
        try lowerBlock(self, i.thenBranch);
        try emitMergeIfUnterminated(self, &mut mergeBlock);

        // Lower the @else block similarly.
        switchToBlock(self, elseBlock);
        try lowerBlock(self, elseNode);
        try emitMergeIfUnterminated(self, &mut mergeBlock);

        // If a merge block was created (at least one branch flows into it),
        // switch to it for subsequent code. The merge block's predecessors
        // are the branches that jumped to it, so we seal it now.
        // If the merge block is `nil`, both branches diverged and there's no
        // continuation point.
        if let blk = mergeBlock {
            try switchToAndSeal(self, blk);
        }
    } else { // If without `else`.
        // The false branch goes directly to @end, which also serves as the
        // merge point after @then completes.
        let endBlock = try createBlock(self, "merge");

        try emitCondBranch(self, i.condition, thenBlock, endBlock);

        // @then has one predecessor (@entry), seal it immediately.
        // @end is not sealed yet because @then might also jump to it.
        try sealBlock(self, thenBlock);

        // Lower the @then block, then jump to @end and seal it.
        // Unlike the if-else case, @end is always created because there is no
        // else branch that can diverge.
        switchToBlock(self, thenBlock);
        try lowerBlock(self, i.thenBranch);
        try emitJmpAndSeal(self, endBlock);

        // Continue execution at @end.
        try switchToAndSeal(self, endBlock);
    }
}

/// Lower an assignment target that designates a memory location, and return
/// the address to store through. Returns `nil` without emitting anything for
/// targets that aren't memory-backed, such as locals tracked in SSA.
unsafe fn lowerPlace 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, target: *ast::Node) -> ?FieldRef throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    match target.value {
        case ast::NodeValue::ScopeAccess(_) => {
            let sym = try symOf(self, target);
            let case resolver::SymbolData::Value { type, .. } = sym.data else {
                throw LowerError::ImmutableAssignment;
            };
            return FieldRef { base: emitDataAddr(self, sym), offset: 0, fieldType: type };
        }
        case ast::NodeValue::FieldAccess(access) => {
            return try lowerFieldRef(self, access);
        }
        case ast::NodeValue::Deref(pointer) => {
            // Dereference: `*ptr` or `*r` on a single-field unlabeled record.
            // Both address offset 0 using the resolver-assigned type.
            let base = emitValToReg(self, try lowerExpr(self, pointer));
            return FieldRef { base, offset: 0, fieldType: try typeOf(self, target) };
        }
        case ast::NodeValue::Subscript { container, index } => {
            // Array or slice element: `arr[i]`.
            let elem = try lowerElemPtr(self, container, index);
            return FieldRef { base: elem.elemReg, offset: 0, fieldType: elem.elemType };
        }
        else => return nil,
    }
}

/// Lower a compound assignment whose target is memory-backed, and report
/// whether it was handled. Nothing is emitted when it isn't.
///
/// Compound assignments share their target node with the left operand of the
/// desugared binary expression. Resolve that place once so side effects in a
/// dereference, field parent, or subscript index are not repeated for the store.
unsafe fn lowerCompoundAssign 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function, expr: *ast::Node, binop: ast::BinOp
) -> bool throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let place = try lowerPlace(self, binop.left) else return false;
    let current = emitRead(self, place.base, place.offset, place.fieldType);
    let left = try applyCoercion(self, binop.left, current);
    let right = try lowerExpr(self, binop.right);
    let exprType = try typeOf(self, expr);
    let result = emitScalarBinOp(
        self, binop.op, ilType(self.low, exprType), left, right, isUnsignedType(exprType)
    );
    let assigned = try applyCoercion(self, expr, result);
    try emitStore(self, place.base, place.offset, place.fieldType, assigned);

    return true;
}

/// Lower an assignment statement.
unsafe fn lowerAssign 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node, a: ast::Assign) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    // Slice assignment: `slice[range] = value`.
    if let info = resolver::sliceRangeInfoFor(self.low.resolver, node) {
        let case ast::NodeValue::Subscript { container, index } = a.left.value
            else panic "lowerAssign: slice assign without subscript";
        let case ast::NodeValue::Range(range) = index.value
            else panic "lowerAssign: slice assign without range";
        try lowerSliceAssign(self, a.right, container, range, info);

        return;
    }
    // Compound assignments are represented as `target = target op rhs`, with
    // the exact same target node used in both places. Memory-backed targets
    // must be resolved once so calls in the target are evaluated once.
    if let case ast::NodeValue::BinOp(binop) = a.right.value {
        if binop.left == a.left and try lowerCompoundAssign(self, a.right, binop) {
            return;
        }
    }
    // Evaluate assignment value.
    let rhs = try lowerExpr(self, a.right);

    match a.left.value {
        case ast::NodeValue::Ident(_) => {
            // First try local variable lookup.
            if let v = lookupLocalVar(self, a.left) {
                if not getVar(self, v).mutable {
                    throw LowerError::ImmutableAssignment;
                }
                let leftTy = try typeOf(self, a.left);
                if isAggregateType(leftTy) or getVar(self, v).addressTaken {
                    // Aggregates and address-taken scalars are represented as
                    // pointers to stack memory. Store through the pointer.
                    let val = try useVar(self, v);
                    let dst = emitValToReg(self, val);

                    try emitStore(self, dst, 0, leftTy, rhs);
                } else {
                    // Scalars are tracked directly in SSA. Each assignment
                    // records a new SSA value.
                    defVar(self, v, rhs);
                }
            } else {
                // Fall back to static variable assignment.
                try lowerStaticAssign(self, a.left, rhs);
            }
        }
        else => {
            let place = try lowerPlace(self, a.left) else {
                throw LowerError::UnexpectedNodeValue(a.left);
            };
            try emitStore(self, place.base, place.offset, place.fieldType, rhs);
        }
    }
}

/// Lower `slice[range] = value`.
unsafe fn lowerSliceAssign 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    rhs: *ast::Node,
    container: *ast::Node,
    range: ast::Range,
    info: resolver::SliceRangeInfo
) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let r = try resolveSliceRangePtr(self, container, range, info);
    let elemSize = resolver::getTypeLayout(*info.itemType).size;
    let rhsTy = try typeOf(self, rhs);

    if let case resolver::Type::Slice { .. } = rhsTy {
        // Copy from source slice.
        let srcReg = emitValToReg(self, try lowerExpr(self, rhs));
        let srcData = loadSlicePtr(self, srcReg);
        let srcLen = loadSliceLen(self, srcReg);

        // Trap if source and destination lengths differ.
        try emitTrapUnlessCmp(self, il::CmpOp::Eq, il::Type::W32, r.count, srcLen);

        let bytes = emitTypedBinOp(
            self, il::BinOp::Mul, il::Type::W32, r.count, il::Val::Imm(elemSize as i64)
        );
        try emitByteCopyLoop(self, r.dataReg, srcData, bytes, "copy");
    } else {
        // Fill with scalar value.
        let fillVal = try lowerExpr(self, rhs);
        try emitFillLoop(self, r.dataReg, fillVal, r.count, *info.itemType, elemSize);
    }
}

/// Emit a typed fill loop: `for i in 0..count { dst[i * stride] = value; }`.
unsafe fn emitFillLoop 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    dst: il::Reg,
    value: il::Val,
    count: il::Val,
    elemType: resolver::Type,
    elemSize: u32
) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let iReg = nextReg(self);
    let header = try createBlockWithParam(
        self, "fill", il::Param { value: iReg, type: il::Type::W32 }
    );
    let body = try createBlock(self, "fill");
    let done = try createBlock(self, "fill");

    try emitJmpWithArg(self, header, il::Val::Imm(0));
    switchToBlock(self, header);
    try emitBrCmp(self, il::CmpOp::Ult, il::Type::W32, il::Val::Reg(iReg), count, body, done);

    try switchToAndSeal(self, body);
    let dstElem = emitElem(self, elemSize, dst, il::Val::Reg(iReg));
    try emitStore(self, dstElem, 0, elemType, value);

    let nextI = emitTypedBinOp(
        self, il::BinOp::Add, il::Type::W32, il::Val::Reg(iReg), il::Val::Imm(1)
    );
    try emitJmpWithArg(self, header, nextI);
    try sealBlock(self, header);
    try switchToAndSeal(self, done);
}

///////////////////
// Loop Lowering //
///////////////////

// All loop forms are lowered to a common structure:
//
// - Loop header block: evaluates condition if any, branches to body or exit.
// - Body block: executes loop body, may contain break/continue.
// - Step block (for `for` loops): increments counter, jumps back to header.
// - Exit block: target for break statements and normal loop exit.
//
// The loop stack tracks break/continue targets for nested loops.

/// Lower an infinite loop: `loop { <body> }`.
///
///   @entry -> @loop -> @loop
///               |
///               `----> @end
///
unsafe fn lowerLoop 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, body: *ast::Node) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let loopBlock = try createBlock(self, "loop");
    let endBlock = try createBlock(self, "merge");

    // Enter the loop with the given break and continue targets.
    // `break` jumps to `endBlock`,
    // `continue` jumps to `loopBlock`.
    enterLoop(self, endBlock, loopBlock);
    // Switch to and jump to the loop block, then lower all loop body statements into it.
    try switchAndJumpTo(self, loopBlock);
    try lowerBlock(self, body);

    // If the loop body doesn't diverge, jump back to the start.
    // This creates the infinite loop.
    // All predecessors are known, we can seal the loop block.
    try emitJmpAndSeal(self, loopBlock);
    // Exit the loop.
    exitLoop(self);

    // Only seal end block if it's actually reachable (ie. has predecessors).
    // If the loop has no breaks and only exits via return, the end block
    // remains unreachable and isn't added to the CFG.
    if getBlock(self, endBlock).preds.len > 0 {
        try switchToAndSeal(self, endBlock);
    }
}

/// Lower a while loop: `while <cond> { <body> }`.
///
///   @entry -> @loop -> (true)  @body -> @loop
///               |
///               `----> (false) @end
///
unsafe fn lowerWhile 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, w: ast::While) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let whileBlock = try createBlock(self, "while");
    let bodyBlock = try createBlock(self, "body");
    let endBlock = try createBlock(self, "merge");

    enterLoop(self, endBlock, whileBlock);

    // Loop condition.
    try switchAndJumpTo(self, whileBlock);

    // Based on the condition, either jump to the body,
    // or to the end of the loop.
    try emitCondBranch(self, w.condition, bodyBlock, endBlock);

    // Lower loop body and jump back to loop condition check.
    try switchToAndSeal(self, bodyBlock);
    try lowerBlock(self, w.body);
    try emitJmpAndSeal(self, whileBlock);

    try switchToAndSeal(self, endBlock);
    exitLoop(self);
}

/// Emit an increment of a variable by `1`.
unsafe fn emitIncrement 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, v: Var, typ: il::Type) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let cur = try useVar(self, v);
    let next = nextReg(self);

    emit(self, il::Instr::BinOp { op: il::BinOp::Add, typ, dst: next, a: cur, b: il::Val::Imm(1) });
    defVar(self, v, il::Val::Reg(next));
}

/// Common for-loop lowering for both range and collection iterators.
///
/// The step block is created lazily (after the loop body) so that it gets
/// a block index higher than all body blocks. This ensures the register
/// allocator processes definitions before uses in forward block order,
/// avoiding stale assignments when a value defined deep in the body flows
/// through the step block as a block argument.
unsafe fn lowerForLoop 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, iter: &ForIter, body: *ast::Node) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let loopBlock = try createBlock(self, "loop");
    let bodyBlock = try createBlock(self, "body");
    let endBlock = try createBlock(self, "merge");

    enterLoop(self, endBlock, nil);
    try switchAndJumpTo(self, loopBlock);

    // Emit condition check.
    match *iter {
        case ForIter::Range { valVar, endVal, valType, unsigned, .. } => {
            let curVal = try useVar(self, valVar);
            let cmp = il::CmpOp::Ult if unsigned else il::CmpOp::Slt;
            try emitBrCmp(self, cmp, valType, curVal, endVal, bodyBlock, endBlock);
        }
        case ForIter::Collection { idxVar, lengthVal, .. } => {
            let curIdx = try useVar(self, idxVar);
            try emitBrCmp(self, il::CmpOp::Slt, il::Type::W32, curIdx, lengthVal, bodyBlock, endBlock);
        }
    }
    // Switch to loop body.
    try switchToAndSeal(self, bodyBlock);

    // Emit element binding, only for collections.
    // Reads the element at the current index.
    if let case ForIter::Collection { valVar, idxVar, dataReg, elemType, .. } = *iter {
        if let v = valVar {
            let curIdx = try useVar(self, idxVar);
            let elemReg = emitElem(self, resolver::getTypeLayout(*elemType).size, dataReg, curIdx);
            let val = emitRead(self, elemReg, 0, *elemType);

            defVar(self, v, val);
        }
    }
    // Lower the loop body.
    try lowerBlock(self, body);

    // Check if a `continue` statement created a step block.
    let ctx = currentLoop(self) else panic;
    if let stepBlock = ctx.continueTarget {
        // Body has `continue` statements: jump to step block, emit increment there.
        try emitJmpAndSeal(self, stepBlock);
        switchToBlock(self, stepBlock);
    }
    // Otherwise, emit increment directly in the current block,
    // saving the jump to a separate step block.
    if not blockHasTerminator(self) {
        match *iter {
            case ForIter::Range { valVar, valType, indexVar, .. } => {
                try emitIncrement(self, valVar, valType);
                if let idxVar = indexVar {
                    try emitIncrement(self, idxVar, il::Type::W32);
                }
            }
            case ForIter::Collection { idxVar, .. } => {
                try emitIncrement(self, idxVar, il::Type::W32);
            }
        }
        try emitJmp(self, loopBlock);
    }
    exitLoop(self);

    try sealBlock(self, loopBlock);
    try sealBlock(self, endBlock);

    switchToBlock(self, endBlock);
}

/// Lower a `for` loop over a range, array, or slice.
unsafe fn lowerFor 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node, f: ast::For) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let savedVarsLen = enterVarScope(self);
    let info = resolver::forLoopInfoFor(self.low.resolver, node) else {
        throw LowerError::MissingMetadata;
    };
    match info {
        case resolver::ForLoopInfo::Range { valType, range, bindingName, indexName } => {
            let endExpr = range.end else {
                throw LowerError::MissingMetadata;
            };
            let mut startVal = il::Val::Imm(0);
            if let start = range.start {
                set startVal = try lowerExpr(self, start);
            }
            let endVal = try lowerExpr(self, endExpr);
            let iterType = ilType(self.low, *valType);
            let valVar = newVar(self, bindingName, iterType, false, startVal);

            let mut indexVar: ?Var = nil;
            if indexName <> nil { // Optional index always starts at zero.
                set indexVar = newVar(self, indexName, il::Type::W32, false, il::Val::Imm(0));
            }
            let iter = ForIter::Range {
                valVar, indexVar, endVal, valType: iterType,
                unsigned: isUnsignedType(*valType),
            };

            try lowerForLoop(self, &iter, f.body);
        }
        case resolver::ForLoopInfo::Collection { elemType, length, bindingName, indexName } => {
            let containerVal = try lowerExpr(self, f.iterable);
            let containerReg = emitValToReg(self, containerVal);

            let mut dataReg = containerReg;
            let mut lengthVal: il::Val = undefined;
            if let len = length { // Array (length is known).
                set lengthVal = il::Val::Imm(len as i64);
            } else { // Slice (length must be loaded).
                set lengthVal = loadSliceLen(self, containerReg);
                set dataReg = loadSlicePtr(self, containerReg);
            }
            // Declare index value binidng.
            let idxVar = newVar(self, indexName, il::Type::W32, false, il::Val::Imm(0));

            // Declare element value binding.
            let mut valVar: ?Var = nil;
            if bindingName <> nil {
                set valVar = newVar(
                    self,
                    bindingName,
                    ilType(self.low, *elemType),
                    false,
                    il::Val::Undef
                );
            }
            let iter = ForIter::Collection { valVar, idxVar, dataReg, lengthVal, elemType };

            try lowerForLoop(self, &iter, f.body);
        }
    }
    exitVarScope(self, savedVarsLen);
}

/// Lower a break statement.
unsafe fn lowerBreak 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let ctx = currentLoop(self) else {
        throw LowerError::OutsideOfLoop;
    };
    try emitJmp(self, ctx.breakTarget);
}

/// Lower a continue statement.
unsafe fn lowerContinue 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let block = try getOrCreateContinueBlock(self);
    try emitJmp(self, block);
}

/// Emit a return, blitting into the caller's return buffer if needed.
///
/// When the function has a return buffer parameter, the value is blitted
/// into the buffer and the buffer pointer is returned. Otherwise, the value is
/// returned directly.
unsafe fn emitRetVal 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, val: il::Val) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    if let retReg = self.returnReg {
        let src = emitValToReg(self, val);
        let size = resolver::getResultLayout(*self.fnType.returnType, self.fnType.throwList).size
            if self.fnType.throwList.len > 0
            else resolver::getTypeLayout(*self.fnType.returnType).size;

        emit(self, il::Instr::Blit { dst: retReg, src, size: il::Val::Imm(size as i64) });
        emit(self, il::Instr::Ret { val: il::Val::Reg(retReg) });
    } else if isSmallAggregate(*self.fnType.returnType) {
        let mut src = emitValToReg(self, val);
        let layout = resolver::getTypeLayout(*self.fnType.returnType);
        if layout.alignment < resolver::PTR_SIZE or layout.size < resolver::PTR_SIZE {
            // The return word must not read beyond the value or its alignment.
            let word = emitReserveLayout(self, resolver::Layout {
                size: resolver::PTR_SIZE, alignment: resolver::PTR_SIZE,
            });
            emitStoreW64At(self, il::Val::Imm(0), word, 0);
            emit(self, il::Instr::Blit { dst: word, src, size: il::Val::Imm(layout.size as i64) });
            set src = word;
        }
        let dst = nextReg(self);

        emit(self, il::Instr::Load { typ: il::Type::W64, dst, src, offset: 0 });
        emit(self, il::Instr::Ret { val: il::Val::Reg(dst) });
    } else {
        emit(self, il::Instr::Ret { val });
    }
}

/// Lower a return statement.
unsafe fn lowerReturnStmt 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node, value: ?*ast::Node) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let mut val = il::Val::Undef;
    if let expr = value {
        set val = try lowerExpr(self, expr);
    }
    if blockHasTerminator(self) {
        return;
    }
    set val = try applyCoercion(self, node, val);
    try emitRetVal(self, val);
}

/// Lower a throw statement.
unsafe fn lowerThrowStmt 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, expr: *ast::Node) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    assert self.fnType.throwList.len > 0;

    let errType = *self.fnType.throwList[0] if self.fnType.throwList.len == 1
        else try typeOf(self, expr);
    let tag = getOrAssignErrorTag(self.low, errType) as i64;
    let errVal = try lowerExpr(self, expr);
    let resultVal = try buildResult(self, tag, errVal, errType);

    try emitRetVal(self, resultVal);
}

/// Ensure a value is in a register (eg. for branch conditions).
unsafe fn emitValToReg 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, val: il::Val) -> il::Reg where 'arena: 'phase, 'phase: 'function {
    match val {
        case il::Val::Reg(r) => return r,
        case il::Val::Imm(_), il::Val::DataSym(_), il::Val::FnAddr(_) => {
            let dst = nextReg(self);
            emit(self, il::Instr::Copy { dst, val });
            return dst;
        }
        case il::Val::Undef => {
            // TODO: We shouldn't hit this case, right? A register shouldn't be needed
            // if the value is undefined.
            return nextReg(self);
        }
    }
}

/// Lower a logical `and`/`or` with short-circuit evaluation.
///
/// Short-circuit evaluation skips evaluating the right operand when the left
/// operand already determines the result:
///
/// - In `a and b`, if `a` is false, result is false without evaluating `b`.
/// - In `a or b`, if `a` is true, result is true without evaluating `b`.
///
/// This matters when `b` has side effects, or is expensive to evaluate.
///
/// Example: `a and b`
///
///     @entry
///       br %a @then @else;
///     @then
///       // Evaluate b into %b
///       // ...
///       jmp @end(%b);
///     @else
///       jmp @end(0);                  // Skip evaluating b, result is false
///     @end(w8 %result)
///       ret %result;
///
/// Example: `a or b`
///
///     @entry
///       br %a @then @else;
///     @then
///       jmp @end(1);                  // Skip evaluating b, result is true
///     @else
///       // Evaluate b into %b
///       // ...
///       jmp @end(%b);
///     @end(w8 %result)
///       ret %result;
///
unsafe fn lowerLogicalOp 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    binop: ast::BinOp,
    thenLabel: *[u8],
    elseLabel: *[u8],
    mergeLabel: *[u8],
    op: LogicalOp
) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let thenBlock = try createBlock(self, thenLabel);
    let elseBlock = try createBlock(self, elseLabel);

    let resultReg = nextReg(self);
    let mergeBlock = try createBlockWithParam(
        self, mergeLabel, il::Param { value: resultReg, type: il::Type::W8 }
    );
    // Evaluate left operand and branch.
    try emitCondBranch(self, binop.left, thenBlock, elseBlock);

    // Block that skips evaluating `b`.
    let mut shortCircuitBlock: BlockId = undefined;
    // Block that evaluates `b`.
    let mut evalBlock: BlockId = undefined;
    // Result when short-circuiting (`0` or `1`).
    let mut shortCircuitVal: i64 = undefined;

    match op {
        case LogicalOp::And => {
            set shortCircuitBlock = elseBlock;
            set evalBlock = thenBlock;
            set shortCircuitVal = 0;
        }
        case LogicalOp::Or => {
            set shortCircuitBlock = thenBlock;
            set evalBlock = elseBlock;
            set shortCircuitVal = 1;
        }
    }
    // Emit short-circuit branch: jump to merge with constant result.
    try switchToAndSeal(self, shortCircuitBlock);
    try emitJmpWithArg(self, mergeBlock, il::Val::Imm(shortCircuitVal));

    // Emit evaluation branch: evaluate right operand and jump to merge.
    try switchToAndSeal(self, evalBlock);
    try emitJmpWithArg(self, mergeBlock, try lowerExpr(self, binop.right));

    try switchToAndSeal(self, mergeBlock);
    return il::Val::Reg(resultReg);
}

/// Lower a conditional expression (`thenExpr if condition else elseExpr`).
unsafe fn lowerCondExpr 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node, cond: ast::CondExpr) -> il::Val
    throws (LowerError) where 'arena: 'phase, 'phase: 'function
{
    let typ = try typeOf(self, node);
    let thenBlock = try createBlock(self, "cond#then");
    let elseBlock = try createBlock(self, "cond#else");

    if isAggregateType(typ) {
        let dst = try emitReserve(self, typ);
        let layout = resolver::getTypeLayout(typ);
        try emitCondBranch(self, cond.condition, thenBlock, elseBlock);

        let mergeBlock = try createBlock(self, "cond#merge");
        try switchToAndSeal(self, thenBlock);

        let thenVal = emitValToReg(self, try lowerExpr(self, cond.thenExpr));
        emit(self, il::Instr::Blit { dst, src: thenVal, size: il::Val::Imm(layout.size as i64) });

        try emitJmp(self, mergeBlock);
        try switchToAndSeal(self, elseBlock);

        let elseVal = emitValToReg(self, try lowerExpr(self, cond.elseExpr));
        emit(self, il::Instr::Blit { dst, src: elseVal, size: il::Val::Imm(layout.size as i64) });

        try emitJmp(self, mergeBlock);
        try switchToAndSeal(self, mergeBlock);

        return il::Val::Reg(dst);
    } else {
        try emitCondBranch(self, cond.condition, thenBlock, elseBlock);

        let resultType = ilType(self.low, typ);
        let resultReg = nextReg(self);
        let mergeBlock = try createBlockWithParam(
            self, "cond#merge", il::Param { value: resultReg, type: resultType }
        );
        try switchToAndSeal(self, thenBlock);
        try emitJmpWithArg(self, mergeBlock, try lowerExpr(self, cond.thenExpr));
        try switchToAndSeal(self, elseBlock);
        try emitJmpWithArg(self, mergeBlock, try lowerExpr(self, cond.elseExpr));
        try switchToAndSeal(self, mergeBlock);

        return il::Val::Reg(resultReg);
    }
}

/// Select the concrete operand type for a scalar comparison.
fn scalarComparisonType(left: resolver::Type, right: resolver::Type) -> resolver::Type {
    return right if left == resolver::Type::Int else left;
}

/// Convert a binary operator to a comparison op, if applicable.
/// For `Gt`, caller must swap operands: `a > b = b < a`.
/// For `Gte`/`Lte`, caller must swap branch labels: `a >= b = !(a < b)`.
/// For `Lte`, caller must also swap operands: `a <= b = !(b < a)`.
fn cmpOpFrom(op: ast::BinaryOp, unsigned: bool) -> ?il::CmpOp {
    match op {
        case ast::BinaryOp::Eq => return il::CmpOp::Eq,
        case ast::BinaryOp::Ne => return il::CmpOp::Ne,
        case ast::BinaryOp::Lt, ast::BinaryOp::Gt,
             ast::BinaryOp::Gte, ast::BinaryOp::Lte =>
            return il::CmpOp::Ult if unsigned else il::CmpOp::Slt,
        else => return nil,
    }
}

/// Lower a binary operation.
unsafe fn lowerBinOp 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node, binop: ast::BinOp) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    // Short-circuit logical operators don't evaluate both operands eagerly.
    if binop.op == ast::BinaryOp::And {
        return try lowerLogicalOp(self, binop, "and#then", "and#else", "and#end", LogicalOp::And);
    } else if binop.op == ast::BinaryOp::Or {
        return try lowerLogicalOp(self, binop, "or#then", "or#else", "or#end", LogicalOp::Or);
    }

    // Handle comparison with a `nil` literal: just check the tag/pointer instead of
    // building a `nil` aggregate and doing full comparison.
    if binop.op == ast::BinaryOp::Eq or binop.op == ast::BinaryOp::Ne {
        let isEq = binop.op == ast::BinaryOp::Eq;
        let leftIsNil = binop.left.value == ast::NodeValue::Nil;
        let rightIsNil = binop.right.value == ast::NodeValue::Nil;

        if leftIsNil {
            return try lowerNilCheck(self, binop.right, isEq);
        } else if rightIsNil {
            return try lowerNilCheck(self, binop.left, isEq);
        }
    }
    // Lower operands.
    let a = try lowerExpr(self, binop.left);
    let b = try lowerExpr(self, binop.right);
    let isComparison = cmpOpFrom(binop.op, false) <> nil;

    // The result type for comparisons is always `bool`, while for arithmetic
    // operations, it's the operand type. We set this appropriately here.
    let nodeTy = try typeOf(self, node);
    let mut resultTy = nodeTy;

    if isComparison {
        let leftTy = try effectiveType(self, binop.left);
        let rightTy = try effectiveType(self, binop.right);
        // Optimize: comparing with a void variant just needs tag comparison.
        if let idx = voidVariantIndex(self.low.resolver, binop.left) {
            return try emitTagCmp(self, binop.op, b, idx, rightTy);
        } else if let idx = voidVariantIndex(self.low.resolver, binop.right) {
            return try emitTagCmp(self, binop.op, a, idx, leftTy);
        }
        // Aggregate types require element-wise comparison.
        // When comparing `?T` with `T`, wrap the scalar side.
        if isAggregateType(leftTy) {
            let mut rhs = b;
            if not isAggregateType(rightTy) {
                set rhs = try wrapInOptional(self, rhs, leftTy);
            }
            return try emitAggregateEqOp(self, binop.op, leftTy, a, rhs);
        }
        if isAggregateType(rightTy) {
            let lhs = try wrapInOptional(self, a, rightTy);
            return try emitAggregateEqOp(self, binop.op, rightTy, lhs, b);
        }
        set resultTy = scalarComparisonType(leftTy, rightTy);
    }
    return emitScalarBinOp(self, binop.op, ilType(self.low, resultTy), a, b, isUnsignedType(resultTy));
}

/// Emit an aggregate equality or inequality comparison.
unsafe fn emitAggregateEqOp 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    op: ast::BinaryOp,
    typ: resolver::Type,
    a: il::Val,
    b: il::Val
) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let regA = emitValToReg(self, a);
    let regB = emitValToReg(self, b);
    let result = try lowerAggregateEq(self, typ, regA, regB, 0);

    if op == ast::BinaryOp::Ne {
        return emitTypedBinOp(self, il::BinOp::Eq, il::Type::W32, result, il::Val::Imm(0));
    }
    return result;
}

/// Emit a scalar binary operation instruction.
unsafe fn emitScalarBinOp 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    op: ast::BinaryOp,
    typ: il::Type,
    a: il::Val,
    b: il::Val,
    unsigned: bool
) -> il::Val where 'arena: 'phase, 'phase: 'function {
    let dst = nextReg(self);
    let mut needsExt: bool = false;
    match op {
        case ast::BinaryOp::Add => {
            emit(self, il::Instr::BinOp { op: il::BinOp::Add, typ, dst, a, b });
            set needsExt = true;
        }
        case ast::BinaryOp::Sub => {
            emit(self, il::Instr::BinOp { op: il::BinOp::Sub, typ, dst, a, b });
            set needsExt = true;
        }
        case ast::BinaryOp::Mul => {
            emit(self, il::Instr::BinOp { op: il::BinOp::Mul, typ, dst, a, b });
            set needsExt = true;
        }
        case ast::BinaryOp::Div => {
            let op = il::BinOp::Udiv if unsigned else il::BinOp::Sdiv;
            emit(self, il::Instr::BinOp { op, typ, dst, a, b });
            set needsExt = true;
        }
        case ast::BinaryOp::Mod => {
            let op = il::BinOp::Urem if unsigned else il::BinOp::Srem;
            emit(self, il::Instr::BinOp { op, typ, dst, a, b });
            set needsExt = true;
        }
        case ast::BinaryOp::BitAnd => emit(self, il::Instr::BinOp { op: il::BinOp::And, typ, dst, a, b }),
        case ast::BinaryOp::BitOr => emit(self, il::Instr::BinOp { op: il::BinOp::Or, typ, dst, a, b }),
        case ast::BinaryOp::BitXor => emit(self, il::Instr::BinOp { op: il::BinOp::Xor, typ, dst, a, b }),
        case ast::BinaryOp::Shl => {
            emit(self, il::Instr::BinOp { op: il::BinOp::Shl, typ, dst, a, b });
            set needsExt = true;
        }
        case ast::BinaryOp::Shr => {
            let op = il::BinOp::Ushr if unsigned else il::BinOp::Sshr;
            emit(self, il::Instr::BinOp { op, typ, dst, a, b });
        }
        case ast::BinaryOp::Eq => emit(self, il::Instr::BinOp { op: il::BinOp::Eq, typ, dst, a, b }),
        case ast::BinaryOp::Ne => emit(self, il::Instr::BinOp { op: il::BinOp::Ne, typ, dst, a, b }),
        case ast::BinaryOp::Lt => {
            let op = il::BinOp::Ult if unsigned else il::BinOp::Slt;
            emit(self, il::Instr::BinOp { op, typ, dst, a, b });
        }
        case ast::BinaryOp::Gt => { // `a > b` = `b < a`
            let op = il::BinOp::Ult if unsigned else il::BinOp::Slt;
            emit(self, il::Instr::BinOp { op, typ, dst, a: b, b: a });
        }
        case ast::BinaryOp::Lte => { // `a <= b` = `b >= a`
            let op = il::BinOp::Uge if unsigned else il::BinOp::Sge;
            emit(self, il::Instr::BinOp { op, typ, dst, a: b, b: a });
        }
        case ast::BinaryOp::Gte => {
            let op = il::BinOp::Uge if unsigned else il::BinOp::Sge;
            emit(self, il::Instr::BinOp { op, typ, dst, a, b });
        }
        // Logical xor on booleans is equivalent to not equal.
        case ast::BinaryOp::Xor => emit(self, il::Instr::BinOp { op: il::BinOp::Ne, typ, dst, a, b }),
        // Short-circuit ops are handled elsewhere.
        case ast::BinaryOp::And, ast::BinaryOp::Or => panic,
    }
    // Normalize sub-word arithmetic results so high bits are well-defined.
    // The lowering knows signedness, so it can pick the right extension.
    // [`il::Type::W32`] is handled in the backend via 32-bit instructions.
    if needsExt {
        return normalizeSubword(self, typ, unsigned, il::Val::Reg(dst));
    }
    return il::Val::Reg(dst);
}

/// Normalize sub-word values to well-defined high bits.
unsafe fn normalizeSubword 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, typ: il::Type, unsigned: bool, val: il::Val) -> il::Val where 'arena: 'phase, 'phase: 'function {
    if typ == il::Type::W8 or typ == il::Type::W16 {
        let extDst: il::Reg = nextReg(self);
        if unsigned {
            emit(self, il::Instr::Zext { typ, dst: extDst, val });
        } else {
            emit(self, il::Instr::Sext { typ, dst: extDst, val });
        }
        return il::Val::Reg(extDst);
    }
    return val;
}

/// Lower a unary operation.
unsafe fn lowerUnOp 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node, unop: ast::UnOp) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    if unop.op == ast::UnaryOp::Neg {
        if let case ast::NodeValue::Number(lit) = unop.value.value {
            return il::Val::Imm((0 - lit.magnitude) as i64);
        }
    }
    let val = try lowerExpr(self, unop.value);
    let t = try typeOf(self, node);
    let typ = ilType(self.low, t);
    let dst = nextReg(self);
    let mut needsExt: bool = false;

    match unop.op {
        case ast::UnaryOp::Not => {
            emit(self, il::Instr::BinOp { op: il::BinOp::Eq, typ, dst, a: val, b: il::Val::Imm(0) });
        }
        case ast::UnaryOp::Neg => {
            emit(self, il::Instr::UnOp { op: il::UnOp::Neg, typ, dst, a: val });
            set needsExt = true;
        }
        case ast::UnaryOp::BitNot => {
            emit(self, il::Instr::UnOp { op: il::UnOp::Not, typ, dst, a: val });
            set needsExt = true;
        }
    }
    if needsExt {
        return normalizeSubword(self, typ, isUnsignedType(t), il::Val::Reg(dst));
    }
    return il::Val::Reg(dst);
}

/// Lower a cast expression (`x as T`).
unsafe fn lowerCast 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node, cast: ast::As) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let val = try lowerExpr(self, cast.value);

    let srcType = try typeOf(self, cast.value);
    let dstType = try typeOf(self, node);
    if resolver::typesEqual(srcType, dstType) {
        return val;
    }
    return lowerNumericCast(self, val, srcType, dstType);
}

/// Check whether a resolver type is a signed integer type.
fn isSignedType(t: resolver::Type) -> bool {
    match t {
        case resolver::Type::I8, resolver::Type::I16, resolver::Type::I32, resolver::Type::I64,
             resolver::Type::Int => return true,
        else => return false,
    }
}

/// Check whether a resolver type is an unsigned integer type.
fn isUnsignedType(t: resolver::Type) -> bool {
    match t {
        case resolver::Type::U8, resolver::Type::U16, resolver::Type::U32, resolver::Type::U64 => return true,
        else => return false,
    }
}

/// Lower a string literal to a slice value.
///
/// String literals are stored as global data and the result is a slice
/// pointing to the data with the appropriate length.
unsafe fn lowerStringLit 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node, s: *[u8]) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    // Get the slice type from the node.
    let sliceTy = try typeOf(self, node);
    let case resolver::Type::Slice { item, mutable, .. } = sliceTy
        else throw LowerError::ExpectedSliceOrArray;
    // Build the string data value.
    let ptr = try! alloc::alloc(
        self.low.arena, @sizeOf(il::DataValue), @alignOf(il::DataValue)
    ) as *mut il::DataValue;

    set *ptr = il::DataValue { item: il::DataItem::Str(s), count: 1 };
    let result = ConstDataResult { values: @sliceOf(ptr, 1), zeroInit: false };

    return try lowerConstDataAsSlice(
        self, &result, 1, true, item, mutable, s.len
    );
}

/// Lower a builtin call expression.
unsafe fn lowerBuiltinCall 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node, kind: ast::Builtin, args: *[*ast::Node]) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    match kind {
        case ast::Builtin::SliceOf => return try lowerSliceOf(self, node, args),
        case ast::Builtin::SizeOf, ast::Builtin::AlignOf => {
            let constVal = resolver::constValueEntry(self.low.resolver, node) else {
                throw LowerError::MissingConst(node);
            };
            return try constValueToVal(self, constVal, node);
        }
    }
}

/// Lower a `@sliceOf(ptr, len)` or `@sliceOf(ptr, len, cap)` builtin call.
unsafe fn lowerSliceOf 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node, args: *[*ast::Node]) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    if args.len <> 2 and args.len <> 3 {
        throw LowerError::InvalidArgCount;
    }
    let sliceTy = try typeOf(self, node);
    let case resolver::Type::Slice { item, mutable, .. } = sliceTy
        else throw LowerError::ExpectedSliceOrArray;
    let ptrVal = try lowerExpr(self, args[0]);
    let lenVal = try lowerExpr(self, args[1]);
    let mut capVal = lenVal;
    if args.len == 3 {
        set capVal = try lowerExpr(self, args[2]);
    }
    if not isLtEq(lenVal, capVal) {
        try emitTrapIfLt(self, il::Type::W32, capVal, lenVal);
    }
    return try buildSliceValue(self, item, mutable, ptrVal, lenVal, capVal);
}

/// Lower a `try` expression.
unsafe fn lowerTry 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node, t: ast::Try) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let case ast::NodeValue::Call(callExpr) = t.expr.value else {
        throw LowerError::ExpectedCall;
    };
    let calleeTy = try typeOf(self, callExpr.callee);
    let case resolver::Type::Fn(calleeInfo) = calleeTy else {
        throw LowerError::ExpectedFunction;
    };
    let okValueTy = *calleeInfo.returnType; // The type of the success payload.

    // Type of the try expression, which is either the return type of the function
    // if successful, or an optional of it, if using `try?`.
    let tryExprTy = try typeOf(self, node);
    // Check for trait method dispatch or standalone method call.
    let mut resVal: il::Val = undefined;
    let mut initialization: ?SessionInitialization = nil;
    let callNodeExtra = resolver::nodeData(self.low.resolver, t.expr).extra;
    if let case resolver::NodeExtra::SessionAllocation(allocation) = callNodeExtra {
        let prepared = try prepareSessionAllocation(self, callExpr, allocation);
        set initialization = prepared;
        set resVal = prepared.result;
    } else if let case resolver::NodeExtra::TraitMethodCall {
        traitInfo, methodIndex
    } = callNodeExtra {
        set resVal = try lowerTraitMethodCall(self, t.expr, callExpr, traitInfo, methodIndex);
    } else if let case resolver::NodeExtra::MethodCall { method } = callNodeExtra {
        set resVal = try lowerMethodCall(self, t.expr, callExpr, method);
    } else {
        set resVal = try lowerCall(self, t.expr, callExpr);
    }
    if blockHasTerminator(self) {
        return il::Val::Undef;
    }
    let base = emitValToReg(self, resVal); // The result value.
    let tagReg = resultTagReg(self, base); // The result tag.

    let okBlock = try createBlock(self, "ok"); // Block if success.
    let errBlock = try createBlock(self, "err"); // Block if failure.

    let mut mergeBlock: ?BlockId = nil;
    let mut resultSlot: ?il::Reg = nil; // `try` result value will be stored here.

    // Check if the `try` returns a success value or not. If so, reserve
    // space for it.
    let isVoid = tryExprTy == resolver::Type::Void or tryExprTy == resolver::Type::Never;
    if not isVoid {
        set resultSlot = try emitReserve(self, tryExprTy);
    }
    // Branch on tag: zero means ok, non-zero means error.
    try emitBr(self, tagReg, errBlock, okBlock);

    // We can now seal the blocks since all predecessors are known.
    try sealBlock(self, okBlock);
    try sealBlock(self, errBlock);

    // Success path: extract the successful value from the result and store it
    // in the result slot for later use after the merge point.
    switchToBlock(self, okBlock);
    if let prepared = initialization {
        try initializeSessionAllocation(self, prepared);
    }

    if okValueTy == resolver::Type::Never {
        emit(self, il::Instr::Unreachable);
    } else if let slot = resultSlot {
        // Extract the success payload. If the result type differs from the payload
        // type (e.g. `try?` wrapping `T` into `?T`), wrap the value.
        let payloadVal = tvalPayloadVal(self, base, okValueTy, RESULT_VAL_OFFSET);
        let mut okVal = payloadVal;

        if t.returnsOptional and tryExprTy <> okValueTy {
            set okVal = try wrapInOptional(self, payloadVal, tryExprTy);
        }
        try emitStore(self, slot, 0, tryExprTy, okVal);
    }
    // Jump to merge block if unterminated.
    try emitMergeIfUnterminated(self, &mut mergeBlock);

    // Error path: handle the failure case based on the try expression variant.
    switchToBlock(self, errBlock);

    if t.returnsOptional {
        // `try?` converts errors to `nil` -- store the `nil` and continue.
        if let slot = resultSlot {
            let errVal = try buildNilOptional(self, tryExprTy);
            try emitStore(self, slot, 0, tryExprTy, errVal);
        }
        try emitMergeIfUnterminated(self, &mut mergeBlock);
    } else if t.catches.len > 0 {
        // `try ... catch` -- handle the error.
        let firstNode = t.catches[0];
        let case ast::NodeValue::CatchClause(first) = firstNode.value
            else panic "lowerTry: expected CatchClause";

        if first.typeNode <> nil or t.catches.len > 1 {
            // Typed multi-catch: switch on global error tag.
            try lowerMultiCatch(self, t.catches, calleeInfo, base, tagReg, &mut mergeBlock);
        } else {
            // Single untyped catch clause.
            let savedVarsLen = enterVarScope(self);
            if let binding = first.binding {
                let case ast::NodeValue::Ident(name) = binding.value else {
                    throw LowerError::ExpectedIdentifier;
                };
                let errTy = *calleeInfo.throwList[0];
                let errVal = tvalPayloadVal(self, base, errTy, RESULT_VAL_OFFSET);
                let _ = newVar(self, name, ilType(self.low, errTy), false, errVal);
            }
            try lowerBlock(self, first.body);
            try emitMergeIfUnterminated(self, &mut mergeBlock);
            exitVarScope(self, savedVarsLen);
        }
    } else if t.shouldPanic {
        // `try!` -- panic on error, emit unreachable since control won't continue.
        // TODO: We should have some kind of `panic` instruction?
        emit(self, il::Instr::Unreachable);
    } else {
        // Plain `try` -- propagate the error to the caller by returning early.
        // Forward the callee's global error tag and payload directly.
        let callerLayout = resolver::getResultLayout(
            *self.fnType.returnType, self.fnType.throwList
        );
        let calleeErrSize = maxErrSize(calleeInfo.throwList);
        let dst = emitReserveLayout(self, callerLayout);

        emitStoreW64At(self, il::Val::Reg(tagReg), dst, TVAL_TAG_OFFSET);
        let srcPayload = emitPtrOffset(self, base, RESULT_VAL_OFFSET);
        let dstPayload = emitPtrOffset(self, dst, RESULT_VAL_OFFSET);
        emit(self, il::Instr::Blit { dst: dstPayload, src: srcPayload, size: il::Val::Imm(calleeErrSize as i64) });

        try emitRetVal(self, il::Val::Reg(dst));
    }

    // Switch to the merge block if one was created. If all paths diverged
    // (e.g both success and error returned), there's no merge block.
    if let blk = mergeBlock {
        try switchToAndSeal(self, blk);
    } else {
        return il::Val::Undef;
    }
    // Return the result value. For `void` expressions, return undefined.
    // For aggregates, return the slot pointer; for scalars, load the value.
    if let slot = resultSlot {
        if isAggregateType(tryExprTy) {
            return il::Val::Reg(slot);
        }
        return emitLoad(self, slot, 0, tryExprTy);
    } else { // Void return.
        return il::Val::Undef;
    }
}

/// Lower typed multi-catch clauses.
///
/// Emits a switch on the global error tag to dispatch to the correct catch
/// clause. Each typed clause extracts the error payload for its specific type
/// and binds it to the clause's identifier.
unsafe fn lowerMultiCatch 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    catches: *[*ast::Node],
    calleeInfo: *resolver::FnType,
    base: il::Reg,
    tagReg: il::Reg,
    mergeBlock: &mut ?BlockId
) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let entry = currentBlock(self);

    // First pass: create blocks, resolve error types, and build switch cases.
    let mut blocks: [BlockId; MAX_CATCH_CLAUSES] = undefined;
    let mut errTypes: [?resolver::Type; MAX_CATCH_CLAUSES] = undefined;
    let mut cases: *unsafe mut [il::SwitchCase] = &mut [];
    let mut defaultIdx: ?u32 = nil;

    for clauseNode, i in catches {
        let case ast::NodeValue::CatchClause(clause) = clauseNode.value
            else panic "lowerMultiCatch: expected CatchClause";

        set blocks[i] = try createBlock(self, "catch");
        addPredecessor(self, blocks[i], entry);

        if let typeNode = clause.typeNode {
            let errTy = try typeOf(self, typeNode);
            set errTypes[i] = errTy;

            cases.append(il::SwitchCase {
                value: getOrAssignErrorTag(self.low, errTy) as i64,
                target: *blocks[i],
                args: &mut []
            }, alloc::arenaAllocator(self.arena));
        } else {
            set errTypes[i] = nil;
            set defaultIdx = i;
        }
    }

    // Emit switch. Default target is the catch-all block, or an unreachable block.
    let mut defaultTarget: BlockId = undefined;
    if let idx = defaultIdx {
        set defaultTarget = blocks[idx];
    } else {
        set defaultTarget = try createBlock(self, "unreachable");
        addPredecessor(self, defaultTarget, entry);
    }
    emit(self, il::Instr::Switch {
        val: il::Val::Reg(tagReg),
        defaultTarget: *defaultTarget,
        defaultArgs: &mut [],
        cases
    });

    // Second pass: emit each catch clause body.
    for clauseNode, i in catches {
        let case ast::NodeValue::CatchClause(clause) = clauseNode.value
            else panic "lowerMultiCatch: expected CatchClause";

        try switchToAndSeal(self, blocks[i]);
        let savedVarsLen = enterVarScope(self);

        if let binding = clause.binding {
            let case ast::NodeValue::Ident(name) = binding.value else {
                throw LowerError::ExpectedIdentifier;
            };
            let errTy = errTypes[i] else panic "lowerMultiCatch: catch-all with binding";
            let errVal = tvalPayloadVal(self, base, errTy, RESULT_VAL_OFFSET);

            newVar(self, name, ilType(self.low, errTy), false, errVal);
        }
        try lowerBlock(self, clause.body);
        try emitMergeIfUnterminated(self, mergeBlock);

        exitVarScope(self, savedVarsLen);
    }

    // Emit unreachable block if no catch-all.
    if defaultIdx == nil {
        try switchToAndSeal(self, defaultTarget);
        emit(self, il::Instr::Unreachable);
    }
}

/// Emit a byte-copy loop: `for i in 0..size { dst[i] = src[i]; }`.
///
/// Used when `blit` cannot be used because the copy size is dynamic.
/// Terminates the current block and leaves the builder positioned
/// after the loop.
unsafe fn emitByteCopyLoop 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    dst: il::Reg,
    src: il::Reg,
    size: il::Val,
    label: *[u8]
) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let iReg = nextReg(self);
    let header = try createBlockWithParam(
        self, label, il::Param { value: iReg, type: il::Type::W32 }
    );
    let body = try createBlock(self, label);
    let done = try createBlock(self, label);

    // Jump to header with initial counter is zero.
    try emitJmpWithArg(self, header, il::Val::Imm(0));

    // Don't seal header yet -- the body will add another predecessor.
    switchToBlock(self, header);
    try emitBrCmp(self, il::CmpOp::Ult, il::Type::W32, il::Val::Reg(iReg), size, body, done);

    // Body: load byte from source, store to destination, increment counter.
    try switchToAndSeal(self, body);

    let srcElem = emitElem(self, 1, src, il::Val::Reg(iReg));
    let byteReg = nextReg(self);
    emit(self, il::Instr::Load { typ: il::Type::W8, dst: byteReg, src: srcElem, offset: 0 });

    let dstElem = emitElem(self, 1, dst, il::Val::Reg(iReg));
    emit(self, il::Instr::Store { typ: il::Type::W8, src: il::Val::Reg(byteReg), dst: dstElem, offset: 0 });

    let nextI = emitTypedBinOp(self, il::BinOp::Add, il::Type::W32, il::Val::Reg(iReg), il::Val::Imm(1));
    // Jump back to header -- this adds body as a predecessor.
    try emitJmpWithArg(self, header, nextI);

    // Now all predecessors of header are known, seal it.
    try sealBlock(self, header);
    try switchToAndSeal(self, done);
}

/// Lower `slice.append(val, allocator)`.
///
/// Emits inline grow-if-needed logic:
///
///     load len, cap from slice header
///     if len < cap: jmp @store
///     else:         jmp @grow
///
///     @grow:
///       newCap = max(cap * 2, 1)
///       call allocator.func(allocator.ctx, newCap * stride, alignment)
///       copy old data to new pointer
///       update slice ptr and cap
///       jmp @store
///
///     @store:
///       store element at ptr + len * stride
///       increment len
///
unsafe fn lowerSliceAppend 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, call: ast::Call, elemType: *resolver::Type) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let case ast::NodeValue::FieldAccess(access) = call.callee.value
        else throw LowerError::MissingMetadata;

    // Get the address of the slice header.
    let sliceVal = try lowerExpr(self, access.parent);
    let sliceReg = emitValToReg(self, sliceVal);

    // Lower the value to append and the allocator.
    let elemVal = try lowerExpr(self, call.args[0]);
    let allocVal = try lowerExpr(self, call.args[1]);
    let allocReg = emitValToReg(self, allocVal);

    let elemLayout = resolver::getTypeLayout(*elemType);
    let stride = elemLayout.size;
    let alignment = elemLayout.alignment;

    // Load current length and capacity.
    let lenVal = loadSliceLen(self, sliceReg);
    let capVal = loadSliceCap(self, sliceReg);

    // Branch: if length is smaller than capacity, go to @store else @grow.
    let storeBlock = try createBlock(self, "append.store");
    let growBlock = try createBlock(self, "append.grow");
    try emitBrCmp(self, il::CmpOp::Ult, il::Type::W32, lenVal, capVal, storeBlock, growBlock);
    try switchToAndSeal(self, growBlock);

    // -- @grow block ----------------------------------------------------------

    // `newCap = max(cap * 2, 1)`.
    // We are only here when at capacity, so we use `or` with `1` to ensure at least capacity `1`.
    let doubledVal = emitTypedBinOp(self, il::BinOp::Shl, il::Type::W32, capVal, il::Val::Imm(1));
    let newCapVal = emitTypedBinOp(self, il::BinOp::Or, il::Type::W32, doubledVal, il::Val::Imm(1));

    // Call allocator: `a.func(a.ctx, newCap * stride, alignment)`.
    let allocFnReg = nextReg(self);
    emitLoadW64At(self, allocFnReg, allocReg, 0);

    let allocCtxReg = nextReg(self);
    emitLoadW64At(self, allocCtxReg, allocReg, 8);

    let byteSize = emitTypedBinOp(self, il::BinOp::Mul, il::Type::W32, newCapVal, il::Val::Imm(stride as i64));
    let args = try allocVals(self, 3);

    set args[0] = il::Val::Reg(allocCtxReg);
    set args[1] = byteSize;
    set args[2] = il::Val::Imm(alignment as i64);

    let newPtrReg = nextReg(self);
    emit(self, il::Instr::Call {
        retTy: il::Type::W64,
        dst: newPtrReg,
        func: il::Val::Reg(allocFnReg),
        args,
    });

    // Copy old data byte-by-byte.
    let oldPtrReg = loadSlicePtr(self, sliceReg);
    let copyBytes = emitTypedBinOp(self, il::BinOp::Mul, il::Type::W32, lenVal, il::Val::Imm(stride as i64));
    try emitByteCopyLoop(self, newPtrReg, oldPtrReg, copyBytes, "append");

    // Update slice header.
    emitStoreW64At(self, il::Val::Reg(newPtrReg), sliceReg, SLICE_PTR_OFFSET);
    emitStoreW32At(self, newCapVal, sliceReg, SLICE_CAP_OFFSET);

    try emitJmp(self, storeBlock);
    try switchToAndSeal(self, storeBlock);

    // -- @store block ---------------------------------------------------------

    // Store element at `ptr + len * stride`.
    let ptrReg = loadSlicePtr(self, sliceReg);
    let elemDst = emitElem(self, stride, ptrReg, lenVal);
    try emitStore(self, elemDst, 0, *elemType, elemVal);

    // Increment len.
    let newLen = emitTypedBinOp(self, il::BinOp::Add, il::Type::W32, lenVal, il::Val::Imm(1));
    emitStoreW32At(self, newLen, sliceReg, SLICE_LEN_OFFSET);

    return il::Val::Reg(sliceReg);
}

/// Lower `slice.delete(index)`.
///
/// Bounds-check the index, shift elements after it by one stride
/// via a byte-copy loop, and decrement `len`.
unsafe fn lowerSliceDelete 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, call: ast::Call, elemType: *resolver::Type) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let case ast::NodeValue::FieldAccess(access) = call.callee.value
        else throw LowerError::MissingMetadata;

    let elemLayout = resolver::getTypeLayout(*elemType);
    let stride = elemLayout.size;

    // Get slice header address.
    let sliceVal = try lowerExpr(self, access.parent);
    let sliceReg = emitValToReg(self, sliceVal);

    // Lower the index argument.
    let indexVal = try lowerExpr(self, call.args[0]);

    // Load len and bounds-check: index must be smaller than length.
    let lenVal = loadSliceLen(self, sliceReg);
    try emitTrapUnlessCmp(self, il::CmpOp::Ult, il::Type::W32, indexVal, lenVal);

    // Compute the destination and source for the shift.
    let ptrReg = loadSlicePtr(self, sliceReg);
    let dst = emitElem(self, stride, ptrReg, indexVal);

    // `src = dst + stride`.
    let src = emitPtrOffset(self, dst, stride as i32);

    // Move `(len - index - 1) * stride`.
    let tailLen = emitTypedBinOp(self, il::BinOp::Sub, il::Type::W32, lenVal, indexVal);
    let tailLenMinusOne = emitTypedBinOp(self, il::BinOp::Sub, il::Type::W32, tailLen, il::Val::Imm(1));
    let moveBytes = emitTypedBinOp(self, il::BinOp::Mul, il::Type::W32, tailLenMinusOne, il::Val::Imm(stride as i64));

    // Shift elements left via byte-copy loop.
    // When deleting the last element, the loop is a no-op.
    try emitByteCopyLoop(self, dst, src, moveBytes, "delete");
    // Decrement length.
    let newLen = emitTypedBinOp(self, il::BinOp::Sub, il::Type::W32, lenVal, il::Val::Imm(1));

    emitStoreW32At(self, newLen, sliceReg, SLICE_LEN_OFFSET);
}

/// Initializer values retained across a fallible session reservation.
record SessionInitialization: Copy {
    /// Resolved allocation operation and element type.
    allocation: resolver::SessionAllocation,
    /// Runtime reservation result.
    result: il::Val,
    /// Evaluated object, fill value, or source slice.
    value: il::Val,
    /// Evaluated number of elements.
    count: il::Val,
}

/// Evaluate initializer values and reserve their session storage.
unsafe fn prepareSessionAllocation 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function, call: ast::Call, allocation: resolver::SessionAllocation
) -> SessionInitialization throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let case ast::NodeValue::FieldAccess(access) = call.callee.value else throw LowerError::ExpectedCall;
    let receiver = try lowerExpr(self, access.parent);
    let sessionReg = emitValToReg(self, receiver);
    let dataReg = nextReg(self);
    emitLoadW64At(self, dataReg, sessionReg, TRAIT_OBJ_DATA_OFFSET);
    let vtableReg = nextReg(self);
    emitLoadW64At(self, vtableReg, sessionReg, TRAIT_OBJ_VTABLE_OFFSET);
    let functionReg = nextReg(self);
    emitLoadW64At(
        self, functionReg, vtableReg,
        (allocation.methodIndex * resolver::PTR_SIZE) as i32,
    );
    let value = try lowerCallArg(self, call.args[0], true);
    if blockHasTerminator(self) {
        return SessionInitialization { allocation, result: il::Val::Undef, value, count: il::Val::Undef };
    }
    let slice = allocation.kind <> resolver::SessionAllocationKind::New;
    let mut count = il::Val::Imm(1);
    if allocation.kind == resolver::SessionAllocationKind::Copy {
        set count = loadSliceLen(self, emitValToReg(self, value));
    } else if allocation.kind == resolver::SessionAllocationKind::Fill {
        set count = try lowerExpr(self, call.args[1]);
        if blockHasTerminator(self) {
            return SessionInitialization { allocation, result: il::Val::Undef, value, count: il::Val::Undef };
        }
    }
    let layout = resolver::getTypeLayout(*allocation.item);
    let size = layout.size if layout.size > 0 else 1;
    let alignment = layout.alignment if layout.alignment > 0 else 1;
    let runtime = allocation.traitInfo.methods[allocation.methodIndex].fnType;
    let argOffset: u32 = 1 if requiresReturnParam(runtime) else 0;
    let args = try allocVals(self, (4 if slice else 3) + argOffset);
    set args[argOffset] = il::Val::Reg(dataReg);
    set args[argOffset + 1] = il::Val::Imm(size as i64);
    set args[argOffset + 2] = il::Val::Imm(alignment as i64);
    if slice {
        set args[argOffset + 3] = count;
    }
    let result = try emitCallValue(self, il::Val::Reg(functionReg), runtime, args);
    return SessionInitialization { allocation, result, value, count };
}

/// Initialize successful reservation storage before its reference is published.
unsafe fn initializeSessionAllocation 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, prepared: SessionInitialization)
    throws (LowerError) where 'arena: 'phase, 'phase: 'function
{
    let allocation = prepared.allocation;
    let base = emitValToReg(self, prepared.result);
    let value = prepared.value;
    let count = prepared.count;
    let layout = resolver::getTypeLayout(*allocation.item);
    let destination = nextReg(self);
    emitLoadW64At(self, destination, base, RESULT_VAL_OFFSET);
    match allocation.kind {
        case resolver::SessionAllocationKind::New =>
            try emitStore(self, destination, 0, *allocation.item, value),
        case resolver::SessionAllocationKind::Copy => {
            let source = loadSlicePtr(self, emitValToReg(self, value));
            let bytes = emitTypedBinOp(self, il::BinOp::Mul, il::Type::W32, count, il::Val::Imm(layout.size as i64));
            try emitByteCopyLoop(self, destination, source, bytes, "allocate.copy");
        }
        case resolver::SessionAllocationKind::Fill =>
            try emitFillLoop(self, destination, value, count, *allocation.item, layout.size),
    }
}

/// Lower a call expression, which may be a function call or type constructor.
unsafe fn lowerCallOrCtor 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node, call: ast::Call) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let nodeData = resolver::nodeData(self.low.resolver, node).extra;

    // Check for slice method dispatch.
    if let case resolver::NodeExtra::SliceAppend { elemType } = nodeData {
        return try lowerSliceAppend(self, call, elemType);
    }
    if let case resolver::NodeExtra::SliceDelete { elemType } = nodeData {
        try lowerSliceDelete(self, call, elemType);
        return il::Val::Undef;
    }
    // Check for trait method dispatch.
    if let case resolver::NodeExtra::TraitMethodCall { traitInfo, methodIndex } = nodeData {
        return try lowerTraitMethodCall(self, node, call, traitInfo, methodIndex);
    }
    // Check for standalone method call.
    if let case resolver::NodeExtra::MethodCall { method } = nodeData {
        return try lowerMethodCall(self, node, call, method);
    }
    if let sym = resolver::nodeData(self.low.resolver, call.callee).sym {
        if let case resolver::SymbolData::Type(_) = sym.data {
            let ty = try typeOf(self, node);
            let case resolver::Type::Nominal(nominal) = ty else throw LowerError::ExpectedRecord;
            return try lowerRecordCtor(self, nominal, call.args);
        }
        if let case resolver::SymbolData::Variant { .. } = sym.data {
            return try lowerUnionCtor(self, node, sym, call);
        }
    }
    return try lowerCall(self, node, call);
}

/// Lower a trait method call through v-table dispatch.
///
/// Given `obj.method(args)` where `obj` is a trait object, emits:
///
///     load w64 %data %obj 0          // data pointer
///     load w64 %vtable %obj 8        // v-table pointer
///     load w64 %fn %vtable <slot>    // function pointer
///     call <retTy> %ret %fn(%data, args...)
///
unsafe fn lowerTraitMethodCall 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    node: *ast::Node,
    call: ast::Call,
    traitInfo: *unsafe resolver::TraitType,
    methodIndex: u32
) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    // Method calls look like field accesses.
    let case ast::NodeValue::FieldAccess(access) = call.callee.value
        else throw LowerError::MissingMetadata;

    // Lower the trait object expression.
    let traitObjVal = try lowerExpr(self, access.parent);
    let traitObjReg = emitValToReg(self, traitObjVal);

    // Load data pointer from trait object.
    let dataReg = nextReg(self);
    emit(self, il::Instr::Load {
        typ: il::Type::W64,
        dst: dataReg,
        src: traitObjReg,
        offset: TRAIT_OBJ_DATA_OFFSET,
    });

    // Load v-table pointer from trait object.
    let vtableReg = nextReg(self);
    emit(self, il::Instr::Load {
        typ: il::Type::W64,
        dst: vtableReg,
        src: traitObjReg,
        offset: TRAIT_OBJ_VTABLE_OFFSET,
    });

    // Load function pointer from v-table at the method's slot offset.
    let fnPtrReg = nextReg(self);
    let slotOffset = (methodIndex * resolver::PTR_SIZE) as i32;

    emit(self, il::Instr::Load {
        typ: il::Type::W64,
        dst: fnPtrReg,
        src: vtableReg,
        offset: slotOffset,
    });
    let methodFnType = traitInfo.methods[methodIndex].fnType;

    // Build args: optional return param slot + data pointer (receiver) + user args.
    let argOffset: u32 = 1 if requiresReturnParam(methodFnType) else 0;
    let args = try allocVals(self, call.args.len + 1 + argOffset);
    set args[argOffset] = il::Val::Reg(dataReg);

    for arg, i in call.args {
        set args[i + 1 + argOffset] = try lowerCallArg(
            self, arg, i + 1 < call.args.len
        );
    }
    return try emitCallValue(self, il::Val::Reg(fnPtrReg), methodFnType, args);
}

/// Lower a call argument, snapshotting aggregate place expressions before
/// evaluating later arguments. Aggregates are represented by addresses in the
/// IL, so retaining the original address would let a later argument mutation
/// change the value already supplied for this argument.
unsafe fn lowerCallArg 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, arg: *ast::Node, hasLater: bool) -> il::Val
    throws (LowerError) where 'arena: 'phase, 'phase: 'function
{
    let val = try lowerExpr(self, arg);

    if hasLater and ast::isPlaceExpr(arg) {
        let argType = try effectiveType(self, arg);
        if isAggregateType(argType) {
            return try emitStackVal(self, argType, val);
        }
    }
    return val;
}

/// Emit a function call with return-parameter and small-aggregate handling.
///
/// All call lowering paths (regular, trait method, standalone method) converge
/// here after preparing the callee value, function type, and argument array.
/// The `args` slice must already include a slot at index zero for the hidden
/// return parameter; that slot is filled by this function.
unsafe fn emitCallValue 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    callee: il::Val,
    fnInfo: *resolver::FnType,
    args: *unsafe mut [il::Val],
) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let retTy = *fnInfo.returnType;

    if requiresReturnParam(fnInfo) {
        if fnInfo.throwList.len > 0 {
            let layout = resolver::getResultLayout(retTy, fnInfo.throwList);
            set args[0] = il::Val::Reg(emitReserveLayout(self, layout));
        } else {
            set args[0] = il::Val::Reg(try emitReserve(self, retTy));
        }
        let dst = nextReg(self);

        emit(self, il::Instr::Call {
            retTy: il::Type::W64,
            dst,
            func: callee,
            args,
        });
        return il::Val::Reg(dst);
    }
    let mut dst: ?il::Reg = nil;
    if retTy <> resolver::Type::Void and retTy <> resolver::Type::Never {
        set dst = nextReg(self);
    }
    emit(self, il::Instr::Call {
        retTy: ilType(self.low, retTy),
        dst,
        func: callee,
        args,
    });

    if retTy == resolver::Type::Never {
        emit(self, il::Instr::Unreachable);
    }
    if let d = dst {
        if isSmallAggregate(retTy) {
            let slot = emitReserveLayout(self, resolver::Layout {
                size: resolver::PTR_SIZE,
                alignment: resolver::PTR_SIZE,
            });
            emit(self, il::Instr::Store {
                typ: il::Type::W64,
                src: il::Val::Reg(d),
                dst: slot,
                offset: 0,
            });
            return il::Val::Reg(slot);
        }
        return il::Val::Reg(d);
    }
    return il::Val::Undef;
}

/// Lower a method receiver expression to a pointer value.
///
/// If the parent is already a pointer type, the value is used directly.
/// If the parent is a value type (eg. a local record), its address is taken.
unsafe fn lowerReceiver 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, parent: *ast::Node, parentTy: resolver::Type) -> il::Val
    throws (LowerError) where 'arena: 'phase, 'phase: 'function
{
    if let case resolver::Type::Pointer { .. } = parentTy {
        // Already a pointer: lower and use directly.
        return try lowerExpr(self, parent);
    }
    // Value type: take its address by lowering it and returning the slot pointer.
    // Aggregate types are already lowered as pointers to stack slots.
    let val = try lowerExpr(self, parent);
    if isAggregateType(parentTy) {
        return val;
    }
    // Scalar value: store to a stack slot and return the slot pointer.
    let layout = resolver::getLayout(self.low.resolver, parent, parentTy);
    let slot = emitReserveLayout(self, layout);
    try emitStore(self, slot, 0, parentTy, val);

    return il::Val::Reg(slot);
}

/// Lower a standalone method call via direct dispatch.
///
/// Given `obj.method(args)` where `method` is a standalone method on a concrete type,
/// emits a direct call with the receiver address as the first argument:
///
///     call <retTy> %ret @Type::method(&obj, args...)
///
unsafe fn lowerMethodCall 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    node: *ast::Node,
    call: ast::Call,
    method: &resolver::MethodEntry,
) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let case ast::NodeValue::FieldAccess(access) = call.callee.value
        else throw LowerError::MissingMetadata;

    // Get the receiver as a pointer.
    let parentTy = try typeOf(self, access.parent);
    let receiverVal = try lowerReceiver(self, access.parent, parentTy);

    let qualName = instanceMethodName(self.low, nil, method.concreteTypeName, method.name);
    let case resolver::SymbolData::Value { type: resolver::Type::Fn(fnInfo), .. } = method.symbol.data
        else panic "lowerMethodCall: expected Fn type on method symbol";

    // Build args: optional return param slot + receiver + user args.
    let argOffset: u32 = 1 if requiresReturnParam(fnInfo) else 0;
    let args = try allocVals(self, call.args.len + 1 + argOffset);
    set args[argOffset] = receiverVal;
    for arg, i in call.args {
        set args[i + 1 + argOffset] = try lowerCallArg(
            self, arg, i + 1 < call.args.len
        );
    }
    return try emitCallValue(self, il::Val::FnAddr(qualName), fnInfo, args);
}

/// Check if a call is to a compiler intrinsic and lower it directly.
unsafe fn lowerIntrinsicCall 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, call: ast::Call) -> ?il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    // Get the callee symbol and check if it's marked as an intrinsic.
    let sym = resolver::nodeData(self.low.resolver, call.callee).sym else {
        // Expressions or function pointers may not have an associated symbol.
        return nil;
    };
    if not ast::hasAttribute(sym.attrs, ast::Attribute::Intrinsic) {
        return nil;
    }
    // Check for known intrinsic names.
    if mem::eq(sym.name, "ecall") {
        return try lowerEcall(self, call);
    } else if mem::eq(sym.name, "deviceRead8") {
        return try lowerDevice(self, call, il::Type::W8, false);
    } else if mem::eq(sym.name, "deviceWrite8") {
        return try lowerDevice(self, call, il::Type::W8, true);
    } else if mem::eq(sym.name, "deviceRead16") {
        return try lowerDevice(self, call, il::Type::W16, false);
    } else if mem::eq(sym.name, "deviceWrite16") {
        return try lowerDevice(self, call, il::Type::W16, true);
    } else if mem::eq(sym.name, "deviceRead32") {
        return try lowerDevice(self, call, il::Type::W32, false);
    } else if mem::eq(sym.name, "deviceWrite32") {
        return try lowerDevice(self, call, il::Type::W32, true);
    } else if mem::eq(sym.name, "deviceRead64") {
        return try lowerDevice(self, call, il::Type::W64, false);
    } else if mem::eq(sym.name, "deviceWrite64") {
        return try lowerDevice(self, call, il::Type::W64, true);
    } else if mem::eq(sym.name, "ebreak") {
        return try lowerEbreak(self, call);
    } else if mem::eq(sym.name, "memoryFence") {
        return try lowerMemoryFence(self, call);
    } else {
        throw LowerError::UnknownIntrinsic;
    }
}

/// Lower an ecall intrinsic: `ecall(num, a0, a1, a2, a3) -> i32`.
unsafe fn lowerEcall 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, call: ast::Call) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    if call.args.len <> 5 {
        throw LowerError::InvalidArgCount;
    }
    let num = try lowerExpr(self, call.args[0]);
    let a0 = try lowerExpr(self, call.args[1]);
    let a1 = try lowerExpr(self, call.args[2]);
    let a2 = try lowerExpr(self, call.args[3]);
    let a3 = try lowerExpr(self, call.args[4]);
    let dst = nextReg(self);

    emit(self, il::Instr::Ecall { dst, num, a0, a1, a2, a3 });

    return il::Val::Reg(dst);
}

/// Lower an ebreak intrinsic: `ebreak()`.
unsafe fn lowerEbreak 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, call: ast::Call) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    if call.args.len <> 0 {
        throw LowerError::InvalidArgCount;
    }
    emit(self, il::Instr::Ebreak);

    return il::Val::Undef;
}

/// Lower `memoryFence()`.
unsafe fn lowerMemoryFence 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, call: ast::Call) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    if call.args.len <> 0 {
        throw LowerError::InvalidArgCount;
    }
    emit(self, il::Instr::MemoryFence);
    return il::Val::Undef;
}

/// Resolve callee to an IL value. For direct function calls, use the symbol name.
/// For variables holding function pointers or complex expressions (eg. `array[i]()`),
/// lower the callee expression.
unsafe fn lowerCallee 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, callee: *ast::Node) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    if let sym = resolver::nodeData(self.low.resolver, callee).sym {
        if let case ast::NodeValue::FnDecl(_) = sym.node.value {
            // First try to look up the symbol in our registered functions.
            // This handles cross-package calls correctly, since packages are
            // lowered in dependency order.
            if let qualName = lookupSymbolName(self.low, sym) {
                return il::Val::FnAddr(qualName);
            }
            // Fall back to computing the qualified name from the module graph.
            // This works for functions in the current package.
            let modId = resolver::moduleIdForSymbol(self.low.resolver, sym);
            return il::Val::FnAddr(qualifyName(self.low, modId, sym.name));
        }
    }
    return try lowerExpr(self, callee);
}

/// Lower a function call expression.
unsafe fn lowerCall 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node, call: ast::Call) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    // Check for intrinsic calls before normal call lowering.
    if let intrinsicVal = try lowerIntrinsicCall(self, call) {
        return intrinsicVal;
    }
    let calleeTy = try typeOf(self, call.callee);
    let case resolver::Type::Fn(fnInfo) = calleeTy else {
        throw LowerError::ExpectedFunction;
    };
    let callee = try lowerCallee(self, call.callee);
    let offset: u32 = 1 if requiresReturnParam(fnInfo) else 0;
    let args = try allocVals(self, call.args.len + offset);
    for arg, i in call.args {
        set args[i + offset] = try lowerCallArg(
            self, arg, i + 1 < call.args.len
        );
    }

    return try emitCallValue(self, callee, fnInfo, args);
}

/// Apply coercions requested by the resolver.
unsafe fn applyCoercion 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node, val: il::Val) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let coerce = resolver::coercionFor(self.low.resolver, node) else {
        return val;
    };
    match coerce {
        case resolver::Coercion::OptionalLift(optType) => {
            if let case ast::NodeValue::Nil = node.value {
                return try buildNilOptional(self, optType);
            }
            return try wrapInOptional(self, val, optType);
        }
        case resolver::Coercion::NumericCast { from, to } => {
            return lowerNumericCast(self, val, from, to);
        }
        case resolver::Coercion::ResultWrap => {
            let payloadType = *self.fnType.returnType;
            return try buildResult(self, 0, val, payloadType);
        }
        case resolver::Coercion::TraitObject { traitInfo, inst } => {
            return try buildTraitObject(self, val, traitInfo, inst);
        }
        case resolver::Coercion::Identity => return val,
    }
}

/// Lower an implicit numeric cast coercion.
///
/// Handles widening conversions between integer types. Uses sign-extension
/// for signed source types and zero-extension for unsigned source types.
unsafe fn lowerNumericCast 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, val: il::Val, srcType: resolver::Type, dstType: resolver::Type) -> il::Val where 'arena: 'phase, 'phase: 'function {
    let srcLayout = resolver::getTypeLayout(srcType);
    let dstLayout = resolver::getTypeLayout(dstType);

    if srcLayout.size == dstLayout.size {
        // Same size: bit pattern is unchanged, value is returned as-is.
        return val;
    }
    // Widening: extend based on source signedness.
    // Narrowing: truncate and normalize to destination width.
    let widening = srcLayout.size < dstLayout.size;
    let extType = ilType(self.low, srcType) if widening else ilType(self.low, dstType);
    let signed = isSignedType(srcType) if widening else isSignedType(dstType);
    let dst = nextReg(self);

    if signed {
        emit(self, il::Instr::Sext { typ: extType, dst, val });
    } else {
        emit(self, il::Instr::Zext { typ: extType, dst, val });
    }
    return il::Val::Reg(dst);
}

/// Lower a global value symbol.
unsafe fn lowerGlobalValue 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, sym: *unsafe resolver::Symbol, ty: resolver::Type) -> il::Val where 'arena: 'phase, 'phase: 'function {
    // Function pointer reference: return the function's address directly.
    // Functions have no separate storage cell in the data section.
    if let case resolver::Type::Fn(_) = ty {
        return il::Val::Reg(emitFnAddr(self, sym));
    }
    let src = emitDataAddr(self, sym);

    return emitRead(self, src, 0, ty);
}

/// Lower an identifier that refers to a global symbol.
unsafe fn lowerGlobalSymbol 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    // First try to get a compile-time constant value.
    if let constVal = resolver::constValueEntry(self.low.resolver, node) {
        return try constValueToVal(self, constVal, node);
    }
    // Otherwise get the symbol.
    let sym = try symOf(self, node);

    match sym.data {
        case resolver::SymbolData::Constant { type, .. } => {
            let src = emitDataAddr(self, sym);

            return emitRead(self, src, 0, type);
        }
        case resolver::SymbolData::Value { type, .. } =>
            return lowerGlobalValue(self, sym, type),
        else => throw LowerError::UnexpectedNodeValue(node),
    }
}

/// Lower an assignment to a static variable.
unsafe fn lowerStaticAssign 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, target: *ast::Node, val: il::Val) throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let sym = try symOf(self, target);
    let case resolver::SymbolData::Value { type, .. } = sym.data else {
        throw LowerError::ImmutableAssignment;
    };
    let dst = emitDataAddr(self, sym);

    try emitStore(self, dst, 0, type, val);
}

/// Lower a scope access expression like `Module::Const` or `Union::Variant`.
/// This doesn't handle record literal variants.
unsafe fn lowerScopeAccess 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    // First try to get a compile-time constant value.
    if let constVal = resolver::constValueEntry(self.low.resolver, node) {
        return try constValueToVal(self, constVal, node);
    }
    // Otherwise get the associated symbol.
    let data = resolver::nodeData(self.low.resolver, node);
    let sym = data.sym else {
        throw LowerError::MissingSymbol(node);
    };
    match sym.data {
        case resolver::SymbolData::Variant { index, .. } => {
            let mut indexValue = index as i64;
            if let idx = voidVariantIndex(self.low.resolver, node) {
                set indexValue = idx;
            }
            // Void union variant like `Option::None`.
            if data.ty == resolver::Type::Unknown {
                throw LowerError::MissingType(node);
            }
            // All-void unions are passed as scalars (the tag byte).
            // Return an immediate instead of building a tagged aggregate.
            if resolver::isVoidUnion(data.ty) {
                return il::Val::Imm(indexValue);
            }
            let unionInfo = unionInfoFromType(data.ty) else {
                throw LowerError::MissingMetadata;
            };
            let valOffset = unionInfo.valOffset as i32;
            return try buildTagged(self, resolver::getTypeLayout(data.ty), indexValue, nil, resolver::Type::Void, 1, valOffset);
        }
        case resolver::SymbolData::Constant { type, .. } => {
            // Constant without compile-time value (e.g. record constant);
            // load from data section.
            let src = emitDataAddr(self, sym);

            // Aggregate constants live in read-only memory.  Return a
            // mutable copy so that callers that assign through the
            // resulting pointer do not fault.
            if isAggregateType(type) {
                let layout = resolver::getTypeLayout(type);
                let dst = emitReserveLayout(self, layout);
                emit(self, il::Instr::Blit { dst, src, size: il::Val::Imm(layout.size as i64) });

                return il::Val::Reg(dst);
            }
            return emitRead(self, src, 0, type);
        }
        case resolver::SymbolData::Value { type, .. } =>
            return lowerGlobalValue(self, sym, type),
        else =>
            throw LowerError::UnexpectedNodeValue(node),
    }
}

/// Lower an expression AST node to an IL value.
/// This is the main expression dispatch, all expression nodes go through here.
unsafe fn lowerExpr 'arena 'phase 'function (self: &mut FnLowerer 'arena 'phase 'function, node: *ast::Node) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    if self.low.options.debug {
        set self.srcLoc.offset = node.span.offset;
    }
    let mut val: il::Val = undefined;

    match node.value {
        case ast::NodeValue::Ident(_) => {
            // First try local variable lookup.
            // Otherwise fall back to global symbol lookup.
            if let v = lookupLocalVar(self, node) {
                set val = try useVar(self, v);
                if self.vars[*v].addressTaken {
                    let typ = try typeOf(self, node);
                    let ptr = emitValToReg(self, val);
                    set val = emitRead(self, ptr, 0, typ);
                }
            } else {
                set val = try lowerGlobalSymbol(self, node);
            }
        }
        case ast::NodeValue::ScopeAccess(_) => {
            set val = try lowerScopeAccess(self, node);
        }
        case ast::NodeValue::Number(lit) => {
            set val = il::Val::Imm(lit.magnitude as i64);
        }
        case ast::NodeValue::Bool(b) => {
            set val = il::Val::Imm(1) if b else il::Val::Imm(0);
        }
        case ast::NodeValue::Char(c) => {
            set val = il::Val::Imm(c as i64);
        }
        case ast::NodeValue::Nil => {
            let typ = try typeOf(self, node);
            if let case resolver::Type::Optional(_) = typ {
                set val = try buildNilOptional(self, typ);
            } else if let case resolver::Type::Nil = typ {
                // Standalone `nil` without a concrete optional type. We can't
                // generate a proper value representation.
                throw LowerError::MissingType(node);
            } else {
                throw LowerError::NilInNonOptional;
            }
        }
        case ast::NodeValue::RecordLit(lit) => {
            set val = try lowerRecordLit(self, node, lit);
        }
        case ast::NodeValue::AddressOf(addr) => {
            set val = try lowerAddressOf(self, node, addr);
        }
        case ast::NodeValue::Deref(target) => {
            set val = try lowerDeref(self, node, target);
        }
        case ast::NodeValue::BinOp(binop) => {
            set val = try lowerBinOp(self, node, binop);
        }
        case ast::NodeValue::UnOp(unop) => {
            set val = try lowerUnOp(self, node, unop);
        }
        case ast::NodeValue::Subscript { container, index } => {
            set val = try lowerSubscript(self, node, container, index);
        }
        case ast::NodeValue::BuiltinCall { kind, args } => {
            set val = try lowerBuiltinCall(self, node, kind, args);
        }
        case ast::NodeValue::Call(call) => {
            set val = try lowerCallOrCtor(self, node, call);
        }
        case ast::NodeValue::Try(t) => {
            set val = try lowerTry(self, node, t);
        }
        case ast::NodeValue::FieldAccess(access) => {
            // Check for compile-time constant (e.g., `arr.len` on fixed-size arrays).
            if let constVal = resolver::constValueEntry(self.low.resolver, node) {
                match constVal {
                    // TODO: Handle `u32` values that don't fit in an `i32`.
                    //       Perhaps just store the `ConstInt`.
                    case resolver::ConstValue::Int(i) => set val = il::Val::Imm(constIntToI64(i)),
                    else => set val = try lowerFieldAccess(self, access),
                }
            } else {
                set val = try lowerFieldAccess(self, access);
            }
        }
        case ast::NodeValue::ArrayLit(elements) => {
            set val = try lowerArrayLit(self, node, elements);
        }
        case ast::NodeValue::ArrayRepeatLit(repeat) => {
            set val = try lowerArrayRepeatLit(self, node, repeat);
        }
        case ast::NodeValue::RegionApply { value, .. } => {
            set val = try lowerExpr(self, value);
        }
        case ast::NodeValue::As(cast) => {
            set val = try lowerCast(self, node, cast);
        }
        case ast::NodeValue::CondExpr(cond) => {
            set val = try lowerCondExpr(self, node, cond);
        }
        case ast::NodeValue::String(s) => {
            set val = try lowerStringLit(self, node, s);
        }
        case ast::NodeValue::Undef => {
            let typ = try typeOf(self, node);
            if isAggregateType(typ) {
                // When `undefined` appears as a stand-alone expression,
                /// we need a stack slot for reads and writes.
                let slot = try emitReserve(self, typ);
                set val = il::Val::Reg(slot);
            } else {
                set val = il::Val::Undef;
            }
        }
        case ast::NodeValue::Panic { .. } => {
            // Panic in expression context (e.g. match arm). Emit unreachable
            // and return a dummy value since control won't continue.
            emit(self, il::Instr::Unreachable);
            set val = il::Val::Undef;
        }
        case ast::NodeValue::Assert { .. } => {
            // Assert in expression context. Lower as statement, return `void`.
            try lowerNode(self, node);
            set val = il::Val::Undef;
        }
        case ast::NodeValue::Block(_) => {
            try lowerBlock(self, node);
            set val = il::Val::Undef;
        }
        case ast::NodeValue::ExprStmt(expr) => {
            let _ = expr;
            set val = il::Val::Undef;
        }
        // Lower these as statements.
        case ast::NodeValue::ConstDecl(decl) => {
            try registerLocalDataDeclName(self, node);
            try lowerDataDecl(self.low, node, decl.value, true);
            set val = il::Val::Undef;
        }
        case ast::NodeValue::StaticDecl(decl) => {
            try registerLocalDataDeclName(self, node);
            try lowerDataDecl(self.low, node, decl.value, false);
            set val = il::Val::Undef;
        }
        case ast::NodeValue::Throw { .. },
             ast::NodeValue::Return { .. },
             ast::NodeValue::Continue,
             ast::NodeValue::Break => {
            try lowerNode(self, node);
            set val = il::Val::Undef;
        }
        else => {
            panic "lowerExpr: node is not an expression";
        }
    }
    return try applyCoercion(self, node, val);
}

/// Translate a Radiance type to an IL type.
///
/// The IL type system is much simpler than Radiance's, only primitive types
/// are used. If a Radiance type doesn't fit in a machine word, it is passed
/// by reference.
///
/// The IL doesn't track signedness - that's encoded in the instructions
/// (e.g., Slt vs Ult).
unsafe fn ilType 'arena 'phase (self: &mut Lowerer 'arena 'phase, typ: resolver::Type) -> il::Type where 'arena: 'phase {
    match typ {
        case resolver::Type::Bool,
             resolver::Type::I8,
             resolver::Type::U8 => return il::Type::W8,
        case resolver::Type::I16,
             resolver::Type::U16 => return il::Type::W16,
        case resolver::Type::I32,
             resolver::Type::U32 => return il::Type::W32,
        case resolver::Type::I64,
             resolver::Type::U64,
             resolver::Type::Pointer { .. },
             resolver::Type::Slice { .. },
             resolver::Type::TraitObject { .. },
             resolver::Type::Array(_),
             resolver::Type::Optional(_),
             resolver::Type::Fn(_),
             resolver::Type::Session(_),
             resolver::Type::Cell { .. } => return il::Type::W64,
        case resolver::Type::Nominal(_) => {
            if resolver::isVoidUnion(typ) {
                return il::Type::W8;
            }
            return il::Type::W64;
        }
        case resolver::Type::Void, resolver::Type::Never => return il::Type::W64,
        // [`Type::Int`] is the type of unsuffixed integer literals and their
        // compound expressions (e.g. `1 + 2`). It defaults to W64 (i64) here,
        // matching the native word size on RV64. It cannot be resolved earlier
        // because the resolver uses [`Type::Int`] to distinguish unsuffixed
        // expressions from explicitly typed ones, which affects coercion
        // behavior (e.g. implicit narrowing).
        case resolver::Type::Int => return il::Type::W64,
        case resolver::Type::Opaque => panic "ilType: opaque type must be behind a pointer",
        else => panic "ilType: type cannot be lowered",
    }
}

/// Lower one fixed-width MMIO operation without exposing a pointer value.
unsafe fn lowerDevice 'arena 'phase 'function (
    self: &mut FnLowerer 'arena 'phase 'function,
    call: ast::Call,
    typ: il::Type,
    writing: bool
) -> il::Val throws (LowerError) where 'arena: 'phase, 'phase: 'function {
    let count: u32 = 3 if writing else 2;
    if call.args.len <> count {
        throw LowerError::InvalidArgCount;
    }
    let handle = try lowerExpr(self, call.args[0]);
    let offset = try lowerExpr(self, call.args[1]);
    if writing {
        let value = try lowerExpr(self, call.args[2]);
        emit(self, il::Instr::DeviceWrite { typ, handle, offset, value });
        return il::Val::Undef;
    }
    let dst = nextReg(self);
    emit(self, il::Instr::DeviceRead { typ, dst, handle, offset });
    return il::Val::Reg(dst);
}