radiance · Radiance self-hosting compiler

compiler/ lib/ examples/ std/ arch/ char/ collections/ lang/ alloc/ ast/ gen/ il/ module/ parser/ resolver/ scanner/ alloc.rad 5.3 KiB ast.rad 23.6 KiB gen.rad 513 B il.rad 15.5 KiB lower.rad 277.7 KiB module.rad 13.5 KiB package.rad 1.3 KiB parser.rad 79.6 KiB resolver.rad 332.7 KiB scanner.rad 17.6 KiB sexpr.rad 6.4 KiB strings.rad 2.2 KiB types.rad 286 B sys/ arch.rad 68 B char.rad 855 B collections.rad 39 B fmt.rad 8.3 KiB intrinsics.rad 467 B io.rad 1.7 KiB lang.rad 276 B mem.rad 2.3 KiB sys.rad 179 B testing.rad 2.4 KiB tests.rad 15.7 KiB vec.rad 3.2 KiB std.rad 281 B scripts/ seed/ sublime/ test/ vim/ .gitignore 336 B .gitsigners 112 B CONTRIBUTING 2.1 KiB LICENSE 1.1 KiB Makefile 4.1 KiB README 2.5 KiB STYLE 2.6 KiB std.lib 1.2 KiB std.lib.test 347 B
lib/std/lang/lower.rad 277.7 KiB raw
//! AST to IL lowering pass.
//!
//! This module converts the typed AST produced by the resolver into a linear
//! SSA-based intermediate language (IL). The IL is suitable for further
//! optimization and code generation.
//!
//! # Design Overview
//!
//! The lowering process works in two main phases:
//!
//! 1. Module-level pass: Iterates over top-level declarations, lowering
//!    each function independently while accumulating global data items (strings,
//!    constants, static arrays) into a shared data section.
//!
//! 2. Function-level pass: For each function, constructs an SSA-form control
//!    flow graph (CFG) with basic blocks connected by jumps and branches. Uses
//!    a simplified SSA construction algorithm where variables are tracked per-block
//!    and block parameters are inserted lazily when a variable is used before
//!    being defined in a block.
//!
//! # Memory Model
//!
//! All allocations use an arena allocator passed through the `Lowerer` context.
//! The IL is entirely stack-based at runtime -- there's no heap allocation during
//! program execution. Aggregate values (records, slices, optionals) are passed
//! by pointer on the stack.
//!
//! # SSA Construction
//!
//! SSA form is built incrementally as the AST is walked. When a variable is
//! defined (via `defVar`), its current value is recorded in the current block.
//! When a variable is used (via `useVar`), the algorithm either:
//! - Returns the value if defined in this block
//! - Recurses to predecessors if the block is sealed (all predecessors known)
//! - Inserts a block parameter if the variable value must come from multiple paths
//!
//! Block "sealing" indicates that all predecessor edges are known, enabling
//! the SSA construction to resolve cross-block variable references.
//!
//! # SSA Variable API
//!
//! 1. `newVar` creates a logical variable that can be defined differently in each block.
//! 2. `defVar` defines the variable's value in the current block.
//! 3. `useVar` reads the variable; if called in a block with multiple
//!             predecessors that defined it differently, the SSA algorithm
//!             automatically creates a block parameter.
//!
//! # Expression Lowering
//!
//! Expressions produce IL values which can be:
//!
//! - Imm(i64): immediate/constant values
//! - Reg(u32): SSA register references
//! - Sym(name): symbol references (for function pointers, data addresses)
//! - Undef: for unused values
//!
//! For aggregate types (records, arrays, slices, optionals), the "value" is
//! actually a pointer to stack-allocated memory containing the aggregate.
//!
//! # Block ordering invariant
//!
//! The register allocator processes blocks in forward index order and uses a
//! single global assignment array. This means a value's definition block must
//! have a lower index than any block that uses the value (except through
//! back-edges, where block parameters handle the merge). The lowerer maintains
//! this by creating blocks in control-flow order. For `for` loops, the step
//! block is created lazily (after the loop body) so that its index is higher
//! than all body blocks -- see [`lowerForLoop`] and [`getOrCreateContinueBlock`].
//!
//! A more robust alternative would be per-block register maps with edge-copy
//! resolution (as in QBE's `rega.c`), which is block-order-independent. That
//! would eliminate this invariant at the cost of a more complex allocator.
//!
//! # Notes on constant data lowering
//!
//! The lowerer flattens constant AST expressions directly into IL data values.
//! Primitive constants use [`resolver::ConstValue`] as an intermediate form.
//! Record padding is explicit: `undef * N;` for N padding bytes.
//!
//!   AST Node (ArrayLit, RecordLit, literals, etc.)
//!       ↓ [`lowerConstData`]
//!   resolver::ConstValue (Bool, Char, String, Int)
//!       ↓ [`constValueToDataItem`]
//!   il::DataItem (Val, Sym, Str, Undef)
//!       ↓
//!   il::DataValue
//!       ↓
//!   il::Data
//!
use std::fmt;
use std::io;
use std::lang::alloc;
use std::lang::types;
use std::mem;
use std::lang::ast;
use std::lang::il;
use std::lang::module;
use std::lang::resolver;

// TODO: Search for all `_ as i32` to ensure that casts from u32 to i32 don't
// happen, since they are potentially truncating values.

// TODO: Support constant union lowering.
// TODO: Void unions should be passed by value.

////////////////////
// Error Handling //
////////////////////

/// Lowering errors are typically unrecoverable since they indicate bugs in
/// the resolver or malformed AST that should have been caught earlier.
export union LowerError: Copy {
    /// A node's symbol was not set before lowering.
    MissingSymbol(*ast::Node),
    /// A node's type was not set before lowering.
    MissingType(*ast::Node),
    /// A node's constant value was not set before lowering.
    MissingConst(*ast::Node),
    /// An optional type was expected.
    ExpectedOptional,
    /// A record type was expected.
    ExpectedRecord,
    /// An array was expected.
    ExpectedArray,
    /// A block was expected.
    ExpectedBlock(*ast::Node),
    /// An identifier was expected.
    ExpectedIdentifier,
    /// A function parameter was expected.
    ExpectedFunctionParam,
    /// A function type was expected.
    ExpectedFunction,
    /// Trying to lower loop construct outside of loop.
    OutsideOfLoop,
    /// Invalid variable use.
    InvalidUse,
    /// Unexpected node value.
    UnexpectedNodeValue(*ast::Node),
    /// Unexpected type.
    UnexpectedType(resolver::Type),

    /// Missing control flow target.
    MissingTarget,
    /// Missing metadata that should have been set by resolver.
    MissingMetadata,
    /// Expected a slice or array type for operation.
    ExpectedSliceOrArray,
    /// Expected a variant symbol.
    ExpectedVariant,
    /// Expected a call expression.
    ExpectedCall,
    /// Field not found in record or invalid field access.
    FieldNotFound,
    /// Assignment to an immutable variable.
    ImmutableAssignment,
    /// Nil used in non-optional context.
    NilInNonOptional,
    /// Invalid argument count for builtin.
    InvalidArgCount,
    /// Feature or pattern not supported by the lowerer.
    Unsupported,
    /// Unknown intrinsic function.
    UnknownIntrinsic,
    /// Allocation failure.
    AllocationFailed,
}

/// Print a LowerError for debugging.
export fn printError(err: LowerError) {
    match err {
        case LowerError::MissingSymbol(_) => io::print("MissingSymbol"),
        case LowerError::MissingType(_) => io::print("MissingType"),
        case LowerError::MissingConst(_) => io::print("MissingConst"),
        case LowerError::ExpectedOptional => io::print("ExpectedOptional"),
        case LowerError::ExpectedRecord => io::print("ExpectedRecord"),
        case LowerError::ExpectedArray => io::print("ExpectedArray"),
        case LowerError::ExpectedBlock(_) => io::print("ExpectedBlock"),
        case LowerError::ExpectedIdentifier => io::print("ExpectedIdentifier"),
        case LowerError::ExpectedFunctionParam => io::print("ExpectedFunctionParam"),
        case LowerError::ExpectedFunction => io::print("ExpectedFunction"),
        case LowerError::OutsideOfLoop => io::print("OutsideOfLoop"),
        case LowerError::InvalidUse => io::print("InvalidUse"),
        case LowerError::UnexpectedNodeValue(_) => io::print("UnexpectedNodeValue"),
        case LowerError::UnexpectedType(_) => io::print("UnexpectedType"),
        case LowerError::MissingTarget => io::print("MissingTarget"),
        case LowerError::MissingMetadata => io::print("MissingMetadata"),
        case LowerError::ExpectedSliceOrArray => io::print("ExpectedSliceOrArray"),
        case LowerError::ExpectedVariant => io::print("ExpectedVariant"),
        case LowerError::ExpectedCall => io::print("ExpectedCall"),
        case LowerError::FieldNotFound => io::print("FieldNotFound"),
        case LowerError::ImmutableAssignment => io::print("ImmutableAssignment"),
        case LowerError::NilInNonOptional => io::print("NilInNonOptional"),
        case LowerError::InvalidArgCount => io::print("InvalidArgCount"),
        case LowerError::Unsupported => io::print("Unsupported"),
        case LowerError::UnknownIntrinsic => io::print("UnknownIntrinsic"),
        case LowerError::AllocationFailed => io::print("AllocationFailed"),
    }
}

///////////////
// Constants //
///////////////

/// Maximum nesting depth of loops.
constant MAX_LOOP_DEPTH: u32 = 16;
/// Maximum number of `catch` clauses per `try`.
constant MAX_CATCH_CLAUSES: u32 = 32;

// Slice Layout
//
// A slice is a fat pointer consisting of a data pointer, a length and a capacity.
// `{ ptr: u32, len: u32, cap: u32 }`.

/// Slice data pointer offset.
constant SLICE_PTR_OFFSET: i32 = 0;
/// Offset of slice length in slice data structure.
constant SLICE_LEN_OFFSET: i32 = resolver::PTR_SIZE as i32;
/// Offset of slice capacity in slice data structure.
constant SLICE_CAP_OFFSET: i32 = resolver::PTR_SIZE as i32 + 4;

// Trait Object Layout
//
// A trait object is a fat pointer consisting of a data pointer and a
// v-table pointer. `{ data: *T, vtable: *VTable }`.

/// Trait object data pointer offset.
constant TRAIT_OBJ_DATA_OFFSET: i32 = 0;
/// Trait object v-table pointer offset.
constant TRAIT_OBJ_VTABLE_OFFSET: i32 = resolver::PTR_SIZE as i32;

// Tagged Value Layout (optionals, tagged unions)
//
// Optionals and unions use 1-byte tags. Results use 8-byte tags.
//
// `{ tag: u8, [padding], payload: T }`
//
// Optionals use `tag: 0` for `nil` and `tag: 1` otherwise.
// When `T` is a pointer, the entire optional is stored as a single pointer.
//
// Tagged unions have a payload the size of the maximum variant size.

/// Offset of tag in tagged value data structure.
constant TVAL_TAG_OFFSET: i32 = 0;
/// Offset of value in result data structure (8-byte tag).
constant RESULT_VAL_OFFSET: i32 = resolver::PTR_SIZE as i32;

//////////////////////////
// Core Data Structures //
//////////////////////////

/// Options controlling the lowering pass.
export record LowerOptions: Copy {
    /// Whether to emit source location info.
    debug: bool,
    /// Whether to lower `@test` functions.
    buildTest: bool,
}

/// Role of a lowered function in the final program.
export union FnRole: Copy {
    /// Regular function.
    Normal,
    /// Program entry function marked with `@default`.
    Default,
}

/// Function sink used by lowerers that consume functions as they are produced.
export record FnSink: Copy {
    /// Opaque context. It must remain valid for every sink callback.
    ctx: *unsafe mut opaque,
    /// Callback invoked for each lowered function.
    emitFn: unsafe fn(*unsafe mut opaque, *unsafe il::Fn, FnRole),
}

/// Destination for functions produced by the lowerer.
export union FnOutput: Copy {
    /// Store lowered functions in the provided slice.
    Accumulate(*unsafe mut [*unsafe il::Fn]),
    /// Send lowered functions to an external consumer immediately.
    Stream(FnSink),
}

/// Module-level lowering context. Shared across all function lowerings.
/// Holds global state like the data section (strings, constants) and provides
/// access to the resolver for type queries.
export record Lowerer {
    /// Arena for persistent lowering state, including data and symbol names.
    arena: *unsafe mut alloc::Arena,
    /// Arena for allocations owned by the function currently being lowered.
    fnArena: *unsafe mut alloc::Arena,
    /// Allocator backed by the arena.
    allocator: alloc::Allocator,
    /// Resolver for type information. Used to query types, symbols, and
    /// compile-time constant values during lowering.
    resolver: *unsafe resolver::Resolver,
    /// Module graph for cross-module symbol resolution.
    moduleGraph: ?*unsafe module::ModuleGraph,
    /// Package name for qualified symbol names.
    pkgName: *[u8],
    /// Current module being lowered.
    currentMod: ?u16,
    /// Global data items (string literals, constants, static arrays).
    /// These become the data sections in the final binary.
    data: *mut [il::Data],
    /// Destination for lowered functions.
    output: FnOutput,
    /// Map of function symbols to qualified names.
    fnSyms: *mut [FnSymEntry],
    /// Global error type tag table. Maps nominal types to unique tags.
    errTags: *mut [ErrTagEntry],
    /// Next error tag to assign (starts at 1; 0 = success).
    errTagCounter: u32,
    /// Lowering options.
    options: LowerOptions,
}

/// Entry mapping a function symbol to its qualified name.
record FnSymEntry: Copy {
    sym: *unsafe resolver::Symbol,
    qualName: *[u8],
}

/// Entry in the global error tag table.
record ErrTagEntry: Copy {
    /// The type of this error, identified by its interned pointer.
    ty: resolver::Type,
    /// The globally unique tag assigned to this error type (non-zero).
    tag: u32,
}

/// Compute the maximum size of any error type in a throw list.
unsafe fn maxErrSize(throwList: *[*resolver::Type]) -> u32 {
    let mut maxSize: u32 = 0;
    for ty in throwList {
        let size = resolver::getTypeLayout(*ty).size;
        if size > maxSize {
            set maxSize = size;
        }
    }
    return maxSize;
}

/// Get or assign a globally unique error tag for the given error type.
/// Tag `0` is reserved for success; error tags start at `1`.
fn getOrAssignErrorTag(self: &mut Lowerer, errType: resolver::Type) -> u32 {
    for entry in &self.errTags[..] {
        if entry.ty == errType {
            return entry.tag;
        }
    }
    let tag = self.errTagCounter;

    set self.errTagCounter += 1;
    self.errTags.append(ErrTagEntry { ty: errType, tag }, self.allocator);

    return tag;
}

/// Emit one function to the active output.
unsafe fn emitFunction(self: &mut Lowerer, func: *unsafe il::Fn, role: FnRole) {
    match self.output {
        case FnOutput::Accumulate(accumulated) => {
            let mut fns = accumulated;
            fns.append(func, self.allocator);
            set self.output = FnOutput::Accumulate(fns);
        }
        case FnOutput::Stream(sink) => {
            sink.emitFn(sink.ctx, func, role);
        }
    }
}

/// Get the function role for a top-level function declaration.
fn lowerFnRole(isRoot: bool, attrs: ?ast::Attributes) -> FnRole {
    if isRoot and checkAttr(attrs, ast::Attribute::Default) {
        return FnRole::Default;
    }
    return FnRole::Normal;
}

/// Builder for accumulating data values during constant lowering.
record DataValueBuilder {
    /// Allocator used to grow the value storage.
    allocator: alloc::Allocator,
    /// Owned values under construction.
    values: *mut [il::DataValue],
    /// Whether all pushed values can be represented by zero-filled memory.
    zeroInit: bool,
}

/// Result of lowering constant data.
record ConstDataResult: Copy {
    /// Immutable initializer values.
    values: *[il::DataValue],
    /// Whether the values can be represented by zero-filled memory.
    zeroInit: bool,
}

/// Create a new builder.
fn dataBuilder(allocator: alloc::Allocator) -> DataValueBuilder {
    return DataValueBuilder { allocator, values: &mut [], zeroInit: true };
}

/// Return whether a data item can be omitted from a zero-filled image.
fn dataIsZero(item: il::DataItem) -> bool {
    match item {
        case il::DataItem::Undef => return true,
        case il::DataItem::Val { val, .. } => return val == 0,
        else => return false,
    }
}

/// Return whether all data values can be omitted from a zero-filled image.
fn dataValuesAreZeroInit(values: *[il::DataValue]) -> bool {
    for value in values {
        if value.count > 0 and not dataIsZero(value.item) {
            return false;
        }
    }
    return true;
}

/// Append a data value to the builder.
fn dataBuilderPush(b: &mut DataValueBuilder, value: il::DataValue) {
    b.values.append(value, b.allocator);

    if value.count > 0 and not dataIsZero(value.item) {
        set b.zeroInit = false;
    }
}

/// Return the accumulated values.
fn dataBuilderFinish(b: DataValueBuilder) -> ConstDataResult {
    let case DataValueBuilder { values, zeroInit, .. } = b
        else panic "expected data builder";
    return ConstDataResult { values, zeroInit };
}

///////////////////////////
// SSA Variable Tracking //
///////////////////////////

// The SSA construction algorithm tracks variable definitions per-block.
// Each source-level variable gets a [`Var`] handle, and each block maintains
// a mapping from [`Var`] to current SSA value. When control flow merges,
// block parameters are inserted to merge values from different control flow paths.

/// A variable handle. Represents a source-level variable during lowering.
/// The same [`Var`] can have different SSA values in different blocks.
export record Var: Copy(u32);

/// Metadata for a source-level variable, stored once per function.
///
/// Each variable declaration in the source creates one [`VarData`] entry in the
/// function's `variables` array, indexed by `id`. This contains static
/// properties that don't change across basic blocks.
///
/// Per-block SSA values are tracked separately in [`BlockData::vars`] as [`?il::Val`],
/// where `nil` means "not yet assigned in this block". Together they implement
/// SSA construction.
///
record VarData: Copy {
    /// Variable name, used by [`lookupVarByName`] to resolve identifiers.
    /// Nil for anonymous variables (e.g., internal loop counters).
    name: ?*[u8],
    /// IL type of this variable. Set at declaration time and used when
    /// generating loads, stores, and type-checking assignments.
    type: il::Type,
    /// Whether this variable was declared with `mut`. Controls whether [`defVar`]
    /// is allowed after the initial definition.
    mutable: bool,
    /// Whether this variable's address has been taken (e.g. via `&mut x`).
    /// When true, the SSA value is a pointer to a stack slot and reads/writes
    /// must go through memory instead of using the cached SSA value directly.
    addressTaken: bool,
}

/// Links a function parameter to its corresponding variable for the entry block.
/// After creating the entry block, we iterate through these to define initial values.
record FnParamBinding: Copy {
    /// The variable that receives this parameter's value.
    var: Var,
    /// SSA register containing the parameter value from the caller.
    reg: il::Reg,
}

////////////////////////////////
// Basic Block Representation //
////////////////////////////////

// During lowering, we build a CFG of basic blocks. Each block accumulates
// instructions until terminated by a jump, branch, return, or unreachable.
// Blocks can be created before they're filled (forward references for jumps).

/// A handle to a basic block within the current function.
/// Block handles are stable, they don't change as more blocks are added.
export record BlockId: Copy(u32);

/// Internal block state during construction.
///
/// The key invariants:
///
/// - A block is "open" if it has no terminator; instructions can be added.
/// - A block is "sealed" when all predecessor edges are known.
/// - Sealing resolves the predecessor arguments for block parameters.
///
/// This differs from the final [`il::Block`] which is immutable and fully formed.
record BlockData: Copy {
    /// Block label for debugging and IL printing.
    label: *[u8],
    /// Block parameters for merging values at control flow joins. These
    /// receive values from predecessor edges when control flow merges.
    params: *unsafe mut [il::Param],
    /// Variable ids in parameter order. Before sealing, these are the variables
    /// whose predecessor arguments must be resolved.
    paramVars: *unsafe mut [u32],
    /// Instructions accumulated so far. The last instruction should eventually
    /// be a terminator.
    instrs: *unsafe mut [il::Instr],
    /// Debug source locations, one per instruction. Only populated when
    /// debug info is enabled.
    locs: *unsafe mut [il::SrcLoc],
    /// Predecessor block ids. Used for SSA construction to propagate values
    /// from predecessors when a variable is used before being defined locally.
    preds: *unsafe mut [u32],
    /// The current SSA value of each variable in this block. Indexed by variable
    /// id. A `nil` means the variable wasn't assigned in this block. Updated by
    /// [`defVar`], queried by [`useVarInBlock`].
    vars: *unsafe mut [?il::Val],
    /// Sealing state. Once sealed, all predecessors are known and we can resolve
    /// variable uses that need to pull values from predecessors.
    sealState: Sealed,
    /// Loop nesting depth when this block was created.
    loopDepth: u32,
}

/// Block sealing state for SSA construction.
///
/// A block is "unsealed" while its predecessors are still being discovered.
/// Before sealing, `BlockData.paramVars` records each variable that needs a
/// block parameter. Once all predecessors are known, the block is sealed and
/// those parameters are resolved via [`resolveBlockArgs`].
union Sealed: Copy {
    /// Block is unsealed; predecessors may still be added.
    No,
    /// Block is sealed; all predecessors are known.
    Yes,
}

///////////////////////////////////
// Loop and Control Flow Context //
///////////////////////////////////

/// Context for break/continue statements within a loop.
/// Each nested loop pushes a new context onto the loop stack.
export record LoopCtx: Copy {
    /// Where `break` should transfer control (the loop's exit block).
    breakTarget: BlockId,
    /// Where `continue` should transfer control.
    continueTarget: ?BlockId,
}

/// Logical operator.
union LogicalOp: Copy { And, Or }

/// Iterator state for for-loop lowering.
union ForIter: Copy {
    /// Range iterator: `for i in 0..n`.
    Range {
        valVar: Var,
        indexVar: ?Var,
        endVal: il::Val,
        valType: il::Type,
        unsigned: bool,
    },
    /// Collection iterator: `for elem in slice`.
    Collection {
        valVar: ?Var,
        idxVar: Var,
        dataReg: il::Reg,
        lengthVal: il::Val,
        elemType: *resolver::Type,
    },
}

//////////////////////////////
// Pattern Matching Support //
//////////////////////////////

// Match expressions are lowered by evaluating the subject once, then emitting
// a chain of comparison-and-branch sequences for each arm. The algorithm
// handles several subject types specially:
//
// - Optional pointers: compared against `null`.
// - Optional aggregates: tag checked then payload extracted.
// - Unions: tag compared against variant indices.

/// Cached information about a match subject. Computed once and reused across
/// all arms to avoid redundant lowering and type queries.
record MatchSubject: Copy {
    /// The lowered subject value.
    val: il::Val,
    /// Source-level type from the resolver.
    type: resolver::Type,
    /// IL-level type for code generation.
    ilType: il::Type,
    /// The type that binding arms should use. For optionals, this is the
    /// inner type; for regular values, it's the same as `type`.
    bindType: resolver::Type,
    /// Classification of how the subject should be compared and destructured.
    kind: MatchSubjectKind,
    /// How bindings are created: by value, or by reference.
    by: resolver::MatchBy,
}

/// Classifies a match subject by how it should be compared and destructured.
union MatchSubjectKind: Copy {
    /// Regular value: direct equality comparison.
    Regular,
    /// Optional with null pointer optimization: `?*T`, `?*[T]`.
    OptionalPtr,
    /// Optional aggregate `?T`: tagged union with payload.
    OptionalAggregate,
    /// Union type: tag compared against variant indices.
    Union(resolver::UnionType),
}

/// Determine the kind of a match subject from its type.
unsafe fn matchSubjectKind(type: resolver::Type) -> MatchSubjectKind {
    if resolver::isOptionalPointer(type) {
        return MatchSubjectKind::OptionalPtr;
    }
    if resolver::isOptionalAggregate(type) {
        return MatchSubjectKind::OptionalAggregate;
    }
    if let info = unionInfoFromType(type) {
        return MatchSubjectKind::Union(info);
    }
    return MatchSubjectKind::Regular;
}

//////////////////////////
// Field Access Support //
//////////////////////////

/// Result of resolving a place expression to a memory location.
record FieldRef: Copy {
    /// Base pointer register (points to the container).
    base: il::Reg,
    /// Byte offset of the value within the container.
    offset: i32,
    /// Type of the value held at that location.
    fieldType: resolver::Type,
}

/// Result of computing an element pointer for array/slice subscript operations.
/// Used by [`lowerElemPtr`] to return both the element address register and
/// the element type for subsequent load or address-of operations.
record ElemPtrResult: Copy {
    /// Register holding the computed element address.
    elemReg: il::Reg,
    /// Source-level type of the element.
    elemType: resolver::Type,
}

/// Result of resolving a slice range to a data pointer and element count.
record SliceRangeResult: Copy {
    /// Register holding the first element address.
    dataReg: il::Reg,
    /// Number of elements in the range.
    count: il::Val,
    /// Available capacity from the first element address.
    capacity: il::Val,
}

/////////////////////////////
// Function Lowering State //
/////////////////////////////

/// Per-function lowering state. Created fresh for each function and contains
/// all the mutable state needed during function body lowering.
record FnLowerer: Copy {
    /// Reference to the module-level lowerer.
    low: *unsafe mut Lowerer,
    /// Allocator for IL allocations.
    allocator: alloc::Allocator,
    /// Type signature of the function being lowered.
    fnType: *resolver::FnType,
    /// Number of SSA variable slots required for each block.
    localCount: u32,
    /// Function name, used as prefix for generated data symbols.
    fnName: *[u8],

    // ~ SSA variable tracking ~ //

    /// Metadata (name, type, mutability) for each variable. Indexed by variable
    /// id. Doesn't change after declaration. For the SSA value of a variable in
    /// a specific block, see [`BlockData::vars`].
    vars: *unsafe mut [VarData],
    /// Parameter-to-variable bindings, initialized in the entry block.
    params: *unsafe mut [FnParamBinding],

    // ~ Basic block management ~ //

    /// Block storage array, indexed by block id.
    blockData: *unsafe mut [BlockData],
    /// The entry block for this function.
    entryBlock: ?BlockId,
    /// The block currently receiving new instructions.
    currentBlock: ?BlockId,

    // ~ Loop management ~ //

    /// Stack of loop contexts for break/continue resolution.
    loopStack: *unsafe mut [LoopCtx],
    /// Current nesting depth (index into loopStack).
    loopDepth: u32,

    // ~ Counters ~ //

    /// Counter for generating unique block labels like `then#0`, `loop#1`, etc.
    labelCounter: u32,
    /// Counter for generating unique data names within this function.
    /// Each literal gets a name like `fnName$literal$N`.
    dataCounter: u32,
    /// Counter for generating SSA register numbers.
    regCounter: u32,
    /// When the function returns an aggregate type, the caller passes a hidden
    /// pointer as the first parameter. The callee writes the return value into
    /// this buffer and returns the pointer.
    returnReg: ?il::Reg,
    /// Whether the function is a leaf.
    isLeaf: bool,

    // ~ Debug info ~ //

    /// Current debug source location, set when processing AST nodes.
    srcLoc: il::SrcLoc,
}

/////////////////////////////////
// Module Lowering Entry Point //
/////////////////////////////////

/// Lower a complete module AST to an IL program.
///
/// This is the main entry point for the lowering pass. It:
/// 1. Counts functions to preallocate the output array.
/// 2. Iterates over top-level declarations, lowering each.
/// 3. Returns the complete IL program with functions and data section.
///
/// The resolver must have already processed the AST -- we rely on its type
/// annotations, symbol table, and constant evaluations.
export unsafe fn lower(
    res: &resolver::Resolver,
    root: *ast::Node,
    pkgName: *[u8],
    arena: &mut alloc::Arena
) -> il::Program throws (LowerError) {
    let mut low = Lowerer {
        arena: (&mut *arena) as *unsafe mut alloc::Arena,
        fnArena: (&mut *arena) as *unsafe mut alloc::Arena,
        allocator: alloc::arenaAllocator(arena),
        resolver: res as *unsafe resolver::Resolver,
        moduleGraph: nil,
        pkgName,
        currentMod: nil,
        data: &mut [],
        output: FnOutput::Accumulate(&mut []),
        fnSyms: &mut [],
        errTags: &mut [],
        errTagCounter: 1,
        options: LowerOptions { debug: false, buildTest: false },
    };
    try lowerDecls(&mut low, root, true);

    return finalize(low);
}

/////////////////////////////////
// Multi-Module Lowering API   //
/////////////////////////////////

/// Create a lowerer for multi-module compilation.
/// The resolver, graph, and both arenas must outlive the returned lowerer.
export unsafe fn lowerer(
    res: *unsafe resolver::Resolver,
    graph: &module::ModuleGraph,
    pkgName: *[u8],
    arena: *unsafe mut alloc::Arena,
    fnArena: *unsafe mut alloc::Arena,
    options: LowerOptions
) -> Lowerer {
    return Lowerer {
        arena,
        fnArena,
        allocator: alloc::arenaAllocator(arena),
        resolver: res,
        moduleGraph: graph as *unsafe module::ModuleGraph,
        pkgName,
        currentMod: nil,
        data: &mut [],
        output: FnOutput::Accumulate(&mut []),
        fnSyms: &mut [],
        errTags: &mut [],
        errTagCounter: 1,
        options,
    };
}

/// Lower a module's AST into the lowerer.
/// Call this for each module in the package, then use `finalize` to get the program.
export unsafe fn lowerModule(
    low: &mut Lowerer,
    moduleId: u16,
    root: *ast::Node,
    isRoot: bool
) throws (LowerError) {
    set low.currentMod = moduleId;
    try lowerDecls(low, root, isRoot);
}

/// Lower all top-level declarations in a block.
unsafe fn lowerDecls(low: &mut Lowerer, root: *ast::Node, isRoot: bool) throws (LowerError) {
    let case ast::NodeValue::Block(block) = root.value else {
        throw LowerError::ExpectedBlock(root);
    };
    let stmtsList = block.statements;

    for node in stmtsList {
        match node.value {
            case ast::NodeValue::FnDecl(decl) => {
                if let f = try lowerFnDecl(low, node, decl) {
                    let role = lowerFnRole(isRoot, decl.attrs);
                    emitFunction(low, f, role);
                }
            }
            case ast::NodeValue::ConstDecl(decl) => {
                try lowerDataDecl(low, node, decl.value, true);
            }
            case ast::NodeValue::StaticDecl(decl) => {
                try lowerDataDecl(low, node, decl.value, false);
            }
            case ast::NodeValue::InstanceDecl { traitName, targetType, methods } => {
                try lowerInstanceDecl(low, node, traitName, targetType, methods);
            }
            case ast::NodeValue::MethodDecl { name, receiverName, receiverType, sig, body, .. } => {
                if let f = try lowerMethodDecl(low, node, name, receiverName, sig, body) {
                    emitFunction(low, f, FnRole::Normal);
                }
            }
            else => {},
        }
    }
}

/// Consume the lowerer and publish its global data as an immutable array.
export fn finalize(low: Lowerer) -> il::Program {
    let case Lowerer { data, output, .. } = low
        else panic "expected lowerer";
    match output {
        case FnOutput::Accumulate(accumulated) => {
            return il::Program {
                data,
                fns: accumulated,
            };
        }
        case FnOutput::Stream(_) => {
            panic "finalize: cannot finalize streaming lowerer";
        }
    }
}

/////////////////////////////////
// Qualified Name Construction //
/////////////////////////////////

/// Get module path segments for the current or specified module.
/// Returns empty slice if no module graph or module not found.
unsafe fn getModulePath(self: &mut Lowerer, modId: ?u16) -> *[*[u8]] {
    let graph = self.moduleGraph else {
        return &[];
    };
    let mut id = modId;
    if id == nil {
        set id = self.currentMod;
    }
    let actualId = id else {
        return &[];
    };
    let entry = module::get(graph, actualId) else {
        return &[];
    };
    return module::moduleQualifiedPath(entry);
}

/// Build a qualified name string for a symbol.
/// If `modId` is nil, uses current module.
unsafe fn qualifyName(self: &mut Lowerer, modId: ?u16, name: *[u8]) -> *[u8] {
    let path = getModulePath(self, modId);
    if path.len == 0 {
        return name;
    }
    return il::formatQualifiedName(self.arena, path, name);
}

/// Register a function symbol with its qualified name.
/// Called when lowering function declarations, so cross-package calls can find
/// the function by name.
fn registerFnSym(self: &mut Lowerer, sym: *unsafe resolver::Symbol, qualName: *[u8]) {
    self.fnSyms.append(FnSymEntry { sym, qualName }, self.allocator);
}

/// Look up a function's qualified name by its symbol.
/// Returns `nil` if the symbol wasn't registered (e.g. callee's module is not yet lowered).
// TODO: This is kind of dubious as an optimization, if it depends on the order
// in which modules are lowered.
// TODO: Use a hash table here?
unsafe fn lookupFnSym(self: &Lowerer, sym: *unsafe resolver::Symbol) -> ?*[u8] {
    for entry in &self.fnSyms[..] {
        if entry.sym == sym {
            return entry.qualName;
        }
    }
    return nil;
}

/// Set the package context for lowering.
/// Called before lowering each package.
/// The graph must outlive later uses of the lowerer.
export unsafe fn setPackage(self: &mut Lowerer, graph: &module::ModuleGraph, pkgName: *[u8]) {
    set self.moduleGraph = graph as *unsafe module::ModuleGraph;
    set self.pkgName = pkgName;
    set self.currentMod = nil;
}

/// Create a new function lowerer for a given function type and name.
unsafe fn fnLowerer(
    self: &mut Lowerer,
    node: *ast::Node,
    fnType: *resolver::FnType,
    qualName: *[u8]
) -> FnLowerer {
    let loopStack = try! alloc::allocRawSlice(self.fnArena, @sizeOf(LoopCtx), @alignOf(LoopCtx), MAX_LOOP_DEPTH) as *unsafe mut [LoopCtx];

    let mut fnLow = FnLowerer {
        low: (&mut *self) as *unsafe mut Lowerer,
        allocator: alloc::arenaAllocator(self.fnArena),
        fnType: fnType,
        localCount: resolver::nodeData(self.resolver, node).localCount,
        fnName: qualName,
        vars: &mut [],
        params: &mut [],
        blockData: &mut [],
        entryBlock: nil,
        currentBlock: nil,
        loopStack,
        loopDepth: 0,
        labelCounter: 0,
        dataCounter: 0,
        regCounter: 0,
        returnReg: nil,
        isLeaf: true,
        srcLoc: undefined,
    };
    if self.options.debug {
        let modId = self.currentMod else {
            panic "fnLowerer: debug enabled but no current module";
        };
        set fnLow.srcLoc = il::SrcLoc {
            moduleId: modId,
            offset: node.span.offset,
        };
    }
    return fnLow;
}

/// Lower a function declaration.
///
/// This sets up the per-function lowering state, processes parameters,
/// then lowers the function body into a CFG of basic blocks.
///
/// For throwing functions, the return type is a result aggregate
/// rather than the declared return type.
unsafe fn lowerFnDecl(self: &mut Lowerer, node: *ast::Node, decl: ast::FnDecl) -> ?*unsafe il::Fn throws (LowerError) {
    if not shouldLowerFn(&decl, self.options.buildTest) {
        return nil;
    }
    let case ast::NodeValue::Ident(name) = decl.name.value else {
        throw LowerError::ExpectedIdentifier;
    };
    let data = resolver::nodeData(self.resolver, node);
    let case resolver::Type::Fn(fnType) = data.ty else {
        throw LowerError::ExpectedFunction;
    };
    let isExtern = checkAttr(decl.attrs, ast::Attribute::Extern);

    // Build qualified function name for multi-module compilation.
    let qualName = qualifyName(self, nil, name);

    // Register function symbol for cross-package call resolution.
    if let sym = data.sym {
        registerFnSym(self, sym, qualName);
    }
    let mut fnLow = fnLowerer(self, node, fnType, qualName);

    // If the function returns an aggregate or is throwing, prepend a hidden
    // return parameter. The caller allocates the buffer and passes it
    // as the first argument; the callee writes the return value into it.
    if requiresReturnParam(fnType) and not isExtern {
        set fnLow.returnReg = nextReg(&mut fnLow);
    }
    let lowParams = try lowerParams(&mut fnLow, *fnType, decl.sig.params, nil);
    let func = try! alloc::allocRaw(self.fnArena, @sizeOf(il::Fn), @alignOf(il::Fn)) as *unsafe mut il::Fn;

    set *func = il::Fn {
        name: qualName,
        params: lowParams,
        returnType: undefined,
        isExtern,
        isLeaf: true,
        blocks: &[],
    };
    // Throwing functions return a result aggregate (word-sized pointer).
    // TODO: The resolver should set an appropriate type that takes into account
    //       the throws list. It shouldn't set the return type to the "success"
    //       value only.
    if fnType.throwList.len > 0 {
        set func.returnType = il::Type::W64;
    } else {
        set func.returnType = ilType(self, *fnType.returnType);
    }
    let body = decl.body else {
        // Extern functions have no body.
        assert isExtern;
        return func;
    };
    set func.blocks = try lowerFnBody(&mut fnLow, body);
    set func.isLeaf = fnLow.isLeaf;

    return func;
}

/// Build a qualified name of the form "Type::method".
unsafe fn instanceMethodName(self: &mut Lowerer, modId: ?u16, typeName: *[u8], methodName: *[u8]) -> *[u8] {
    let sepLen: u32 = 2; // "::"
    let totalLen = typeName.len + sepLen + methodName.len;
    let buf = try! alloc::allocSlice(self.arena, 1, 1, totalLen) as *mut [u8];
    let mut pos: u32 = 0;

    set pos += try! mem::copy(&mut buf[pos..], typeName);
    set pos += try! mem::copy(&mut buf[pos..], "::");
    set pos += try! mem::copy(&mut buf[pos..], methodName);
    assert pos == totalLen;

    return qualifyName(self, modId, &buf[..totalLen]);
}

/// Build a v-table data name of the form "vtable::Type::Trait".
unsafe fn vtableName(self: &mut Lowerer, modId: ?u16, typeName: *[u8], traitName: *[u8]) -> *[u8] {
    let prefix = "vtable::";
    let sepLen: u32 = 2; // "::"
    let totalLen = prefix.len + typeName.len + sepLen + traitName.len;
    let buf = try! alloc::allocSlice(self.arena, 1, 1, totalLen) as *mut [u8];
    let mut pos: u32 = 0;

    set pos += try! mem::copy(&mut buf[pos..], prefix);
    set pos += try! mem::copy(&mut buf[pos..], typeName);
    set pos += try! mem::copy(&mut buf[pos..], "::");
    set pos += try! mem::copy(&mut buf[pos..], traitName);
    assert pos == totalLen;

    return qualifyName(self, modId, &buf[..totalLen]);
}

/// Lower an instance declaration (`instance Trait for Type { ... }`).
///
/// Each method in the instance block is lowered as a standalone function
/// with a qualified name of the form `Type::method`. A read-only v-table
/// data record is emitted containing pointers to these functions, ordered
/// by the trait's method indices. The v-table is later referenced when
/// constructing trait objects for dynamic dispatch.
unsafe fn lowerInstanceDecl(
    self: &mut Lowerer,
    node: *ast::Node,
    traitNameNode: *ast::Node,
    targetTypeNode: *ast::Node,
    methods: *[*ast::Node]
) throws (LowerError) {
    // Look up the trait and type from the resolver.
    let traitSym = resolver::nodeData(self.resolver, traitNameNode).sym
        else throw LowerError::MissingSymbol(traitNameNode);
    let case resolver::SymbolData::Trait(traitInfo) = traitSym.data
        else throw LowerError::MissingMetadata;
    let typeSym = resolver::nodeData(self.resolver, targetTypeNode).sym
        else throw LowerError::MissingSymbol(targetTypeNode);

    let tName = traitSym.name;
    let typeName = typeSym.name;

    // Lower each instance method as a regular function.
    // Collect qualified names for the v-table. Empty entries are filled
    // later from inherited supertrait methods.
    let mut methodNames: [*[u8]; ast::MAX_TRAIT_METHODS] = undefined;
    let mut methodNameSet: [bool; ast::MAX_TRAIT_METHODS] = [false; ast::MAX_TRAIT_METHODS];

    for methodNode in methods {
        let case ast::NodeValue::MethodDecl {
            name, receiverName, receiverType, sig, body, ..
        } = methodNode.value else continue;

        let case ast::NodeValue::Ident(mName) = name.value else {
            throw LowerError::ExpectedIdentifier;
        };
        let qualName = instanceMethodName(self, nil, typeName, mName);
        let func = try lowerMethod(self, methodNode, qualName, receiverName, sig, body)
            else continue;
        emitFunction(self, func, FnRole::Normal);

        let method = resolver::findTraitMethod(traitInfo, mName)
            else panic "lowerInstanceDecl: method not found in trait";

        set methodNames[method.index] = qualName;
        set methodNameSet[method.index] = true;
    }

    // Fill inherited method slots from supertraits.
    // These methods were already lowered as part of the supertrait instance
    // declarations and use the same `Type::method` qualified name.
    for method, i in traitInfo.methods {
        if not methodNameSet[i] {
            set methodNames[i] = instanceMethodName(self, nil, typeName, method.name);
        }
    }

    // Create v-table in data section, used for dynamic dispatch.
    let vName = vtableName(self, nil, typeName, tName);
    let values = try! alloc::allocSlice(
        self.arena, @sizeOf(il::DataValue), @alignOf(il::DataValue), traitInfo.methods.len as u32
    ) as *mut [il::DataValue];

    for i in 0..traitInfo.methods.len {
        set values[i] = il::DataValue {
            item: il::DataItem::Fn(methodNames[i]),
            count: 1,
        };
    }
    self.data.append(il::Data {
        name: vName,
        size: traitInfo.methods.len as u32 * resolver::PTR_SIZE,
        alignment: resolver::PTR_SIZE,
        readOnly: true,
        isZeroInit: false,
        values: &values[..traitInfo.methods.len as u32],
    }, self.allocator);
}

/// Lower a method node into an IL function with the given qualified name.
/// Shared by both instance methods and standalone methods.
unsafe fn lowerMethod(
    self: &mut Lowerer,
    node: *ast::Node,
    qualName: *[u8],
    receiverName: *ast::Node,
    sig: ast::FnSig,
    body: *ast::Node,
) -> ?*unsafe il::Fn throws (LowerError) {
    let data = resolver::nodeData(self.resolver, node);
    let case resolver::Type::Fn(fnType) = data.ty else {
        throw LowerError::ExpectedFunction;
    };
    let sym = data.sym else throw LowerError::MissingSymbol(node);
    registerFnSym(self, sym, qualName);

    let mut fnLow = fnLowerer(self, node, fnType, qualName);
    if requiresReturnParam(fnType) {
        set fnLow.returnReg = nextReg(&mut fnLow);
    }
    let lowParams = try lowerParams(&mut fnLow, *fnType, sig.params, receiverName);
    let func = try! alloc::allocRaw(self.fnArena, @sizeOf(il::Fn), @alignOf(il::Fn)) as *unsafe mut il::Fn;

    set *func = il::Fn {
        name: qualName,
        params: lowParams,
        returnType: ilType(self, *fnType.returnType),
        isExtern: false,
        isLeaf: true,
        blocks: &[],
    };
    if fnType.throwList.len > 0 {
        set func.returnType = il::Type::W64;
    }
    set func.blocks = try lowerFnBody(&mut fnLow, body);
    set func.isLeaf = fnLow.isLeaf;

    return func;
}

/// Lower a standalone method declaration.
/// Produces a function with qualified name `Type::method`.
unsafe fn lowerMethodDecl(
    self: &mut Lowerer,
    node: *ast::Node,
    name: *ast::Node,
    receiverName: *ast::Node,
    sig: ast::FnSig,
    body: *ast::Node,
) -> ?*unsafe il::Fn throws (LowerError) {
    let sym = resolver::nodeData(self.resolver, node).sym
        else throw LowerError::MissingSymbol(node);
    let case ast::NodeValue::Ident(mName) = name.value
        else throw LowerError::ExpectedIdentifier;
    let me = resolver::findMethodBySymbol(self.resolver, sym)
        else throw LowerError::MissingMetadata;
    let qualName = instanceMethodName(self, nil, me.concreteTypeName, mName);

    return try lowerMethod(self, node, qualName, receiverName, sig, body);
}

/// Check if a function should be lowered.
fn shouldLowerFn(decl: &ast::FnDecl, buildTest: bool) -> bool {
    if checkAttr(decl.attrs, ast::Attribute::Test) {
        return buildTest;
    }
    return true;
}

/// Check if a specific attribute is present in the attribute set.
fn checkAttr(attrs: ?ast::Attributes, attr: ast::Attribute) -> bool {
    if let a = attrs {
        return ast::attributesContains(&a, attr);
    }
    return false;
}

/// Create a label with a numeric suffix, eg. `@base0`.
/// This ensures unique labels like `@then0`, `@then1`, etc.
unsafe fn labelWithSuffix(self: &mut FnLowerer, base: *[u8], suffix: u32) -> *[u8] throws (LowerError) {
    let mut digits: [u8; fmt::U32_STR_LEN] = undefined;
    let start = fmt::formatU32(suffix, &mut digits[..]);
    let totalLen = base.len + digits.len - start;
    let buf = try! alloc::allocSlice(self.low.fnArena, 1, 1, totalLen) as *mut [u8];

    try! mem::copy(&mut buf[..base.len], base);
    try! mem::copy(&mut buf[base.len..totalLen], &digits[start..]);

    return &buf[..totalLen];
}

/// Generate a unique label by appending the global counter to the base.
unsafe fn nextLabel(self: &mut FnLowerer, base: *[u8]) -> *[u8] throws (LowerError) {
    let idx = self.labelCounter;
    set self.labelCounter += 1;

    return try labelWithSuffix(self, base, idx);
}

///////////////////////////////
// Data Section Construction //
///////////////////////////////

// Functions for building the data section from constant/static
// declarations and inline literals.

/// Convert a constant integer payload to a signed 64-bit value.
fn constIntToI64(intVal: resolver::ConstInt) -> i64 {
    return (0 - intVal.magnitude as i64) if intVal.negative else intVal.magnitude as i64;
}

/// Convert a scalar constant value to an i64.
/// String constants are not handled here and should be checked before calling.
/// Panics if a non-scalar value is passed.
fn constToScalar(val: resolver::ConstValue) -> i64 {
    match val {
        case resolver::ConstValue::Bool(b) => return 1 if b else 0,
        case resolver::ConstValue::Char(c) => return c as i64,
        case resolver::ConstValue::Int(i) => return constIntToI64(i),
        else => panic,
    }
}

/// Convert a constant value to an IL value.
unsafe fn constValueToVal(self: &mut FnLowerer, val: resolver::ConstValue, node: *ast::Node) -> il::Val throws (LowerError) {
    if let case resolver::ConstValue::String(s) = val {
        return try lowerStringLit(self, node, s);
    }
    return il::Val::Imm(constToScalar(val));
}

/// Convert a resolver constant value to an IL data initializer item.
unsafe fn constValueToDataItem(self: &mut Lowerer, val: resolver::ConstValue, typ: resolver::Type) -> il::DataItem {
    if let case resolver::ConstValue::String(s) = val {
        return il::DataItem::Str(s);
    }
    // Bool and char are byte-sized; integer uses the declared type.
    let mut irTyp = il::Type::W8;
    if let case resolver::ConstValue::Int(_) = val {
        set irTyp = ilType(self, typ);
    }
    return il::DataItem::Val { typ: irTyp, val: constToScalar(val) };
}

/// Lower scalar-like constant nodes into data values, including fallback handling
/// for void-variant tags and slice string initializers.
unsafe fn lowerConstScalarDataInto(
    self: &mut Lowerer,
    node: *ast::Node,
    ty: resolver::Type,
    dataPrefix: *[u8],
    b: &mut DataValueBuilder
) throws (LowerError) {
    let val = resolver::constValueEntry(self.resolver, node) else {
        if let idx = voidVariantIndex(self.resolver, node) {
            dataBuilderPush(b, il::DataValue {
                item: il::DataItem::Val { typ: il::Type::W8, val: idx },
                count: 1
            });
            return;
        }
        throw LowerError::MissingConst(node);
    };

    if let case resolver::ConstValue::String(s) = val {
        if let case resolver::Type::Slice { .. } = ty {
            let strSym = try getOrCreateStringData(self, s, dataPrefix);
            dataSliceHeader(b, strSym, s.len);
            return;
        }
    }
    dataBuilderPush(b, il::DataValue {
        item: constValueToDataItem(self, val, ty),
        count: 1
    });
}

/// Lower a constant or static declaration to the data section.
unsafe fn lowerDataDecl(
    self: &mut Lowerer,
    node: *ast::Node,
    value: *ast::Node,
    readOnly: bool
) throws (LowerError) {
    let data = resolver::nodeData(self.resolver, node);
    let sym = data.sym else {
        throw LowerError::MissingSymbol(node);
    };
    if data.ty == resolver::Type::Unknown {
        throw LowerError::MissingType(node);
    }
    let layout = resolver::getTypeLayout(data.ty);
    let qualName = qualifyName(self, nil, sym.name);
    let mut b = dataBuilder(self.allocator);
    try lowerConstDataInto(self, value, data.ty, layout.size, qualName, &mut b);
    let result = dataBuilderFinish(b);

    self.data.append(il::Data {
        name: qualName,
        size: layout.size,
        alignment: layout.alignment,
        readOnly,
        isZeroInit: not readOnly and result.zeroInit,
        values: result.values,
    }, self.allocator);
}

/// Emit the in-memory representation of a slice header: `{ ptr, len, cap }`.
fn dataSliceHeader(b: &mut DataValueBuilder, dataSym: *[u8], len: u32) {
    dataBuilderPush(b, il::DataValue {
        item: il::DataItem::Sym(dataSym),
        count: 1
    });
    dataBuilderPush(b, il::DataValue {
        item: il::DataItem::Val {
            typ: il::Type::W32,
            val: len as i64
        },
        count: 1
    });
    dataBuilderPush(b, il::DataValue {
        item: il::DataItem::Val {
            typ: il::Type::W32,
            val: len as i64
        },
        count: 1
    });
}

/// Lower a compile-time `&[...]` expression to a concrete slice header.
unsafe fn lowerConstAddressSliceInto(
    self: &mut Lowerer,
    addr: ast::AddressOf,
    ty: resolver::Type,
    dataPrefix: *[u8],
    b: &mut DataValueBuilder
) throws (LowerError) {
    let case resolver::Type::Slice { mutable, .. } = ty
        else throw LowerError::ExpectedSliceOrArray;
    let targetTy = resolver::typeFor(self.resolver, addr.target)
        else throw LowerError::MissingType(addr.target);
    let case resolver::Type::Array(arrInfo) = targetTy
        else throw LowerError::ExpectedArray;

    let mut nested = dataBuilder(self.allocator);
    let layout = resolver::getTypeLayout(targetTy);
    try lowerConstDataInto(self, addr.target, targetTy, layout.size, dataPrefix, &mut nested);

    let backing = dataBuilderFinish(nested);
    let readOnly = not mutable;
    let mut dataName: *[u8] = undefined;
    if readOnly {
        if let found = findConstData(self, backing.values, layout.alignment) {
            set dataName = found;
        } else {
            set dataName = try pushDeclData(self, layout.size, layout.alignment, readOnly, backing.values, dataPrefix);
        }
    } else {
        set dataName = try pushDeclData(self, layout.size, layout.alignment, readOnly, backing.values, dataPrefix);
    }
    dataSliceHeader(b, dataName, arrInfo.length);
}

/// Lower a constant expression payload into a builder without slot padding.
/// Compute the type layout when nil or undefined data needs a byte count.
unsafe fn lowerConstDataPayloadInto(
    self: &mut Lowerer,
    node: *ast::Node,
    ty: resolver::Type,
    dataPrefix: *[u8],
    b: &mut DataValueBuilder
) throws (LowerError) {
    // Optional coercions define the tag and payload layout of constant values.
    if let case resolver::Type::Optional(inner) = ty {
        if let coercion = resolver::coercionFor(self.resolver, node) {
            if let case resolver::Coercion::OptionalLift(target) = coercion {
                if resolver::typesEqual(target, ty) {
                    if resolver::isNullableType(*inner) {
                        try lowerConstDataPayloadInto(self, node, *inner, dataPrefix, b);
                    } else {
                        let layout = resolver::getTypeLayout(ty);
                        let offset = resolver::getOptionalValOffset(*inner);
                        dataBuilderPush(b, il::DataValue {
                            item: il::DataItem::Val { typ: il::Type::W8, val: 1 }, count: 1,
                        });
                        if offset > 1 {
                            dataBuilderPush(b, il::DataValue { item: il::DataItem::Undef, count: offset - 1 });
                        }
                        try lowerConstDataInto(self, node, *inner, layout.size - offset, dataPrefix, b);
                    }
                    return;
                }
            }
        }
    }
    // Function pointer references in constant data.
    if let case resolver::Type::Fn(_) = ty {
        let sym = resolver::nodeData(self.resolver, node).sym
            else throw LowerError::MissingSymbol(node);
        let modId = resolver::moduleIdForSymbol(self.resolver, sym);
        let qualName = qualifyName(self, modId, sym.name);
        dataBuilderPush(b, il::DataValue {
            item: il::DataItem::Fn(qualName), count: 1,
        });
        return;
    }
    // In constant data, a void variant of a mixed union still occupies the
    // full tagged-union slot. Emitting only the tag corrupts following fields.
    if let sym = resolver::nodeData(self.resolver, node).sym {
        if let case resolver::SymbolData::Variant { type: resolver::Type::Void, .. } = sym.data {
            if let case resolver::Type::Nominal(resolver::NominalType::Union(_)) = ty {
                try lowerConstUnionVariantInto(self, node, sym, ty, &mut [], dataPrefix, b);
                return;
            }
        }
    }
    match node.value {
        case ast::NodeValue::Nil => {
            let case resolver::Type::Optional(_) = ty else throw LowerError::NilInNonOptional;
            let layout = resolver::getTypeLayout(ty);
            dataBuilderPush(b, il::DataValue {
                item: il::DataItem::Val { typ: il::Type::W8, val: 0 }, count: layout.size,
            });
        },
        case ast::NodeValue::Undef => {
            let layout = resolver::getTypeLayout(ty);
            dataBuilderPush(b, il::DataValue {
                item: il::DataItem::Undef,
                count: layout.size
            });
        }
        case ast::NodeValue::ArrayLit(elems) =>
            try lowerConstArrayLitInto(self, elems, ty, dataPrefix, b),
        case ast::NodeValue::ArrayRepeatLit(repeat) =>
            try lowerConstArrayRepeatInto(self, repeat, ty, dataPrefix, b),
        case ast::NodeValue::RecordLit(recLit) =>
            try lowerConstRecordLitInto(self, node, recLit, ty, dataPrefix, b),
        case ast::NodeValue::Call(call) => {
            let calleeSym = resolver::nodeData(self.resolver, call.callee).sym
                else throw LowerError::MissingSymbol(call.callee);
            match calleeSym.data {
                case resolver::SymbolData::Variant { .. } =>
                    try lowerConstUnionVariantInto(self, node, calleeSym, ty, call.args, dataPrefix, b),
                case resolver::SymbolData::Type(resolver::NominalType::Record(recInfo)) => {
                    try lowerConstRecordCtorInto(self, call.args, recInfo, dataPrefix, b);
                }
                else => throw LowerError::MissingConst(node)
            }
        }
        case ast::NodeValue::AddressOf(addr) => {
            try lowerConstAddressSliceInto(self, addr, ty, dataPrefix, b);
        }
        case ast::NodeValue::Ident(_) => {
            // Identifier referencing a constant.
            let sym = resolver::nodeData(self.resolver, node).sym
                else throw LowerError::MissingSymbol(node);
            let case ast::NodeValue::ConstDecl(decl) = sym.node.value
                else throw LowerError::MissingConst(node);

            try lowerConstDataPayloadInto(self, decl.value, ty, dataPrefix, b);
        },
        case ast::NodeValue::ScopeAccess(_) => {
            let sym = resolver::nodeData(self.resolver, node).sym
                else throw LowerError::MissingSymbol(node);
            if let case ast::NodeValue::ConstDecl(decl) = sym.node.value {
                try lowerConstDataPayloadInto(self, decl.value, ty, dataPrefix, b);
            } else {
                try lowerConstScalarDataInto(self, node, ty, dataPrefix, b);
            }
        }
        else => {
            // Scalar values: integers, bools, strings, void union variants, etc.
            try lowerConstScalarDataInto(self, node, ty, dataPrefix, b);
        }
    }
}

/// Lower a constant expression into a builder, padding to the given slot size.
unsafe fn lowerConstDataInto(
    self: &mut Lowerer,
    node: *ast::Node,
    ty: resolver::Type,
    slotSize: u32,
    dataPrefix: *[u8],
    b: &mut DataValueBuilder
) throws (LowerError) {
    let layout = resolver::getTypeLayout(ty);
    try lowerConstDataPayloadInto(self, node, ty, dataPrefix, b);
    // Pad to fill the enclosing slot.
    let padding = slotSize - layout.size;
    if padding > 0 {
        dataBuilderPush(b, il::DataValue { item: il::DataItem::Undef, count: padding });
    }
}

/// Flatten a constant array literal `[a, b, c]` into a builder.
/// Each element payload fills its type size; no extra slot padding is needed.
unsafe fn lowerConstArrayLitInto(
    self: &mut Lowerer,
    elems: *[*ast::Node],
    ty: resolver::Type,
    dataPrefix: *[u8],
    b: &mut DataValueBuilder
) throws (LowerError) {
    let case resolver::Type::Array(arrInfo) = ty
        else throw LowerError::ExpectedArray;
    let elemTy = *arrInfo.item;

    for elem in elems {
        try lowerConstDataPayloadInto(self, elem, elemTy, dataPrefix, b);
    }
}

/// Build data values for a constant array repeat literal `[item; count]`.
/// Repeat element payloads without extra slot padding. Undefined data uses
/// the element layout to compute the total byte count.
unsafe fn lowerConstArrayRepeatInto(
    self: &mut Lowerer,
    repeat: ast::ArrayRepeatLit,
    ty: resolver::Type,
    dataPrefix: *[u8],
    b: &mut DataValueBuilder
) throws (LowerError) {
    let case resolver::Type::Array(arrInfo) = ty
        else throw LowerError::ExpectedArray;
    let length = arrInfo.length;
    let elemTy = *arrInfo.item;

    if let case resolver::Type::Optional(_) = elemTy {
        for _ in 0..length {
            try lowerConstDataPayloadInto(self, repeat.item, elemTy, dataPrefix, b);
        }
        return;
    }

    if let case ast::NodeValue::Undef = repeat.item.value {
        let elemLayout = resolver::getTypeLayout(elemTy);
        dataBuilderPush(b, il::DataValue {
            item: il::DataItem::Undef,
            count: elemLayout.size * length
        });
    } else if let val = resolver::constValueEntry(self.resolver, repeat.item) {
        if let case resolver::ConstValue::String(_) = val {
            // A string used as a slice is represented by a three-word slice
            // header, not by the bytes of the string itself.
            for _ in 0..length {
                try lowerConstDataPayloadInto(self, repeat.item, elemTy, dataPrefix, b);
            }
        } else {
            dataBuilderPush(b, il::DataValue {
                item: constValueToDataItem(self, val, elemTy),
                count: length
            });
        }
    } else {
        for _ in 0..length {
            try lowerConstDataPayloadInto(self, repeat.item, elemTy, dataPrefix, b);
        }
    }
}

/// Build data values for a constant record literal.
/// Each field is lowered with a slot size that includes trailing padding.
unsafe fn lowerConstRecordLitInto(
    self: &mut Lowerer,
    node: *ast::Node,
    recLit: ast::RecordLit,
    ty: resolver::Type,
    dataPrefix: *[u8],
    b: &mut DataValueBuilder
) throws (LowerError) {
    match ty {
        case resolver::Type::Nominal(resolver::NominalType::Record(recInfo)) => {
            try lowerConstRecordCtorInto(self, recLit.fields, recInfo, dataPrefix, b);
        }
        case resolver::Type::Nominal(resolver::NominalType::Union(_)) => {
            let typeName = recLit.typeName else {
                throw LowerError::ExpectedVariant;
            };
            let sym = resolver::nodeData(self.resolver, typeName).sym else {
                throw LowerError::MissingSymbol(typeName);
            };
            try lowerConstUnionVariantInto(self, node, sym, ty, recLit.fields, dataPrefix, b);
        }
        else => throw LowerError::ExpectedRecord,
    }
}

/// Build data values for record constants.
unsafe fn lowerConstRecordCtorInto(
    self: &mut Lowerer,
    args: *[*ast::Node],
    recInfo: resolver::RecordType,
    dataPrefix: *[u8],
    b: &mut DataValueBuilder
) throws (LowerError) {
    let layout = recInfo.layout;
    for argNode, i in args {
        let mut valueNode = argNode;
        if let case ast::NodeValue::RecordLitField(fieldLit) = argNode.value {
            set valueNode = fieldLit.value;
        }
        let fieldInfo = recInfo.fields[i];
        let fieldOffset = fieldInfo.offset as u32;

        // Slot extends to the next field's offset,
        // or record size for the last field.
        let slotEnd = recInfo.fields[i + 1].offset as u32 if i + 1 < recInfo.fields.len else layout.size;
        let slotSize = slotEnd - fieldOffset;

        try lowerConstDataInto(self, valueNode, fieldInfo.fieldType, slotSize, dataPrefix, b);
    }
}

/// Build data values for a constant union variant value from payload fields/args.
unsafe fn lowerConstUnionVariantInto(
    self: &mut Lowerer,
    node: *ast::Node,
    variantSym: *unsafe mut resolver::Symbol,
    ty: resolver::Type,
    payloadArgs: *[*ast::Node],
    dataPrefix: *[u8],
    b: &mut DataValueBuilder
) throws (LowerError) {
    let case resolver::SymbolData::Variant { type: payloadType, index, .. } = variantSym.data
        else throw LowerError::UnexpectedNodeValue(node);

    let unionInfo = unionInfoFromType(ty) else {
        throw LowerError::MissingMetadata;
    };
    let unionLayout = resolver::getTypeLayout(ty);
    let payloadSlotSize = unionLayout.size - unionInfo.valOffset;

    // Tag byte.
    dataBuilderPush(b, il::DataValue {
        item: il::DataItem::Val {
            typ: il::Type::W8,
            val: index as i64
        },
        count: 1
    });
    // Padding between tag and payload.
    if unionInfo.valOffset > 1 {
        dataBuilderPush(b, il::DataValue {
            item: il::DataItem::Undef,
            count: unionInfo.valOffset - 1
        });
    }
    if payloadType == resolver::Type::Void {
        if payloadSlotSize > 0 {
            dataBuilderPush(b, il::DataValue {
                item: il::DataItem::Undef,
                count: payloadSlotSize
            });
        }
        return;
    }

    let case resolver::Type::Nominal(resolver::NominalType::Record(payloadRec)) = payloadType else {
        throw LowerError::ExpectedRecord;
    };
    let payloadLayout = payloadRec.layout;
    try lowerConstRecordCtorInto(self, payloadArgs, payloadRec, dataPrefix, b);

    // Unused bytes in the union payload slot for smaller variants.
    if payloadSlotSize > payloadLayout.size {
        dataBuilderPush(b, il::DataValue {
            item: il::DataItem::Undef,
            count: payloadSlotSize - payloadLayout.size
        });
    }
}

/// Find an existing string data entry with matching content.
// TODO: Optimize with hash table or remove?
fn findStringData(self: &Lowerer, s: *[u8]) -> ?*[u8] {
    for d in &self.data[..] {
        if d.values.len == 1 {
            if let case il::DataItem::Str(existing) = d.values[0].item {
                if mem::eq(existing, s) {
                    return d.name;
                }
            }
        }
    }
    return nil;
}

/// Compose a segmented symbol name from a list of path segments.
/// Example: `["func", "nominal", "VALUE"]` -> `func$nominal$VALUE`.
unsafe fn buildSegmentedName(
    self: &mut Lowerer,
    segments: &[*[u8]]
) -> *[u8] throws (LowerError) {
    assert segments.len > 0;

    let mut totalLen: u32 = 0;
    for segment in segments {
        set totalLen += segment.len;
    }
    if segments.len > 1 {
        set totalLen += segments.len - 1;
    }
    let buf = try! alloc::allocSlice(self.arena, 1, 1, totalLen) as *mut [u8];
    let mut pos: u32 = 0;

    for segment, i in segments {
        set pos += try! mem::copy(&mut buf[pos..], segment);
        if i + 1 <> segments.len {
            set buf[pos] = '$';
            set pos += 1;
        }
    }
    assert pos == totalLen;

    return &buf[..totalLen];
}

/// Generate a unique name for declaration-local backing data entries.
unsafe fn nextDeclDataName(
    self: &mut Lowerer,
    prefix: *[u8],
    count: u32,
    namespace: *[u8]
) -> *[u8] throws (LowerError) {
    let mut digits: [u8; fmt::U32_STR_LEN] = undefined;
    let start = fmt::formatU32(count, &mut digits[..]);
    let suffix = try! alloc::allocSlice(self.arena, 1, 1, digits.len - start) as *mut [u8];
    try! mem::copy(suffix, &digits[start..]);
    let segments = [prefix, namespace, suffix];

    return try buildSegmentedName(self, &segments[..]);
}

/// Append a data entry using a function-local literal namespace (`prefix$literal$N`).
unsafe fn pushDeclData(
    self: &mut Lowerer,
    size: u32,
    alignment: u32,
    readOnly: bool,
    values: *[il::DataValue],
    dataPrefix: *[u8]
) -> *[u8] throws (LowerError) {
    let dataCount = self.data.len;
    let name = try nextDeclDataName(self, dataPrefix, dataCount, "literal");
    self.data.append(il::Data {
        name,
        size,
        alignment,
        readOnly,
        isZeroInit: not readOnly and dataValuesAreZeroInit(values),
        values,
    }, self.allocator);

    return name;
}

/// Find or create read-only string data and return its symbol name.
unsafe fn getOrCreateStringData(
    self: &mut Lowerer,
    s: *[u8],
    dataPrefix: *[u8]
) -> *[u8] throws (LowerError) {
    if let existing = findStringData(self, s) {
        return existing;
    }
    let values = try! alloc::allocSlice(
        self.arena, @sizeOf(il::DataValue), @alignOf(il::DataValue), 1
    ) as *mut [il::DataValue];

    set values[0] = il::DataValue {
        item: il::DataItem::Str(s),
        count: 1
    };
    return try pushDeclData(self, s.len, 1, true, &values[..1], dataPrefix);
}

/// Compare two data items for structural equality.
/// Unlike raw byte comparison, this correctly ignores padding bytes in unions.
fn dataItemEq(a: il::DataItem, b: il::DataItem) -> bool {
    match a {
        case il::DataItem::Val { typ: aTyp, val: aVal } =>
            if let case il::DataItem::Val { typ: bTyp, val: bVal } = b {
                return aTyp == bTyp and aVal == bVal;
            } else {
                return false;
            },
        case il::DataItem::Sym(aPtr) =>
            if let case il::DataItem::Sym(bPtr) = b {
                return mem::eq(aPtr, bPtr);
            } else {
                return false;
            },
        case il::DataItem::Fn(aName) =>
            if let case il::DataItem::Fn(bName) = b {
                return mem::eq(aName, bName);
            } else {
                return false;
            },
        case il::DataItem::Str(aStr) =>
            if let case il::DataItem::Str(bStr) = b {
                return mem::eq(aStr, bStr);
            } else {
                return false;
            },
        case il::DataItem::Undef =>
            if let case il::DataItem::Undef = b {
                return true;
            } else {
                return false;
            },
    }
}

/// Compare two data value slices for structural equality.
fn dataValuesEq(a: &[il::DataValue], b: &[il::DataValue]) -> bool {
    if a.len <> b.len {
        return false;
    }
    for i in 0..a.len {
        if a[i].count <> b[i].count {
            return false;
        }
        if not dataItemEq(a[i].item, b[i].item) {
            return false;
        }
    }
    return true;
}

/// Find an existing read-only slice data entry with matching values.
// TODO: Optimize with hash table or remove?
fn findSliceData(self: &Lowerer, values: *[il::DataValue], alignment: u32) -> ?*[u8] {
    for d in &self.data[..] {
        if d.alignment == alignment and d.readOnly and dataValuesEq(d.values, values) {
            return d.name;
        }
    }
    return nil;
}

/// Find existing constant data entry with matching content.
/// Handles both string data and slice data.
fn findConstData(self: &Lowerer, values: *[il::DataValue], alignment: u32) -> ?*[u8] {
    // Fast path for strings.
    if values.len == 1 and alignment == 1 {
        if let case il::DataItem::Str(s) = values[0].item {
            return findStringData(self, s);
        }
    }
    // General case: byte comparison of data values.
    return findSliceData(self, values, alignment);
}

/// Lower constant data to a slice value.
/// Creates or reuses a data section entry, then builds a slice header on the stack.
unsafe fn lowerConstDataAsSlice(
    self: &mut FnLowerer,
    result: &ConstDataResult,
    alignment: u32,
    readOnly: bool,
    elemTy: *resolver::Type,
    mutable: bool,
    length: u32
) -> il::Val throws (LowerError) {
    let values = result.values;
    let elemLayout = resolver::getTypeLayout(*elemTy);
    let size = elemLayout.size * length;
    let mut dataName: *[u8] = undefined;
    let mut found: ?*[u8] = nil;
    if readOnly {
        set found = findConstData(self.low, values, alignment);
    }
    if let name = found {
        set dataName = name;
    } else {
        set dataName = try nextDataName(self);
        self.low.data.append(il::Data {
            name: dataName,
            size,
            alignment,
            readOnly,
            isZeroInit: not readOnly and result.zeroInit,
            values,
        }, self.low.allocator);
    }

    // Get data address.
    let ptrReg = nextReg(self);
    emit(self, il::Instr::Copy { dst: ptrReg, val: il::Val::DataSym(dataName) });

    return try buildSliceValue(
        self, elemTy, mutable, il::Val::Reg(ptrReg), il::Val::Imm(length as i64), il::Val::Imm(length as i64)
    );
}

/// Generate a unique data name for inline literals, eg. `fnName$literal$N`.
unsafe fn nextDataName(self: &mut FnLowerer) -> *[u8] throws (LowerError) {
    let counter = self.dataCounter;
    set self.dataCounter += 1;
    let fnName = self.fnName;
    return try nextDeclDataName(self.low, fnName, counter, "literal");
}

/// Assign a unique function-local data symbol name.
unsafe fn registerLocalDataDeclName(self: &mut FnLowerer, node: *ast::Node) throws (LowerError) {
    let sym = resolver::nodeData(self.low.resolver, node).sym
        else throw LowerError::MissingSymbol(node);

    let prefix = self.fnName;
    let segments = [prefix, "nominal", sym.name];
    let name = try buildSegmentedName(self.low, &segments[..]);

    set sym.name = name;
}

/// Get the next available SSA register.
fn nextReg(self: &mut FnLowerer) -> il::Reg {
    let reg = il::Reg { n: self.regCounter };
    set self.regCounter += 1;
    return reg;
}

/// Look up the resolved type of an AST node, or throw `MissingType`.
unsafe fn typeOf(self: &mut FnLowerer, node: *ast::Node) -> resolver::Type throws (LowerError) {
    let ty = resolver::typeFor(self.low.resolver, node)
        else throw LowerError::MissingType(node);
    return ty;
}

/// Look up the symbol for an AST node, or throw `MissingSymbol`.
unsafe fn symOf(self: &mut FnLowerer, node: *ast::Node) -> *unsafe mut resolver::Symbol throws (LowerError) {
    let sym = resolver::nodeData(self.low.resolver, node).sym
        else throw LowerError::MissingSymbol(node);
    return sym;
}

/// Remove the last block parameter and its associated variable.
/// Used when detecting a trivial phi that can be eliminated.
unsafe fn removeLastBlockParam(self: &mut FnLowerer, block: BlockId) {
    let blk = getBlockMut(self, block);
    if blk.params.len > 0 {
        // TODO: Use `pop`?
        set blk.params = @sliceOf(blk.params.ptr, blk.params.len - 1, blk.params.cap);
    }
    if blk.paramVars.len > 0 {
        // TODO: Use `pop`?
        set blk.paramVars = @sliceOf(blk.paramVars.ptr, blk.paramVars.len - 1, blk.paramVars.cap);
    }
}

/// Rewrite cached SSA values for a variable across all blocks, and also
/// rewrite any terminator arguments that reference the provisional register.
/// The latter is necessary because recursive SSA resolution may have already
/// patched terminator arguments with the provisional value before it was
/// found to be trivial.
unsafe fn rewriteCachedVarValue(self: &mut FnLowerer, v: Var, from: il::Val, to: il::Val) {
    for i in 0..self.blockData.len {
        let blk = getBlockMut(self, BlockId(i));
        if blk.vars[*v] == from {
            set blk.vars[*v] = to;
        }
        if blk.instrs.len > 0 {
            let ix = blk.instrs.len - 1;
            match &mut blk.instrs[ix] {
                case il::Instr::Jmp { args, .. } =>
                    rewriteValInSlice(*args, from, to),
                case il::Instr::Br { thenArgs, elseArgs, .. } => {
                    rewriteValInSlice(*thenArgs, from, to);
                    rewriteValInSlice(*elseArgs, from, to);
                }
                case il::Instr::Switch { defaultArgs, cases, .. } => {
                    rewriteValInSlice(*defaultArgs, from, to);
                    for j in 0..cases.len {
                        rewriteValInSlice(cases[j].args, from, to);
                    }
                }
                else => {}
            }
        }
    }
}

/// Replace all occurrences of `from` with `to` in an args slice.
unsafe fn rewriteValInSlice(args: *unsafe mut [il::Val], from: il::Val, to: il::Val) {
    for i in 0..args.len {
        if args[i] == from {
            set args[i] = to;
        }
    }
}

////////////////////////////
// Basic Block Management //
////////////////////////////

// Basic blocks are the fundamental unit of the CFG. Each block contains a
// sequence of instructions ending in a terminator (jump, branch, return).
//
// The block management API supports forward references -- you can create a
// block before switching to it and emitting instructions.
// This is essential for control flow where we need to reference target blocks
// before we've built them (e.g., the "else" block when building "then").
//
// Sealing is a key concept for SSA construction: a block is sealed once all
// its predecessor edges are known. Before sealing, we can't resolve variable
// uses that require looking up values from predecessors.

/// Create a new basic block with the given label base.
///
/// The block is initially unsealed (predecessors may be added later) and empty.
/// Returns a [`BlockId`] that can be used for jumps and branches. The block must
/// be switched to via [`switchToBlock`] before instructions can be emitted.
unsafe fn createBlock(self: &mut FnLowerer, labelBase: *[u8]) -> BlockId throws (LowerError) {
    let label = try nextLabel(self, labelBase);
    let id = BlockId(self.blockData.len);
    let varCount = self.localCount;
    let vars = try! alloc::allocRawSlice(self.low.fnArena, @sizeOf(?il::Val), @alignOf(?il::Val), varCount) as *unsafe mut [?il::Val];

    for i in 0..varCount {
        set vars[i] = nil;
    }
    self.blockData.append(BlockData {
        label,
        params: &mut [],
        paramVars: &mut [],
        instrs: &mut [],
        locs: &mut [],
        preds: &mut [],
        vars,
        sealState: Sealed::No,
        loopDepth: self.loopDepth,
    }, self.allocator);

    return id;
}

/// Create a new block with a single parameter.
unsafe fn createBlockWithParam(
    self: &mut FnLowerer,
    labelBase: *[u8],
    param: il::Param
) -> BlockId throws (LowerError) {
    let block = try createBlock(self, labelBase);
    let blk = getBlockMut(self, block);
    blk.params.append(param, self.allocator);

    return block;
}

/// Switch to building a different block.
/// All subsequent `emit` calls will add instructions to this block.
fn switchToBlock(self: &mut FnLowerer, block: BlockId) {
    set self.currentBlock = block;
}

/// Seal a block, indicating all predecessor edges are now known.
///
/// Sealing enables SSA construction to resolve variable uses by looking up
/// values from predecessors and inserting block parameters as needed. It
/// does not prevent instructions from being added to the block.
unsafe fn sealBlock(self: &mut FnLowerer, block: BlockId) throws (LowerError) {
    let blk = getBlockMut(self, block);
    let case Sealed::No = blk.sealState else {
        return; // Already sealed.
    };
    // Keep the current parameter list. Resolution can add more parameters.
    let paramVars = blk.paramVars;
    set blk.sealState = Sealed::Yes;

    // Complete each parameter that was created before sealing.
    for varId, paramIdx in paramVars {
        try resolveBlockArgs(self, block, Var(varId), paramIdx);
    }
}

/// Seal a block and switch to it.
unsafe fn switchToAndSeal(self: &mut FnLowerer, block: BlockId) throws (LowerError) {
    try sealBlock(self, block);
    switchToBlock(self, block);
}

/// Get block data by block id.
unsafe fn getBlock(self: &FnLowerer, block: BlockId) -> *unsafe BlockData {
    return &self.blockData[*block];
}

/// Get mutable block data by block id.
unsafe fn getBlockMut(self: &mut FnLowerer, block: BlockId) -> *unsafe mut BlockData {
    return &mut self.blockData[*block];
}

/// Get the current block being built.
fn currentBlock(self: &FnLowerer) -> BlockId {
    let block = self.currentBlock else {
        panic "currentBlock: no current block";
    };
    return block;
}

//////////////////////////
// Instruction Emission //
//////////////////////////

/// Emit an instruction to the current block.
unsafe fn emit(self: &mut FnLowerer, instr: il::Instr) {
    let blk = self.currentBlock else panic;
    let mut block = getBlockMut(self, blk);

    // Track whether this function is a leaf.
    if self.isLeaf {
        match instr {
            case il::Instr::Call { .. },
                 il::Instr::Ecall { .. } => set self.isLeaf = false,
            else => {},
        }
    }
    // Record source location alongside instruction when enabled.
    if self.low.options.debug {
        block.locs.append(self.srcLoc, self.allocator);
    }
    block.instrs.append(instr, self.allocator);
}

/// Emit an unconditional jump to `target`.
unsafe fn emitJmp(self: &mut FnLowerer, target: BlockId) throws (LowerError) {
    emit(self, il::Instr::Jmp { target: *target, args: &mut [] });
    addPredecessor(self, target, currentBlock(self));
}

/// Emit an unconditional jump to `target` with a single argument.
unsafe fn emitJmpWithArg(self: &mut FnLowerer, target: BlockId, arg: il::Val) throws (LowerError) {
    let args = try allocVal(self, arg);
    emit(self, il::Instr::Jmp { target: *target, args });
    addPredecessor(self, target, currentBlock(self));
}

/// Emit an unconditional jump to `target` and switch to it.
unsafe fn switchAndJumpTo(self: &mut FnLowerer, target: BlockId) throws (LowerError) {
    try emitJmp(self, target);
    switchToBlock(self, target);
}

/// Emit a conditional branch based on `cond`.
unsafe fn emitBr(self: &mut FnLowerer, cond: il::Reg, thenBlock: BlockId, elseBlock: BlockId) throws (LowerError) {
    assert thenBlock <> elseBlock;
    emit(self, il::Instr::Br {
        op: il::CmpOp::Ne,
        typ: il::Type::W32,
        a: il::Val::Reg(cond),
        b: il::Val::Imm(0),
        thenTarget: *thenBlock,
        thenArgs: &mut [],
        elseTarget: *elseBlock,
        elseArgs: &mut [],
    });
    addPredecessor(self, thenBlock, currentBlock(self));
    addPredecessor(self, elseBlock, currentBlock(self));
}

/// Emit a compare-and-branch instruction with the given comparison op.
unsafe fn emitBrCmp(
    self: &mut FnLowerer,
    op: il::CmpOp,
    typ: il::Type,
    a: il::Val,
    b: il::Val,
    thenBlock: BlockId,
    elseBlock: BlockId
) throws (LowerError) {
    assert thenBlock <> elseBlock;
    emit(self, il::Instr::Br {
        op, typ, a, b,
        thenTarget: *thenBlock, thenArgs: &mut [],
        elseTarget: *elseBlock, elseArgs: &mut [],
    });
    addPredecessor(self, thenBlock, currentBlock(self));
    addPredecessor(self, elseBlock, currentBlock(self));
}

/// Emit a guard that traps with `ebreak` when a comparison is false.
unsafe fn emitTrapUnlessCmp(
    self: &mut FnLowerer,
    op: il::CmpOp,
    typ: il::Type,
    a: il::Val,
    b: il::Val
) throws (LowerError) {
    let passBlock = try createBlock(self, "guard#pass");
    let trapBlock = try createBlock(self, "guard#trap");

    try emitBrCmp(self, op, typ, a, b, passBlock, trapBlock);
    try switchToAndSeal(self, trapBlock);

    emit(self, il::Instr::Ebreak);
    emit(self, il::Instr::Unreachable);

    try switchToAndSeal(self, passBlock);
}

/// Return whether two IL values are known `u32` immediates satisfying `a <= b`.
fn isLtEq(a: il::Val, b: il::Val) -> bool {
    if a == b {
        return true;
    }
    if a == il::Val::Imm(0) {
        return true;
    }
    let case il::Val::Imm(aa) = a else {
        return false;
    };
    let case il::Val::Imm(bb) = b else {
        return false;
    };
    if aa < 0 or bb < 0 {
        return false;
    }
    return aa <= bb;
}

/// Emit an `ebreak` when `a < b` holds.
unsafe fn emitTrapIfLt(
    self: &mut FnLowerer,
    typ: il::Type,
    a: il::Val,
    b: il::Val
) throws (LowerError) {
    let trapBlock = try createBlock(self, "guard#trap");
    let passBlock = try createBlock(self, "guard#pass");

    try emitBrCmp(self, il::CmpOp::Ult, typ, a, b, trapBlock, passBlock);
    try switchToAndSeal(self, trapBlock);

    emit(self, il::Instr::Ebreak);
    emit(self, il::Instr::Unreachable);

    try switchToAndSeal(self, passBlock);
}

/// Emit a conditional branch. Uses fused compare-and-branch for simple scalar
/// comparisons, falls back to separate comparison plus branch otherwise.
unsafe fn emitCondBranch(
    self: &mut FnLowerer,
    cond: *ast::Node,
    thenBlock: BlockId,
    elseBlock: BlockId
) throws (LowerError) {
    // Try fused compare-and-branch for simple scalar comparisons.
    if let case ast::NodeValue::BinOp(binop) = cond.value {
        let leftTy = try typeOf(self, binop.left);
        let rightTy = try typeOf(self, binop.right);
        if not isAggregateType(leftTy) and not isAggregateType(rightTy) {
            let operandTy = scalarComparisonType(leftTy, rightTy);
            let unsigned = isUnsignedType(operandTy);
            if let op = cmpOpFrom(binop.op, unsigned) {
                let a = try lowerExpr(self, binop.left);
                let b = try lowerExpr(self, binop.right);
                let typ = ilType(self.low, operandTy);

                // Swap operands if needed.
                match binop.op {
                    case ast::BinaryOp::Gt => // `a > b` = `b < a`
                        try emitBrCmp(self, op, typ, b, a, thenBlock, elseBlock),
                    case ast::BinaryOp::Lte => // `a <= b` = `!(b < a)`
                        try emitBrCmp(self, op, typ, b, a, elseBlock, thenBlock),
                    case ast::BinaryOp::Gte => // `a >= b` = `!(a < b)`
                        try emitBrCmp(self, op, typ, a, b, elseBlock, thenBlock),
                    else =>
                        try emitBrCmp(self, op, typ, a, b, thenBlock, elseBlock),
                }
                return;
            }
        }
    }
    // Fallback: evaluate condition and emit boolean branch.
    let condVal = try lowerExpr(self, cond);
    let condReg = emitValToReg(self, condVal);

    try emitBr(self, condReg, thenBlock, elseBlock);
}

/// Emit a 32-bit store instruction at the given offset.
unsafe fn emitStoreW32At(self: &mut FnLowerer, src: il::Val, dst: il::Reg, offset: i32) {
    emit(self, il::Instr::Store { typ: il::Type::W32, src, dst, offset });
}

/// Emit a 32-bit load instruction at the given offset.
unsafe fn emitLoadW32At(self: &mut FnLowerer, dst: il::Reg, src: il::Reg, offset: i32) {
    emit(self, il::Instr::Load { typ: il::Type::W32, dst, src, offset });
}

/// Emit an 8-bit store instruction at the given offset.
unsafe fn emitStoreW8At(self: &mut FnLowerer, src: il::Val, dst: il::Reg, offset: i32) {
    emit(self, il::Instr::Store { typ: il::Type::W8, src, dst, offset });
}

/// Emit an 8-bit load instruction at the given offset.
unsafe fn emitLoadW8At(self: &mut FnLowerer, dst: il::Reg, src: il::Reg, offset: i32) {
    emit(self, il::Instr::Load { typ: il::Type::W8, dst, src, offset });
}

/// Emit a 64-bit store instruction at the given offset.
unsafe fn emitStoreW64At(self: &mut FnLowerer, src: il::Val, dst: il::Reg, offset: i32) {
    emit(self, il::Instr::Store { typ: il::Type::W64, src, dst, offset });
}

/// Emit a 64-bit load instruction at the given offset.
unsafe fn emitLoadW64At(self: &mut FnLowerer, dst: il::Reg, src: il::Reg, offset: i32) {
    emit(self, il::Instr::Load { typ: il::Type::W64, dst, src, offset });
}

/// Load a tag from memory at `src` plus `offset` with the given IL type.
unsafe fn loadTag(self: &mut FnLowerer, src: il::Reg, offset: i32, tagType: il::Type) -> il::Val {
    let dst = nextReg(self);
    emit(self, il::Instr::Load { typ: tagType, dst, src, offset });
    return il::Val::Reg(dst);
}

/// Load the data pointer from a slice value.
unsafe fn loadSlicePtr(self: &mut FnLowerer, sliceReg: il::Reg) -> il::Reg {
    let ptrReg = nextReg(self);
    emitLoadW64At(self, ptrReg, sliceReg, SLICE_PTR_OFFSET);
    return ptrReg;
}

/// Load the length from a slice value.
unsafe fn loadSliceLen(self: &mut FnLowerer, sliceReg: il::Reg) -> il::Val {
    let lenReg = nextReg(self);
    emitLoadW32At(self, lenReg, sliceReg, SLICE_LEN_OFFSET);
    return il::Val::Reg(lenReg);
}

/// Load the capacity from a slice value.
unsafe fn loadSliceCap(self: &mut FnLowerer, sliceReg: il::Reg) -> il::Val {
    let capReg = nextReg(self);
    emitLoadW32At(self, capReg, sliceReg, SLICE_CAP_OFFSET);
    return il::Val::Reg(capReg);
}

/// Emit a load instruction for a scalar value at `src` plus `offset`.
/// For reading values that may be aggregates, use `emitRead` instead.
unsafe fn emitLoad(self: &mut FnLowerer, src: il::Reg, offset: i32, typ: resolver::Type) -> il::Val {
    let dst = nextReg(self);
    let ilTyp = ilType(self.low, typ);

    if isSignedType(typ) {
        emit(self, il::Instr::Sload { typ: ilTyp, dst, src, offset });
    } else {
        emit(self, il::Instr::Load { typ: ilTyp, dst, src, offset });
    }
    return il::Val::Reg(dst);
}

/// Read a value from memory at `src` plus `offset`. Aggregates are represented
/// as pointers, so we return the address directly. Scalars are loaded via [`emitLoad`].
unsafe fn emitRead(self: &mut FnLowerer, src: il::Reg, offset: i32, typ: resolver::Type) -> il::Val {
    if isAggregateType(typ) {
        let ptr = emitPtrOffset(self, src, offset);
        return il::Val::Reg(ptr);
    }
    return emitLoad(self, src, offset, typ);
}

/// Emit a copy instruction that loads a data symbol's address into a register.
unsafe fn emitDataAddr(self: &mut FnLowerer, sym: *unsafe resolver::Symbol) -> il::Reg {
    let dst = nextReg(self);
    let modId = resolver::moduleIdForSymbol(self.low.resolver, sym);
    let qualName = qualifyName(self.low, modId, sym.name);

    emit(self, il::Instr::Copy { dst, val: il::Val::DataSym(qualName) });

    return dst;
}

/// Emit a copy instruction that loads a function's address into a register.
unsafe fn emitFnAddr(self: &mut FnLowerer, sym: *unsafe resolver::Symbol) -> il::Reg {
    let dst = nextReg(self);
    let modId = resolver::moduleIdForSymbol(self.low.resolver, sym);
    let qualName = qualifyName(self.low, modId, sym.name);

    emit(self, il::Instr::Copy { dst, val: il::Val::FnAddr(qualName) });

    return dst;
}

/// Emit pattern tests for a single pattern.
unsafe fn emitPatternMatch(
    self: &mut FnLowerer,
    subject: &MatchSubject,
    pattern: *ast::Node,
    matchBlock: BlockId,
    fallthrough: BlockId
) throws (LowerError) {
    // Wildcards always match; array patterns are tested element-by-element
    // during binding, so they also unconditionally enter the match block.
    if isWildcardPattern(pattern) {
        try emitJmp(self, matchBlock);
        return;
    }
    if let case ast::NodeValue::ArrayLit(_) = pattern.value {
        try emitJmp(self, matchBlock);
        return;
    }
    if let case resolver::Type::Nominal(resolver::NominalType::Record(_)) = subject.type {
        if let case ast::NodeValue::RecordLit(_) = pattern.value {
            try emitJmp(self, matchBlock);
            return;
        }
    }
    let isNil = pattern.value == ast::NodeValue::Nil;

    match subject.kind {
        case MatchSubjectKind::OptionalPtr if isNil => {
            // Null pointer optimization: branch on the data pointer being null.
            let nilReg = try optionalNilReg(self, subject.val, subject.type);
            try emitBrCmp(self, il::CmpOp::Eq, il::Type::W64, il::Val::Reg(nilReg), il::Val::Imm(0), matchBlock, fallthrough);
        }
        case MatchSubjectKind::OptionalAggregate => {
            let base = emitValToReg(self, subject.val);

            if isNil { // Optional aggregate: `nil` means tag is zero.
                let tagReg = tvalTagReg(self, base);
                try emitBr(self, tagReg, fallthrough, matchBlock);
            } else {
                // `lowerExpr` applies the resolver's optional-lift coercion,
                // materializing the tagged representation for value patterns.
                let pattVal = try lowerExpr(self, pattern);
                let pattReg = emitValToReg(self, pattVal);
                let eq = try lowerOptionalEq(self, subject.bindType, base, pattReg, 0);
                let eqReg = emitValToReg(self, eq);

                try emitBr(self, eqReg, matchBlock, fallthrough);
            }
        }
        case MatchSubjectKind::Union(unionInfo) => {
            assert not isNil;

            let case resolver::NodeExtra::UnionVariant { tag: variantTag, .. } =
                resolver::nodeData(self.low.resolver, pattern).extra
            else {
                throw LowerError::ExpectedVariant;
            };
            // Void unions are passed by value (the tag itself).
            // Non-void unions are passed by reference (need to load tag).
            // When matching by reference, always load from the pointer.
            if unionInfo.isAllVoid {
                let mut tagVal = subject.val;
                match subject.by {
                    case resolver::MatchBy::Ref, resolver::MatchBy::MutRef => {
                        let base = emitValToReg(self, subject.val);
                        set tagVal = loadTag(self, base, 0, il::Type::W8);
                    }
                    case resolver::MatchBy::Value => {}
                }
                try emitBrCmp(self, il::CmpOp::Eq, il::Type::W8, tagVal, il::Val::Imm(variantTag as i64), matchBlock, fallthrough);
            } else {
                let base = emitValToReg(self, subject.val);
                let tagReg = tvalTagReg(self, base);

                try emitBrCmp(self, il::CmpOp::Eq, il::Type::W8, il::Val::Reg(tagReg), il::Val::Imm(variantTag as i64), matchBlock, fallthrough);
            }
        }
        else => { // Value comparison.
            assert not isNil;
            let pattVal = try lowerExpr(self, pattern);
            if isAggregateType(subject.type) {
                // Aggregate types need structural comparison rather than
                // scalar compare.
                let subjectReg = emitValToReg(self, subject.val);
                let pattReg = emitValToReg(self, pattVal);
                let eq = try lowerAggregateEq(self, subject.type, subjectReg, pattReg, 0);
                let eqReg = emitValToReg(self, eq);

                try emitBr(self, eqReg, matchBlock, fallthrough);
            } else {
                try emitBrCmp(self, il::CmpOp::Eq, subject.ilType, subject.val, pattVal, matchBlock, fallthrough);
            }
        }
    }
}

/// Emit branches for multiple patterns. The first pattern that matches
/// causes a jump to the match block. If no patterns match, we jump to the
/// fallthrough block.
unsafe fn emitPatternMatches(
    self: &mut FnLowerer,
    subject: &MatchSubject,
    patterns: &[*ast::Node],
    matchBlock: BlockId,
    fallthrough: BlockId
) throws (LowerError) {
    assert patterns.len > 0;

    for i in 0..(patterns.len - 1) {
        let pattern = patterns[i];
        let nextArm = try createBlock(self, "arm");
        try emitPatternMatch(self, subject, pattern, matchBlock, nextArm);

        // Seal the intermediate arm block: all predecessor edges are known
        // This ensures SSA construction can resolve variable uses through
        // single-predecessor optimization instead of creating unresolved block
        // parameters.
        try switchToAndSeal(self, nextArm);
    }
    // Handle last pattern: go to fallthrough block on failure.
    let last = patterns[patterns.len - 1];
    try emitPatternMatch(self, subject, last, matchBlock, fallthrough);
}

/// Emit a match binding pattern.
/// Binding patterns always match for regular values, but for optionals they
/// check for the presence of a value. Jumps to `valuePresent` on success,
/// `valueAbsent` on failure.
unsafe fn emitBindingTest(
    self: &mut FnLowerer,
    subject: &MatchSubject,
    valuePresent: BlockId,
    valueAbsent: BlockId
) throws (LowerError) {
    match subject.kind {
        case MatchSubjectKind::OptionalPtr, MatchSubjectKind::OptionalAggregate => {
            let nilReg = try optionalNilReg(self, subject.val, subject.type);
            try emitBr(self, nilReg, valuePresent, valueAbsent);
        }
        else => {
            // Regular values always match binding patterns unconditionally.
            try emitJmp(self, valuePresent);
        }
    }
}

/// Emit a jump to target if the current block hasn't terminated, then seal the target block.
unsafe fn emitJmpAndSeal(self: &mut FnLowerer, target: BlockId) throws (LowerError) {
    if not blockHasTerminator(self) {
        try emitJmp(self, target);
    }
    try sealBlock(self, target);
}

/// Check if the current block already has a terminator instruction.
unsafe fn blockHasTerminator(self: &FnLowerer) -> bool {
    let blk = getBlock(self, currentBlock(self));
    if blk.instrs.len == 0 {
        return false;
    }
    match blk.instrs[blk.instrs.len - 1] {
        case il::Instr::Ret { .. },
             il::Instr::Jmp { .. },
             il::Instr::Br { .. },
             il::Instr::Switch { .. },
             il::Instr::Unreachable =>
            return true,
        else =>
            return false,
    }
}

/// Emit a jump to merge block if the current block hasn't terminated.
///
/// This is used after lowering branches of an if-else, for example. If the
/// branch diverges, the block already has a terminator and no jump is needed.
///
/// This handles cases like:
///
///     if cond {
///         return 1;   // @then diverges, no jump to merge.
///     } else {
///         return 0;   // @else diverges, no jump to merge.
///     }
///
/// In the above example, the merge block stays `nil`, and no code is generated
/// after the `if`. The merge block is created on first use.
unsafe fn emitMergeIfUnterminated(self: &mut FnLowerer, mergeBlock: &mut ?BlockId) throws (LowerError) {
    if not blockHasTerminator(self) {
        if *mergeBlock == nil {
            set *mergeBlock = try createBlock(self, "merge");
        }
        let target = *mergeBlock else { throw LowerError::MissingTarget; };
        try emitJmp(self, target);
    }
}

//////////////////////////////////
// Control Flow Edge Management //
//////////////////////////////////

/// Add a predecessor edge from `pred` to `target`.
/// Must be called before the target block is sealed. Duplicates are ignored.
unsafe fn addPredecessor(self: &mut FnLowerer, target: BlockId, pred: BlockId) {
    let blk = getBlockMut(self, target);
    assert blk.sealState <> Sealed::Yes, "addPredecessor: adding predecessor to sealed block";
    let preds = &mut blk.preds;
    for i in 0..preds.len {
        if preds[i] == *pred { // Avoid duplicate predecessor entries.
            return;
        }
    }
    preds.append(*pred, self.allocator);
}

/// Finalize all blocks and return the block array.
unsafe fn finalizeBlocks(self: &mut FnLowerer) -> *unsafe [il::Block] throws (LowerError) {
    let blockCount = self.blockData.len;
    let blocks = try! alloc::allocRawSlice(
        self.low.fnArena, @sizeOf(il::Block), @alignOf(il::Block), blockCount
    ) as *unsafe mut [il::Block];

    for i in 0..self.blockData.len {
        let data = &self.blockData[i];

        set blocks[i] = il::Block {
            label: data.label,
            params: &data.params[..],
            instrs: data.instrs,
            locs: &data.locs[..],
            preds: &data.preds[..],
            loopDepth: data.loopDepth,
        };
    }
    return &blocks[..self.blockData.len];
}

/////////////////////
// Loop Management //
/////////////////////

/// Enter a loop context for break/continue handling.
/// `continueBlock` is `nil` when the continue target is created lazily.
unsafe fn enterLoop(self: &mut FnLowerer, breakBlock: BlockId, continueBlock: ?BlockId) {
    assert self.loopDepth < self.loopStack.len, "enterLoop: loop depth overflow";
    let slot = &mut self.loopStack[self.loopDepth];

    set slot.breakTarget = breakBlock;
    set slot.continueTarget = continueBlock;
    set self.loopDepth += 1;
}

/// Exit the current loop context.
fn exitLoop(self: &mut FnLowerer) {
    assert self.loopDepth <> 0, "exitLoop: loopDepth is zero";
    set self.loopDepth -= 1;
}

/// Get the current loop context.
unsafe fn currentLoop(self: &mut FnLowerer) -> ?*unsafe mut LoopCtx {
    if self.loopDepth == 0 {
        return nil;
    }
    return &mut self.loopStack[self.loopDepth - 1];
}

/// Get or lazily create the continue target block for the current loop.
unsafe fn getOrCreateContinueBlock(self: &mut FnLowerer) -> BlockId throws (LowerError) {
    let ctx = currentLoop(self) else {
        throw LowerError::OutsideOfLoop;
    };
    if let block = ctx.continueTarget {
        return block;
    }
    let block = try createBlock(self, "step");
    set ctx.continueTarget = block;
    return block;
}

/// Allocate a slice of values in the lowering arena.
unsafe fn allocVals(self: &mut FnLowerer, len: u32) -> *unsafe mut [il::Val] throws (LowerError) {
    return try! alloc::allocRawSlice(self.low.fnArena, @sizeOf(il::Val), @alignOf(il::Val), len) as *unsafe mut [il::Val];
}

/// Allocate a single-value slice in the lowering arena.
unsafe fn allocVal(self: &mut FnLowerer, val: il::Val) -> *unsafe mut [il::Val] throws (LowerError) {
    let args = try allocVals(self, 1);
    set args[0] = val;
    return args;
}

////////////////////////
// SSA Var Management //
////////////////////////

// This section implements SSA construction following "Simple and Efficient
// Construction of Static Single Assignment Form" (Braun et al., 2013). The IL
// uses block parameters (equivalent to phi nodes) that receive values via
// terminator arguments. Block args are resolved eagerly at seal time via
// [`resolveBlockArgs`], matching Braun's on-the-fly approach.
//
// In SSA form, each variable definition creates a unique value. When control
// flow diverges and merges (like after an if-else), a variable might have
// different definitions from different paths.
//
// # What "Value" means
//
// An [`il::Val`] is a compile-time representation of *where* a runtime value
// lives, not the runtime value itself. Values can live in registers, as symbol
// references, or as immediate values, ie. static constants.
//
// When we "find the value" of a variable, we're answering: "Which SSA register
// (or constant) represents this variable at this program point?"
//
// Each source-level variable gets a `Var` handle on declaration. When a
// variable is defined via [`defVar`], its SSA value is recorded in the current
// block's variable mapping. When a variable is used with [`useVar`] or
// [`useVarInBlock`], we either return the local definition or recursively look
// up the value from predecessor blocks. When multiple predecessors define
// different values for a given variable, we insert a block parameter
// (equivalent to a phi node).
//
// The algorithm used by [`useVarInBlock`] handles three cases:
//
// 1. **Local definition exists**: If the variable was assigned in this block,
//    return that value immediately (fast path).
//
// 2. **Sealed block with single predecessor**: If all incoming edges are known
//    and there's exactly one predecessor, recurse to that predecessor. No merge
//    is needed since there's only one path. The result is cached.
//
// 3. **Unsealed block or multiple predecessors**: Create a block parameter to
//    receive the merged value. If sealed, look up each predecessor's value and
//    patch its terminator via [`resolveBlockArgs`]. If unsealed, `paramVars`
//    records the variable in parameter order. [`sealBlock`] resolves these
//    parameters after all incoming edges are known.
//
// Consider this code:
//
//     let mut x = 1;
//     if cond {
//         x = 2;
//     } else {
//         x = f();   // Result in register %r.
//     }
//     print(x);      // Which value?
//
// The Control Flow Graph (CFG) looks like:
//
//         [entry]
//           _|_
//          /   \
//     [then]   [else]
//     x = 2    x = %r
//         \    /
//        [merge]
//         use(x)
//
// At the `print(x)` point, the compiler doesn't know which branch ran (that's
// a runtime decision), but it needs to emit code that works for either case.
// This is where [`useVarInBlock`] is called.
//
// The generated IL looks like this:
//
//     @then
//       jmp @merge(2);           // pass immediate `2`
//     @else
//       jmp @merge(%r);          // pass register `%r`
//     @merge(w32 %m)             // `%m` receives whichever value arrives at runtime
//       call w32 $print(%m);     // `print` is called with runtime value in `%m`
//
// A block is "sealed" when all predecessor edges are known. This is crucial
// because until sealed, we can't know how many paths merge into the block,
// and thus can't create the right number of block parameter arguments.
//
// If a block isn't sealed but we need a variable's value, we still create
// a block parameter, but defer filling in the terminator arguments. When the
// block is later sealed via [`sealBlock`], all incomplete block params are resolved.

/// Declare a new source-level variable and define its initial value.
/// If called before any block exists (e.g., for parameters), the definition is skipped.
unsafe fn newVar(
    self: &mut FnLowerer,
    name: ?*[u8],
    type: il::Type,
    mutable: bool,
    val: il::Val
) -> Var {
    let id = self.vars.len;
    self.vars.append(VarData { name, type, mutable, addressTaken: false }, self.allocator);

    let v = Var(id);
    if self.currentBlock <> nil {
        defVar(self, v, val);
    }
    return v;
}

/// Define (write) a variable. Record the SSA value of a variable in the
/// current block. Called when a variable is assigned or initialized (`let`
/// bindings, assignments, loop updates). When [`useVar`] is later called,
/// it will retrieve this value.
unsafe fn defVar(self: &mut FnLowerer, v: Var, val: il::Val) {
    assert *v < self.vars.len;
    set getBlockMut(self, currentBlock(self)).vars[*v] = val;
}

/// Use (read) the current value of a variable in the current block.
/// May insert block parameters if the value must come from predecessors.
unsafe fn useVar(self: &mut FnLowerer, v: Var) -> il::Val throws (LowerError) {
    return try useVarInBlock(self, currentBlock(self), v);
}

/// Resolve which SSA definition of a variable reaches a use point in a given block.
///
/// Given a variable and a block where it's used, this function finds the
/// correct [`il::Val`] that holds the variable's value at that program point.
/// When control flow merges from multiple predecessors with different
/// definitions, it creates a block parameter to unify them.
unsafe fn useVarInBlock(self: &mut FnLowerer, block: BlockId, v: Var) -> il::Val throws (LowerError) {
    assert *v < self.vars.len;

    let blk = getBlockMut(self, block);
    if let val = blk.vars[*v] {
        return val;
    }
    // Entry block cannot have block parameters. If variable isn't defined
    // in entry, we return undefined.
    if block == self.entryBlock {
        return il::Val::Undef;
    }
    if blk.sealState == Sealed::Yes {
        if blk.preds.len == 0 {
            // Variable used in sealed block with no predecessors.
            throw LowerError::InvalidUse;
        }
        // Single predecessor means no merge needed, variable is implicitly
        // available without a block parameter.
        if blk.preds.len == 1 {
            let pred = BlockId(blk.preds[0]);
            if *pred <> *block {
                let val = try useVarInBlock(self, pred, v);
                set blk.vars[*v] = val; // Cache.
                return val;
            }
        }
    }
    // Multiple predecessors or unsealed block: need a block parameter to merge
    // the control flow paths.
    return try createBlockParam(self, block, v);
}

/// Look up a variable by name in the current scope.
/// Searches from most recently declared to first, enabling shadowing.
unsafe fn lookupVarByName(self: &FnLowerer, name: *[u8]) -> ?Var {
    let mut id = self.vars.len;
    while id > 0 {
        set id -= 1;
        if let varName = self.vars[id].name {
            // Names are interned strings, so pointer comparison suffices.
            if varName == name {
                return Var(id);
            }
        }
    }
    return nil;
}

/// Look up a local variable bound to an identifier node.
unsafe fn lookupLocalVar(self: &FnLowerer, node: *ast::Node) -> ?Var {
    let case ast::NodeValue::Ident(name) = node.value else {
        return nil;
    };
    return lookupVarByName(self, name);
}

/// Save current lexical variable scope depth.
unsafe fn enterVarScope(self: &FnLowerer) -> u32 {
    return self.vars.len;
}

/// Restore lexical variable scope depth.
unsafe fn exitVarScope(self: &mut FnLowerer, savedVarsLen: u32) {
    set self.vars = @sliceOf(self.vars.ptr, savedVarsLen, self.vars.cap);
}

/// Get the metadata for a variable.
unsafe fn getVar(self: &FnLowerer, v: Var) -> *unsafe VarData {
    assert *v < self.vars.len;
    return &self.vars[*v];
}

/// Create a block parameter to merge a variable's value from multiple
/// control flow paths.
///
/// Called when [`useVarInBlock`] cannot find a local definition and the block is
/// unsealed or has multiple predecessors. For example, `x` can be used in `@end`
/// but defined differently in `@then` and `@else`:
///
///     @then
///       jmp @end(1);            // x = 1
///     @else
///       jmp @end(2);            // x = 2
///     @end(w32 %1)              // x = %1, merged from predecessors
///       ret %1;
///
/// Create a register `%1` as a block parameter. In a sealed block, patch each
/// predecessor's jump to pass its value of `x`. Otherwise, defer until sealing.
unsafe fn createBlockParam(self: &mut FnLowerer, block: BlockId, v: Var) -> il::Val throws (LowerError) {
    // Entry block must not have block parameters.
    assert block <> self.entryBlock, "createBlockParam: entry block must not have block parameters";
    // Allocate a register to hold the merged value.
    let reg = nextReg(self);
    let type = getVar(self, v).type;

    // Create block parameter and add it to the block.
    let param = il::Param { value: reg, type };
    let blk = getBlockMut(self, block);
    let paramIdx = blk.paramVars.len;
    blk.params.append(param, self.allocator);
    blk.paramVars.append(*v, self.allocator); // Associate variable with parameter.

    // Record that this variable's value in this block is now the parameter register.
    // This must happen before the predecessor loop to handle self-referential loops.
    set blk.vars[*v] = il::Val::Reg(reg);

    if blk.sealState == Sealed::Yes {
        // Block sealed: check for trivial phi before committing. If all
        // predecessors provide the same value, we can remove the param we
        // just created and use that value directly.
        if let trivial = try getTrivialPhiVal(self, block, v) {
            let provisional = il::Val::Reg(reg);
            removeLastBlockParam(self, block);
            rewriteCachedVarValue(self, v, provisional, trivial);
            set getBlockMut(self, block).vars[*v] = trivial;
            return trivial;
        }
        // Non-trivial phi: patch predecessors to pass their values.
        try resolveBlockArgs(self, block, v, paramIdx);
    }
    return il::Val::Reg(reg);
}

/// Complete the block parameter at `paramIdx`. Look up the variable's value in
/// each predecessor and patch its terminator with the edge argument.
///
/// This is the block-parameter equivalent of adding operands to a phi-function in
/// traditional SSA. Where a phi-function merges values at the join point:
///
///     x3 = phi(x1, x2)
///
/// This representation avoids the need for phi nodes to reference their
/// predecessor blocks explicitly, since the control flow edges already encode
/// that information.
unsafe fn resolveBlockArgs(self: &mut FnLowerer, block: BlockId, v: Var, paramIdx: u32) throws (LowerError) {
    let blk = getBlock(self, block);

    // For each predecessor, recursively look up the variable's reaching definition
    // in that block, then patch the predecessor's terminator to pass that value
    // as an argument to this block's parameter.
    for predId in blk.preds {
        let pred = BlockId(predId);
        // This may recursively trigger more block arg resolution if the
        // predecessor also needs to look up the variable from its predecessors.
        let val = try useVarInBlock(self, pred, v);
        assert val <> il::Val::Undef, "createBlockParam: predecessor provides undef value for block parameter";
        patchTerminatorArg(self, pred, *block, paramIdx, val);
    }
}

/// Check if a block parameter is trivial, i.e. all predecessors provide
/// the same value. Returns the trivial value if so.
unsafe fn getTrivialPhiVal(self: &mut FnLowerer, block: BlockId, v: Var) -> ?il::Val throws (LowerError) {
    let blk = getBlock(self, block);
    // Get the block parameter register.
    let paramReg = blk.vars[*v];
    // Check if all predecessors provide the same value.
    let mut sameVal: ?il::Val = nil;

    for predId in blk.preds {
        let pred = BlockId(predId);
        let val = try useVarInBlock(self, pred, v);

        // Check if this is a self-reference.
        // This happens in cycles where the loop back-edge passes the phi to
        // itself. We skip self-references when checking for trivial phis.
        if let reg = paramReg {
            if val == reg {
                // Self-reference, skip this predecessor.
            } else if let sv = sameVal {
                if val <> sv {
                    // Multiple different values, not trivial.
                    return nil;
                }
            } else {
                set sameVal = val;
            }
        } else { // No param reg set yet, can't be trivial.
            return nil;
        }
    }
    return sameVal;
}

/// Patch a single terminator argument for a specific edge. This is used during
/// SSA construction to pass variable values along control flow edges.
unsafe fn patchTerminatorArg(
    self: &mut FnLowerer,
    from: BlockId,         // The predecessor block containing the terminator to patch.
    target: u32,           // The index of the target block we're passing the value to.
    paramIdx: u32,         // The index of the block parameter to set.
    val: il::Val           // The value to pass as the argument.
) {
    let data = getBlockMut(self, from);
    let ix = data.instrs.len - 1; // The terminator is always the last instruction.

    // TODO: We shouldn't need to use a mutable subscript here, given that the
    // fields are already mutable.
    match &mut data.instrs[ix] {
        case il::Instr::Jmp { args, .. } => {
            set *args = growArgs(self, *args, paramIdx + 1);
            set args[paramIdx] = val;
        }
        case il::Instr::Br { thenTarget, thenArgs, elseTarget, elseArgs, .. } => {
            // Nb. both branches could target the same block (e.g. `if cond { x } else { x }`).
            if *thenTarget == target {
                set *thenArgs = growArgs(self, *thenArgs, paramIdx + 1);
                set thenArgs[paramIdx] = val;
            }
            if *elseTarget == target {
                set *elseArgs = growArgs(self, *elseArgs, paramIdx + 1);
                set elseArgs[paramIdx] = val;
            }
        }
        case il::Instr::Switch { defaultTarget, defaultArgs, cases, .. } => {
            if *defaultTarget == target {
                set *defaultArgs = growArgs(self, *defaultArgs, paramIdx + 1);
                set defaultArgs[paramIdx] = val;
            }
            let idx = paramIdx + 1;
            for ci in 0..cases.len {
                if cases[ci].target == target {
                    set cases[ci].args = growArgs(self, cases[ci].args, idx);
                    set cases[ci].args[paramIdx] = val;
                }
            }
        }
        else => {
            // Other terminators (e.g. `Ret`, `Unreachable`) don't have successor blocks.
        }
    }
}

/// Grow an args array to hold at least the given capacity.
unsafe fn growArgs(self: &mut FnLowerer, args: *unsafe mut [il::Val], capacity: u32) -> *unsafe mut [il::Val] {
    if args.len >= capacity {
        return args;
    }
    let newArgs = try! alloc::allocRawSlice(
        self.low.fnArena, @sizeOf(il::Val), @alignOf(il::Val), capacity
    ) as *unsafe mut [il::Val];

    for arg, i in args {
        set newArgs[i] = arg;
    }
    for i in args.len..capacity {
        set newArgs[i] = il::Val::Undef;
    }
    return newArgs;
}

/// Extract the parameter name from an [`FnParam`] AST node value.
fn paramName(value: *ast::NodeValue) -> *[u8] throws (LowerError) {
    let case ast::NodeValue::FnParam(param) = *value else {
        throw LowerError::ExpectedFunctionParam;
    };
    let case ast::NodeValue::Ident(name) = param.name.value else {
        throw LowerError::ExpectedIdentifier;
    };
    return name;
}

/// Lower function parameters. Declares variables for each parameter.
/// When a receiver name is passed, we're handling a trait method.
unsafe fn lowerParams(
    self: &mut FnLowerer,
    fnType: resolver::FnType,
    astParams: *[*ast::Node],
    receiverName: ?*ast::Node
) -> *unsafe [il::Param] throws (LowerError) {
    let offset: u32 = 1 if self.returnReg <> nil else 0;
    let totalLen = fnType.paramTypes.len as u32 + offset;
    if totalLen == 0 {
        return &[];
    }
    assert fnType.paramTypes.len as u32 <= resolver::MAX_FN_PARAMS;

    let params = try! alloc::allocRawSlice(
        self.low.fnArena, @sizeOf(il::Param), @alignOf(il::Param), totalLen
    ) as *unsafe mut [il::Param];

    if let reg = self.returnReg {
        set params[0] = il::Param { value: reg, type: il::Type::W64 };
    }
    for i in 0..fnType.paramTypes.len as u32 {
        let type = ilType(self.low, *fnType.paramTypes[i]);
        let reg = nextReg(self);

        set params[i + offset] = il::Param { value: reg, type };

        // Declare the parameter variable. For the receiver, the name comes
        // from the receiver node.
        // For all other parameters, the name comes from the AST params.
        let mut name: *[u8] = undefined;
        if let recNode = receiverName {
            if i == 0 {
                let case ast::NodeValue::Ident(recName) = recNode.value else {
                    throw LowerError::ExpectedIdentifier;
                };
                set name = recName;
            } else {
                set name = try paramName(&astParams[i - 1].value);
            }
        } else {
            set name = try paramName(&astParams[i].value);
        }
        let v = newVar(self, name, type, false, il::Val::Undef);

        self.params.append(FnParamBinding { var: v, reg }, self.allocator);
    }
    return params;
}

/// Resolve match subject.
unsafe fn lowerMatchSubject(self: &mut FnLowerer, subject: *ast::Node) -> MatchSubject throws (LowerError) {
    let mut val = try lowerExpr(self, subject);
    let subjectType = try typeOf(self, subject);
    let unwrapped = resolver::unwrapMatchSubject(subjectType);

    // When matching an aggregate by value, copy it to a fresh stack slot so
    // that bindings are independent of the original memory.  Without this,
    // the lowerer returns a pointer into the source and mutations to the
    // source silently corrupt the bound variables.
    if unwrapped.by == resolver::MatchBy::Value and isAggregateType(unwrapped.effectiveTy) {
        set val = try emitStackVal(self, unwrapped.effectiveTy, val);
    }

    let mut bindType = unwrapped.effectiveTy;
    if let case resolver::Type::Optional(inner) = unwrapped.effectiveTy {
        set bindType = *inner;
    }
    let ilType = ilType(self.low, unwrapped.effectiveTy);
    let kind = matchSubjectKind(unwrapped.effectiveTy);

    return MatchSubject { val, type: unwrapped.effectiveTy, ilType, bindType, kind, by: unwrapped.by };
}

/// Check whether a case pattern is an unconditional wildcard.
fn isWildcardPattern(pattern: *ast::Node) -> bool {
    match pattern.value {
        case ast::NodeValue::Placeholder => return true,
        else => return false,
    }
}

/// Check whether an AST node is the `undefined` literal.
fn isUndef(node: *ast::Node) -> bool {
    match node.value {
        case ast::NodeValue::Undef => return true,
        else => return false,
    }
}

/// Load the tag byte from a tagged value aggregate (optionals and unions).
unsafe fn tvalTagReg(self: &mut FnLowerer, base: il::Reg) -> il::Reg {
    let tagReg = nextReg(self);
    emitLoadW8At(self, tagReg, base, TVAL_TAG_OFFSET);
    return tagReg;
}

/// Load the tag word from a result aggregate.
unsafe fn resultTagReg(self: &mut FnLowerer, base: il::Reg) -> il::Reg {
    let tagReg = nextReg(self);
    emitLoadW64At(self, tagReg, base, TVAL_TAG_OFFSET);
    return tagReg;
}

/// Get the register to compare against `0` for optional `nil` checking.
/// For null-ptr-optimized types, loads the data pointer, or returns it
/// directly for scalar pointers. For aggregates, returns the tag register.
unsafe fn optionalNilReg(self: &mut FnLowerer, val: il::Val, typ: resolver::Type) -> il::Reg throws (LowerError) {
    let reg = emitValToReg(self, val);

    match typ {
        case resolver::Type::Optional(resolver::Type::Slice { .. }) => {
            let ptrReg = nextReg(self);
            emitLoadW64At(self, ptrReg, reg, SLICE_PTR_OFFSET);
            return ptrReg;
        }
        case resolver::Type::Optional(resolver::Type::Pointer { .. }) => return reg,
        case resolver::Type::Optional(_) => return tvalTagReg(self, reg),
        else => return reg,
    }
}

/// Lower an optional nil check (`opt == nil` or `opt <> nil`).
unsafe fn lowerNilCheck(self: &mut FnLowerer, opt: *ast::Node, isEq: bool) -> il::Val throws (LowerError) {
    let optTy = try typeOf(self, opt);
    // Handle `nil == nil` or `nil <> nil`.
    if optTy == resolver::Type::Nil {
        return il::Val::Imm(1) if isEq else il::Val::Imm(0);
    }
    let val = try lowerExpr(self, opt);
    let cmpReg = try optionalNilReg(self, val, optTy);

    // Null-pointer-optimized types compare a 64-bit pointer against zero.
    // Aggregate optionals compare an 8-bit tag byte against zero.
    let cmpType = il::Type::W64 if resolver::isOptionalPointer(optTy) else il::Type::W8;

    let op = il::BinOp::Eq if isEq else il::BinOp::Ne;
    return emitTypedBinOp(self, op, cmpType, il::Val::Reg(cmpReg), il::Val::Imm(0));
}

/// Load the payload value from a tagged value aggregate at the given offset.
unsafe fn tvalPayloadVal(self: &mut FnLowerer, base: il::Reg, payload: resolver::Type, valOffset: i32) -> il::Val {
    if payload == resolver::Type::Void {
        return il::Val::Undef;
    }
    return emitRead(self, base, valOffset, payload);
}

/// Compute the address of the payload in a tagged value aggregate.
unsafe fn tvalPayloadAddr(self: &mut FnLowerer, base: il::Reg, valOffset: i32) -> il::Val {
    return il::Val::Reg(emitPtrOffset(self, base, valOffset));
}

/// Bind a variable to a tagged value's payload.
unsafe fn bindPayloadVariable(
    self: &mut FnLowerer,
    name: *[u8],
    subjectVal: il::Val,
    bindType: resolver::Type,
    matchBy: resolver::MatchBy,
    valOffset: i32,
    mutable: bool
) -> Var throws (LowerError) {
    let base = emitValToReg(self, subjectVal);
    let mut payload: il::Val = undefined;

    match matchBy {
        case resolver::MatchBy::Value =>
            set payload = tvalPayloadVal(self, base, bindType, valOffset),
        case resolver::MatchBy::Ref, resolver::MatchBy::MutRef =>
            set payload = tvalPayloadAddr(self, base, valOffset),
    };
    return newVar(self, name, ilType(self.low, bindType), mutable, payload);
}

/// Bind an identifier from a matched subject.
unsafe fn bindMatchVariable(
    self: &mut FnLowerer,
    subject: &MatchSubject,
    binding: *ast::Node,
    mutable: bool
) -> ?Var throws (LowerError) {
    // Only bind if the pattern is an identifier.
    let case ast::NodeValue::Ident(name) = binding.value else {
        return nil;
    };
    // For optional aggregates, extract the payload from the tagged value.
    // The tag check already passed, so we know the payload is valid.
    if let case MatchSubjectKind::OptionalAggregate = subject.kind {
        let valOffset = resolver::getOptionalValOffset(subject.bindType) as i32;
        return try bindPayloadVariable(self, name, subject.val, subject.bindType, subject.by, valOffset, mutable);
    }
    // Declare the variable in the current block's scope.
    return newVar(self, name, ilType(self.low, subject.bindType), mutable, subject.val);
}

/// Bind variables from inside case patterns (union variants, records, slices).
/// `failBlock` is passed when nested patterns may require additional tests
/// that branch on mismatch (e.g. nested union variant tests).
unsafe fn bindPatternVariables(self: &mut FnLowerer, subject: &MatchSubject, patterns: &[*ast::Node], failBlock: BlockId) throws (LowerError) {
    for pattern in patterns {

        // Handle simple variant patterns like `Variant(x)`.
        if let arg = resolver::variantPatternBinding(self.low.resolver, pattern) {
            let case MatchSubjectKind::Union(unionInfo) = subject.kind
                else panic "bindPatternVariables: expected union subject";
            let valOffset = unionInfo.valOffset as i32;

            // Get the actual field type from the variant's record info.
            // This preserves the original data layout type (e.g. `*T`) even when
            // the resolver resolved the pattern against a dereferenced type (`T`).
            let variantExtra = resolver::nodeData(self.low.resolver, pattern).extra;
            let case resolver::NodeExtra::UnionVariant { ordinal, .. } = variantExtra
                else panic "bindPatternVariables: expected variant extra";
            let payloadType = unionInfo.variants[ordinal].valueType;
            let payloadRec = resolver::getRecord(payloadType)
                else panic "bindPatternVariables: expected record payload";
            let fieldType = payloadRec.fields[0].fieldType;

            match arg.value {
                case ast::NodeValue::Ident(name) => {
                    try bindPayloadVariable(self, name, subject.val, fieldType, subject.by, valOffset, false);
                }
                case ast::NodeValue::Placeholder => {}
                else => {
                    // Nested pattern inside a variant call, e.g. `Variant(Inner { x, y })`.
                    let base = emitValToReg(self, subject.val);
                    let payloadBase = emitPtrOffset(self, base, valOffset);
                    let fieldInfo = resolver::RecordField {
                        name: nil,
                        fieldType,
                        offset: 0,
                    };
                    try bindFieldVariable(self, arg, payloadBase, fieldInfo, subject.by, failBlock);
                }
            }
        }
        match pattern.value {
            // Compound variant patterns like `Variant { a, b }`.
            case ast::NodeValue::RecordLit(lit) =>
                try bindRecordPatternFields(self, subject, pattern, lit, failBlock),
            // Array patterns like `[a, b, c]`.
            case ast::NodeValue::ArrayLit(items) =>
                try bindArrayPatternElements(self, subject, items, failBlock),
            // Literals, wildcards, identifiers: no bindings needed.
            else => {},
        }
    }
}

/// Bind variables from an array literal pattern (e.g., `[a, 1, c]`).
/// Each element is either bound as a variable, skipped (placeholder), or
/// tested against the subject element, branching to `failBlock` on mismatch.
unsafe fn bindArrayPatternElements(
    self: &mut FnLowerer,
    subject: &MatchSubject,
    items: *[*ast::Node],
    failBlock: BlockId
) throws (LowerError) {
    let case resolver::Type::Array(arrInfo) = subject.type
        else throw LowerError::ExpectedSliceOrArray;

    let elemTy = *arrInfo.item;
    let elemLayout = resolver::getTypeLayout(elemTy);
    let stride = elemLayout.size as i32;
    let base = emitValToReg(self, subject.val);

    for elem, i in items {
        let fieldInfo = resolver::RecordField {
            name: nil,
            fieldType: elemTy,
            offset: (i as i32) * stride,
        };
        try bindFieldVariable(self, elem, base, fieldInfo, subject.by, failBlock);
    }
}

/// Bind fields from a record pattern.
unsafe fn bindRecordPatternFields(self: &mut FnLowerer, subject: &MatchSubject, pattern: *ast::Node, lit: ast::RecordLit, failBlock: BlockId) throws (LowerError) {
    // No fields to bind (e.g., `{ .. }`).
    if lit.fields.len == 0 {
        return;
    }
    // Optional value patterns were already compared structurally by
    // `emitPatternMatch`; unlike union record patterns, they have no payload
    // bindings to extract here.
    if let case MatchSubjectKind::OptionalAggregate = subject.kind {
        return;
    }
    if let recInfo = resolver::getRecord(subject.type) {
        let base = emitValToReg(self, subject.val);
        try bindNestedRecordFields(self, base, lit, recInfo, subject.by, failBlock);
        return;
    }
    // Get the union type info from the subject.
    let case MatchSubjectKind::Union(unionInfo) = subject.kind
        else panic "bindRecordPatternFields: expected union subject";

    // Get the variant index from the pattern node.
    let case resolver::NodeExtra::UnionVariant { ordinal: variantOrdinal, .. } =
        resolver::nodeData(self.low.resolver, pattern).extra
    else throw LowerError::MissingMetadata;

    // Get the record type from the variant's payload type.
    let payloadType = unionInfo.variants[variantOrdinal].valueType;
    let recInfo = resolver::getRecord(payloadType)
        else throw LowerError::ExpectedRecord;

    // Get the payload base pointer which points to the record within the tagged union.
    let base = emitValToReg(self, subject.val);
    let valOffset = unionInfo.valOffset as i32;
    let payloadBase = emitPtrOffset(self, base, valOffset);

    try bindNestedRecordFields(self, payloadBase, lit, recInfo, subject.by, failBlock);
}

/// Bind a single record field to a pattern variable, with support for nested
/// pattern tests that branch to `failBlock` on mismatch.
unsafe fn bindFieldVariable(
    self: &mut FnLowerer,
    binding: *ast::Node,
    base: il::Reg,
    fieldInfo: resolver::RecordField,
    matchBy: resolver::MatchBy,
    failBlock: BlockId
) throws (LowerError) {
    match binding.value {
        case ast::NodeValue::Ident(name) => {
            let val = emitRead(self, base, fieldInfo.offset, fieldInfo.fieldType)
                if matchBy == resolver::MatchBy::Value
                else il::Val::Reg(emitPtrOffset(self, base, fieldInfo.offset));
            newVar(self, name, ilType(self.low, fieldInfo.fieldType), false, val);
        }
        case ast::NodeValue::Placeholder => {}
        case ast::NodeValue::RecordLit(lit) => {
            // Check if this record literal is a union variant pattern.
            if let keyNode = resolver::patternVariantKeyNode(binding) {
                if let case resolver::NodeExtra::UnionVariant { .. } = resolver::nodeData(self.low.resolver, keyNode).extra {
                    try emitNestedFieldTest(self, binding, base, fieldInfo, matchBy, failBlock);
                    return;
                }
            }
            // Plain nested record destructuring pattern.
            // Auto-deref: if the field is a pointer, load it first.
            let mut derefType = fieldInfo.fieldType;
            let mut nestedBase = emitPtrOffset(self, base, fieldInfo.offset);
            if let case resolver::Type::Pointer { target, .. } = fieldInfo.fieldType {
                let ptrReg = nextReg(self);
                emitLoadW64At(self, ptrReg, nestedBase, 0);
                set nestedBase = ptrReg;
                set derefType = *target;
            }
            let recInfo = resolver::getRecord(derefType)
                else throw LowerError::ExpectedRecord;

            try bindNestedRecordFields(self, nestedBase, lit, recInfo, matchBy, failBlock);
        }
        else => {
            // Nested pattern requiring a test (union variant scope access, literal, etc).
            try emitNestedFieldTest(self, binding, base, fieldInfo, matchBy, failBlock);
        }
    }
}

/// Emit a nested pattern test for a record field value, branching to
/// `failBlock` if the pattern does not match. On success, continues in
/// a fresh block and binds any nested variables.
unsafe fn emitNestedFieldTest(
    self: &mut FnLowerer,
    pattern: *ast::Node,
    base: il::Reg,
    fieldInfo: resolver::RecordField,
    matchBy: resolver::MatchBy,
    failBlock: BlockId
) throws (LowerError) {
    let mut fieldType = fieldInfo.fieldType;
    let fieldPtr = emitPtrOffset(self, base, fieldInfo.offset);

    // Auto-deref: when the field is a pointer and the pattern destructures
    // the pointed-to value, load the pointer and use the target type.
    // The loaded pointer becomes the base address for the nested subject.
    let mut derefBase: ?il::Reg = nil;
    if let case resolver::Type::Pointer { target, .. } = fieldType {
        if resolver::isDestructuringPattern(pattern) {
            let ptrReg = nextReg(self);
            emitLoadW64At(self, ptrReg, fieldPtr, 0);
            set derefBase = ptrReg;
            set fieldType = *target;
        }
    }
    // Build a MatchSubject for the nested field.
    let ilTy = ilType(self.low, fieldType);
    let kind = matchSubjectKind(fieldType);

    // Determine the subject value.
    let mut val: il::Val = undefined;
    if let reg = derefBase {
        // Auto-deref: the loaded pointer is the address of the target value.
        set val = il::Val::Reg(reg);
    } else if isAggregateType(fieldType) {
        // Aggregate: use the pointer.
        set val = il::Val::Reg(fieldPtr);
    } else {
        // Scalar: load the value.
        set val = emitRead(self, base, fieldInfo.offset, fieldType);
    }
    let nestedSubject = MatchSubject {
        val,
        type: fieldType,
        ilType: ilTy,
        bindType: fieldType,
        kind,
        by: matchBy,
    };

    // Emit the pattern test: on success jump to `continueBlock`, on fail to `failBlock`.
    let continueBlock = try createBlock(self, "nest");
    try emitPatternMatch(self, &nestedSubject, pattern, continueBlock, failBlock);
    try switchToAndSeal(self, continueBlock);

    // After the test succeeds, bind any nested variables.
    let patterns = [pattern];
    try bindPatternVariables(self, &nestedSubject, &patterns[..], failBlock);
}

/// Bind variables from a nested record literal pattern.
unsafe fn bindNestedRecordFields(
    self: &mut FnLowerer,
    base: il::Reg,
    lit: ast::RecordLit,
    recInfo: resolver::RecordType,
    matchBy: resolver::MatchBy,
    failBlock: BlockId
) throws (LowerError) {
    for fieldNode in lit.fields {
        let case ast::NodeValue::RecordLitField(field) = fieldNode.value else {
            throw LowerError::UnexpectedNodeValue(fieldNode);
        };
        let fieldIdx = resolver::recordFieldIndexFor(self.low.resolver, fieldNode)
            else throw LowerError::MissingMetadata;
        if fieldIdx >= recInfo.fields.len {
            throw LowerError::MissingMetadata;
        }
        let fieldInfo = recInfo.fields[fieldIdx];

        try bindFieldVariable(self, field.value, base, fieldInfo, matchBy, failBlock);
    }
}

/// Lower function body to a list of basic blocks.
unsafe fn lowerFnBody(self: &mut FnLowerer, body: *ast::Node) -> *unsafe [il::Block] throws (LowerError) {
    // Create and switch to entry block.
    let entry = try createBlock(self, "entry");
    set self.entryBlock = entry;
    switchToBlock(self, entry);

    /// Bind parameter registers to variables in the entry block.
    for def in self.params {
        defVar(self, def.var, il::Val::Reg(def.reg));
    }
    /// Lower function body.
    try lowerBlock(self, body);

    // Add implicit return if body doesn't diverge.
    if not blockHasTerminator(self) {
        if *self.fnType.returnType == resolver::Type::Never {
            emit(self, il::Instr::Unreachable);
        } else if self.fnType.throwList.len > 0 {
            if *self.fnType.returnType == resolver::Type::Void {
                // Implicit `void` return in throwing function: wrap in result success.
                let val = try buildResult(self, 0, nil, resolver::Type::Void);
                try emitRetVal(self, val);
            } else {
                // Non-void throwing function without explicit return should
                // not happen.
                panic "lowerFnBody: missing return in non-void function";
            }
        } else {
            emit(self, il::Instr::Ret { val: nil });
        }
    }
    return try finalizeBlocks(self);
}

/// Lower a scalar match as a switch instruction.
unsafe fn lowerMatchSwitch(self: &mut FnLowerer, prongs: *[*ast::Node], subject: &MatchSubject, mergeBlock: &mut ?BlockId) throws (LowerError) {
    let mut blocks: [BlockId; 32] = undefined;
    let mut cases: *unsafe mut [il::SwitchCase] = &mut [];
    let mut defaultIdx: u32 = 0;
    let entry = currentBlock(self);

    for p, i in prongs {
        let case ast::NodeValue::MatchProng(prong) = p.value
            else throw LowerError::UnexpectedNodeValue(p);

        match prong.arm {
            case ast::ProngArm::Binding(_), ast::ProngArm::Else => {
                set blocks[i] = try createBlock(self, "default");
                set defaultIdx = i;
            }
            case ast::ProngArm::Case(pats) => {
                set blocks[i] = try createBlock(self, "case");
                for pat in pats {
                    let cv = resolver::constValueEntry(self.low.resolver, pat)
                        else throw LowerError::MissingConst(pat);

                    cases.append(il::SwitchCase {
                        value: constToScalar(cv),
                        target: *blocks[i],
                        args: &mut []
                    }, self.allocator);
                }
            }
        }
        addPredecessor(self, blocks[i], entry);
    }
    emit(self, il::Instr::Switch {
        val: subject.val,
        defaultTarget: *blocks[defaultIdx],
        defaultArgs: &mut [],
        cases: &mut cases[..]
    });

    for p, i in prongs {
        let case ast::NodeValue::MatchProng(prong) = p.value
            else throw LowerError::UnexpectedNodeValue(p);

        try switchToAndSeal(self, blocks[i]);
        try lowerNode(self, prong.body);
        try emitMergeIfUnterminated(self, mergeBlock);
    }
    if let blk = *mergeBlock {
        try switchToAndSeal(self, blk);
    }
}

/// Lower a match statement.
///
/// Processes prongs sequentially, generating comparison code and branches for each.
/// Prongs are processed in source order, so earlier prongs take precedence.
///
/// Guards are handled by emitting an additional branch after pattern matching
/// but before the body.
///
/// Example:
///
///   match x {
///       case 0 => return 0,
///       case 1 => return 1,
///       else => return 2,
///   }
///
/// Generates:
///
///   arm#0:
///       br.eq w32 %x 0 case#0 arm#1;    // if `x == 0`, jump to case#0, else arm#1
///   case#0:
///       ret 0;                          // `case 0` body
///   arm#1:
///       br.eq w32 %x 1 case#1 arm#2;    // if `x == 1`, jump to case#1, else arm#2
///   case#1:
///       ret 1;                          // `case 1` body
///   arm#2:
///       jmp else#0;                     // fallthrough to `else`
///   else#0:
///       ret 2;                          // `else` body
///
/// Example: Binding with guard
///
///   match x {
///       y if y > 0 => return y,
///       else => return 0,
///   }
///
/// Generates:
///
///   arm#0:
///       jmp guard#0;                    // catch-all binding, jump to guard
///   case#0(w32 %y):
///       ret %y;                         // guarded case body, receives bound var
///   guard#0:
///       sgt w32 %cmp %x 0;              // evaluate guard `y > 0`
///       br.ne w32 %cmp 0 case#0 arm#1;  // if `true`, jump to case body
///   arm#1:
///       jmp else#0;                     // guard failed, fallthrough to `else`
///   else#0:
///       ret 0;                          // `else` body
///
unsafe fn lowerMatch(self: &mut FnLowerer, node: *ast::Node, m: ast::Match) throws (LowerError) {
    assert m.prongs.len > 0;

    let prongs = m.prongs;
    // Lower the subject expression once; reused across all arms.
    let subject = try lowerMatchSubject(self, m.subject);
    // Merge block created lazily if any arm needs it (i.e., doesn't diverge).
    let mut mergeBlock: ?BlockId = nil;

    // Use `switch` instruction for matches with constant patterns.
    if resolver::isMatchConst(self.low.resolver, node) {
        try lowerMatchSwitch(self, prongs, &subject, &mut mergeBlock);
        return;
    }
    // Fallback: chained branches.
    let firstArm = try createBlock(self, "arm");
    try emitJmp(self, firstArm);
    try switchToAndSeal(self, firstArm);

    for prongNode, i in prongs {
        let prongScope = enterVarScope(self);
        let case ast::NodeValue::MatchProng(prong) = prongNode.value
            else panic "lowerMatch: expected match prong";

        let isLastArm = i + 1 == prongs.len;
        let hasGuard = prong.guard <> nil;
        let catchAll = resolver::isProngCatchAll(self.low.resolver, prongNode);

        // Entry block: guard block if present, otherwise the body block.
        // The guard block must be created before the body block so that
        // block indices are in reverse post-order (RPO), which the register
        // allocator requires.
        let mut entryBlock: BlockId = undefined;
        if hasGuard {
            set entryBlock = try createBlock(self, "guard");
        }
        // Body block: where the case body lives.
        let mut bodyLabel = "case";
        if prong.arm == ast::ProngArm::Else {
            set bodyLabel = "else";
        }
        let mut bodyBlock = try createBlock(self, bodyLabel);
        if not hasGuard {
            set entryBlock = bodyBlock;
        }
        // Fallthrough block: jumped to when pattern or guard fails.
        let nextArm = try createBlock(self, "arm");

        // Emit pattern test: branch to entry block on match, next arm on fail.
        match prong.arm {
            case ast::ProngArm::Binding(_) if not catchAll =>
                try emitBindingTest(self, &subject, entryBlock, nextArm),
            case ast::ProngArm::Case(patterns) if not catchAll =>
                try emitPatternMatches(self, &subject, patterns, entryBlock, nextArm),
            else =>
                try emitJmp(self, entryBlock),
        }
        // Switch to entry block, where any variable bindings need to be created.
        try switchToAndSeal(self, entryBlock);

        // Bind pattern variables after successful match. Note that the guard
        // has not been evaluated yet. Nested patterns may emit additional
        // tests that branch to `nextArm` on failure, switching the current
        // block.
        match prong.arm {
            case ast::ProngArm::Binding(pat) =>
                try bindMatchVariable(self, &subject, pat, false),
            case ast::ProngArm::Case(patterns) =>
                try bindPatternVariables(self, &subject, patterns, nextArm),
            else => {},
        }

        // Evaluate guard if present; can still fail to next arm.
        if let g = prong.guard {
            try emitCondBranch(self, g, bodyBlock, nextArm);
        } else if *currentBlock(self) <> *bodyBlock {
            // Nested tests changed the current block. Create a new body block
            // after the nest blocks to maintain RPO ordering, and jump to it.
            set bodyBlock = try createBlock(self, bodyLabel);
            try emitJmp(self, bodyBlock);
        }
        // Lower prong body and jump to merge if unterminated.
        try switchToAndSeal(self, bodyBlock);
        try lowerNode(self, prong.body);
        try emitMergeIfUnterminated(self, &mut mergeBlock);
        exitVarScope(self, prongScope);

        // Switch to next arm, unless last arm without guard.
        if not isLastArm or hasGuard {
            try switchToAndSeal(self, nextArm);
            if isLastArm {
                // Last arm with guard: guard failure jumps to merge.
                try emitMergeIfUnterminated(self, &mut mergeBlock);
            }
        }
    }
    // Continue in merge block if we have one, ie. if at least one arm doesn't
    // diverge.
    if let blk = mergeBlock {
        try switchToAndSeal(self, blk);
    }
}

/// Lower an `if let` statement.
unsafe fn lowerIfLet(self: &mut FnLowerer, cond: ast::IfLet) throws (LowerError) {
    let savedVarsLen = enterVarScope(self);
    let subject = try lowerMatchSubject(self, cond.pattern.scrutinee);
    let mut thenBlock: BlockId = undefined;
    if cond.pattern.guard == nil {
        set thenBlock = try createBlock(self, "then");
    }
    let elseBlock = try createBlock(self, "else");
    let mut mergeBlock: ?BlockId = nil;

    // Pattern match: jump to @then on success, @else on failure.
    try lowerPatternMatch(self, &subject, &cond.pattern, &mut thenBlock, "then", elseBlock);

    // Lower then branch.
    try lowerNode(self, cond.thenBranch);
    try emitMergeIfUnterminated(self, &mut mergeBlock);
    // Pattern bindings are visible only in the success branch. Restore the
    // outer variable scope before lowering `else`, where a same-named outer
    // variable may be referenced.
    exitVarScope(self, savedVarsLen);

    // Lower else branch.
    try switchToAndSeal(self, elseBlock);
    if let elseBranch = cond.elseBranch {
        try lowerNode(self, elseBranch);
    }
    try emitMergeIfUnterminated(self, &mut mergeBlock);

    if let blk = mergeBlock {
        try switchToAndSeal(self, blk);
    }
}

/// Emit pattern match branch with optional guard, and bind variables.
/// Used by `if-let`, `let-else`, and `while-let` lowering.
///
/// When a guard is present, the guard block is created before `successBlock`
/// to ensure block indices are in RPO.
unsafe fn lowerPatternMatch(
    self: &mut FnLowerer,
    subject: &MatchSubject,
    pat: &ast::PatternMatch,
    successBlock: &mut BlockId,
    successLabel: *[u8],
    failBlock: BlockId
) throws (LowerError) {
    // If guard present, pattern match jumps to @guard, then guard evaluation
    // jumps to `successBlock` or `failBlock`. Otherwise, jump directly to
    // `successBlock`.
    let mut targetBlock: BlockId = undefined;
    if pat.guard <> nil {
        set targetBlock = try createBlock(self, "guard");
        set *successBlock = try createBlock(self, successLabel);
    } else {
        set targetBlock = *successBlock;
    }
    match pat.kind {
        case ast::PatternKind::Case => {
            let patterns = [pat.pattern];
            // Jump to `targetBlock` if the pattern matches, `failBlock` otherwise.
            try emitPatternMatches(self, subject, &patterns[..], targetBlock, failBlock);
            try switchToAndSeal(self, targetBlock);
            // Bind any variables inside the pattern. Nested patterns may
            // emit additional tests that branch to `failBlock`, switching
            // the current block.
            try bindPatternVariables(self, subject, &patterns[..], failBlock);
        }
        case ast::PatternKind::Binding => {
            // Jump to `targetBlock` if there is a value present, `failBlock` otherwise.
            try emitBindingTest(self, subject, targetBlock, failBlock);
            try switchToAndSeal(self, targetBlock);
            // Bind the matched value to the pattern variable.
            try bindMatchVariable(self, subject, pat.pattern, pat.mutable);
        }
    }
    // Handle guard: on success jump to `successBlock`, on failure jump to `failBlock`.
    if let g = pat.guard {
        try emitCondBranch(self, g, *successBlock, failBlock);
        try switchToAndSeal(self, *successBlock);
    } else if *currentBlock(self) <> *targetBlock {
        // Nested tests changed the current block. Create a new success block
        // after the nest blocks to maintain RPO ordering, and jump to it.
        set *successBlock = try createBlock(self, successLabel);

        try emitJmp(self, *successBlock);
        try switchToAndSeal(self, *successBlock);
    }
}

/// Lower a `let-else` statement.
unsafe fn lowerLetElse(self: &mut FnLowerer, letElse: ast::LetElse) throws (LowerError) {
    let subject = try lowerMatchSubject(self, letElse.pattern.scrutinee);
    let mut successBlock: BlockId = undefined;
    if letElse.pattern.guard == nil {
        set successBlock = try createBlock(self, "success");
    }
    // The else branch executes when the pattern fails to match.
    let elseBlock = try createBlock(self, "else");

    // Evaluate the pattern and jump to @success or @else.
    try lowerPatternMatch(
        self,
        &subject,
        &letElse.pattern,
        &mut successBlock,
        "success",
        elseBlock,
    );
    let mut bindingVar: ?Var = nil;
    if let case ast::PatternKind::Binding = letElse.pattern.kind {
        set bindingVar = lookupLocalVar(self, letElse.pattern.pattern);
    }
    let mergeBlock = try createBlock(self, "merge");
    try emitJmp(self, mergeBlock);

    try switchToAndSeal(self, elseBlock);
    if try typeOf(self, letElse.elseBranch) == resolver::Type::Never {
        try lowerNode(self, letElse.elseBranch);
    } else {
        let fallback = try lowerExpr(self, letElse.elseBranch);
        if let variable = bindingVar {
            defVar(self, variable, fallback);
        }
        try emitJmp(self, mergeBlock);
    }

    // Continue at @merge after a successful match or value-producing fallback.
    try switchToAndSeal(self, mergeBlock);
}

/// Lower a `while let` loop as a match-driven loop.
unsafe fn lowerWhileLet(self: &mut FnLowerer, w: ast::WhileLet) throws (LowerError) {
    let savedVarsLen = enterVarScope(self);
    // Create control flow blocks: loop header, body (created lazily when
    // there's a guard), and exit.
    let whileBlock = try createBlock(self, "while");
    let mut bodyBlock: BlockId = undefined;
    if w.pattern.guard == nil {
        set bodyBlock = try createBlock(self, "body");
    }
    let endBlock = try createBlock(self, "merge");

    // Enter loop context and jump to loop header.
    enterLoop(self, endBlock, whileBlock);
    try switchAndJumpTo(self, whileBlock);
    let subject = try lowerMatchSubject(self, w.pattern.scrutinee);

    // Evaluate pattern and jump to loop body or loop end.
    try lowerPatternMatch(self, &subject, &w.pattern, &mut bodyBlock, "body", endBlock);

    // Lower loop body, jump back to loop header, and exit loop context.
    try lowerBlock(self, w.body);
    try emitJmpAndSeal(self, whileBlock);

    exitLoop(self);
    try switchToAndSeal(self, endBlock);
    exitVarScope(self, savedVarsLen);
}

///////////////////
// Node Lowering //
///////////////////

/// Lower an AST node.
unsafe fn lowerNode(self: &mut FnLowerer, node: *ast::Node) throws (LowerError) {
    if self.low.options.debug {
        set self.srcLoc.offset = node.span.offset;
    }
    match node.value {
        case ast::NodeValue::Block(_) => {
            try lowerBlock(self, node);
        }
        case ast::NodeValue::Return { value } => {
            try lowerReturnStmt(self, node, value);
        }
        case ast::NodeValue::Throw { expr } => {
            try lowerThrowStmt(self, expr);
        }
        case ast::NodeValue::Let(l) => {
            try lowerLet(self, node, l);
        }
        case ast::NodeValue::ConstDecl(decl) => {
            // Local constants lower to data declarations and emit no runtime code.
            try registerLocalDataDeclName(self, node);
            try lowerDataDecl(self.low, node, decl.value, true);
        }
        case ast::NodeValue::StaticDecl(decl) => {
            // Local statics lower to data declarations and emit no runtime code.
            try registerLocalDataDeclName(self, node);
            try lowerDataDecl(self.low, node, decl.value, false);
        }
        case ast::NodeValue::If(i) => {
            try lowerIf(self, i);
        }
        case ast::NodeValue::IfLet(i) => {
            try lowerIfLet(self, i);
        }
        case ast::NodeValue::Assign(a) => {
            try lowerAssign(self, node, a);
        }
        case ast::NodeValue::Loop { body } => {
            try lowerLoop(self, body);
        }
        case ast::NodeValue::While(w) => {
            try lowerWhile(self, w);
        }
        case ast::NodeValue::WhileLet(w) => {
            try lowerWhileLet(self, w);
        }
        case ast::NodeValue::For(f) => {
            try lowerFor(self, node, f);
        }
        case ast::NodeValue::Break => {
            try lowerBreak(self);
        }
        case ast::NodeValue::Continue => {
            try lowerContinue(self);
        }
        case ast::NodeValue::Match(m) => {
            try lowerMatch(self, node, m);
        }
        case ast::NodeValue::LetElse(letElse) => {
            try lowerLetElse(self, letElse);
        }
        case ast::NodeValue::ExprStmt(expr) => {
            let _ = try lowerExpr(self, expr);
        }
        case ast::NodeValue::Panic { .. } => {
            emit(self, il::Instr::Unreachable);
        }
        case ast::NodeValue::Assert { condition, .. } => {
            // Lower `assert <cond>` as: `if not cond { unreachable; }`.
            let thenBlock = try createBlock(self, "assert.fail");
            let endBlock = try createBlock(self, "assert.ok");

            // Branch: if condition is `true`, go to `endBlock`; if `false`, go to `thenBlock`.
            try emitCondBranch(self, condition, endBlock, thenBlock);
            try sealBlock(self, thenBlock);

            // Emit `unreachable` in the failure block.
            switchToBlock(self, thenBlock);
            emit(self, il::Instr::Unreachable);

            // Continue after the assert.
            try switchToAndSeal(self, endBlock);
        }
        else => {
            // Treat as expression statement, discard result.
            let _ = try lowerExpr(self, node);
        }
    }
}

/// Lower a code block.
unsafe fn lowerBlock(self: &mut FnLowerer, node: *ast::Node) throws (LowerError) {
    let case ast::NodeValue::Block(blk) = node.value else {
        throw LowerError::ExpectedBlock(node);
    };
    let savedVarsLen = enterVarScope(self);
    for stmt in blk.statements {
        try lowerNode(self, stmt);

        // If the statement diverges, further statements are unreachable.
        if blockHasTerminator(self) {
            exitVarScope(self, savedVarsLen);
            return;
        }
    }
    exitVarScope(self, savedVarsLen);
}

///////////////////////////////////////
// Record and Aggregate Type Helpers //
///////////////////////////////////////

/// Extract the nominal record info from a resolver type.
unsafe fn recordInfoFromType(typ: resolver::Type) -> ?resolver::RecordType {
    let case resolver::Type::Nominal(resolver::NominalType::Record(recInfo)) = typ
        else return nil;

    return recInfo;
}

/// Extract the nominal union info from a resolver type.
unsafe fn unionInfoFromType(typ: resolver::Type) -> ?resolver::UnionType {
    let case resolver::Type::Nominal(resolver::NominalType::Union(unionInfo)) = typ
        else return nil;

    return unionInfo;
}

/// Return the effective type of a node after any coercion applied by
/// the resolver. `lowerExpr` already materializes the coercion in the
/// IL value, so the lowerer must use the post-coercion type when
/// choosing how to compare or store that value.
unsafe fn effectiveType(self: &mut FnLowerer, node: *ast::Node) -> resolver::Type throws (LowerError) {
    let ty = try typeOf(self, node);
    if let coerce = resolver::coercionFor(self.low.resolver, node) {
        if let case resolver::Coercion::OptionalLift(optTy) = coerce {
            return optTy;
        }
    }
    return ty;
}

/// Check if a resolver type lowers to an aggregate in memory.
unsafe fn isAggregateType(typ: resolver::Type) -> bool {
    match typ {
        case resolver::Type::Slice { .. },
             resolver::Type::TraitObject { .. } => return true,
        case resolver::Type::Optional(resolver::Type::Pointer { .. }) => {
            // Optional pointers are scalar due to NPO.
            return false;
        }
        case resolver::Type::Optional(_) => {
            // All other optionals, including optional slices, are aggregates.
            return true;
        }
        case resolver::Type::Nominal(_) => {
            // Void unions are small enough to pass by value.
            return not resolver::isVoidUnion(typ);
        }
        case resolver::Type::Array(_),
             resolver::Type::Nil => return true,
        else => return false,
    }
}

/// Check if a resolver type is a small aggregate that can be
/// passed or returned by value in a register.
unsafe fn isSmallAggregate(typ: resolver::Type) -> bool {
    match typ {
        case resolver::Type::Nominal(_) => {
            if resolver::isVoidUnion(typ) {
                return false;
            }
            let layout = resolver::getTypeLayout(typ);
            return layout.size <= resolver::PTR_SIZE;
        }
        else => return false,
    }
}

/// Whether a function needs a hidden return parameter.
///
/// This is the case for throwing functions, which return a result aggregate,
/// and for functions returning large aggregates that cannot be passed in
/// registers.
unsafe fn requiresReturnParam(fnType: *resolver::FnType) -> bool {
    return fnType.throwList.len > 0
        or (isAggregateType(*fnType.returnType)
        and not isSmallAggregate(*fnType.returnType));
}

/// Check if a node is a void union variant literal (e.g. `Color::Red`).
/// If so, returns the variant's tag index. This enables optimized comparisons
/// that only check the tag instead of doing full aggregate comparison.
unsafe fn voidVariantIndex(res: &resolver::Resolver, node: *ast::Node) -> ?i64 {
    let data = resolver::nodeData(res, node);
    // Optional equality checks both the presence tag and the union value.
    if let case resolver::Coercion::OptionalLift(_) = data.coercion {
        return nil;
    }
    let sym = data.sym else {
        return nil;
    };
    let case resolver::SymbolData::Variant { type: payloadType, index, .. } = sym.data else {
        return nil;
    };
    // Only void variants can use tag-only comparison.
    if payloadType <> resolver::Type::Void {
        return nil;
    }
    return index as i64;
}

/// Reserve stack storage for a value of the given type.
unsafe fn emitReserve(self: &mut FnLowerer, typ: resolver::Type) -> il::Reg throws (LowerError) {
    let layout = resolver::getTypeLayout(typ);
    return emitReserveLayout(self, layout);
}

/// Reserve stack storage with an explicit layout.
unsafe fn emitReserveLayout(self: &mut FnLowerer, layout: resolver::Layout) -> il::Reg {
    let dst = nextReg(self);

    emit(self, il::Instr::Reserve {
        dst,
        size: il::Val::Imm(layout.size as i64),
        alignment: layout.alignment,
    });
    return dst;
}

/// Store a value into an address.
unsafe fn emitStore(self: &mut FnLowerer, base: il::Reg, offset: i32, typ: resolver::Type, src: il::Val) throws (LowerError) {
    // `undefined` values need no store.
    if let case il::Val::Undef = src {
        return;
    }
    if isAggregateType(typ) {
        let dst = emitPtrOffset(self, base, offset);
        let src = emitValToReg(self, src);
        let layout = resolver::getTypeLayout(typ);

        emit(self, il::Instr::Blit { dst, src, size: il::Val::Imm(layout.size as i64) });
    } else {
        emit(self, il::Instr::Store {
            typ: ilType(self.low, typ),
            src,
            dst: base,
            offset,
        });
    }
}

/// Allocate stack space for a value and store it. Returns a pointer to the value.
unsafe fn emitStackVal(self: &mut FnLowerer, typ: resolver::Type, val: il::Val) -> il::Val throws (LowerError) {
    let ptr = try emitReserve(self, typ);
    try emitStore(self, ptr, 0, typ, val);
    return il::Val::Reg(ptr);
}

/// Generic helper to build any tagged aggregate.
/// Reserves space based on the provided layout, stores the tag, and optionally
/// stores the payload value at `valOffset`.
unsafe fn buildTagged(
    self: &mut FnLowerer,
    layout: resolver::Layout,
    tag: i64,
    payload: ?il::Val,
    payloadType: resolver::Type,
    tagSize: u32,
    valOffset: i32
) -> il::Val throws (LowerError) {
    let dst = nextReg(self);
    emit(self, il::Instr::Reserve {
        dst,
        size: il::Val::Imm(layout.size as i64),
        alignment: layout.alignment,
    });
    if tagSize == 1 {
        emitStoreW8At(self, il::Val::Imm(tag), dst, TVAL_TAG_OFFSET);
    } else {
        emitStoreW64At(self, il::Val::Imm(tag), dst, TVAL_TAG_OFFSET);
    }

    if let val = payload {
        if payloadType <> resolver::Type::Void {
            try emitStore(self, dst, valOffset, payloadType, val);
        }
    }
    return il::Val::Reg(dst);
}

/// Wrap a value in an optional type.
///
/// For optional pointers (`?*T`), the value is returned as-is since pointers
/// use zero to represent `nil`. For other optionals, builds a tagged aggregate.
/// with the tag set to `1`, and the value as payload.
unsafe fn wrapInOptional(self: &mut FnLowerer, val: il::Val, optType: resolver::Type) -> il::Val throws (LowerError) {
    let case resolver::Type::Optional(inner) = optType else {
        throw LowerError::ExpectedOptional;
    };
    // Null-pointer-optimized (NPO) types are used as-is -- valid values are never null.
    if resolver::isNullableType(*inner) {
        return val;
    }
    let layout = resolver::getTypeLayout(optType);
    let valOffset = resolver::getOptionalValOffset(*inner) as i32;

    return try buildTagged(self, layout, 1, val, *inner, 1, valOffset);
}

/// Build a `nil` value for an optional type.
///
/// For optional pointers (`?*T`), returns an immediate `0` (null pointer).
/// For other optionals, builds a tagged aggregate with tag set to `0` (absent).
unsafe fn buildNilOptional(self: &mut FnLowerer, optType: resolver::Type) -> il::Val throws (LowerError) {
    let case resolver::Type::Optional(inner) = optType
        else throw LowerError::ExpectedOptional;
    if let case resolver::Type::Pointer { .. } = *inner {
        return il::Val::Imm(0);
    }
    if let case resolver::Type::Slice { item, mutable, .. } = *inner {
        return try buildSliceValue(
            self, item, mutable, il::Val::Imm(0), il::Val::Imm(0), il::Val::Imm(0)
        );
    }
    let valOffset = resolver::getOptionalValOffset(*inner) as i32;
    return try buildTagged(self, resolver::getTypeLayout(optType), 0, nil, *inner, 1, valOffset);
}

/// Build a result value for throwing functions.
unsafe fn buildResult(
    self: &mut FnLowerer,
    tag: i64,
    payload: ?il::Val,
    payloadType: resolver::Type
) -> il::Val throws (LowerError) {
    let successType = *self.fnType.returnType;
    let layout = resolver::getResultLayout(
        successType, self.fnType.throwList
    );
    return try buildTagged(self, layout, tag, payload, payloadType, resolver::PTR_SIZE as i32, RESULT_VAL_OFFSET);
}

/// Build a slice aggregate from a data pointer, length and capacity.
unsafe fn buildSliceValue(
    self: &mut FnLowerer,
    elemTy: *resolver::Type,
    mutable: bool,
    ptrVal: il::Val,
    lenVal: il::Val,
    capVal: il::Val
) -> il::Val throws (LowerError) {
    let sliceType = resolver::Type::Slice {
        class: types::PointerClass::Unsafe,
        item: elemTy,
        mutable,
    };
    let dst = try emitReserve(self, sliceType);
    let ptrTy = resolver::Type::Pointer {
        class: types::PointerClass::Unsafe,
        target: elemTy,
        mutable,
    };

    try emitStore(self, dst, SLICE_PTR_OFFSET, ptrTy, ptrVal);
    try emitStore(self, dst, SLICE_LEN_OFFSET, resolver::Type::U32, lenVal);
    try emitStore(self, dst, SLICE_CAP_OFFSET, resolver::Type::U32, capVal);

    return il::Val::Reg(dst);
}

/// Build a trait object fat pointer from a data pointer and a v-table.
unsafe fn buildTraitObject(
    self: &mut FnLowerer,
    dataVal: il::Val,
    traitInfo: *unsafe resolver::TraitType,
    inst: &resolver::InstanceEntry
) -> il::Val throws (LowerError) {
    let vName = vtableName(self.low, inst.moduleId, inst.concreteTypeName, traitInfo.name);

    // Reserve space for the trait object on the stack.
    let slot = emitReserveLayout(self, resolver::Layout {
        size: resolver::PTR_SIZE * 2,
        alignment: resolver::PTR_SIZE,
    });

    // Store data pointer.
    emit(self, il::Instr::Store {
        typ: il::Type::W64,
        src: dataVal,
        dst: slot,
        offset: TRAIT_OBJ_DATA_OFFSET,
    });

    // Store v-table address.
    emit(self, il::Instr::Store {
        typ: il::Type::W64,
        src: il::Val::DataSym(vName),
        dst: slot,
        offset: TRAIT_OBJ_VTABLE_OFFSET,
    });
    return il::Val::Reg(slot);
}

/// Compute a field pointer by adding a byte offset to a base address.
unsafe fn emitPtrOffset(self: &mut FnLowerer, base: il::Reg, offset: i32) -> il::Reg {
    if offset == 0 {
        return base;
    }
    let dst = nextReg(self);

    emit(self, il::Instr::BinOp {
        op: il::BinOp::Add,
        typ: il::Type::W64,
        dst,
        a: il::Val::Reg(base),
        b: il::Val::Imm(offset as i64),
    });
    return dst;
}

/// Emit an element address computation for array/slice indexing.
/// Computes: `base + idx * stride`.
unsafe fn emitElem(self: &mut FnLowerer, stride: u32, base: il::Reg, idx: il::Val) -> il::Reg {
    // If index is zero, return base directly.
    if idx == il::Val::Imm(0) {
        return base;
    }
    // If stride is `1`, skip the multiply.
    if stride == 1 {
        let dst = nextReg(self);

        emit(self, il::Instr::BinOp {
            op: il::BinOp::Add,
            typ: il::Type::W64,
            dst,
            a: il::Val::Reg(base),
            b: idx
        });
        return dst;
    }
    // Compute `offset = idx * stride`.
    let offset = nextReg(self);

    emit(self, il::Instr::BinOp {
        op: il::BinOp::Mul,
        typ: il::Type::W64,
        dst: offset,
        a: idx,
        b: il::Val::Imm(stride as i64)
    });
    // Compute `dst = base + offset`.
    let dst = nextReg(self);
    emit(self, il::Instr::BinOp {
        op: il::BinOp::Add,
        typ: il::Type::W64,
        dst,
        a: il::Val::Reg(base),
        b: il::Val::Reg(offset)
    });
    return dst;
}

/// Emit a typed binary operation, returning the result as a value.
unsafe fn emitTypedBinOp(self: &mut FnLowerer, op: il::BinOp, typ: il::Type, a: il::Val, b: il::Val) -> il::Val {
    let dst = nextReg(self);
    emit(self, il::Instr::BinOp { op, typ, dst, a, b });
    return il::Val::Reg(dst);
}

/// Emit a tag comparison for void variant equality/inequality.
unsafe fn emitTagCmp(self: &mut FnLowerer, op: ast::BinaryOp, val: il::Val, tagIdx: i64, valType: resolver::Type) -> il::Val
    throws (LowerError)
{
    let reg = emitValToReg(self, val);

    // For all-void unions, the value *is* the tag, not a pointer.
    let mut tag: il::Val = undefined;
    if resolver::isVoidUnion(valType) {
        set tag = il::Val::Reg(reg);
    } else {
        set tag = loadTag(self, reg, TVAL_TAG_OFFSET, il::Type::W8);
    }
    let binOp = il::BinOp::Eq if op == ast::BinaryOp::Eq else il::BinOp::Ne;
    return emitTypedBinOp(self, binOp, il::Type::W8, tag, il::Val::Imm(tagIdx));
}

/// Logical "and" between two values. Returns the result in a register.
unsafe fn emitLogicalAnd(self: &mut FnLowerer, left: ?il::Val, right: il::Val) -> il::Val {
    let prev = left else {
        return right;
    };
    return emitTypedBinOp(self, il::BinOp::And, il::Type::W32, prev, right);
}

//////////////////////////
// Aggregate Comparison //
//////////////////////////

/// Emit an equality test for values at an offset of the given base registers.
unsafe fn emitEqAtOffset(
    self: &mut FnLowerer,
    left: il::Reg,
    right: il::Reg,
    offset: i32,
    fieldType: resolver::Type
) -> il::Val throws (LowerError) {
    // For aggregate types, pass offset through and compare recursively.
    if isAggregateType(fieldType) {
        return try lowerAggregateEq(self, fieldType, left, right, offset);
    }
    // For scalar types, load and compare directly.
    let a = emitLoad(self, left, offset, fieldType);
    let b = emitLoad(self, right, offset, fieldType);
    let dst = nextReg(self);
    emit(self, il::Instr::BinOp { op: il::BinOp::Eq, typ: ilType(self.low, fieldType), dst, a, b });

    return il::Val::Reg(dst);
}

/// Compare two record values for equality.
unsafe fn lowerRecordEq(
    self: &mut FnLowerer,
    recInfo: resolver::RecordType,
    a: il::Reg,
    b: il::Reg,
    offset: i32
) -> il::Val throws (LowerError) {
    let mut result: ?il::Val = nil;

    for field in recInfo.fields {
        let cmp = try emitEqAtOffset(self, a, b, offset + field.offset, field.fieldType);

        set result = emitLogicalAnd(self, result, cmp);
    }
    if let r = result {
        return r;
    }
    return il::Val::Imm(1);
}

/// Compare two slice values for equality.
unsafe fn lowerSliceEq(
    self: &mut FnLowerer,
    elemTy: *resolver::Type,
    mutable: bool,
    a: il::Reg,
    b: il::Reg,
    offset: i32
) -> il::Val throws (LowerError) {
    let ptrTy = resolver::Type::Pointer {
        class: types::PointerClass::Unsafe,
        target: elemTy,
        mutable,
    };
    let ptrEq = try emitEqAtOffset(self, a, b, offset + SLICE_PTR_OFFSET, ptrTy);
    let lenEq = try emitEqAtOffset(self, a, b, offset + SLICE_LEN_OFFSET, resolver::Type::U32);

    return emitTypedBinOp(self, il::BinOp::And, il::Type::W32, ptrEq, lenEq);
}

/// Compare two optional aggregate values for equality.
///
/// Two optionals are equal when their tags match and either both are `nil` or
/// their payloads are equal.
///
/// For inner types that are safe to compare even when uninitialised, we use a
/// branchless formulation: `tagEq AND (tagNil OR payloadEq)`
///
/// For inner types that may contain uninitialized data when `nil` (unions,
/// nested optionals), the payload comparison is guarded behind a branch
/// so that `nil` payloads are never inspected.
unsafe fn lowerOptionalEq(
    self: &mut FnLowerer,
    inner: resolver::Type,
    a: il::Reg,
    b: il::Reg,
    offset: i32
) -> il::Val throws (LowerError) {
    let valOffset = resolver::getOptionalValOffset(inner) as i32;

    // Load tags.
    let tagA = loadTag(self, a, offset + TVAL_TAG_OFFSET, il::Type::W8);
    let tagB = loadTag(self, b, offset + TVAL_TAG_OFFSET, il::Type::W8);

    // For simple inner types (no unions/nested optionals), use branchless comparison.
    // Unions and nested optionals may contain uninitialized payload bytes
    // when nil, so they need a guarded comparison.
    let isUnion = unionInfoFromType(inner) <> nil;
    let mut isOptional = false;
    if let case resolver::Type::Optional(_) = inner {
        set isOptional = true;
    }
    if not isUnion and not isOptional {
        let tagEq = emitTypedBinOp(self, il::BinOp::Eq, il::Type::W8, tagA, tagB);
        let tagNil = emitTypedBinOp(self, il::BinOp::Eq, il::Type::W8, tagA, il::Val::Imm(0));
        let payloadEq = try emitEqAtOffset(self, a, b, offset + valOffset, inner);

        return emitTypedBinOp(self, il::BinOp::And, il::Type::W32, tagEq,
            emitTypedBinOp(self, il::BinOp::Or, il::Type::W32, tagNil, payloadEq));
    }

    // For complex inner types, use branching comparison to avoid inspecting
    // uninitialized payload bytes.
    let resultReg = nextReg(self);
    let mergeBlock = try createBlockWithParam(self, "opteq#merge", il::Param {
        value: resultReg, type: il::Type::W8
    });
    let nilCheck = try createBlock(self, "opteq#nil");
    let payloadCmp = try createBlock(self, "opteq#payload");

    let falseArgs = try allocVal(self, il::Val::Imm(0));
    let trueArgs = try allocVal(self, il::Val::Imm(1));

    // Check if tags differ.
    emit(self, il::Instr::Br {
        op: il::CmpOp::Eq, typ: il::Type::W8, a: tagA, b: tagB,
        thenTarget: *nilCheck, thenArgs: &mut [],
        elseTarget: *mergeBlock, elseArgs: falseArgs,
    });
    addPredecessor(self, nilCheck, currentBlock(self));
    addPredecessor(self, mergeBlock, currentBlock(self));

    // Check if both are `nil`.
    try switchToAndSeal(self, nilCheck);
    emit(self, il::Instr::Br {
        op: il::CmpOp::Ne, typ: il::Type::W8, a: tagA, b: il::Val::Imm(0),
        thenTarget: *payloadCmp, thenArgs: &mut [],
        elseTarget: *mergeBlock, elseArgs: trueArgs,
    });
    addPredecessor(self, payloadCmp, currentBlock(self));
    addPredecessor(self, mergeBlock, currentBlock(self));

    // Both are non-`nil`, compare payloads.
    try switchToAndSeal(self, payloadCmp);
    let payloadEq = try emitEqAtOffset(self, a, b, offset + valOffset, inner);
    try emitJmpWithArg(self, mergeBlock, payloadEq);
    try switchToAndSeal(self, mergeBlock);

    return il::Val::Reg(resultReg);
}

/// Compare two union values for equality.
///
/// Two unions are equal iff their tags match and, for non-void variants,
/// their payloads are also equal. The comparison proceeds as follows.
///
/// First, compare the tags. If they differ, the unions are not equal, so
/// jump to the merge block with `false`. If they match, jump to the tag
/// block to determine which variant we're dealing with.
///
/// The tag block uses a switch on the tag value to dispatch to the appropriate
/// comparison block. Void variants jump directly to merge with `true`.
/// Non-void variants each have their own payload block that compares the
/// payload and jumps to the merge block with the result.
///
/// The merge block collects results from all paths via a block parameter
/// and returns the final equality result.
///
/// For all-void unions, we skip the control flow entirely and just compare
/// the tags directly.
///
/// TODO: Could be optimized to branchless when all non-void variants share
/// the same payload type: `tagEq AND (isVoidVariant OR payloadEq)`.
unsafe fn lowerUnionEq(
    self: &mut FnLowerer,
    unionInfo: resolver::UnionType,
    a: il::Reg,
    b: il::Reg,
    offset: i32
) -> il::Val throws (LowerError) {
    // Compare tags.
    let tagA = loadTag(self, a, offset + TVAL_TAG_OFFSET, il::Type::W8);
    let tagB = loadTag(self, b, offset + TVAL_TAG_OFFSET, il::Type::W8);

    // Fast path: all-void union just needs tag comparison.
    if unionInfo.isAllVoid {
        return emitTypedBinOp(self, il::BinOp::Eq, il::Type::W8, tagA, tagB);
    }
    // Holds the equality result.
    let resultReg = nextReg(self);

    // Where control flow continues after equality check is done. Receives
    // the result as a parameter.
    let mergeBlock = try createBlockWithParam(self, "eq#merge", il::Param {
        value: resultReg, type: il::Type::W8
    });
    // Where we switch on the tag to compare payloads.
    let tagBlock = try createBlock(self, "eq#tag");

    // Compare tags: if they differ, jump to merge with `false`; otherwise check payloads.
    let falseArgs = try allocVal(self, il::Val::Imm(0));

    assert tagBlock <> mergeBlock;

    // TODO: Use the helper once the compiler supports more than eight function params.
    emit(self, il::Instr::Br {
        op: il::CmpOp::Eq, typ: il::Type::W8, a: tagA, b: tagB,
        thenTarget: *tagBlock, thenArgs: &mut [],
        elseTarget: *mergeBlock, elseArgs: falseArgs,
    });
    addPredecessor(self, tagBlock, currentBlock(self));
    addPredecessor(self, mergeBlock, currentBlock(self));

    // Create comparison blocks for each non-void variant and build switch cases.
    // Void variants jump directly to merge with `true`.
    let trueArgs = try allocVal(self, il::Val::Imm(1));
    let cases = try! alloc::allocRawSlice(
        self.low.fnArena, @sizeOf(il::SwitchCase), @alignOf(il::SwitchCase), unionInfo.variants.len as u32
    ) as *unsafe mut [il::SwitchCase];

    let mut caseBlocks: [?BlockId; resolver::MAX_UNION_VARIANTS] = undefined;
    for variant, i in unionInfo.variants {
        if variant.valueType == resolver::Type::Void {
            set cases[i] = il::SwitchCase {
                value: i as i64,
                target: *mergeBlock,
                args: trueArgs
            };
            set caseBlocks[i] = nil;
        } else {
            let payloadBlock = try createBlock(self, "eq#payload");
            set cases[i] = il::SwitchCase {
                value: i as i64,
                target: *payloadBlock,
                args: &mut []
            };
            set caseBlocks[i] = payloadBlock;
        }
    }

    // Emit switch in @tag block. Default arm is unreachable since we cover all variants.
    let unreachableBlock = try createBlock(self, "eq#unreachable");
    try switchToAndSeal(self, tagBlock);
    emit(self, il::Instr::Switch {
        val: tagA,
        defaultTarget: *unreachableBlock,
        defaultArgs: &mut [],
        cases
    });

    // Add predecessor edges for switch targets.
    addPredecessor(self, unreachableBlock, tagBlock);
    for i in 0..unionInfo.variants.len {
        if let caseBlock = caseBlocks[i] {
            addPredecessor(self, caseBlock, tagBlock);
        } else {
            addPredecessor(self, mergeBlock, tagBlock);
        }
    }
    let valOffset = unionInfo.valOffset as i32;

    // Emit payload comparison blocks for non-void variants.
    for variant, i in unionInfo.variants {
        if let caseBlock = caseBlocks[i] {
            try switchToAndSeal(self, caseBlock);
            let payloadEq = try emitEqAtOffset(
                self, a, b, offset + valOffset, variant.valueType
            );
            try emitJmpWithArg(self, mergeBlock, payloadEq);
        }
    }
    // Emit unreachable block.
    try switchToAndSeal(self, unreachableBlock);
    emit(self, il::Instr::Unreachable);

    try switchToAndSeal(self, mergeBlock);
    return il::Val::Reg(resultReg);
}

/// Compare two array values for equality, element by element.
unsafe fn lowerArrayEq(
    self: &mut FnLowerer,
    arr: resolver::ArrayType,
    a: il::Reg,
    b: il::Reg,
    offset: i32
) -> il::Val throws (LowerError) {
    let elemLayout = resolver::getTypeLayout(*arr.item);
    let stride = elemLayout.size as i32;
    let mut result: ?il::Val = nil;

    for i in 0..arr.length {
        let elemOffset = offset + (i as i32) * stride;
        let cmp = try emitEqAtOffset(self, a, b, elemOffset, *arr.item);
        set result = emitLogicalAnd(self, result, cmp);
    }
    if let r = result {
        return r;
    }
    // Empty arrays are always equal.
    return il::Val::Imm(1);
}

/// Compare two aggregate values for equality.
unsafe fn lowerAggregateEq(
    self: &mut FnLowerer,
    typ: resolver::Type,
    a: il::Reg,
    b: il::Reg,
    offset: i32
) -> il::Val throws (LowerError) {
    match typ {
        case resolver::Type::Slice { item, mutable, .. } =>
            return try lowerSliceEq(self, item, mutable, a, b, offset),
        case resolver::Type::Optional(inner) => {
            if let case resolver::Type::Slice { item, mutable, .. } = *inner {
                // Optional slices use null pointer optimization.
                return try lowerSliceEq(self, item, mutable, a, b, offset);
            }
            return try lowerOptionalEq(self, *inner, a, b, offset);
        }
        case resolver::Type::Array(arr) =>
            return try lowerArrayEq(self, arr, a, b, offset),
        case resolver::Type::Nominal(resolver::NominalType::Record(recInfo)) =>
            return try lowerRecordEq(self, recInfo, a, b, offset),
        case resolver::Type::Nominal(resolver::NominalType::Union(unionInfo)) =>
            return try lowerUnionEq(self, unionInfo, a, b, offset),
        else => {
            let recInfo = recordInfoFromType(typ) else {
                throw LowerError::ExpectedRecord;
            };
            return try lowerRecordEq(self, recInfo, a, b, offset);
        }
    }
}

/// Lower a record literal expression. Handles both plain records and union variant
/// record literals like `Union::Variant { field: value }`.
unsafe fn lowerRecordLit(self: &mut FnLowerer, node: *ast::Node, lit: ast::RecordLit) -> il::Val throws (LowerError) {
    let typ = try typeOf(self, node);
    match typ {
        case resolver::Type::Nominal(resolver::NominalType::Record(recInfo)) => {
            let dst = try emitReserve(self, typ);
            try lowerRecordFields(self, dst, &recInfo, lit.fields, 0);

            return il::Val::Reg(dst);
        }
        case resolver::Type::Nominal(resolver::NominalType::Union(_)) => {
            let typeName = lit.typeName else {
                throw LowerError::ExpectedVariant;
            };
            let sym = try symOf(self, typeName);
            let case resolver::SymbolData::Variant { type: payloadType, index, .. } = sym.data else {
                throw LowerError::ExpectedVariant;
            };
            let recInfo = recordInfoFromType(payloadType) else {
                throw LowerError::ExpectedRecord;
            };
            let unionInfo = unionInfoFromType(typ) else {
                throw LowerError::MissingMetadata;
            };
            let valOffset = unionInfo.valOffset as i32;
            let dst = try emitReserve(self, typ);

            emitStoreW8At(self, il::Val::Imm(index as i64), dst, TVAL_TAG_OFFSET);
            try lowerRecordFields(self, dst, &recInfo, lit.fields, valOffset);

            return il::Val::Reg(dst);
        }
        else => throw LowerError::UnexpectedType(typ),
    }
}

/// Lower fields of a record literal into a destination register.
/// The `offset` is added to each field's offset when storing.
unsafe fn lowerRecordFields(
    self: &mut FnLowerer,
    dst: il::Reg,
    recInfo: &resolver::RecordType,
    fields: *[*ast::Node],
    offset: i32
) throws (LowerError) {
    for fieldNode, i in fields {
        let case ast::NodeValue::RecordLitField(field) = fieldNode.value else {
            throw LowerError::UnexpectedNodeValue(fieldNode);
        };
        let mut fieldIdx: u32 = i;
        if recInfo.labeled {
            let idx = resolver::recordFieldIndexFor(self.low.resolver, fieldNode) else {
                throw LowerError::MissingMetadata;
            };
            set fieldIdx = idx;
        }
        // Skip `undefined` fields, they need no initialization.
        // Emitting a blit from an uninitialised reserve produces a
        // phantom SSA source value that the backend cannot handle.
        if not isUndef(field.value) {
            let fieldTy = recInfo.fields[fieldIdx].fieldType;
            let fieldVal = try lowerExpr(self, field.value);
            try emitStore(self, dst, offset + recInfo.fields[fieldIdx].offset, fieldTy, fieldVal);
        }
    }
}

/// Lower an unlabeled record constructor call.
unsafe fn lowerRecordCtor(self: &mut FnLowerer, nominal: *unsafe resolver::NominalType, args: *[*ast::Node]) -> il::Val throws (LowerError) {
    let case resolver::NominalType::Record(recInfo) = *nominal else {
        throw LowerError::ExpectedRecord;
    };
    let typ = resolver::Type::Nominal(nominal);
    let dst = try emitReserve(self, typ);

    for argNode, i in args {
        // Skip `undefined` arguments.
        if not isUndef(argNode) {
            let fieldTy = recInfo.fields[i].fieldType;
            let argVal = try lowerExpr(self, argNode);
            try emitStore(self, dst, recInfo.fields[i].offset, fieldTy, argVal);
        }
    }
    return il::Val::Reg(dst);
}

/// Lower an array literal expression like `[1, 2, 3]`.
unsafe fn lowerArrayLit(self: &mut FnLowerer, node: *ast::Node, elements: *[*ast::Node]) -> il::Val
    throws (LowerError)
{
    let typ = try typeOf(self, node);
    let case resolver::Type::Array(arrInfo) = typ else {
        throw LowerError::ExpectedArray;
    };
    let elemTy = *arrInfo.item;
    let elemLayout = resolver::getTypeLayout(elemTy);
    let dst = try emitReserve(self, typ);

    for elemNode, i in elements {
        let elemVal = try lowerExpr(self, elemNode);
        let offset = i * elemLayout.size;

        try emitStore(self, dst, offset as i32, elemTy, elemVal);
    }
    return il::Val::Reg(dst);
}

/// Lower an array repeat literal expression like `[42; 3]`.
/// Unrolls the initialization at compile time.
// TODO: Beyond a certain length, lower this to a loop.
unsafe fn lowerArrayRepeatLit(self: &mut FnLowerer, node: *ast::Node, repeat: ast::ArrayRepeatLit) -> il::Val
    throws (LowerError)
{
    let typ = try typeOf(self, node);
    let case resolver::Type::Array(arrInfo) = typ else {
        throw LowerError::ExpectedArray;
    };
    let elemTy = *arrInfo.item;
    let length = arrInfo.length;
    let elemLayout = resolver::getTypeLayout(elemTy);
    let dst = try emitReserve(self, typ);

    // Evaluate the repeated item once.
    let repeatVal = try lowerExpr(self, repeat.item);

    // Unroll: store at each offset.
    for i in 0..length {
        let offset = i * elemLayout.size;
        try emitStore(self, dst, offset as i32, elemTy, repeatVal);
    }
    return il::Val::Reg(dst);
}

/// Lower a union constructor call like `Union::Variant(...)`.
unsafe fn lowerUnionCtor(self: &mut FnLowerer, node: *ast::Node, sym: *unsafe mut resolver::Symbol, call: ast::Call) -> il::Val
    throws (LowerError)
{
    let unionTy = try typeOf(self, node);
    let case resolver::SymbolData::Variant { type: payloadType, index, .. } = sym.data else {
        throw LowerError::ExpectedVariant;
    };
    let unionInfo = unionInfoFromType(unionTy) else {
        throw LowerError::MissingMetadata;
    };
    let valOffset = unionInfo.valOffset as i32;
    let mut payloadVal: ?il::Val = nil;
    if payloadType <> resolver::Type::Void {
        let case resolver::Type::Nominal(payloadNominal) = payloadType else {
            throw LowerError::MissingMetadata;
        };
        set payloadVal = try lowerRecordCtor(self, payloadNominal, call.args);
    }
    return try buildTagged(self, resolver::getTypeLayout(unionTy), index as i64, payloadVal, payloadType, 1, valOffset);
}

/// Lower a field access into a pointer to the field.
unsafe fn lowerFieldRef(self: &mut FnLowerer, access: ast::Access) -> FieldRef throws (LowerError) {
    let parentTy = try typeOf(self, access.parent);
    let subjectTy = resolver::autoDeref(parentTy);
    let fieldIdx = resolver::recordFieldIndexFor(self.low.resolver, access.child) else {
        throw LowerError::MissingMetadata;
    };
    let fieldInfo = resolver::getRecordField(subjectTy, fieldIdx) else {
        throw LowerError::FieldNotFound;
    };
    let baseVal = try lowerExpr(self, access.parent);
    let baseReg = emitValToReg(self, baseVal);

    return FieldRef {
        base: baseReg,
        offset: fieldInfo.offset,
        fieldType: fieldInfo.fieldType,
    };
}

/// Lower a field access expression.
unsafe fn lowerFieldAccess(self: &mut FnLowerer, access: ast::Access) -> il::Val throws (LowerError) {
    let fieldRef = try lowerFieldRef(self, access);
    return emitRead(self, fieldRef.base, fieldRef.offset, fieldRef.fieldType);
}

/// Compute data pointer and element count for a range into a container.
/// Used by both slice range expressions (`&a[start..end]`) and slice
/// assignments (`a[start..end] = value`).
unsafe fn resolveSliceRangePtr(
    self: &mut FnLowerer,
    container: *ast::Node,
    range: ast::Range,
    info: resolver::SliceRangeInfo
) -> SliceRangeResult throws (LowerError) {
    let baseVal = try lowerExpr(self, container);
    let baseReg = emitValToReg(self, baseVal);

    // Extract data pointer and container length.
    let mut dataReg = baseReg;
    let mut containerLen: il::Val = undefined;
    let mut capacity: il::Val = undefined;
    if let cap = info.capacity { // Slice from array.
        set containerLen = il::Val::Imm(cap as i64);
        set capacity = containerLen;
    } else { // Slice from slice.
        set dataReg = loadSlicePtr(self, baseReg);
        set containerLen = loadSliceLen(self, baseReg);
        set capacity = loadSliceCap(self, baseReg);
    }

    // Compute range bounds.
    let mut startVal: il::Val = il::Val::Imm(0);
    if let start = range.start {
        set startVal = try lowerExpr(self, start);
    }
    let mut endVal = containerLen;
    if let end = range.end {
        set endVal = try lowerExpr(self, end);
    }

    // Runtime slice bounds checks for dynamic range expressions.
    if not isLtEq(startVal, containerLen) {
        try emitTrapIfLt(self, il::Type::W32, containerLen, startVal);
    }
    if not isLtEq(endVal, containerLen) {
        try emitTrapIfLt(self, il::Type::W32, containerLen, endVal);
    }
    if startVal <> il::Val::Imm(0) and endVal <> containerLen and not isLtEq(startVal, endVal) {
        try emitTrapIfLt(self, il::Type::W32, endVal, startVal);
    }

    // If the start value is known to be zero, the count is just the end
    // value. Otherwise, we have to compute it.
    let mut count = endVal;

    // Only compute range offset and count if the start value is not
    // statically known to be zero.
    if startVal <> il::Val::Imm(0) {
        // Offset the data pointer by the start value.
        set dataReg = emitElem(
            self, resolver::getTypeLayout(*info.itemType).size, dataReg, startVal
        );
        // Compute the count as `end - start`.
        let lenReg = nextReg(self);
        emit(self, il::Instr::BinOp {
            op: il::BinOp::Sub,
            typ: il::Type::W32,
            dst: lenReg,
            a: endVal,
            b: startVal,
        });
        set count = il::Val::Reg(lenReg);
        if capacity == endVal {
            set capacity = count;
        } else {
            set capacity = emitTypedBinOp(self, il::BinOp::Sub, il::Type::W32, capacity, startVal);
        }
    }
    return SliceRangeResult { dataReg, count, capacity };
}

/// Lower a slice range expression into a slice header value.
unsafe fn lowerSliceRange(
    self: &mut FnLowerer,
    container: *ast::Node,
    range: ast::Range,
    sliceNode: *ast::Node
) -> il::Val throws (LowerError) {
    let info = resolver::sliceRangeInfoFor(self.low.resolver, sliceNode) else {
        throw LowerError::MissingMetadata;
    };
    let r = try resolveSliceRangePtr(self, container, range, info);
    return try buildSliceValue(
        self, info.itemType, info.mutable, il::Val::Reg(r.dataReg), r.count, r.capacity
    );
}

/// Lower an address-of (`&x`) expression.
unsafe fn lowerAddressOf(self: &mut FnLowerer, node: *ast::Node, addr: ast::AddressOf) -> il::Val throws (LowerError) {
    // Handle subscript: `&ary[i]` or `&ary[start..end]`.
    if let case ast::NodeValue::Subscript { container, index } = addr.target.value {
        if let case ast::NodeValue::Range(range) = index.value {
            return try lowerSliceRange(self, container, range, node);
        }
        let result = try lowerElemPtr(self, container, index);

        return il::Val::Reg(result.elemReg);
    }
    // Handle field address: `&x.field`.
    if let case ast::NodeValue::FieldAccess(access) = addr.target.value {
        let fieldRef = try lowerFieldRef(self, access);
        let ptr = emitPtrOffset(self, fieldRef.base, fieldRef.offset);

        return il::Val::Reg(ptr);
    }
    // Handle variable address: `&x`
    if let case ast::NodeValue::Ident(_) = addr.target.value {
        if let v = lookupLocalVar(self, addr.target) {
            let val = try useVar(self, v);
            let typ = try typeOf(self, addr.target);
            // For aggregates, the value is already a pointer.
            if isAggregateType(typ) {
                return val;
            }
            // For scalars, if we've already materialized a stack slot for this
            // variable, the SSA value is that slot pointer.
            if self.vars[*v].addressTaken {
                // Already address-taken; return existing stack pointer.
                return val;
            }
            // Materialize a stack slot using the declaration's resolved
            // layout so `align(N)` on locals is honored.
            let layout = resolver::getLayout(self.low.resolver, addr.target, typ);
            let slot = emitReserveLayout(self, layout);
            try emitStore(self, slot, 0, typ, val);
            let stackVal = il::Val::Reg(slot);

            set self.vars[*v].addressTaken = true;
            defVar(self, v, stackVal);

            return stackVal;
        }
        // Fall back to symbol lookup for constants/statics.
        if let sym = resolver::nodeData(self.low.resolver, addr.target).sym {
            return il::Val::Reg(emitDataAddr(self, sym));
        } else {
            throw LowerError::MissingSymbol(node);
        }
    }
    // Handle dereference address: `&(*ptr) = ptr`.
    if let case ast::NodeValue::Deref(target) = addr.target.value {
        return try lowerExpr(self, target);
    }
    // Array literals become slices when their address is taken.
    match addr.target.value {
        case ast::NodeValue::ArrayLit(_),
             ast::NodeValue::ArrayRepeatLit(_) =>
        {
            return try lowerArrayLiteralSlice(self, node, addr.target);
        }
        else => {}
    }
    throw LowerError::UnexpectedNodeValue(addr.target);
}

/// Lower an addressed array literal as a slice.
unsafe fn lowerArrayLiteralSlice(
    self: &mut FnLowerer,
    sliceNode: *ast::Node,
    arrayNode: *ast::Node
) -> il::Val throws (LowerError) {
    let sliceTy = try typeOf(self, sliceNode);
    let case resolver::Type::Slice { item, mutable, .. } = sliceTy else {
        throw LowerError::UnexpectedType(sliceTy);
    };
    let arrayTy = try typeOf(self, arrayNode);
    let case resolver::Type::Array(arrayInfo) = arrayTy else {
        throw LowerError::ExpectedArray;
    };
    let length = arrayInfo.length;
    if length == 0 {
        return try buildSliceValue(
            self, item, mutable, il::Val::Imm(0), il::Val::Imm(0), il::Val::Imm(0)
        );
    }
    if resolver::isConstExpr(self.low.resolver, arrayNode) {
        let fnName = self.fnName;
        let mut b = dataBuilder(self.low.allocator);
        match arrayNode.value {
            case ast::NodeValue::ArrayLit(elements) =>
                try lowerConstArrayLitInto(self.low, elements, arrayTy, fnName, &mut b),
            case ast::NodeValue::ArrayRepeatLit(repeat) =>
                try lowerConstArrayRepeatInto(self.low, repeat, arrayTy, fnName, &mut b),
            else => throw LowerError::UnexpectedNodeValue(arrayNode),
        }
        let result = dataBuilderFinish(b);
        let alignment = resolver::getTypeLayout(*item).alignment;
        return try lowerConstDataAsSlice(
            self, &result, alignment, not mutable,
            item, mutable, length
        );
    }
    let data = try lowerExpr(self, arrayNode);
    let count = il::Val::Imm(length as i64);
    return try buildSliceValue(self, item, mutable, data, count, count);
}

/// Lower the common element pointer computation for subscript operations.
/// Handles both arrays and slices by resolving the container type, extracting
/// the data pointer (for slices), and emitting an [`il::Instr::Elem`] to compute
/// the element address.
unsafe fn lowerElemPtr(
    self: &mut FnLowerer, container: *ast::Node, index: *ast::Node
) -> ElemPtrResult throws (LowerError) {
    let containerTy = try typeOf(self, container);
    let subjectTy = resolver::autoDeref(containerTy);
    let baseVal = try lowerExpr(self, container);
    let indexVal = try lowerExpr(self, index);
    let baseReg = emitValToReg(self, baseVal);

    let mut dataReg = baseReg;
    let mut elemType: resolver::Type = undefined;

    match subjectTy {
        case resolver::Type::Slice { item, .. } => {
            set elemType = *item;
            let sliceLen = loadSliceLen(self, baseReg);
            // Runtime safety check: index must be strictly less than slice length.
            try emitTrapUnlessCmp(self, il::CmpOp::Ult, il::Type::W32, indexVal, sliceLen);

            set dataReg = loadSlicePtr(self, baseReg);
        }
        case resolver::Type::Array(arrInfo) => {
            set elemType = *arrInfo.item;
            // Runtime safety check: index must be strictly less than array length.
            // Skip when the index is a compile-time constant, since we check
            // that in the resolver.
            if not resolver::isConstExpr(self.low.resolver, index) {
                let arrLen = il::Val::Imm(arrInfo.length as i64);
                try emitTrapUnlessCmp(self, il::CmpOp::Ult, il::Type::W32, indexVal, arrLen);
            }
        }
        else => throw LowerError::ExpectedSliceOrArray,
    }
    let elemLayout = resolver::getTypeLayout(elemType);
    let elemReg = emitElem(self, elemLayout.size, dataReg, indexVal);

    return ElemPtrResult { elemReg, elemType };
}

/// Lower a dereference expression.
/// Handles both pointer deref (`*ptr`) and record deref (`*r` on single-field
/// unlabeled record). Both read at offset 0 using the resolver-assigned type.
unsafe fn lowerDeref(self: &mut FnLowerer, node: *ast::Node, target: *ast::Node) -> il::Val throws (LowerError) {
    let type = try typeOf(self, node);
    let ptrVal = try lowerExpr(self, target);
    let ptrReg = emitValToReg(self, ptrVal);

    return emitRead(self, ptrReg, 0, type);
}

/// Lower a subscript expression.
unsafe fn lowerSubscript(self: &mut FnLowerer, node: *ast::Node, container: *ast::Node, index: *ast::Node) -> il::Val
    throws (LowerError)
{
    if let case ast::NodeValue::Range(_) = index.value {
        panic "lowerSubscript: range subscript must use address-of (&)";
    }
    let result = try lowerElemPtr(self, container, index);

    return emitRead(self, result.elemReg, 0, result.elemType);
}

/// Lower a let binding.
unsafe fn lowerLet(self: &mut FnLowerer, node: *ast::Node, l: ast::Let) throws (LowerError) {
    // Evaluate value.
    let val = try lowerExpr(self, l.value);
    if blockHasTerminator(self) {
        return;
    }
    // Handle placeholder pattern: `let _ = expr;`
    if let case ast::NodeValue::Placeholder = l.ident.value {
        return;
    }
    let case ast::NodeValue::Ident(name) = l.ident.value else {
        throw LowerError::ExpectedIdentifier;
    };
    let typ = try typeOf(self, l.value);
    let ilType = ilType(self.low, typ);
    let mut varVal = val;

    // Aggregates with persistent storage need a local copy to avoid aliasing.
    // Temporaries such as literals or call results can be adopted directly.
    // This is because aggregates are represented as memory addresses
    // internally, even though they have value semantics, so without an explicit
    // copy, only the address is is written. Function calls on the other hand
    // reserve their own local stack space, so copying would be redundant.
    // Void variant literals (e.g. `Option::None`) use scope access syntax and
    // are flagged as place expressions, but they are freshly constructed
    // temporaries with no persistent storage.
    if isAggregateType(typ) and
        ast::isPlaceExpr(l.value) and
        voidVariantIndex(self.low.resolver, l.value) == nil {
        set varVal = try emitStackVal(self, typ, val);
    }

    // If the resolver determined that this variable's address is taken
    // anywhere in the function, allocate a stack slot immediately so the
    // SSA value is always a pointer. This avoids mixing integer and pointer
    // values in loop phis when `&var` or `&mut var` appears inside a loop.
    if not isAggregateType(typ) {
        if let sym = resolver::nodeData(self.low.resolver, node).sym {
            if let case resolver::SymbolData::Value { addressTaken, .. } = sym.data; addressTaken {
                let layout = resolver::getLayout(self.low.resolver, node, typ);
                let slot = emitReserveLayout(self, layout);
                try emitStore(self, slot, 0, typ, varVal);

                let v = newVar(self, name, ilType, l.mutable, il::Val::Reg(slot));
                set self.vars[*v].addressTaken = true;

                return;
            }
        }
    }
    let _ = newVar(self, name, ilType, l.mutable, varVal);
}

/// Lower an if statement: `if <cond> { <then> } else { <else> }`.
///
/// With else branch:
///
///     @entry -> (true)  @then ---> @merge <--.
///         |                                   )
///         `---> (false) @else ---------------'
///
/// Without else branch:
///
///     @entry -> (true)  @then ---> @end <--.
///         |                                 )
///         `---- (false) -------------------'
///
unsafe fn lowerIf(self: &mut FnLowerer, i: ast::If) throws (LowerError) {
    let thenBlock = try createBlock(self, "then");

    if let elseNode = i.elseBranch { // If-else case.
        let elseBlock = try createBlock(self, "else");
        try emitCondBranch(self, i.condition, thenBlock, elseBlock);

        // Both @then and @else have exactly one predecessor (@entry), so we can
        // seal them immediately.
        try sealBlock(self, thenBlock);
        try sealBlock(self, elseBlock);

        // The merge block is created lazily by [`emitMergeIfUnterminated`]. We
        // only need it if at least one branch doesn't diverge (i.e., needs to
        // continue execution after the `if`). If both branches diverge (eg.
        // both `return`), no merge block is created and control flow
        // doesn't continue past the `if` statement.
        let mut mergeBlock: ?BlockId = nil;

        // Lower the @then block: switch to it, emit its code, then jump to
        // merge if the block doesn't diverge.
        switchToBlock(self, thenBlock);
        try lowerBlock(self, i.thenBranch);
        try emitMergeIfUnterminated(self, &mut mergeBlock);

        // Lower the @else block similarly.
        switchToBlock(self, elseBlock);
        try lowerBlock(self, elseNode);
        try emitMergeIfUnterminated(self, &mut mergeBlock);

        // If a merge block was created (at least one branch flows into it),
        // switch to it for subsequent code. The merge block's predecessors
        // are the branches that jumped to it, so we seal it now.
        // If the merge block is `nil`, both branches diverged and there's no
        // continuation point.
        if let blk = mergeBlock {
            try switchToAndSeal(self, blk);
        }
    } else { // If without `else`.
        // The false branch goes directly to @end, which also serves as the
        // merge point after @then completes.
        let endBlock = try createBlock(self, "merge");

        try emitCondBranch(self, i.condition, thenBlock, endBlock);

        // @then has one predecessor (@entry), seal it immediately.
        // @end is not sealed yet because @then might also jump to it.
        try sealBlock(self, thenBlock);

        // Lower the @then block, then jump to @end and seal it.
        // Unlike the if-else case, @end is always created because there is no
        // else branch that can diverge.
        switchToBlock(self, thenBlock);
        try lowerBlock(self, i.thenBranch);
        try emitJmpAndSeal(self, endBlock);

        // Continue execution at @end.
        try switchToAndSeal(self, endBlock);
    }
}

/// Lower an assignment target that designates a memory location, and return
/// the address to store through. Returns `nil` without emitting anything for
/// targets that aren't memory-backed, such as locals tracked in SSA.
unsafe fn lowerPlace(self: &mut FnLowerer, target: *ast::Node) -> ?FieldRef throws (LowerError) {
    match target.value {
        case ast::NodeValue::ScopeAccess(_) => {
            let sym = try symOf(self, target);
            let case resolver::SymbolData::Value { type, .. } = sym.data else {
                throw LowerError::ImmutableAssignment;
            };
            return FieldRef { base: emitDataAddr(self, sym), offset: 0, fieldType: type };
        }
        case ast::NodeValue::FieldAccess(access) => {
            return try lowerFieldRef(self, access);
        }
        case ast::NodeValue::Deref(pointer) => {
            // Dereference: `*ptr` or `*r` on a single-field unlabeled record.
            // Both address offset 0 using the resolver-assigned type.
            let base = emitValToReg(self, try lowerExpr(self, pointer));
            return FieldRef { base, offset: 0, fieldType: try typeOf(self, target) };
        }
        case ast::NodeValue::Subscript { container, index } => {
            // Array or slice element: `arr[i]`.
            let elem = try lowerElemPtr(self, container, index);
            return FieldRef { base: elem.elemReg, offset: 0, fieldType: elem.elemType };
        }
        else => return nil,
    }
}

/// Lower a compound assignment whose target is memory-backed, and report
/// whether it was handled. Nothing is emitted when it isn't.
///
/// Compound assignments share their target node with the left operand of the
/// desugared binary expression. Resolve that place once so side effects in a
/// dereference, field parent, or subscript index are not repeated for the store.
unsafe fn lowerCompoundAssign(
    self: &mut FnLowerer, expr: *ast::Node, binop: ast::BinOp
) -> bool throws (LowerError) {
    let place = try lowerPlace(self, binop.left) else return false;
    let current = emitRead(self, place.base, place.offset, place.fieldType);
    let left = try applyCoercion(self, binop.left, current);
    let right = try lowerExpr(self, binop.right);
    let exprType = try typeOf(self, expr);
    let result = emitScalarBinOp(
        self, binop.op, ilType(self.low, exprType), left, right, isUnsignedType(exprType)
    );
    let assigned = try applyCoercion(self, expr, result);
    try emitStore(self, place.base, place.offset, place.fieldType, assigned);

    return true;
}

/// Lower an assignment statement.
unsafe fn lowerAssign(self: &mut FnLowerer, node: *ast::Node, a: ast::Assign) throws (LowerError) {
    // Slice assignment: `slice[range] = value`.
    if let info = resolver::sliceRangeInfoFor(self.low.resolver, node) {
        let case ast::NodeValue::Subscript { container, index } = a.left.value
            else panic "lowerAssign: slice assign without subscript";
        let case ast::NodeValue::Range(range) = index.value
            else panic "lowerAssign: slice assign without range";
        try lowerSliceAssign(self, a.right, container, range, info);

        return;
    }
    // Compound assignments are represented as `target = target op rhs`, with
    // the exact same target node used in both places. Memory-backed targets
    // must be resolved once so calls in the target are evaluated once.
    if let case ast::NodeValue::BinOp(binop) = a.right.value {
        if binop.left == a.left and try lowerCompoundAssign(self, a.right, binop) {
            return;
        }
    }
    // Evaluate assignment value.
    let rhs = try lowerExpr(self, a.right);

    match a.left.value {
        case ast::NodeValue::Ident(_) => {
            // First try local variable lookup.
            if let v = lookupLocalVar(self, a.left) {
                if not getVar(self, v).mutable {
                    throw LowerError::ImmutableAssignment;
                }
                let leftTy = try typeOf(self, a.left);
                if isAggregateType(leftTy) or getVar(self, v).addressTaken {
                    // Aggregates and address-taken scalars are represented as
                    // pointers to stack memory. Store through the pointer.
                    let val = try useVar(self, v);
                    let dst = emitValToReg(self, val);

                    try emitStore(self, dst, 0, leftTy, rhs);
                } else {
                    // Scalars are tracked directly in SSA. Each assignment
                    // records a new SSA value.
                    defVar(self, v, rhs);
                }
            } else {
                // Fall back to static variable assignment.
                try lowerStaticAssign(self, a.left, rhs);
            }
        }
        else => {
            let place = try lowerPlace(self, a.left) else {
                throw LowerError::UnexpectedNodeValue(a.left);
            };
            try emitStore(self, place.base, place.offset, place.fieldType, rhs);
        }
    }
}

/// Lower `slice[range] = value`.
unsafe fn lowerSliceAssign(
    self: &mut FnLowerer,
    rhs: *ast::Node,
    container: *ast::Node,
    range: ast::Range,
    info: resolver::SliceRangeInfo
) throws (LowerError) {
    let r = try resolveSliceRangePtr(self, container, range, info);
    let elemSize = resolver::getTypeLayout(*info.itemType).size;
    let rhsTy = try typeOf(self, rhs);

    if let case resolver::Type::Slice { .. } = rhsTy {
        // Copy from source slice.
        let srcReg = emitValToReg(self, try lowerExpr(self, rhs));
        let srcData = loadSlicePtr(self, srcReg);
        let srcLen = loadSliceLen(self, srcReg);

        // Trap if source and destination lengths differ.
        try emitTrapUnlessCmp(self, il::CmpOp::Eq, il::Type::W32, r.count, srcLen);

        let bytes = emitTypedBinOp(
            self, il::BinOp::Mul, il::Type::W32, r.count, il::Val::Imm(elemSize as i64)
        );
        try emitByteCopyLoop(self, r.dataReg, srcData, bytes, "copy");
    } else {
        // Fill with scalar value.
        let fillVal = try lowerExpr(self, rhs);
        try emitFillLoop(self, r.dataReg, fillVal, r.count, *info.itemType, elemSize);
    }
}

/// Emit a typed fill loop: `for i in 0..count { dst[i * stride] = value; }`.
unsafe fn emitFillLoop(
    self: &mut FnLowerer,
    dst: il::Reg,
    value: il::Val,
    count: il::Val,
    elemType: resolver::Type,
    elemSize: u32
) throws (LowerError) {
    let iReg = nextReg(self);
    let header = try createBlockWithParam(
        self, "fill", il::Param { value: iReg, type: il::Type::W32 }
    );
    let body = try createBlock(self, "fill");
    let done = try createBlock(self, "fill");

    try emitJmpWithArg(self, header, il::Val::Imm(0));
    switchToBlock(self, header);
    try emitBrCmp(self, il::CmpOp::Ult, il::Type::W32, il::Val::Reg(iReg), count, body, done);

    try switchToAndSeal(self, body);
    let dstElem = emitElem(self, elemSize, dst, il::Val::Reg(iReg));
    try emitStore(self, dstElem, 0, elemType, value);

    let nextI = emitTypedBinOp(
        self, il::BinOp::Add, il::Type::W32, il::Val::Reg(iReg), il::Val::Imm(1)
    );
    try emitJmpWithArg(self, header, nextI);
    try sealBlock(self, header);
    try switchToAndSeal(self, done);
}

///////////////////
// Loop Lowering //
///////////////////

// All loop forms are lowered to a common structure:
//
// - Loop header block: evaluates condition if any, branches to body or exit.
// - Body block: executes loop body, may contain break/continue.
// - Step block (for `for` loops): increments counter, jumps back to header.
// - Exit block: target for break statements and normal loop exit.
//
// The loop stack tracks break/continue targets for nested loops.

/// Lower an infinite loop: `loop { <body> }`.
///
///   @entry -> @loop -> @loop
///               |
///               `----> @end
///
unsafe fn lowerLoop(self: &mut FnLowerer, body: *ast::Node) throws (LowerError) {
    let loopBlock = try createBlock(self, "loop");
    let endBlock = try createBlock(self, "merge");

    // Enter the loop with the given break and continue targets.
    // `break` jumps to `endBlock`,
    // `continue` jumps to `loopBlock`.
    enterLoop(self, endBlock, loopBlock);
    // Switch to and jump to the loop block, then lower all loop body statements into it.
    try switchAndJumpTo(self, loopBlock);
    try lowerBlock(self, body);

    // If the loop body doesn't diverge, jump back to the start.
    // This creates the infinite loop.
    // All predecessors are known, we can seal the loop block.
    try emitJmpAndSeal(self, loopBlock);
    // Exit the loop.
    exitLoop(self);

    // Only seal end block if it's actually reachable (ie. has predecessors).
    // If the loop has no breaks and only exits via return, the end block
    // remains unreachable and isn't added to the CFG.
    if getBlock(self, endBlock).preds.len > 0 {
        try switchToAndSeal(self, endBlock);
    }
}

/// Lower a while loop: `while <cond> { <body> }`.
///
///   @entry -> @loop -> (true)  @body -> @loop
///               |
///               `----> (false) @end
///
unsafe fn lowerWhile(self: &mut FnLowerer, w: ast::While) throws (LowerError) {
    let whileBlock = try createBlock(self, "while");
    let bodyBlock = try createBlock(self, "body");
    let endBlock = try createBlock(self, "merge");

    enterLoop(self, endBlock, whileBlock);

    // Loop condition.
    try switchAndJumpTo(self, whileBlock);

    // Based on the condition, either jump to the body,
    // or to the end of the loop.
    try emitCondBranch(self, w.condition, bodyBlock, endBlock);

    // Lower loop body and jump back to loop condition check.
    try switchToAndSeal(self, bodyBlock);
    try lowerBlock(self, w.body);
    try emitJmpAndSeal(self, whileBlock);

    try switchToAndSeal(self, endBlock);
    exitLoop(self);
}

/// Emit an increment of a variable by `1`.
unsafe fn emitIncrement(self: &mut FnLowerer, v: Var, typ: il::Type) throws (LowerError) {
    let cur = try useVar(self, v);
    let next = nextReg(self);

    emit(self, il::Instr::BinOp { op: il::BinOp::Add, typ, dst: next, a: cur, b: il::Val::Imm(1) });
    defVar(self, v, il::Val::Reg(next));
}

/// Common for-loop lowering for both range and collection iterators.
///
/// The step block is created lazily (after the loop body) so that it gets
/// a block index higher than all body blocks. This ensures the register
/// allocator processes definitions before uses in forward block order,
/// avoiding stale assignments when a value defined deep in the body flows
/// through the step block as a block argument.
unsafe fn lowerForLoop(self: &mut FnLowerer, iter: &ForIter, body: *ast::Node) throws (LowerError) {
    let loopBlock = try createBlock(self, "loop");
    let bodyBlock = try createBlock(self, "body");
    let endBlock = try createBlock(self, "merge");

    enterLoop(self, endBlock, nil);
    try switchAndJumpTo(self, loopBlock);

    // Emit condition check.
    match *iter {
        case ForIter::Range { valVar, endVal, valType, unsigned, .. } => {
            let curVal = try useVar(self, valVar);
            let cmp = il::CmpOp::Ult if unsigned else il::CmpOp::Slt;
            try emitBrCmp(self, cmp, valType, curVal, endVal, bodyBlock, endBlock);
        }
        case ForIter::Collection { idxVar, lengthVal, .. } => {
            let curIdx = try useVar(self, idxVar);
            try emitBrCmp(self, il::CmpOp::Slt, il::Type::W32, curIdx, lengthVal, bodyBlock, endBlock);
        }
    }
    // Switch to loop body.
    try switchToAndSeal(self, bodyBlock);

    // Emit element binding, only for collections.
    // Reads the element at the current index.
    if let case ForIter::Collection { valVar, idxVar, dataReg, elemType, .. } = *iter {
        if let v = valVar {
            let curIdx = try useVar(self, idxVar);
            let elemReg = emitElem(self, resolver::getTypeLayout(*elemType).size, dataReg, curIdx);
            let val = emitRead(self, elemReg, 0, *elemType);

            defVar(self, v, val);
        }
    }
    // Lower the loop body.
    try lowerBlock(self, body);

    // Check if a `continue` statement created a step block.
    let ctx = currentLoop(self) else panic;
    if let stepBlock = ctx.continueTarget {
        // Body has `continue` statements: jump to step block, emit increment there.
        try emitJmpAndSeal(self, stepBlock);
        switchToBlock(self, stepBlock);
    }
    // Otherwise, emit increment directly in the current block,
    // saving the jump to a separate step block.
    if not blockHasTerminator(self) {
        match *iter {
            case ForIter::Range { valVar, valType, indexVar, .. } => {
                try emitIncrement(self, valVar, valType);
                if let idxVar = indexVar {
                    try emitIncrement(self, idxVar, il::Type::W32);
                }
            }
            case ForIter::Collection { idxVar, .. } => {
                try emitIncrement(self, idxVar, il::Type::W32);
            }
        }
        try emitJmp(self, loopBlock);
    }
    exitLoop(self);

    try sealBlock(self, loopBlock);
    try sealBlock(self, endBlock);

    switchToBlock(self, endBlock);
}

/// Lower a `for` loop over a range, array, or slice.
unsafe fn lowerFor(self: &mut FnLowerer, node: *ast::Node, f: ast::For) throws (LowerError) {
    let savedVarsLen = enterVarScope(self);
    let info = resolver::forLoopInfoFor(self.low.resolver, node) else {
        throw LowerError::MissingMetadata;
    };
    match info {
        case resolver::ForLoopInfo::Range { valType, range, bindingName, indexName } => {
            let endExpr = range.end else {
                throw LowerError::MissingMetadata;
            };
            let mut startVal = il::Val::Imm(0);
            if let start = range.start {
                set startVal = try lowerExpr(self, start);
            }
            let endVal = try lowerExpr(self, endExpr);
            let iterType = ilType(self.low, *valType);
            let valVar = newVar(self, bindingName, iterType, false, startVal);

            let mut indexVar: ?Var = nil;
            if indexName <> nil { // Optional index always starts at zero.
                set indexVar = newVar(self, indexName, il::Type::W32, false, il::Val::Imm(0));
            }
            let iter = ForIter::Range {
                valVar, indexVar, endVal, valType: iterType,
                unsigned: isUnsignedType(*valType),
            };

            try lowerForLoop(self, &iter, f.body);
        }
        case resolver::ForLoopInfo::Collection { elemType, length, bindingName, indexName } => {
            let containerVal = try lowerExpr(self, f.iterable);
            let containerReg = emitValToReg(self, containerVal);

            let mut dataReg = containerReg;
            let mut lengthVal: il::Val = undefined;
            if let len = length { // Array (length is known).
                set lengthVal = il::Val::Imm(len as i64);
            } else { // Slice (length must be loaded).
                set lengthVal = loadSliceLen(self, containerReg);
                set dataReg = loadSlicePtr(self, containerReg);
            }
            // Declare index value binidng.
            let idxVar = newVar(self, indexName, il::Type::W32, false, il::Val::Imm(0));

            // Declare element value binding.
            let mut valVar: ?Var = nil;
            if bindingName <> nil {
                set valVar = newVar(
                    self,
                    bindingName,
                    ilType(self.low, *elemType),
                    false,
                    il::Val::Undef
                );
            }
            let iter = ForIter::Collection { valVar, idxVar, dataReg, lengthVal, elemType };

            try lowerForLoop(self, &iter, f.body);
        }
    }
    exitVarScope(self, savedVarsLen);
}

/// Lower a break statement.
unsafe fn lowerBreak(self: &mut FnLowerer) throws (LowerError) {
    let ctx = currentLoop(self) else {
        throw LowerError::OutsideOfLoop;
    };
    try emitJmp(self, ctx.breakTarget);
}

/// Lower a continue statement.
unsafe fn lowerContinue(self: &mut FnLowerer) throws (LowerError) {
    let block = try getOrCreateContinueBlock(self);
    try emitJmp(self, block);
}

/// Emit a return, blitting into the caller's return buffer if needed.
///
/// When the function has a return buffer parameter, the value is blitted
/// into the buffer and the buffer pointer is returned. Otherwise, the value is
/// returned directly.
unsafe fn emitRetVal(self: &mut FnLowerer, val: il::Val) throws (LowerError) {
    if let retReg = self.returnReg {
        let src = emitValToReg(self, val);
        let size = resolver::getResultLayout(*self.fnType.returnType, self.fnType.throwList).size
            if self.fnType.throwList.len > 0
            else resolver::getTypeLayout(*self.fnType.returnType).size;

        emit(self, il::Instr::Blit { dst: retReg, src, size: il::Val::Imm(size as i64) });
        emit(self, il::Instr::Ret { val: il::Val::Reg(retReg) });
    } else if isSmallAggregate(*self.fnType.returnType) {
        let src = emitValToReg(self, val);
        let dst = nextReg(self);

        emit(self, il::Instr::Load { typ: il::Type::W64, dst, src, offset: 0 });
        emit(self, il::Instr::Ret { val: il::Val::Reg(dst) });
    } else {
        emit(self, il::Instr::Ret { val });
    }
}

/// Lower a return statement.
unsafe fn lowerReturnStmt(self: &mut FnLowerer, node: *ast::Node, value: ?*ast::Node) throws (LowerError) {
    let mut val = il::Val::Undef;
    if let expr = value {
        set val = try lowerExpr(self, expr);
    }
    if blockHasTerminator(self) {
        return;
    }
    set val = try applyCoercion(self, node, val);
    try emitRetVal(self, val);
}

/// Lower a throw statement.
unsafe fn lowerThrowStmt(self: &mut FnLowerer, expr: *ast::Node) throws (LowerError) {
    assert self.fnType.throwList.len > 0;

    let errType = *self.fnType.throwList[0] if self.fnType.throwList.len == 1
        else try typeOf(self, expr);
    let tag = getOrAssignErrorTag(self.low, errType) as i64;
    let errVal = try lowerExpr(self, expr);
    let resultVal = try buildResult(self, tag, errVal, errType);

    try emitRetVal(self, resultVal);
}

/// Ensure a value is in a register (eg. for branch conditions).
unsafe fn emitValToReg(self: &mut FnLowerer, val: il::Val) -> il::Reg {
    match val {
        case il::Val::Reg(r) => return r,
        case il::Val::Imm(_), il::Val::DataSym(_), il::Val::FnAddr(_) => {
            let dst = nextReg(self);
            emit(self, il::Instr::Copy { dst, val });
            return dst;
        }
        case il::Val::Undef => {
            // TODO: We shouldn't hit this case, right? A register shouldn't be needed
            // if the value is undefined.
            return nextReg(self);
        }
    }
}

/// Lower a logical `and`/`or` with short-circuit evaluation.
///
/// Short-circuit evaluation skips evaluating the right operand when the left
/// operand already determines the result:
///
/// - In `a and b`, if `a` is false, result is false without evaluating `b`.
/// - In `a or b`, if `a` is true, result is true without evaluating `b`.
///
/// This matters when `b` has side effects, or is expensive to evaluate.
///
/// Example: `a and b`
///
///     @entry
///       br %a @then @else;
///     @then
///       // Evaluate b into %b
///       // ...
///       jmp @end(%b);
///     @else
///       jmp @end(0);                  // Skip evaluating b, result is false
///     @end(w8 %result)
///       ret %result;
///
/// Example: `a or b`
///
///     @entry
///       br %a @then @else;
///     @then
///       jmp @end(1);                  // Skip evaluating b, result is true
///     @else
///       // Evaluate b into %b
///       // ...
///       jmp @end(%b);
///     @end(w8 %result)
///       ret %result;
///
unsafe fn lowerLogicalOp(
    self: &mut FnLowerer,
    binop: ast::BinOp,
    thenLabel: *[u8],
    elseLabel: *[u8],
    mergeLabel: *[u8],
    op: LogicalOp
) -> il::Val throws (LowerError) {
    let thenBlock = try createBlock(self, thenLabel);
    let elseBlock = try createBlock(self, elseLabel);

    let resultReg = nextReg(self);
    let mergeBlock = try createBlockWithParam(
        self, mergeLabel, il::Param { value: resultReg, type: il::Type::W8 }
    );
    // Evaluate left operand and branch.
    try emitCondBranch(self, binop.left, thenBlock, elseBlock);

    // Block that skips evaluating `b`.
    let mut shortCircuitBlock: BlockId = undefined;
    // Block that evaluates `b`.
    let mut evalBlock: BlockId = undefined;
    // Result when short-circuiting (`0` or `1`).
    let mut shortCircuitVal: i64 = undefined;

    match op {
        case LogicalOp::And => {
            set shortCircuitBlock = elseBlock;
            set evalBlock = thenBlock;
            set shortCircuitVal = 0;
        }
        case LogicalOp::Or => {
            set shortCircuitBlock = thenBlock;
            set evalBlock = elseBlock;
            set shortCircuitVal = 1;
        }
    }
    // Emit short-circuit branch: jump to merge with constant result.
    try switchToAndSeal(self, shortCircuitBlock);
    try emitJmpWithArg(self, mergeBlock, il::Val::Imm(shortCircuitVal));

    // Emit evaluation branch: evaluate right operand and jump to merge.
    try switchToAndSeal(self, evalBlock);
    try emitJmpWithArg(self, mergeBlock, try lowerExpr(self, binop.right));

    try switchToAndSeal(self, mergeBlock);
    return il::Val::Reg(resultReg);
}

/// Lower a conditional expression (`thenExpr if condition else elseExpr`).
unsafe fn lowerCondExpr(self: &mut FnLowerer, node: *ast::Node, cond: ast::CondExpr) -> il::Val
    throws (LowerError)
{
    let typ = try typeOf(self, node);
    let thenBlock = try createBlock(self, "cond#then");
    let elseBlock = try createBlock(self, "cond#else");

    if isAggregateType(typ) {
        let dst = try emitReserve(self, typ);
        let layout = resolver::getTypeLayout(typ);
        try emitCondBranch(self, cond.condition, thenBlock, elseBlock);

        let mergeBlock = try createBlock(self, "cond#merge");
        try switchToAndSeal(self, thenBlock);

        let thenVal = emitValToReg(self, try lowerExpr(self, cond.thenExpr));
        emit(self, il::Instr::Blit { dst, src: thenVal, size: il::Val::Imm(layout.size as i64) });

        try emitJmp(self, mergeBlock);
        try switchToAndSeal(self, elseBlock);

        let elseVal = emitValToReg(self, try lowerExpr(self, cond.elseExpr));
        emit(self, il::Instr::Blit { dst, src: elseVal, size: il::Val::Imm(layout.size as i64) });

        try emitJmp(self, mergeBlock);
        try switchToAndSeal(self, mergeBlock);

        return il::Val::Reg(dst);
    } else {
        try emitCondBranch(self, cond.condition, thenBlock, elseBlock);

        let resultType = ilType(self.low, typ);
        let resultReg = nextReg(self);
        let mergeBlock = try createBlockWithParam(
            self, "cond#merge", il::Param { value: resultReg, type: resultType }
        );
        try switchToAndSeal(self, thenBlock);
        try emitJmpWithArg(self, mergeBlock, try lowerExpr(self, cond.thenExpr));
        try switchToAndSeal(self, elseBlock);
        try emitJmpWithArg(self, mergeBlock, try lowerExpr(self, cond.elseExpr));
        try switchToAndSeal(self, mergeBlock);

        return il::Val::Reg(resultReg);
    }
}

/// Select the concrete operand type for a scalar comparison.
fn scalarComparisonType(left: resolver::Type, right: resolver::Type) -> resolver::Type {
    return right if left == resolver::Type::Int else left;
}

/// Convert a binary operator to a comparison op, if applicable.
/// For `Gt`, caller must swap operands: `a > b = b < a`.
/// For `Gte`/`Lte`, caller must swap branch labels: `a >= b = !(a < b)`.
/// For `Lte`, caller must also swap operands: `a <= b = !(b < a)`.
fn cmpOpFrom(op: ast::BinaryOp, unsigned: bool) -> ?il::CmpOp {
    match op {
        case ast::BinaryOp::Eq => return il::CmpOp::Eq,
        case ast::BinaryOp::Ne => return il::CmpOp::Ne,
        case ast::BinaryOp::Lt, ast::BinaryOp::Gt,
             ast::BinaryOp::Gte, ast::BinaryOp::Lte =>
            return il::CmpOp::Ult if unsigned else il::CmpOp::Slt,
        else => return nil,
    }
}

/// Lower a binary operation.
unsafe fn lowerBinOp(self: &mut FnLowerer, node: *ast::Node, binop: ast::BinOp) -> il::Val throws (LowerError) {
    // Short-circuit logical operators don't evaluate both operands eagerly.
    if binop.op == ast::BinaryOp::And {
        return try lowerLogicalOp(self, binop, "and#then", "and#else", "and#end", LogicalOp::And);
    } else if binop.op == ast::BinaryOp::Or {
        return try lowerLogicalOp(self, binop, "or#then", "or#else", "or#end", LogicalOp::Or);
    }

    // Handle comparison with a `nil` literal: just check the tag/pointer instead of
    // building a `nil` aggregate and doing full comparison.
    if binop.op == ast::BinaryOp::Eq or binop.op == ast::BinaryOp::Ne {
        let isEq = binop.op == ast::BinaryOp::Eq;
        let leftIsNil = binop.left.value == ast::NodeValue::Nil;
        let rightIsNil = binop.right.value == ast::NodeValue::Nil;

        if leftIsNil {
            return try lowerNilCheck(self, binop.right, isEq);
        } else if rightIsNil {
            return try lowerNilCheck(self, binop.left, isEq);
        }
    }
    // Lower operands.
    let a = try lowerExpr(self, binop.left);
    let b = try lowerExpr(self, binop.right);
    let isComparison = cmpOpFrom(binop.op, false) <> nil;

    // The result type for comparisons is always `bool`, while for arithmetic
    // operations, it's the operand type. We set this appropriately here.
    let nodeTy = try typeOf(self, node);
    let mut resultTy = nodeTy;

    if isComparison {
        let leftTy = try effectiveType(self, binop.left);
        let rightTy = try effectiveType(self, binop.right);
        // Optimize: comparing with a void variant just needs tag comparison.
        if let idx = voidVariantIndex(self.low.resolver, binop.left) {
            return try emitTagCmp(self, binop.op, b, idx, rightTy);
        } else if let idx = voidVariantIndex(self.low.resolver, binop.right) {
            return try emitTagCmp(self, binop.op, a, idx, leftTy);
        }
        // Aggregate types require element-wise comparison.
        // When comparing `?T` with `T`, wrap the scalar side.
        if isAggregateType(leftTy) {
            let mut rhs = b;
            if not isAggregateType(rightTy) {
                set rhs = try wrapInOptional(self, rhs, leftTy);
            }
            return try emitAggregateEqOp(self, binop.op, leftTy, a, rhs);
        }
        if isAggregateType(rightTy) {
            let lhs = try wrapInOptional(self, a, rightTy);
            return try emitAggregateEqOp(self, binop.op, rightTy, lhs, b);
        }
        set resultTy = scalarComparisonType(leftTy, rightTy);
    }
    return emitScalarBinOp(self, binop.op, ilType(self.low, resultTy), a, b, isUnsignedType(resultTy));
}

/// Emit an aggregate equality or inequality comparison.
unsafe fn emitAggregateEqOp(
    self: &mut FnLowerer,
    op: ast::BinaryOp,
    typ: resolver::Type,
    a: il::Val,
    b: il::Val
) -> il::Val throws (LowerError) {
    let regA = emitValToReg(self, a);
    let regB = emitValToReg(self, b);
    let result = try lowerAggregateEq(self, typ, regA, regB, 0);

    if op == ast::BinaryOp::Ne {
        return emitTypedBinOp(self, il::BinOp::Eq, il::Type::W32, result, il::Val::Imm(0));
    }
    return result;
}

/// Emit a scalar binary operation instruction.
unsafe fn emitScalarBinOp(
    self: &mut FnLowerer,
    op: ast::BinaryOp,
    typ: il::Type,
    a: il::Val,
    b: il::Val,
    unsigned: bool
) -> il::Val {
    let dst = nextReg(self);
    let mut needsExt: bool = false;
    match op {
        case ast::BinaryOp::Add => {
            emit(self, il::Instr::BinOp { op: il::BinOp::Add, typ, dst, a, b });
            set needsExt = true;
        }
        case ast::BinaryOp::Sub => {
            emit(self, il::Instr::BinOp { op: il::BinOp::Sub, typ, dst, a, b });
            set needsExt = true;
        }
        case ast::BinaryOp::Mul => {
            emit(self, il::Instr::BinOp { op: il::BinOp::Mul, typ, dst, a, b });
            set needsExt = true;
        }
        case ast::BinaryOp::Div => {
            let op = il::BinOp::Udiv if unsigned else il::BinOp::Sdiv;
            emit(self, il::Instr::BinOp { op, typ, dst, a, b });
            set needsExt = true;
        }
        case ast::BinaryOp::Mod => {
            let op = il::BinOp::Urem if unsigned else il::BinOp::Srem;
            emit(self, il::Instr::BinOp { op, typ, dst, a, b });
            set needsExt = true;
        }
        case ast::BinaryOp::BitAnd => emit(self, il::Instr::BinOp { op: il::BinOp::And, typ, dst, a, b }),
        case ast::BinaryOp::BitOr => emit(self, il::Instr::BinOp { op: il::BinOp::Or, typ, dst, a, b }),
        case ast::BinaryOp::BitXor => emit(self, il::Instr::BinOp { op: il::BinOp::Xor, typ, dst, a, b }),
        case ast::BinaryOp::Shl => {
            emit(self, il::Instr::BinOp { op: il::BinOp::Shl, typ, dst, a, b });
            set needsExt = true;
        }
        case ast::BinaryOp::Shr => {
            let op = il::BinOp::Ushr if unsigned else il::BinOp::Sshr;
            emit(self, il::Instr::BinOp { op, typ, dst, a, b });
        }
        case ast::BinaryOp::Eq => emit(self, il::Instr::BinOp { op: il::BinOp::Eq, typ, dst, a, b }),
        case ast::BinaryOp::Ne => emit(self, il::Instr::BinOp { op: il::BinOp::Ne, typ, dst, a, b }),
        case ast::BinaryOp::Lt => {
            let op = il::BinOp::Ult if unsigned else il::BinOp::Slt;
            emit(self, il::Instr::BinOp { op, typ, dst, a, b });
        }
        case ast::BinaryOp::Gt => { // `a > b` = `b < a`
            let op = il::BinOp::Ult if unsigned else il::BinOp::Slt;
            emit(self, il::Instr::BinOp { op, typ, dst, a: b, b: a });
        }
        case ast::BinaryOp::Lte => { // `a <= b` = `b >= a`
            let op = il::BinOp::Uge if unsigned else il::BinOp::Sge;
            emit(self, il::Instr::BinOp { op, typ, dst, a: b, b: a });
        }
        case ast::BinaryOp::Gte => {
            let op = il::BinOp::Uge if unsigned else il::BinOp::Sge;
            emit(self, il::Instr::BinOp { op, typ, dst, a, b });
        }
        // Logical xor on booleans is equivalent to not equal.
        case ast::BinaryOp::Xor => emit(self, il::Instr::BinOp { op: il::BinOp::Ne, typ, dst, a, b }),
        // Short-circuit ops are handled elsewhere.
        case ast::BinaryOp::And, ast::BinaryOp::Or => panic,
    }
    // Normalize sub-word arithmetic results so high bits are well-defined.
    // The lowering knows signedness, so it can pick the right extension.
    // [`il::Type::W32`] is handled in the backend via 32-bit instructions.
    if needsExt {
        return normalizeSubword(self, typ, unsigned, il::Val::Reg(dst));
    }
    return il::Val::Reg(dst);
}

/// Normalize sub-word values to well-defined high bits.
unsafe fn normalizeSubword(self: &mut FnLowerer, typ: il::Type, unsigned: bool, val: il::Val) -> il::Val {
    if typ == il::Type::W8 or typ == il::Type::W16 {
        let extDst: il::Reg = nextReg(self);
        if unsigned {
            emit(self, il::Instr::Zext { typ, dst: extDst, val });
        } else {
            emit(self, il::Instr::Sext { typ, dst: extDst, val });
        }
        return il::Val::Reg(extDst);
    }
    return val;
}

/// Lower a unary operation.
unsafe fn lowerUnOp(self: &mut FnLowerer, node: *ast::Node, unop: ast::UnOp) -> il::Val throws (LowerError) {
    if unop.op == ast::UnaryOp::Neg {
        if let case ast::NodeValue::Number(lit) = unop.value.value {
            return il::Val::Imm((0 - lit.magnitude) as i64);
        }
    }
    let val = try lowerExpr(self, unop.value);
    let t = try typeOf(self, node);
    let typ = ilType(self.low, t);
    let dst = nextReg(self);
    let mut needsExt: bool = false;

    match unop.op {
        case ast::UnaryOp::Not => {
            emit(self, il::Instr::BinOp { op: il::BinOp::Eq, typ, dst, a: val, b: il::Val::Imm(0) });
        }
        case ast::UnaryOp::Neg => {
            emit(self, il::Instr::UnOp { op: il::UnOp::Neg, typ, dst, a: val });
            set needsExt = true;
        }
        case ast::UnaryOp::BitNot => {
            emit(self, il::Instr::UnOp { op: il::UnOp::Not, typ, dst, a: val });
            set needsExt = true;
        }
    }
    if needsExt {
        return normalizeSubword(self, typ, isUnsignedType(t), il::Val::Reg(dst));
    }
    return il::Val::Reg(dst);
}

/// Lower a cast expression (`x as T`).
unsafe fn lowerCast(self: &mut FnLowerer, node: *ast::Node, cast: ast::As) -> il::Val throws (LowerError) {
    let val = try lowerExpr(self, cast.value);

    let srcType = try typeOf(self, cast.value);
    let dstType = try typeOf(self, node);
    if resolver::typesEqual(srcType, dstType) {
        return val;
    }
    return lowerNumericCast(self, val, srcType, dstType);
}

/// Check whether a resolver type is a signed integer type.
fn isSignedType(t: resolver::Type) -> bool {
    match t {
        case resolver::Type::I8, resolver::Type::I16, resolver::Type::I32, resolver::Type::I64,
             resolver::Type::Int => return true,
        else => return false,
    }
}

/// Check whether a resolver type is an unsigned integer type.
fn isUnsignedType(t: resolver::Type) -> bool {
    match t {
        case resolver::Type::U8, resolver::Type::U16, resolver::Type::U32, resolver::Type::U64 => return true,
        else => return false,
    }
}

/// Lower a string literal to a slice value.
///
/// String literals are stored as global data and the result is a slice
/// pointing to the data with the appropriate length.
unsafe fn lowerStringLit(self: &mut FnLowerer, node: *ast::Node, s: *[u8]) -> il::Val throws (LowerError) {
    // Get the slice type from the node.
    let sliceTy = try typeOf(self, node);
    let case resolver::Type::Slice { item, mutable, .. } = sliceTy
        else throw LowerError::ExpectedSliceOrArray;
    // Build the string data value.
    let ptr = try! alloc::alloc(
        self.low.arena, @sizeOf(il::DataValue), @alignOf(il::DataValue)
    ) as *mut il::DataValue;

    set *ptr = il::DataValue { item: il::DataItem::Str(s), count: 1 };
    let result = ConstDataResult { values: @sliceOf(ptr, 1), zeroInit: false };

    return try lowerConstDataAsSlice(
        self, &result, 1, true, item, mutable, s.len
    );
}

/// Lower a builtin call expression.
unsafe fn lowerBuiltinCall(self: &mut FnLowerer, node: *ast::Node, kind: ast::Builtin, args: *[*ast::Node]) -> il::Val throws (LowerError) {
    match kind {
        case ast::Builtin::SliceOf => return try lowerSliceOf(self, node, args),
        case ast::Builtin::SizeOf, ast::Builtin::AlignOf => {
            let constVal = resolver::constValueEntry(self.low.resolver, node) else {
                throw LowerError::MissingConst(node);
            };
            return try constValueToVal(self, constVal, node);
        }
    }
}

/// Lower a `@sliceOf(ptr, len)` or `@sliceOf(ptr, len, cap)` builtin call.
unsafe fn lowerSliceOf(self: &mut FnLowerer, node: *ast::Node, args: *[*ast::Node]) -> il::Val throws (LowerError) {
    if args.len <> 2 and args.len <> 3 {
        throw LowerError::InvalidArgCount;
    }
    let sliceTy = try typeOf(self, node);
    let case resolver::Type::Slice { item, mutable, .. } = sliceTy
        else throw LowerError::ExpectedSliceOrArray;
    let ptrVal = try lowerExpr(self, args[0]);
    let lenVal = try lowerExpr(self, args[1]);
    let mut capVal = lenVal;
    if args.len == 3 {
        set capVal = try lowerExpr(self, args[2]);
    }
    if not isLtEq(lenVal, capVal) {
        try emitTrapIfLt(self, il::Type::W32, capVal, lenVal);
    }
    return try buildSliceValue(self, item, mutable, ptrVal, lenVal, capVal);
}

/// Lower a `try` expression.
unsafe fn lowerTry(self: &mut FnLowerer, node: *ast::Node, t: ast::Try) -> il::Val throws (LowerError) {
    let case ast::NodeValue::Call(callExpr) = t.expr.value else {
        throw LowerError::ExpectedCall;
    };
    let calleeTy = try typeOf(self, callExpr.callee);
    let case resolver::Type::Fn(calleeInfo) = calleeTy else {
        throw LowerError::ExpectedFunction;
    };
    let okValueTy = *calleeInfo.returnType; // The type of the success payload.

    // Type of the try expression, which is either the return type of the function
    // if successful, or an optional of it, if using `try?`.
    let tryExprTy = try typeOf(self, node);
    // Check for trait method dispatch or standalone method call.
    let mut resVal: il::Val = undefined;
    let callNodeExtra = resolver::nodeData(self.low.resolver, t.expr).extra;
    if let case resolver::NodeExtra::TraitMethodCall {
        traitInfo, methodIndex
    } = callNodeExtra {
        set resVal = try lowerTraitMethodCall(self, t.expr, callExpr, traitInfo, methodIndex);
    } else if let case resolver::NodeExtra::MethodCall { method } = callNodeExtra {
        set resVal = try lowerMethodCall(self, t.expr, callExpr, method);
    } else {
        set resVal = try lowerCall(self, t.expr, callExpr);
    }
    let base = emitValToReg(self, resVal); // The result value.
    let tagReg = resultTagReg(self, base); // The result tag.

    let okBlock = try createBlock(self, "ok"); // Block if success.
    let errBlock = try createBlock(self, "err"); // Block if failure.

    let mut mergeBlock: ?BlockId = nil;
    let mut resultSlot: ?il::Reg = nil; // `try` result value will be stored here.

    // Check if the `try` returns a success value or not. If so, reserve
    // space for it.
    let isVoid = tryExprTy == resolver::Type::Void or tryExprTy == resolver::Type::Never;
    if not isVoid {
        set resultSlot = try emitReserve(self, tryExprTy);
    }
    // Branch on tag: zero means ok, non-zero means error.
    try emitBr(self, tagReg, errBlock, okBlock);

    // We can now seal the blocks since all predecessors are known.
    try sealBlock(self, okBlock);
    try sealBlock(self, errBlock);

    // Success path: extract the successful value from the result and store it
    // in the result slot for later use after the merge point.
    switchToBlock(self, okBlock);

    if okValueTy == resolver::Type::Never {
        emit(self, il::Instr::Unreachable);
    } else if let slot = resultSlot {
        // Extract the success payload. If the result type differs from the payload
        // type (e.g. `try?` wrapping `T` into `?T`), wrap the value.
        let payloadVal = tvalPayloadVal(self, base, okValueTy, RESULT_VAL_OFFSET);
        let mut okVal = payloadVal;

        if t.returnsOptional and tryExprTy <> okValueTy {
            set okVal = try wrapInOptional(self, payloadVal, tryExprTy);
        }
        try emitStore(self, slot, 0, tryExprTy, okVal);
    }
    // Jump to merge block if unterminated.
    try emitMergeIfUnterminated(self, &mut mergeBlock);

    // Error path: handle the failure case based on the try expression variant.
    switchToBlock(self, errBlock);

    if t.returnsOptional {
        // `try?` converts errors to `nil` -- store the `nil` and continue.
        if let slot = resultSlot {
            let errVal = try buildNilOptional(self, tryExprTy);
            try emitStore(self, slot, 0, tryExprTy, errVal);
        }
        try emitMergeIfUnterminated(self, &mut mergeBlock);
    } else if t.catches.len > 0 {
        // `try ... catch` -- handle the error.
        let firstNode = t.catches[0];
        let case ast::NodeValue::CatchClause(first) = firstNode.value
            else panic "lowerTry: expected CatchClause";

        if first.typeNode <> nil or t.catches.len > 1 {
            // Typed multi-catch: switch on global error tag.
            try lowerMultiCatch(self, t.catches, calleeInfo, base, tagReg, &mut mergeBlock);
        } else {
            // Single untyped catch clause.
            let savedVarsLen = enterVarScope(self);
            if let binding = first.binding {
                let case ast::NodeValue::Ident(name) = binding.value else {
                    throw LowerError::ExpectedIdentifier;
                };
                let errTy = *calleeInfo.throwList[0];
                let errVal = tvalPayloadVal(self, base, errTy, RESULT_VAL_OFFSET);
                let _ = newVar(self, name, ilType(self.low, errTy), false, errVal);
            }
            try lowerBlock(self, first.body);
            try emitMergeIfUnterminated(self, &mut mergeBlock);
            exitVarScope(self, savedVarsLen);
        }
    } else if t.shouldPanic {
        // `try!` -- panic on error, emit unreachable since control won't continue.
        // TODO: We should have some kind of `panic` instruction?
        emit(self, il::Instr::Unreachable);
    } else {
        // Plain `try` -- propagate the error to the caller by returning early.
        // Forward the callee's global error tag and payload directly.
        let callerLayout = resolver::getResultLayout(
            *self.fnType.returnType, self.fnType.throwList
        );
        let calleeErrSize = maxErrSize(calleeInfo.throwList);
        let dst = emitReserveLayout(self, callerLayout);

        emitStoreW64At(self, il::Val::Reg(tagReg), dst, TVAL_TAG_OFFSET);
        let srcPayload = emitPtrOffset(self, base, RESULT_VAL_OFFSET);
        let dstPayload = emitPtrOffset(self, dst, RESULT_VAL_OFFSET);
        emit(self, il::Instr::Blit { dst: dstPayload, src: srcPayload, size: il::Val::Imm(calleeErrSize as i64) });

        try emitRetVal(self, il::Val::Reg(dst));
    }

    // Switch to the merge block if one was created. If all paths diverged
    // (e.g both success and error returned), there's no merge block.
    if let blk = mergeBlock {
        try switchToAndSeal(self, blk);
    } else {
        return il::Val::Undef;
    }
    // Return the result value. For `void` expressions, return undefined.
    // For aggregates, return the slot pointer; for scalars, load the value.
    if let slot = resultSlot {
        if isAggregateType(tryExprTy) {
            return il::Val::Reg(slot);
        }
        return emitLoad(self, slot, 0, tryExprTy);
    } else { // Void return.
        return il::Val::Undef;
    }
}

/// Lower typed multi-catch clauses.
///
/// Emits a switch on the global error tag to dispatch to the correct catch
/// clause. Each typed clause extracts the error payload for its specific type
/// and binds it to the clause's identifier.
unsafe fn lowerMultiCatch(
    self: &mut FnLowerer,
    catches: *[*ast::Node],
    calleeInfo: *resolver::FnType,
    base: il::Reg,
    tagReg: il::Reg,
    mergeBlock: &mut ?BlockId
) throws (LowerError) {
    let entry = currentBlock(self);

    // First pass: create blocks, resolve error types, and build switch cases.
    let mut blocks: [BlockId; MAX_CATCH_CLAUSES] = undefined;
    let mut errTypes: [?resolver::Type; MAX_CATCH_CLAUSES] = undefined;
    let mut cases: *unsafe mut [il::SwitchCase] = &mut [];
    let mut defaultIdx: ?u32 = nil;

    for clauseNode, i in catches {
        let case ast::NodeValue::CatchClause(clause) = clauseNode.value
            else panic "lowerMultiCatch: expected CatchClause";

        set blocks[i] = try createBlock(self, "catch");
        addPredecessor(self, blocks[i], entry);

        if let typeNode = clause.typeNode {
            let errTy = try typeOf(self, typeNode);
            set errTypes[i] = errTy;

            cases.append(il::SwitchCase {
                value: getOrAssignErrorTag(self.low, errTy) as i64,
                target: *blocks[i],
                args: &mut []
            }, self.allocator);
        } else {
            set errTypes[i] = nil;
            set defaultIdx = i;
        }
    }

    // Emit switch. Default target is the catch-all block, or an unreachable block.
    let mut defaultTarget: BlockId = undefined;
    if let idx = defaultIdx {
        set defaultTarget = blocks[idx];
    } else {
        set defaultTarget = try createBlock(self, "unreachable");
        addPredecessor(self, defaultTarget, entry);
    }
    emit(self, il::Instr::Switch {
        val: il::Val::Reg(tagReg),
        defaultTarget: *defaultTarget,
        defaultArgs: &mut [],
        cases
    });

    // Second pass: emit each catch clause body.
    for clauseNode, i in catches {
        let case ast::NodeValue::CatchClause(clause) = clauseNode.value
            else panic "lowerMultiCatch: expected CatchClause";

        try switchToAndSeal(self, blocks[i]);
        let savedVarsLen = enterVarScope(self);

        if let binding = clause.binding {
            let case ast::NodeValue::Ident(name) = binding.value else {
                throw LowerError::ExpectedIdentifier;
            };
            let errTy = errTypes[i] else panic "lowerMultiCatch: catch-all with binding";
            let errVal = tvalPayloadVal(self, base, errTy, RESULT_VAL_OFFSET);

            newVar(self, name, ilType(self.low, errTy), false, errVal);
        }
        try lowerBlock(self, clause.body);
        try emitMergeIfUnterminated(self, mergeBlock);

        exitVarScope(self, savedVarsLen);
    }

    // Emit unreachable block if no catch-all.
    if defaultIdx == nil {
        try switchToAndSeal(self, defaultTarget);
        emit(self, il::Instr::Unreachable);
    }
}

/// Emit a byte-copy loop: `for i in 0..size { dst[i] = src[i]; }`.
///
/// Used when `blit` cannot be used because the copy size is dynamic.
/// Terminates the current block and leaves the builder positioned
/// after the loop.
unsafe fn emitByteCopyLoop(
    self: &mut FnLowerer,
    dst: il::Reg,
    src: il::Reg,
    size: il::Val,
    label: *[u8]
) throws (LowerError) {
    let iReg = nextReg(self);
    let header = try createBlockWithParam(
        self, label, il::Param { value: iReg, type: il::Type::W32 }
    );
    let body = try createBlock(self, label);
    let done = try createBlock(self, label);

    // Jump to header with initial counter is zero.
    try emitJmpWithArg(self, header, il::Val::Imm(0));

    // Don't seal header yet -- the body will add another predecessor.
    switchToBlock(self, header);
    try emitBrCmp(self, il::CmpOp::Ult, il::Type::W32, il::Val::Reg(iReg), size, body, done);

    // Body: load byte from source, store to destination, increment counter.
    try switchToAndSeal(self, body);

    let srcElem = emitElem(self, 1, src, il::Val::Reg(iReg));
    let byteReg = nextReg(self);
    emit(self, il::Instr::Load { typ: il::Type::W8, dst: byteReg, src: srcElem, offset: 0 });

    let dstElem = emitElem(self, 1, dst, il::Val::Reg(iReg));
    emit(self, il::Instr::Store { typ: il::Type::W8, src: il::Val::Reg(byteReg), dst: dstElem, offset: 0 });

    let nextI = emitTypedBinOp(self, il::BinOp::Add, il::Type::W32, il::Val::Reg(iReg), il::Val::Imm(1));
    // Jump back to header -- this adds body as a predecessor.
    try emitJmpWithArg(self, header, nextI);

    // Now all predecessors of header are known, seal it.
    try sealBlock(self, header);
    try switchToAndSeal(self, done);
}

/// Lower `slice.append(val, allocator)`.
///
/// Emits inline grow-if-needed logic:
///
///     load len, cap from slice header
///     if len < cap: jmp @store
///     else:         jmp @grow
///
///     @grow:
///       newCap = max(cap * 2, 1)
///       call allocator.func(allocator.ctx, newCap * stride, alignment)
///       copy old data to new pointer
///       update slice ptr and cap
///       jmp @store
///
///     @store:
///       store element at ptr + len * stride
///       increment len
///
unsafe fn lowerSliceAppend(self: &mut FnLowerer, call: ast::Call, elemType: *resolver::Type) -> il::Val throws (LowerError) {
    let case ast::NodeValue::FieldAccess(access) = call.callee.value
        else throw LowerError::MissingMetadata;

    // Get the address of the slice header.
    let sliceVal = try lowerExpr(self, access.parent);
    let sliceReg = emitValToReg(self, sliceVal);

    // Lower the value to append and the allocator.
    let elemVal = try lowerExpr(self, call.args[0]);
    let allocVal = try lowerExpr(self, call.args[1]);
    let allocReg = emitValToReg(self, allocVal);

    let elemLayout = resolver::getTypeLayout(*elemType);
    let stride = elemLayout.size;
    let alignment = elemLayout.alignment;

    // Load current length and capacity.
    let lenVal = loadSliceLen(self, sliceReg);
    let capVal = loadSliceCap(self, sliceReg);

    // Branch: if length is smaller than capacity, go to @store else @grow.
    let storeBlock = try createBlock(self, "append.store");
    let growBlock = try createBlock(self, "append.grow");
    try emitBrCmp(self, il::CmpOp::Ult, il::Type::W32, lenVal, capVal, storeBlock, growBlock);
    try switchToAndSeal(self, growBlock);

    // -- @grow block ----------------------------------------------------------

    // `newCap = max(cap * 2, 1)`.
    // We are only here when at capacity, so we use `or` with `1` to ensure at least capacity `1`.
    let doubledVal = emitTypedBinOp(self, il::BinOp::Shl, il::Type::W32, capVal, il::Val::Imm(1));
    let newCapVal = emitTypedBinOp(self, il::BinOp::Or, il::Type::W32, doubledVal, il::Val::Imm(1));

    // Call allocator: `a.func(a.ctx, newCap * stride, alignment)`.
    let allocFnReg = nextReg(self);
    emitLoadW64At(self, allocFnReg, allocReg, 0);

    let allocCtxReg = nextReg(self);
    emitLoadW64At(self, allocCtxReg, allocReg, 8);

    let byteSize = emitTypedBinOp(self, il::BinOp::Mul, il::Type::W32, newCapVal, il::Val::Imm(stride as i64));
    let args = try allocVals(self, 3);

    set args[0] = il::Val::Reg(allocCtxReg);
    set args[1] = byteSize;
    set args[2] = il::Val::Imm(alignment as i64);

    let newPtrReg = nextReg(self);
    emit(self, il::Instr::Call {
        retTy: il::Type::W64,
        dst: newPtrReg,
        func: il::Val::Reg(allocFnReg),
        args,
    });

    // Copy old data byte-by-byte.
    let oldPtrReg = loadSlicePtr(self, sliceReg);
    let copyBytes = emitTypedBinOp(self, il::BinOp::Mul, il::Type::W32, lenVal, il::Val::Imm(stride as i64));
    try emitByteCopyLoop(self, newPtrReg, oldPtrReg, copyBytes, "append");

    // Update slice header.
    emitStoreW64At(self, il::Val::Reg(newPtrReg), sliceReg, SLICE_PTR_OFFSET);
    emitStoreW32At(self, newCapVal, sliceReg, SLICE_CAP_OFFSET);

    try emitJmp(self, storeBlock);
    try switchToAndSeal(self, storeBlock);

    // -- @store block ---------------------------------------------------------

    // Store element at `ptr + len * stride`.
    let ptrReg = loadSlicePtr(self, sliceReg);
    let elemDst = emitElem(self, stride, ptrReg, lenVal);
    try emitStore(self, elemDst, 0, *elemType, elemVal);

    // Increment len.
    let newLen = emitTypedBinOp(self, il::BinOp::Add, il::Type::W32, lenVal, il::Val::Imm(1));
    emitStoreW32At(self, newLen, sliceReg, SLICE_LEN_OFFSET);

    return il::Val::Reg(sliceReg);
}

/// Lower `slice.delete(index)`.
///
/// Bounds-check the index, shift elements after it by one stride
/// via a byte-copy loop, and decrement `len`.
unsafe fn lowerSliceDelete(self: &mut FnLowerer, call: ast::Call, elemType: *resolver::Type) throws (LowerError) {
    let case ast::NodeValue::FieldAccess(access) = call.callee.value
        else throw LowerError::MissingMetadata;

    let elemLayout = resolver::getTypeLayout(*elemType);
    let stride = elemLayout.size;

    // Get slice header address.
    let sliceVal = try lowerExpr(self, access.parent);
    let sliceReg = emitValToReg(self, sliceVal);

    // Lower the index argument.
    let indexVal = try lowerExpr(self, call.args[0]);

    // Load len and bounds-check: index must be smaller than length.
    let lenVal = loadSliceLen(self, sliceReg);
    try emitTrapUnlessCmp(self, il::CmpOp::Ult, il::Type::W32, indexVal, lenVal);

    // Compute the destination and source for the shift.
    let ptrReg = loadSlicePtr(self, sliceReg);
    let dst = emitElem(self, stride, ptrReg, indexVal);

    // `src = dst + stride`.
    let src = emitPtrOffset(self, dst, stride as i32);

    // Move `(len - index - 1) * stride`.
    let tailLen = emitTypedBinOp(self, il::BinOp::Sub, il::Type::W32, lenVal, indexVal);
    let tailLenMinusOne = emitTypedBinOp(self, il::BinOp::Sub, il::Type::W32, tailLen, il::Val::Imm(1));
    let moveBytes = emitTypedBinOp(self, il::BinOp::Mul, il::Type::W32, tailLenMinusOne, il::Val::Imm(stride as i64));

    // Shift elements left via byte-copy loop.
    // When deleting the last element, the loop is a no-op.
    try emitByteCopyLoop(self, dst, src, moveBytes, "delete");
    // Decrement length.
    let newLen = emitTypedBinOp(self, il::BinOp::Sub, il::Type::W32, lenVal, il::Val::Imm(1));

    emitStoreW32At(self, newLen, sliceReg, SLICE_LEN_OFFSET);
}

/// Lower a call expression, which may be a function call or type constructor.
unsafe fn lowerCallOrCtor(self: &mut FnLowerer, node: *ast::Node, call: ast::Call) -> il::Val throws (LowerError) {
    let nodeData = resolver::nodeData(self.low.resolver, node).extra;

    // Check for slice method dispatch.
    if let case resolver::NodeExtra::SliceAppend { elemType } = nodeData {
        return try lowerSliceAppend(self, call, elemType);
    }
    if let case resolver::NodeExtra::SliceDelete { elemType } = nodeData {
        try lowerSliceDelete(self, call, elemType);
        return il::Val::Undef;
    }
    // Check for trait method dispatch.
    if let case resolver::NodeExtra::TraitMethodCall { traitInfo, methodIndex } = nodeData {
        return try lowerTraitMethodCall(self, node, call, traitInfo, methodIndex);
    }
    // Check for standalone method call.
    if let case resolver::NodeExtra::MethodCall { method } = nodeData {
        return try lowerMethodCall(self, node, call, method);
    }
    if let sym = resolver::nodeData(self.low.resolver, call.callee).sym {
        if let case resolver::SymbolData::Type(nominal) = sym.data {
            let case resolver::NominalType::Record(_) = *nominal else {
                throw LowerError::ExpectedRecord;
            };
            return try lowerRecordCtor(self, nominal, call.args);
        }
        if let case resolver::SymbolData::Variant { .. } = sym.data {
            return try lowerUnionCtor(self, node, sym, call);
        }
    }
    return try lowerCall(self, node, call);
}

/// Lower a trait method call through v-table dispatch.
///
/// Given `obj.method(args)` where `obj` is a trait object, emits:
///
///     load w64 %data %obj 0          // data pointer
///     load w64 %vtable %obj 8        // v-table pointer
///     load w64 %fn %vtable <slot>    // function pointer
///     call <retTy> %ret %fn(%data, args...)
///
unsafe fn lowerTraitMethodCall(
    self: &mut FnLowerer,
    node: *ast::Node,
    call: ast::Call,
    traitInfo: *unsafe resolver::TraitType,
    methodIndex: u32
) -> il::Val throws (LowerError) {
    // Method calls look like field accesses.
    let case ast::NodeValue::FieldAccess(access) = call.callee.value
        else throw LowerError::MissingMetadata;

    // Lower the trait object expression.
    let traitObjVal = try lowerExpr(self, access.parent);
    let traitObjReg = emitValToReg(self, traitObjVal);

    // Load data pointer from trait object.
    let dataReg = nextReg(self);
    emit(self, il::Instr::Load {
        typ: il::Type::W64,
        dst: dataReg,
        src: traitObjReg,
        offset: TRAIT_OBJ_DATA_OFFSET,
    });

    // Load v-table pointer from trait object.
    let vtableReg = nextReg(self);
    emit(self, il::Instr::Load {
        typ: il::Type::W64,
        dst: vtableReg,
        src: traitObjReg,
        offset: TRAIT_OBJ_VTABLE_OFFSET,
    });

    // Load function pointer from v-table at the method's slot offset.
    let fnPtrReg = nextReg(self);
    let slotOffset = (methodIndex * resolver::PTR_SIZE) as i32;

    emit(self, il::Instr::Load {
        typ: il::Type::W64,
        dst: fnPtrReg,
        src: vtableReg,
        offset: slotOffset,
    });
    let methodFnType = traitInfo.methods[methodIndex].fnType;

    // Build args: optional return param slot + data pointer (receiver) + user args.
    let argOffset: u32 = 1 if requiresReturnParam(methodFnType) else 0;
    let args = try allocVals(self, call.args.len + 1 + argOffset);
    set args[argOffset] = il::Val::Reg(dataReg);

    for arg, i in call.args {
        set args[i + 1 + argOffset] = try lowerCallArg(
            self, arg, i + 1 < call.args.len
        );
    }
    return try emitCallValue(self, il::Val::Reg(fnPtrReg), methodFnType, args);
}

/// Lower a call argument, snapshotting aggregate place expressions before
/// evaluating later arguments. Aggregates are represented by addresses in the
/// IL, so retaining the original address would let a later argument mutation
/// change the value already supplied for this argument.
unsafe fn lowerCallArg(self: &mut FnLowerer, arg: *ast::Node, hasLater: bool) -> il::Val
    throws (LowerError)
{
    let val = try lowerExpr(self, arg);

    if hasLater and ast::isPlaceExpr(arg) {
        let argType = try effectiveType(self, arg);
        if isAggregateType(argType) {
            return try emitStackVal(self, argType, val);
        }
    }
    return val;
}

/// Emit a function call with return-parameter and small-aggregate handling.
///
/// All call lowering paths (regular, trait method, standalone method) converge
/// here after preparing the callee value, function type, and argument array.
/// The `args` slice must already include a slot at index zero for the hidden
/// return parameter; that slot is filled by this function.
unsafe fn emitCallValue(
    self: &mut FnLowerer,
    callee: il::Val,
    fnInfo: *resolver::FnType,
    args: *unsafe mut [il::Val],
) -> il::Val throws (LowerError) {
    let retTy = *fnInfo.returnType;

    if requiresReturnParam(fnInfo) {
        if fnInfo.throwList.len > 0 {
            let layout = resolver::getResultLayout(retTy, fnInfo.throwList);
            set args[0] = il::Val::Reg(emitReserveLayout(self, layout));
        } else {
            set args[0] = il::Val::Reg(try emitReserve(self, retTy));
        }
        let dst = nextReg(self);

        emit(self, il::Instr::Call {
            retTy: il::Type::W64,
            dst,
            func: callee,
            args,
        });
        return il::Val::Reg(dst);
    }
    let mut dst: ?il::Reg = nil;
    if retTy <> resolver::Type::Void and retTy <> resolver::Type::Never {
        set dst = nextReg(self);
    }
    emit(self, il::Instr::Call {
        retTy: ilType(self.low, retTy),
        dst,
        func: callee,
        args,
    });

    if retTy == resolver::Type::Never {
        emit(self, il::Instr::Unreachable);
    }
    if let d = dst {
        if isSmallAggregate(retTy) {
            let slot = emitReserveLayout(self, resolver::Layout {
                size: resolver::PTR_SIZE,
                alignment: resolver::PTR_SIZE,
            });
            emit(self, il::Instr::Store {
                typ: il::Type::W64,
                src: il::Val::Reg(d),
                dst: slot,
                offset: 0,
            });
            return il::Val::Reg(slot);
        }
        return il::Val::Reg(d);
    }
    return il::Val::Undef;
}

/// Lower a method receiver expression to a pointer value.
///
/// If the parent is already a pointer type, the value is used directly.
/// If the parent is a value type (eg. a local record), its address is taken.
unsafe fn lowerReceiver(self: &mut FnLowerer, parent: *ast::Node, parentTy: resolver::Type) -> il::Val
    throws (LowerError)
{
    if let case resolver::Type::Pointer { .. } = parentTy {
        // Already a pointer: lower and use directly.
        return try lowerExpr(self, parent);
    }
    // Value type: take its address by lowering it and returning the slot pointer.
    // Aggregate types are already lowered as pointers to stack slots.
    let val = try lowerExpr(self, parent);
    if isAggregateType(parentTy) {
        return val;
    }
    // Scalar value: store to a stack slot and return the slot pointer.
    let layout = resolver::getLayout(self.low.resolver, parent, parentTy);
    let slot = emitReserveLayout(self, layout);
    try emitStore(self, slot, 0, parentTy, val);

    return il::Val::Reg(slot);
}

/// Lower a standalone method call via direct dispatch.
///
/// Given `obj.method(args)` where `method` is a standalone method on a concrete type,
/// emits a direct call with the receiver address as the first argument:
///
///     call <retTy> %ret @Type::method(&obj, args...)
///
unsafe fn lowerMethodCall(
    self: &mut FnLowerer,
    node: *ast::Node,
    call: ast::Call,
    method: &resolver::MethodEntry,
) -> il::Val throws (LowerError) {
    let case ast::NodeValue::FieldAccess(access) = call.callee.value
        else throw LowerError::MissingMetadata;

    // Get the receiver as a pointer.
    let parentTy = try typeOf(self, access.parent);
    let receiverVal = try lowerReceiver(self, access.parent, parentTy);

    let qualName = instanceMethodName(self.low, nil, method.concreteTypeName, method.name);
    let case resolver::SymbolData::Value { type: resolver::Type::Fn(fnInfo), .. } = method.symbol.data
        else panic "lowerMethodCall: expected Fn type on method symbol";

    // Build args: optional return param slot + receiver + user args.
    let argOffset: u32 = 1 if requiresReturnParam(fnInfo) else 0;
    let args = try allocVals(self, call.args.len + 1 + argOffset);
    set args[argOffset] = receiverVal;
    for arg, i in call.args {
        set args[i + 1 + argOffset] = try lowerCallArg(
            self, arg, i + 1 < call.args.len
        );
    }
    return try emitCallValue(self, il::Val::FnAddr(qualName), fnInfo, args);
}

/// Check if a call is to a compiler intrinsic and lower it directly.
unsafe fn lowerIntrinsicCall(self: &mut FnLowerer, call: ast::Call) -> ?il::Val throws (LowerError) {
    // Get the callee symbol and check if it's marked as an intrinsic.
    let sym = resolver::nodeData(self.low.resolver, call.callee).sym else {
        // Expressions or function pointers may not have an associated symbol.
        return nil;
    };
    if not ast::hasAttribute(sym.attrs, ast::Attribute::Intrinsic) {
        return nil;
    }
    // Check for known intrinsic names.
    if mem::eq(sym.name, "ecall") {
        return try lowerEcall(self, call);
    } else if mem::eq(sym.name, "ebreak") {
        return try lowerEbreak(self, call);
    } else if mem::eq(sym.name, "memoryFence") {
        return try lowerMemoryFence(self, call);
    } else {
        throw LowerError::UnknownIntrinsic;
    }
}

/// Lower an ecall intrinsic: `ecall(num, a0, a1, a2, a3) -> i32`.
unsafe fn lowerEcall(self: &mut FnLowerer, call: ast::Call) -> il::Val throws (LowerError) {
    if call.args.len <> 5 {
        throw LowerError::InvalidArgCount;
    }
    let num = try lowerExpr(self, call.args[0]);
    let a0 = try lowerExpr(self, call.args[1]);
    let a1 = try lowerExpr(self, call.args[2]);
    let a2 = try lowerExpr(self, call.args[3]);
    let a3 = try lowerExpr(self, call.args[4]);
    let dst = nextReg(self);

    emit(self, il::Instr::Ecall { dst, num, a0, a1, a2, a3 });

    return il::Val::Reg(dst);
}

/// Lower an ebreak intrinsic: `ebreak()`.
unsafe fn lowerEbreak(self: &mut FnLowerer, call: ast::Call) -> il::Val throws (LowerError) {
    if call.args.len <> 0 {
        throw LowerError::InvalidArgCount;
    }
    emit(self, il::Instr::Ebreak);

    return il::Val::Undef;
}

/// Lower `memoryFence()`.
unsafe fn lowerMemoryFence(self: &mut FnLowerer, call: ast::Call) -> il::Val throws (LowerError) {
    if call.args.len <> 0 {
        throw LowerError::InvalidArgCount;
    }
    emit(self, il::Instr::MemoryFence);
    return il::Val::Undef;
}

/// Resolve callee to an IL value. For direct function calls, use the symbol name.
/// For variables holding function pointers or complex expressions (eg. `array[i]()`),
/// lower the callee expression.
unsafe fn lowerCallee(self: &mut FnLowerer, callee: *ast::Node) -> il::Val throws (LowerError) {
    if let sym = resolver::nodeData(self.low.resolver, callee).sym {
        if let case ast::NodeValue::FnDecl(_) = sym.node.value {
            // First try to look up the symbol in our registered functions.
            // This handles cross-package calls correctly, since packages are
            // lowered in dependency order.
            if let qualName = lookupFnSym(self.low, sym) {
                return il::Val::FnAddr(qualName);
            }
            // Fall back to computing the qualified name from the module graph.
            // This works for functions in the current package.
            let modId = resolver::moduleIdForSymbol(self.low.resolver, sym) else {
                throw LowerError::MissingMetadata;
            };
            return il::Val::FnAddr(qualifyName(self.low, modId, sym.name));
        }
    }
    return try lowerExpr(self, callee);
}

/// Lower a function call expression.
unsafe fn lowerCall(self: &mut FnLowerer, node: *ast::Node, call: ast::Call) -> il::Val throws (LowerError) {
    // Check for intrinsic calls before normal call lowering.
    if let intrinsicVal = try lowerIntrinsicCall(self, call) {
        return intrinsicVal;
    }
    let calleeTy = try typeOf(self, call.callee);
    let case resolver::Type::Fn(fnInfo) = calleeTy else {
        throw LowerError::ExpectedFunction;
    };
    let callee = try lowerCallee(self, call.callee);
    let offset: u32 = 1 if requiresReturnParam(fnInfo) else 0;
    let args = try allocVals(self, call.args.len + offset);
    for arg, i in call.args {
        set args[i + offset] = try lowerCallArg(
            self, arg, i + 1 < call.args.len
        );
    }

    return try emitCallValue(self, callee, fnInfo, args);
}

/// Apply coercions requested by the resolver.
unsafe fn applyCoercion(self: &mut FnLowerer, node: *ast::Node, val: il::Val) -> il::Val throws (LowerError) {
    let coerce = resolver::coercionFor(self.low.resolver, node) else {
        return val;
    };
    match coerce {
        case resolver::Coercion::OptionalLift(optType) => {
            if let case ast::NodeValue::Nil = node.value {
                return try buildNilOptional(self, optType);
            }
            return try wrapInOptional(self, val, optType);
        }
        case resolver::Coercion::NumericCast { from, to } => {
            return lowerNumericCast(self, val, from, to);
        }
        case resolver::Coercion::ResultWrap => {
            let payloadType = *self.fnType.returnType;
            return try buildResult(self, 0, val, payloadType);
        }
        case resolver::Coercion::TraitObject { traitInfo, inst } => {
            return try buildTraitObject(self, val, traitInfo, inst);
        }
        case resolver::Coercion::Identity => return val,
    }
}

/// Lower an implicit numeric cast coercion.
///
/// Handles widening conversions between integer types. Uses sign-extension
/// for signed source types and zero-extension for unsigned source types.
unsafe fn lowerNumericCast(self: &mut FnLowerer, val: il::Val, srcType: resolver::Type, dstType: resolver::Type) -> il::Val {
    let srcLayout = resolver::getTypeLayout(srcType);
    let dstLayout = resolver::getTypeLayout(dstType);

    if srcLayout.size == dstLayout.size {
        // Same size: bit pattern is unchanged, value is returned as-is.
        return val;
    }
    // Widening: extend based on source signedness.
    // Narrowing: truncate and normalize to destination width.
    let widening = srcLayout.size < dstLayout.size;
    let extType = ilType(self.low, srcType) if widening else ilType(self.low, dstType);
    let signed = isSignedType(srcType) if widening else isSignedType(dstType);
    let dst = nextReg(self);

    if signed {
        emit(self, il::Instr::Sext { typ: extType, dst, val });
    } else {
        emit(self, il::Instr::Zext { typ: extType, dst, val });
    }
    return il::Val::Reg(dst);
}

/// Lower a global value symbol.
unsafe fn lowerGlobalValue(self: &mut FnLowerer, sym: *unsafe resolver::Symbol, ty: resolver::Type) -> il::Val {
    // Function pointer reference: return the function's address directly.
    // Functions have no separate storage cell in the data section.
    if let case resolver::Type::Fn(_) = ty {
        return il::Val::Reg(emitFnAddr(self, sym));
    }
    let src = emitDataAddr(self, sym);

    return emitRead(self, src, 0, ty);
}

/// Lower an identifier that refers to a global symbol.
unsafe fn lowerGlobalSymbol(self: &mut FnLowerer, node: *ast::Node) -> il::Val throws (LowerError) {
    // First try to get a compile-time constant value.
    if let constVal = resolver::constValueEntry(self.low.resolver, node) {
        return try constValueToVal(self, constVal, node);
    }
    // Otherwise get the symbol.
    let sym = try symOf(self, node);

    match sym.data {
        case resolver::SymbolData::Constant { type, .. } => {
            let src = emitDataAddr(self, sym);

            return emitRead(self, src, 0, type);
        }
        case resolver::SymbolData::Value { type, .. } =>
            return lowerGlobalValue(self, sym, type),
        else => throw LowerError::UnexpectedNodeValue(node),
    }
}

/// Lower an assignment to a static variable.
unsafe fn lowerStaticAssign(self: &mut FnLowerer, target: *ast::Node, val: il::Val) throws (LowerError) {
    let sym = try symOf(self, target);
    let case resolver::SymbolData::Value { type, .. } = sym.data else {
        throw LowerError::ImmutableAssignment;
    };
    let dst = emitDataAddr(self, sym);

    try emitStore(self, dst, 0, type, val);
}

/// Lower a scope access expression like `Module::Const` or `Union::Variant`.
/// This doesn't handle record literal variants.
unsafe fn lowerScopeAccess(self: &mut FnLowerer, node: *ast::Node) -> il::Val throws (LowerError) {
    // First try to get a compile-time constant value.
    if let constVal = resolver::constValueEntry(self.low.resolver, node) {
        return try constValueToVal(self, constVal, node);
    }
    // Otherwise get the associated symbol.
    let data = resolver::nodeData(self.low.resolver, node);
    let sym = data.sym else {
        throw LowerError::MissingSymbol(node);
    };
    match sym.data {
        case resolver::SymbolData::Variant { index, .. } => {
            let mut indexValue = index as i64;
            if let idx = voidVariantIndex(self.low.resolver, node) {
                set indexValue = idx;
            }
            // Void union variant like `Option::None`.
            if data.ty == resolver::Type::Unknown {
                throw LowerError::MissingType(node);
            }
            // All-void unions are passed as scalars (the tag byte).
            // Return an immediate instead of building a tagged aggregate.
            if resolver::isVoidUnion(data.ty) {
                return il::Val::Imm(indexValue);
            }
            let unionInfo = unionInfoFromType(data.ty) else {
                throw LowerError::MissingMetadata;
            };
            let valOffset = unionInfo.valOffset as i32;
            return try buildTagged(self, resolver::getTypeLayout(data.ty), indexValue, nil, resolver::Type::Void, 1, valOffset);
        }
        case resolver::SymbolData::Constant { type, .. } => {
            // Constant without compile-time value (e.g. record constant);
            // load from data section.
            let src = emitDataAddr(self, sym);

            // Aggregate constants live in read-only memory.  Return a
            // mutable copy so that callers that assign through the
            // resulting pointer do not fault.
            if isAggregateType(type) {
                let layout = resolver::getTypeLayout(type);
                let dst = emitReserveLayout(self, layout);
                emit(self, il::Instr::Blit { dst, src, size: il::Val::Imm(layout.size as i64) });

                return il::Val::Reg(dst);
            }
            return emitRead(self, src, 0, type);
        }
        case resolver::SymbolData::Value { type, .. } =>
            return lowerGlobalValue(self, sym, type),
        else =>
            throw LowerError::UnexpectedNodeValue(node),
    }
}

/// Lower an expression AST node to an IL value.
/// This is the main expression dispatch, all expression nodes go through here.
unsafe fn lowerExpr(self: &mut FnLowerer, node: *ast::Node) -> il::Val throws (LowerError) {
    if self.low.options.debug {
        set self.srcLoc.offset = node.span.offset;
    }
    let mut val: il::Val = undefined;

    match node.value {
        case ast::NodeValue::Ident(_) => {
            // First try local variable lookup.
            // Otherwise fall back to global symbol lookup.
            if let v = lookupLocalVar(self, node) {
                set val = try useVar(self, v);
                if self.vars[*v].addressTaken {
                    let typ = try typeOf(self, node);
                    let ptr = emitValToReg(self, val);
                    set val = emitRead(self, ptr, 0, typ);
                }
            } else {
                set val = try lowerGlobalSymbol(self, node);
            }
        }
        case ast::NodeValue::ScopeAccess(_) => {
            set val = try lowerScopeAccess(self, node);
        }
        case ast::NodeValue::Number(lit) => {
            set val = il::Val::Imm(lit.magnitude as i64);
        }
        case ast::NodeValue::Bool(b) => {
            set val = il::Val::Imm(1) if b else il::Val::Imm(0);
        }
        case ast::NodeValue::Char(c) => {
            set val = il::Val::Imm(c as i64);
        }
        case ast::NodeValue::Nil => {
            let typ = try typeOf(self, node);
            if let case resolver::Type::Optional(_) = typ {
                set val = try buildNilOptional(self, typ);
            } else if let case resolver::Type::Nil = typ {
                // Standalone `nil` without a concrete optional type. We can't
                // generate a proper value representation.
                throw LowerError::MissingType(node);
            } else {
                throw LowerError::NilInNonOptional;
            }
        }
        case ast::NodeValue::RecordLit(lit) => {
            set val = try lowerRecordLit(self, node, lit);
        }
        case ast::NodeValue::AddressOf(addr) => {
            set val = try lowerAddressOf(self, node, addr);
        }
        case ast::NodeValue::Deref(target) => {
            set val = try lowerDeref(self, node, target);
        }
        case ast::NodeValue::BinOp(binop) => {
            set val = try lowerBinOp(self, node, binop);
        }
        case ast::NodeValue::UnOp(unop) => {
            set val = try lowerUnOp(self, node, unop);
        }
        case ast::NodeValue::Subscript { container, index } => {
            set val = try lowerSubscript(self, node, container, index);
        }
        case ast::NodeValue::BuiltinCall { kind, args } => {
            set val = try lowerBuiltinCall(self, node, kind, args);
        }
        case ast::NodeValue::Call(call) => {
            set val = try lowerCallOrCtor(self, node, call);
        }
        case ast::NodeValue::Try(t) => {
            set val = try lowerTry(self, node, t);
        }
        case ast::NodeValue::FieldAccess(access) => {
            // Check for compile-time constant (e.g., `arr.len` on fixed-size arrays).
            if let constVal = resolver::constValueEntry(self.low.resolver, node) {
                match constVal {
                    // TODO: Handle `u32` values that don't fit in an `i32`.
                    //       Perhaps just store the `ConstInt`.
                    case resolver::ConstValue::Int(i) => set val = il::Val::Imm(constIntToI64(i)),
                    else => set val = try lowerFieldAccess(self, access),
                }
            } else {
                set val = try lowerFieldAccess(self, access);
            }
        }
        case ast::NodeValue::ArrayLit(elements) => {
            set val = try lowerArrayLit(self, node, elements);
        }
        case ast::NodeValue::ArrayRepeatLit(repeat) => {
            set val = try lowerArrayRepeatLit(self, node, repeat);
        }
        case ast::NodeValue::As(cast) => {
            set val = try lowerCast(self, node, cast);
        }
        case ast::NodeValue::CondExpr(cond) => {
            set val = try lowerCondExpr(self, node, cond);
        }
        case ast::NodeValue::String(s) => {
            set val = try lowerStringLit(self, node, s);
        }
        case ast::NodeValue::Undef => {
            let typ = try typeOf(self, node);
            if isAggregateType(typ) {
                // When `undefined` appears as a stand-alone expression,
                /// we need a stack slot for reads and writes.
                let slot = try emitReserve(self, typ);
                set val = il::Val::Reg(slot);
            } else {
                set val = il::Val::Undef;
            }
        }
        case ast::NodeValue::Panic { .. } => {
            // Panic in expression context (e.g. match arm). Emit unreachable
            // and return a dummy value since control won't continue.
            emit(self, il::Instr::Unreachable);
            set val = il::Val::Undef;
        }
        case ast::NodeValue::Assert { .. } => {
            // Assert in expression context. Lower as statement, return `void`.
            try lowerNode(self, node);
            set val = il::Val::Undef;
        }
        case ast::NodeValue::Block(_) => {
            try lowerBlock(self, node);
            set val = il::Val::Undef;
        }
        case ast::NodeValue::ExprStmt(expr) => {
            let _ = expr;
            set val = il::Val::Undef;
        }
        // Lower these as statements.
        case ast::NodeValue::ConstDecl(decl) => {
            try registerLocalDataDeclName(self, node);
            try lowerDataDecl(self.low, node, decl.value, true);
            set val = il::Val::Undef;
        }
        case ast::NodeValue::StaticDecl(decl) => {
            try registerLocalDataDeclName(self, node);
            try lowerDataDecl(self.low, node, decl.value, false);
            set val = il::Val::Undef;
        }
        case ast::NodeValue::Throw { .. },
             ast::NodeValue::Return { .. },
             ast::NodeValue::Continue,
             ast::NodeValue::Break => {
            try lowerNode(self, node);
            set val = il::Val::Undef;
        }
        else => {
            panic "lowerExpr: node is not an expression";
        }
    }
    return try applyCoercion(self, node, val);
}

/// Translate a Radiance type to an IL type.
///
/// The IL type system is much simpler than Radiance's, only primitive types
/// are used. If a Radiance type doesn't fit in a machine word, it is passed
/// by reference.
///
/// The IL doesn't track signedness - that's encoded in the instructions
/// (e.g., Slt vs Ult).
unsafe fn ilType(self: &mut Lowerer, typ: resolver::Type) -> il::Type {
    match typ {
        case resolver::Type::Bool,
             resolver::Type::I8,
             resolver::Type::U8 => return il::Type::W8,
        case resolver::Type::I16,
             resolver::Type::U16 => return il::Type::W16,
        case resolver::Type::I32,
             resolver::Type::U32 => return il::Type::W32,
        case resolver::Type::I64,
             resolver::Type::U64,
             resolver::Type::Pointer { .. },
             resolver::Type::Slice { .. },
             resolver::Type::TraitObject { .. },
             resolver::Type::Array(_),
             resolver::Type::Optional(_),
             resolver::Type::Fn(_) => return il::Type::W64,
        case resolver::Type::Nominal(_) => {
            if resolver::isVoidUnion(typ) {
                return il::Type::W8;
            }
            return il::Type::W64;
        }
        case resolver::Type::Void, resolver::Type::Never => return il::Type::W64,
        // [`Type::Int`] is the type of unsuffixed integer literals and their
        // compound expressions (e.g. `1 + 2`). It defaults to W64 (i64) here,
        // matching the native word size on RV64. It cannot be resolved earlier
        // because the resolver uses [`Type::Int`] to distinguish unsuffixed
        // expressions from explicitly typed ones, which affects coercion
        // behavior (e.g. implicit narrowing).
        case resolver::Type::Int => return il::Type::W64,
        case resolver::Type::Opaque => panic "ilType: opaque type must be behind a pointer",
        else => panic "ilType: type cannot be lowered",
    }
}