kernel: preserve recipient page lifetimes across handle changes

53c5af21ba197b25b7741bbd29a418ab41865fbe5e92314153ba39abe96b54e6
Verified: make -C kernel check with the machine-capable emulator; all pass.
Alexis Sellier committed ago 1 parent 806b7604
kernel/Makefile +4 -3
2 2
EMU ?= $(or $(RAD_EMULATOR),emulator)
3 3
HOST_EMU ?= $(EMU)
4 4
COMPILER := ../bin/radiance.rv64.dev
5 5
COMPILE := $(HOST_EMU) -memory-size=385024 -data-size=348160 -stack-size=512 -run $(COMPILER)
6 6
MODULES := core/fdt.rad core/platform.rad core/frames.rad core/abi.rad core/handles.rad \
7 -
	core/events.rad core/domains.rad core/resources.rad core/capabilities.rad
7 +
	core/events.rad core/domains.rad core/resources.rad core/capabilities.rad \
8 +
	core/memory.rad core/state.rad core/pages.rad
8 9
CORE := -pkg core -mod core.rad $(addprefix -mod ,$(MODULES))
9 10
CHECK_MODULES := check/boot.rad check/fixture.rad check/frames.rad check/handles.rad \
10 -
	check/domains.rad check/capabilities.rad
11 +
	check/domains.rad check/capabilities.rad check/pages.rad
11 12
12 13
.PHONY: all check clean compiler-check
13 14
all: kernel.rv64
14 15
15 16
compiler-check:
23 24
check.rv64: check.rad core.rad $(MODULES) $(CHECK_MODULES) $(COMPILER)
24 25
	$(COMPILE) $(CORE) -pkg check -mod check.rad $(addprefix -mod ,$(CHECK_MODULES)) -entry check -o $@
25 26
26 27
check: all check.rv64
27 28
	$(HOST_EMU) -run check.rv64
28 -
	$(EMU) -machine -no-guard-stack -max-steps=1000000 -count-instructions -run kernel.rv64
29 +
	$(EMU) -machine -no-guard-stack -max-steps=100000000 -count-instructions -run kernel.rv64
29 30
30 31
clean:
31 32
	rm -f kernel.rv64 check.rv64
kernel/NOTES.md +16 -5
1 1
# Kernel implementation decisions
2 2
3 3
The specification at https://radiant.computer/system/kernel takes precedence
4 4
for fixed call numbers, handle layout, rights, and object behavior. These notes
5 -
record the contracts established through step 7 of the 22-step plan.
5 +
record the contracts established through step 8 of the 22-step plan.
6 6
7 7
## Source and trust boundary
8 8
9 9
- Kernel mechanisms use freestanding Radiance; RAS owns machine entry, register
10 10
  state, atomics, and MMIO. Hosted checks exercise the same mechanism modules.
74 74
- Transfer requires Transfer and a different target domain. Check the source,
75 75
  live target, and destination capacity before moving authority. Failure leaves
76 76
  the source usable. Object liveness is checked independently of the handle.
77 77
- Events cannot cross domains; the installed Events handle cannot be dropped.
78 78
  Interrupt authority cannot be granted and moves exclusively by transfer.
79 -
- Dropping a handle removes authority to invoke operations through that name
80 -
  and releases its object reference.
79 +
- Dropping a handle removes authority to invoke operations through that name;
80 +
  it does not revoke memory pointers already derived from a persistent grant.
81 81
82 82
## Aligned aggregate code generation
83 83
84 84
- Aggregate copies retain their proven alignment. Small aggregate register
85 85
  returns load exactly their layout bytes with valid alignment. This does not
87 87
- Machine execution uses the current machine-capable sibling emulator:
88 88
  FDT boot registers, independent hart state, shared coherent RAM, WFI wakeup,
89 89
  CLINT, PLIC, UART, and atomics. Explicit M-mode finish writes end checks;
90 90
  they are not a domain-exit operation. The boot checks for this boundary leave secondary harts idle.
91 91
92 +
## Persistent Page lifetime and splitting
93 +
94 +
- Persistent grants pin frames for each recipient domain lifetime, not merely
95 +
  the origin lifetime. A recipient can retain a derived pointer after dropping
96 +
  its Page handle and after the origin dies. Splits preserve these pins.
97 +
- Domain-major frame bitmaps and per-frame pin counts track lifetime access.
98 +
  A frame returns to the pool only after its Page object claim and all pins
99 +
  disappear. Clear a quiescent domain's bitmap before reusing its identity.
100 +
- Allocation preflights capability/object slots before taking RAM. PageSplit
101 +
  checks exclusivity and both destination slots before changing the source.
102 +
  Readable coverage may span adjacent grants; a gap never authorizes access.
103 +
92 104
## Validation
93 105
94 106
Use the current machine-capable sibling emulator. Set `RAD_EMULATOR`, pass
95 107
`EMU` to the kernel Make invocation, or put `emulator` on PATH. The kernel build
96 108
checks compiler dependencies. From the repository root, run:
97 109
98 110
```sh
99 111
make -C kernel check
100 -
make std-test bin-test
101 112
```
102 113
103 -
Exercise aggregate-alignment machine regression and firmware boot on 1 and 8 harts, with secondary harts idle.
114 +
Exercise split exclusivity, failed allocation/split transactions, pointers retained after handle drop, and reclamation after the final recipient lifetime.
kernel/check.rad +2 -0
4 4
export mod fixture;
5 5
mod frames;
6 6
mod handles;
7 7
mod domains;
8 8
mod capabilities;
9 +
mod pages;
9 10
10 11
/// Run the available kernel mechanism checks.
11 12
@default fn main() -> u32 {
12 13
    frames::run();
13 14
    boot::run();
14 15
    handles::run();
15 16
    domains::run();
16 17
    capabilities::run();
18 +
    pages::run();
17 19
    return 0;
18 20
}
kernel/check/capabilities.rad +38 -17
4 4
use core::capabilities;
5 5
use core::domains;
6 6
use core::fdt;
7 7
use core::handles;
8 8
use core::resources;
9 +
use core::frames;
10 +
use core::memory;
11 +
use core::platform;
12 +
use core::state;
9 13
10 14
/// Domain storage for capability transactions.
11 15
static DOMAINS: [domains::Domain; 3] = undefined;
12 16
/// Physical resources used by the transactions.
13 17
static OBJECTS: [resources::Slot; 4] = undefined;
18 +
/// Empty RAM pool for device-only capability transactions.
19 +
static POOL: frames::Pool = undefined;
20 +
/// Frame pin backing storage.
21 +
static PINS: [u16; 1] = undefined;
22 +
/// Frame object backing storage.
23 +
static ASSIGNED: [bool; 1] = undefined;
24 +
/// Per-domain grant backing storage.
25 +
static GRANTS: [u64; 3] = undefined;
26 +
/// Shared mechanism state for each transaction.
27 +
static KERNEL: state::State = undefined;
14 28
15 29
/// Return a grant error as a test value.
16 30
fn grantError(source: abi::Handle, target: abi::Handle) -> abi::Error {
17 -
    let _handle = try capabilities::grant(&mut DOMAINS[..], &mut OBJECTS[..], 0, source, target, abi::RIGHTS as u64) catch error {
31 +
    let _handle = try capabilities::grant(&mut KERNEL, 0, source, target, abi::RIGHTS as u64) catch error {
18 32
        return error;
19 33
    };
20 34
    return abi::Error::Ok;
21 35
}
22 36
23 37
/// Check attenuation, non-delegable queues, unique IRQs, and failed moves.
24 38
export fn run() {
25 39
    domains::init(&mut DOMAINS[..]);
26 40
    resources::init(&mut OBJECTS[..]);
41 +
    let mut machine: platform::Platform = undefined;
42 +
    set machine.memoryCount = 0;
43 +
    set machine.reservedCount = 0;
44 +
    try! frames::init(&mut POOL, &machine);
45 +
    let mut ram: memory::Memory = undefined;
46 +
    memory::init(&mut ram, &mut POOL, &mut PINS[..], &mut ASSIGNED[..], &mut GRANTS[..], 3);
47 +
    set KERNEL = state::State { domains: &mut DOMAINS[..], resources: &mut OBJECTS[..], memory: ram };
27 48
    let rootHandle = domains::root(&mut DOMAINS[..], 0);
28 49
    let root = try! domains::resolve(&DOMAINS[..], 0, rootHandle, abi::CREATE);
29 50
    let child = try! domains::create(&mut DOMAINS[..], root, 1);
30 51
    let childHandle = handles::install(&mut DOMAINS[0].handles, 2, child, abi::DOMAIN_RIGHTS);
31 52
    let device = try! resources::create(&mut OBJECTS[..], resources::Value::Device(fdt::Range { base: 0x10000000, size: 256 }));
32 53
    let source = handles::install(&mut DOMAINS[0].handles, 3, device, abi::READ | abi::GRANT | abi::TRANSFER);
33 54
    resources::retain(&mut OBJECTS[..], device);
34 -
    let shared = try! capabilities::grant(&mut DOMAINS[..], &mut OBJECTS[..], 0, source, childHandle, (abi::READ | abi::WRITE) as u64);
35 -
    let authority = try! capabilities::lookup(&DOMAINS[..], &OBJECTS[..], child.index, shared);
55 +
    let shared = try! capabilities::grant(&mut KERNEL, 0, source, childHandle, (abi::READ | abi::WRITE) as u64);
56 +
    let authority = try! capabilities::lookup(&KERNEL, child.index, shared);
36 57
    assert authority.rights == abi::READ;
37 -
    let _source = try! capabilities::lookup(&DOMAINS[..], &OBJECTS[..], 0, source);
38 -
    let narrowed = try! capabilities::grant(&mut DOMAINS[..], &mut OBJECTS[..], child.index, shared, abi::Handle { bits: 0 }, abi::READ as u64);
39 -
    let _local = try! capabilities::lookup(&DOMAINS[..], &OBJECTS[..], child.index, narrowed);
58 +
    let _source = try! capabilities::lookup(&KERNEL, 0, source);
59 +
    let narrowed = try! capabilities::grant(&mut KERNEL, child.index, shared, abi::Handle { bits: 0 }, abi::READ as u64);
60 +
    let _local = try! capabilities::lookup(&KERNEL, child.index, narrowed);
40 61
41 62
    let queue = try! handles::get(&DOMAINS[0].handles, 1, abi::Kind::Events);
42 63
    assert grantError(queue, childHandle) == abi::Error::Denied;
43 -
    let readQueue = try! capabilities::grant(&mut DOMAINS[..], &mut OBJECTS[..], 0, queue, abi::Handle { bits: 0 }, abi::READ as u64);
44 -
    try! capabilities::drop(&mut DOMAINS[..], &mut OBJECTS[..], 0, readQueue);
45 -
    try capabilities::drop(&mut DOMAINS[..], &mut OBJECTS[..], 0, queue) catch error {
64 +
    let readQueue = try! capabilities::grant(&mut KERNEL, 0, queue, abi::Handle { bits: 0 }, abi::READ as u64);
65 +
    try! capabilities::drop(&mut KERNEL, 0, readQueue);
66 +
    try capabilities::drop(&mut KERNEL, 0, queue) catch error {
46 67
        assert error == abi::Error::Denied;
47 68
    };
48 -
    let _queue = try! capabilities::lookup(&DOMAINS[..], &OBJECTS[..], 0, queue);
69 +
    let _queue = try! capabilities::lookup(&KERNEL, 0, queue);
49 70
50 71
    let irq = try! resources::create(&mut OBJECTS[..], resources::Value::Interrupt(resources::Interrupt { number: 10, target: root }));
51 72
    let irqSlot = try! handles::vacant(&DOMAINS[0].handles);
52 73
    let interrupt = handles::install(&mut DOMAINS[0].handles, irqSlot, irq, abi::TRANSFER);
53 74
    resources::retain(&mut OBJECTS[..], irq);
54 75
    assert grantError(interrupt, childHandle) == abi::Error::Denied;
55 76
    assert grantError(interrupt, abi::Handle { bits: 0 }) == abi::Error::Denied;
56 -
    let moved = try! capabilities::transfer(&mut DOMAINS[..], &mut OBJECTS[..], 0, interrupt, childHandle, abi::TRANSFER as u64);
57 -
    let _receiver = try! capabilities::lookup(&DOMAINS[..], &OBJECTS[..], child.index, moved);
58 -
    let _old = try capabilities::lookup(&DOMAINS[..], &OBJECTS[..], 0, interrupt) catch error {
77 +
    let moved = try! capabilities::transfer(&mut KERNEL, 0, interrupt, childHandle, abi::TRANSFER as u64);
78 +
    let _receiver = try! capabilities::lookup(&KERNEL, child.index, moved);
79 +
    let _old = try capabilities::lookup(&KERNEL, 0, interrupt) catch error {
59 80
        assert error == abi::Error::BadHandle;
60 81
        for i in 0..abi::MAX_HANDLES - 4 {
61 -
            let _copy = try! capabilities::grant(&mut DOMAINS[..], &mut OBJECTS[..], 0, source, childHandle, abi::READ as u64);
82 +
            let _copy = try! capabilities::grant(&mut KERNEL, 0, source, childHandle, abi::READ as u64);
62 83
        }
63 -
        let _failed = try capabilities::transfer(&mut DOMAINS[..], &mut OBJECTS[..], 0, source, childHandle, abi::READ as u64) catch full {
84 +
        let _failed = try capabilities::transfer(&mut KERNEL, 0, source, childHandle, abi::READ as u64) catch full {
64 85
            assert full == abi::Error::InvalidArg;
65 -
            let _stillLive = try! capabilities::lookup(&DOMAINS[..], &OBJECTS[..], 0, source);
66 -
            try! capabilities::drop(&mut DOMAINS[..], &mut OBJECTS[..], 0, childHandle);
86 +
            let _stillLive = try! capabilities::lookup(&KERNEL, 0, source);
87 +
            try! capabilities::drop(&mut KERNEL, 0, childHandle);
67 88
            assert domains::live(&DOMAINS[..], child);
68 89
            return;
69 90
        };
70 91
        panic "run: transfer into a full table succeeded";
71 92
    };
kernel/check/pages.rad added +86 -0
1 +
//! Page authority, split exclusivity, and persistent recipient lifetime checks.
2 +
3 +
use core::abi;
4 +
use core::capabilities;
5 +
use core::domains;
6 +
use core::fdt;
7 +
use core::frames;
8 +
use core::handles;
9 +
use core::memory;
10 +
use core::pages;
11 +
use core::platform;
12 +
use core::resources;
13 +
use core::state;
14 +
15 +
/// Test-owned physical RAM.
16 +
static RAM: [u8; 16 * frames::PAGE_SIZE] = undefined;
17 +
/// Domain storage for allocator and recipient authority.
18 +
static DOMAINS: [domains::Domain; 3] = undefined;
19 +
/// Physical Page object slots.
20 +
static OBJECTS: [resources::Slot; 8] = undefined;
21 +
/// Frame availability metadata.
22 +
static POOL: frames::Pool = undefined;
23 +
/// Recipient pin counts.
24 +
static PINS: [u16; 16] = undefined;
25 +
/// Live Page claims on each frame.
26 +
static ASSIGNED: [bool; 16] = undefined;
27 +
/// Per-domain persistent grant sets.
28 +
static GRANTS: [u64; 3] = undefined;
29 +
/// State shared by all tested control-plane operations.
30 +
static KERNEL: state::State = undefined;
31 +
32 +
/// Clear a physical allocation before it is exposed through a handle.
33 +
fn clear(base: u64, size: u32) {
34 +
    assert base <= RAM.len as u64 and size as u64 <= RAM.len as u64 - base;
35 +
    for i in base as u32..base as u32 + size { set RAM[i] = 0; }
36 +
}
37 +
38 +
/// Check split metadata, attenuation, and reclamation after every recipient dies.
39 +
export fn run() {
40 +
    let mut machine: platform::Platform = undefined;
41 +
    set machine.memoryCount = 1;
42 +
    set machine.memory[0] = fdt::Range { base: 0, size: RAM.len as u64 };
43 +
    set machine.reservedCount = 0;
44 +
    try! frames::init(&mut POOL, &machine);
45 +
    let mut ram: memory::Memory = undefined;
46 +
    memory::init(&mut ram, &mut POOL, &mut PINS[..], &mut ASSIGNED[..], &mut GRANTS[..], 3);
47 +
    domains::init(&mut DOMAINS[..]);
48 +
    resources::init(&mut OBJECTS[..]);
49 +
    set KERNEL = state::State { domains: &mut DOMAINS[..], resources: &mut OBJECTS[..], memory: ram };
50 +
    let rootHandle = domains::root(&mut DOMAINS[..], 0);
51 +
    let root = try! domains::resolve(&DOMAINS[..], 0, rootHandle, abi::CREATE);
52 +
    let child = try! domains::create(&mut DOMAINS[..], root, 1);
53 +
    let childHandle = handles::install(&mut DOMAINS[0].handles, 2, child, abi::DOMAIN_RIGHTS);
54 +
    let left = try! pages::allocate(&mut KERNEL, 0, abi::Handle { bits: 0 }, 4, clear);
55 +
    let right = try! pages::split(&mut KERNEL, 0, left, 2);
56 +
    let leftRange = try! pages::access(&KERNEL, 0, left, abi::WRITE);
57 +
    let rightRange = try! pages::access(&KERNEL, 0, right, abi::WRITE);
58 +
    assert leftRange.base == 0 and leftRange.size == 8192;
59 +
    assert rightRange.base == 8192 and rightRange.size == 8192;
60 +
    assert pages::covers(&KERNEL, 0, 0, 16384, abi::WRITE);
61 +
    assert not pages::covers(&KERNEL, 0, 0, 16385, abi::WRITE);
62 +
    assert not pages::covers(&KERNEL, 0, 0xfffffffffffffff0, 32, abi::READ);
63 +
    let shared = try! capabilities::grant(&mut KERNEL, 0, left, childHandle, abi::READ as u64);
64 +
    let _split = try pages::split(&mut KERNEL, 0, left, 1) catch error {
65 +
        assert error == abi::Error::Busy;
66 +
        let borrowed = try! pages::access(&KERNEL, child.index, shared, abi::READ);
67 +
        assert borrowed.base == leftRange.base and borrowed.size == leftRange.size;
68 +
        assert not pages::covers(&KERNEL, child.index, 0, 8192, abi::WRITE);
69 +
        try! capabilities::drop(&mut KERNEL, child.index, shared);
70 +
        try! capabilities::drop(&mut KERNEL, 0, left);
71 +
        try! capabilities::drop(&mut KERNEL, 0, right);
72 +
        assert POOL.available == 12;
73 +
        memory::retire(&mut KERNEL.memory, 0);
74 +
        assert POOL.available == 14;
75 +
        let allocation = try! frames::allocate(&mut POOL, 14, clear);
76 +
        let untouched = frames::install(allocation);
77 +
        assert untouched.first == 2;
78 +
        frames::reclaim(&mut POOL, untouched);
79 +
        memory::retire(&mut KERNEL.memory, child.index);
80 +
        assert POOL.available == 16;
81 +
        let entire = try! frames::allocate(&mut POOL, 16, clear);
82 +
        frames::discard(&mut POOL, entire);
83 +
        return;
84 +
    };
85 +
    panic "run: shared Page split succeeded";
86 +
}
kernel/core.rad +3 -0
7 7
export mod handles;
8 8
export mod events;
9 9
export mod domains;
10 10
export mod resources;
11 11
export mod capabilities;
12 +
export mod memory;
13 +
export mod state;
14 +
export mod pages;
kernel/core/capabilities.rad +28 -11
2 2
3 3
use core::abi;
4 4
use core::domains;
5 5
use core::handles;
6 6
use core::resources;
7 +
use core::memory;
8 +
use core::state;
7 9
8 10
/// Resolve source authority and check the named object's incarnation.
9 -
export fn lookup(store: *[domains::Domain], objects: *[resources::Slot], caller: u32, handle: abi::Handle) -> handles::Entry throws (abi::Error) {
10 -
    let slot = try handles::slot(&store[caller].handles, handle);
11 -
    let entry = store[caller].handles.entries[slot];
12 -
    if not resources::live(objects, store, entry.object) { throw abi::Error::BadHandle; }
11 +
export fn lookup(kernel: *state::State, caller: u32, handle: abi::Handle) -> handles::Entry throws (abi::Error) {
12 +
    let slot = try handles::slot(&kernel.domains[caller].handles, handle);
13 +
    let entry = kernel.domains[caller].handles.entries[slot];
14 +
    if not resources::live(kernel.resources, kernel.domains, entry.object) { throw abi::Error::BadHandle; }
13 15
    return entry;
14 16
}
15 17
16 18
/// Copy authority with attenuation after all failure checks complete.
17 -
export fn grant(store: *mut [domains::Domain], objects: *mut [resources::Slot], caller: u32, source: abi::Handle, target: abi::Handle, rights: u64) -> abi::Handle throws (abi::Error) {
18 -
    let entry = try lookup(store, objects, caller, source);
19 +
export fn grant(kernel: *mut state::State, caller: u32, source: abi::Handle, target: abi::Handle, rights: u64) -> abi::Handle throws (abi::Error) {
20 +
    let entry = try lookup(kernel, caller, source);
21 +
    let store = kernel.domains;
22 +
    let objects = kernel.resources;
19 23
    let mut recipient = abi::Object { kind: abi::Kind::Domain, index: caller, epoch: store[caller].epoch };
20 24
    if target.bits <> 0 { set recipient = try domains::resolve(store, caller, target, 0); }
21 25
    if entry.object.kind == abi::Kind::Interrupt { throw abi::Error::Denied; }
22 26
    if recipient.index <> caller {
23 27
        if entry.object.kind == abi::Kind::Events or entry.rights & abi::GRANT == 0 {
24 28
            throw abi::Error::Denied;
25 29
        }
26 30
    }
27 31
    let slot = try handles::vacant(&store[recipient.index].handles) catch { throw abi::Error::InvalidArg; };
32 +
    if entry.object.kind == abi::Kind::Page {
33 +
        let case resources::Value::Page(page) = objects[entry.object.index].value else { panic "grant: page kind mismatch"; };
34 +
        memory::pin(&mut kernel.memory, recipient.index, page.run);
35 +
    }
28 36
    resources::retain(objects, entry.object);
29 37
    return handles::install(&mut store[recipient.index].handles, slot, entry.object, entry.rights & rights as u16);
30 38
}
31 39
32 40
/// Move authority as one transaction; failure leaves the source live.
33 -
export fn transfer(store: *mut [domains::Domain], objects: *mut [resources::Slot], caller: u32, source: abi::Handle, target: abi::Handle, rights: u64) -> abi::Handle throws (abi::Error) {
34 -
    let entry = try lookup(store, objects, caller, source);
41 +
export fn transfer(kernel: *mut state::State, caller: u32, source: abi::Handle, target: abi::Handle, rights: u64) -> abi::Handle throws (abi::Error) {
42 +
    let entry = try lookup(kernel, caller, source);
43 +
    let store = kernel.domains;
44 +
    let objects = kernel.resources;
35 45
    let recipient = try domains::resolve(store, caller, target, 0);
36 46
    if recipient.index == caller { throw abi::Error::InvalidArg; }
37 47
    if entry.object.kind == abi::Kind::Events or entry.rights & abi::TRANSFER == 0 {
38 48
        throw abi::Error::Denied;
39 49
    }
40 50
    let slot = try handles::vacant(&store[recipient.index].handles) catch { throw abi::Error::InvalidArg; };
41 51
    let sourceSlot = try handles::slot(&store[caller].handles, source);
52 +
    if entry.object.kind == abi::Kind::Page {
53 +
        let case resources::Value::Page(page) = objects[entry.object.index].value else { panic "transfer: page kind mismatch"; };
54 +
        memory::pin(&mut kernel.memory, recipient.index, page.run);
55 +
    }
42 56
    resources::retarget(objects, entry.object, recipient);
43 57
    let result = handles::install(&mut store[recipient.index].handles, slot, entry.object, entry.rights & rights as u16);
44 58
    let _previous = handles::remove(&mut store[caller].handles, sourceSlot);
45 59
    return result;
46 60
}
47 61
48 62
/// Drop a live local capability without revoking persistent pointer access.
49 -
export fn drop(store: *mut [domains::Domain], objects: *mut [resources::Slot], caller: u32, handle: abi::Handle) throws (abi::Error) {
50 -
    let entry = try lookup(store, objects, caller, handle);
63 +
export fn drop(kernel: *mut state::State, caller: u32, handle: abi::Handle) throws (abi::Error) {
64 +
    let entry = try lookup(kernel, caller, handle);
65 +
    let store = kernel.domains;
51 66
    let slot = try handles::slot(&store[caller].handles, handle);
52 67
    if entry.object.kind == abi::Kind::Events and slot == 1 { throw abi::Error::Denied; }
53 68
    let _previous = handles::remove(&mut store[caller].handles, slot);
54 -
    resources::release(objects, entry.object);
69 +
    if let run = resources::release(kernel.resources, entry.object) {
70 +
        memory::release(&mut kernel.memory, run);
71 +
    }
55 72
}
kernel/core/fdt.rad +1 -1
154 154
/// Store one relevant property after validating its representation.
155 155
fn property(node: *mut Node, name: *[u8], value: *[u8]) throws (Error) {
156 156
    if equal(name, "#address-cells") {
157 157
        if value.len <> 4 { throw Error::Invalid; }
158 158
        set node.addressCells = try word(value, 0);
159 -
        if node.addressCells == 0 or node.addressCells > 2 { throw Error::Invalid; }
159 +
        if node.addressCells > 2 { throw Error::Invalid; }
160 160
    } else if equal(name, "#size-cells") {
161 161
        if value.len <> 4 { throw Error::Invalid; }
162 162
        set node.sizeCells = try word(value, 0);
163 163
        if node.sizeCells > 2 { throw Error::Invalid; }
164 164
    } else if equal(name, "reg") {
kernel/core/memory.rad added +99 -0
1 +
//! Physical frame pins for persistent domain-lifetime memory access.
2 +
3 +
use core::abi;
4 +
use core::frames;
5 +
6 +
/// Borrowed frame lifetime metadata allocated at kernel boot.
7 +
export record Memory: Copy {
8 +
    /// Global availability bitmap.
9 +
    pool: *mut frames::Pool,
10 +
    /// Number of live domain-lifetime pins for each physical frame.
11 +
    pins: *mut [u16],
12 +
    /// Whether each frame still belongs to a live Page object.
13 +
    objects: *mut [bool],
14 +
    /// Domain-major bitmaps of persistent grants.
15 +
    grants: *mut [u64],
16 +
    /// Bitmap words per domain.
17 +
    words: u32,
18 +
    /// Number of domain slots with lifetime metadata.
19 +
    domains: u32,
20 +
}
21 +
22 +
/// Initialize frame lifetime metadata before the first user allocation.
23 +
export fn init(memory: *mut Memory, pool: *mut frames::Pool, pins: *mut [u16], objects: *mut [bool], grants: *mut [u64], domainCount: u32) {
24 +
    let words = (pool.limit + 63) / 64;
25 +
    assert domainCount > 0 and domainCount <= abi::MAX_DOMAINS;
26 +
    assert pins.len >= pool.limit and objects.len >= pool.limit;
27 +
    assert grants.len >= words * domainCount;
28 +
    for i in 0..pool.limit {
29 +
        set pins[i] = 0;
30 +
        set objects[i] = false;
31 +
    }
32 +
    for i in 0..words * domainCount { set grants[i] = 0; }
33 +
    set *memory = Memory { pool, pins, objects, grants, words, domains: domainCount };
34 +
}
35 +
36 +
/// Pin each frame once for the receiving domain's complete lifetime.
37 +
export fn pin(memory: *mut Memory, domain: u32, run: frames::Run) {
38 +
    assert domain < memory.domains and run.first <= memory.pool.limit;
39 +
    assert run.count > 0 and run.count <= memory.pool.limit - run.first;
40 +
    for frame in run.first..run.first + run.count {
41 +
        let word = domain * memory.words + frame / 64;
42 +
        let bit = 1 as u64 << (frame as u64 & 63);
43 +
        if memory.grants[word] & bit == 0 {
44 +
            assert memory.objects[frame];
45 +
            assert memory.pins[frame] < memory.domains as u16;
46 +
            set memory.grants[word] |= bit;
47 +
            set memory.pins[frame] += 1;
48 +
        }
49 +
    }
50 +
}
51 +
52 +
/// Allocate cleared frames and establish the origin domain's lifetime pin.
53 +
export fn allocate(memory: *mut Memory, domain: u32, count: u32, clear: fn(u64, u32)) -> frames::Run throws (abi::Error) {
54 +
    let allocation = try frames::allocate(memory.pool, count, clear) catch error {
55 +
        if error == frames::Error::Invalid { throw abi::Error::InvalidArg; }
56 +
        throw abi::Error::OutOfMemory;
57 +
    };
58 +
    let run = frames::install(allocation);
59 +
    for frame in run.first..run.first + run.count {
60 +
        assert not memory.objects[frame] and memory.pins[frame] == 0;
61 +
        set memory.objects[frame] = true;
62 +
    }
63 +
    pin(memory, domain, run);
64 +
    return run;
65 +
}
66 +
67 +
/// Release a Page object's claim; existing domain pins still protect its data.
68 +
export fn release(memory: *mut Memory, run: frames::Run) {
69 +
    for frame in run.first..run.first + run.count {
70 +
        assert memory.objects[frame];
71 +
        set memory.objects[frame] = false;
72 +
        if memory.pins[frame] == 0 {
73 +
            frames::reclaim(memory.pool, frames::Run { first: frame, count: 1 });
74 +
        }
75 +
    }
76 +
}
77 +
78 +
/// Clear a quiescent domain's persistent grants before its slot can be reused.
79 +
export fn retire(memory: *mut Memory, domain: u32) {
80 +
    assert domain < memory.domains;
81 +
    for word in 0..memory.words {
82 +
        let index = domain * memory.words + word;
83 +
        let mut bits = memory.grants[index];
84 +
        set memory.grants[index] = 0;
85 +
        let mut bit: u32 = 0;
86 +
        while bits <> 0 {
87 +
            if bits & 1 <> 0 {
88 +
                let frame = word * 64 + bit;
89 +
                assert memory.pins[frame] > 0;
90 +
                set memory.pins[frame] -= 1;
91 +
                if memory.pins[frame] == 0 and not memory.objects[frame] {
92 +
                    frames::reclaim(memory.pool, frames::Run { first: frame, count: 1 });
93 +
                }
94 +
            }
95 +
            set bits >>= 1;
96 +
            set bit += 1;
97 +
        }
98 +
    }
99 +
}
kernel/core/pages.rad added +84 -0
1 +
//! Capability-authorized physical Pages and bounded access metadata.
2 +
3 +
use core::abi;
4 +
use core::capabilities;
5 +
use core::domains;
6 +
use core::fdt;
7 +
use core::frames;
8 +
use core::handles;
9 +
use core::memory;
10 +
use core::resources;
11 +
use core::state;
12 +
13 +
/// Allocate cleared frames under the caller's explicit Allocate authority.
14 +
export fn allocate(kernel: *mut state::State, caller: u32, authority: abi::Handle, count: u64, clear: fn(u64, u32)) -> abi::Handle throws (abi::Error) {
15 +
    let owner = try domains::authority(kernel.domains, caller, authority, abi::ALLOCATE);
16 +
    if owner.index <> caller { throw abi::Error::Denied; }
17 +
    if count == 0 or count > frames::MAX_FRAMES as u64 { throw abi::Error::InvalidArg; }
18 +
    let slot = try handles::vacant(&kernel.domains[caller].handles) catch { throw abi::Error::InvalidArg; };
19 +
    let objectSlot = try resources::vacant(kernel.resources) catch { throw abi::Error::OutOfMemory; };
20 +
    let run = try memory::allocate(&mut kernel.memory, caller, count as u32, clear);
21 +
    let object = resources::install(kernel.resources, objectSlot, resources::Value::Page(resources::Page { run, origin: owner }));
22 +
    resources::retain(kernel.resources, object);
23 +
    return handles::install(&mut kernel.domains[caller].handles, slot, object, abi::PAGE_RIGHTS);
24 +
}
25 +
26 +
/// Split an exclusive allocation without changing its frame lifetime pins.
27 +
export fn split(kernel: *mut state::State, caller: u32, source: abi::Handle, leftCount: u64) -> abi::Handle throws (abi::Error) {
28 +
    let entry = try capabilities::lookup(kernel, caller, source);
29 +
    if entry.object.kind <> abi::Kind::Page { throw abi::Error::BadHandle; }
30 +
    let objectSlot = entry.object.index;
31 +
    let object = &kernel.resources[objectSlot];
32 +
    let case resources::Value::Page(page) = object.value else { panic "split: page kind mismatch"; };
33 +
    if object.references <> 1 { throw abi::Error::Busy; }
34 +
    if leftCount == 0 or leftCount >= page.run.count as u64 { throw abi::Error::InvalidArg; }
35 +
    let slot = try handles::vacant(&kernel.domains[caller].handles) catch { throw abi::Error::Busy; };
36 +
    let rightSlot = try resources::vacant(kernel.resources) catch { throw abi::Error::Busy; };
37 +
    let left = leftCount as u32;
38 +
    let remainder = resources::Page {
39 +
        run: frames::Run { first: page.run.first + left, count: page.run.count - left },
40 +
        origin: page.origin,
41 +
    };
42 +
    let right = resources::install(kernel.resources, rightSlot, resources::Value::Page(remainder));
43 +
    set kernel.resources[objectSlot].value = resources::Value::Page(resources::Page {
44 +
        run: frames::Run { first: page.run.first, count: left }, origin: page.origin,
45 +
    });
46 +
    resources::retain(kernel.resources, right);
47 +
    return handles::install(&mut kernel.domains[caller].handles, slot, right, entry.rights);
48 +
}
49 +
50 +
/// Query Page metadata after a live kind and rights check.
51 +
export fn access(kernel: *state::State, caller: u32, handle: abi::Handle, rights: u16) -> fdt::Range throws (abi::Error) {
52 +
    let entry = try capabilities::lookup(kernel, caller, handle);
53 +
    if entry.object.kind <> abi::Kind::Page { throw abi::Error::BadHandle; }
54 +
    if entry.rights & rights <> rights { throw abi::Error::Denied; }
55 +
    let case resources::Value::Page(page) = kernel.resources[entry.object.index].value else { panic "access: page kind mismatch"; };
56 +
    return fdt::Range {
57 +
        base: page.run.first as u64 * frames::PAGE_SIZE as u64,
58 +
        size: page.run.count as u64 * frames::PAGE_SIZE as u64,
59 +
    };
60 +
}
61 +
62 +
/// Check that live Page grants cover an entire byte range with the given rights.
63 +
/// Adjacent granted ranges can contribute to the same checked interval.
64 +
export fn covers(kernel: *state::State, domain: u32, base: u64, size: u64, rights: u16) -> bool {
65 +
    if size == 0 { return true; }
66 +
    if base > 0xffffffffffffffff - size { return false; }
67 +
    let end = base + size;
68 +
    let mut cursor = base;
69 +
    while cursor < end {
70 +
        let mut covered = cursor;
71 +
        for i in 0..abi::MAX_HANDLES {
72 +
            let entry = &kernel.domains[domain].handles.entries[i];
73 +
            if entry.object.kind <> abi::Kind::Page or entry.rights & rights <> rights
74 +
                or not resources::live(kernel.resources, kernel.domains, entry.object) { continue; }
75 +
            let case resources::Value::Page(page) = kernel.resources[entry.object.index].value else { panic "covers: page kind mismatch"; };
76 +
            let start = page.run.first as u64 * frames::PAGE_SIZE as u64;
77 +
            let limit = start + page.run.count as u64 * frames::PAGE_SIZE as u64;
78 +
            if start <= cursor and limit > covered { set covered = limit; }
79 +
        }
80 +
        if covered == cursor { return false; }
81 +
        set cursor = covered;
82 +
    }
83 +
    return true;
84 +
}
kernel/core/resources.rad +41 -12
1 1
//! Physical object identities shared by capability entries.
2 2
3 3
use core::abi;
4 4
use core::domains;
5 5
use core::fdt;
6 +
use core::frames;
7 +
8 +
/// One contiguous physical Page object.
9 +
export record Page: Copy {
10 +
    /// Allocated physical frames.
11 +
    run: frames::Run,
12 +
    /// Allocation origin identity, independent of handle holders.
13 +
    origin: abi::Object,
14 +
}
6 15
7 16
/// An exclusive external interrupt source.
8 17
export record Interrupt: Copy {
9 18
    /// PLIC source number.
10 19
    number: u32,
14 23
15 24
/// Resource data that determines the authoritative object kind.
16 25
export union Value: Copy {
17 26
    /// Unoccupied object slot.
18 27
    Free,
28 +
    /// Contiguous RAM with persistent domain-lifetime grants.
29 +
    Page(Page),
19 30
    /// User-visible MMIO register range.
20 31
    Device(fdt::Range),
21 32
    /// Exclusively held external interrupt source.
22 33
    Interrupt(Interrupt),
23 34
}
34 45
35 46
/// Get the kind associated with a resource value.
36 47
export fn kind(value: Value) -> abi::Kind {
37 48
    match value {
38 49
        case Value::Free => return abi::Kind::Empty,
50 +
        case Value::Page(_) => return abi::Kind::Page,
39 51
        case Value::Device(_) => return abi::Kind::Device,
40 52
        case Value::Interrupt(_) => return abi::Kind::Interrupt,
41 53
    }
42 54
}
43 55
46 58
    for i in 0..slots.len {
47 59
        set slots[i] = Slot { value: Value::Free, epoch: 0, references: 0 };
48 60
    }
49 61
}
50 62
51 -
/// Install a physical resource before its first capability is published.
52 -
export fn create(slots: *mut [Slot], value: Value) -> abi::Object throws (abi::Error) {
53 -
    assert kind(value) <> abi::Kind::Empty;
63 +
/// Find a reusable resource slot before a fallible allocation begins.
64 +
export fn vacant(slots: *[Slot]) -> u32 throws (abi::Error) {
54 65
    for i in 0..slots.len {
55 -
        let slot = &mut slots[i];
56 -
        if kind(slot.value) == abi::Kind::Empty and slot.epoch < 0xffffffff {
57 -
            set slot.epoch += 1;
58 -
            set slot.value = value;
59 -
            set slot.references = 0;
60 -
            return abi::Object { kind: kind(value), index: i, epoch: slot.epoch };
66 +
        if kind(slots[i].value) == abi::Kind::Empty and slots[i].epoch < 0xffffffff {
67 +
            return i;
61 68
        }
62 69
    }
63 70
    throw abi::Error::Exhausted;
64 71
}
65 72
73 +
/// Install a resource into a preflighted empty slot.
74 +
export fn install(slots: *mut [Slot], index: u32, value: Value) -> abi::Object {
75 +
    assert kind(value) <> abi::Kind::Empty;
76 +
    let slot = &mut slots[index];
77 +
    assert kind(slot.value) == abi::Kind::Empty and slot.epoch < 0xffffffff;
78 +
    set slot.epoch += 1;
79 +
    set slot.value = value;
80 +
    set slot.references = 0;
81 +
    return abi::Object { kind: kind(value), index, epoch: slot.epoch };
82 +
}
83 +
84 +
/// Install a physical resource before its first capability is published.
85 +
export fn create(slots: *mut [Slot], value: Value) -> abi::Object throws (abi::Error) {
86 +
    let index = try vacant(slots);
87 +
    return install(slots, index, value);
88 +
}
89 +
66 90
/// Check a physical-resource incarnation or a live domain-owned object.
67 91
export fn live(slots: *[Slot], domainStore: *[domains::Domain], object: abi::Object) -> bool {
68 92
    if object.kind == abi::Kind::Domain { return domains::live(domainStore, object); }
69 93
    if object.kind == abi::Kind::Events {
70 94
        return domains::live(domainStore, abi::Object {
83 107
    assert slot.epoch == object.epoch and kind(slot.value) == object.kind;
84 108
    assert slot.references < abi::MAX_DOMAINS * abi::MAX_HANDLES;
85 109
    set slot.references += 1;
86 110
}
87 111
88 -
/// Account for one removed physical-resource handle.
89 -
export fn release(slots: *mut [Slot], object: abi::Object) {
90 -
    if object.kind == abi::Kind::Domain or object.kind == abi::Kind::Events { return; }
112 +
/// Remove one resource reference and return any released Page's frames.
113 +
export fn release(slots: *mut [Slot], object: abi::Object) -> ?frames::Run {
114 +
    if object.kind == abi::Kind::Domain or object.kind == abi::Kind::Events { return nil; }
91 115
    let slot = &mut slots[object.index];
92 116
    assert slot.epoch == object.epoch and kind(slot.value) == object.kind and slot.references > 0;
93 117
    set slot.references -= 1;
118 +
    if let case Value::Page(page) = slot.value; slot.references == 0 {
119 +
        set slot.value = Value::Free;
120 +
        return page.run;
121 +
    }
94 122
    if let case Value::Interrupt(irq) = &mut slot.value {
95 123
        assert slot.references == 0;
96 124
        set irq.target = domains::none();
97 125
    }
126 +
    return nil;
98 127
}
99 128
100 129
/// Change the exclusive receiver before an Interrupt transfer becomes visible.
101 130
export fn retarget(slots: *mut [Slot], object: abi::Object, target: abi::Object) {
102 131
    if object.kind <> abi::Kind::Interrupt { return; }
kernel/core/state.rad added +15 -0
1 +
//! Bounded storage shared by kernel control-plane mechanisms.
2 +
3 +
use core::domains;
4 +
use core::memory;
5 +
use core::resources;
6 +
7 +
/// Kernel-owned state accessed while the machine critical section is held.
8 +
export record State: Copy {
9 +
    /// Protection-domain slots and their capability tables.
10 +
    domains: *mut [domains::Domain],
11 +
    /// Physical resource identities and live handle counts.
12 +
    resources: *mut [resources::Slot],
13 +
    /// Physical allocations and persistent domain-lifetime pins.
14 +
    memory: memory::Memory,
15 +
}
kernel/main.rad +17 -3
1 1
//! Freestanding machine initialization.
2 2
3 -
3 +
use core::fdt;
4 +
use core::frames;
5 +
use core::platform;
4 6
/// Maximum online harts in this kernel build.
5 7
constant MAX_HARTS: u64 = 8;
6 8
7 -
/// Validate the entry hart before machine initialization.
8 -
@default fn main(hart: u64) -> u32 {
9 +
/// Decode trusted firmware memory before any resource can reach user space.
10 +
/// Firmware supplies a mapped FDT header; subsequent lengths are validated.
11 +
@default unsafe fn main(hart: u64, description: *u8) -> u32 {
9 12
    assert hart < MAX_HARTS;
13 +
    let header = @sliceOf(description, 40);
14 +
    let size = try! fdt::word(header, 4);
15 +
    assert size >= 40 and size <= fdt::MAX_BYTES;
16 +
    let bytes = @sliceOf(description, size);
17 +
    let mut tree: fdt::Tree = undefined;
18 +
    try! fdt::decode(bytes, &mut tree);
19 +
    let mut machine: platform::Platform = undefined;
20 +
    try! platform::discover(&tree, &mut machine);
21 +
    let mut pool: frames::Pool = undefined;
22 +
    try! frames::init(&mut pool, &machine);
23 +
    assert pool.available > 0;
10 24
    return 0;
11 25
}