Initial broker design and implementation by wdcui · Pull Request #880 · microsoft/litebox

wdcui · 2026-05-29T19:00:28Z

This PR implements the broker based on a new design and adds the end-to-end implementation for the broker to provide the support for non-blocking evenfd.

jaybosamiya-ms

Bunch of comments all throughout the place about lower-level details. As discussed IRL, the plan is to merge this almost as-is (modulo removing the docs/ dir); a lot of the other comments are mostly to be read right now, but not really change the PR; future PRs will be doing the necessary cleanup. As a 1st order approximation, this current PR is directionally correct, but the agents have left a bunch of code smells that we do need to clean up to keep the project healthy; that's what most of the suggestions are focusing on.

On major bit (other than the docs/) that we should try to remove from this PR before merging, if possible, is the readv/writev changes that are mostly irrelevant to this PR at least as I understand it.

If one of my comments mentions just adding a tiny TODO or such trivially tweaked thing, might be good to just resolve that up front rn; but anything more significant than that probably should go into its own PR.

I'm making a tracking issue for the changes needed btw (#911)

jaybosamiya-ms · 2026-06-11T00:45:31Z

+pub use channel::{
+    HostControlChannel, LocalControlChannel, PeerCredential, ReceivedBrokerRequest,
+    ReceivedBrokerResponse,
+};
+pub use error::ErrorCode;
+pub use event::{
+    AddEventRequest, AddEventResponse, ConsumeEventRequest, ConsumeEventResponse,
+    CreateEventRequest, CreateEventResponse, EventConsumeMode, EventConsumption, ReadinessState,
+    WaitEventRequest, WaitEventResponse, WaitOutcome,
+};
+pub use message::{
+    BrokerRequest, BrokerResponse, CoreRequest, CoreResponse, EventRequest, EventResponse,
+};
+pub use object::{ObjectHandle, ObjectReferenceGeneration, ObjectReferenceId};


A large number of pub use make it difficult to follow things along, basically a bunch of modularity gets lost if everything is re-exposed publicly

jaybosamiya-ms · 2026-06-11T00:49:41Z

+/// Broker request sent over the control channel.
+///
+/// The outer broker request is intentionally small. Object-family and
+/// domain-specific operations are grouped below it so new object families do not
+/// accumulate as unrelated top-level broker variants.
+#[derive(Clone, Debug, PartialEq, Eq)]
+#[non_exhaustive]
+pub enum BrokerRequest {
+    /// Protocol negotiation request.
+    Negotiate {
+        /// Required protocol version.
+        protocol_version: ProtocolVersion,
+    },
+    /// BrokerCore authority request.
+    Core(CoreRequest),
+}
+
+/// Request adapted by the broker host into a BrokerCore domain call.
+#[derive(Clone, Debug, PartialEq, Eq)]
+#[non_exhaustive]
+pub enum CoreRequest {
+    /// Event object request family.
+    Event(EventRequest),
+}
+
+/// Broker-owned event object request.
+#[derive(Clone, Debug, PartialEq, Eq)]
+#[non_exhaustive]
+pub enum EventRequest {
+    /// Create a broker-owned event object.
+    Create(CreateEventRequest),
+    /// Check whether an event wait would complete now.
+    Wait(WaitEventRequest),
+    /// Add readiness credits to an event.
+    Add(AddEventRequest),
+    /// Consume readiness credits from an event.
+    Consume(ConsumeEventRequest),
+}


I'm not immediately seeing the benefits of this 3 layer indirection

jaybosamiya-ms · 2026-06-11T00:51:26Z

+use crate::{
+    AddEventRequest, AddEventResponse, ConsumeEventRequest, ConsumeEventResponse,
+    CreateEventRequest, CreateEventResponse, ErrorCode, WaitEventRequest, WaitEventResponse,
+};


this is the sort of stuff that becomes unreadable / unfollowable when the crate top becomes pub use everything

agents really prefer making everything pub and everything just into a big top level thing, we need to steer them away from that :)

jaybosamiya-ms · 2026-06-12T21:58:44Z

+            // Broker control-channel I/O runs through a host Unix socket in the
+            // current POC. The transport refreshes read/write timeouts around
+            // each request.


This seems like an unnecessary weakening of our seccomp filters. I'm not quite sure why read/write timeouts are needed if the broker is healthy

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Introduce modular broker protocol, transport, wire codec, client, core, server, and Unix socket implementation for the initial broker-owned event object path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Track effective protocol negotiation in client and server state, expose version mismatch responses for downgrade retries, and make public broker interface enums forward-compatible. Map future core outcomes to an Internal protocol error and keep wire tag details contained in the wire layer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Expose broker handles as reference capabilities containing only reference ID and reference generation. Keep object IDs internal to BrokerCore and remove ObjectGeneration from the protocol, wire codec, and server adapters. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Exclude litebox_broker_unix_socket from the confirm_no_std sweep because it is the concrete hosted Unix-domain-socket transport and broker executable for the split-broker POC. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rename the broker transport boundary to a channel boundary and tighten the protocol/channel naming so future IPC implementations can share the same semantic broker contract. Document the protocol, channel, wire, and core separation, including the future notification lane and kernel broker relationship. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Keep server, core, and Unix socket tests focused on distinct interface and security behavior while removing redundant private implementation coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Separate the Unix socket channel adapter from the hosted userland broker deployment, replace the redundant broker connection wrapper with associations, and align broker interface naming. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Restore shim-local nonblocking eventfd behavior when no broker control channel is configured, while still using broker-backed counters when available. Update the ratchet and missing test C-file copyright header. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove redundant timeout and deadline transport tests while keeping framing coverage and a single wall-clock partial-frame timeout case. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Simplify the userland broker entry point, move core-request envelope handling into broker-local/LiteBox layers, and keep the Linux runner focused on endpoint setup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Cover poll readiness transitions and nonblocking counter saturation in the broker-backed eventfd runner fixture. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Count event requests observed by the test broker so the broker-backed eventfd integration test proves eventfd operations used the broker. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Enable broker-backed event counters to support blocking waits so eventfd status flags can clear nonblocking mode. Special-case pipe and socket readv to perform one backend read, and coalesce small pipe writev calls to preserve PIPE_BUF atomicity. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Keep the spawned eventfd writer handle and join it after epoll_wait so Windows test processes do not remain alive after the test body reports success. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The docs now live on wdcui/ulitebox/broker-design-docs. Drop them from the implementation branch so the PR contains only code changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Keep eventfd-specific readv/writev handling while routing pipes and sockets through the existing generic fallback paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Use plain read/write in the eventfd fixture and remove eventfd-specific readv/writev shim handling that is not needed for the broker POC. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-06-17T20:37:55Z

🤖 SemverChecks 🤖 ⚠️ Potential breaking API changes detected ⚠️

Click for details

--- failure constructible_struct_adds_field: externally-constructible struct adds field ---

Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field CliArgs.broker_socket in /home/runner/work/litebox/litebox/litebox_runner_linux_userland/src/lib.rs:90

wdcui added discussion Open questions must-not-merge:prototype An experimental/proof-of-concept PR that must not be merged. expmt:ulitebox uLiteBox experiment labels May 29, 2026

wdcui force-pushed the wdcui/ulitebox/broker-design branch 2 times, most recently from 8d6f5d2 to bc2c10e Compare June 4, 2026 17:25

wdcui changed the title ~~broker design/poc~~ Initial broker design and implementation Jun 5, 2026

wdcui force-pushed the wdcui/ulitebox/broker-design branch from c889ab4 to 56fe76c Compare June 5, 2026 23:24

wdcui marked this pull request as ready for review June 5, 2026 23:24

wdcui requested a review from jaybosamiya-ms June 5, 2026 23:25

wdcui removed discussion Open questions must-not-merge:prototype An experimental/proof-of-concept PR that must not be merged. labels Jun 8, 2026

jaybosamiya-ms reviewed Jun 12, 2026

View reviewed changes

jaybosamiya-ms mentioned this pull request Jun 12, 2026

Tracking issue for cleaning up ulitebox broker design #911

Open

26 tasks

wdcui force-pushed the wdcui/ulitebox/broker-design branch 2 times, most recently from 154dda9 to 0f9601c Compare June 17, 2026 20:25

wdcui and others added 15 commits June 17, 2026 13:27

broker design v0.1

6b1b897

updated

6259a2b

Document broker split design

b45215f

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Document broker implementation plan

605c1c1

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Incorporate sandbox architecture findings

4c7a391

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add split broker event POC

76cd13f

Introduce modular broker protocol, transport, wire codec, client, core, server, and Unix socket implementation for the initial broker-owned event object path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

fix cargo lock

ec6163c

Decouple broker core from protocol layers

551ca42

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Hide foreign broker object references

c90b69e

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Allow Unix broker transport in no_std CI

f0fb824

Exclude litebox_broker_unix_socket from the confirm_no_std sweep because it is the concrete hosted Unix-domain-socket transport and broker executable for the split-broker POC. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Prune broker unit tests

5965b0f

Keep server, core, and Unix socket tests focused on distinct interface and security behavior while removing redundant private implementation coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

wdcui and others added 22 commits June 17, 2026 13:27

Simplify broker identity model

aacb196

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Update eventfd fcntl test expectation

02f7d70

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Restore eventfd fcntl test coverage

5d1cdf7

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Avoid extra broker core test global

d702489

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Prune redundant broker tests

6f310fe

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rename broker local and host errors

6a520d1

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove broker local worker

4bc3077

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Align broker local and host naming

83117fc

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Fold broker negotiation modules into crate roots

45732e1

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Prune broker local and host tests

24529d7

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Centralize initial broker protocol version

d5b0f49

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Organize broker wire codec by protocol layer

2620a48

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Prune Unix socket transport tests

4063b18

Remove redundant timeout and deadline transport tests while keeping framing coverage and a single wall-clock partial-frame timeout case. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Clean up broker local integration layering

84e8588

Simplify the userland broker entry point, move core-request envelope handling into broker-local/LiteBox layers, and keep the Linux runner focused on endpoint setup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Expand broker eventfd fixture coverage

928acfa

Cover poll readiness transitions and nonblocking counter saturation in the broker-backed eventfd runner fixture. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Assert broker eventfd traffic in runner test

fe92183

Count event requests observed by the test broker so the broker-backed eventfd integration test proves eventfd operations used the broker. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Join epoll eventfd test writer thread

f0d9e07

Keep the spawned eventfd writer handle and join it after epoll_wait so Windows test processes do not remain alive after the test body reports success. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove broker design docs from implementation PR

5dcc8fd

The docs now live on wdcui/ulitebox/broker-design-docs. Drop them from the implementation branch so the PR contains only code changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Back out non-eventfd vectored IO changes

ea52efe

Keep eventfd-specific readv/writev handling while routing pipes and sockets through the existing generic fallback paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Simplify eventfd fixture IO paths

e1d7407

Use plain read/write in the eventfd fixture and remove eventfd-specific readv/writev shim handling that is not needed for the broker POC. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

wdcui force-pushed the wdcui/ulitebox/broker-design branch from 0f9601c to e1d7407 Compare June 17, 2026 20:27

wdcui enabled auto-merge June 17, 2026 20:28

wdcui added this pull request to the merge queue Jun 17, 2026

Merged via the queue into ulitebox with commit e91d236 Jun 17, 2026
7 checks passed

wdcui deleted the wdcui/ulitebox/broker-design branch June 17, 2026 20:48

wdcui mentioned this pull request Jun 22, 2026

Simplify broker protocol version handling #944

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial broker design and implementation#880

Initial broker design and implementation#880
wdcui merged 66 commits into
uliteboxfrom
wdcui/ulitebox/broker-design

wdcui commented May 29, 2026 •

edited

Loading

Uh oh!

jaybosamiya-ms left a comment •

edited

Loading

Uh oh!

jaybosamiya-ms Jun 11, 2026

Uh oh!

Uh oh!

Uh oh!

jaybosamiya-ms Jun 11, 2026

Uh oh!

jaybosamiya-ms Jun 11, 2026

Uh oh!

jaybosamiya-ms Jun 12, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

wdcui commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jaybosamiya-ms left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jaybosamiya-ms Jun 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jaybosamiya-ms Jun 11, 2026

Choose a reason for hiding this comment

Uh oh!

jaybosamiya-ms Jun 11, 2026

Choose a reason for hiding this comment

Uh oh!

jaybosamiya-ms Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

wdcui commented May 29, 2026 •

edited

Loading

jaybosamiya-ms left a comment •

edited

Loading