Design Documents¶
This section preserves the full design rationale behind go-codex's foundational,
cross-cutting architectural patterns — decisions that shaped how MULTIPLE apis/ports
work, not a single feature. Unlike docs/roadmap/, everything
here IS implemented and shipped; unlike docs/concepts/, these
are not usage guides — they keep the ORIGINAL reasoning, rejected alternatives, and the
review history that led to the shipped design, for future maintainers extending the same
pattern to a new boundary.
A document graduates here from docs/roadmap/ — and only from there — when it meets
either bar:
- It is fully shipped AND establishes a pattern multiple apis/ports/packages are expected to follow (not a single-adapter feature); or
- The design fundamentally changes how an existing api/port/package works.
This is a deliberately high bar, reserved for bigger architecture designs and reworks
worth keeping in full. Most shipped roadmap docs still just follow docs/roadmap/'s own
existing lifecycle (removed once shipped, or kept in place if a follow-on phase remains
open) — see .github/skills/plan-a-new-codex-feature/SKILL.md for the exact policy.
Numbering convention¶
Every document here is filed as d-NNNN-<slug>.md — a sequential, zero-padded number
assigned in the CHRONOLOGICAL order design docs were WRITTEN (not necessarily the order
they shipped), mirroring an ADR-style numbering scheme. d-0001 is REST's own
middleware/workflow simplification — the FIRST of these two designs, and the one the
second one adopted and adapted concepts from. d-0002 is pub/sub's — written second,
after REST's d-0001 had already landed, deliberately reusing and adapting as many of
its concepts as pub/sub's own structural differences allow (role model, security
merge/coverage, Client.Attach), with two-way sync back onto d-0001 itself whenever a
gap surfaced in ONE that the other had already solved (see either doc's own addenda for
the concrete back-and-forth). When promoting a NEW roadmap doc here, assign it the next
sequential number — do not renumber existing documents.
Documents¶
| Document | Package | Summary |
|---|---|---|
| D-0001 — REST Middleware Workflow Simplification | middleware, api/rest, adapters/nethttp, adapters/chi |
The declare/implement middleware split (HandleMW/ClientMW) plus whole-API declarative wiring (Route.Register(builder) + Serve/ServeOne/ServeSSE as the sole server-side entry points, Call/CallWithHandle as the sole client-side ones) — fully shipped, including removal of the older per-route Handler/Register/SSEHandler/RegisterSSE functions. Establishes the pattern the ports.Pattern binding layer (RESTPattern) reuses unchanged, and the one d-0002 (pub/sub) adopted and adapted next. |
| D-0002 — Pub/Sub Workflow Simplification | api/events, ports, adapters/mqtt5, adapters/mqtt, adapters/zeromq |
Pub/sub's client-centric role model — no fixed server/client pairing, since a broker is the intermediary and both publisher and subscriber are CLIENTS of a channel — resolved via events.Client + role-scoped Subscriber[T]/Publisher[T] builders (WithSubscribe/WithPublish, .Use/.SubscribeMW/.PublishMW, .Handle(client)/.Register(client)), unconditional security-coverage enforcement (CheckCoverage/checkImplementationsDeclared), and a reflection-based Client.Attach/.Publish/.Subscribe/.ServeSubscribers convenience layer unified with d-0001's own Server.Attach/rest.Client/nethttp.Attach design. Format resolution (JSON/YAML/TOML/Gob/custom binary) is centralized on ChannelHandle/RouteHandle themselves (EncodeWithFormats/DecodeMergedWithFormats), so every adapter — escape-hatch primitive AND Client.Attach shim alike — is a thin caller of one canonical method. Fully shipped across all three pub/sub adapters plus REST/nethttp/chi, with every old, call-time-competing public primitive removed (confirmed exceptions kept for genuine advanced needs). Establishes the pattern api/reqreply's own rework (see D-0004, now shipped) follows. |
| D-0004 — ReqReply Workflow Simplification | api/reqreply, ports, adapters/mqtt5, adapters/zeromq |
Extends d-0001/d-0002's Server/Client+Attach pattern to request-reply's third API boundary: reqreply.Server UNIFIES what a separate, now-deprecated Builder type did (AsyncAPI 3.0 spec accumulation, AddGlobalSecurity) with dispatch/transport — ONE type, mirroring rest.Server. Server-side declaration is ONE fluent chain, route.WithHandler(fn).Register(server), matching rest.Route's dominant idiom byte-for-byte. Client.Call accepts EITHER a raw, unregistered Route (REST-style, GlobalSecurity invisible — same accepted limitation as rest.Route.ClientHandle()) OR an already-registered *RouteHandle (GlobalSecurity enforced) — dispatched via a type-switch, a real choice REST itself cannot offer. An ADDITIVE, reqreply-only async Client.CallAsync/Future[Resp] exists alongside the blocking Call, needed because reqreply's underlying transport (MQTT5/ZeroMQ correlation-based reply matching) is genuinely asynchronous, unlike REST's/pub-sub's dispatch — confirmed via a FutureFactory interface letting an adapter recover a correctly-typed future via a plain interface type assertion, zero reflect-based generic instantiation needed. adapters/mqtt5's AttachServer/AttachClient and adapters/zeromq's AttachServer/AttachClient/AttachRouterServer/AttachDealerClient (covering REQ/REP AND ROUTER/DEALER, the latter pair with a new MissingSocketError validating full topic/socket coverage upfront at Attach time) implement the reflection-shim idiom adapters/mqtt5's own events.Transport implementation already established. Builder/NewBuilder/BuilderOption stay DELIBERATELY KEPT as deprecated aliases (zero-cost, no duplicate logic). mqtt5's AND zeromq's lower-level Serve/Call/CallHandle/ServeRouter/CallDealer — REOPENED, decision REVERSED, then RESHIPPED via delegation, KEPT unchanged: originally kept as documented escape hatches by analogy to REST's ServeOne/CallWithHandle; that analogy was found flawed during a later review (recorded in this doc's own "Addendum: reqreply-middleware.md and zeromq-security.md" section — both source roadmap docs have since shipped and been deleted) — these functions shared no code with AttachServer/AttachClient/AttachRouterServer/AttachDealerClient, a genuine duplicate dispatch implementation — but rather than deleting them (the initial plan), Phase 0b found reqreply.ServerTransport.Serve/ClientTransport.Call are already single-route/single-call scoped, so every one of these functions' BODIES were rewritten to delegate directly to the matching serverTransport/clientTransport/routerServerTransport/dealerClientTransport — zero duplicate logic, EXACT SAME signatures, no breaking change, no caller migration, for BOTH adapters. Fully shipped across api/reqreply core types, adapters/mqtt5, adapters/zeromq, and the examples/reqreply-api mini-project (7 demos spanning mqtt5 AND zeromq REQ/REP + ROUTER/DEALER, replacing 3 deleted/rebuilt examples). |
| D-0003 — Codec-Declared Middlewares | middleware, api/rest, api/events, api/reqreply, adapters/nethttp, adapters/chi, adapters/mqtt, adapters/mqtt5, adapters/zeromq |
A codec-backed middleware.Declaration[In,Out] core (Input/Output shape, like a route/channel's own Req/Resp) + per-pattern derived types rest.Middleware[In,Out]/events.Middleware[In,Out], additive via a middleware.RouteMiddleware marker interface — zero breaking changes to middleware.Middleware/SecurityScheme/existing .Use(...) call sites. Two attachment styles, one shared vocabulary across Route, SSERoute, and Subscriber/Publisher: Transform/ClientTransform (route/channel-BOUND, fn gets req/msg access plus a codec-validated In) and plain .Use(mw) (route/channel-AGNOSTIC, via WithReceive/WithSend-bundled Fns, reused verbatim across many routes/channels). Establishes the guiding principle that middleware is fundamentally a spec-layering mechanism (a declare-time value that can layer into the final spec and/or carry runtime dispatch behavior; "non-spec-adding" middleware is simply the case where the spec column is empty) — governing REST/events middleware today and a confirmed-feasible future ports.Middleware[In,Out] extension. Resolved 7 design decisions (D1-D7): pre-handler-only timing (D1); fn errors are ErrorPattern/ErrorChannel-eligible, falling back to a new MiddlewareError (D2); 3-tier client precedence, explicit > middleware-derived > route/channel-derived (D3); a mandatory spec-layering conflict-check guard covering multiple Middleware[In,Out] contributions (D4); stats.ReportErrors integration at "middleware:in"/"middleware:fn" (D5); Declaration.Name uniqueness enforcement, a new DuplicateMiddlewareNameError (D6); rejection of combining both attachment styles on one value, a new AmbiguousMiddlewareAttachmentError (D7). Fully shipped across REST (Route + SSERoute, server AND client dispatch), events (subscribe AND publish, all three pub/sub adapters), and reqreply (property vocabulary axis, Addendum). Supersedes the now-deleted docs/roadmap/common-middleware-architecture.md (its finding fully absorbed here); complementary to (not competing with) docs/design/d-0006-protocol-native-capabilities.md. Addendum 3 (absorbing the now-deleted docs/roadmap/middleware-consolidation.md): the legacy middleware.Middleware type is NO LONGER unchanged — its former RequestParams/ResponseParams fields were deleted (fully replaced by this design's own presence-only WithRequestHeaderSpec-family methods), while its Security half survives PERMANENTLY, shrunk to {Name, Security} — a genuinely cross-API-shared Security declaration cannot be built from a per-pattern codec-backed type (each pattern's internal dispatch only recognizes its own concrete type; a foreign value would compile but be silently dropped). Addendum 4 (api/rest only, docs/design/d-0007-declarative-middleware-layering.md's Rollout Phase A): Transform/ClientTransform/TransformSSE/ClientTransformSSE REMOVED — folded into HandleMW/ClientMW via reflected-fn-shape detection; rest.SecurityMiddleware[In,Out] generalized over In/Out; Security dispatches through the SAME mechanism as ordinary middleware via the new GrantedScopes conventional-field convention, retiring adapters/nethttp/adapters/chi's separate runSecurityMiddleware code path; codex.EncodeMergeVars + the full omit-empty merge-field constructor family added; middleware.ContextFieldSetter fixes a Phase-3 sketch that didn't compile. Addendum 5 (api/events only, Rollout Phase B): events.Transform/events.ClientTransform REMOVED — folded into Subscriber.SubscribeMW/Publisher.PublishMW via reflected-fn-shape detection (events' own legacy-vs-bound ambiguity needed a 3rd-param check, not just REST's 2nd-param one); events.SecurityMiddleware[In,Out] generalized over In/Out with the zero-value-codec fix applied from the start; events.CheckCoverage gained a handlers parameter; Middleware[In,Out].SetContextFieldFromIn/SetContextFieldFromOut added (Publish-only for the latter); Client.AddConnectSecurityScheme closes the connection-level-auth spec-registration gap; adapters/mqtt5.ConnectError gained ReasonCode/ReasonString, ConnectOptions gained Observer. Addendum 6 (api/reqreply only, Rollout Phase C — the FINAL of the 3 planned phases): reqreply.Transform/reqreply.ClientTransform REMOVED — folded into Route.HandleMW/Route.ClientMW via reflected-fn-shape detection; reqreply.SecurityMiddleware[In,Out] generalized over In/Out with the zero-value-codec fix applied from the start; reqreply.CheckCoverage gained a handlers parameter; Middleware[In,Out].SetContextFieldFromIn/SetContextFieldFromOut added (fully duplex, unlike events' Publish-only asymmetry); Server.AddConnectSecurityScheme mirrors events' identical method; GrantedScopes merge-and-enforce wired receiving (Serve)-side only via the shared adapters/internal/scopesmerge package (also retrofitted onto events' Subscribe side as a prerequisite patch); a confirmed, cross-package .Use()+bound-HandleMW duplicate-name gap (also affecting api/rest) is documented but not yet fixed. |
| D-0005 — Unified Error Handling: REST, Events, ReqReply | api/rest, api/events, api/reqreply, render/asyncapi/v3, stats |
A single, extensible error-response enforcement point (ErrorPattern/ErrorChannel) that BOTH declares typed error responses AND observes every error flowing through it, uniformly across every failure point — reducing handler-author observability effort while keeping adapters thin (one repeated dispatch pattern instead of N special-cased ones). Extends ErrorPattern/ErrorChannel eligibility from just handler + general-purpose-middleware errors to EVERY Category-A dispatch failure (security middleware Fn, decode/param-validation, MiddlewareInputError/MiddlewareOutputError, response/merge-field encode) across all 3 APIs — codex's own error taxonomy and ports/adapter infrastructure errors remain permanently excluded (structural boundaries, not gaps). Ratifies reqreply's same-reply-topic error-reply behavior (industry-standard gRPC/AMQP/NATS precedent) and migrates its AsyncAPI rendering from "separate channel per error type" to AsyncAPI 3.0's native channel-level messages map (Operation.Messages []Message) — the direct analogue of OpenAPI's per-status responses object. Adds a declarative dead-letter mechanism (events.DeadLetter/reqreply.DeadLetter), scoped to genuinely UNMATCHED failures only, complementary to (not replacing) ErrorChannel/ErrorPattern — global-default support mirroring GlobalSecurity, two-tier failure enumeration, failed publishes/replies also dead-letterable. Adds an optional stats.ErrorPatternObserver (RecordErrorPatternMatch/RecordErrorPatternMiss) and a separate stats.SpanTagger, consolidated (alongside DeadLetter's envelope-building) into two core-layer methods — ObserveErrorResponseFor/DeadLetterFor — so adapters make ONE call each instead of independently type-asserting observers at ~26+ call sites. Ships all 3 client-side matching ergonomics together — ErrorPatternAs[B], a .Match method on the declaration value, and a HandleErrorPattern/Case dispatch table — and states an explicit Breaking Changes Policy (single maintainer/user, breaking changes acceptable when they aid maintainability) used to reopen REST's previously-permanent same-status ErrorPattern exclusion (DuplicateErrorStatusError, matching reqreply's stricter DuplicateErrorPatternCodeError). Hardening rounds before graduation: closed a REST-specific gap where handlerFunc (the SEPARATE dispatch behind port/stream-adapter bindings — IngestAdapter/LatestAdapter/HandlerLatest/PipelineHandler — distinct from serve.go's already-wired serve) never consulted a declared ErrorPattern; closed an events-side gap where PublishAdapter's upstream-pipeline-error path skipped ErrorPatternObserver observability; and — per this codebase's "adapters implement wire protocols only" guardrail (see docs/concepts/ports-and-adapters.md) — relocated ErrorPatternAs/HandleErrorPattern/Case OUT of adapters/nethttp/adapters/mqtt5/adapters/zeromq (the reqreply pair was byte-for-byte duplicated code) into api/rest/api/reqreply, since none of that logic touches anything transport-specific. |
| D-0006 — Protocol-Native Capabilities | api/rest, api/events, api/reqreply, internal/middleware, stats, adapters/nethttp, adapters/chi, adapters/websocket, adapters/mqtt, adapters/mqtt5, adapters/zeromq |
The COMPLETE three-tier capability model across all 3 api packages: Tier 1 (Explicit) — a SEALED, per-adapter Capability interface for api/events/api/reqreply (MQTT QoS/Retained, ZeroMQ HWM/Conflate), declared via CapabilityRequirement/RequireQoS/etc. and checked automatically via CheckCapabilityCoverage (VALUE-aware via LeveledCapability); Tier 2 (Implicit) — api/rest's HeaderCapableTransport/CookieCapableTransport/QueryCapableTransport, real Extract-shaped methods on each adapter's per-request carrier (httpCarrier/wsCarrier); Tier 3a — the concrete adapter capability values themselves. Also covers the Attach-factory redesign (every adapter exposes New*Transport factories; all 12 adapter-namespaced Attach* convenience functions removed), ErrorResponseWriter (REST's interface-level ErrorPattern write step), and Handler Disposition (stats.Disposition, shared by events+reqreply). §9 merges in, wholesale, the FULL phase-by-phase implementation history of the capability-requirement-composition rework that built all of this on top of the original events-only mechanism. Resolves its relationship to D-0003 via a confirmed 4-stage lifecycle model — Middleware[In,Out] and Capability are both stage-2, spec-contributing declarations sharing a lifecycle stage, without merging into one Go type. |
| D-0007 — Declarative Middleware as Partial Route/Channel Definitions | middleware, api/rest, api/events, api/reqreply, adapters/mqtt5, adapters/mqtt, adapters/zeromq | States the general principle (confirmed via code, not aspirational) that a Middleware[In,Out] is a partial route/channel definition — its own merge-field-declared In/Out, stacked and aggregated with the route/channel's own declaration by the SAME mechanism in all 3 packages. Extracts a shared middleware.DecodeLayer/EncodeLayer mechanism (replacing 3 independently-duplicated per-package implementations); generalizes SecurityMiddleware[In,Out] in all 3 packages to dispatch its credential Fn through that shared mechanism via a conventional GrantedScopes Out field, merged (adapters/internal/scopesmerge) with any legacy Fn's own grants before ONE unified middleware.CheckScopes call, RECEIVING-side only (events' Subscribe, reqreply's Serve — never the sending side, confirmed via REST's own ClientMW, which never does this either). Extends middleware.ContextField[V] with SetContextFieldFromIn/SetContextFieldFromOut linking, so a middleware-derived value reaches the handler/caller automatically via context.Context, with zero manual re-extraction. Adds connection-level auth spec registration (Client.AddConnectSecurityScheme/Server.AddConnectSecurityScheme), closing a gap where a scheme referenced only via a Server.Security list (never by an individual route/channel) could not be spec-registered. Fully shipped in 3 rollout phases — Phase A (api/rest), Phase B (api/events), Phase C (api/reqreply) — each folding its own Transform/ClientTransform free functions into HandleMW/ClientMW/SubscribeMW/PublishMW directly via reflected-fn-shape detection (see D-0003's Addenda ⅘/6 for the compact per-phase shipped-changelog). Known, tracked, unfixed gap: .Use(mw).HandleMW(&mw, boundFn) for a Security-only mw throws DuplicateMiddlewareNameError in both api/rest and api/reqreply (not api/events) — a nontrivial design question (distinguishing a legitimate .Use()+bound combo from a genuine duplicate-name collision) deferred to a future session. Post-ship regression found and fixed (while adding real examples showcasing these capabilities): events.Client.Attach+ServeSubscribers — the PRIMARY workflow for all 3 pub/sub adapters — never dispatched bound SubscribeMW middleware or ContextField propagation at all; only the lower-level escape hatch (unit-tested, never used by Client.Attach) had the fix. Fixed across mqtt5/mqtt/zeromq; see this doc's own "Post-ship regression found and fixed" section for the full finding. |
| D-0008 — Declarative Router Groups | api/rest, api/events, api/reqreply | A THIRD, group-level construct (distinct from Middleware) modeled on chi's own Router/Mount/Group/With/Routes/Walk: declares a path/topic PREFIX once and groups independently-declared Route/Channel-derived leaves (Route[Req,Resp]/SSERoute[Req,Event] for REST; Subscriber[T]/Publisher[T] for events; Route[Req,Resp] again for reqreply, the simplest variant with no second axis) under it, optionally attaching reusable-class middleware.RouteMiddleware to every leaf in one declaration instead of repeating .Use(mw) per route/channel. Router is a fully IMMUTABLE VALUE type (matching Route/Channel/Subscriber/Publisher exactly) — a deliberate, mid-design reconsideration that eliminated an earlier mutex-plus-pendingRoute-type draft entirely. Supports nesting (Mount), same-path/topic scoped-middleware subsets (Group, keeping chi's own baseline semantics for reqreply, method-scoped for REST, role-scoped for events), one-shot ad hoc middleware (With, accumulating not overwriting), and a Routes()/Walk() pair for a holistic, pre-registration view of the fully-assembled tree. WithHandleCallback (free function, all 3 patterns; events gets a role-targeted 3-way split mirroring Formats/SubscribeFormats/PublishFormats) recovers a grouped leaf's typed handle; ClientHandle's new variadic WithRouter opt (REST/reqreply only — events has no bare zero-argument accessor needing it) avoids a silent client/server path mismatch when a leaf is grouped under a Router on the server side. Fully shipped in 3 rollout phases — Phase A (api/rest), Phase B (api/events), Phase C (api/reqreply) — followed by a dedicated deferred-item review (6 "out of scope" candidates re-examined post-implementation, including concretely rejected alternatives for each: variable prefixes stay deferred as a deliberate simplicity-over-capability tradeoff; api/mcp support spun off as a prefix-less ToolGroup sketch in mcp-ports-declarative-middleware.md; bound-class middleware at Router level confirmed permanently impossible for this heterogeneous Router shape, with the strongly-typed Router[Req,Resp] alternative spun off into its own idea-stage doc, typed-router-groups.md; WithRouterTags and a ports.Pattern integration both gained concrete design sketches — including found gotchas/blockers — for a future implementer to start from; a Router-scoped fallback handler confirmed redundant for REST/events and structurally impossible for reqreply, given each pattern's real dispatch architecture). |
| D-0009 — Internalizing Shared Cross-Pattern Mechanics | internal/router, internal/route, internal/middleware, api/rest, api/events, api/reqreply, stats | Relocated the public router/route/middleware packages under internal/, making "never imported by an end user of go-codex" a compiler-enforced fact (Go's internal/ import rule) instead of a documentation convention. Each pattern package gained its own thin public wrapper (type aliases + forwarding constructors) exposing the identical vocabulary — e.g. rest.BearerScheme/rest.SecurityMiddleware. Disposition/ResolveDisposition relocated to stats instead (not internal/middleware), since stats.DispositionObserver is a public, user-implementable extensibility interface. Migrated ~190 consumer files (~100 in-module, ~35 examples) with full build/vet/test/staticcheck/gosec/example verification at every phase. Establishes the standing design rule for future shared mechanics (codified in .github/instructions/go-codex.instructions.md, 2 skills, and review-go-codex's checklist §16). Addendum: SharedMiddleware/FromSecurityScheme (the legacy cross-pattern-shareable-value door shipped as part of this doc's original scope) were SUBSEQUENTLY REMOVED entirely in a later round — a from-scratch investigation found zero real callers needing a literal shared value across patterns; "shared config, declared twice per pattern" via each pattern's own SecurityMiddleware constructor is the sole surviving cross-pattern-sharing mechanism. Second Addendum: docs/roadmap/shared-api-layer-mechanics.md (the roadmap that originally phased the Router/Bound-Middleware/Error-Pattern consolidations feeding into this doc) was retired once all its phases resolved — its full phase-by-phase record (Bound Middleware's BoundCore-dropped correction and the real api/events spec-population bug found; the Declaration/Dispatch core phase's REJECTION, not deferral, once REST's and events' merge-field shapes were confirmed genuinely different-arity; Error-Pattern client helpers' consolidation) is folded into this doc's own second Addendum. |
How to read these documents¶
Each document here was originally a docs/roadmap/ design doc, refined through one or
more critical review passes before and during implementation. Expect:
- Motivation — the problem that drove the pattern, and why a narrower fix wasn't enough
- Rejected alternatives — approaches considered and why they were set aside
- API surface — the actual shipped type signatures, as-built
- Known limitations and open risks — a running punch list, resolved item by item during implementation (kept, not deleted, so the reasoning survives)
- Coverage — how the pattern extends (or is planned to extend) across every api/port boundary it applies to, not just the first one it shipped for