Are you an LLM? Read llms.txt for a summary of the docs, or llms-full.txt for the full context.
Skip to content

wshub — WebSocket aggregation

enc-wshub is a WebSocket aggregator hub for the ENC protocol. It collapses per-enclave client sockets into one client connection, multiplexed over shared, refcounted per-enclave upstream sockets to the node:

client ──(1 WS)──> HubDO ──(1 refcounted WS per enclave)──> enc-node DOs

Why

Cloudflare binds an accepted WebSocket to the single Durable Object that accepted it, and each enclave is its own DO. A client that watches its DM + Personal + N group enclaves needs N+2 sockets; a 100-member group pins 100 sockets to that enclave's DO. Connection count is O(clients × enclaves).

The ENC wire protocol was already built for multiplexing — auth is per-message (every Query carries its own session) and the node tags every frame with a sub_id. The hub reclaims that capacity:

  • One client socket carries unlimited subscriptions to any mix of enclaves.
  • The hub keeps exactly one upstream socket per enclave, shared by every client of this hub shard, refcounted open/closed.
  • Query frames are forwarded verbatim — {session, filter} stays opaque; the node verifies each client's session and enforces per-client RBAC. The hub is a metadata-only sub_id router, never an auth point.
  • sub_ids are translated client-local ↔ hub-global, so two clients can both use "s1" without colliding on the shared upstream.

Result: connection count drops to O(clients) + O(distinct live enclaves). This is the network-aggregation layer described in the litepaper.

Architecture

The hub is a Cloudflare Worker plus one Durable Object class, HubDO:

  • The Worker's fetch() routes WS upgrades to a hub shard (?shard=, default hub-0) via idFromName. Non-WS requests (Commit / Query / Pull / Info POSTs, GET /enclave/<id>/sth) are transparently proxied to the node, so clients can point a single base URL at the hub.
  • HubDO accepts the WS upgrade as hibernatable and assigns a unique connection id.
  • On each Query it writes a route gid = <connId>.<clientSubId>, ensures an upstream socket for that enclave (opening one if needed, bumping its refcount otherwise), and forwards the Query with sub_id := gid.
  • On every upstream frame it translates sub_id back to the client's local id and forwards. On Closed it drops the route and releases the upstream refcount; an upstream that hits refcount 0 is closed.
  • If an upstream socket dies, affected clients are notified with a per-sub Closed { reason: "upstream_…" } so they re-subscribe (and re-backfill). A client disconnect tears down all of that client's routes.

Wire contract

The hub is the boring side of the contract. Because one upstream WS now carries many subs from different clients, the node must accept the caller-supplied sub_id and echo it on every frame so the hub can demux. Required node behaviors:

  1. Query accepts a caller-supplied sub_id (used verbatim) and echoes it on Event / EOSE / Closed / Error frames.
  2. Close { sub_id } removes only that subscription, leaving the other subs on the same WS alive.
  3. DO hibernation preserves all subs on the WS, not just the last one — the full sub-list is restored on wake-up.
  4. Tags persist across the storage round-trip, so SQL backfill returns the same tags (["to"], ["enclave_id"], …) that strict-wire decryption needs — not just the live in-memory broadcast.

ENC's reference node implementation satisfies all four, with protocol regression guards in its test suite; a node that drops these behaviors will fail them.

Develop

# 1. spawn the mock node (default :18890)
yarn mock-node
 
# 2. spawn the hub (default :8787 via wrangler dev)
yarn dev
 
# 3. drive 100 clients through it
N=100 yarn bench
 
# pure-model numbers (no I/O)
yarn bench:sim
 
# type-check
yarn typecheck

Deploy

# point at the live node first
wrangler secret put ENC_NODE_URL   # or set it in wrangler.jsonc vars
yarn deploy

The Worker is exposed at enc-wshub.<account>.workers.dev (Cloudflare auto-subdomain). Clients then use wss://enc-wshub.<account>.workers.dev/?shard=<id> as their WS endpoint and the same hostname for HTTP — the hub transparently proxies non-WS traffic to ENC_NODE_URL.

Sharding

?shard=<name> picks the HubDO instance; the default hub-0 is a single shard. For horizontal scale, hash the client's pubkey (or any stable per-client key) and pass it as shard — each shard is its own DO with its own upstream sockets.

The number of distinct enclaves seen across all clients on a shard is that shard's upstream socket budget — roughly N_clients_on_shard × avg_enclaves_per_client / overlap_factor.

Client integration

A client integrates the hub behind the same subscribe() call it already uses; a single HubWS instance per node URL multiplexes every subscription over one socket. Each subscription gets a caller-supplied sub_id (derived from the enclave id plus a per-instance counter) that is unique within the HubWS instance, so the switch between per-enclave sockets and hub multiplexing is transparent to application code.

Heartbeats

Both directions ping every 25 s with a 10 s pong deadline:

  • Client → hub — keeps the client WS alive across Cloudflare edge idle timeouts; on a missed pong the client closes and reconnects.
  • Hub → upstream node — Cloudflare Workers silently idle-close an outbound WS after ~100 s without always firing a close event, so the hub force-closes a stale upstream on a missed pong, notifies routed clients with Closed { reason: "upstream_closed" }, and they re-subscribe (new upstream + backfill catches up missed events).

Plain 'ping' gets a plain 'pong' (no JSON envelope), and the hub forwards client pings to the upstream so a silent idle drop surfaces on either side.

Limits

  • HubDO hibernation. When all client WSes are idle, Cloudflare can hibernate the HubDO; the in-memory route/upstream maps are gone on wake-up and the first Query rebuilds them and reconnects a fresh upstream. Clients see a Closed { upstream_closed } on their next frame and re-subscribe.
  • Single-DO hot spot. A shard handles all its clients in one (single-threaded) DO. For high-traffic regions, shard horizontally via ?shard=.
  • No bandwidth metering. The hub doesn't enforce per-client throughput; abuse mitigation lives at the Cloudflare edge (rate limiting / WAF).