cloud/redapricot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

break: replace muxed workers with 1:1 tunnels

Browse Source 
Worker frames are now FrameType + payload; there is no stream id.
Each player gets its own worker conn. maxTunnels (default 256)
caps concurrent tunnels. The old maxConn pool size is ignored so
existing configs do not silently admit only a handful of players.

Resume, per-direction windows, the control session, and the
DATA-only shaper stay. A dropped worker still hangs that one
player and reattaches over a fresh conn.

Add a hub-side per-IP limiter for player intents only (default
8/s, burst 16, 64 concurrent). Unmatched hostnames consume a
token; Intent 17 is never counted. 0 disables each knob.
This commit is contained in:
iceBear67
2026-08-15 18:32:51 +08:00
parent da17140583
commit 4df2560331
27 changed files with 1174 additions and 856 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+74 -89
View File
@@ -15,9 +15,9 @@ on the wire, read [PROTOCOL.md](../PROTOCOL.md).
┌──────────┐ Intent 17, magic 0x01 │ │ │
│ Client │ ◀──────── control session ──────│ • pattern reg │ │
│ (Go) │ ────────────────────────────────│ • CID table │ │
│ │ Intent 17, magic 0x02 │ • mux demux │ │
│ │ Intent 17, magic 0x02 │ • 1:1 workers │ │
│ │ ═════════ worker conns ═════════│ │ │
└──────────┘ multiplexed player streams └───────────────┘ │
└──────────┘ one TCP conn per player └───────────────┘ │
│ │
▼ MC bytes (+ optional HAProxy v2) │
┌───────────────┐ │
@@ -68,15 +68,15 @@ Player Hub Client Destinatio
│ │ pause player socket,
│ │ buffer bytes, mint CID
│ │─ ControlRequest(CID, pattern, ip:port) ─▶
│ │ allocate worker+stream
│ │◀──────── SYN(streamId, CID) ────────────│
│ │ dial dedicated worker
│ │◀──────────── SYN(CID) ──────────────────│
│ │ takePending(CID) → bind dial destination,
│ │ forward buffered bytes write HAProxy v2 hdr
│ │─ DATA(streamId, handshake…) ───▶ ── handshake ──▶│
│ resume ─────────────────────────────│ bridge stream ⇄ dest
│ │─ DATA(handshake…) ─────────────▶ ── handshake ──▶│
│ resume ─────────────────────────────│ bridge conn ⇄ dest
│══════════════ player bytes ══ DATA ══▶│════ DATA ═══▶ dest.write │
│◀═══════════ dest bytes ═══ DATA ══════│◀═══ DATA ════ dest.read │
│ player closes ──────────────────────│─ FIN(streamId) ────────▶ close dest │
│ player closes ──────────────────────│─ FIN ──────────────────▶ close dest │
```
Key points:
@@ -87,6 +87,11 @@ Key points:
player's hostname — in `ControlRequest`, so the client can look it straight up
in its own route table. Invalid patterns are rejected at registration with a
non-zero `RegisterAck` status.
* A **per-IP limiter** runs first: a token bucket (`playerRatePerSec` /
`playerBurst`) and a concurrent-socket cap (`maxPlayersPerIp`). It applies
only to player intents — never Intent 17 — and unmatched hostnames still
consume a token, so a hostname scan is not a free flood. A refusal closes
the socket before CID minting or pause.
* The hub **pauses** the player socket the instant it matches, so no player
bytes are lost while the takeover is arranged; the buffered handshake is
forwarded **verbatim**, so the real server sees exactly what the player sent
@@ -94,54 +99,38 @@ Key points:
* **CID** is 16 random bytes minted by the hub and delivered only over the
encrypted control session, so only the intended client learns it. Any worker
connection presenting the correct CID is allowed to take over — that secrecy
is what binds a worker stream to the right pending player without any explicit
is what binds a worker conn to the right pending player without any explicit
client identity.
* Disconnects are symmetric: player-close → hub sends `FIN` → client closes the
destination; destination-close → client sends `FIN` → hub closes the player.
## 3. Multiplexing (worker connections)
## 3. Worker connections (1:1)
A worker connection is one encrypted TCP link carrying many **streams**. The
frame is intentionally tiny (PROTOCOL.md §7):
A worker connection is one encrypted TCP link carrying **exactly one player**.
The frame is intentionally tiny (PROTOCOL.md §7):
```
[plaintext VarInt length][ FrameType u8 | StreamID VarInt | Data… ] (payload encrypted)
[plaintext VarInt length][ FrameType u8 | Data… ] (payload encrypted)
```
Only the client opens streams (`SYN`), so stream-id allocation is a simple
per-connection counter with no coordination.
There is no stream id. The client dials a fresh worker for each
`ControlRequest` and binds it with `SYN(CID)` after `SessionReady`. A second
bind on the same conn is a protocol violation.
### 3.1 Pool & allocation
### 3.1 Allocation
The client keeps 1…`maxConn` worker connections and places each new stream on
the **least-loaded** one. The pool grows **breadth-first**: it dials out to
`maxConn` before stacking streams, so that no single TCP connection ever becomes
the shared point of failure for every player on the tunnel (PROTOCOL.md §7.1):
The client holds at most `maxTunnels` live worker connections (default 256). A
`ControlRequest` either gets its own TCP connection or is dropped — the hub
then closes the player when `pendingTimeoutMs` fires.
```
pick least-loaded conn; use it
if leastLoaded.streams >= 1 and pool.size + dialsInFlight < maxConn:
dial another worker conn in the background # the stream just placed does not wait
```
A dial is **never performed while holding the live-set lock** — session
establishment is network I/O, and one unresponsive hub must not park every
other player behind it. Each caller dials independently; callers are not
serialized behind a single in-flight handshake.
The new connection becomes the least-loaded one and picks up subsequent streams.
Once the pool is at `maxConn`, streams stack on the least-loaded connection;
going past 8 active streams there is logged as pool saturation but is not an
error.
Two properties of the dialing path matter as much as the placement rule:
* A dial is **never performed while holding the pool lock** — session
establishment is network I/O, and one unresponsive hub must not park every
other player behind it.
* Only when the pool is *empty* does a caller dial synchronously, and then
exactly one caller dials while the others wait on its result, so a burst of
arrivals cannot open a burst of redundant connections.
The e2e test `TestConcurrentStreamsUseMultipleConns` drives 20 simultaneous
streams with `maxConn=4` and confirms they spread over more than one connection
without exceeding the cap; `TestAllocateDoesNotWedgePoolOnStalledHub` covers the
stalled-dial path.
The e2e test `TestEachPlayerGetsOwnWorker` drives concurrent players and
confirms one worker conn each, without exceeding `maxTunnels`;
`TestDialDoesNotWedgeOnStalledHub` covers the stalled-dial path.
## 4. Encryption
@@ -166,31 +155,28 @@ stalled-dial path.
non-blocking; crypto is CPU-cheap. This trades multi-core scaling for
simplicity and correctness.
* **Client:** goroutine-per-concern. One goroutine reads each connection
(control or worker); `WriteFrame` is mutex-serialized so many stream goroutines
can share a worker connection safely. Each stream has two goroutines: `run`
pumps destination → hub, and `writeLoop` is the only writer to the
destination, draining a per-stream queue fed by the worker readLoop. The
readLoop itself never writes to a destination, so a stalled destination can
never block frame dispatch for other streams.
(control or worker); `WriteFrame` is mutex-serialized. Each tunnel has two
goroutines: `run` pumps destination → hub, and `writeLoop` is the only writer
to the destination, draining a queue fed by the worker readLoop. The
readLoop itself never writes to a destination, so a stalled destination cannot
stall WND / FIN / heartbeat dispatch on that conn.
## 6. Back-pressure & flow control
Three mechanisms operate at different granularities:
* **Per-stream credit windows** (PROTOCOL.md §7.3; the windows are exchanged
at session establishment): each stream direction has an independent byte
* **Per-connection credit windows** (PROTOCOL.md §7.3; the windows are exchanged
at session establishment): each tunnel direction has an independent byte
budget equal to the receiver's advertised window (default 256 KiB). A sender
that exhausts a
stream's window pauses *only that stream's source* — the hub pauses the one
player socket, the client parks the one destination-reader goroutine. Credit
is granted back (`WND` frames, batched at half-window) as bytes are actually
written to the terminal socket. The result: a slow player or slow destination
jams its own stream at a bounded buffer size and nothing else. This is what
eliminates head-of-line blocking between streams.
* **Aggregate TCP back-pressure** on each worker connection: when the shared
socket itself is congested (total bandwidth, not one stream), the hub parks
all sending players until it drains, and the client's `WriteFrame` blocks.
This is fair — when the pipe is genuinely full, everyone should slow down.
that exhausts a window pauses *only that player's source* — the hub pauses the
one player socket, the client parks the one destination-reader goroutine.
Credit is granted back (`WND` frames, batched at half-window) as bytes are
actually written to the terminal socket. The result: a slow player or slow
destination jams its own tunnel at a bounded buffer size and nothing else.
* **TCP back-pressure** on each worker connection: when that socket is
congested, the hub parks that one player until it drains, and the client's
`WriteFrame` blocks. Because the conn is 1:1, TCP HOL cannot stall another
player.
* **Client egress shaping** (optional, `maxBandwidth`; `client/shaper.go`): a
rate cap on everything the client sends to the hub, across all worker conns.
@@ -198,18 +184,18 @@ The first two mechanisms have no time dimension. A credit window bounds how many
bytes are *in flight*, and TCP back-pressure only reacts once the pipe is already
full — which on a residential uplink is too late. One player loading chunks fills
the line, the standing queue grows to seconds, and every other player's keepalive
times out. Nothing in §7.3 prevents that: each stream is individually
times out. Nothing in §7.3 prevents that: each tunnel is individually
well-behaved, and collectively they still overrun the link.
The shaper closes that gap with a token bucket for the rate and start-time fair
queueing for the split. A global virtual clock advances with each grant; every
stream remembers where its last request finished, and a new request is stamped
`max(stream.vfinish, vclock)`. Lowest stamp wins. A stream that keeps sending
pushes its own stamp further out and yields; a stream returning from idle is
tunnel remembers where its last request finished, and a new request is stamped
`max(tunnel.vfinish, vclock)`. Lowest stamp wins. A tunnel that keeps sending
pushes its own stamp further out and yields; a tunnel returning from idle is
clamped back to the clock, so it cannot bank credit for time it did not use, but
is not penalised for the idleness either. A stream sending a few hundred bytes
is not penalised for the idleness either. A tunnel sending a few hundred bytes
gets a nearer stamp than one sending a full chunk, so keepalives and chat overtake
bulk terrain data for free. One stream alone still gets the entire rate.
bulk terrain data for free. One tunnel alone still gets the entire rate.
Two details keep bursts cheap. The bucket banks 200 ms of transmission, so a
player joining spends it at once instead of paying for the cap in visible
@@ -222,7 +208,7 @@ This is entirely client-local: nothing about it appears on the wire, and the hub
is unaware. Only DATA is shaped — delaying a `FIN`, `WND` or `PONG` would cause
the false-death detection §7.4 exists to avoid.
The window also bounds memory: a stream can hold at most one window of
The window also bounds memory: a tunnel can hold at most one window of
undelivered data per direction (the client's pre-connect handshake buffer is
covered by the same bound). With stream resumption enabled (§7) the *sender*
holds a second window — the bytes it has sent but the peer has not yet credited,
@@ -230,44 +216,42 @@ kept so they can be retransmitted after an outage. That is not a new bound so
much as the existing one made symmetric: the region is exactly what flow control
already declared outstanding, which is why resumption needs no cap of its own.
Per-stream flow control is mandatory: the hub rejects a session whose Rekey
Per-connection flow control is mandatory: the hub rejects a session whose Rekey
lacks the STREAM_FC flag, and the client rejects a hub that does not echo it —
peers that predate the mechanism cannot connect at all.
What remains (by design) is TCP-level head-of-line blocking: a lost packet on
a worker connection stalls all its streams for one retransmit. That is inherent
to mux-over-TCP; the connection pool is the mitigation, and a datagram
transport (QUIC) would be the escape hatch if it ever matters.
TCP-level head-of-line blocking is now confined to one player: a lost packet
stalls only that player's tunnel for one retransmit. The cost is one handshake
and one NAT mapping per player instead of per pool slot.
## 7. Failure & recovery
* **Control session drop:** the client retries immediately, then backs off to a
10s cap, and re-registers all patterns. Existing worker connections and their
live streams are unaffected — they ride worker conns, which a control-session
players are unaffected — they ride worker conns, which a control-session
close never touches. The hub meanwhile keeps that session's routes as
*orphaned* for `registrationGraceMs` (PROTOCOL.md §5.2) and **holds** players
arriving on them instead of refusing them, replaying the control request once
a client re-registers the pattern. Without that, the reconnect window is one
in which every new player is told there is no such server.
* **Worker connection drop:** the connection leaves the pool either way. What
happens to its streams depends on whether STREAM_RESUME was negotiated:
* *without it* — every stream is torn down (destinations closed) and the hub
closes the corresponding player sockets, as it always did;
* *with it* — the streams are **hung** instead (PROTOCOL.md §7.5). The
destination sockets stay open, the hub pauses the player sockets and holds
them for its grace period, and the client reattaches each stream over a
freshly dialed connection, replaying byte-exactly from the offset the peer
reports. Players see a stall rather than a disconnect.
* **Worker connection drop:** the connection leaves the live set either way. What
happens to its player depends on whether STREAM_RESUME was negotiated:
* *without it* — the tunnel is torn down (destination closed) and the hub
closes the player socket, as it always did;
* *with it* — the player is **hung** instead (PROTOCOL.md §7.5). The
destination socket stays open, the hub pauses the player socket and holds
it for its grace period, and the client reattaches over a freshly dialed
connection, replaying byte-exactly from the offset the peer reports. The
player sees a stall rather than a disconnect.
* **Pending timeout:** if no worker takes over a matched player within
`pendingTimeoutMs`, the hub drops the pending entry and closes the player.
* **Bad PSK / bad timestamp / bad magic:** the hub closes the TCP connection;
the client's session establishment fails fast.
Resumption is worth the machinery because a worker connection is only the
*middle* leg of every stream it carries. When it dies both terminal sockets are
*middle* leg of the player it carries. When it dies both terminal sockets are
usually still healthy, so the old behaviour discarded working connections
because a replaceable transport failed — one conntrack expiry disconnected every
player sharing that connection. It also has to be byte-exact rather than
because a replaceable transport failed. It also has to be byte-exact rather than
best-effort: bytes handed to a dying socket are lost with no notification and the
frame cipher cannot be resynchronized, so an approximate reattach would splice
the tunneled protocol mid-packet, which is worse than a clean close.
@@ -279,13 +263,14 @@ wait on the control reconnect backoff.
## 8. Known limitations
1. No AEAD — payload integrity/authenticity is not cryptographically guaranteed.
2. TCP-level head-of-line blocking within a worker connection (lost packets;
see §6) — per-stream flow control removes the application-level variant only.
2. One handshake and one NAT mapping per player (the cost of dropping mux).
3. Single-event-loop hub (see §5) bounds throughput to one core.
4. `Intent 18` is reserved but only stubbed (the hub logs and closes).
4. `Intent 18` is reserved for a plaintext status probe.
5. Pattern ownership is last-writer-wins; two clients registering the identical
pattern string will silently reassign it. Overlapping-but-distinct regexes are
both kept, and when several match one hostname the winner is unspecified.
6. The player IP limiter matches exact addresses. A `/64` of IPv6 clients looks
like many independent IPs; aggregation is a later, isolated change.
These are deliberate scope choices for a connectivity-focused P2P tool, not
oversights; each is a small, well-isolated change away from being hardened.