break: replace muxed workers with 1:1 tunnels

Worker frames are now FrameType + payload; there is no stream id.
Each player gets its own worker conn. maxTunnels (default 256)
caps concurrent tunnels. The old maxConn pool size is ignored so
existing configs do not silently admit only a handful of players.

Resume, per-direction windows, the control session, and the
DATA-only shaper stay. A dropped worker still hangs that one
player and reattaches over a fresh conn.

Add a hub-side per-IP limiter for player intents only (default
8/s, burst 16, 64 concurrent). Unmatched hostnames consume a
token; Intent 17 is never counted. 0 disables each knob.
This commit is contained in:
iceBear67
2026-08-15 18:32:51 +08:00
parent da17140583
commit 4df2560331
27 changed files with 1174 additions and 856 deletions
+20 -17
View File
@@ -57,7 +57,7 @@ they exercise, so run them a few times rather than trusting a single pass.
| First handshake | Becomes | | First handshake | Becomes |
|---|---| |---|---|
| `Intent 17`, rekey magic `0x01` | **control session** — pattern registration, `ControlRequest`, Ping/Pong | | `Intent 17`, rekey magic `0x01` | **control session** — pattern registration, `ControlRequest`, Ping/Pong |
| `Intent 17`, rekey magic `0x02` | **worker conn**multiplexed player streams | | `Intent 17`, rekey magic `0x02` | **worker conn**one player, 1:1 |
| any other intent | **player** — hostname regex-matched, then tunneled | | any other intent | **player** — hostname regex-matched, then tunneled |
Session establishment (both kinds): plaintext handshake whose `Server Address` is Session establishment (both kinds): plaintext handshake whose `Server Address` is
@@ -69,8 +69,8 @@ from `rand‖ts` → hub replies `SessionReady` echoing accepted flags and its w
The hub pauses and buffers a matched player socket, mints a random 16-byte **CID**, and The hub pauses and buffers a matched player socket, mints a random 16-byte **CID**, and
sends `ControlRequest(CID, matchedPattern, ip, port)` down the control session. The client sends `ControlRequest(CID, matchedPattern, ip, port)` down the control session. The client
allocates a worker stream, `SYN`s it with that CID, dials the destination, and the hub dials a dedicated worker conn, `SYN`s it with that CID, dials the destination, and the hub
binds the pending player to that stream. The CID's secrecy (it only ever travels encrypted binds the pending player to that conn. The CID's secrecy (it only ever travels encrypted
over the control session) is what authorizes the takeover — there is no other client over the control session) is what authorizes the takeover — there is no other client
identity check. The hub echoes **the matched pattern string, not the player's hostname**, identity check. The hub echoes **the matched pattern string, not the player's hostname**,
so the client can look it up directly in its route table; the buffered handshake is so the client can look it up directly in its route table; the buffered handshake is
@@ -80,10 +80,10 @@ forwarded verbatim so the backend sees the original hostname.
- `server/src/main/java/io/icybear/redapricot/``HubConnection` (per-socket state machine: - `server/src/main/java/io/icybear/redapricot/``HubConnection` (per-socket state machine:
handshake parse → dispatch → rekey → control/worker/player), `Hub` (pattern registry, CID handshake parse → dispatch → rekey → control/worker/player), `Hub` (pattern registry, CID
table, pending players), `ControlSession`, `WorkerConn` (mux demux + per-stream state), table, pending players), `ControlSession`, `WorkerConn` (1:1 tunnel + flow-control state),
`net/EncryptedFrames`, `crypto/Crypto`, `util/` (VarInt, ProtoReader/Writer, Hex). `net/EncryptedFrames`, `crypto/Crypto`, `util/` (VarInt, ProtoReader/Writer, Hex).
- `client/``client.go` (control session, reconnect, dispatch), `worker.go` (pool, - `client/``client.go` (control session, reconnect, dispatch), `worker.go` (1:1
allocation, `WorkerConn`, `Stream` and its two goroutines), `shaper.go` (egress rate cap), dial, `WorkerConn`, `Stream` and its two goroutines), `shaper.go` (egress rate cap),
`velocity.go` (Velocity modern-forwarding interception), `proxyproto.go` (HAProxy v2), `velocity.go` (Velocity modern-forwarding interception), `proxyproto.go` (HAProxy v2),
`config.go` (config + all protocol constants), `wire/` (VarInt/MC codec, SHA3+ChaCha20, `config.go` (config + all protocol constants), `wire/` (VarInt/MC codec, SHA3+ChaCha20,
`FramedConn`). `FramedConn`).
@@ -95,10 +95,10 @@ forwarded verbatim so the backend sees the original hostname.
per-connection state are touched by one event loop and need no locking. Never introduce a per-connection state are touched by one event loop and need no locking. Never introduce a
blocking call there. blocking call there.
- **Client:** one goroutine reads each connection; `WriteFrame` is mutex-serialized. Each - **Client:** one goroutine reads each connection; `WriteFrame` is mutex-serialized. Each
stream has exactly two goroutines — `run` (destination → hub) and `writeLoop` (the *only* tunnel has exactly two goroutines — `run` (destination → hub) and `writeLoop` (the *only*
writer to the destination, draining a queue fed by the worker readLoop). The readLoop must writer to the destination, draining a queue fed by the worker readLoop). The readLoop must
never write to a destination, or a stalled backend blocks frame dispatch for every other never write to a destination: WND is granted only after the dest write completes, and a
stream on that conn. stalled backend must not stall heartbeat / FIN dispatch on that conn.
## Invariants to preserve when editing ## Invariants to preserve when editing
@@ -111,25 +111,28 @@ forwarded verbatim so the backend sees the original hostname.
`‖ 0x02` s2c). Both directions must never share a keystream. Go `x/crypto/chacha20` and `‖ 0x02` s2c). Both directions must never share a keystream. Go `x/crypto/chacha20` and
Java JCE `ChaCha20` are byte-identical here, and unit tests on both sides pin the same Java JCE `ChaCha20` are byte-identical here, and unit tests on both sides pin the same
SHA3-224 vector — keep that pinning if you touch crypto. SHA3-224 vector — keep that pinning if you touch crypto.
- **Per-stream flow control is mandatory.** The hub rejects a session whose rekey lacks - **Per-connection flow control is mandatory.** The hub rejects a session whose rekey lacks
`FLAG_STREAM_FC`; the client rejects a hub that does not echo it. Credit is granted back `FLAG_STREAM_FC`; the client rejects a hub that does not echo it. Credit is granted back
(`WND`) only as bytes are actually written to the terminal socket, batched at half-window. (`WND`) only as bytes are actually written to the terminal socket, batched at half-window.
- **Only stream ids allocated by the client exist** (a per-conn counter starting at 1); - **One worker conn carries one player.** There is no stream id. The first business frame
sid `0` is reserved for connection-scoped `PING`/`PONG`. after `SessionReady` is `SYN` or `RESUME`; a second bind on the same conn is a protocol
violation. `PING`/`PONG` are connection-scoped frames.
- **Only DATA is shaped** by `client/shaper.go`. Delaying `FIN`, `WND`, or `PONG` would trip - **Only DATA is shaped** by `client/shaper.go`. Delaying `FIN`, `WND`, or `PONG` would trip
the very liveness detection the heartbeat exists for. The shaper is client-local and the very liveness detection the heartbeat exists for. The shaper is client-local and
invisible on the wire. invisible on the wire.
- **The pool grows breadth-first** (`StreamsBeforeGrowing = 1`): dial up to `maxConn` before - **Each player gets its own worker dial**, up to `maxTunnels` (default 256). A dial is
stacking streams, so one TCP connection is never the shared point of failure for every never performed while holding the live-set lock: session establishment is network I/O,
player. `SaturationThreshold = 8` only logs, once the pool is already at `maxConn`. and one unresponsive hub must not block unrelated players.
A dial is never performed while holding the pool lock. - **The hub IP limiter is player-only.** Intent 17 (control + workers) is never
admitted through it — those sockets share the client's one address. Unmatched
player hostnames still consume a token. `0` turns each knob off.
- **Liveness is explicit everywhere:** every session heartbeats (drop after - **Liveness is explicit everywhere:** every session heartbeats (drop after
`3 × pingIntervalMs`), every socket write is bounded, establishment has a deadline. A `3 × pingIntervalMs`), every socket write is bounded, establishment has a deadline. A
silently blackholed path (NAT forgetting a flow, no FIN/RST) must recover without operator silently blackholed path (NAT forgetting a flow, no FIN/RST) must recover without operator
action — `TestBlackholedPathRecovers` guards this. action — `TestBlackholedPathRecovers` guards this.
- **Stream resumption is byte-exact or it is nothing** (`PROTOCOL.md §7.5`, - **Stream resumption is byte-exact or it is nothing** (`PROTOCOL.md §7.5`,
`client/resume.go`, `WorkerConn.handleResume`). A worker-conn drop hangs the player and `client/resume.go`, `WorkerConn.handleResume`). A worker-conn drop hangs the player and
reattaches the stream over a fresh conn. Three offsets are tracked per direction and are reattaches over a fresh conn. Three offsets are tracked per direction and are
*not* interchangeable: replay from the peer's **accepted** offset, restate the window from *not* interchangeable: replay from the peer's **accepted** offset, restate the window from
its **delivered** offset, and never size the window from **credited** — the grants in its **delivered** offset, and never size the window from **credited** — the grants in
flight when the conn died are gone for good, and a window derived from them can be flight when the conn died are gone for good, and a window derived from them can be
+102 -91
View File
@@ -15,7 +15,7 @@ There are three roles:
| **Player** | any | An ordinary Minecraft client connecting through the hub. | | **Player** | any | An ordinary Minecraft client connecting through the hub. |
``` ```
Player ──MC──▶ Hub(server) ══WorkerConn(mux)══▶ Client ──MC──▶ Destination Player ──MC──▶ Hub(server) ══WorkerConn(1:1)══▶ Client ──MC──▶ Destination
▲ registers patterns / receives control requests │ ▲ registers patterns / receives control requests │
└────────────── Control Session ────────────────────┘ └────────────── Control Session ────────────────────┘
``` ```
@@ -154,18 +154,18 @@ RandLen : VarInt # 8 ≤ RandLen ≤ 64
Rand : Bytes[RandLen] # cryptographically random Rand : Bytes[RandLen] # cryptographically random
Timestamp : I64 # client's epoch milliseconds Timestamp : I64 # client's epoch milliseconds
Flags : VarInt # feature flags; bit 0x01 (STREAM_FC) MUST be set Flags : VarInt # feature flags; bit 0x01 (STREAM_FC) MUST be set
RecvWindow: VarInt # client's per-stream receive window, bytes (§7.3) RecvWindow: VarInt # client's per-connection receive window, bytes (§7.3)
``` ```
`Flags` is a bitfield of features. `Flags` is a bitfield of features.
| Bit | Name | Meaning | | Bit | Name | Meaning |
|-----|------|---------| |-----|------|---------|
| `0x01` | STREAM_FC | **Per-stream flow control** (§7.3). Mandatory. | | `0x01` | STREAM_FC | **Per-connection flow control** (§7.3). Mandatory. |
| `0x02` | WORKER_HEARTBEAT | Mux-level `PING`/`PONG` on worker conns (§7.4). Optional. | | `0x02` | WORKER_HEARTBEAT | Connection-level `PING`/`PONG` on worker conns (§7.4). Optional. |
| `0x04` | STREAM_RESUME | **Stream resumption** (§7.5): a worker-conn drop hangs the player rather than closing it. Optional. | | `0x04` | STREAM_RESUME | **Stream resumption** (§7.5): a worker-conn drop hangs the player rather than closing it. Optional. |
STREAM_FC is mandatory: `RecvWindow` advertises the client's per-stream receive STREAM_FC is mandatory: `RecvWindow` advertises the client's per-connection receive
window in bytes and must be positive. The hub closes the connection if the flag window in bytes and must be positive. The hub closes the connection if the flag
is missing, `RecvWindow` is absent or non-positive, or the fields are malformed. is missing, `RecvWindow` is absent or non-positive, or the fields are malformed.
@@ -195,7 +195,7 @@ SessionReady : payload = [ 0x00, Flags: VarInt, RecvWindow: VarInt,
``` ```
The hub echoes the accepted flags (STREAM_FC set) followed by its own The hub echoes the accepted flags (STREAM_FC set) followed by its own
per-stream receive window. A client must reject a SessionReady without the per-connection receive window. A client must reject a SessionReady without the
STREAM_FC flag or without a positive window (an unsupported hub). STREAM_FC flag or without a positive window (an unsupported hub).
`ResumeGraceMs` is present only when the hub accepts STREAM_RESUME, and states `ResumeGraceMs` is present only when the hub accepts STREAM_RESUME, and states
@@ -314,35 +314,36 @@ Msg : String
Purely informational; the receiver logs it. Purely informational; the receiver logs it.
## 7. Worker conn & multiplexing ## 7. Worker conn
A **Worker Conn** (`Magic == 0x02`) carries player↔destination traffic for many A **Worker Conn** (`Magic == 0x02`) carries player↔destination traffic for
players over one TCP connection using a minimal stream multiplexer. The unit of **exactly one player**. The TCP connection *is* the tunnel: there is no stream
work is a **stream**. Stream ids are assigned by the **client** (the only side id and no multiplexer. The client dials a fresh worker conn for each
that opens streams), unique per worker conn, starting at 1 and increasing. `ControlRequest` (and for each resumption attempt).
Each encrypted frame on a worker conn carries one **mux frame**: Each encrypted frame on a worker conn carries one **tunnel frame**:
``` ```
FrameType : u8 FrameType : u8
StreamID : VarInt
Data : Bytes[...] # remainder of the frame payload Data : Bytes[...] # remainder of the frame payload
``` ```
| FrameType | Name | Direction | Data | | FrameType | Name | Direction | Data |
|-----------|------|-----------|------| |-----------|------|-----------|------|
| `0x00` | SYN | C → S | `CID: Bytes[16]` open a stream to take over the pending player identified by CID. | | `0x00` | SYN | C → S | `CID: Bytes[16]` — take over the pending player identified by CID. |
| `0x01` | DATA | both | raw tunneled bytes for the stream. | | `0x01` | DATA | both | raw tunneled bytes. |
| `0x02` | FIN | both | *(empty)* — graceful close of the stream (both directions). This is the "disconnect" the hub sends when the player leaves. | | `0x02` | FIN | both | *(empty)* — graceful close (both directions). This is the "disconnect" the hub sends when the player leaves. |
| `0x03` | RST | both | *(optional 1 byte reason)* — abnormal close (e.g. CID unknown/expired, destination dial failed). | | `0x03` | RST | both | *(optional 1 byte reason)* — abnormal close (e.g. CID unknown/expired, destination dial failed). |
| `0x04` | WND | both | `Delta: VarInt` — flow-control credit grant (§7.3). | | `0x04` | WND | both | `Delta: VarInt` — flow-control credit grant (§7.3). |
| `0x05` | PING | both | `Nonce: I64` — liveness probe on reserved StreamID `0` (§7.4). | | `0x05` | PING | both | `Nonce: I64` — liveness probe (§7.4). |
| `0x06` | PONG | both | `Nonce: I64` — echoes the probe's nonce (§7.4). | | `0x06` | PONG | both | `Nonce: I64` — echoes the probe's nonce (§7.4). |
| `0x07` | RESUME | C → S | `CID: Bytes[16]`, `Accepted: I64`, `Delivered: I64` — reattach a hung stream to this conn (§7.5). | | `0x07` | RESUME | C → S | `CID: Bytes[16]`, `Accepted: I64`, `Delivered: I64` — reattach a hung player to this conn (§7.5). |
| `0x08` | RESUME_ACK | S → C | `Accepted: I64`, `Delivered: I64`, `NewCID: Bytes[16]` — the reattach succeeded (§7.5). | | `0x08` | RESUME_ACK | S → C | `Accepted: I64`, `Delivered: I64`, `NewCID: Bytes[16]` — the reattach succeeded (§7.5). |
There is no explicit SYN-ACK: success is implied by the hub forwarding the The first business frame after `SessionReady` must be `SYN` or `RESUME`. A
buffered Handshake as the stream's first `DATA`; failure is an `RST`. second bind on an already-bound conn is a protocol violation: the hub replies
`RST` and closes. There is no explicit SYN-ACK: success is implied by the hub
forwarding the buffered Handshake as the first `DATA`; failure is an `RST`.
`RST` reason codes. The byte remains optional — a peer that predates it sends `RST` reason codes. The byte remains optional — a peer that predates it sends
none, and a receiver must tolerate its absence — but distinguishing the reasons none, and a receiver must tolerate its absence — but distinguishing the reasons
@@ -358,25 +359,17 @@ took it" call for opposite responses.
| `0x04` | FLOW_CONTROL | The peer exceeded its advertised window. | | `0x04` | FLOW_CONTROL | The peer exceeded its advertised window. |
| `0x05` | DIAL_FAILED | The client could not reach the destination. | | `0x05` | DIAL_FAILED | The client could not reach the destination. |
### 7.1 Stream allocation (client side) ### 7.1 Tunnel allocation (client side)
The client keeps a pool of `1 ≤ N ≤ max_conn` worker conns (`max_conn` The client dials one worker conn per player, up to a configurable
configurable, `1..8`). The pool grows **breadth-first**: spreading streams over `maxTunnels` cap (default 256, clamped to `[1, 4096]`). There is no pool and no
several connections keeps any single TCP connection from becoming the shared least-loaded placement: a `ControlRequest` either gets its own TCP connection or
point of failure for every player on the tunnel. To place a new stream: is dropped (the hub then closes the player when `pendingTimeoutMs` fires).
1. Pick the worker conn with the **fewest active streams**, and use it. A dial is never performed while holding the live-set lock: session establishment
2. If that conn already carries at least one stream and is network I/O, and one unresponsive hub must not be able to block unrelated
`poolSize + dialsInFlight < max_conn`, dial another worker conn **in the players. Each dial is independent and bounded by the session-establishment
background**. The stream just placed is not delayed by that dial; the new deadline (§7.4); callers are not serialized behind a single in-flight handshake.
conn becomes the least-loaded one and picks up subsequent streams.
3. Once the pool is at `max_conn`, streams stack on the least-loaded conn.
Exceeding `8` active streams there is logged as pool saturation.
Only when the pool is *empty* does a caller dial synchronously, and then exactly
one caller dials while the others wait for its result. A dial is never performed
while holding the pool lock: session establishment is network I/O, and one
unresponsive hub must not be able to block unrelated players.
### 7.2 End-to-end player flow ### 7.2 End-to-end player flow
@@ -388,45 +381,43 @@ unresponsive hub must not be able to block unrelated players.
control session. If no SYN arrives within `pendingTimeoutMs` (default 10000) control session. If no SYN arrives within `pendingTimeoutMs` (default 10000)
the pending entry is dropped and the player socket closed. the pending entry is dropped and the player socket closed.
3. The client receives `ControlRequest`, looks up the destination for `Pattern`, 3. The client receives `ControlRequest`, looks up the destination for `Pattern`,
allocates a worker conn + `StreamID`, and sends `SYN(StreamID, CID)`. In dials a dedicated worker conn, and sends `SYN(CID)`. In parallel it dials the
parallel it dials the destination and (if configured) writes a HAProxy v2 destination and (if configured) writes a HAProxy v2 header (§8) carrying
header (§8) carrying `PlayerIP:PlayerPort`. `PlayerIP:PlayerPort`.
4. The hub matches `CID` to the pending player, binds 4. The hub matches `CID` to the pending player, binds
`(workerConn, StreamID) ↔ playerSocket`, forwards the buffered bytes as `workerConn ↔ playerSocket`, forwards the buffered bytes as `DATA`, and
`DATA`, and resumes the player socket. Subsequent player bytes become `DATA` resumes the player socket. Subsequent player bytes become `DATA` frames;
frames; `DATA` frames from the client are written to the player socket. If `DATA` frames from the client are written to the player socket. If `CID` is
`CID` is unknown/expired the hub replies `RST`. unknown/expired the hub replies `RST`.
5. When the player disconnects the hub sends `FIN` on the stream; the client 5. When the player disconnects the hub sends `FIN`; the client closes the
closes the destination. When the destination closes, the client sends `FIN`; destination. When the destination closes, the client sends `FIN`; the hub
the hub closes the player socket. `RST` is treated the same way (hard close). closes the player socket. `RST` is treated the same way (hard close).
Data on a worker conn is subject to that TCP connection's back-pressure for Data on a worker conn is subject to that TCP connection's back-pressure. Credit
its **aggregate** bandwidth; *per-stream* fairness is governed by the credit windows of §7.3 bound how many bytes may be in flight on that one tunnel.
windows of §7.3.
### 7.3 Per-stream flow control ### 7.3 Per-connection flow control
Every stream carries an independent credit window per direction: Every worker conn carries an independent credit window per direction:
* Each side advertised its **receive window** W (bytes) at session setup. A * Each side advertised its **receive window** W (bytes) at session setup. A
sender may have at most W un-credited DATA bytes outstanding per stream; the sender may have at most W un-credited DATA bytes outstanding; the initial
initial budget is W, spent as DATA is sent (`Data` length only — SYN/FIN/RST budget is W, spent as DATA is sent (`Data` length only — SYN/FIN/RST frames
frames are free) starting with the very first DATA on the stream (including are free) starting with the very first DATA (including the hub's forwarded
the hub's forwarded handshake). handshake).
* The receiver returns credit with `WND(Delta)` once bytes are **delivered to * The receiver returns credit with `WND(Delta)` once bytes are **delivered to
the terminal socket** (written to the player / destination connection), not the terminal socket** (written to the player / destination connection), not
when they are merely buffered. Receivers should batch grants (the reference when they are merely buffered. Receivers should batch grants (the reference
implementations send one `WND` per W/2 bytes consumed). implementations send one `WND` per W/2 bytes consumed).
* A sender whose window is exhausted pauses reading **that stream's source * A sender whose window is exhausted pauses reading **that player's source
socket only**; the shared worker conn is never paused because of a single socket only**. A receiver that observes more than W un-credited bytes may
stream. A receiver that observes more than W un-credited bytes on a stream reset the tunnel (`RST`) as a protocol violation.
may reset it (`RST`) as a protocol violation.
* Senders should also cap individual DATA payloads (the reference * Senders should also cap individual DATA payloads (the reference
implementations use 32 KiB) so one stream cannot monopolize the link for a implementations use 32 KiB) so one write cannot occupy the link for a full
full 1-MiB frame. 1-MiB frame.
Both windows may differ (each side enforces the one its peer advertised). Both windows may differ (each side enforces the one its peer advertised).
`Delta` must be positive; a `WND` for an unknown stream id is ignored. `Delta` must be positive; a `WND` on an unbound worker conn is ignored.
### 7.4 Liveness ### 7.4 Liveness
@@ -444,10 +435,9 @@ Every established session is therefore covered by a heartbeat:
hub answers `Pong`. If no `Pong` arrives for `3 × pingIntervalMs`, the client hub answers `Pong`. If no `Pong` arrives for `3 × pingIntervalMs`, the client
closes the session, which triggers its normal reconnect with backoff. closes the session, which triggers its normal reconnect with backoff.
* **Worker conns** — when WORKER_HEARTBEAT was negotiated, the same exchange * **Worker conns** — when WORKER_HEARTBEAT was negotiated, the same exchange
runs as mux `PING`/`PONG` frames on the reserved StreamID `0`. On timeout the runs as connection-level `PING`/`PONG` frames. On timeout the client closes
client closes the conn and drops it from the pool. Its streams are reset, the conn. The player is reset, unless STREAM_RESUME was negotiated, in which
unless STREAM_RESUME was negotiated, in which case they are hung and reattached case it is hung and reattached over a fresh conn instead (§7.5).
over a fresh conn instead (§7.5).
* **Hub side** — an established redapricot session that receives no frame for * **Hub side** — an established redapricot session that receives no frame for
`sessionIdleTimeoutMs` (default 90000, `0` disables) is closed. Player `sessionIdleTimeoutMs` (default 90000, `0` disables) is closed. Player
connections are never subject to this. connections are never subject to this.
@@ -457,26 +447,24 @@ that has become unreachable at the IP layer.
Session establishment (§4) is bounded by a single deadline covering the dial, Session establishment (§4) is bounded by a single deadline covering the dial,
the `Rekey` write and the `SessionReady` read, and every frame write is bounded the `Rekey` write and the `SessionReady` read, and every frame write is bounded
too — a peer that stops reading must not be able to park a whole multiplexed too — a peer that stops reading must not be able to park a write forever.
connection inside one write.
### 7.5 Stream resumption (STREAM_RESUME) ### 7.5 Stream resumption (STREAM_RESUME)
A worker conn is only the middle leg of every stream it carries. When it dies, A worker conn is only the middle leg of the player it carries. When it dies,
both terminal sockets — the player's and the destination's — are usually still both terminal sockets — the player's and the destination's — are usually still
healthy, so resetting the streams discards working connections because a healthy, so resetting the tunnel discards working connections because a
replaceable transport failed. One conntrack expiry disconnects every player on replaceable transport failed.
that conn.
With STREAM_RESUME negotiated, a worker-conn drop instead **hangs** each stream: With STREAM_RESUME negotiated, a worker-conn drop instead **hangs** the player:
* the hub pauses the player socket, keeps its state, and holds it for * the hub pauses the player socket, keeps its state, and holds it for
`ResumeGraceMs` from the moment of the *first* hang (an absolute deadline, so a `ResumeGraceMs` from the moment of the *first* hang (an absolute deadline, so a
flapping client cannot extend it indefinitely); flapping client cannot extend it indefinitely);
* the client keeps the destination socket open and reattaches the stream over a * the client keeps the destination socket open and reattaches over a fresh
fresh worker conn by sending `RESUME` with the stream's CID; worker conn by sending `RESUME` with the player's CID;
* the hub answers `RESUME_ACK`, or `RST(UNKNOWN_STREAM)` if it holds no such * the hub answers `RESUME_ACK`, or `RST(UNKNOWN_STREAM)` if it holds no such
stream — which is also what a client gets from a hub that has restarted. player — which is also what a client gets from a hub that has restarted.
**Resumption is byte-exact, and must be.** Frames handed to a dying socket are **Resumption is byte-exact, and must be.** Frames handed to a dying socket are
lost with no notification, and the frame cipher cannot be resynchronized, so each lost with no notification, and the frame cipher cannot be resynchronized, so each
@@ -512,13 +500,13 @@ Two rules deserve emphasis, because the obvious simplifications are wrong:
credit can ever come back. credit can ever come back.
`RESUME_ACK` carries a freshly minted `NewCID`, which replaces the old one. A CID `RESUME_ACK` carries a freshly minted `NewCID`, which replaces the old one. A CID
therefore stays single-use even though a stream may be reattached many times, so therefore stays single-use even though a player may be reattached many times, so
a leaked CID grants no more than the outage in which it was observed. a leaked CID grants no more than the outage in which it was observed.
Resumption is **hub-instance-affine**: a CID means nothing to a second hub behind Resumption is **hub-instance-affine**: a CID means nothing to a second hub behind
an L4 load balancer, which answers `RST(UNKNOWN_STREAM)` and lets the client tear an L4 load balancer, which answers `RST(UNKNOWN_STREAM)` and lets the client tear
down at once. Because the grace period holds player sockets and their buffers, a down at once. Because the grace period holds player sockets and their buffers, a
hub bounds the number of hung streams and the bytes they retain, dropping the hub bounds the number of hung players and the bytes they retain, dropping the
oldest first when either cap is reached. oldest first when either cap is reached.
## 8. HAProxy protocol v2 (optional) ## 8. HAProxy protocol v2 (optional)
@@ -555,12 +543,15 @@ big-endian.
"resumeGraceMs": 20000, "resumeGraceMs": 20000,
"maxParkedStreams": 256, "maxParkedStreams": 256,
"statsIntervalMs": 0, "statsIntervalMs": 0,
"registrationGraceMs": 15000 "registrationGraceMs": 15000,
"playerRatePerSec": 8,
"playerBurst": 16,
"maxPlayersPerIp": 64
} }
``` ```
`streamWindowBytes` (optional, default 262144, clamped to [32768, 8388608]) is `streamWindowBytes` (optional, default 262144, clamped to [32768, 8388608]) is
the hub's advertised per-stream receive window (§7.3). the hub's advertised per-connection receive window (§7.3).
`sessionIdleTimeoutMs` (optional, default 90000) closes an established control `sessionIdleTimeoutMs` (optional, default 90000) closes an established control
session or worker conn that has gone silent for that long (§7.4). It must stay session or worker conn that has gone silent for that long (§7.4). It must stay
@@ -587,13 +578,27 @@ session's routes are kept as **orphaned** rather than deleted (§5.2). `0`
disables it, restoring the behaviour of dropping routes the instant a session disables it, restoring the behaviour of dropping routes the instant a session
closes. closes.
`playerRatePerSec` (optional, default 8) and `playerBurst` (optional, default
16) are a per-IP token bucket applied **only to player connections** (any
handshake whose `Intent` is not 17 or 18). One arrival consumes one token;
without a token the socket is closed after the handshake, before `match`, CID
minting, or pause. Unmatched hostnames still consume a token — otherwise a
hostname scan is a free flood. Intent 17 is never admitted through the
limiter: every worker conn comes from the client's one address, and limiting
those would be the hub throttling its own client. `playerRatePerSec: 0` turns
the bucket off.
`maxPlayersPerIp` (optional, default 64) caps concurrent player sockets from
one address (pending + live + parked). `0` disables the cap. Addresses are
matched exactly; IPv6 `/64` aggregation is out of scope.
### 9.2 Client — JSON ### 9.2 Client — JSON
```json ```json
{ {
"server": "127.0.0.1:25565", "server": "127.0.0.1:25565",
"psk": "change-me", "psk": "change-me",
"maxConn": 4, "maxTunnels": 256,
"pingIntervalMs": 20000, "pingIntervalMs": 20000,
"streamWindowBytes": 262144, "streamWindowBytes": 262144,
"maxBandwidth": "20mbps", "maxBandwidth": "20mbps",
@@ -606,8 +611,14 @@ closes.
} }
``` ```
`maxTunnels` (optional, default 256, clamped to [1, 4096]) is how many
concurrent 1:1 worker connections the client will hold. A `ControlRequest`
arriving at the cap is dropped. The older `maxConn` key (the mux-era pool size,
clamped 18) is ignored if present: treating it as a player cap would silently
limit a previously-working config to a handful of players.
`streamWindowBytes` (optional, default 262144, clamped to [32768, 8388608]) is `streamWindowBytes` (optional, default 262144, clamped to [32768, 8388608]) is
the client's advertised per-stream receive window (§7.3). the client's advertised per-connection receive window (§7.3).
`streamResume` (optional, default true) offers STREAM_RESUME (§7.5). With it `streamResume` (optional, default true) offers STREAM_RESUME (§7.5). With it
false the client never offers the flag, never retains a byte for retransmission, false the client never offers the flag, never retains a byte for retransmission,
@@ -626,7 +637,7 @@ time per conn. These distinguish a slow backend from a saturated uplink from a
bad path, which throughput alone cannot. bad path, which throughput alone cannot.
`maxBandwidth` (optional, default unlimited) caps the aggregate rate at which the `maxBandwidth` (optional, default unlimited) caps the aggregate rate at which the
client sends `DATA` to the hub, shared fairly across streams. Accepts `"20mbps"` client sends `DATA` to the hub, shared fairly across tunnels. Accepts `"20mbps"`
(decimal bit units), `"2MB/s"` (binary byte units), or a bare number of bytes per (decimal bit units), `"2MB/s"` (binary byte units), or a bare number of bytes per
second; the minimum is 8192 B/s. **This is a purely local policy and has no second; the minimum is 8192 B/s. **This is a purely local policy and has no
effect on the wire format** — a shaped client is indistinguishable from a slow effect on the wire format** — a shaped client is indistinguishable from a slow
@@ -664,17 +675,17 @@ not appear on the tunnel wire, and the exchange is invisible to the player.
| pattern matching | case-insensitive, whole-string regex; first match wins | | pattern matching | case-insensitive, whole-string regex; first match wins |
| CID length | 16 bytes | | CID length | 16 bytes |
| max frame payload | 1 MiB | | max frame payload | 1 MiB |
| pool growth | breadth-first: grow to `max_conn` before stacking | | worker framing | `FrameType` + payload; no stream id |
| saturation threshold (logged) | active streams `> 8` at `max_conn` | | max concurrent worker conns | `maxTunnels`, default 256, clamped `[1, 4096]` |
| max worker conns | `max_conn ∈ [1,8]` | | player IP rate / burst / concurrent | `8 /s`, burst `16`, `maxPlayersPerIp` `64` (Intent ∉ {17, 18} only) |
| feature flag: per-stream flow control | `0x01` (mandatory) | | feature flag: per-connection flow control | `0x01` (mandatory) |
| feature flag: worker heartbeat | `0x02` (negotiated) | | feature flag: worker heartbeat | `0x02` (negotiated) |
| heartbeat timeout | `3 × pingIntervalMs` | | heartbeat timeout | `3 × pingIntervalMs` |
| hub session idle timeout | 90000 ms (`0` disables) | | hub session idle timeout | 90000 ms (`0` disables) |
| stream window default / bounds | 256 KiB, clamped to [32 KiB, 8 MiB] | | connection window default / bounds | 256 KiB, clamped to [32 KiB, 8 MiB] |
| feature flag: stream resumption | 0x04 (negotiated) | | feature flag: stream resumption | 0x04 (negotiated) |
| mux RESUME / RESUME_ACK | 0x07 / 0x08 | | RESUME / RESUME_ACK | 0x07 / 0x08 |
| resume grace: hub / client default | 20000 ms / 15000 ms (hub value advertised) | | resume grace: hub / client default | 20000 ms / 15000 ms (hub value advertised) |
| retained region per stream per direction | bounded by the stream window | | retained region per tunnel per direction | bounded by the connection window |
| WND grant batching (reference) | one grant per window/2 consumed | | WND grant batching (reference) | one grant per window/2 consumed |
| DATA chunk cap (reference) | 32 KiB | | DATA chunk cap (reference) | 32 KiB |
+42 -39
View File
@@ -13,9 +13,9 @@ field, so a vanilla Minecraft client needs no modification.
> made reachable from the outside. > made reachable from the outside.
``` ```
Player ──MC──▶ Hub (Java) ══ Worker Conn (mux) ══▶ Client (Go) ──MC──▶ Real MC server Player ──MC──▶ Hub (Java) ══ Worker Conn (1:1) ══▶ Client (Go) ──MC──▶ Real MC server
vanilla client public IP encrypted, pooled behind NAT (localhost) vanilla client public IP one encrypted TCP behind NAT (localhost)
per player
└────────────── Control Session ─────────┘ └────────────── Control Session ─────────┘
(pattern registration + control requests) (pattern registration + control requests)
``` ```
@@ -40,13 +40,11 @@ one or more hostname patterns — each a **regular expression**. When a player
connects to the hub with a hostname that matches a registered pattern (and any connects to the hub with a hostname that matches a registered pattern (and any
normal `Intent`), the hub assigns a random **CID**, buffers normal `Intent`), the hub assigns a random **CID**, buffers
the player's bytes, and asks the client (via the control session) to take over. the player's bytes, and asks the client (via the control session) to take over.
The client picks a **worker connection**a multiplexed, encrypted TCP link The client dials a fresh **worker connection**one encrypted TCP link per
that carries many players as lightweight *streams* — opens a stream for that CID, player — announces the CID with `SYN`, dials the real destination (optionally
dials the real destination (optionally announcing the player's real IP with the announcing the player's real IP with the **HAProxy v2** protocol), and bridges
**HAProxy v2** protocol), and bridges the two ends. Worker connections are the two ends. `maxTunnels` is the only player cap; the retired `maxConn` field
pooled: the client uses up to `maxConn` of them and always places a new stream is ignored so an old config does not silently admit only four players.
on the least-loaded one, growing the pool to `maxConn` before stacking streams
so no single TCP connection carries every player.
## Repository layout ## Repository layout
@@ -114,7 +112,7 @@ cp client/config.example.json client.json
# { # {
# "server": "hub.example.com:25565", # "server": "hub.example.com:25565",
# "psk": "same-as-the-hub", # "psk": "same-as-the-hub",
# "maxConn": 4, # "maxTunnels": 256,
# "mappings": [ # "mappings": [
# { "pattern": "mc\\.example\\.com", "destination": "127.0.0.1:25566", "proxyProtocol": true } # { "pattern": "mc\\.example\\.com", "destination": "127.0.0.1:25566", "proxyProtocol": true }
# ] # ]
@@ -175,6 +173,9 @@ secrets. The base image and build flags live in `.ko.yaml`.
| `maxParkedStreams` | `256` | Cap on simultaneously hung players; `maxParkedBytes` (default `maxParkedStreams × 2 × streamWindowBytes`) caps what they retain. Past either, the oldest are dropped. | | `maxParkedStreams` | `256` | Cap on simultaneously hung players; `maxParkedBytes` (default `maxParkedStreams × 2 × streamWindowBytes`) caps what they retain. Past either, the oldest are dropped. |
| `statsIntervalMs` | `0` (off) | Log a periodic line with live/hung stream counts, retained bytes and pattern count. | | `statsIntervalMs` | `0` (off) | Log a periodic line with live/hung stream counts, retained bytes and pattern count. |
| `registrationGraceMs` | `15000` | Keep a closed control session's routes as *orphaned* for this long, holding players that arrive on them instead of refusing them, and replaying their requests once the client re-registers. `0` disables it. | | `registrationGraceMs` | `15000` | Keep a closed control session's routes as *orphaned* for this long, holding players that arrive on them instead of refusing them, and replaying their requests once the client re-registers. `0` disables it. |
| `playerRatePerSec` | `8` | Per-IP token-bucket rate for **player** connections only (Intent ∉ {17, 18}). Unmatched hostnames still consume a token. `0` disables. |
| `playerBurst` | `16` | Token-bucket depth for `playerRatePerSec`. |
| `maxPlayersPerIp` | `64` | Concurrent player sockets (pending + live + parked) from one IP. `0` disables. Intent 17 is never counted. |
### Client (`client/config.example.json`) ### Client (`client/config.example.json`)
@@ -182,7 +183,7 @@ secrets. The base image and build flags live in `.ko.yaml`.
|------------------|--------------------|---------| |------------------|--------------------|---------|
| `server` | *(required)* | Hub `host:port`. | | `server` | *(required)* | Hub `host:port`. |
| `psk` | *(required)* | Shared secret; must match the hub. | | `psk` | *(required)* | Shared secret; must match the hub. |
| `maxConn` | `1` (clamped 18) | Max worker connections in the pool. | | `maxTunnels` | `256` (clamped 14096) | Max concurrent player tunnels. The retired `maxConn` field is ignored. |
| `pingIntervalMs` | `20000` (min 1000) | Heartbeat interval for the control session and every worker conn. A session with no reply for `3×` this is dropped and re-established. | | `pingIntervalMs` | `20000` (min 1000) | Heartbeat interval for the control session and every worker conn. A session with no reply for `3×` this is dropped and re-established. |
| `maxBandwidth` | *(unlimited)* | Caps what the client uploads to the hub, summed over every player — the direction carrying the game server's output, and the one a home uplink runs out of first. `"20mbps"`, `"512kbps"`, `"2MB/s"`, or a bare number of bytes/sec. **Bit units are decimal (`20mbps` = 20,000,000 bit/s); byte units are binary (`2MB/s` = 2 MiB/s).** Set it slightly below your real upload speed — framing and TCP/IP overhead are not counted. The budget is shared fairly across players, so one person loading chunks cannot time the others out. | | `maxBandwidth` | *(unlimited)* | Caps what the client uploads to the hub, summed over every player — the direction carrying the game server's output, and the one a home uplink runs out of first. `"20mbps"`, `"512kbps"`, `"2MB/s"`, or a bare number of bytes/sec. **Bit units are decimal (`20mbps` = 20,000,000 bit/s); byte units are binary (`2MB/s` = 2 MiB/s).** Set it slightly below your real upload speed — framing and TCP/IP overhead are not counted. The budget is shared fairly across players, so one person loading chunks cannot time the others out. |
| `streamWindowBytes` | `262144` | Advertised per-stream receive window, clamped to [32 KiB, 8 MiB]. | | `streamWindowBytes` | `262144` | Advertised per-stream receive window, clamped to [32 KiB, 8 MiB]. |
@@ -217,20 +218,23 @@ go test ./e2e/... -v
The e2e suite covers: a full player round-trip with verbatim handshake The e2e suite covers: a full player round-trip with verbatim handshake
forwarding and case-insensitive matching, regex wildcard pattern routing, forwarding and case-insensitive matching, regex wildcard pattern routing,
multi-megabyte transfers, concurrent multi-megabyte transfers, concurrent players each on their own worker
streams spreading across multiple worker connections, HAProxy v2 source-address connection, HAProxy v2 source-address
propagation, Velocity modern-forwarding interception (signed player-info propagation, Velocity modern-forwarding interception (signed player-info
handoff to a mock Paper backend), player- and destination-initiated disconnect propagation, wrong-PSK handoff to a mock Paper backend), player- and destination-initiated disconnect
rejection, dropping of unmatched hostnames, stream isolation under a slow propagation, wrong-PSK rejection, dropping of unmatched hostnames, isolation
player and under a slow destination (no head-of-line blocking), rejection under a slow player and under a slow destination (no head-of-line blocking),
of pre-flow-control peers, and stream resumption — a tunnel hard-reset rejection of pre-flow-control peers, and stream resumption — a tunnel
mid-transfer with the player connection held open, asserting the byte stream hard-reset mid-transfer with the player connection held open, asserting the
neither gains nor loses a byte, across concurrent streams, plus grace expiry and byte stream neither gains nor loses a byte, across concurrent players, plus
the resume-disabled path, and control-outage handling — a player arriving while grace expiry and the resume-disabled path, and control-outage handling — a
the client's control session is down is held and then served once it player arriving while the client's control session is down is held and then
re-registers, with the grace-disabled and grace-expired paths pinned too. The Go and Java crypto layers are served once it re-registers, with the grace-disabled and grace-expired paths
independently pinned to the same SHA3-224 test vector so they cannot silently pinned too, and the per-IP player limiter — burst overflow and
drift apart. `maxPlayersPerIp` drop extras before they become pending, Intent 17 is never
counted, and both knobs at `0` restore the unlimited path. The Go and Java
crypto layers are independently pinned to the same SHA3-224 test vector so they
cannot silently drift apart.
## Design notes & limitations ## Design notes & limitations
@@ -238,29 +242,28 @@ drift apart.
ChaCha20-encrypted (no AEAD tag) to minimize overhead. This protects against ChaCha20-encrypted (no AEAD tag) to minimize overhead. This protects against
casual sniffing, not a determined active attacker (see the note at the top of casual sniffing, not a determined active attacker (see the note at the top of
[PROTOCOL.md](PROTOCOL.md) and [docs/architecture.md](docs/architecture.md) §8). [PROTOCOL.md](PROTOCOL.md) and [docs/architecture.md](docs/architecture.md) §8).
* **Per-stream flow control.** Each stream has credit-based windows in both * **Per-tunnel flow control.** Each worker connection has credit-based windows
directions (windows exchanged at session setup, default 256 KiB), so a slow in both directions (windows exchanged at session setup, default 256 KiB), so
player or slow destination jams only its own stream at a bounded buffer — no a slow player or slow destination jams only its own tunnel at a bounded
application-level head-of-line blocking between streams. What remains is buffer. There is no application-level head-of-line blocking: each player
TCP-level HOL (packet loss stalls a whole worker connection briefly); owns a TCP connection.
raising `maxConn` spreads that.
* **Liveness is explicit.** Every session heartbeats, every socket write is * **Liveness is explicit.** Every session heartbeats, every socket write is
bounded, and session establishment has a deadline. A path that dies silently — bounded, and session establishment has a deadline. A path that dies silently —
no `FIN`, no `RST`, as when a NAT or firewall forgets an established flow — is no `FIN`, no `RST`, as when a NAT or firewall forgets an established flow — is
detected within `3 × pingIntervalMs`, the dead connection is dropped from the detected within `3 × pingIntervalMs`, the dead connection is dropped, and
pool, and service is restored without operator action. TCP keepalive is on as service is restored without operator action. TCP keepalive is on as a
a second line of defence. second line of defence.
* **A control-session reconnect no longer refuses new players.** While a client * **A control-session reconnect no longer refuses new players.** While a client
is reconnecting the hub has no route for it, so arriving players used to be is reconnecting the hub has no route for it, so arriving players used to be
told there is no such server. Those routes are now held briefly and the told there is no such server. Those routes are now held briefly and the
players with them, then served once the client re-registers. players with them, then served once the client re-registers.
* **A dropped tunnel no longer drops the players.** A worker connection is only * **A dropped tunnel no longer drops the players.** A worker connection is only
the middle leg of the streams it carries; when it dies both terminal sockets the middle leg of the player it carries; when it dies both terminal sockets
are usually still healthy. The hub now hangs those players while the client are usually still healthy. The hub now hangs that player while the client
reattaches their streams over a fresh connection, replaying byte-exactly from reattaches the tunnel over a fresh connection, replaying byte-exactly from
the offset the peer reports, so a conntrack expiry costs a stall rather than the offset the peer reports, so a conntrack expiry costs a stall rather than
disconnecting everyone on that connection. Negotiated, and `streamResume: a disconnect. Negotiated, and `streamResume: false` on either side restores
false` on either side restores the old behaviour. the old behaviour.
* **Single hub event loop.** The hub deploys one Vert.x verticle, so all state * **Single hub event loop.** The hub deploys one Vert.x verticle, so all state
is confined to one event loop (no locking). Throughput is bounded by one core; is confined to one event loop (no locking). Throughput is bounded by one core;
ample for hundreds of players, not designed for tens of thousands. ample for hundreds of players, not designed for tens of thousands.
+41 -31
View File
@@ -14,8 +14,8 @@ import (
"github.com/iceBear67/redapricot/client/wire" "github.com/iceBear67/redapricot/client/wire"
) )
// Client is a redapricot client: it holds a control session with the hub and a // Client is a redapricot client: it holds a control session with the hub and
// pool of worker connections used to serve player streams. // dials one worker connection per player.
type Client struct { type Client struct {
cfg *Config cfg *Config
pskBytes []byte pskBytes []byte
@@ -25,7 +25,7 @@ type Client struct {
mappings map[string]Mapping // normalized pattern -> mapping mappings map[string]Mapping // normalized pattern -> mapping
pool *WorkerPool pool *WorkerPool
// ctx/cancel own every pool dial: Close cancels it so an in-flight dial // ctx/cancel own every worker dial: Close cancels it so an in-flight dial
// aborts instead of holding a goroutine for the whole handshake timeout. // aborts instead of holding a goroutine for the whole handshake timeout.
// The control path uses the caller's context from Start, which is the same // The control path uses the caller's context from Start, which is the same
// shutdown signal by convention. // shutdown signal by convention.
@@ -34,10 +34,10 @@ type Client struct {
// closing is set by Close; a conn-loss teardown checks it and closes // closing is set by Close; a conn-loss teardown checks it and closes
// streams outright rather than parking them for a reattach that is never // streams outright rather than parking them for a reattach that is never
// coming. Allocate also consults it through the pool's own flag. // coming. Dial also consults it through the pool's own flag.
closing atomic.Bool closing atomic.Bool
streamWnd int // our advertised per-stream receive window (bytes) streamWnd int // our advertised per-connection receive window (bytes)
shaper *Shaper // caps aggregate egress to the hub; nil when unlimited shaper *Shaper // caps aggregate egress to the hub; nil when unlimited
chunk int // DATA payload cap; shrinks below DataChunkSize at low rates chunk int // DATA payload cap; shrinks below DataChunkSize at low rates
@@ -78,10 +78,20 @@ func New(cfg *Config) *Client {
log.Printf("egress shaped to %d B/s (burst %d B, chunk %d B)", log.Printf("egress shaped to %d B/s (burst %d B, chunk %d B)",
bps, int64(c.shaper.burst), c.shaper.chunk) bps, int64(c.shaper.burst), c.shaper.chunk)
} }
c.pool = newWorkerPool(c, cfg.MaxConn) c.pool = newWorkerPool(c, clampMaxTunnels(cfg.MaxTunnels))
return c return c
} }
func clampMaxTunnels(n int) int {
if n < 1 {
return DefaultMaxTunnels
}
if n > MaxMaxTunnels {
return MaxMaxTunnels
}
return n
}
func clampWindow(w int) int { func clampWindow(w int) int {
if w <= 0 { if w <= 0 {
return DefaultStreamWindow return DefaultStreamWindow
@@ -99,8 +109,8 @@ func clampWindow(w int) int {
// was negotiated during establishment. // was negotiated during establishment.
type session struct { type session struct {
fc *wire.FramedConn fc *wire.FramedConn
peerWnd int // hub's advertised per-stream receive window peerWnd int // hub's advertised per-connection receive window
heartbeat bool // hub accepted mux-level PING/PONG on worker conns heartbeat bool // hub accepted connection-level PING/PONG on worker conns
resume bool // hub accepted stream resumption (§7.5) resume bool // hub accepted stream resumption (§7.5)
// hubGrace is how long the hub will hang a parked player, as advertised in // hubGrace is how long the hub will hang a parked player, as advertised in
// SessionReady. Zero when resumption was not negotiated. // SessionReady. Zero when resumption was not negotiated.
@@ -108,7 +118,7 @@ type session struct {
} }
// dialSession opens a TCP connection, performs the Intent-17 handshake, the // dialSession opens a TCP connection, performs the Intent-17 handshake, the
// Phase-A rekey, and reads SessionReady. Per-stream flow control is mandatory: // Phase-A rekey, and reads SessionReady. Per-connection flow control is mandatory:
// a hub that does not echo the STREAM_FC flag is rejected. // a hub that does not echo the STREAM_FC flag is rejected.
// //
// The whole exchange is bounded by HandshakeTimeout. A hub that accepts the // The whole exchange is bounded by HandshakeTimeout. A hub that accepts the
@@ -116,7 +126,7 @@ type session struct {
// behalf) must fail fast rather than park the caller forever. // behalf) must fail fast rather than park the caller forever.
// //
// ctx bounds the dial: the control path passes the caller's context so a // ctx bounds the dial: the control path passes the caller's context so a
// shutdown mid-handshake aborts the attempt, and the pool passes the client's // shutdown mid-handshake aborts the attempt, and worker dials pass the client's
// own context so Close cancels in-flight dials. After the dial, a cancelled // own context so Close cancels in-flight dials. After the dial, a cancelled
// ctx keeps aborting by closing the conn underneath the deadline-bounded // ctx keeps aborting by closing the conn underneath the deadline-bounded
// handshake I/O. // handshake I/O.
@@ -158,7 +168,7 @@ func (c *Client) dialSession(ctx context.Context, magic byte) (sess *session, er
) )
// 3. Rekey frame (Phase A), including the mandatory feature flags and our // 3. Rekey frame (Phase A), including the mandatory feature flags and our
// per-stream receive window. // per-connection receive window.
rnd := make([]byte, 16) rnd := make([]byte, 16)
if _, err := crand.Read(rnd); err != nil { if _, err := crand.Read(rnd); err != nil {
return nil, err return nil, err
@@ -186,7 +196,7 @@ func (c *Client) dialSession(ctx context.Context, magic byte) (sess *session, er
) )
// 5. SessionReady: the type byte followed by the hub's accepted flags and // 5. SessionReady: the type byte followed by the hub's accepted flags and
// its per-stream receive window. Both are required. // its per-connection receive window. Both are required.
payload, err := fc.ReadFrame() payload, err := fc.ReadFrame()
if err != nil { if err != nil {
return nil, err return nil, err
@@ -198,7 +208,7 @@ func (c *Client) dialSession(ctx context.Context, magic byte) (sess *session, er
flags, ferr := r.VarInt() flags, ferr := r.VarInt()
hubWnd, werr := r.VarInt() hubWnd, werr := r.VarInt()
if ferr != nil || werr != nil || flags&FlagStreamFC == 0 || hubWnd <= 0 { if ferr != nil || werr != nil || flags&FlagStreamFC == 0 || hubWnd <= 0 {
return nil, fmt.Errorf("hub did not accept per-stream flow control (unsupported hub version?)") return nil, fmt.Errorf("hub did not accept per-connection flow control (unsupported hub version?)")
} }
if hubWnd > MaxStreamWindow { if hubWnd > MaxStreamWindow {
hubWnd = MaxStreamWindow hubWnd = MaxStreamWindow
@@ -398,8 +408,8 @@ func (c *Client) pingLoop(ctx context.Context, ctrl *ctrlSession) {
} }
} }
// handleControlRequest reacts to a matched player: allocate a worker stream, // handleControlRequest reacts to a matched player: dial a dedicated worker
// SYN it, and bridge it to the mapped destination. // conn, SYN it, and bridge it to the mapped destination.
func (c *Client) handleControlRequest(cid []byte, pattern, ip string, port int) { func (c *Client) handleControlRequest(cid []byte, pattern, ip string, port int) {
mapping, ok := c.mappings[NormalizeAddress(pattern)] mapping, ok := c.mappings[NormalizeAddress(pattern)]
if !ok { if !ok {
@@ -408,37 +418,37 @@ func (c *Client) handleControlRequest(cid []byte, pattern, ip string, port int)
} }
log.Printf("player %s:%d joined via pattern %q -> %s", ip, port, pattern, mapping.Destination) log.Printf("player %s:%d joined via pattern %q -> %s", ip, port, pattern, mapping.Destination)
// Allocate and publish must agree on a live conn: Allocate hands out a // Dial and attach must agree on a live conn: Dial hands out a conn that can
// (conn, sid) pair that can die before we register on it, which would strand // die before we attach on it, which would strand the stream on a conn
// the stream in a map nothing iterates. registerStream reports that, and we // nothing iterates. attach reports that, and we simply dial another.
// simply pick another conn.
var st *Stream var st *Stream
var lg *leg var wc *WorkerConn
for attempt := 0; attempt < allocateAttempts; attempt++ { for attempt := 0; attempt < dialAttempts; attempt++ {
wc, sid, err := c.pool.Allocate() var err error
wc, err = c.pool.Dial()
if err != nil { if err != nil {
log.Printf("worker allocate failed: %v", err) log.Printf("worker dial failed: %v", err)
return return
} }
st = newStream(c, wc, sid, cid, mapping, ip, port) st = newStream(c, wc, cid, mapping, ip, port)
// Register before SYN so inbound DATA can never race ahead of the table, // Attach before SYN so inbound DATA can never race ahead of the binding,
// and start the pumps before the (bounded) SYN write so a failed or slow // and start the pumps before the (bounded) SYN write so a failed or slow
// SYN cannot strand a stream that nothing would ever tear down. // SYN cannot strand a stream that nothing would ever tear down.
if wc.registerStream(sid, st) { if wc.attach(st) {
lg = st.conn()
break break
} }
_ = wc.fc.Close()
st = nil st = nil
} }
if st == nil { if st == nil {
log.Printf("worker allocate failed: no live conn after %d attempts", allocateAttempts) log.Printf("worker dial failed: no live conn after %d attempts", dialAttempts)
return return
} }
go st.writeLoop() go st.writeLoop()
go st.run() go st.run()
if err := lg.wc.sendSyn(lg.sid, cid); err != nil { if err := wc.sendSyn(cid); err != nil {
log.Printf("stream %s: SYN failed: %v", lg, err) log.Printf("stream %s: SYN failed: %v", st.name(), err)
st.teardown(false) st.teardown(false)
} }
} }
@@ -452,7 +462,7 @@ func (c *Client) WorkerConnCount() int { return c.pool.count() }
// open. // open.
func (c *Client) Close() { func (c *Client) Close() {
c.closing.Store(true) c.closing.Store(true)
c.cancel() // aborts in-flight pool dials c.cancel() // aborts in-flight worker dials
c.mu.Lock() c.mu.Lock()
fc := c.ctrl fc := c.ctrl
c.mu.Unlock() c.mu.Unlock()
+1 -1
View File
@@ -1,7 +1,7 @@
{ {
"server": "hub.example.com:25565", "server": "hub.example.com:25565",
"psk": "change-me-to-a-long-random-passphrase", "psk": "change-me-to-a-long-random-passphrase",
"maxConn": 4, "maxTunnels": 256,
"pingIntervalMs": 20000, "pingIntervalMs": 20000,
"streamWindowBytes": 262144, "streamWindowBytes": 262144,
"maxBandwidth": "", "maxBandwidth": "",
+41 -34
View File
@@ -43,19 +43,15 @@ const (
MuxWnd = 0x04 MuxWnd = 0x04
MuxPing = 0x05 MuxPing = 0x05
MuxPong = 0x06 MuxPong = 0x06
// MuxResume reattaches a parked stream to this conn (CID + our accepted // MuxResume reattaches a parked player to this conn (CID + our accepted
// offset); MuxResumeAck carries the hub's accepted offset and a fresh CID. // offset); MuxResumeAck carries the hub's accepted offset and a fresh CID.
MuxResume = 0x07 MuxResume = 0x07
MuxResumeAck = 0x08 MuxResumeAck = 0x08
// MuxCtlSid is the reserved stream id carrying connection-scoped mux frames
// (PING/PONG). Real streams are numbered from 1.
MuxCtlSid = 0
// RST reason codes (optional trailing byte; absence means "unspecified"). // RST reason codes (optional trailing byte; absence means "unspecified").
// Distinguishing them matters for resume: an unknown stream is terminal, // Distinguishing them matters for resume: an unknown stream is terminal,
// while "already bound" means the hub has the stream on another conn — the // while "already bound" means the hub has the player on another conn — the
// reattach retries until that bind dies and the hub re-parks the stream. // reattach retries until that bind dies and the hub re-parks the player.
RstUnspecified = 0x00 RstUnspecified = 0x00
RstUnknownStream = 0x01 RstUnknownStream = 0x01
RstAlreadyBound = 0x02 RstAlreadyBound = 0x02
@@ -65,33 +61,33 @@ const (
FrameError = 0x7F FrameError = 0x7F
// SaturationThreshold caps how many streams share one worker conn once the // DefaultMaxTunnels / MaxMaxTunnels bound concurrent 1:1 worker conns.
// pool has grown to maxConn. Below maxConn the pool grows first (§7.1), so a // The old mux-era maxConn cap of 8 would silently become "8 players".
// single connection is never a shared point of failure for every player. DefaultMaxTunnels = 256
SaturationThreshold = 8 MaxMaxTunnels = 4096
// Session-establishment feature flags (trailing VarInt on the Rekey message). // Session-establishment feature flags (trailing VarInt on the Rekey message).
FlagStreamFC = 0x01 FlagStreamFC = 0x01
// FlagWorkerHeartbeat enables mux-level PING/PONG on worker conns. Without // FlagWorkerHeartbeat enables connection-level PING/PONG on worker conns.
// it a worker conn whose path is silently blackholed (NAT/conntrack drop, // Without it a worker conn whose path is silently blackholed (NAT/conntrack
// firewall) is never detected: the read loop parks forever, the dead conn // drop, firewall) is never detected: the read loop parks forever and that
// stays in the pool, and no player can be served until the client restarts. // player is stuck until the client restarts.
FlagWorkerHeartbeat = 0x02 FlagWorkerHeartbeat = 0x02
// FlagStreamResume enables stream resumption (PROTOCOL.md §7.5): a worker // FlagStreamResume enables stream resumption (PROTOCOL.md §7.5): a worker
// conn drop parks its streams instead of killing them, the hub hangs the // conn drop parks the player instead of killing them, the hub hangs the
// player sockets, and the client reattaches each stream byte-exactly over a // player socket, and the client reattaches byte-exactly over a fresh conn.
// fresh conn. Negotiated, so either side may decline and get today's // Negotiated, so either side may decline and get today's behaviour
// behaviour (immediate teardown) unchanged. // (immediate teardown) unchanged.
FlagStreamResume = 0x04 FlagStreamResume = 0x04
// Per-stream flow-control window bounds (bytes). The advertised window is the // Per-connection flow-control window bounds (bytes). The advertised window
// receiver's promise of how much un-credited DATA it will buffer per stream. // is the receiver's promise of how much un-credited DATA it will buffer.
DefaultStreamWindow = 256 * 1024 DefaultStreamWindow = 256 * 1024
MinStreamWindow = 32 * 1024 MinStreamWindow = 32 * 1024
MaxStreamWindow = 8 << 20 MaxStreamWindow = 8 << 20
// DataChunkSize caps a single DATA frame's payload so no stream monopolizes // DataChunkSize caps a single DATA frame's payload so one write cannot
// the shared worker connection for long. // occupy the link for a full 1-MiB frame.
DataChunkSize = 32 * 1024 DataChunkSize = 32 * 1024
) )
@@ -211,14 +207,20 @@ type Mapping struct {
// Config is the client configuration (PROTOCOL.md §9.2). // Config is the client configuration (PROTOCOL.md §9.2).
type Config struct { type Config struct {
Server string `json:"server"` Server string `json:"server"`
PSK string `json:"psk"` PSK string `json:"psk"`
MaxConn int `json:"maxConn"` // MaxTunnels is the max concurrent 1:1 worker connections (PROTOCOL.md §7.1).
PingIntervalMs int `json:"pingIntervalMs"` // 0 means the default. Clamped to [1, 4096].
StreamWindowBytes int `json:"streamWindowBytes"` // per-stream receive window; 0 = default MaxTunnels int `json:"maxTunnels"`
// MaxConn is the retired mux-era pool size. Ignored when loading a file:
// honouring a value of 4 as a player cap would silently break existing
// configs. Tests that construct a Config should set MaxTunnels instead.
MaxConn int `json:"maxConn"`
PingIntervalMs int `json:"pingIntervalMs"`
StreamWindowBytes int `json:"streamWindowBytes"` // per-connection receive window; 0 = default
// MaxBandwidth caps what the client sends to the hub, aggregated over every // MaxBandwidth caps what the client sends to the hub, aggregated over every
// stream on every worker conn — the direction that carries the game server's // worker conn — the direction that carries the game server's output to the
// output to the players, and the one a residential uplink runs out of first. // players, and the one a residential uplink runs out of first.
// Empty means no limit. See parseBandwidth for the accepted syntax. // Empty means no limit. See parseBandwidth for the accepted syntax.
MaxBandwidth string `json:"maxBandwidth"` MaxBandwidth string `json:"maxBandwidth"`
// StreamResume enables stream resumption (PROTOCOL.md §7.5). A pointer so an // StreamResume enables stream resumption (PROTOCOL.md §7.5). A pointer so an
@@ -310,11 +312,16 @@ func LoadConfig(path string) (*Config, error) {
if c.PSK == "" { if c.PSK == "" {
return nil, fmt.Errorf("psk is required") return nil, fmt.Errorf("psk is required")
} }
if c.MaxConn < 1 { if c.MaxConn != 0 && c.MaxTunnels == 0 {
c.MaxConn = 1 // Old mux pool size. Must not become the player cap: a previously-working
// maxConn: 4 would admit only four players.
fmt.Fprintf(os.Stderr, "redapricot-client: maxConn is ignored (it was the mux pool size); use maxTunnels (default %d)\n", DefaultMaxTunnels)
} }
if c.MaxConn > 8 { if c.MaxTunnels < 1 {
c.MaxConn = 8 c.MaxTunnels = DefaultMaxTunnels
}
if c.MaxTunnels > MaxMaxTunnels {
c.MaxTunnels = MaxMaxTunnels
} }
if c.PingIntervalMs <= 0 { if c.PingIntervalMs <= 0 {
c.PingIntervalMs = DefaultPingIntervalMs c.PingIntervalMs = DefaultPingIntervalMs
+19 -47
View File
@@ -40,24 +40,25 @@ func stalledHub(t *testing.T) string {
return ln.Addr().String() return ln.Addr().String()
} }
// TestAllocateDoesNotWedgePoolOnStalledHub is the regression guard for the // TestDialDoesNotWedgeOnStalledHub is the regression guard for the worst
// worst failure mode found in the stability audit: Allocate used to dial while // failure mode found in the stability audit: Dial used to share a single
// holding the pool mutex, and the handshake read had no deadline. One // in-flight handshake, and the handshake read had no deadline. One
// unresponsive hub therefore parked every present and future allocation // unresponsive hub therefore parked every present and future allocation
// forever, so no player could be served again until the process restarted. // forever. 1:1 dials independently, but each must still fail on its own
func TestAllocateDoesNotWedgePoolOnStalledHub(t *testing.T) { // HandshakeTimeout rather than block the other.
func TestDialDoesNotWedgeOnStalledHub(t *testing.T) {
c := New(&Config{ c := New(&Config{
Server: stalledHub(t), Server: stalledHub(t),
PSK: "pool-test", PSK: "pool-test",
MaxConn: 8, MaxTunnels: 8,
PingIntervalMs: 20000, PingIntervalMs: 20000,
Mappings: []Mapping{{Pattern: "mc.local", Destination: "127.0.0.1:1"}}, Mappings: []Mapping{{Pattern: "mc.local", Destination: "127.0.0.1:1"}},
}) })
done := make(chan error, 2) done := make(chan error, 2)
go func() { _, _, err := c.pool.Allocate(); done <- err }() go func() { _, err := c.pool.Dial(); done <- err }()
time.Sleep(200 * time.Millisecond) // let the first caller get into the dial time.Sleep(200 * time.Millisecond) // let the first caller get into the dial
go func() { _, _, err := c.pool.Allocate(); done <- err }() go func() { _, err := c.pool.Dial(); done <- err }()
// Both must give up on their own; neither may be stuck behind the other. // Both must give up on their own; neither may be stuck behind the other.
limit := time.After(HandshakeTimeout + 15*time.Second) limit := time.After(HandshakeTimeout + 15*time.Second)
@@ -65,50 +66,21 @@ func TestAllocateDoesNotWedgePoolOnStalledHub(t *testing.T) {
select { select {
case err := <-done: case err := <-done:
if err == nil { if err == nil {
t.Fatal("Allocate succeeded against a hub that never answers") t.Fatal("Dial succeeded against a hub that never answers")
} }
case <-limit: case <-limit:
t.Fatalf("Allocate #%d never returned: the pool is wedged again", i+1) t.Fatalf("Dial #%d never returned: a stalled hub wedged the other caller", i+1)
} }
} }
} }
// TestAllocateSpreadsAcrossConns guards the allocation rule: the pool must fan // TestDialRespectsMaxTunnels pins the concurrency cap: once live+dialing
// out to maxConn before stacking streams, so a single worker connection is // equals maxTunnels, further Dial calls fail immediately rather than stacking.
// never the shared point of failure for every player. Seven players used to all func TestDialRespectsMaxTunnels(t *testing.T) {
// land on one conn, which meant one dead TCP connection dropped everybody. p := newWorkerPool(&Client{}, 2)
func TestAllocateSpreadsAcrossConns(t *testing.T) { p.conns[&WorkerConn{id: 1}] = struct{}{}
p := &WorkerPool{maxConn: 4} p.conns[&WorkerConn{id: 2}] = struct{}{}
p.cond = sync.NewCond(&p.mu) if _, err := p.Dial(); err != errTooManyTunnels {
t.Fatalf("Dial at cap: got %v, want %v", err, errTooManyTunnels)
newConn := func() *WorkerConn {
return &WorkerConn{pool: p, streams: make(map[int]*Stream), nextSid: 1, done: make(chan struct{})}
}
p.conns = []*WorkerConn{newConn()}
if !p.conns[0].registerStream(1, &Stream{}) {
t.Fatal("registerStream refused on a live conn")
}
// One conn holding a stream, pool below maxConn: growth is warranted.
_, bestCount := p.leastLoadedLocked()
if bestCount < StreamsBeforeGrowing {
t.Fatalf("a conn with %d stream(s) should trigger growth", bestCount)
}
// Once the pool is at maxConn, growth stops and streams stack on the
// least-loaded conn instead.
for len(p.conns) < p.maxConn {
p.conns = append(p.conns, newConn())
}
best, bestCount := p.leastLoadedLocked()
if bestCount != 0 {
t.Fatalf("expected an empty conn to be least-loaded, got %d streams", bestCount)
}
p.maybeGrowLocked(bestCount)
if p.dialing != 0 {
t.Fatalf("pool dialed past maxConn=%d", p.maxConn)
}
if best == nil {
t.Fatal("no conn selected")
} }
} }
+41 -41
View File
@@ -10,11 +10,10 @@ import (
// Stream resumption (PROTOCOL.md §7.5). // Stream resumption (PROTOCOL.md §7.5).
// //
// A worker conn carries many players but is only the middle leg of each: when // A worker conn is only the middle leg of the player it carries: when it dies,
// it dies, both terminal sockets are usually still perfectly healthy. Tearing // both terminal sockets are usually still perfectly healthy. Tearing the
// the streams down therefore throws away working connections because a // tunnel down therefore throws away working connections because a replaceable
// replaceable transport failed — one conntrack expiry disconnects everyone on // transport failed.
// that conn.
// //
// Instead the stream parks: the destination socket stays open, the hub hangs the // Instead the stream parks: the destination socket stays open, the hub hangs the
// player socket, and the client reattaches over a fresh conn. Correctness rests // player socket, and the client reattaches over a fresh conn. Correctness rests
@@ -104,7 +103,7 @@ func (s *Stream) resumeLoop(grace time.Duration) {
return return
} }
// errResumeRaced falls through to the retry below. The hub has this // errResumeRaced falls through to the retry below. The hub has this
// stream bound to a conn that is not ours — a half-open conn whose // player bound to a conn that is not ours — a half-open conn whose
// death the hub has not yet learned, or a bind left behind by a racing // death the hub has not yet learned, or a bind left behind by a racing
// attempt on a now-dead conn. There is no other live attempt: park is // attempt on a now-dead conn. There is no other live attempt: park is
// the only resumeLoop starter and it refuses to double-start. Retrying // the only resumeLoop starter and it refuses to double-start. Retrying
@@ -131,10 +130,10 @@ func (s *Stream) resumeLoop(grace time.Duration) {
s.teardown(false) s.teardown(false)
} }
// tryResume performs one reattach attempt: find a live conn, claim a stream id // tryResume performs one reattach attempt: dial a fresh worker conn, send
// on it, send RESUME, and replay from wherever the hub says it got to. // RESUME, and replay from wherever the hub says it got to.
func (s *Stream) tryResume() error { func (s *Stream) tryResume() error {
wc, sid, err := s.allocateForResume() wc, err := s.allocateForResume()
if err != nil { if err != nil {
return err return err
} }
@@ -147,10 +146,10 @@ func (s *Stream) tryResume() error {
s.resumeWait = wait s.resumeWait = wait
s.mu.Unlock() s.mu.Unlock()
msg := wire.NewWriter().U8(MuxResume).VarInt(sid).Bytes(cid). msg := wire.NewWriter().U8(MuxResume).Bytes(cid).
I64(accepted).I64(delivered).Out() I64(accepted).I64(delivered).Out()
if err := wc.fc.WriteFrame(msg); err != nil { if err := wc.fc.WriteFrame(msg); err != nil {
s.abandonAttempt(wc, sid) s.abandonAttempt(wc)
return err return err
} }
@@ -158,47 +157,51 @@ func (s *Stream) tryResume() error {
select { select {
case res = <-wait: case res = <-wait:
case <-s.done: case <-s.done:
// Torn down while waiting. teardown only deregisters the leg the stream // Torn down while waiting. teardown only detaches the conn the stream
// was bound to, which is not this one, so the claim made above has to be // was bound to, which is not this one, so the claim made above has to be
// withdrawn here or it stays in the new conn's table forever. // withdrawn here or the new conn stays bound forever.
s.abandonAttempt(wc, sid) s.abandonAttempt(wc)
return errResumeRefused return errResumeRefused
case <-time.After(ResumeAckTimeout): case <-time.After(ResumeAckTimeout):
s.abandonAttempt(wc, sid) s.abandonAttempt(wc)
return errResumeTimeout return errResumeTimeout
} }
if res.err != nil { if res.err != nil {
s.abandonAttempt(wc, sid) s.abandonAttempt(wc)
return res.err return res.err
} }
return s.completeResume(wc, sid, res) return s.completeResume(wc, res)
} }
// allocateForResume picks a live conn that will honour a reattach. // allocateForResume dials a dedicated worker conn that will honour a reattach.
func (s *Stream) allocateForResume() (*WorkerConn, int, error) { // 1:1: this must never land on someone else's tunnel.
for attempt := 0; attempt < allocateAttempts; attempt++ { func (s *Stream) allocateForResume() (*WorkerConn, error) {
wc, sid, err := s.client.pool.Allocate() for attempt := 0; attempt < dialAttempts; attempt++ {
wc, err := s.client.pool.Dial()
if err != nil { if err != nil {
return nil, 0, err return nil, err
} }
// Re-checked per conn, not assumed from the dead one: this may be a // Re-checked per conn, not assumed from the dead one: this may be a
// different or restarted hub. Sending RESUME to a hub that does not know // different or restarted hub. Sending RESUME to a hub that does not know
// the frame type would hang the player for the rest of the grace waiting // the frame type would hang the player for the rest of the grace waiting
// for an answer that is never coming. // for an answer that is never coming.
if !wc.resume { if !wc.resume {
return nil, 0, errResumeNoResume _ = wc.fc.Close()
return nil, errResumeNoResume
} }
if wc.registerStream(sid, s) { if wc.attach(s) {
return wc, sid, nil return wc, nil
} }
_ = wc.fc.Close()
} }
return nil, 0, errResumeRefused return nil, errResumeRefused
} }
// abandonAttempt withdraws a failed attempt from the conn it was made on, so a // abandonAttempt withdraws a failed attempt from the conn it was made on and
// retry can never leave two RESUMEs outstanding for one stream. // closes that conn — 1:1, it exists only for this attempt.
func (s *Stream) abandonAttempt(wc *WorkerConn, sid int) { func (s *Stream) abandonAttempt(wc *WorkerConn) {
wc.removeStream(sid) wc.detach()
_ = wc.fc.Close()
s.mu.Lock() s.mu.Lock()
s.resumeWait = nil s.resumeWait = nil
s.mu.Unlock() s.mu.Unlock()
@@ -206,7 +209,7 @@ func (s *Stream) abandonAttempt(wc *WorkerConn, sid int) {
// completeResume rebinds the stream to its new conn and replays what the hub is // completeResume rebinds the stream to its new conn and replays what the hub is
// missing, holding sendMu throughout so live traffic cannot overtake the replay. // missing, holding sendMu throughout so live traffic cannot overtake the replay.
func (s *Stream) completeResume(wc *WorkerConn, sid int, res resumeResult) error { func (s *Stream) completeResume(wc *WorkerConn, res resumeResult) error {
s.sendMu.Lock() s.sendMu.Lock()
// Delivery is a strictly stronger fact than credit — the hub only credits what // Delivery is a strictly stronger fact than credit — the hub only credits what
// it has delivered — so the reported offset can be adopted wholesale. Doing so // it has delivered — so the reported offset can be adopted wholesale. Doing so
@@ -218,7 +221,7 @@ func (s *Stream) completeResume(wc *WorkerConn, sid int, res resumeResult) error
replay := s.un.from(res.accepted) replay := s.un.from(res.accepted)
if replay == nil { if replay == nil {
s.sendMu.Unlock() s.sendMu.Unlock()
s.abandonAttempt(wc, sid) s.abandonAttempt(wc)
return errResumeTooOld return errResumeTooOld
} }
// Three offsets, three jobs, and conflating any two of them breaks something // Three offsets, three jobs, and conflating any two of them breaks something
@@ -235,10 +238,8 @@ func (s *Stream) completeResume(wc *WorkerConn, sid int, res resumeResult) error
outstanding := s.un.length() outstanding := s.un.length()
replayed := s.un.end() - res.accepted replayed := s.un.end() - res.accepted
// Publish the new binding before any frame goes out on it, and as one value: // Publish the new binding before any frame goes out on it.
// stream ids restart at 1 per conn, so a half-updated pair would address a s.wc.Store(wc)
// different player's stream.
s.leg.Store(&leg{wc: wc, sid: sid})
s.mu.Lock() s.mu.Lock()
// Restated, not patched. The window is a delta ledger and the outage tore a // Restated, not patched. The window is a delta ledger and the outage tore a
@@ -251,7 +252,6 @@ func (s *Stream) completeResume(wc *WorkerConn, sid int, res resumeResult) error
// Symmetrically, our own pending credit is discarded rather than flushed: // Symmetrically, our own pending credit is discarded rather than flushed:
// the delivered offset we reported already tells the hub everything those // the delivered offset we reported already tells the hub everything those
// deltas would have, and sending both would grant the same bytes twice. // deltas would have, and sending both would grant the same bytes twice.
// Counting resumes from this baseline.
s.consumed = 0 s.consumed = 0
if len(res.cid) == CIDLen { if len(res.cid) == CIDLen {
s.cid = res.cid // fresh capability, so a CID is never reusable twice s.cid = res.cid // fresh capability, so a CID is never reusable twice
@@ -267,9 +267,9 @@ func (s *Stream) completeResume(wc *WorkerConn, sid int, res resumeResult) error
if n > s.client.chunk { if n > s.client.chunk {
n = s.client.chunk n = s.client.chunk
} }
if err := wc.sendData(sid, replay[:n]); err != nil { if err := wc.sendData(replay[:n]); err != nil {
// The conn died mid-replay. The stream is still resumable, but not // The conn died mid-replay. The stream is still resumable, but not
// from this leg — put it back in the parked state before returning // from this conn — put it back in the parked state before returning
// so the conn's teardown takes park()'s already-branch instead of // so the conn's teardown takes park()'s already-branch instead of
// starting a second resumeLoop. The loop we came from keeps // starting a second resumeLoop. The loop we came from keeps
// retrying with the fresh CID, which the hub re-parked alongside // retrying with the fresh CID, which the hub re-parked alongside
@@ -300,11 +300,11 @@ func (s *Stream) completeResume(wc *WorkerConn, sid int, res resumeResult) error
// A destination that closed while we were parked owed the hub a FIN that had // A destination that closed while we were parked owed the hub a FIN that had
// nowhere to go at the time. // nowhere to go at the time.
if owedFin { if owedFin {
wc.sendFin(sid) wc.sendFin()
s.teardown(false) s.teardown(false)
return nil return nil
} }
log.Printf("stream conn%d/sid%d resumed (%d bytes replayed, %d outstanding)", wc.id, sid, replayed, outstanding) log.Printf("stream %s resumed (%d bytes replayed, %d outstanding)", s.name(), replayed, outstanding)
return nil return nil
} }
+16 -18
View File
@@ -57,11 +57,9 @@ type connStats struct {
framesOut atomic.Int64 framesOut atomic.Int64
writeErrs atomic.Int64 writeErrs atomic.Int64
// Round-trip time of the mux heartbeat. The probe already carries a // Round-trip time of the worker heartbeat. The probe already carries a
// timestamp that the peer echoes and both sides currently throw away, so // timestamp that the peer echoes and both sides currently throw away, so
// this measures tunnel latency for no added cost — and it is the best signal // this measures tunnel latency for no added cost.
// available for head-of-line blocking, where one stream's backlog delays
// every other stream sharing the connection.
mu sync.Mutex mu sync.Mutex
rttLast time.Duration rttLast time.Duration
rttMin time.Duration rttMin time.Duration
@@ -137,22 +135,19 @@ func (c *Client) StatsLine() string {
live, parked := 0, 0 live, parked := 0, 0
for _, wc := range conns { for _, wc := range conns {
wc.mu.Lock() st := wc.getStream()
streams := make([]*Stream, 0, len(wc.streams)) bound := 0
for _, s := range wc.streams { if st != nil {
streams = append(streams, s) bound = 1
} live++
wc.mu.Unlock() st.mu.Lock()
live += len(streams) if st.parked {
for _, s := range streams {
s.mu.Lock()
if s.parked {
parked++ parked++
} }
s.mu.Unlock() st.mu.Unlock()
} }
fmt.Fprintf(&b, " | conn%d streams=%d", wc.id, len(streams)) fmt.Fprintf(&b, " | conn%d bound=%d", wc.id, bound)
if cs := wc.stats; cs != nil { if cs := wc.stats; cs != nil {
_, mn, avg, mx := cs.rtt() _, mn, avg, mx := cs.rtt()
fmt.Fprintf(&b, " frames=%d/%d rtt=%s/%s/%s", fmt.Fprintf(&b, " frames=%d/%d rtt=%s/%s/%s",
@@ -162,7 +157,7 @@ func (c *Client) StatsLine() string {
} }
} }
} }
fmt.Fprintf(&b, " | streams=%d parked=%d", live, parked) fmt.Fprintf(&b, " | tunnels=%d parked=%d", live, parked)
return b.String() return b.String()
} }
@@ -189,7 +184,10 @@ func (s *Stream) logSummary() {
func (p *WorkerPool) snapshot() []*WorkerConn { func (p *WorkerPool) snapshot() []*WorkerConn {
p.mu.Lock() p.mu.Lock()
conns := append([]*WorkerConn(nil), p.conns...) conns := make([]*WorkerConn, 0, len(p.conns))
for wc := range p.conns {
conns = append(conns, wc)
}
p.mu.Unlock() p.mu.Unlock()
sort.Slice(conns, func(i, j int) bool { return conns[i].id < conns[j].id }) sort.Slice(conns, func(i, j int) bool { return conns[i].id < conns[j].id })
return conns return conns
+3 -3
View File
@@ -15,9 +15,9 @@ import (
const MaxFrame = 1 << 20 const MaxFrame = 1 << 20
// WriteTimeout bounds a single frame write. A peer that stops reading must not // WriteTimeout bounds a single frame write. A peer that stops reading must not
// be able to park every stream on the connection inside WriteFrame forever: the // be able to park this connection inside WriteFrame forever: the write mutex
// write mutex is held for the whole socket write, so one stalled write would // is held for the whole socket write, so one stalled write would otherwise
// otherwise wedge the entire multiplexed connection. // wedge liveness (PING/PONG) and WND/FIN on this tunnel.
const WriteTimeout = 30 * time.Second const WriteTimeout = 30 * time.Second
var ( var (
+172 -291
View File
@@ -12,166 +12,87 @@ import (
"github.com/iceBear67/redapricot/client/wire" "github.com/iceBear67/redapricot/client/wire"
) )
// errPoolClosed is returned by Allocate after Close: the pool is shutting down // errPoolClosed is returned by Dial after Close: the set is shutting down
// and must not start new dials, so a caller (handleControlRequest, // and must not start new dials, so a caller (handleControlRequest,
// allocateForResume) gives up rather than wait on a cond no one will satisfy. // allocateForResume) gives up rather than waiting on a hub that will never
var errPoolClosed = errors.New("worker pool closed") // be used.
var errPoolClosed = errors.New("worker set closed")
// StreamsBeforeGrowing is how many streams a worker conn may carry before the // errTooManyTunnels is returned when live+in-flight worker conns already
// pool starts opening another one. Set to 1 so the pool fans out to maxConn // equal maxTunnels. The ControlRequest is dropped; the hub closes the player
// under load *before* stacking streams: concentrating every player on a single // when pendingTimeoutMs fires.
// TCP connection makes that connection a shared point of failure, which is var errTooManyTunnels = errors.New("maxTunnels reached")
// exactly how a whole server's worth of players used to drop at once.
const StreamsBeforeGrowing = 1
// allocateAttempts bounds how many times a caller retries Allocate when the // dialAttempts bounds how many times a caller retries Dial when the conn it
// conn it was handed dies before the stream could be registered on it. The race // was handed dies before the stream could be attached to it. The race is
// is narrow and each retry picks a different conn, so a small bound is enough; // narrow; an unbounded loop would spin against a hub that is refusing every
// an unbounded loop would spin against a hub that is refusing every connection. // connection.
const allocateAttempts = 3 const dialAttempts = 3
// WorkerPool manages up to maxConn worker connections and allocates streams // WorkerPool tracks live 1:1 worker connections up to maxTunnels
// using the least-loaded strategy (PROTOCOL.md §7.1). // (PROTOCOL.md §7.1). There is no least-loaded placement and no sharing:
// every player gets its own TCP connection.
type WorkerPool struct { type WorkerPool struct {
client *Client client *Client
maxConn int maxTunnels int
connSeq atomic.Int64 // conn ids, for log correlation connSeq atomic.Int64 // conn ids, for log correlation
mu sync.Mutex mu sync.Mutex
cond *sync.Cond conns map[*WorkerConn]struct{}
conns []*WorkerConn dialing int // dials currently in flight
dialing int // dials currently in flight (foreground + background) closed bool // closeAll ran; no new conns may join
dialGen uint64
dialErr error // most recent dial failure
closed bool // closeAll ran; no new conns may join the pool
} }
func newWorkerPool(c *Client, maxConn int) *WorkerPool { func newWorkerPool(c *Client, maxTunnels int) *WorkerPool {
p := &WorkerPool{client: c, maxConn: maxConn} return &WorkerPool{
p.cond = sync.NewCond(&p.mu) client: c,
return p maxTunnels: maxTunnels,
conns: make(map[*WorkerConn]struct{}),
}
} }
// Allocate returns a worker conn and a fresh stream id to place a new stream on. // Dial opens a dedicated worker conn for one player.
// //
// A dial is never performed while holding p.mu: session establishment involves // A dial is never performed while holding p.mu: session establishment involves
// network I/O, and holding the pool lock across it would park every other // network I/O, and holding the lock across it would park every other player
// player behind one unresponsive hub. When the pool is empty exactly one caller // behind one unresponsive hub. Each caller dials independently; unlike the
// dials and the rest wait on the condition variable; when the pool is merely // old mux pool there is no shared conn to wait for.
// below maxConn, growth happens in the background and the caller is served func (p *WorkerPool) Dial() (*WorkerConn, error) {
// immediately by an existing conn.
func (p *WorkerPool) Allocate() (*WorkerConn, int, error) {
p.mu.Lock() p.mu.Lock()
for {
if p.closed {
// Close won. No new conn may join the pool, so no stream may be
// placed; waiting on the cond could only be satisfied by a dial we
// must not start.
p.mu.Unlock()
return nil, 0, errPoolClosed
}
best, bestCount := p.leastLoadedLocked()
if best != nil {
p.maybeGrowLocked(bestCount)
p.mu.Unlock()
return best, best.newSid(), nil
}
if p.dialing > 0 {
// Someone is already dialing the first conn; wait for it rather
// than piling up redundant connections.
gen := p.dialGen
p.cond.Wait()
if len(p.conns) == 0 && p.dialGen != gen && p.dialErr != nil {
err := p.dialErr
p.mu.Unlock()
return nil, 0, err
}
continue
}
p.dialing++
p.mu.Unlock()
wc, err := p.dialWorker()
p.mu.Lock()
p.dialing--
p.dialGen++
p.dialErr = err
if err != nil {
p.cond.Broadcast()
p.mu.Unlock()
return nil, 0, err
}
if p.closed {
// Close raced this dial: the conn must not enter the pool. Closing
// it here, under p.mu, is a raw socket close — fine, and it makes
// the shutdown atomic with the pool state.
p.cond.Broadcast()
p.mu.Unlock()
_ = wc.fc.Close()
return nil, 0, errPoolClosed
}
p.conns = append(p.conns, wc)
p.cond.Broadcast()
}
}
// leastLoadedLocked returns the worker conn carrying the fewest streams.
func (p *WorkerPool) leastLoadedLocked() (*WorkerConn, int) {
var best *WorkerConn
bestCount := 0
for _, wc := range p.conns {
n := wc.streamCount()
if best == nil || n < bestCount {
best = wc
bestCount = n
}
}
return best, bestCount
}
// maybeGrowLocked opens one more worker conn in the background when the pool is
// below maxConn and the least-loaded conn is already carrying streams. The
// caller does not wait for it: it keeps using the conn it already has, and the
// new one picks up subsequent players.
func (p *WorkerPool) maybeGrowLocked(bestCount int) {
if bestCount < StreamsBeforeGrowing {
return
}
if p.closed { if p.closed {
return // shutdown; do not start dials nobody will join the pool p.mu.Unlock()
return nil, errPoolClosed
} }
if len(p.conns)+p.dialing >= p.maxConn { if len(p.conns)+p.dialing >= p.maxTunnels {
if bestCount > SaturationThreshold { p.mu.Unlock()
log.Printf("worker pool at maxConn=%d with %d streams on the least-loaded conn", p.maxConn, bestCount) return nil, errTooManyTunnels
}
return
} }
p.dialing++ p.dialing++
go func() { p.mu.Unlock()
wc, err := p.dialWorker()
var surplus *WorkerConn wc, err := p.dialWorker()
p.mu.Lock()
p.dialing-- p.mu.Lock()
p.dialGen++ p.dialing--
p.dialErr = err if err != nil {
switch {
case err != nil:
case p.closed:
surplus = wc // Close raced this background dial
case len(p.conns) < p.maxConn:
p.conns = append(p.conns, wc)
default:
surplus = wc // raced with another dial
}
p.cond.Broadcast()
p.mu.Unlock() p.mu.Unlock()
if err != nil { return nil, err
log.Printf("worker pool: background dial failed: %v", err) }
} if p.closed {
if surplus != nil { // Close raced this dial: the conn must not enter the set.
_ = surplus.fc.Close() p.mu.Unlock()
} _ = wc.fc.Close()
}() return nil, errPoolClosed
}
if len(p.conns) >= p.maxTunnels {
p.mu.Unlock()
_ = wc.fc.Close()
return nil, errTooManyTunnels
}
p.conns[wc] = struct{}{}
p.mu.Unlock()
return wc, nil
} }
// dialWorker establishes one worker conn. It must be called without p.mu held. // dialWorker establishes one worker conn. It must be called without p.mu held.
@@ -188,8 +109,6 @@ func (p *WorkerPool) dialWorker() (*WorkerConn, error) {
resume: sess.resume, resume: sess.resume,
grace: p.client.resumeGrace(sess.hubGrace), grace: p.client.resumeGrace(sess.hubGrace),
id: int(p.connSeq.Add(1)), id: int(p.connSeq.Add(1)),
streams: make(map[int]*Stream),
nextSid: 1,
done: make(chan struct{}), done: make(chan struct{}),
} }
if p.client.statsOn() { if p.client.statsOn() {
@@ -200,7 +119,7 @@ func (p *WorkerPool) dialWorker() (*WorkerConn, error) {
if sess.heartbeat { if sess.heartbeat {
go wc.heartbeatLoop(p.client.cfg.pingInterval(), p.client.cfg.heartbeatTimeout()) go wc.heartbeatLoop(p.client.cfg.pingInterval(), p.client.cfg.heartbeatTimeout())
} else { } else {
log.Printf("worker conn: hub does not support the mux heartbeat; " + log.Printf("worker conn: hub does not support the worker heartbeat; " +
"a silently dropped path will only be caught by TCP keepalive") "a silently dropped path will only be caught by TCP keepalive")
} }
log.Printf("opened worker conn (send window %d, recv window %d, heartbeat %v, resume %v)", log.Printf("opened worker conn (send window %d, recv window %d, heartbeat %v, resume %v)",
@@ -217,41 +136,32 @@ func (p *WorkerPool) count() int {
func (p *WorkerPool) remove(wc *WorkerConn) { func (p *WorkerPool) remove(wc *WorkerConn) {
p.mu.Lock() p.mu.Lock()
defer p.mu.Unlock() defer p.mu.Unlock()
for i, c := range p.conns { delete(p.conns, wc)
if c == wc {
p.conns = append(p.conns[:i], p.conns[i+1:]...)
// A waiter parked on an empty pool must re-evaluate: it may now
// need to dial rather than keep waiting for this conn.
p.cond.Broadcast()
return
}
}
} }
func (p *WorkerPool) closeAll() { func (p *WorkerPool) closeAll() {
p.mu.Lock() p.mu.Lock()
p.closed = true p.closed = true
conns := append([]*WorkerConn(nil), p.conns...) conns := make([]*WorkerConn, 0, len(p.conns))
// Wake waiters parked in Allocate: the closed flag they re-check is the for wc := range p.conns {
// only thing that can release them now that no conn will ever join. conns = append(conns, wc)
p.cond.Broadcast() }
p.mu.Unlock() p.mu.Unlock()
for _, wc := range conns { for _, wc := range conns {
_ = wc.fc.Close() _ = wc.fc.Close()
} }
} }
// WorkerConn is one multiplexed worker connection to the hub. // WorkerConn is one 1:1 worker connection to the hub: it carries exactly one
// // player. Only genuinely per-connection state lives here. Client-wide values
// Only genuinely per-connection state lives here. Client-wide values (the // (the shaper, the advertised receive window, the DATA chunk cap) belong to
// shaper, the advertised receive window, the DATA chunk cap) belong to Client: // Client: reading them through a connection pointer would make every such
// reading them through a connection pointer would make every such read a // read a re-parenting hazard once a stream can migrate between conns.
// re-parenting hazard once a stream can migrate between conns.
type WorkerConn struct { type WorkerConn struct {
pool *WorkerPool pool *WorkerPool
fc *wire.FramedConn fc *wire.FramedConn
sendWndInit int // hub's advertised per-stream receive window (our send budget) sendWndInit int // hub's advertised receive window (our send budget)
// resume is whether this conn negotiated stream resumption, and grace how // resume is whether this conn negotiated stream resumption, and grace how
// long a stream parked from it may keep trying to reattach. Both are // long a stream parked from it may keep trying to reattach. Both are
@@ -266,17 +176,16 @@ type WorkerConn struct {
id int // for log correlation only id int // for log correlation only
stats *connStats // nil unless diagnostics are enabled stats *connStats // nil unless diagnostics are enabled
mu sync.Mutex mu sync.Mutex
streams map[int]*Stream stream *Stream
nextSid int closed bool // readLoop has exited; attach must refuse
closed bool // readLoop has exited; registerStream must refuse
} }
// heartbeatLoop proves the worker conn is still carrying frames end to end. TCP // heartbeatLoop proves the worker conn is still carrying frames end to end. TCP
// alone cannot tell us: a middlebox that drops an established flow (conntrack // alone cannot tell us: a middlebox that drops an established flow (conntrack
// expiry, firewall state loss) sends no FIN or RST, so the read loop would park // expiry, firewall state loss) sends no FIN or RST, so the read loop would park
// forever, the dead conn would stay in the pool, and every player routed to it // forever and the player on this conn would silently fail until the process
// would silently fail until the process restarted. // restarted.
func (wc *WorkerConn) heartbeatLoop(interval, timeout time.Duration) { func (wc *WorkerConn) heartbeatLoop(interval, timeout time.Duration) {
ticker := time.NewTicker(interval) ticker := time.NewTicker(interval)
defer ticker.Stop() defer ticker.Stop()
@@ -286,12 +195,11 @@ func (wc *WorkerConn) heartbeatLoop(interval, timeout time.Duration) {
return return
case <-ticker.C: case <-ticker.C:
if silent := time.Since(time.UnixMilli(wc.lastPong.Load())); silent > timeout { if silent := time.Since(time.UnixMilli(wc.lastPong.Load())); silent > timeout {
log.Printf("worker conn silent for %s; dropping it and its %d stream(s)", log.Printf("worker conn silent for %s; dropping it", silent.Round(time.Second))
silent.Round(time.Second), wc.streamCount())
_ = wc.fc.Close() // readLoop unblocks and tears everything down _ = wc.fc.Close() // readLoop unblocks and tears everything down
return return
} }
msg := wire.NewWriter().U8(MuxPing).VarInt(MuxCtlSid).I64(time.Now().UnixMilli()).Out() msg := wire.NewWriter().U8(MuxPing).I64(time.Now().UnixMilli()).Out()
if err := wc.fc.WriteFrame(msg); err != nil { if err := wc.fc.WriteFrame(msg); err != nil {
return return
} }
@@ -299,56 +207,39 @@ func (wc *WorkerConn) heartbeatLoop(interval, timeout time.Duration) {
} }
} }
func (wc *WorkerConn) streamCount() int { // attach publishes a stream on this conn, or reports false if the conn has
wc.mu.Lock() // already died (or is already bound — which would be a caller bug).
defer wc.mu.Unlock()
return len(wc.streams)
}
func (wc *WorkerConn) newSid() int {
wc.mu.Lock()
defer wc.mu.Unlock()
sid := wc.nextSid
wc.nextSid++
return sid
}
// registerStream publishes a stream in the conn's table, or reports false if
// the conn has already died.
// //
// The check is not advisory. Allocate hands out a (conn, sid) pair under the // The check is not advisory. Dial hands out a conn, and the conn's readLoop
// pool lock, and the conn's readLoop can exit before the caller gets here — it // can exit before the caller gets here. A blind store would land on a conn
// has then already swapped the stream map, so a blind insert would land in a map // nothing iterates and the stream would never be torn down.
// nothing iterates and the stream would never be torn down. That normally hides func (wc *WorkerConn) attach(st *Stream) bool {
// behind a failing SYN, but not on a half-open conn whose readLoop died on a
// framing error while the socket is still writable. Callers must re-Allocate.
func (wc *WorkerConn) registerStream(sid int, st *Stream) bool {
wc.mu.Lock() wc.mu.Lock()
defer wc.mu.Unlock() defer wc.mu.Unlock()
if wc.closed { if wc.closed || wc.stream != nil {
return false return false
} }
wc.streams[sid] = st wc.stream = st
return true return true
} }
func (wc *WorkerConn) getStream(sid int) *Stream { func (wc *WorkerConn) getStream() *Stream {
wc.mu.Lock() wc.mu.Lock()
defer wc.mu.Unlock() defer wc.mu.Unlock()
return wc.streams[sid] return wc.stream
} }
func (wc *WorkerConn) removeStream(sid int) *Stream { func (wc *WorkerConn) detach() *Stream {
wc.mu.Lock() wc.mu.Lock()
defer wc.mu.Unlock() defer wc.mu.Unlock()
st := wc.streams[sid] st := wc.stream
delete(wc.streams, sid) wc.stream = nil
return st return st
} }
// readLoop dispatches inbound mux frames. It must never block on a stream's // readLoop dispatches inbound tunnel frames. DATA is only enqueued (the
// destination: DATA is only enqueued (the per-stream writeLoop does the actual // stream's writeLoop does the actual destination writes), so a stalled
// destination writes), so one slow destination cannot stall other streams. // destination cannot stall liveness / WND / FIN dispatch on this conn.
func (wc *WorkerConn) readLoop() { func (wc *WorkerConn) readLoop() {
for { for {
payload, err := wc.fc.ReadFrame() payload, err := wc.fc.ReadFrame()
@@ -363,24 +254,20 @@ func (wc *WorkerConn) readLoop() {
if err != nil { if err != nil {
continue continue
} }
sid, err := r.VarInt()
if err != nil {
continue
}
switch ftype { switch ftype {
case MuxData: case MuxData:
if st := wc.getStream(sid); st != nil { if st := wc.getStream(); st != nil {
st.deliverFromHub(r.Remaining()) st.deliverFromHub(r.Remaining())
} }
case MuxWnd: case MuxWnd:
if delta, err := r.VarInt(); err == nil && delta > 0 { if delta, err := r.VarInt(); err == nil && delta > 0 {
if st := wc.getStream(sid); st != nil { if st := wc.getStream(); st != nil {
st.grantSendWnd(delta) st.grantSendWnd(delta)
} }
} }
case MuxFin: case MuxFin:
// Graceful: drain what is already queued to the destination first. // Graceful: drain what is already queued to the destination first.
if st := wc.removeStream(sid); st != nil { if st := wc.detach(); st != nil {
st.gracefulFin() st.gracefulFin()
} }
case MuxRst: case MuxRst:
@@ -389,7 +276,7 @@ func (wc *WorkerConn) readLoop() {
if b, err := r.U8(); err == nil { if b, err := r.U8(); err == nil {
reason = int(b) reason = int(b)
} }
if st := wc.removeStream(sid); st != nil { if st := wc.detach(); st != nil {
st.onRst(reason) st.onRst(reason)
} }
case MuxResumeAck: case MuxResumeAck:
@@ -399,12 +286,12 @@ func (wc *WorkerConn) readLoop() {
if aerr != nil || derr != nil || cerr != nil { if aerr != nil || derr != nil || cerr != nil {
continue continue
} }
if st := wc.getStream(sid); st != nil { if st := wc.getStream(); st != nil {
st.deliverResume(resumeResult{accepted: accepted, delivered: delivered, cid: cid}) st.deliverResume(resumeResult{accepted: accepted, delivered: delivered, cid: cid})
} }
case MuxPing: case MuxPing:
nonce, _ := r.I64() nonce, _ := r.I64()
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxPong).VarInt(MuxCtlSid).I64(nonce).Out()) _ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxPong).I64(nonce).Out())
case MuxPong: case MuxPong:
now := time.Now() now := time.Now()
wc.lastPong.Store(now.UnixMilli()) wc.lastPong.Store(now.UnixMilli())
@@ -416,46 +303,42 @@ func (wc *WorkerConn) readLoop() {
} }
} }
default: default:
log.Printf("worker: unknown mux type %d", ftype) log.Printf("worker: unknown frame type %d", ftype)
} }
} }
// Connection lost: tear down all streams and drop from pool. // Connection lost: tear down the bound stream and drop from the set.
close(wc.done) close(wc.done)
wc.pool.remove(wc) wc.pool.remove(wc)
wc.mu.Lock() wc.mu.Lock()
// Marked before the map is swapped, under the same lock, so a concurrent // Marked before the pointer is cleared, under the same lock, so a concurrent
// registerStream either lands in the map we are about to drain or is refused. // attach either lands in the field we are about to drain or is refused.
wc.closed = true wc.closed = true
streams := make([]*Stream, 0, len(wc.streams)) st := wc.stream
for _, st := range wc.streams { wc.stream = nil
streams = append(streams, st)
}
wc.streams = make(map[int]*Stream)
wc.mu.Unlock() wc.mu.Unlock()
if st == nil {
return
}
// Only the tunnel leg died. Where the session negotiated resumption the // Only the tunnel leg died. Where the session negotiated resumption the
// destination sockets are kept open and each stream reattaches over a fresh // destination socket is kept open and the stream reattaches over a fresh
// conn (§7.5); otherwise this is the old, unconditional teardown. // conn (§7.5); otherwise this is the old, unconditional teardown.
// During Close there is no reattach to come: a stream that parked now would // During Close there is no reattach to come: a stream that parked now would
// hold its destination socket open past shutdown, so teardown instead. // hold its destination socket open past shutdown, so teardown instead.
if wc.pool.client.closing.Load() { if wc.pool.client.closing.Load() {
for _, st := range streams { st.teardown(false)
st.teardown(false)
}
return return
} }
for _, st := range streams { if !st.park(wc.grace) {
if !st.park(wc.grace) { st.teardown(false)
st.teardown(false)
}
} }
} }
func (wc *WorkerConn) sendSyn(sid int, cid []byte) error { func (wc *WorkerConn) sendSyn(cid []byte) error {
return wc.fc.WriteFrame(wire.NewWriter().U8(MuxSyn).VarInt(sid).Bytes(cid).Out()) return wc.fc.WriteFrame(wire.NewWriter().U8(MuxSyn).Bytes(cid).Out())
} }
func (wc *WorkerConn) sendData(sid int, data []byte) error { func (wc *WorkerConn) sendData(data []byte) error {
err := wc.fc.WriteFrame(wire.NewWriter().U8(MuxData).VarInt(sid).Bytes(data).Out()) err := wc.fc.WriteFrame(wire.NewWriter().U8(MuxData).Bytes(data).Out())
if wc.stats != nil { if wc.stats != nil {
wc.stats.framesOut.Add(1) wc.stats.framesOut.Add(1)
if err != nil { if err != nil {
@@ -465,42 +348,27 @@ func (wc *WorkerConn) sendData(sid int, data []byte) error {
return err return err
} }
func (wc *WorkerConn) sendFin(sid int) { func (wc *WorkerConn) sendFin() {
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxFin).VarInt(sid).Out()) _ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxFin).Out())
} }
func (wc *WorkerConn) sendRst(sid int) { func (wc *WorkerConn) sendRst() {
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxRst).VarInt(sid).Out()) _ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxRst).Out())
} }
func (wc *WorkerConn) sendWndUpdate(sid, delta int) { func (wc *WorkerConn) sendWndUpdate(delta int) {
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxWnd).VarInt(sid).VarInt(delta).Out()) _ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxWnd).VarInt(delta).Out())
} }
// leg binds a stream to one worker conn. The two fields are only meaningful
// together: stream ids are per-conn and restart at 1, so conn A's sid 3 and
// conn B's sid 3 belong to different players. A torn read across the two would
// credit, reset or FIN a stranger's stream, so the pair is swapped as one
// immutable value rather than as two fields.
type leg struct {
wc *WorkerConn
sid int
}
// String formats one (conn, sid) snapshot for log correlation. Stream ids
// restart at 1 per conn, so a bare sid cannot be traced across a reattach —
// the conn id is what ties the log lines together.
func (lg *leg) String() string { return fmt.Sprintf("conn%d/sid%d", lg.wc.id, lg.sid) }
// Stream bridges one player (via the hub) to one destination connection. // Stream bridges one player (via the hub) to one destination connection.
// //
// Data from the hub is queued and written to the destination by a dedicated // Data from the hub is queued and written to the destination by a dedicated
// writeLoop goroutine. The queue is bounded by the advertised receive window — // writeLoop goroutine. The queue is bounded by the advertised receive window —
// the hub never sends more un-credited bytes, so overflow is a protocol // the hub never sends more un-credited bytes, so overflow is a protocol
// violation and resets the stream. // violation and resets the tunnel.
type Stream struct { type Stream struct {
client *Client client *Client
leg atomic.Pointer[leg] wc atomic.Pointer[WorkerConn]
mapping Mapping mapping Mapping
srcIP string srcIP string
srcPort int srcPort int
@@ -527,8 +395,8 @@ type Stream struct {
// never interleave with live ones. // never interleave with live ones.
// //
// Deliberately not s.mu: deliverFromHub takes s.mu from the worker readLoop, // Deliberately not s.mu: deliverFromHub takes s.mu from the worker readLoop,
// and holding s.mu across a WriteFrame would stall frame dispatch for every // and holding s.mu across a WriteFrame would stall frame dispatch (WND/FIN
// other stream on the connection. // /heartbeat replies) on this conn.
sendMu sync.Mutex sendMu sync.Mutex
un unackedBuf // guarded by sendMu un unackedBuf // guarded by sendMu
@@ -578,10 +446,10 @@ type qentry struct {
fromHub bool fromHub bool
} }
func newStream(c *Client, wc *WorkerConn, sid int, cid []byte, m Mapping, ip string, port int) *Stream { func newStream(c *Client, wc *WorkerConn, cid []byte, m Mapping, ip string, port int) *Stream {
s := &Stream{client: c, cid: cid, mapping: m, srcIP: ip, srcPort: port, s := &Stream{client: c, cid: cid, mapping: m, srcIP: ip, srcPort: port,
resumable: wc.resume, sendWnd: wc.sendWndInit, done: make(chan struct{})} resumable: wc.resume, sendWnd: wc.sendWndInit, done: make(chan struct{})}
s.leg.Store(&leg{wc: wc, sid: sid}) s.wc.Store(wc)
if c.statsOn() { if c.statsOn() {
s.stats = &streamStats{opened: time.Now()} s.stats = &streamStats{opened: time.Now()}
} }
@@ -592,20 +460,28 @@ func newStream(c *Client, wc *WorkerConn, sid int, cid []byte, m Mapping, ip str
return s return s
} }
// conn returns the stream's current binding. Every caller must take exactly one // conn returns the stream's current worker conn. May be the original or a
// snapshot and use both fields from it; re-loading mid-operation reintroduces // reattach; callers that send must take one snapshot and use it for the
// the torn-pair hazard the leg exists to prevent. // whole operation so a concurrent rebind cannot split a write across conns.
func (s *Stream) conn() *leg { return s.leg.Load() } func (s *Stream) conn() *WorkerConn { return s.wc.Load() }
func (s *Stream) name() string {
if wc := s.conn(); wc != nil {
return fmt.Sprintf("conn%d", wc.id)
}
return "conn?"
}
// run dials the destination, optionally writes the PROXY v2 header, then pumps // run dials the destination, optionally writes the PROXY v2 header, then pumps
// destination -> hub (respecting the stream send window when negotiated). // destination -> hub (respecting the send window when negotiated).
func (s *Stream) run() { func (s *Stream) run() {
dest, err := net.DialTimeout("tcp", s.mapping.Destination, 10*time.Second) dest, err := net.DialTimeout("tcp", s.mapping.Destination, 10*time.Second)
if err != nil { if err != nil {
lg := s.conn() log.Printf("stream %s: dial %s failed: %v", s.name(), s.mapping.Destination, err)
log.Printf("stream %s: dial %s failed: %v", lg, s.mapping.Destination, err) if wc := s.conn(); wc != nil {
lg.wc.removeStream(lg.sid) wc.detach()
lg.wc.sendRst(lg.sid) wc.sendRst()
}
s.teardown(false) s.teardown(false)
return return
} }
@@ -616,7 +492,7 @@ func (s *Stream) run() {
if s.mapping.ProxyProtocol { if s.mapping.ProxyProtocol {
if hdr := s.buildProxyHeader(dest); hdr != nil { if hdr := s.buildProxyHeader(dest); hdr != nil {
if _, err := dest.Write(hdr); err != nil { if _, err := dest.Write(hdr); err != nil {
log.Printf("stream %s: proxy header write: %v", s.conn(), err) log.Printf("stream %s: proxy header write: %v", s.name(), err)
} }
} }
} }
@@ -663,7 +539,7 @@ func (s *Stream) run() {
} }
// sendToHub forwards destination bytes to the hub in bounded DATA frames, // sendToHub forwards destination bytes to the hub in bounded DATA frames,
// honoring both the stream send window and the client-wide bandwidth cap. // honoring both the send window and the client-wide bandwidth cap.
// Returns false once the stream closed or the worker conn failed. // Returns false once the stream closed or the worker conn failed.
func (s *Stream) sendToHub(data []byte) bool { func (s *Stream) sendToHub(data []byte) bool {
for len(data) > 0 { for len(data) > 0 {
@@ -674,7 +550,7 @@ func (s *Stream) sendToHub(data []byte) bool {
// Credit first, bandwidth second. The reverse order would charge the // Credit first, bandwidth second. The reverse order would charge the
// budget for bytes still parked on an exhausted window, so the client // budget for bytes still parked on an exhausted window, so the client
// would throttle itself below the configured rate. Holding credit while // would throttle itself below the configured rate. Holding credit while
// waiting for tokens is free — credit is per-stream, and the hub returns // waiting for tokens is free — credit is per-tunnel, and the hub returns
// it as it drains data to the player, independent of our pacing. // it as it drains data to the player, independent of our pacing.
if !s.acquireSendWnd(n) { if !s.acquireSendWnd(n) {
return false return false
@@ -714,9 +590,6 @@ func (s *Stream) sendToHub(data []byte) bool {
// one taken before the attempt. That also makes a failure survivable: while the // one taken before the attempt. That also makes a failure survivable: while the
// stream can still be resumed the bytes are already safe, and the reattach // stream can still be resumed the bytes are already safe, and the reattach
// replays them from wherever the hub says it got to. // replays them from wherever the hub says it got to.
//
// Note the old code let a failed write drop the rest of the chunk on the floor —
// the caller's slice advance sat after the error return.
func (s *Stream) emit(chunk []byte) bool { func (s *Stream) emit(chunk []byte) bool {
s.sendMu.Lock() s.sendMu.Lock()
if s.resumable { if s.resumable {
@@ -725,8 +598,13 @@ func (s *Stream) emit(chunk []byte) bool {
s.un.advance(s.ackedOffset.Load()) s.un.advance(s.ackedOffset.Load())
s.un.append(chunk) s.un.append(chunk)
} }
lg := s.conn() wc := s.conn()
err := lg.wc.sendData(lg.sid, chunk) var err error
if wc != nil {
err = wc.sendData(chunk)
} else {
err = errPoolClosed
}
s.sendMu.Unlock() s.sendMu.Unlock()
if err == nil { if err == nil {
@@ -792,7 +670,7 @@ func (s *Stream) writeLoop() {
// deliverFromHub enqueues hub bytes for the destination. Called from the worker // deliverFromHub enqueues hub bytes for the destination. Called from the worker
// readLoop; it never blocks — a peer that exceeds the advertised window is a // readLoop; it never blocks — a peer that exceeds the advertised window is a
// protocol violator and gets the stream reset. // protocol violator and gets the tunnel reset.
func (s *Stream) deliverFromHub(data []byte) { func (s *Stream) deliverFromHub(data []byte) {
if s.vel != nil { if s.vel != nil {
s.vel.ObserveC2S(data) // observation only; bytes still forwarded verbatim s.vel.ObserveC2S(data) // observation only; bytes still forwarded verbatim
@@ -804,10 +682,11 @@ func (s *Stream) deliverFromHub(data []byte) {
} }
if s.qBytes+len(data) > s.client.streamWnd { if s.qBytes+len(data) > s.client.streamWnd {
s.mu.Unlock() s.mu.Unlock()
lg := s.conn() log.Printf("stream %s: peer exceeded flow-control window; resetting", s.name())
log.Printf("stream %s: peer exceeded flow-control window; resetting", lg) if wc := s.conn(); wc != nil {
lg.wc.removeStream(lg.sid) wc.detach()
lg.wc.sendRst(lg.sid) wc.sendRst()
}
s.teardown(false) s.teardown(false)
return return
} }
@@ -891,8 +770,9 @@ func (s *Stream) credit(n int) {
delta := s.consumed delta := s.consumed
s.consumed = 0 s.consumed = 0
s.mu.Unlock() s.mu.Unlock()
lg := s.conn() if wc := s.conn(); wc != nil {
lg.wc.sendWndUpdate(lg.sid, delta) wc.sendWndUpdate(delta)
}
} }
func (s *Stream) buildProxyHeader(dest net.Conn) []byte { func (s *Stream) buildProxyHeader(dest net.Conn) []byte {
@@ -957,10 +837,11 @@ func (s *Stream) teardown(notifyHub bool) {
if dest != nil { if dest != nil {
_ = dest.Close() _ = dest.Close()
} }
lg := s.conn() if wc := s.conn(); wc != nil {
lg.wc.removeStream(lg.sid) wc.detach()
if notifyHub { if notifyHub {
lg.wc.sendFin(lg.sid) wc.sendFin()
}
} }
s.logSummary() s.logSummary()
} }
+74 -89
View File
@@ -15,9 +15,9 @@ on the wire, read [PROTOCOL.md](../PROTOCOL.md).
┌──────────┐ Intent 17, magic 0x01 │ │ │ ┌──────────┐ Intent 17, magic 0x01 │ │ │
│ Client │ ◀──────── control session ──────│ • pattern reg │ │ │ Client │ ◀──────── control session ──────│ • pattern reg │ │
│ (Go) │ ────────────────────────────────│ • CID table │ │ │ (Go) │ ────────────────────────────────│ • CID table │ │
│ │ Intent 17, magic 0x02 │ • mux demux │ │ │ │ Intent 17, magic 0x02 │ • 1:1 workers │ │
│ │ ═════════ worker conns ═════════│ │ │ │ │ ═════════ worker conns ═════════│ │ │
└──────────┘ multiplexed player streams └───────────────┘ │ └──────────┘ one TCP conn per player └───────────────┘ │
│ │ │ │
▼ MC bytes (+ optional HAProxy v2) │ ▼ MC bytes (+ optional HAProxy v2) │
┌───────────────┐ │ ┌───────────────┐ │
@@ -68,15 +68,15 @@ Player Hub Client Destinatio
│ │ pause player socket, │ │ pause player socket,
│ │ buffer bytes, mint CID │ │ buffer bytes, mint CID
│ │─ ControlRequest(CID, pattern, ip:port) ─▶ │ │─ ControlRequest(CID, pattern, ip:port) ─▶
│ │ allocate worker+stream │ │ dial dedicated worker
│ │◀──────── SYN(streamId, CID) ────────────│ │ │◀──────────── SYN(CID) ──────────────────│
│ │ takePending(CID) → bind dial destination, │ │ takePending(CID) → bind dial destination,
│ │ forward buffered bytes write HAProxy v2 hdr │ │ forward buffered bytes write HAProxy v2 hdr
│ │─ DATA(streamId, handshake…) ───▶ ── handshake ──▶│ │ │─ DATA(handshake…) ─────────────▶ ── handshake ──▶│
│ resume ─────────────────────────────│ bridge stream ⇄ dest │ resume ─────────────────────────────│ bridge conn ⇄ dest
│══════════════ player bytes ══ DATA ══▶│════ DATA ═══▶ dest.write │ │══════════════ player bytes ══ DATA ══▶│════ DATA ═══▶ dest.write │
│◀═══════════ dest bytes ═══ DATA ══════│◀═══ DATA ════ dest.read │ │◀═══════════ dest bytes ═══ DATA ══════│◀═══ DATA ════ dest.read │
│ player closes ──────────────────────│─ FIN(streamId) ────────▶ close dest │ │ player closes ──────────────────────│─ FIN ──────────────────▶ close dest │
``` ```
Key points: Key points:
@@ -87,6 +87,11 @@ Key points:
player's hostname — in `ControlRequest`, so the client can look it straight up player's hostname — in `ControlRequest`, so the client can look it straight up
in its own route table. Invalid patterns are rejected at registration with a in its own route table. Invalid patterns are rejected at registration with a
non-zero `RegisterAck` status. non-zero `RegisterAck` status.
* A **per-IP limiter** runs first: a token bucket (`playerRatePerSec` /
`playerBurst`) and a concurrent-socket cap (`maxPlayersPerIp`). It applies
only to player intents — never Intent 17 — and unmatched hostnames still
consume a token, so a hostname scan is not a free flood. A refusal closes
the socket before CID minting or pause.
* The hub **pauses** the player socket the instant it matches, so no player * The hub **pauses** the player socket the instant it matches, so no player
bytes are lost while the takeover is arranged; the buffered handshake is bytes are lost while the takeover is arranged; the buffered handshake is
forwarded **verbatim**, so the real server sees exactly what the player sent forwarded **verbatim**, so the real server sees exactly what the player sent
@@ -94,54 +99,38 @@ Key points:
* **CID** is 16 random bytes minted by the hub and delivered only over the * **CID** is 16 random bytes minted by the hub and delivered only over the
encrypted control session, so only the intended client learns it. Any worker encrypted control session, so only the intended client learns it. Any worker
connection presenting the correct CID is allowed to take over — that secrecy connection presenting the correct CID is allowed to take over — that secrecy
is what binds a worker stream to the right pending player without any explicit is what binds a worker conn to the right pending player without any explicit
client identity. client identity.
* Disconnects are symmetric: player-close → hub sends `FIN` → client closes the * Disconnects are symmetric: player-close → hub sends `FIN` → client closes the
destination; destination-close → client sends `FIN` → hub closes the player. destination; destination-close → client sends `FIN` → hub closes the player.
## 3. Multiplexing (worker connections) ## 3. Worker connections (1:1)
A worker connection is one encrypted TCP link carrying many **streams**. The A worker connection is one encrypted TCP link carrying **exactly one player**.
frame is intentionally tiny (PROTOCOL.md §7): The frame is intentionally tiny (PROTOCOL.md §7):
``` ```
[plaintext VarInt length][ FrameType u8 | StreamID VarInt | Data… ] (payload encrypted) [plaintext VarInt length][ FrameType u8 | Data… ] (payload encrypted)
``` ```
Only the client opens streams (`SYN`), so stream-id allocation is a simple There is no stream id. The client dials a fresh worker for each
per-connection counter with no coordination. `ControlRequest` and binds it with `SYN(CID)` after `SessionReady`. A second
bind on the same conn is a protocol violation.
### 3.1 Pool & allocation ### 3.1 Allocation
The client keeps 1…`maxConn` worker connections and places each new stream on The client holds at most `maxTunnels` live worker connections (default 256). A
the **least-loaded** one. The pool grows **breadth-first**: it dials out to `ControlRequest` either gets its own TCP connection or is dropped — the hub
`maxConn` before stacking streams, so that no single TCP connection ever becomes then closes the player when `pendingTimeoutMs` fires.
the shared point of failure for every player on the tunnel (PROTOCOL.md §7.1):
``` A dial is **never performed while holding the live-set lock** — session
pick least-loaded conn; use it establishment is network I/O, and one unresponsive hub must not park every
if leastLoaded.streams >= 1 and pool.size + dialsInFlight < maxConn: other player behind it. Each caller dials independently; callers are not
dial another worker conn in the background # the stream just placed does not wait serialized behind a single in-flight handshake.
```
The new connection becomes the least-loaded one and picks up subsequent streams. The e2e test `TestEachPlayerGetsOwnWorker` drives concurrent players and
Once the pool is at `maxConn`, streams stack on the least-loaded connection; confirms one worker conn each, without exceeding `maxTunnels`;
going past 8 active streams there is logged as pool saturation but is not an `TestDialDoesNotWedgeOnStalledHub` covers the stalled-dial path.
error.
Two properties of the dialing path matter as much as the placement rule:
* A dial is **never performed while holding the pool lock** — session
establishment is network I/O, and one unresponsive hub must not park every
other player behind it.
* Only when the pool is *empty* does a caller dial synchronously, and then
exactly one caller dials while the others wait on its result, so a burst of
arrivals cannot open a burst of redundant connections.
The e2e test `TestConcurrentStreamsUseMultipleConns` drives 20 simultaneous
streams with `maxConn=4` and confirms they spread over more than one connection
without exceeding the cap; `TestAllocateDoesNotWedgePoolOnStalledHub` covers the
stalled-dial path.
## 4. Encryption ## 4. Encryption
@@ -166,31 +155,28 @@ stalled-dial path.
non-blocking; crypto is CPU-cheap. This trades multi-core scaling for non-blocking; crypto is CPU-cheap. This trades multi-core scaling for
simplicity and correctness. simplicity and correctness.
* **Client:** goroutine-per-concern. One goroutine reads each connection * **Client:** goroutine-per-concern. One goroutine reads each connection
(control or worker); `WriteFrame` is mutex-serialized so many stream goroutines (control or worker); `WriteFrame` is mutex-serialized. Each tunnel has two
can share a worker connection safely. Each stream has two goroutines: `run` goroutines: `run` pumps destination → hub, and `writeLoop` is the only writer
pumps destination → hub, and `writeLoop` is the only writer to the to the destination, draining a queue fed by the worker readLoop. The
destination, draining a per-stream queue fed by the worker readLoop. The readLoop itself never writes to a destination, so a stalled destination cannot
readLoop itself never writes to a destination, so a stalled destination can stall WND / FIN / heartbeat dispatch on that conn.
never block frame dispatch for other streams.
## 6. Back-pressure & flow control ## 6. Back-pressure & flow control
Three mechanisms operate at different granularities: Three mechanisms operate at different granularities:
* **Per-stream credit windows** (PROTOCOL.md §7.3; the windows are exchanged * **Per-connection credit windows** (PROTOCOL.md §7.3; the windows are exchanged
at session establishment): each stream direction has an independent byte at session establishment): each tunnel direction has an independent byte
budget equal to the receiver's advertised window (default 256 KiB). A sender budget equal to the receiver's advertised window (default 256 KiB). A sender
that exhausts a that exhausts a window pauses *only that player's source* — the hub pauses the
stream's window pauses *only that stream's source* — the hub pauses the one one player socket, the client parks the one destination-reader goroutine.
player socket, the client parks the one destination-reader goroutine. Credit Credit is granted back (`WND` frames, batched at half-window) as bytes are
is granted back (`WND` frames, batched at half-window) as bytes are actually actually written to the terminal socket. The result: a slow player or slow
written to the terminal socket. The result: a slow player or slow destination destination jams its own tunnel at a bounded buffer size and nothing else.
jams its own stream at a bounded buffer size and nothing else. This is what * **TCP back-pressure** on each worker connection: when that socket is
eliminates head-of-line blocking between streams. congested, the hub parks that one player until it drains, and the client's
* **Aggregate TCP back-pressure** on each worker connection: when the shared `WriteFrame` blocks. Because the conn is 1:1, TCP HOL cannot stall another
socket itself is congested (total bandwidth, not one stream), the hub parks player.
all sending players until it drains, and the client's `WriteFrame` blocks.
This is fair — when the pipe is genuinely full, everyone should slow down.
* **Client egress shaping** (optional, `maxBandwidth`; `client/shaper.go`): a * **Client egress shaping** (optional, `maxBandwidth`; `client/shaper.go`): a
rate cap on everything the client sends to the hub, across all worker conns. rate cap on everything the client sends to the hub, across all worker conns.
@@ -198,18 +184,18 @@ The first two mechanisms have no time dimension. A credit window bounds how many
bytes are *in flight*, and TCP back-pressure only reacts once the pipe is already bytes are *in flight*, and TCP back-pressure only reacts once the pipe is already
full — which on a residential uplink is too late. One player loading chunks fills full — which on a residential uplink is too late. One player loading chunks fills
the line, the standing queue grows to seconds, and every other player's keepalive the line, the standing queue grows to seconds, and every other player's keepalive
times out. Nothing in §7.3 prevents that: each stream is individually times out. Nothing in §7.3 prevents that: each tunnel is individually
well-behaved, and collectively they still overrun the link. well-behaved, and collectively they still overrun the link.
The shaper closes that gap with a token bucket for the rate and start-time fair The shaper closes that gap with a token bucket for the rate and start-time fair
queueing for the split. A global virtual clock advances with each grant; every queueing for the split. A global virtual clock advances with each grant; every
stream remembers where its last request finished, and a new request is stamped tunnel remembers where its last request finished, and a new request is stamped
`max(stream.vfinish, vclock)`. Lowest stamp wins. A stream that keeps sending `max(tunnel.vfinish, vclock)`. Lowest stamp wins. A tunnel that keeps sending
pushes its own stamp further out and yields; a stream returning from idle is pushes its own stamp further out and yields; a tunnel returning from idle is
clamped back to the clock, so it cannot bank credit for time it did not use, but clamped back to the clock, so it cannot bank credit for time it did not use, but
is not penalised for the idleness either. A stream sending a few hundred bytes is not penalised for the idleness either. A tunnel sending a few hundred bytes
gets a nearer stamp than one sending a full chunk, so keepalives and chat overtake gets a nearer stamp than one sending a full chunk, so keepalives and chat overtake
bulk terrain data for free. One stream alone still gets the entire rate. bulk terrain data for free. One tunnel alone still gets the entire rate.
Two details keep bursts cheap. The bucket banks 200 ms of transmission, so a Two details keep bursts cheap. The bucket banks 200 ms of transmission, so a
player joining spends it at once instead of paying for the cap in visible player joining spends it at once instead of paying for the cap in visible
@@ -222,7 +208,7 @@ This is entirely client-local: nothing about it appears on the wire, and the hub
is unaware. Only DATA is shaped — delaying a `FIN`, `WND` or `PONG` would cause is unaware. Only DATA is shaped — delaying a `FIN`, `WND` or `PONG` would cause
the false-death detection §7.4 exists to avoid. the false-death detection §7.4 exists to avoid.
The window also bounds memory: a stream can hold at most one window of The window also bounds memory: a tunnel can hold at most one window of
undelivered data per direction (the client's pre-connect handshake buffer is undelivered data per direction (the client's pre-connect handshake buffer is
covered by the same bound). With stream resumption enabled (§7) the *sender* covered by the same bound). With stream resumption enabled (§7) the *sender*
holds a second window — the bytes it has sent but the peer has not yet credited, holds a second window — the bytes it has sent but the peer has not yet credited,
@@ -230,44 +216,42 @@ kept so they can be retransmitted after an outage. That is not a new bound so
much as the existing one made symmetric: the region is exactly what flow control much as the existing one made symmetric: the region is exactly what flow control
already declared outstanding, which is why resumption needs no cap of its own. already declared outstanding, which is why resumption needs no cap of its own.
Per-stream flow control is mandatory: the hub rejects a session whose Rekey Per-connection flow control is mandatory: the hub rejects a session whose Rekey
lacks the STREAM_FC flag, and the client rejects a hub that does not echo it — lacks the STREAM_FC flag, and the client rejects a hub that does not echo it —
peers that predate the mechanism cannot connect at all. peers that predate the mechanism cannot connect at all.
What remains (by design) is TCP-level head-of-line blocking: a lost packet on TCP-level head-of-line blocking is now confined to one player: a lost packet
a worker connection stalls all its streams for one retransmit. That is inherent stalls only that player's tunnel for one retransmit. The cost is one handshake
to mux-over-TCP; the connection pool is the mitigation, and a datagram and one NAT mapping per player instead of per pool slot.
transport (QUIC) would be the escape hatch if it ever matters.
## 7. Failure & recovery ## 7. Failure & recovery
* **Control session drop:** the client retries immediately, then backs off to a * **Control session drop:** the client retries immediately, then backs off to a
10s cap, and re-registers all patterns. Existing worker connections and their 10s cap, and re-registers all patterns. Existing worker connections and their
live streams are unaffected — they ride worker conns, which a control-session players are unaffected — they ride worker conns, which a control-session
close never touches. The hub meanwhile keeps that session's routes as close never touches. The hub meanwhile keeps that session's routes as
*orphaned* for `registrationGraceMs` (PROTOCOL.md §5.2) and **holds** players *orphaned* for `registrationGraceMs` (PROTOCOL.md §5.2) and **holds** players
arriving on them instead of refusing them, replaying the control request once arriving on them instead of refusing them, replaying the control request once
a client re-registers the pattern. Without that, the reconnect window is one a client re-registers the pattern. Without that, the reconnect window is one
in which every new player is told there is no such server. in which every new player is told there is no such server.
* **Worker connection drop:** the connection leaves the pool either way. What * **Worker connection drop:** the connection leaves the live set either way. What
happens to its streams depends on whether STREAM_RESUME was negotiated: happens to its player depends on whether STREAM_RESUME was negotiated:
* *without it* — every stream is torn down (destinations closed) and the hub * *without it* — the tunnel is torn down (destination closed) and the hub
closes the corresponding player sockets, as it always did; closes the player socket, as it always did;
* *with it* — the streams are **hung** instead (PROTOCOL.md §7.5). The * *with it* — the player is **hung** instead (PROTOCOL.md §7.5). The
destination sockets stay open, the hub pauses the player sockets and holds destination socket stays open, the hub pauses the player socket and holds
them for its grace period, and the client reattaches each stream over a it for its grace period, and the client reattaches over a freshly dialed
freshly dialed connection, replaying byte-exactly from the offset the peer connection, replaying byte-exactly from the offset the peer reports. The
reports. Players see a stall rather than a disconnect. player sees a stall rather than a disconnect.
* **Pending timeout:** if no worker takes over a matched player within * **Pending timeout:** if no worker takes over a matched player within
`pendingTimeoutMs`, the hub drops the pending entry and closes the player. `pendingTimeoutMs`, the hub drops the pending entry and closes the player.
* **Bad PSK / bad timestamp / bad magic:** the hub closes the TCP connection; * **Bad PSK / bad timestamp / bad magic:** the hub closes the TCP connection;
the client's session establishment fails fast. the client's session establishment fails fast.
Resumption is worth the machinery because a worker connection is only the Resumption is worth the machinery because a worker connection is only the
*middle* leg of every stream it carries. When it dies both terminal sockets are *middle* leg of the player it carries. When it dies both terminal sockets are
usually still healthy, so the old behaviour discarded working connections usually still healthy, so the old behaviour discarded working connections
because a replaceable transport failed — one conntrack expiry disconnected every because a replaceable transport failed. It also has to be byte-exact rather than
player sharing that connection. It also has to be byte-exact rather than
best-effort: bytes handed to a dying socket are lost with no notification and the best-effort: bytes handed to a dying socket are lost with no notification and the
frame cipher cannot be resynchronized, so an approximate reattach would splice frame cipher cannot be resynchronized, so an approximate reattach would splice
the tunneled protocol mid-packet, which is worse than a clean close. the tunneled protocol mid-packet, which is worse than a clean close.
@@ -279,13 +263,14 @@ wait on the control reconnect backoff.
## 8. Known limitations ## 8. Known limitations
1. No AEAD — payload integrity/authenticity is not cryptographically guaranteed. 1. No AEAD — payload integrity/authenticity is not cryptographically guaranteed.
2. TCP-level head-of-line blocking within a worker connection (lost packets; 2. One handshake and one NAT mapping per player (the cost of dropping mux).
see §6) — per-stream flow control removes the application-level variant only.
3. Single-event-loop hub (see §5) bounds throughput to one core. 3. Single-event-loop hub (see §5) bounds throughput to one core.
4. `Intent 18` is reserved but only stubbed (the hub logs and closes). 4. `Intent 18` is reserved for a plaintext status probe.
5. Pattern ownership is last-writer-wins; two clients registering the identical 5. Pattern ownership is last-writer-wins; two clients registering the identical
pattern string will silently reassign it. Overlapping-but-distinct regexes are pattern string will silently reassign it. Overlapping-but-distinct regexes are
both kept, and when several match one hostname the winner is unspecified. both kept, and when several match one hostname the winner is unspecified.
6. The player IP limiter matches exact addresses. A `/64` of IPv6 clients looks
like many independent IPs; aggregation is a later, isolated change.
These are deliberate scope choices for a connectivity-focused P2P tool, not These are deliberate scope choices for a connectivity-focused P2P tool, not
oversights; each is a small, well-isolated change away from being hardened. oversights; each is a small, well-isolated change away from being hardened.
+5 -5
View File
@@ -14,12 +14,12 @@ import (
// startClientWithBandwidth is startClient with an egress cap, for the shaping // startClientWithBandwidth is startClient with an egress cap, for the shaping
// tests. Only the client→hub direction is shaped, which in this harness is the // tests. Only the client→hub direction is shaped, which in this harness is the
// leg carrying the mock destination's echo back to the player. // leg carrying the mock destination's echo back to the player.
func startClientWithBandwidth(t *testing.T, hubAddr, psk string, maxConn int, bandwidth string, mappings []client.Mapping) *client.Client { func startClientWithBandwidth(t *testing.T, hubAddr, psk string, maxTunnels int, bandwidth string, mappings []client.Mapping) *client.Client {
t.Helper() t.Helper()
cfg := &client.Config{ cfg := &client.Config{
Server: hubAddr, Server: hubAddr,
PSK: psk, PSK: psk,
MaxConn: maxConn, MaxTunnels: maxTunnels,
PingIntervalMs: 20000, PingIntervalMs: 20000,
MaxBandwidth: bandwidth, MaxBandwidth: bandwidth,
Mappings: mappings, Mappings: mappings,
@@ -104,9 +104,9 @@ func TestCappedBandwidthDoesNotStarveLightStreams(t *testing.T) {
hubAddr := fmt.Sprintf("127.0.0.1:%d", port) hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
startHub(t, port, psk) startHub(t, port, psk)
dest := newMockDest(t, modeEcho) dest := newMockDest(t, modeEcho)
// maxConn=1 forces every stream onto one worker conn, so nothing but the // Each player has its own worker conn; the shaper is still the only thing
// shaper is deciding who gets the link. // splitting the shared uplink budget.
startClientWithBandwidth(t, hubAddr, psk, 1, "1MB/s", []client.Mapping{ startClientWithBandwidth(t, hubAddr, psk, 4, "1MB/s", []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr}, {Pattern: "mc.local", Destination: dest.addr},
}) })
+17 -20
View File
@@ -13,19 +13,19 @@ import (
// startClient builds and starts an in-process client against the hub, with the // startClient builds and starts an in-process client against the hub, with the
// given mappings, returning the running client. // given mappings, returning the running client.
func startClient(t *testing.T, hubAddr, psk string, maxConn int, mappings []client.Mapping) *client.Client { func startClient(t *testing.T, hubAddr, psk string, maxTunnels int, mappings []client.Mapping) *client.Client {
t.Helper() t.Helper()
return startClientWithPing(t, hubAddr, psk, maxConn, 20000, mappings) return startClientWithPing(t, hubAddr, psk, maxTunnels, 20000, mappings)
} }
// startClientWithPing is startClient with an explicit heartbeat interval, for // startClientWithPing is startClient with an explicit heartbeat interval, for
// tests that need liveness detection to trigger quickly. // tests that need liveness detection to trigger quickly.
func startClientWithPing(t *testing.T, hubAddr, psk string, maxConn, pingMs int, mappings []client.Mapping) *client.Client { func startClientWithPing(t *testing.T, hubAddr, psk string, maxTunnels, pingMs int, mappings []client.Mapping) *client.Client {
t.Helper() t.Helper()
return startClientCfg(t, &client.Config{ return startClientCfg(t, &client.Config{
Server: hubAddr, Server: hubAddr,
PSK: psk, PSK: psk,
MaxConn: maxConn, MaxTunnels: maxTunnels,
PingIntervalMs: pingMs, PingIntervalMs: pingMs,
Mappings: mappings, Mappings: mappings,
}) })
@@ -107,7 +107,7 @@ func TestRegexPatternMatch(t *testing.T) {
} }
} }
// TestLargeTransfer pushes a multi-megabyte payload both ways to exercise mux // TestLargeTransfer pushes a multi-megabyte payload both ways to exercise
// framing and back-pressure. // framing and back-pressure.
func TestLargeTransfer(t *testing.T) { func TestLargeTransfer(t *testing.T) {
const psk = "e2e-large" const psk = "e2e-large"
@@ -147,18 +147,17 @@ func TestLargeTransfer(t *testing.T) {
} }
} }
// TestConcurrentStreamsUseMultipleConns confirms the least-loaded allocator // TestEachPlayerGetsOwnWorker confirms the 1:1 rule: N concurrent players
// grows the pool breadth-first: concurrent streams spread over several worker // produce N worker connections, and the maxTunnels cap is honoured.
// connections rather than stacking on one, without ever exceeding maxConn. func TestEachPlayerGetsOwnWorker(t *testing.T) {
func TestConcurrentStreamsUseMultipleConns(t *testing.T) {
const psk = "e2e-concurrent" const psk = "e2e-concurrent"
const n = 20 const n = 8
const maxConn = 4 const maxTunnels = 16
port := freePort(t) port := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", port) hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
startHub(t, port, psk) startHub(t, port, psk)
dest := newMockDest(t, modeEcho) dest := newMockDest(t, modeEcho)
c := startClient(t, hubAddr, psk, maxConn, []client.Mapping{ c := startClient(t, hubAddr, psk, maxTunnels, []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr}, {Pattern: "mc.local", Destination: dest.addr},
}) })
@@ -168,8 +167,6 @@ func TestConcurrentStreamsUseMultipleConns(t *testing.T) {
_ = pc.Close() _ = pc.Close()
} }
}() }()
// Establish streams sequentially so allocation is deterministic; keep them
// all open to hold streams active.
for i := 0; i < n; i++ { for i := 0; i < n; i++ {
pc := dialPlayer(t, hubAddr, "mc.local") pc := dialPlayer(t, hubAddr, "mc.local")
playerEcho(t, pc, []byte(fmt.Sprintf("hello-%d", i))) playerEcho(t, pc, []byte(fmt.Sprintf("hello-%d", i)))
@@ -177,13 +174,13 @@ func TestConcurrentStreamsUseMultipleConns(t *testing.T) {
} }
got := c.WorkerConnCount() got := c.WorkerConnCount()
if got < 2 { if got != n {
t.Fatalf("expected >=2 worker conns for %d concurrent streams, got %d", n, got) t.Fatalf("expected %d worker conns for %d players, got %d", n, n, got)
} }
if got > maxConn { if got > maxTunnels {
t.Fatalf("worker conns %d exceed maxConn %d", got, maxConn) t.Fatalf("worker conns %d exceed maxTunnels %d", got, maxTunnels)
} }
t.Logf("%d concurrent streams spread over %d worker conn(s)", n, got) t.Logf("%d players on %d worker conn(s)", n, got)
} }
// TestProxyProtocol checks that the client prepends a correct HAProxy v2 header // TestProxyProtocol checks that the client prepends a correct HAProxy v2 header
@@ -278,7 +275,7 @@ func TestBadPSK(t *testing.T) {
cfg := &client.Config{ cfg := &client.Config{
Server: hubAddr, Server: hubAddr,
PSK: "totally-wrong", PSK: "totally-wrong",
MaxConn: 2, MaxTunnels: 2,
PingIntervalMs: 20000, PingIntervalMs: 20000,
Mappings: []client.Mapping{{Pattern: "mc.local", Destination: dest.addr}}, Mappings: []client.Mapping{{Pattern: "mc.local", Destination: dest.addr}},
} }
+174
View File
@@ -0,0 +1,174 @@
package e2e
import (
"fmt"
"net"
"testing"
"time"
"github.com/iceBear67/redapricot/client"
)
// expectClosedSoon fails unless conn is closed (or reset) within d. A player
// that entered pending stays open until pendingTimeoutMs, so a fast EOF is
// how we tell the limiter dropped the socket before match-side bookkeeping.
func expectClosedSoon(t *testing.T, conn net.Conn, d time.Duration) {
t.Helper()
_ = conn.SetReadDeadline(time.Now().Add(d))
n, err := conn.Read(make([]byte, 16))
if err == nil {
t.Fatalf("expected the hub to close the player, read %d bytes", n)
}
}
// TestPlayerBurstDropsExtraHandshakes: more arrivals than playerBurst from the
// same IP are closed after the handshake and never become pending.
func TestPlayerBurstDropsExtraHandshakes(t *testing.T) {
const psk = "e2e-rate-burst"
port := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
startHubCfg(t, port, psk, map[string]any{
"playerRatePerSec": 1,
"playerBurst": 2,
"maxPlayersPerIp": 64,
})
dest := newMockDest(t, modeEcho)
startClient(t, hubAddr, psk, 8, []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr},
})
kept := make([]net.Conn, 0, 2)
defer func() {
for _, c := range kept {
_ = c.Close()
}
}()
for i := 0; i < 2; i++ {
pc := dialPlayer(t, hubAddr, "mc.local")
playerEcho(t, pc, []byte(fmt.Sprintf("ok-%d", i)))
kept = append(kept, pc)
}
// Tokens spent, first two still held: extras must die well inside pendingTimeout.
for i := 0; i < 2; i++ {
extra := dialPlayer(t, hubAddr, "mc.local")
expectClosedSoon(t, extra, 1500*time.Millisecond)
_ = extra.Close()
}
// The admitted players are unaffected.
playerEcho(t, kept[0], []byte("still-here"))
}
// TestMaxPlayersPerIpCapsConcurrentSockets: a second player from the same IP
// is refused while the first is still open, and admitted again after it closes.
func TestMaxPlayersPerIpCapsConcurrentSockets(t *testing.T) {
const psk = "e2e-rate-conc"
port := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
startHubCfg(t, port, psk, map[string]any{
"playerRatePerSec": 0,
"playerBurst": 16,
"maxPlayersPerIp": 1,
})
dest := newMockDest(t, modeEcho)
startClient(t, hubAddr, psk, 4, []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr},
})
first := dialPlayer(t, hubAddr, "mc.local")
defer first.Close()
playerEcho(t, first, []byte("first"))
second := dialPlayer(t, hubAddr, "mc.local")
expectClosedSoon(t, second, 1500*time.Millisecond)
_ = second.Close()
playerEcho(t, first, []byte("still-first"))
_ = first.Close()
// The closeHandler runs on the hub event loop; give it a beat to release.
time.Sleep(200 * time.Millisecond)
third := dialPlayer(t, hubAddr, "mc.local")
defer third.Close()
playerEcho(t, third, []byte("after-release"))
}
// TestUnmatchedHostConsumesRateBudget: a hostname scan is not a free flood.
// Two unmatched handshakes spend the burst, so a later matching player is
// also dropped.
func TestUnmatchedHostConsumesRateBudget(t *testing.T) {
const psk = "e2e-rate-scan"
port := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
startHubCfg(t, port, psk, map[string]any{
"playerRatePerSec": 1,
"playerBurst": 2,
"maxPlayersPerIp": 64,
})
dest := newMockDest(t, modeEcho)
startClient(t, hubAddr, psk, 4, []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr},
})
for i := 0; i < 2; i++ {
miss := dialPlayer(t, hubAddr, "no.such.host")
expectClosedSoon(t, miss, 1500*time.Millisecond)
_ = miss.Close()
}
matched := dialPlayer(t, hubAddr, "mc.local")
expectClosedSoon(t, matched, 1500*time.Millisecond)
_ = matched.Close()
}
// TestIntent17IgnoresPlayerLimiter: the control session and worker conns are
// Intent 17, so a limiter tight enough to refuse a second player must not
// prevent the client from connecting or taking over the first player.
func TestIntent17IgnoresPlayerLimiter(t *testing.T) {
const psk = "e2e-rate-intent17"
port := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
startHubCfg(t, port, psk, map[string]any{
"playerRatePerSec": 1,
"playerBurst": 1,
"maxPlayersPerIp": 1,
})
dest := newMockDest(t, modeEcho)
startClient(t, hubAddr, psk, 4, []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr},
})
pc := dialPlayer(t, hubAddr, "mc.local")
defer pc.Close()
playerEcho(t, pc, []byte("intent17-ok"))
}
// TestPlayerLimiterOffSwitch: both knobs at 0 restore phase-1 behaviour —
// many players from 127.0.0.1 all get through.
func TestPlayerLimiterOffSwitch(t *testing.T) {
const psk = "e2e-rate-off"
const n = 6
port := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
startHubCfg(t, port, psk, map[string]any{
"playerRatePerSec": 0,
"maxPlayersPerIp": 0,
})
dest := newMockDest(t, modeEcho)
startClient(t, hubAddr, psk, n, []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr},
})
conns := make([]net.Conn, 0, n)
defer func() {
for _, c := range conns {
_ = c.Close()
}
}()
for i := 0; i < n; i++ {
pc := dialPlayer(t, hubAddr, "mc.local")
playerEcho(t, pc, []byte(fmt.Sprintf("off-%d", i)))
conns = append(conns, pc)
}
}
+6 -7
View File
@@ -131,10 +131,9 @@ func TestResumePreservesByteStream(t *testing.T) {
} }
// TestResumeWithConcurrentStreams covers the failure the single-stream test // TestResumeWithConcurrentStreams covers the failure the single-stream test
// cannot reach: stream ids restart at 1 on every conn, so after a reattach two // cannot reach: two independent tunnels resume at once. A torn rebind that
// players can hold the same id on different conns. A binding that updates the // writes a frame onto the wrong conn would credit or reset the other player's
// conn and the id separately will credit or reset the wrong player's stream, and // tunnel, and that only shows up when a second player is there to be corrupted.
// that only shows up when a second stream is there to be corrupted.
func TestResumeWithConcurrentStreams(t *testing.T) { func TestResumeWithConcurrentStreams(t *testing.T) {
const psk = "e2e-resume-multi" const psk = "e2e-resume-multi"
hubPort := freePort(t) hubPort := freePort(t)
@@ -143,7 +142,7 @@ func TestResumeWithConcurrentStreams(t *testing.T) {
dest := newMockDest(t, modeEcho) dest := newMockDest(t, modeEcho)
relay := newBlackholeRelay(t, hubAddr) relay := newBlackholeRelay(t, hubAddr)
startClientWithPing(t, relay.addr, psk, 2, 400, []client.Mapping{ startClientWithPing(t, relay.addr, psk, 4, 400, []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr}, {Pattern: "mc.local", Destination: dest.addr},
}) })
@@ -197,7 +196,7 @@ func TestResumeDisabledClosesImmediately(t *testing.T) {
startClientCfg(t, &client.Config{ startClientCfg(t, &client.Config{
Server: relay.addr, Server: relay.addr,
PSK: psk, PSK: psk,
MaxConn: 1, MaxTunnels: 1,
PingIntervalMs: 400, PingIntervalMs: 400,
StreamResume: &off, StreamResume: &off,
Mappings: []client.Mapping{{Pattern: "mc.local", Destination: dest.addr}}, Mappings: []client.Mapping{{Pattern: "mc.local", Destination: dest.addr}},
@@ -237,7 +236,7 @@ func TestResumeGraceExpiryClosesPlayer(t *testing.T) {
startClientCfg(t, &client.Config{ startClientCfg(t, &client.Config{
Server: relay.addr, Server: relay.addr,
PSK: psk, PSK: psk,
MaxConn: 1, MaxTunnels: 1,
PingIntervalMs: 400, PingIntervalMs: 400,
ResumeGraceMs: 2000, ResumeGraceMs: 2000,
Mappings: []client.Mapping{{Pattern: "mc.local", Destination: dest.addr}}, Mappings: []client.Mapping{{Pattern: "mc.local", Destination: dest.addr}},
+5 -7
View File
@@ -27,17 +27,15 @@ func echoRounds(t *testing.T, hubAddr string, rounds, size int) {
} }
// TestSlowPlayerDoesNotStallOthers: a player that stops reading while megabytes // TestSlowPlayerDoesNotStallOthers: a player that stops reading while megabytes
// are echoed back to it must not stall another stream on the same worker // are echoed back to it must not stall another player. Flow control and the
// connection (maxConn=1 forces sharing). Before per-stream flow control, the // 1:1 worker model keep a jammed tunnel from taking anyone else with it.
// hub paused the whole worker socket once that player's write queue filled,
// freezing every other stream's downstream data.
func TestSlowPlayerDoesNotStallOthers(t *testing.T) { func TestSlowPlayerDoesNotStallOthers(t *testing.T) {
const psk = "e2e-slowplayer" const psk = "e2e-slowplayer"
port := freePort(t) port := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", port) hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
startHub(t, port, psk) startHub(t, port, psk)
dest := newMockDest(t, modeEcho) dest := newMockDest(t, modeEcho)
startClient(t, hubAddr, psk, 1, []client.Mapping{ startClient(t, hubAddr, psk, 2, []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr}, {Pattern: "mc.local", Destination: dest.addr},
}) })
@@ -52,7 +50,7 @@ func TestSlowPlayerDoesNotStallOthers(t *testing.T) {
// Let the slow stream jam: its flow-control window fills and stays full. // Let the slow stream jam: its flow-control window fills and stays full.
time.Sleep(1 * time.Second) time.Sleep(1 * time.Second)
// The fast player shares the single worker conn and must still round-trip. // The fast player has its own worker conn and must still round-trip.
echoRounds(t, hubAddr, 10, 8*1024) echoRounds(t, hubAddr, 10, 8*1024)
} }
@@ -67,7 +65,7 @@ func TestSlowDestinationDoesNotStallOthers(t *testing.T) {
startHub(t, port, psk) startHub(t, port, psk)
dest := newMockDest(t, modeEcho) dest := newMockDest(t, modeEcho)
hole := newMockDest(t, modeBlackhole) hole := newMockDest(t, modeBlackhole)
startClient(t, hubAddr, psk, 1, []client.Mapping{ startClient(t, hubAddr, psk, 2, []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr}, {Pattern: "mc.local", Destination: dest.addr},
{Pattern: "hole.local", Destination: hole.addr}, {Pattern: "hole.local", Destination: hole.addr},
}) })
+4 -1
View File
@@ -9,5 +9,8 @@
"resumeGraceMs": 20000, "resumeGraceMs": 20000,
"maxParkedStreams": 256, "maxParkedStreams": 256,
"statsIntervalMs": 0, "statsIntervalMs": 0,
"registrationGraceMs": 15000 "registrationGraceMs": 15000,
"playerRatePerSec": 8,
"playerBurst": 16,
"maxPlayersPerIp": 64
} }
@@ -19,7 +19,10 @@ public record Config(
int maxParkedStreams, int maxParkedStreams,
long maxParkedBytes, long maxParkedBytes,
long statsIntervalMs, long statsIntervalMs,
long registrationGraceMs long registrationGraceMs,
double playerRatePerSec,
int playerBurst,
int maxPlayersPerIp
) { ) {
public static Config load(Path file) throws Exception { public static Config load(Path file) throws Exception {
JsonObject json = new JsonObject(Files.readString(file)); JsonObject json = new JsonObject(Files.readString(file));
@@ -68,6 +71,14 @@ public record Config(
// (its backoff caps at 10s) without holding a player so long // (its backoff caps at 10s) without holding a player so long
// that they give up anyway; 0 disables and restores the old // that they give up anyway; 0 disables and restores the old
// behaviour of dropping routes the moment a session closes. // behaviour of dropping routes the moment a session closes.
json.getLong("registrationGraceMs", 15_000L)); json.getLong("registrationGraceMs", 15_000L),
// Player-only (Intent ∉ {17, 18}). 0 turns that mechanism off.
// Unmatched hostnames still consume a token — otherwise a
// hostname scan is a free flood. Intent 17 is never admitted
// through the limiter: every worker comes from the client's
// one IP.
Math.max(0, json.getDouble("playerRatePerSec", 8.0)),
Math.max(1, json.getInteger("playerBurst", 16)),
Math.max(0, json.getInteger("maxPlayersPerIp", 64)));
} }
} }
@@ -33,6 +33,7 @@ public final class Hub {
public final Config config; public final Config config;
public final byte[] pskBytes; public final byte[] pskBytes;
public final String pskAddress; public final String pskAddress;
public final IpRateLimiter limiter;
private final Map<String, Registration> patterns = new ConcurrentHashMap<>(); private final Map<String, Registration> patterns = new ConcurrentHashMap<>();
private final Map<String, PendingPlayer> pending = new ConcurrentHashMap<>(); private final Map<String, PendingPlayer> pending = new ConcurrentHashMap<>();
@@ -77,6 +78,25 @@ public final class Hub {
this.config = config; this.config = config;
this.pskBytes = config.psk().getBytes(StandardCharsets.UTF_8); this.pskBytes = config.psk().getBytes(StandardCharsets.UTF_8);
this.pskAddress = Crypto.pskAddress(config.psk()); this.pskAddress = Crypto.pskAddress(config.psk());
this.limiter = new IpRateLimiter(
config.playerRatePerSec(), config.playerBurst(), config.maxPlayersPerIp());
// Unit tests construct a Hub with a null Vertx (no event loop).
if (limiter.enabled() && vertx != null) {
vertx.setPeriodic(IpRateLimiter.SWEEP_MS, id -> limiter.sweep(System.currentTimeMillis()));
}
}
/**
* Admit one player socket from {@code ip}. {@code null} means allowed;
* the caller must {@link #releasePlayer} on every close path (pending
* timeout, unmatched host, player FIN, park eviction).
*/
public IpRateLimiter.Deny admitPlayer(String ip) {
return limiter.admit(ip, System.currentTimeMillis());
}
public void releasePlayer(String ip) {
limiter.release(ip, System.currentTimeMillis());
} }
// ---- pattern registry ---- // ---- pattern registry ----
@@ -297,6 +317,10 @@ public final class Hub {
if (streams.remove(st.cidHex) == null) return; if (streams.remove(st.cidHex) == null) return;
if (st.parked) unpark(st); if (st.parked) unpark(st);
st.unacked.clear(); st.unacked.clear();
// The player's closeHandler was replaced at SYN/RESUME bind, so the
// HubConnection cleanup never runs. This is the live/parked release
// path; pending/unmatched still go through HubConnection.closeCleanup.
releasePlayer(st.playerIp);
} }
/** Look up a stream by the CID a client presented, live or parked. */ /** Look up a stream by the CID a client presented, live or parked. */
@@ -280,6 +280,19 @@ public final class HubConnection {
// ---- player connection ---- // ---- player connection ----
private void handlePlayer(String address) { private void handlePlayer(String address) {
// Before match / CID / pause: unmatched hostnames still consume a token,
// otherwise a hostname scan is a free flood. Intent 17 never reaches
// this method (PROTOCOL.md §9.1).
String ip = socket.remoteAddress() != null ? socket.remoteAddress().host() : "0.0.0.0";
if (hub.admitPlayer(ip) != null) {
socket.close();
return;
}
// Release on every close of this socket: pending timeout, unmatched
// host, player FIN, park eviction. Later cleanups wrap this, they
// must not replace it.
closeCleanup = () -> hub.releasePlayer(ip);
String host = Hub.normalizeAddress(address); String host = Hub.normalizeAddress(address);
Hub.Match matched = hub.match(address); Hub.Match matched = hub.match(address);
if (matched == null) { if (matched == null) {
@@ -293,14 +306,16 @@ public final class HubConnection {
String pattern = matched.pattern(); String pattern = matched.pattern();
byte[] cid = hub.newCid(); byte[] cid = hub.newCid();
String cidHex = Hex.encode(cid); String cidHex = Hex.encode(cid);
String ip = socket.remoteAddress() != null ? socket.remoteAddress().host() : "0.0.0.0";
int port = socket.remoteAddress() != null ? socket.remoteAddress().port() : 0; int port = socket.remoteAddress() != null ? socket.remoteAddress().port() : 0;
socket.pause(); socket.pause();
Buffer buffered = hs.copy(); // handshake + any pipelined bytes, forwarded verbatim Buffer buffered = hs.copy(); // handshake + any pipelined bytes, forwarded verbatim
PendingPlayer p = new PendingPlayer(cid, cidHex, socket, buffered, pattern, ip, port); PendingPlayer p = new PendingPlayer(cid, cidHex, socket, buffered, pattern, ip, port);
closeCleanup = () -> hub.removePending(cidHex); closeCleanup = () -> {
hub.removePending(cidHex);
hub.releasePlayer(ip);
};
if (session == null) { if (session == null) {
// The route is orphaned: its client's control session has closed and // The route is orphaned: its client's control session has closed and
@@ -0,0 +1,134 @@
package io.icybear.redapricot;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.Map;
/**
* Per-IP admission for <em>player</em> connections only (PROTOCOL.md §9.1).
*
* <p>Intent 17 (control session + every worker conn) is never admitted through
* here. Those sockets all come from the client's one address; limiting them
* would be the hub throttling its own client. e2e shares {@code 127.0.0.1}
* between players and the client for the same reason.
*
* <p>Event-loop confined: no locking. A {@code 0} rate or concurrent cap turns
* that mechanism off; both {@code 0} makes {@link #admit} a no-op.
*/
public final class IpRateLimiter {
private static final Logger LOG = LogManager.getLogger("redapricot.limit");
/** Idle buckets older than this are dropped so a one-shot flood cannot leak. */
static final long SWEEP_MS = 60_000;
private static final long DENY_LOG_INTERVAL_MS = 2_000;
public enum Deny { RATE, CONCURRENT }
private final double ratePerSec; // 0 = token bucket off
private final double burst;
private final int maxConcurrent; // 0 = concurrent cap off
private final Map<String, Bucket> buckets = new LinkedHashMap<>();
static final class Bucket {
double tokens;
long lastRefillMs;
int concurrent;
long lastActivityMs;
long lastDenyLogMs;
int deniesSinceLog;
}
public IpRateLimiter(double ratePerSec, double burst, int maxConcurrent) {
this.ratePerSec = Math.max(0, ratePerSec);
this.burst = Math.max(1, burst);
this.maxConcurrent = Math.max(0, maxConcurrent);
}
public boolean enabled() {
return ratePerSec > 0 || maxConcurrent > 0;
}
/**
* Consume one player admission for {@code ip}. {@code null} means allowed
* and the caller <em>must</em> {@link #release} when the socket closes.
*/
public Deny admit(String ip, long nowMs) {
if (!enabled()) return null;
Bucket b = bucket(ip, nowMs);
b.lastActivityMs = nowMs;
refill(b, nowMs);
if (ratePerSec > 0 && b.tokens < 1.0) {
noteDeny(ip, b, nowMs, Deny.RATE);
return Deny.RATE;
}
if (maxConcurrent > 0 && b.concurrent >= maxConcurrent) {
noteDeny(ip, b, nowMs, Deny.CONCURRENT);
return Deny.CONCURRENT;
}
if (ratePerSec > 0) b.tokens -= 1.0;
b.concurrent++;
return null;
}
public void release(String ip, long nowMs) {
Bucket b = buckets.get(ip);
if (b == null) return;
if (b.concurrent > 0) b.concurrent--;
b.lastActivityMs = nowMs;
}
/** Drop idle empty buckets. Safe to call on a timer. */
public void sweep(long nowMs) {
Iterator<Map.Entry<String, Bucket>> it = buckets.entrySet().iterator();
while (it.hasNext()) {
Bucket b = it.next().getValue();
if (b.concurrent == 0 && nowMs - b.lastActivityMs >= SWEEP_MS) {
it.remove();
}
}
}
/** Visible for tests. */
int bucketCount() {
return buckets.size();
}
/** Visible for tests. */
int concurrent(String ip) {
Bucket b = buckets.get(ip);
return b == null ? 0 : b.concurrent;
}
private Bucket bucket(String ip, long nowMs) {
Bucket b = buckets.get(ip);
if (b != null) return b;
b = new Bucket();
b.tokens = burst;
b.lastRefillMs = nowMs;
b.lastActivityMs = nowMs;
buckets.put(ip, b);
return b;
}
private void refill(Bucket b, long nowMs) {
if (ratePerSec <= 0) return;
double elapsed = (nowMs - b.lastRefillMs) / 1000.0;
if (elapsed <= 0) return;
b.tokens = Math.min(burst, b.tokens + elapsed * ratePerSec);
b.lastRefillMs = nowMs;
}
private void noteDeny(String ip, Bucket b, long nowMs, Deny why) {
b.deniesSinceLog++;
if (b.lastDenyLogMs != 0 && nowMs - b.lastDenyLogMs < DENY_LOG_INTERVAL_MS) {
return;
}
LOG.warn("dropping player from {}: {} ({} similar since last log)",
ip, why == Deny.RATE ? "rate" : "maxPlayersPerIp", b.deniesSinceLog);
b.lastDenyLogMs = nowMs;
b.deniesSinceLog = 0;
}
}
@@ -4,10 +4,10 @@ import io.vertx.core.buffer.Buffer;
import io.vertx.core.net.NetSocket; import io.vertx.core.net.NetSocket;
/** /**
* One tunneled player: the player socket plus the flow-control state of the mux * One tunneled player: the player socket plus the flow-control state of the
* stream carrying it (PROTOCOL.md §7.3). * worker conn carrying it (PROTOCOL.md §7.3).
* *
* <p>This is deliberately <em>not</em> owned by {@link WorkerConn}. A stream's * <p>This is deliberately <em>not</em> owned by {@link WorkerConn}. A tunnel's
* identity is the player, not the connection it happens to ride: the worker conn * identity is the player, not the connection it happens to ride: the worker conn
* is a replaceable transport, and state that dies with it cannot be recovered * is a replaceable transport, and state that dies with it cannot be recovered
* when it drops. * when it drops.
@@ -25,9 +25,8 @@ public final class PlayerStream {
final String playerIp; final String playerIp;
final int playerPort; final int playerPort;
/** The conn currently carrying this stream, and its id there. */ /** The conn currently carrying this player. */
WorkerConn worker; WorkerConn worker;
int sid;
/** Budget for player -> client DATA. */ /** Budget for player -> client DATA. */
int sendWnd; int sendWnd;
@@ -39,11 +38,11 @@ public final class PlayerStream {
// Pause reasons. Vert.x pause() is a flag rather than a counter, so a socket // Pause reasons. Vert.x pause() is a flag rather than a counter, so a socket
// can be paused for several reasons at once and must only be resumed once // can be paused for several reasons at once and must only be resumed once
// none of them hold — see WorkerConn#maybeResumePlayer. // none of them hold — see WorkerConn#maybeResumePlayer.
boolean pausedForWindow; // this stream's send window is exhausted boolean pausedForWindow; // this tunnel's send window is exhausted
boolean pausedForAggregate; // the shared worker socket's write queue is full boolean pausedForAggregate; // the worker socket's write queue is full
boolean parked; // the worker conn died; hanging until a reattach boolean parked; // the worker conn died; hanging until a reattach
/** Whether the conn carrying this stream negotiated resumption (§7.5). */ /** Whether the conn carrying this player negotiated resumption (§7.5). */
boolean resumable; boolean resumable;
// Resumption bookkeeping (§7.5). Three distinct offsets, and conflating them // Resumption bookkeeping (§7.5). Three distinct offsets, and conflating them
@@ -64,7 +63,7 @@ public final class PlayerStream {
long graceDeadline; long graceDeadline;
long timerId = -1; long timerId = -1;
PlayerStream(PendingPlayer p, WorkerConn worker, int sid, int sendWnd) { PlayerStream(PendingPlayer p, WorkerConn worker, int sendWnd) {
this.cid = p.getCid(); this.cid = p.getCid();
this.cidHex = p.getCidHex(); this.cidHex = p.getCidHex();
this.player = p.getSocket(); this.player = p.getSocket();
@@ -72,7 +71,6 @@ public final class PlayerStream {
this.playerIp = p.getPlayerIp(); this.playerIp = p.getPlayerIp();
this.playerPort = p.getPlayerPort(); this.playerPort = p.getPlayerPort();
this.worker = worker; this.worker = worker;
this.sid = sid;
this.sendWnd = sendWnd; this.sendWnd = sendWnd;
} }
@@ -81,7 +79,7 @@ public final class PlayerStream {
return !pausedForWindow && !pausedForAggregate && !parked; return !pausedForWindow && !pausedForAggregate && !parked;
} }
/** Roughly how much this stream holds while parked, for the hub-wide cap. */ /** Roughly how much this tunnel holds while parked, for the hub-wide cap. */
int parkedBytes() { int parkedBytes() {
return unacked.length() + (pendingUp != null ? pendingUp.length() : 0); return unacked.length() + (pendingUp != null ? pendingUp.length() : 0);
} }
@@ -26,22 +26,19 @@ public final class Protocol {
public static final int REGISTER_OK = 0x00; public static final int REGISTER_OK = 0x00;
public static final int REGISTER_ERR_PATTERN = 0x01; // pattern is not a valid regular expression public static final int REGISTER_ERR_PATTERN = 0x01; // pattern is not a valid regular expression
// Worker-conn mux frame types // Worker-conn frame types (one player per conn; no stream id).
public static final int MUX_SYN = 0x00; public static final int MUX_SYN = 0x00;
public static final int MUX_DATA = 0x01; public static final int MUX_DATA = 0x01;
public static final int MUX_FIN = 0x02; public static final int MUX_FIN = 0x02;
public static final int MUX_RST = 0x03; public static final int MUX_RST = 0x03;
public static final int MUX_WND = 0x04; // per-stream flow-control credit grant public static final int MUX_WND = 0x04; // per-connection flow-control credit grant
public static final int MUX_PING = 0x05; // liveness probe, StreamID 0 public static final int MUX_PING = 0x05; // liveness probe
public static final int MUX_PONG = 0x06; // liveness reply, echoes the nonce public static final int MUX_PONG = 0x06; // liveness reply, echoes the nonce
/** Reattach a parked stream to this conn: CID + the client's accepted offset (§7.5). */ /** Reattach a parked player to this conn: CID + the client's accepted offset (§7.5). */
public static final int MUX_RESUME = 0x07; public static final int MUX_RESUME = 0x07;
/** Hub's answer to RESUME: its accepted offset plus a freshly minted CID. */ /** Hub's answer to RESUME: its accepted offset plus a freshly minted CID. */
public static final int MUX_RESUME_ACK = 0x08; public static final int MUX_RESUME_ACK = 0x08;
/** Reserved stream id for connection-scoped mux frames (PING/PONG). Streams start at 1. */
public static final int MUX_CTL_SID = 0;
// RST reason codes (optional trailing byte; absence means "unspecified"). // RST reason codes (optional trailing byte; absence means "unspecified").
// Distinguishing them matters for resume: "unknown stream" is terminal, // Distinguishing them matters for resume: "unknown stream" is terminal,
// "already bound" means a racing attempt won and this one must not tear down. // "already bound" means a racing attempt won and this one must not tear down.
@@ -55,18 +52,18 @@ public final class Protocol {
// Session-establishment feature flags (trailing VarInt on the Rekey message, // Session-establishment feature flags (trailing VarInt on the Rekey message,
// echoed after the SessionReady type byte when accepted). // echoed after the SessionReady type byte when accepted).
public static final int FLAG_STREAM_FC = 0x01; public static final int FLAG_STREAM_FC = 0x01;
/** Mux-level PING/PONG on worker conns, so a silently dropped path is detected. */ /** Connection-level PING/PONG on worker conns, so a silently dropped path is detected. */
public static final int FLAG_WORKER_HEARTBEAT = 0x02; public static final int FLAG_WORKER_HEARTBEAT = 0x02;
/** /**
* Stream resumption (§7.5): on a worker-conn drop the hub hangs the player * Stream resumption (§7.5): on a worker-conn drop the hub hangs the player
* socket instead of closing it, and the client reattaches the stream * socket instead of closing it, and the client reattaches that player
* byte-exactly over a fresh conn. When accepted, the hub appends its resume * byte-exactly over a fresh conn. When accepted, the hub appends its resume
* grace period to SessionReady so the client can bound its own retry budget * grace period to SessionReady so the client can bound its own retry budget
* against it. * against it.
*/ */
public static final int FLAG_STREAM_RESUME = 0x04; public static final int FLAG_STREAM_RESUME = 0x04;
// Per-stream flow-control window bounds (bytes). // Per-connection flow-control window bounds (bytes).
public static final int DEFAULT_STREAM_WINDOW = 256 * 1024; public static final int DEFAULT_STREAM_WINDOW = 256 * 1024;
public static final int MIN_STREAM_WINDOW = 32 * 1024; public static final int MIN_STREAM_WINDOW = 32 * 1024;
public static final int MAX_STREAM_WINDOW = 8 << 20; public static final int MAX_STREAM_WINDOW = 8 << 20;
@@ -8,29 +8,26 @@ import lombok.RequiredArgsConstructor;
import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger; import org.apache.logging.log4j.Logger;
import java.util.HashMap;
import java.util.Map;
/** /**
* An authenticated worker connection (Magic 0x02). Multiplexes many player * An authenticated worker connection (Magic 0x02). Carries exactly one player:
* streams; the client opens streams via SYN(CID) to take over pending players. * the TCP connection is the tunnel (PROTOCOL.md §7). The client binds it with
* SYN(CID) (or RESUME) after SessionReady.
* *
* <p>Every stream has a credit window in both directions (PROTOCOL.md §7.3), * <p>The connection has a credit window in both directions (PROTOCOL.md §7.3),
* so one slow player only ever stalls its own stream — the shared worker * so a slow player only ever stalls itself.
* socket is never paused because of a single stream.
*/ */
@RequiredArgsConstructor @RequiredArgsConstructor
public final class WorkerConn { public final class WorkerConn {
private static final Logger LOG = LogManager.getLogger("redapricot.worker"); private static final Logger LOG = LogManager.getLogger("redapricot.worker");
/** Cap on a single DATA frame so no stream monopolizes the shared link for long. */ /** Cap on a single DATA frame so one write cannot occupy the link for long. */
private static final int CHUNK = 32 * 1024; private static final int CHUNK = 32 * 1024;
private final Hub hub; private final Hub hub;
private final EncryptedFrames frames; private final EncryptedFrames frames;
private final String id; private final String id;
private final int sendWndInit; // client's advertised per-stream receive window (our send budget) private final int sendWndInit; // client's advertised receive window (our send budget)
private final int recvWndInit; // our advertised per-stream receive window (basis for credit grants) private final int recvWndInit; // our advertised receive window (basis for credit grants)
/** /**
* Whether this conn negotiated stream resumption. Gates the park path: a * Whether this conn negotiated stream resumption. Gates the park path: a
* client that will never reattach is better served by an immediate close than * client that will never reattach is better served by an immediate close than
@@ -38,46 +35,52 @@ public final class WorkerConn {
*/ */
private final boolean resume; private final boolean resume;
private final Map<Integer, PlayerStream> streams = new HashMap<>(); /** The one player bound to this conn, or null until SYN/RESUME. */
private PlayerStream stream;
private boolean workerDrainArmed = false; // whether the worker socket's single drainHandler is set private boolean workerDrainArmed = false; // whether the worker socket's drainHandler is set
public void onFrame(byte[] payload) { public void onFrame(byte[] payload) {
ProtoReader r = new ProtoReader(payload); ProtoReader r = new ProtoReader(payload);
int type = r.readUByte(); int type = r.readUByte();
int sid = r.readVarInt();
switch (type) { switch (type) {
case Protocol.MUX_SYN -> handleSyn(sid, r.readBytes(Protocol.CID_LEN)); case Protocol.MUX_SYN -> handleSyn(r.readBytes(Protocol.CID_LEN));
case Protocol.MUX_DATA -> handleData(sid, r.readBuffer(r.remaining())); case Protocol.MUX_DATA -> handleData(r.readBuffer(r.remaining()));
case Protocol.MUX_WND -> handleWnd(sid, r.readVarInt()); case Protocol.MUX_WND -> handleWnd(r.readVarInt());
case Protocol.MUX_RESUME -> case Protocol.MUX_RESUME ->
handleResume(sid, r.readBytes(Protocol.CID_LEN), r.readI64(), r.readI64()); handleResume(r.readBytes(Protocol.CID_LEN), r.readI64(), r.readI64());
case Protocol.MUX_FIN, Protocol.MUX_RST -> closeStream(sid); case Protocol.MUX_FIN, Protocol.MUX_RST -> closeBound();
case Protocol.MUX_PING -> sendPong(r.readI64()); case Protocol.MUX_PING -> sendPong(r.readI64());
case Protocol.MUX_PONG -> { /* liveness only; arrival is what matters */ } case Protocol.MUX_PONG -> { /* liveness only; arrival is what matters */ }
case Protocol.FRAME_ERROR -> LOG.warn("worker {} error frame", id); case Protocol.FRAME_ERROR -> LOG.warn("worker {} error frame", id);
default -> LOG.warn("worker {} unknown mux type {}", id, type); default -> LOG.warn("worker {} unknown frame type {}", id, type);
} }
} }
private void handleSyn(int sid, byte[] cid) { private void handleSyn(byte[] cid) {
if (stream != null) {
LOG.warn("worker {} SYN on an already-bound conn; closing", id);
sendRst(Protocol.RST_ALREADY_BOUND);
frames.close();
return;
}
PendingPlayer p = hub.takePending(cid); PendingPlayer p = hub.takePending(cid);
if (p == null) { if (p == null) {
LOG.warn("worker {} SYN for unknown CID", id); LOG.warn("worker {} SYN for unknown CID", id);
sendRst(sid, Protocol.RST_UNKNOWN_STREAM); sendRst(Protocol.RST_UNKNOWN_STREAM);
return; return;
} }
PlayerStream st = new PlayerStream(p, this, sid, sendWndInit); PlayerStream st = new PlayerStream(p, this, sendWndInit);
st.resumable = resume; st.resumable = resume;
streams.put(sid, st); stream = st;
hub.addStream(st); hub.addStream(st);
// From now on the player socket belongs to this stream. The handlers are // From now on the player socket belongs to this tunnel. The handlers are
// installed once and route through the hub, which dispatches to whichever // installed once and route through the hub, which dispatches to whichever
// conn currently carries the stream. // conn currently carries the player.
// //
// They must not call this conn's methods directly: a lambda defined here // They must not call this conn's methods directly: a lambda defined here
// captures `this`, so after the stream moves to another conn it would keep // captures `this`, so after the player moves to another conn it would keep
// writing into the dead one's transport, where sends are silently dropped // writing into the dead one's transport, where sends are silently dropped
// and the player goes mute with nothing logged. Re-installing handlers on // and the player goes mute with nothing logged. Re-installing handlers on
// every reattach would be the other option, but a Vert.x socket resumed // every reattach would be the other option, but a Vert.x socket resumed
@@ -91,17 +94,17 @@ public final class WorkerConn {
sendUpstream(st, p.getBuffered()); sendUpstream(st, p.getBuffered());
maybeResumePlayer(st); maybeResumePlayer(st);
checkAggregate(st); checkAggregate(st);
LOG.info("worker {} stream {} bound to {}", id, sid, st.pattern); LOG.info("worker {} bound to {}", id, st.pattern);
} }
/** Player bytes arrived on a stream this conn currently carries. */ /** Player bytes arrived on the player this conn currently carries. */
void playerData(PlayerStream st, Buffer buf) { void playerData(PlayerStream st, Buffer buf) {
sendUpstream(st, buf); sendUpstream(st, buf);
checkAggregate(st); checkAggregate(st);
} }
/** /**
* Send player bytes to the client, chunked and clipped to the stream window; * Send player bytes to the client, chunked and clipped to the send window;
* the overflow is parked in {@code pendingUp} and the player socket paused * the overflow is parked in {@code pendingUp} and the player socket paused
* until the client grants more credit. * until the client grants more credit.
*/ */
@@ -120,7 +123,7 @@ public final class WorkerConn {
} }
} }
/** Send from {@code buf[off..]} within the stream window, chunked; returns the new offset. */ /** Send from {@code buf[off..]} within the send window, chunked; returns the new offset. */
private int drainUpstream(PlayerStream st, Buffer buf, int off) { private int drainUpstream(PlayerStream st, Buffer buf, int off) {
while (off < buf.length() && st.sendWnd > 0) { while (off < buf.length() && st.sendWnd > 0) {
int n = Math.min(Math.min(CHUNK, st.sendWnd), buf.length() - off); int n = Math.min(Math.min(CHUNK, st.sendWnd), buf.length() - off);
@@ -134,16 +137,16 @@ public final class WorkerConn {
st.unacked.append(chunk); st.unacked.append(chunk);
} }
st.sentOffset += n; st.sentOffset += n;
sendData(st.sid, chunk); sendData(chunk);
st.sendWnd -= n; st.sendWnd -= n;
off += n; off += n;
} }
return off; return off;
} }
/** The client granted {@code delta} more bytes of credit on a stream. */ /** The client granted {@code delta} more bytes of credit. */
private void handleWnd(int sid, int delta) { private void handleWnd(int delta) {
PlayerStream st = streams.get(sid); PlayerStream st = stream;
if (st == null || delta <= 0) return; if (st == null || delta <= 0) return;
// The running sum doubles as the acked offset: the client grants credit // The running sum doubles as the acked offset: the client grants credit
// exactly as bytes reach the destination socket, so a credited byte can // exactly as bytes reach the destination socket, so a credited byte can
@@ -160,13 +163,13 @@ public final class WorkerConn {
checkAggregate(st); checkAggregate(st);
} }
private void handleData(int sid, Buffer data) { private void handleData(Buffer data) {
PlayerStream st = streams.get(sid); PlayerStream st = stream;
if (st == null) return; if (st == null) return;
// Never pause the shared socket: the client bounds what it sends per // Never pause the worker socket: the client bounds what it sends to our
// stream to our advertised window, so a slow player only piles up a // advertised window, so a slow player only piles up a bounded amount in
// bounded amount in its own write queue; credit is granted back as the // its own write queue; credit is granted back as the write completes
// write completes (i.e. the bytes reached the player socket). // (i.e. the bytes reached the player socket).
int len = data.length(); int len = data.length();
// Accepted the moment the bytes are taken off the wire, not when the write // Accepted the moment the bytes are taken off the wire, not when the write
// completes. Completion is asynchronous and suppressed once the connection // completes. Completion is asynchronous and suppressed once the connection
@@ -176,7 +179,7 @@ public final class WorkerConn {
st.acceptedOffset += len; st.acceptedOffset += len;
st.player.write(data).onComplete(ar -> { st.player.write(data).onComplete(ar -> {
if (ar.failed()) return; if (ar.failed()) return;
// Both counters advance even if this conn has since died or the stream // Both counters advance even if this conn has since died or the player
// has moved on. Discarding them would destroy up to half a window of // has moved on. Discarding them would destroy up to half a window of
// credit per outage, and — worse — leave the delivered offset that a // credit per outage, and — worse — leave the delivered offset that a
// reattach restates the window from permanently short. // reattach restates the window from permanently short.
@@ -186,28 +189,34 @@ public final class WorkerConn {
if (st.credited * 2 >= recvWndInit) { if (st.credited * 2 >= recvWndInit) {
int delta = st.credited; int delta = st.credited;
st.credited = 0; st.credited = 0;
sendWnd(st.sid, delta); sendWnd(delta);
} }
}); });
} }
/** /**
* Reattach a parked stream to this connection (§7.5). * Reattach a parked player to this connection (§7.5).
* *
* <p>Runs to completion in one event-loop turn — rebind, acknowledge, replay — * <p>Runs to completion in one event-loop turn — rebind, acknowledge, replay —
* so the hub's single-threaded model makes the ordering race-free by * so the hub's single-threaded model makes the ordering race-free by
* construction, with no interleaving of live and replayed bytes to reason * construction, with no interleaving of live and replayed bytes to reason
* about. * about.
*/ */
private void handleResume(int sid, byte[] cid, long clientAccepted, long clientDelivered) { private void handleResume(byte[] cid, long clientAccepted, long clientDelivered) {
if (stream != null) {
LOG.warn("worker {} RESUME on an already-bound conn; closing", id);
sendRst(Protocol.RST_ALREADY_BOUND);
frames.close();
return;
}
PlayerStream st = hub.takeParked(cid); PlayerStream st = hub.takeParked(cid);
if (st == null) { if (st == null) {
// Tell a stream we have never heard of apart from one that is still // Tell a player we have never heard of apart from one that is still
// bound: the first is terminal for the client, the second means a // bound: the first is terminal for the client, the second means a
// racing attempt won and this one must leave the player alone. // racing attempt won and this one must leave the player alone.
boolean bound = hub.streamByCid(cid) != null; boolean bound = hub.streamByCid(cid) != null;
LOG.warn("worker {} RESUME for {} CID", id, bound ? "still-bound" : "unknown"); LOG.warn("worker {} RESUME for {} CID", id, bound ? "still-bound" : "unknown");
sendRst(sid, bound ? Protocol.RST_ALREADY_BOUND : Protocol.RST_UNKNOWN_STREAM); sendRst(bound ? Protocol.RST_ALREADY_BOUND : Protocol.RST_UNKNOWN_STREAM);
return; return;
} }
@@ -222,16 +231,15 @@ public final class WorkerConn {
if (replay == null) { if (replay == null) {
LOG.warn("worker {} RESUME at offset {} outside the retained region [{}, {}]; closing player", LOG.warn("worker {} RESUME at offset {} outside the retained region [{}, {}]; closing player",
id, clientAccepted, st.unacked.base(), st.unacked.end()); id, clientAccepted, st.unacked.base(), st.unacked.end());
sendRst(sid, Protocol.RST_UNKNOWN_STREAM); sendRst(Protocol.RST_UNKNOWN_STREAM);
hub.removeStream(st); hub.removeStream(st);
st.player.close(); st.player.close();
return; return;
} }
st.worker = this; st.worker = this;
st.sid = sid;
st.resumable = resume; st.resumable = resume;
streams.put(sid, st); stream = st;
// Restate the window rather than patching it. Three offsets, three jobs: // Restate the window rather than patching it. Three offsets, three jobs:
// the replay above is measured from what the client *accepted*, the window // the replay above is measured from what the client *accepted*, the window
@@ -253,7 +261,7 @@ public final class WorkerConn {
byte[] newCid = hub.newCid(); byte[] newCid = hub.newCid();
hub.rekeyStream(st, newCid); hub.rekeyStream(st, newCid);
frames.send(new ProtoWriter() frames.send(new ProtoWriter()
.u8(Protocol.MUX_RESUME_ACK).varInt(sid) .u8(Protocol.MUX_RESUME_ACK)
.i64(st.acceptedOffset) .i64(st.acceptedOffset)
.i64(st.deliveredOffset) .i64(st.deliveredOffset)
.bytes(newCid) .bytes(newCid)
@@ -264,7 +272,7 @@ public final class WorkerConn {
// drainUpstream would do. // drainUpstream would do.
for (int off = 0; off < replay.length(); off += CHUNK) { for (int off = 0; off < replay.length(); off += CHUNK) {
int end = Math.min(off + CHUNK, replay.length()); int end = Math.min(off + CHUNK, replay.length());
sendData(sid, replay.getBytes(off, end)); sendData(replay.getBytes(off, end));
} }
if (st.pendingUp != null) { if (st.pendingUp != null) {
@@ -275,12 +283,13 @@ public final class WorkerConn {
if (st.pendingUp == null) st.pausedForWindow = false; if (st.pendingUp == null) st.pausedForWindow = false;
maybeResumePlayer(st); maybeResumePlayer(st);
checkAggregate(st); checkAggregate(st);
LOG.info("worker {} stream {} resumed ({} bytes replayed, {} outstanding)", LOG.info("worker {} resumed ({} bytes replayed, {} outstanding)",
id, sid, replay.length(), outstanding); id, replay.length(), outstanding);
} }
private void closeStream(int sid) { private void closeBound() {
PlayerStream st = streams.remove(sid); PlayerStream st = stream;
stream = null;
if (st != null) { if (st != null) {
st.worker = null; st.worker = null;
hub.removeStream(st); hub.removeStream(st);
@@ -288,7 +297,7 @@ public final class WorkerConn {
} }
} }
/** Park the player if the shared worker socket's write queue is congested. */ /** Park the player if the worker socket's write queue is congested. */
private void checkAggregate(PlayerStream st) { private void checkAggregate(PlayerStream st) {
if (!st.pausedForAggregate && frames.writeQueueFull()) { if (!st.pausedForAggregate && frames.writeQueueFull()) {
st.pausedForAggregate = true; st.pausedForAggregate = true;
@@ -297,17 +306,16 @@ public final class WorkerConn {
} }
} }
/** Register (once) the shared worker socket's single drain handler; on drain, wake parked players. */ /** Register (once) the worker socket's drain handler; on drain, wake the player. */
private void armWorkerDrain() { private void armWorkerDrain() {
if (workerDrainArmed) return; if (workerDrainArmed) return;
workerDrainArmed = true; workerDrainArmed = true;
frames.socket().drainHandler(v -> { frames.socket().drainHandler(v -> {
workerDrainArmed = false; workerDrainArmed = false;
for (PlayerStream st : streams.values()) { PlayerStream st = stream;
if (!st.pausedForAggregate) continue; if (st == null || !st.pausedForAggregate) return;
st.pausedForAggregate = false; st.pausedForAggregate = false;
maybeResumePlayer(st); maybeResumePlayer(st);
}
}); });
} }
@@ -322,53 +330,56 @@ public final class WorkerConn {
if (st.shouldFlow()) st.player.resume(); if (st.shouldFlow()) st.player.resume();
} }
/** The player side of a stream this conn carries vanished: unbind it and FIN the client. */ /** The player side of the tunnel this conn carries vanished: unbind it and FIN the client. */
void playerGone(PlayerStream st) { void playerGone(PlayerStream st) {
boolean wasLive = streams.remove(st.sid) == st; boolean wasLive = stream == st;
if (wasLive) stream = null;
st.worker = null; st.worker = null;
if (wasLive) sendFin(st.sid); if (wasLive) sendFin();
} }
private void sendData(int sid, byte[] data) { private void sendData(byte[] data) {
frames.send(new ProtoWriter().u8(Protocol.MUX_DATA).varInt(sid).bytes(data).toBytes()); frames.send(new ProtoWriter().u8(Protocol.MUX_DATA).bytes(data).toBytes());
} }
private void sendFin(int sid) { private void sendFin() {
frames.send(new ProtoWriter().u8(Protocol.MUX_FIN).varInt(sid).toBytes()); frames.send(new ProtoWriter().u8(Protocol.MUX_FIN).toBytes());
} }
/** The reason is a trailing byte, optional on the wire; peers that predate it send none. */ /** The reason is a trailing byte, optional on the wire; peers that predate it send none. */
private void sendRst(int sid, int reason) { private void sendRst(int reason) {
frames.send(new ProtoWriter().u8(Protocol.MUX_RST).varInt(sid).u8(reason).toBytes()); frames.send(new ProtoWriter().u8(Protocol.MUX_RST).u8(reason).toBytes());
} }
private void sendWnd(int sid, int delta) { private void sendWnd(int delta) {
frames.send(new ProtoWriter().u8(Protocol.MUX_WND).varInt(sid).varInt(delta).toBytes()); frames.send(new ProtoWriter().u8(Protocol.MUX_WND).varInt(delta).toBytes());
} }
/** Answer the client's liveness probe, echoing its nonce. */ /** Answer the client's liveness probe, echoing its nonce. */
private void sendPong(long nonce) { private void sendPong(long nonce) {
frames.send(new ProtoWriter().u8(Protocol.MUX_PONG).varInt(Protocol.MUX_CTL_SID).i64(nonce).toBytes()); frames.send(new ProtoWriter().u8(Protocol.MUX_PONG).i64(nonce).toBytes());
} }
/** /**
* Only the tunnel leg died. Where the session negotiated resumption the * Only the tunnel leg died. Where the session negotiated resumption the
* player sockets are hung rather than closed, and wait for the client to * player socket is hung rather than closed, and waits for the client to
* reattach their streams over a fresh conn (§7.5); otherwise this is the old, * reattach over a fresh conn (§7.5); otherwise this is the old,
* unconditional close. * unconditional close.
*/ */
public void onClose() { public void onClose() {
int parked = 0; PlayerStream st = stream;
for (PlayerStream st : streams.values()) { stream = null;
st.worker = null; if (st == null) {
if (hub.park(st)) { LOG.info("worker {} closed (unbound)", id);
parked++; return;
} else { }
hub.removeStream(st); st.worker = null;
st.player.close(); if (hub.park(st)) {
} LOG.info("worker {} closed (player hung for reattach)", id);
} else {
hub.removeStream(st);
st.player.close();
LOG.info("worker {} closed (player dropped)", id);
} }
LOG.info("worker {} closed ({} of {} player(s) hung for reattach)", id, parked, streams.size());
streams.clear();
} }
} }
@@ -79,7 +79,8 @@ class CryptoCodecTest {
private static Hub testHub() { private static Hub testHub() {
return new Hub(null, new Config("0.0.0.0", 25565, "test-psk", 30_000L, 10_000L, return new Hub(null, new Config("0.0.0.0", 25565, "test-psk", 30_000L, 10_000L,
Protocol.DEFAULT_STREAM_WINDOW, 90_000L, Protocol.DEFAULT_STREAM_WINDOW, 90_000L,
true, 20_000L, 256, 256L * 2 * Protocol.DEFAULT_STREAM_WINDOW, 0L, 15_000L)); true, 20_000L, 256, 256L * 2 * Protocol.DEFAULT_STREAM_WINDOW, 0L, 15_000L,
0, 1, 0));
} }
private static ControlSession testSession(Hub hub, String id) { private static ControlSession testSession(Hub hub, String id) {
@@ -0,0 +1,86 @@
package io.icybear.redapricot;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
class IpRateLimiterTest {
@Test
void burstThenRefill() {
IpRateLimiter lim = new IpRateLimiter(2, 2, 0);
long t = 1_000;
assertNull(lim.admit("1.1.1.1", t));
assertNull(lim.admit("1.1.1.1", t));
assertEquals(IpRateLimiter.Deny.RATE, lim.admit("1.1.1.1", t));
// 500 ms at 2/s = 1 token.
assertNull(lim.admit("1.1.1.1", t + 500));
assertEquals(IpRateLimiter.Deny.RATE, lim.admit("1.1.1.1", t + 500));
}
@Test
void concurrentCapIndependentOfRate() {
IpRateLimiter lim = new IpRateLimiter(0, 16, 1);
long t = 1_000;
assertNull(lim.admit("10.0.0.1", t));
assertEquals(1, lim.concurrent("10.0.0.1"));
assertEquals(IpRateLimiter.Deny.CONCURRENT, lim.admit("10.0.0.1", t));
lim.release("10.0.0.1", t);
assertEquals(0, lim.concurrent("10.0.0.1"));
assertNull(lim.admit("10.0.0.1", t));
}
@Test
void ipsAreIndependent() {
IpRateLimiter lim = new IpRateLimiter(1, 1, 1);
long t = 1_000;
assertNull(lim.admit("a", t));
assertNull(lim.admit("b", t));
assertEquals(IpRateLimiter.Deny.RATE, lim.admit("a", t));
assertEquals(IpRateLimiter.Deny.RATE, lim.admit("b", t));
}
@Test
void bothOffIsNoOp() {
IpRateLimiter lim = new IpRateLimiter(0, 16, 0);
long t = 1_000;
for (int i = 0; i < 100; i++) {
assertNull(lim.admit("1.2.3.4", t));
}
assertEquals(0, lim.bucketCount());
}
@Test
void sweepDropsIdleEmptyBuckets() {
IpRateLimiter lim = new IpRateLimiter(8, 8, 64);
long t = 1_000;
assertNull(lim.admit("9.9.9.9", t));
lim.release("9.9.9.9", t);
assertEquals(1, lim.bucketCount());
lim.sweep(t + IpRateLimiter.SWEEP_MS - 1);
assertEquals(1, lim.bucketCount());
lim.sweep(t + IpRateLimiter.SWEEP_MS);
assertEquals(0, lim.bucketCount());
}
@Test
void sweepKeepsLiveBuckets() {
IpRateLimiter lim = new IpRateLimiter(8, 8, 64);
long t = 1_000;
assertNull(lim.admit("9.9.9.9", t));
lim.sweep(t + IpRateLimiter.SWEEP_MS * 2);
assertEquals(1, lim.bucketCount());
assertEquals(1, lim.concurrent("9.9.9.9"));
}
@Test
void denyDoesNotConsumeASlot() {
IpRateLimiter lim = new IpRateLimiter(1, 1, 8);
long t = 1_000;
assertNull(lim.admit("x", t));
assertNotNull(lim.admit("x", t));
assertEquals(1, lim.concurrent("x"));
}
}