break: replace muxed workers with 1:1 tunnels
Worker frames are now FrameType + payload; there is no stream id. Each player gets its own worker conn. maxTunnels (default 256) caps concurrent tunnels. The old maxConn pool size is ignored so existing configs do not silently admit only a handful of players. Resume, per-direction windows, the control session, and the DATA-only shaper stay. A dropped worker still hangs that one player and reattaches over a fresh conn. Add a hub-side per-IP limiter for player intents only (default 8/s, burst 16, 64 concurrent). Unmatched hostnames consume a token; Intent 17 is never counted. 0 disables each knob.
This commit is contained in:
iceBear67
@@ -57,7 +57,7 @@ they exercise, so run them a few times rather than trusting a single pass.
|
||||
| First handshake | Becomes |
|
||||
|---|---|
|
||||
| `Intent 17`, rekey magic `0x01` | **control session** — pattern registration, `ControlRequest`, Ping/Pong |
|
||||
| `Intent 17`, rekey magic `0x02` | **worker conn** — multiplexed player streams |
|
||||
| `Intent 17`, rekey magic `0x02` | **worker conn** — one player, 1:1 |
|
||||
| any other intent | **player** — hostname regex-matched, then tunneled |
|
||||
|
||||
Session establishment (both kinds): plaintext handshake whose `Server Address` is
|
||||
@@ -69,8 +69,8 @@ from `rand‖ts` → hub replies `SessionReady` echoing accepted flags and its w
|
||||
|
||||
The hub pauses and buffers a matched player socket, mints a random 16-byte **CID**, and
|
||||
sends `ControlRequest(CID, matchedPattern, ip, port)` down the control session. The client
|
||||
allocates a worker stream, `SYN`s it with that CID, dials the destination, and the hub
|
||||
binds the pending player to that stream. The CID's secrecy (it only ever travels encrypted
|
||||
dials a dedicated worker conn, `SYN`s it with that CID, dials the destination, and the hub
|
||||
binds the pending player to that conn. The CID's secrecy (it only ever travels encrypted
|
||||
over the control session) is what authorizes the takeover — there is no other client
|
||||
identity check. The hub echoes **the matched pattern string, not the player's hostname**,
|
||||
so the client can look it up directly in its route table; the buffered handshake is
|
||||
@@ -80,10 +80,10 @@ forwarded verbatim so the backend sees the original hostname.
|
||||
|
||||
- `server/src/main/java/io/icybear/redapricot/` — `HubConnection` (per-socket state machine:
|
||||
handshake parse → dispatch → rekey → control/worker/player), `Hub` (pattern registry, CID
|
||||
table, pending players), `ControlSession`, `WorkerConn` (mux demux + per-stream state),
|
||||
table, pending players), `ControlSession`, `WorkerConn` (1:1 tunnel + flow-control state),
|
||||
`net/EncryptedFrames`, `crypto/Crypto`, `util/` (VarInt, ProtoReader/Writer, Hex).
|
||||
- `client/` — `client.go` (control session, reconnect, dispatch), `worker.go` (pool,
|
||||
allocation, `WorkerConn`, `Stream` and its two goroutines), `shaper.go` (egress rate cap),
|
||||
- `client/` — `client.go` (control session, reconnect, dispatch), `worker.go` (1:1
|
||||
dial, `WorkerConn`, `Stream` and its two goroutines), `shaper.go` (egress rate cap),
|
||||
`velocity.go` (Velocity modern-forwarding interception), `proxyproto.go` (HAProxy v2),
|
||||
`config.go` (config + all protocol constants), `wire/` (VarInt/MC codec, SHA3+ChaCha20,
|
||||
`FramedConn`).
|
||||
@@ -95,10 +95,10 @@ forwarded verbatim so the backend sees the original hostname.
|
||||
per-connection state are touched by one event loop and need no locking. Never introduce a
|
||||
blocking call there.
|
||||
- **Client:** one goroutine reads each connection; `WriteFrame` is mutex-serialized. Each
|
||||
stream has exactly two goroutines — `run` (destination → hub) and `writeLoop` (the *only*
|
||||
tunnel has exactly two goroutines — `run` (destination → hub) and `writeLoop` (the *only*
|
||||
writer to the destination, draining a queue fed by the worker readLoop). The readLoop must
|
||||
never write to a destination, or a stalled backend blocks frame dispatch for every other
|
||||
stream on that conn.
|
||||
never write to a destination: WND is granted only after the dest write completes, and a
|
||||
stalled backend must not stall heartbeat / FIN dispatch on that conn.
|
||||
|
||||
## Invariants to preserve when editing
|
||||
|
||||
@@ -111,25 +111,28 @@ forwarded verbatim so the backend sees the original hostname.
|
||||
`‖ 0x02` s2c). Both directions must never share a keystream. Go `x/crypto/chacha20` and
|
||||
Java JCE `ChaCha20` are byte-identical here, and unit tests on both sides pin the same
|
||||
SHA3-224 vector — keep that pinning if you touch crypto.
|
||||
- **Per-stream flow control is mandatory.** The hub rejects a session whose rekey lacks
|
||||
- **Per-connection flow control is mandatory.** The hub rejects a session whose rekey lacks
|
||||
`FLAG_STREAM_FC`; the client rejects a hub that does not echo it. Credit is granted back
|
||||
(`WND`) only as bytes are actually written to the terminal socket, batched at half-window.
|
||||
- **Only stream ids allocated by the client exist** (a per-conn counter starting at 1);
|
||||
sid `0` is reserved for connection-scoped `PING`/`PONG`.
|
||||
- **One worker conn carries one player.** There is no stream id. The first business frame
|
||||
after `SessionReady` is `SYN` or `RESUME`; a second bind on the same conn is a protocol
|
||||
violation. `PING`/`PONG` are connection-scoped frames.
|
||||
- **Only DATA is shaped** by `client/shaper.go`. Delaying `FIN`, `WND`, or `PONG` would trip
|
||||
the very liveness detection the heartbeat exists for. The shaper is client-local and
|
||||
invisible on the wire.
|
||||
- **The pool grows breadth-first** (`StreamsBeforeGrowing = 1`): dial up to `maxConn` before
|
||||
stacking streams, so one TCP connection is never the shared point of failure for every
|
||||
player. `SaturationThreshold = 8` only logs, once the pool is already at `maxConn`.
|
||||
A dial is never performed while holding the pool lock.
|
||||
- **Each player gets its own worker dial**, up to `maxTunnels` (default 256). A dial is
|
||||
never performed while holding the live-set lock: session establishment is network I/O,
|
||||
and one unresponsive hub must not block unrelated players.
|
||||
- **The hub IP limiter is player-only.** Intent 17 (control + workers) is never
|
||||
admitted through it — those sockets share the client's one address. Unmatched
|
||||
player hostnames still consume a token. `0` turns each knob off.
|
||||
- **Liveness is explicit everywhere:** every session heartbeats (drop after
|
||||
`3 × pingIntervalMs`), every socket write is bounded, establishment has a deadline. A
|
||||
silently blackholed path (NAT forgetting a flow, no FIN/RST) must recover without operator
|
||||
action — `TestBlackholedPathRecovers` guards this.
|
||||
- **Stream resumption is byte-exact or it is nothing** (`PROTOCOL.md §7.5`,
|
||||
`client/resume.go`, `WorkerConn.handleResume`). A worker-conn drop hangs the player and
|
||||
reattaches the stream over a fresh conn. Three offsets are tracked per direction and are
|
||||
reattaches over a fresh conn. Three offsets are tracked per direction and are
|
||||
*not* interchangeable: replay from the peer's **accepted** offset, restate the window from
|
||||
its **delivered** offset, and never size the window from **credited** — the grants in
|
||||
flight when the conn died are gone for good, and a window derived from them can be
|
||||
|
||||
+102
-91
@@ -15,7 +15,7 @@ There are three roles:
|
||||
| **Player** | any | An ordinary Minecraft client connecting through the hub. |
|
||||
|
||||
```
|
||||
Player ──MC──▶ Hub(server) ══WorkerConn(mux)══▶ Client ──MC──▶ Destination
|
||||
Player ──MC──▶ Hub(server) ══WorkerConn(1:1)══▶ Client ──MC──▶ Destination
|
||||
▲ registers patterns / receives control requests │
|
||||
└────────────── Control Session ────────────────────┘
|
||||
```
|
||||
@@ -154,18 +154,18 @@ RandLen : VarInt # 8 ≤ RandLen ≤ 64
|
||||
Rand : Bytes[RandLen] # cryptographically random
|
||||
Timestamp : I64 # client's epoch milliseconds
|
||||
Flags : VarInt # feature flags; bit 0x01 (STREAM_FC) MUST be set
|
||||
RecvWindow: VarInt # client's per-stream receive window, bytes (§7.3)
|
||||
RecvWindow: VarInt # client's per-connection receive window, bytes (§7.3)
|
||||
```
|
||||
|
||||
`Flags` is a bitfield of features.
|
||||
|
||||
| Bit | Name | Meaning |
|
||||
|-----|------|---------|
|
||||
| `0x01` | STREAM_FC | **Per-stream flow control** (§7.3). Mandatory. |
|
||||
| `0x02` | WORKER_HEARTBEAT | Mux-level `PING`/`PONG` on worker conns (§7.4). Optional. |
|
||||
| `0x01` | STREAM_FC | **Per-connection flow control** (§7.3). Mandatory. |
|
||||
| `0x02` | WORKER_HEARTBEAT | Connection-level `PING`/`PONG` on worker conns (§7.4). Optional. |
|
||||
| `0x04` | STREAM_RESUME | **Stream resumption** (§7.5): a worker-conn drop hangs the player rather than closing it. Optional. |
|
||||
|
||||
STREAM_FC is mandatory: `RecvWindow` advertises the client's per-stream receive
|
||||
STREAM_FC is mandatory: `RecvWindow` advertises the client's per-connection receive
|
||||
window in bytes and must be positive. The hub closes the connection if the flag
|
||||
is missing, `RecvWindow` is absent or non-positive, or the fields are malformed.
|
||||
|
||||
@@ -195,7 +195,7 @@ SessionReady : payload = [ 0x00, Flags: VarInt, RecvWindow: VarInt,
|
||||
```
|
||||
|
||||
The hub echoes the accepted flags (STREAM_FC set) followed by its own
|
||||
per-stream receive window. A client must reject a SessionReady without the
|
||||
per-connection receive window. A client must reject a SessionReady without the
|
||||
STREAM_FC flag or without a positive window (an unsupported hub).
|
||||
|
||||
`ResumeGraceMs` is present only when the hub accepts STREAM_RESUME, and states
|
||||
@@ -314,35 +314,36 @@ Msg : String
|
||||
|
||||
Purely informational; the receiver logs it.
|
||||
|
||||
## 7. Worker conn & multiplexing
|
||||
## 7. Worker conn
|
||||
|
||||
A **Worker Conn** (`Magic == 0x02`) carries player↔destination traffic for many
|
||||
players over one TCP connection using a minimal stream multiplexer. The unit of
|
||||
work is a **stream**. Stream ids are assigned by the **client** (the only side
|
||||
that opens streams), unique per worker conn, starting at 1 and increasing.
|
||||
A **Worker Conn** (`Magic == 0x02`) carries player↔destination traffic for
|
||||
**exactly one player**. The TCP connection *is* the tunnel: there is no stream
|
||||
id and no multiplexer. The client dials a fresh worker conn for each
|
||||
`ControlRequest` (and for each resumption attempt).
|
||||
|
||||
Each encrypted frame on a worker conn carries one **mux frame**:
|
||||
Each encrypted frame on a worker conn carries one **tunnel frame**:
|
||||
|
||||
```
|
||||
FrameType : u8
|
||||
StreamID : VarInt
|
||||
Data : Bytes[...] # remainder of the frame payload
|
||||
```
|
||||
|
||||
| FrameType | Name | Direction | Data |
|
||||
|-----------|------|-----------|------|
|
||||
| `0x00` | SYN | C → S | `CID: Bytes[16]` — open a stream to take over the pending player identified by CID. |
|
||||
| `0x01` | DATA | both | raw tunneled bytes for the stream. |
|
||||
| `0x02` | FIN | both | *(empty)* — graceful close of the stream (both directions). This is the "disconnect" the hub sends when the player leaves. |
|
||||
| `0x00` | SYN | C → S | `CID: Bytes[16]` — take over the pending player identified by CID. |
|
||||
| `0x01` | DATA | both | raw tunneled bytes. |
|
||||
| `0x02` | FIN | both | *(empty)* — graceful close (both directions). This is the "disconnect" the hub sends when the player leaves. |
|
||||
| `0x03` | RST | both | *(optional 1 byte reason)* — abnormal close (e.g. CID unknown/expired, destination dial failed). |
|
||||
| `0x04` | WND | both | `Delta: VarInt` — flow-control credit grant (§7.3). |
|
||||
| `0x05` | PING | both | `Nonce: I64` — liveness probe on reserved StreamID `0` (§7.4). |
|
||||
| `0x05` | PING | both | `Nonce: I64` — liveness probe (§7.4). |
|
||||
| `0x06` | PONG | both | `Nonce: I64` — echoes the probe's nonce (§7.4). |
|
||||
| `0x07` | RESUME | C → S | `CID: Bytes[16]`, `Accepted: I64`, `Delivered: I64` — reattach a hung stream to this conn (§7.5). |
|
||||
| `0x07` | RESUME | C → S | `CID: Bytes[16]`, `Accepted: I64`, `Delivered: I64` — reattach a hung player to this conn (§7.5). |
|
||||
| `0x08` | RESUME_ACK | S → C | `Accepted: I64`, `Delivered: I64`, `NewCID: Bytes[16]` — the reattach succeeded (§7.5). |
|
||||
|
||||
There is no explicit SYN-ACK: success is implied by the hub forwarding the
|
||||
buffered Handshake as the stream's first `DATA`; failure is an `RST`.
|
||||
The first business frame after `SessionReady` must be `SYN` or `RESUME`. A
|
||||
second bind on an already-bound conn is a protocol violation: the hub replies
|
||||
`RST` and closes. There is no explicit SYN-ACK: success is implied by the hub
|
||||
forwarding the buffered Handshake as the first `DATA`; failure is an `RST`.
|
||||
|
||||
`RST` reason codes. The byte remains optional — a peer that predates it sends
|
||||
none, and a receiver must tolerate its absence — but distinguishing the reasons
|
||||
@@ -358,25 +359,17 @@ took it" call for opposite responses.
|
||||
| `0x04` | FLOW_CONTROL | The peer exceeded its advertised window. |
|
||||
| `0x05` | DIAL_FAILED | The client could not reach the destination. |
|
||||
|
||||
### 7.1 Stream allocation (client side)
|
||||
### 7.1 Tunnel allocation (client side)
|
||||
|
||||
The client keeps a pool of `1 ≤ N ≤ max_conn` worker conns (`max_conn`
|
||||
configurable, `1..8`). The pool grows **breadth-first**: spreading streams over
|
||||
several connections keeps any single TCP connection from becoming the shared
|
||||
point of failure for every player on the tunnel. To place a new stream:
|
||||
The client dials one worker conn per player, up to a configurable
|
||||
`maxTunnels` cap (default 256, clamped to `[1, 4096]`). There is no pool and no
|
||||
least-loaded placement: a `ControlRequest` either gets its own TCP connection or
|
||||
is dropped (the hub then closes the player when `pendingTimeoutMs` fires).
|
||||
|
||||
1. Pick the worker conn with the **fewest active streams**, and use it.
|
||||
2. If that conn already carries at least one stream and
|
||||
`poolSize + dialsInFlight < max_conn`, dial another worker conn **in the
|
||||
background**. The stream just placed is not delayed by that dial; the new
|
||||
conn becomes the least-loaded one and picks up subsequent streams.
|
||||
3. Once the pool is at `max_conn`, streams stack on the least-loaded conn.
|
||||
Exceeding `8` active streams there is logged as pool saturation.
|
||||
|
||||
Only when the pool is *empty* does a caller dial synchronously, and then exactly
|
||||
one caller dials while the others wait for its result. A dial is never performed
|
||||
while holding the pool lock: session establishment is network I/O, and one
|
||||
unresponsive hub must not be able to block unrelated players.
|
||||
A dial is never performed while holding the live-set lock: session establishment
|
||||
is network I/O, and one unresponsive hub must not be able to block unrelated
|
||||
players. Each dial is independent and bounded by the session-establishment
|
||||
deadline (§7.4); callers are not serialized behind a single in-flight handshake.
|
||||
|
||||
### 7.2 End-to-end player flow
|
||||
|
||||
@@ -388,45 +381,43 @@ unresponsive hub must not be able to block unrelated players.
|
||||
control session. If no SYN arrives within `pendingTimeoutMs` (default 10000)
|
||||
the pending entry is dropped and the player socket closed.
|
||||
3. The client receives `ControlRequest`, looks up the destination for `Pattern`,
|
||||
allocates a worker conn + `StreamID`, and sends `SYN(StreamID, CID)`. In
|
||||
parallel it dials the destination and (if configured) writes a HAProxy v2
|
||||
header (§8) carrying `PlayerIP:PlayerPort`.
|
||||
dials a dedicated worker conn, and sends `SYN(CID)`. In parallel it dials the
|
||||
destination and (if configured) writes a HAProxy v2 header (§8) carrying
|
||||
`PlayerIP:PlayerPort`.
|
||||
4. The hub matches `CID` to the pending player, binds
|
||||
`(workerConn, StreamID) ↔ playerSocket`, forwards the buffered bytes as
|
||||
`DATA`, and resumes the player socket. Subsequent player bytes become `DATA`
|
||||
frames; `DATA` frames from the client are written to the player socket. If
|
||||
`CID` is unknown/expired the hub replies `RST`.
|
||||
5. When the player disconnects the hub sends `FIN` on the stream; the client
|
||||
closes the destination. When the destination closes, the client sends `FIN`;
|
||||
the hub closes the player socket. `RST` is treated the same way (hard close).
|
||||
`workerConn ↔ playerSocket`, forwards the buffered bytes as `DATA`, and
|
||||
resumes the player socket. Subsequent player bytes become `DATA` frames;
|
||||
`DATA` frames from the client are written to the player socket. If `CID` is
|
||||
unknown/expired the hub replies `RST`.
|
||||
5. When the player disconnects the hub sends `FIN`; the client closes the
|
||||
destination. When the destination closes, the client sends `FIN`; the hub
|
||||
closes the player socket. `RST` is treated the same way (hard close).
|
||||
|
||||
Data on a worker conn is subject to that TCP connection's back-pressure for
|
||||
its **aggregate** bandwidth; *per-stream* fairness is governed by the credit
|
||||
windows of §7.3.
|
||||
Data on a worker conn is subject to that TCP connection's back-pressure. Credit
|
||||
windows of §7.3 bound how many bytes may be in flight on that one tunnel.
|
||||
|
||||
### 7.3 Per-stream flow control
|
||||
### 7.3 Per-connection flow control
|
||||
|
||||
Every stream carries an independent credit window per direction:
|
||||
Every worker conn carries an independent credit window per direction:
|
||||
|
||||
* Each side advertised its **receive window** W (bytes) at session setup. A
|
||||
sender may have at most W un-credited DATA bytes outstanding per stream; the
|
||||
initial budget is W, spent as DATA is sent (`Data` length only — SYN/FIN/RST
|
||||
frames are free) starting with the very first DATA on the stream (including
|
||||
the hub's forwarded handshake).
|
||||
sender may have at most W un-credited DATA bytes outstanding; the initial
|
||||
budget is W, spent as DATA is sent (`Data` length only — SYN/FIN/RST frames
|
||||
are free) starting with the very first DATA (including the hub's forwarded
|
||||
handshake).
|
||||
* The receiver returns credit with `WND(Delta)` once bytes are **delivered to
|
||||
the terminal socket** (written to the player / destination connection), not
|
||||
when they are merely buffered. Receivers should batch grants (the reference
|
||||
implementations send one `WND` per W/2 bytes consumed).
|
||||
* A sender whose window is exhausted pauses reading **that stream's source
|
||||
socket only**; the shared worker conn is never paused because of a single
|
||||
stream. A receiver that observes more than W un-credited bytes on a stream
|
||||
may reset it (`RST`) as a protocol violation.
|
||||
* A sender whose window is exhausted pauses reading **that player's source
|
||||
socket only**. A receiver that observes more than W un-credited bytes may
|
||||
reset the tunnel (`RST`) as a protocol violation.
|
||||
* Senders should also cap individual DATA payloads (the reference
|
||||
implementations use 32 KiB) so one stream cannot monopolize the link for a
|
||||
full 1-MiB frame.
|
||||
implementations use 32 KiB) so one write cannot occupy the link for a full
|
||||
1-MiB frame.
|
||||
|
||||
Both windows may differ (each side enforces the one its peer advertised).
|
||||
`Delta` must be positive; a `WND` for an unknown stream id is ignored.
|
||||
`Delta` must be positive; a `WND` on an unbound worker conn is ignored.
|
||||
|
||||
### 7.4 Liveness
|
||||
|
||||
@@ -444,10 +435,9 @@ Every established session is therefore covered by a heartbeat:
|
||||
hub answers `Pong`. If no `Pong` arrives for `3 × pingIntervalMs`, the client
|
||||
closes the session, which triggers its normal reconnect with backoff.
|
||||
* **Worker conns** — when WORKER_HEARTBEAT was negotiated, the same exchange
|
||||
runs as mux `PING`/`PONG` frames on the reserved StreamID `0`. On timeout the
|
||||
client closes the conn and drops it from the pool. Its streams are reset,
|
||||
unless STREAM_RESUME was negotiated, in which case they are hung and reattached
|
||||
over a fresh conn instead (§7.5).
|
||||
runs as connection-level `PING`/`PONG` frames. On timeout the client closes
|
||||
the conn. The player is reset, unless STREAM_RESUME was negotiated, in which
|
||||
case it is hung and reattached over a fresh conn instead (§7.5).
|
||||
* **Hub side** — an established redapricot session that receives no frame for
|
||||
`sessionIdleTimeoutMs` (default 90000, `0` disables) is closed. Player
|
||||
connections are never subject to this.
|
||||
@@ -457,26 +447,24 @@ that has become unreachable at the IP layer.
|
||||
|
||||
Session establishment (§4) is bounded by a single deadline covering the dial,
|
||||
the `Rekey` write and the `SessionReady` read, and every frame write is bounded
|
||||
too — a peer that stops reading must not be able to park a whole multiplexed
|
||||
connection inside one write.
|
||||
too — a peer that stops reading must not be able to park a write forever.
|
||||
|
||||
### 7.5 Stream resumption (STREAM_RESUME)
|
||||
|
||||
A worker conn is only the middle leg of every stream it carries. When it dies,
|
||||
A worker conn is only the middle leg of the player it carries. When it dies,
|
||||
both terminal sockets — the player's and the destination's — are usually still
|
||||
healthy, so resetting the streams discards working connections because a
|
||||
replaceable transport failed. One conntrack expiry disconnects every player on
|
||||
that conn.
|
||||
healthy, so resetting the tunnel discards working connections because a
|
||||
replaceable transport failed.
|
||||
|
||||
With STREAM_RESUME negotiated, a worker-conn drop instead **hangs** each stream:
|
||||
With STREAM_RESUME negotiated, a worker-conn drop instead **hangs** the player:
|
||||
|
||||
* the hub pauses the player socket, keeps its state, and holds it for
|
||||
`ResumeGraceMs` from the moment of the *first* hang (an absolute deadline, so a
|
||||
flapping client cannot extend it indefinitely);
|
||||
* the client keeps the destination socket open and reattaches the stream over a
|
||||
fresh worker conn by sending `RESUME` with the stream's CID;
|
||||
* the client keeps the destination socket open and reattaches over a fresh
|
||||
worker conn by sending `RESUME` with the player's CID;
|
||||
* the hub answers `RESUME_ACK`, or `RST(UNKNOWN_STREAM)` if it holds no such
|
||||
stream — which is also what a client gets from a hub that has restarted.
|
||||
player — which is also what a client gets from a hub that has restarted.
|
||||
|
||||
**Resumption is byte-exact, and must be.** Frames handed to a dying socket are
|
||||
lost with no notification, and the frame cipher cannot be resynchronized, so each
|
||||
@@ -512,13 +500,13 @@ Two rules deserve emphasis, because the obvious simplifications are wrong:
|
||||
credit can ever come back.
|
||||
|
||||
`RESUME_ACK` carries a freshly minted `NewCID`, which replaces the old one. A CID
|
||||
therefore stays single-use even though a stream may be reattached many times, so
|
||||
therefore stays single-use even though a player may be reattached many times, so
|
||||
a leaked CID grants no more than the outage in which it was observed.
|
||||
|
||||
Resumption is **hub-instance-affine**: a CID means nothing to a second hub behind
|
||||
an L4 load balancer, which answers `RST(UNKNOWN_STREAM)` and lets the client tear
|
||||
down at once. Because the grace period holds player sockets and their buffers, a
|
||||
hub bounds the number of hung streams and the bytes they retain, dropping the
|
||||
hub bounds the number of hung players and the bytes they retain, dropping the
|
||||
oldest first when either cap is reached.
|
||||
|
||||
## 8. HAProxy protocol v2 (optional)
|
||||
@@ -555,12 +543,15 @@ big-endian.
|
||||
"resumeGraceMs": 20000,
|
||||
"maxParkedStreams": 256,
|
||||
"statsIntervalMs": 0,
|
||||
"registrationGraceMs": 15000
|
||||
"registrationGraceMs": 15000,
|
||||
"playerRatePerSec": 8,
|
||||
"playerBurst": 16,
|
||||
"maxPlayersPerIp": 64
|
||||
}
|
||||
```
|
||||
|
||||
`streamWindowBytes` (optional, default 262144, clamped to [32768, 8388608]) is
|
||||
the hub's advertised per-stream receive window (§7.3).
|
||||
the hub's advertised per-connection receive window (§7.3).
|
||||
|
||||
`sessionIdleTimeoutMs` (optional, default 90000) closes an established control
|
||||
session or worker conn that has gone silent for that long (§7.4). It must stay
|
||||
@@ -587,13 +578,27 @@ session's routes are kept as **orphaned** rather than deleted (§5.2). `0`
|
||||
disables it, restoring the behaviour of dropping routes the instant a session
|
||||
closes.
|
||||
|
||||
`playerRatePerSec` (optional, default 8) and `playerBurst` (optional, default
|
||||
16) are a per-IP token bucket applied **only to player connections** (any
|
||||
handshake whose `Intent` is not 17 or 18). One arrival consumes one token;
|
||||
without a token the socket is closed after the handshake, before `match`, CID
|
||||
minting, or pause. Unmatched hostnames still consume a token — otherwise a
|
||||
hostname scan is a free flood. Intent 17 is never admitted through the
|
||||
limiter: every worker conn comes from the client's one address, and limiting
|
||||
those would be the hub throttling its own client. `playerRatePerSec: 0` turns
|
||||
the bucket off.
|
||||
|
||||
`maxPlayersPerIp` (optional, default 64) caps concurrent player sockets from
|
||||
one address (pending + live + parked). `0` disables the cap. Addresses are
|
||||
matched exactly; IPv6 `/64` aggregation is out of scope.
|
||||
|
||||
### 9.2 Client — JSON
|
||||
|
||||
```json
|
||||
{
|
||||
"server": "127.0.0.1:25565",
|
||||
"psk": "change-me",
|
||||
"maxConn": 4,
|
||||
"maxTunnels": 256,
|
||||
"pingIntervalMs": 20000,
|
||||
"streamWindowBytes": 262144,
|
||||
"maxBandwidth": "20mbps",
|
||||
@@ -606,8 +611,14 @@ closes.
|
||||
}
|
||||
```
|
||||
|
||||
`maxTunnels` (optional, default 256, clamped to [1, 4096]) is how many
|
||||
concurrent 1:1 worker connections the client will hold. A `ControlRequest`
|
||||
arriving at the cap is dropped. The older `maxConn` key (the mux-era pool size,
|
||||
clamped 1–8) is ignored if present: treating it as a player cap would silently
|
||||
limit a previously-working config to a handful of players.
|
||||
|
||||
`streamWindowBytes` (optional, default 262144, clamped to [32768, 8388608]) is
|
||||
the client's advertised per-stream receive window (§7.3).
|
||||
the client's advertised per-connection receive window (§7.3).
|
||||
|
||||
`streamResume` (optional, default true) offers STREAM_RESUME (§7.5). With it
|
||||
false the client never offers the flag, never retains a byte for retransmission,
|
||||
@@ -626,7 +637,7 @@ time per conn. These distinguish a slow backend from a saturated uplink from a
|
||||
bad path, which throughput alone cannot.
|
||||
|
||||
`maxBandwidth` (optional, default unlimited) caps the aggregate rate at which the
|
||||
client sends `DATA` to the hub, shared fairly across streams. Accepts `"20mbps"`
|
||||
client sends `DATA` to the hub, shared fairly across tunnels. Accepts `"20mbps"`
|
||||
(decimal bit units), `"2MB/s"` (binary byte units), or a bare number of bytes per
|
||||
second; the minimum is 8192 B/s. **This is a purely local policy and has no
|
||||
effect on the wire format** — a shaped client is indistinguishable from a slow
|
||||
@@ -664,17 +675,17 @@ not appear on the tunnel wire, and the exchange is invisible to the player.
|
||||
| pattern matching | case-insensitive, whole-string regex; first match wins |
|
||||
| CID length | 16 bytes |
|
||||
| max frame payload | 1 MiB |
|
||||
| pool growth | breadth-first: grow to `max_conn` before stacking |
|
||||
| saturation threshold (logged) | active streams `> 8` at `max_conn` |
|
||||
| max worker conns | `max_conn ∈ [1,8]` |
|
||||
| feature flag: per-stream flow control | `0x01` (mandatory) |
|
||||
| worker framing | `FrameType` + payload; no stream id |
|
||||
| max concurrent worker conns | `maxTunnels`, default 256, clamped `[1, 4096]` |
|
||||
| player IP rate / burst / concurrent | `8 /s`, burst `16`, `maxPlayersPerIp` `64` (Intent ∉ {17, 18} only) |
|
||||
| feature flag: per-connection flow control | `0x01` (mandatory) |
|
||||
| feature flag: worker heartbeat | `0x02` (negotiated) |
|
||||
| heartbeat timeout | `3 × pingIntervalMs` |
|
||||
| hub session idle timeout | 90000 ms (`0` disables) |
|
||||
| stream window default / bounds | 256 KiB, clamped to [32 KiB, 8 MiB] |
|
||||
| connection window default / bounds | 256 KiB, clamped to [32 KiB, 8 MiB] |
|
||||
| feature flag: stream resumption | 0x04 (negotiated) |
|
||||
| mux RESUME / RESUME_ACK | 0x07 / 0x08 |
|
||||
| RESUME / RESUME_ACK | 0x07 / 0x08 |
|
||||
| resume grace: hub / client default | 20000 ms / 15000 ms (hub value advertised) |
|
||||
| retained region per stream per direction | bounded by the stream window |
|
||||
| retained region per tunnel per direction | bounded by the connection window |
|
||||
| WND grant batching (reference) | one grant per window/2 consumed |
|
||||
| DATA chunk cap (reference) | 32 KiB |
|
||||
|
||||
@@ -13,9 +13,9 @@ field, so a vanilla Minecraft client needs no modification.
|
||||
> made reachable from the outside.
|
||||
|
||||
```
|
||||
Player ──MC──▶ Hub (Java) ══ Worker Conn (mux) ══▶ Client (Go) ──MC──▶ Real MC server
|
||||
vanilla client public IP encrypted, pooled behind NAT (localhost)
|
||||
│ ▲
|
||||
Player ──MC──▶ Hub (Java) ══ Worker Conn (1:1) ══▶ Client (Go) ──MC──▶ Real MC server
|
||||
vanilla client public IP one encrypted TCP behind NAT (localhost)
|
||||
│ per player ▲
|
||||
└────────────── Control Session ─────────┘
|
||||
(pattern registration + control requests)
|
||||
```
|
||||
@@ -40,13 +40,11 @@ one or more hostname patterns — each a **regular expression**. When a player
|
||||
connects to the hub with a hostname that matches a registered pattern (and any
|
||||
normal `Intent`), the hub assigns a random **CID**, buffers
|
||||
the player's bytes, and asks the client (via the control session) to take over.
|
||||
The client picks a **worker connection** — a multiplexed, encrypted TCP link
|
||||
that carries many players as lightweight *streams* — opens a stream for that CID,
|
||||
dials the real destination (optionally announcing the player's real IP with the
|
||||
**HAProxy v2** protocol), and bridges the two ends. Worker connections are
|
||||
pooled: the client uses up to `maxConn` of them and always places a new stream
|
||||
on the least-loaded one, growing the pool to `maxConn` before stacking streams
|
||||
so no single TCP connection carries every player.
|
||||
The client dials a fresh **worker connection** — one encrypted TCP link per
|
||||
player — announces the CID with `SYN`, dials the real destination (optionally
|
||||
announcing the player's real IP with the **HAProxy v2** protocol), and bridges
|
||||
the two ends. `maxTunnels` is the only player cap; the retired `maxConn` field
|
||||
is ignored so an old config does not silently admit only four players.
|
||||
|
||||
## Repository layout
|
||||
|
||||
@@ -114,7 +112,7 @@ cp client/config.example.json client.json
|
||||
# {
|
||||
# "server": "hub.example.com:25565",
|
||||
# "psk": "same-as-the-hub",
|
||||
# "maxConn": 4,
|
||||
# "maxTunnels": 256,
|
||||
# "mappings": [
|
||||
# { "pattern": "mc\\.example\\.com", "destination": "127.0.0.1:25566", "proxyProtocol": true }
|
||||
# ]
|
||||
@@ -175,6 +173,9 @@ secrets. The base image and build flags live in `.ko.yaml`.
|
||||
| `maxParkedStreams` | `256` | Cap on simultaneously hung players; `maxParkedBytes` (default `maxParkedStreams × 2 × streamWindowBytes`) caps what they retain. Past either, the oldest are dropped. |
|
||||
| `statsIntervalMs` | `0` (off) | Log a periodic line with live/hung stream counts, retained bytes and pattern count. |
|
||||
| `registrationGraceMs` | `15000` | Keep a closed control session's routes as *orphaned* for this long, holding players that arrive on them instead of refusing them, and replaying their requests once the client re-registers. `0` disables it. |
|
||||
| `playerRatePerSec` | `8` | Per-IP token-bucket rate for **player** connections only (Intent ∉ {17, 18}). Unmatched hostnames still consume a token. `0` disables. |
|
||||
| `playerBurst` | `16` | Token-bucket depth for `playerRatePerSec`. |
|
||||
| `maxPlayersPerIp` | `64` | Concurrent player sockets (pending + live + parked) from one IP. `0` disables. Intent 17 is never counted. |
|
||||
|
||||
### Client (`client/config.example.json`)
|
||||
|
||||
@@ -182,7 +183,7 @@ secrets. The base image and build flags live in `.ko.yaml`.
|
||||
|------------------|--------------------|---------|
|
||||
| `server` | *(required)* | Hub `host:port`. |
|
||||
| `psk` | *(required)* | Shared secret; must match the hub. |
|
||||
| `maxConn` | `1` (clamped 1–8) | Max worker connections in the pool. |
|
||||
| `maxTunnels` | `256` (clamped 1–4096) | Max concurrent player tunnels. The retired `maxConn` field is ignored. |
|
||||
| `pingIntervalMs` | `20000` (min 1000) | Heartbeat interval for the control session and every worker conn. A session with no reply for `3×` this is dropped and re-established. |
|
||||
| `maxBandwidth` | *(unlimited)* | Caps what the client uploads to the hub, summed over every player — the direction carrying the game server's output, and the one a home uplink runs out of first. `"20mbps"`, `"512kbps"`, `"2MB/s"`, or a bare number of bytes/sec. **Bit units are decimal (`20mbps` = 20,000,000 bit/s); byte units are binary (`2MB/s` = 2 MiB/s).** Set it slightly below your real upload speed — framing and TCP/IP overhead are not counted. The budget is shared fairly across players, so one person loading chunks cannot time the others out. |
|
||||
| `streamWindowBytes` | `262144` | Advertised per-stream receive window, clamped to [32 KiB, 8 MiB]. |
|
||||
@@ -217,20 +218,23 @@ go test ./e2e/... -v
|
||||
|
||||
The e2e suite covers: a full player round-trip with verbatim handshake
|
||||
forwarding and case-insensitive matching, regex wildcard pattern routing,
|
||||
multi-megabyte transfers, concurrent
|
||||
streams spreading across multiple worker connections, HAProxy v2 source-address
|
||||
multi-megabyte transfers, concurrent players each on their own worker
|
||||
connection, HAProxy v2 source-address
|
||||
propagation, Velocity modern-forwarding interception (signed player-info
|
||||
handoff to a mock Paper backend), player- and destination-initiated disconnect propagation, wrong-PSK
|
||||
rejection, dropping of unmatched hostnames, stream isolation under a slow
|
||||
player and under a slow destination (no head-of-line blocking), rejection
|
||||
of pre-flow-control peers, and stream resumption — a tunnel hard-reset
|
||||
mid-transfer with the player connection held open, asserting the byte stream
|
||||
neither gains nor loses a byte, across concurrent streams, plus grace expiry and
|
||||
the resume-disabled path, and control-outage handling — a player arriving while
|
||||
the client's control session is down is held and then served once it
|
||||
re-registers, with the grace-disabled and grace-expired paths pinned too. The Go and Java crypto layers are
|
||||
independently pinned to the same SHA3-224 test vector so they cannot silently
|
||||
drift apart.
|
||||
handoff to a mock Paper backend), player- and destination-initiated disconnect
|
||||
propagation, wrong-PSK rejection, dropping of unmatched hostnames, isolation
|
||||
under a slow player and under a slow destination (no head-of-line blocking),
|
||||
rejection of pre-flow-control peers, and stream resumption — a tunnel
|
||||
hard-reset mid-transfer with the player connection held open, asserting the
|
||||
byte stream neither gains nor loses a byte, across concurrent players, plus
|
||||
grace expiry and the resume-disabled path, and control-outage handling — a
|
||||
player arriving while the client's control session is down is held and then
|
||||
served once it re-registers, with the grace-disabled and grace-expired paths
|
||||
pinned too, and the per-IP player limiter — burst overflow and
|
||||
`maxPlayersPerIp` drop extras before they become pending, Intent 17 is never
|
||||
counted, and both knobs at `0` restore the unlimited path. The Go and Java
|
||||
crypto layers are independently pinned to the same SHA3-224 test vector so they
|
||||
cannot silently drift apart.
|
||||
|
||||
## Design notes & limitations
|
||||
|
||||
@@ -238,29 +242,28 @@ drift apart.
|
||||
ChaCha20-encrypted (no AEAD tag) to minimize overhead. This protects against
|
||||
casual sniffing, not a determined active attacker (see the note at the top of
|
||||
[PROTOCOL.md](PROTOCOL.md) and [docs/architecture.md](docs/architecture.md) §8).
|
||||
* **Per-stream flow control.** Each stream has credit-based windows in both
|
||||
directions (windows exchanged at session setup, default 256 KiB), so a slow
|
||||
player or slow destination jams only its own stream at a bounded buffer — no
|
||||
application-level head-of-line blocking between streams. What remains is
|
||||
TCP-level HOL (packet loss stalls a whole worker connection briefly);
|
||||
raising `maxConn` spreads that.
|
||||
* **Per-tunnel flow control.** Each worker connection has credit-based windows
|
||||
in both directions (windows exchanged at session setup, default 256 KiB), so
|
||||
a slow player or slow destination jams only its own tunnel at a bounded
|
||||
buffer. There is no application-level head-of-line blocking: each player
|
||||
owns a TCP connection.
|
||||
* **Liveness is explicit.** Every session heartbeats, every socket write is
|
||||
bounded, and session establishment has a deadline. A path that dies silently —
|
||||
no `FIN`, no `RST`, as when a NAT or firewall forgets an established flow — is
|
||||
detected within `3 × pingIntervalMs`, the dead connection is dropped from the
|
||||
pool, and service is restored without operator action. TCP keepalive is on as
|
||||
a second line of defence.
|
||||
detected within `3 × pingIntervalMs`, the dead connection is dropped, and
|
||||
service is restored without operator action. TCP keepalive is on as a
|
||||
second line of defence.
|
||||
* **A control-session reconnect no longer refuses new players.** While a client
|
||||
is reconnecting the hub has no route for it, so arriving players used to be
|
||||
told there is no such server. Those routes are now held briefly and the
|
||||
players with them, then served once the client re-registers.
|
||||
* **A dropped tunnel no longer drops the players.** A worker connection is only
|
||||
the middle leg of the streams it carries; when it dies both terminal sockets
|
||||
are usually still healthy. The hub now hangs those players while the client
|
||||
reattaches their streams over a fresh connection, replaying byte-exactly from
|
||||
the middle leg of the player it carries; when it dies both terminal sockets
|
||||
are usually still healthy. The hub now hangs that player while the client
|
||||
reattaches the tunnel over a fresh connection, replaying byte-exactly from
|
||||
the offset the peer reports, so a conntrack expiry costs a stall rather than
|
||||
disconnecting everyone on that connection. Negotiated, and `streamResume:
|
||||
false` on either side restores the old behaviour.
|
||||
a disconnect. Negotiated, and `streamResume: false` on either side restores
|
||||
the old behaviour.
|
||||
* **Single hub event loop.** The hub deploys one Vert.x verticle, so all state
|
||||
is confined to one event loop (no locking). Throughput is bounded by one core;
|
||||
ample for hundreds of players, not designed for tens of thousands.
|
||||
|
||||
+41
-31
@@ -14,8 +14,8 @@ import (
|
||||
"github.com/iceBear67/redapricot/client/wire"
|
||||
)
|
||||
|
||||
// Client is a redapricot client: it holds a control session with the hub and a
|
||||
// pool of worker connections used to serve player streams.
|
||||
// Client is a redapricot client: it holds a control session with the hub and
|
||||
// dials one worker connection per player.
|
||||
type Client struct {
|
||||
cfg *Config
|
||||
pskBytes []byte
|
||||
@@ -25,7 +25,7 @@ type Client struct {
|
||||
mappings map[string]Mapping // normalized pattern -> mapping
|
||||
pool *WorkerPool
|
||||
|
||||
// ctx/cancel own every pool dial: Close cancels it so an in-flight dial
|
||||
// ctx/cancel own every worker dial: Close cancels it so an in-flight dial
|
||||
// aborts instead of holding a goroutine for the whole handshake timeout.
|
||||
// The control path uses the caller's context from Start, which is the same
|
||||
// shutdown signal by convention.
|
||||
@@ -34,10 +34,10 @@ type Client struct {
|
||||
|
||||
// closing is set by Close; a conn-loss teardown checks it and closes
|
||||
// streams outright rather than parking them for a reattach that is never
|
||||
// coming. Allocate also consults it through the pool's own flag.
|
||||
// coming. Dial also consults it through the pool's own flag.
|
||||
closing atomic.Bool
|
||||
|
||||
streamWnd int // our advertised per-stream receive window (bytes)
|
||||
streamWnd int // our advertised per-connection receive window (bytes)
|
||||
shaper *Shaper // caps aggregate egress to the hub; nil when unlimited
|
||||
chunk int // DATA payload cap; shrinks below DataChunkSize at low rates
|
||||
|
||||
@@ -78,10 +78,20 @@ func New(cfg *Config) *Client {
|
||||
log.Printf("egress shaped to %d B/s (burst %d B, chunk %d B)",
|
||||
bps, int64(c.shaper.burst), c.shaper.chunk)
|
||||
}
|
||||
c.pool = newWorkerPool(c, cfg.MaxConn)
|
||||
c.pool = newWorkerPool(c, clampMaxTunnels(cfg.MaxTunnels))
|
||||
return c
|
||||
}
|
||||
|
||||
func clampMaxTunnels(n int) int {
|
||||
if n < 1 {
|
||||
return DefaultMaxTunnels
|
||||
}
|
||||
if n > MaxMaxTunnels {
|
||||
return MaxMaxTunnels
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func clampWindow(w int) int {
|
||||
if w <= 0 {
|
||||
return DefaultStreamWindow
|
||||
@@ -99,8 +109,8 @@ func clampWindow(w int) int {
|
||||
// was negotiated during establishment.
|
||||
type session struct {
|
||||
fc *wire.FramedConn
|
||||
peerWnd int // hub's advertised per-stream receive window
|
||||
heartbeat bool // hub accepted mux-level PING/PONG on worker conns
|
||||
peerWnd int // hub's advertised per-connection receive window
|
||||
heartbeat bool // hub accepted connection-level PING/PONG on worker conns
|
||||
resume bool // hub accepted stream resumption (§7.5)
|
||||
// hubGrace is how long the hub will hang a parked player, as advertised in
|
||||
// SessionReady. Zero when resumption was not negotiated.
|
||||
@@ -108,7 +118,7 @@ type session struct {
|
||||
}
|
||||
|
||||
// dialSession opens a TCP connection, performs the Intent-17 handshake, the
|
||||
// Phase-A rekey, and reads SessionReady. Per-stream flow control is mandatory:
|
||||
// Phase-A rekey, and reads SessionReady. Per-connection flow control is mandatory:
|
||||
// a hub that does not echo the STREAM_FC flag is rejected.
|
||||
//
|
||||
// The whole exchange is bounded by HandshakeTimeout. A hub that accepts the
|
||||
@@ -116,7 +126,7 @@ type session struct {
|
||||
// behalf) must fail fast rather than park the caller forever.
|
||||
//
|
||||
// ctx bounds the dial: the control path passes the caller's context so a
|
||||
// shutdown mid-handshake aborts the attempt, and the pool passes the client's
|
||||
// shutdown mid-handshake aborts the attempt, and worker dials pass the client's
|
||||
// own context so Close cancels in-flight dials. After the dial, a cancelled
|
||||
// ctx keeps aborting by closing the conn underneath the deadline-bounded
|
||||
// handshake I/O.
|
||||
@@ -158,7 +168,7 @@ func (c *Client) dialSession(ctx context.Context, magic byte) (sess *session, er
|
||||
)
|
||||
|
||||
// 3. Rekey frame (Phase A), including the mandatory feature flags and our
|
||||
// per-stream receive window.
|
||||
// per-connection receive window.
|
||||
rnd := make([]byte, 16)
|
||||
if _, err := crand.Read(rnd); err != nil {
|
||||
return nil, err
|
||||
@@ -186,7 +196,7 @@ func (c *Client) dialSession(ctx context.Context, magic byte) (sess *session, er
|
||||
)
|
||||
|
||||
// 5. SessionReady: the type byte followed by the hub's accepted flags and
|
||||
// its per-stream receive window. Both are required.
|
||||
// its per-connection receive window. Both are required.
|
||||
payload, err := fc.ReadFrame()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -198,7 +208,7 @@ func (c *Client) dialSession(ctx context.Context, magic byte) (sess *session, er
|
||||
flags, ferr := r.VarInt()
|
||||
hubWnd, werr := r.VarInt()
|
||||
if ferr != nil || werr != nil || flags&FlagStreamFC == 0 || hubWnd <= 0 {
|
||||
return nil, fmt.Errorf("hub did not accept per-stream flow control (unsupported hub version?)")
|
||||
return nil, fmt.Errorf("hub did not accept per-connection flow control (unsupported hub version?)")
|
||||
}
|
||||
if hubWnd > MaxStreamWindow {
|
||||
hubWnd = MaxStreamWindow
|
||||
@@ -398,8 +408,8 @@ func (c *Client) pingLoop(ctx context.Context, ctrl *ctrlSession) {
|
||||
}
|
||||
}
|
||||
|
||||
// handleControlRequest reacts to a matched player: allocate a worker stream,
|
||||
// SYN it, and bridge it to the mapped destination.
|
||||
// handleControlRequest reacts to a matched player: dial a dedicated worker
|
||||
// conn, SYN it, and bridge it to the mapped destination.
|
||||
func (c *Client) handleControlRequest(cid []byte, pattern, ip string, port int) {
|
||||
mapping, ok := c.mappings[NormalizeAddress(pattern)]
|
||||
if !ok {
|
||||
@@ -408,37 +418,37 @@ func (c *Client) handleControlRequest(cid []byte, pattern, ip string, port int)
|
||||
}
|
||||
log.Printf("player %s:%d joined via pattern %q -> %s", ip, port, pattern, mapping.Destination)
|
||||
|
||||
// Allocate and publish must agree on a live conn: Allocate hands out a
|
||||
// (conn, sid) pair that can die before we register on it, which would strand
|
||||
// the stream in a map nothing iterates. registerStream reports that, and we
|
||||
// simply pick another conn.
|
||||
// Dial and attach must agree on a live conn: Dial hands out a conn that can
|
||||
// die before we attach on it, which would strand the stream on a conn
|
||||
// nothing iterates. attach reports that, and we simply dial another.
|
||||
var st *Stream
|
||||
var lg *leg
|
||||
for attempt := 0; attempt < allocateAttempts; attempt++ {
|
||||
wc, sid, err := c.pool.Allocate()
|
||||
var wc *WorkerConn
|
||||
for attempt := 0; attempt < dialAttempts; attempt++ {
|
||||
var err error
|
||||
wc, err = c.pool.Dial()
|
||||
if err != nil {
|
||||
log.Printf("worker allocate failed: %v", err)
|
||||
log.Printf("worker dial failed: %v", err)
|
||||
return
|
||||
}
|
||||
st = newStream(c, wc, sid, cid, mapping, ip, port)
|
||||
// Register before SYN so inbound DATA can never race ahead of the table,
|
||||
st = newStream(c, wc, cid, mapping, ip, port)
|
||||
// Attach before SYN so inbound DATA can never race ahead of the binding,
|
||||
// and start the pumps before the (bounded) SYN write so a failed or slow
|
||||
// SYN cannot strand a stream that nothing would ever tear down.
|
||||
if wc.registerStream(sid, st) {
|
||||
lg = st.conn()
|
||||
if wc.attach(st) {
|
||||
break
|
||||
}
|
||||
_ = wc.fc.Close()
|
||||
st = nil
|
||||
}
|
||||
if st == nil {
|
||||
log.Printf("worker allocate failed: no live conn after %d attempts", allocateAttempts)
|
||||
log.Printf("worker dial failed: no live conn after %d attempts", dialAttempts)
|
||||
return
|
||||
}
|
||||
|
||||
go st.writeLoop()
|
||||
go st.run()
|
||||
if err := lg.wc.sendSyn(lg.sid, cid); err != nil {
|
||||
log.Printf("stream %s: SYN failed: %v", lg, err)
|
||||
if err := wc.sendSyn(cid); err != nil {
|
||||
log.Printf("stream %s: SYN failed: %v", st.name(), err)
|
||||
st.teardown(false)
|
||||
}
|
||||
}
|
||||
@@ -452,7 +462,7 @@ func (c *Client) WorkerConnCount() int { return c.pool.count() }
|
||||
// open.
|
||||
func (c *Client) Close() {
|
||||
c.closing.Store(true)
|
||||
c.cancel() // aborts in-flight pool dials
|
||||
c.cancel() // aborts in-flight worker dials
|
||||
c.mu.Lock()
|
||||
fc := c.ctrl
|
||||
c.mu.Unlock()
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"server": "hub.example.com:25565",
|
||||
"psk": "change-me-to-a-long-random-passphrase",
|
||||
"maxConn": 4,
|
||||
"maxTunnels": 256,
|
||||
"pingIntervalMs": 20000,
|
||||
"streamWindowBytes": 262144,
|
||||
"maxBandwidth": "",
|
||||
|
||||
+41
-34
@@ -43,19 +43,15 @@ const (
|
||||
MuxWnd = 0x04
|
||||
MuxPing = 0x05
|
||||
MuxPong = 0x06
|
||||
// MuxResume reattaches a parked stream to this conn (CID + our accepted
|
||||
// MuxResume reattaches a parked player to this conn (CID + our accepted
|
||||
// offset); MuxResumeAck carries the hub's accepted offset and a fresh CID.
|
||||
MuxResume = 0x07
|
||||
MuxResumeAck = 0x08
|
||||
|
||||
// MuxCtlSid is the reserved stream id carrying connection-scoped mux frames
|
||||
// (PING/PONG). Real streams are numbered from 1.
|
||||
MuxCtlSid = 0
|
||||
|
||||
// RST reason codes (optional trailing byte; absence means "unspecified").
|
||||
// Distinguishing them matters for resume: an unknown stream is terminal,
|
||||
// while "already bound" means the hub has the stream on another conn — the
|
||||
// reattach retries until that bind dies and the hub re-parks the stream.
|
||||
// while "already bound" means the hub has the player on another conn — the
|
||||
// reattach retries until that bind dies and the hub re-parks the player.
|
||||
RstUnspecified = 0x00
|
||||
RstUnknownStream = 0x01
|
||||
RstAlreadyBound = 0x02
|
||||
@@ -65,33 +61,33 @@ const (
|
||||
|
||||
FrameError = 0x7F
|
||||
|
||||
// SaturationThreshold caps how many streams share one worker conn once the
|
||||
// pool has grown to maxConn. Below maxConn the pool grows first (§7.1), so a
|
||||
// single connection is never a shared point of failure for every player.
|
||||
SaturationThreshold = 8
|
||||
// DefaultMaxTunnels / MaxMaxTunnels bound concurrent 1:1 worker conns.
|
||||
// The old mux-era maxConn cap of 8 would silently become "8 players".
|
||||
DefaultMaxTunnels = 256
|
||||
MaxMaxTunnels = 4096
|
||||
|
||||
// Session-establishment feature flags (trailing VarInt on the Rekey message).
|
||||
FlagStreamFC = 0x01
|
||||
// FlagWorkerHeartbeat enables mux-level PING/PONG on worker conns. Without
|
||||
// it a worker conn whose path is silently blackholed (NAT/conntrack drop,
|
||||
// firewall) is never detected: the read loop parks forever, the dead conn
|
||||
// stays in the pool, and no player can be served until the client restarts.
|
||||
// FlagWorkerHeartbeat enables connection-level PING/PONG on worker conns.
|
||||
// Without it a worker conn whose path is silently blackholed (NAT/conntrack
|
||||
// drop, firewall) is never detected: the read loop parks forever and that
|
||||
// player is stuck until the client restarts.
|
||||
FlagWorkerHeartbeat = 0x02
|
||||
// FlagStreamResume enables stream resumption (PROTOCOL.md §7.5): a worker
|
||||
// conn drop parks its streams instead of killing them, the hub hangs the
|
||||
// player sockets, and the client reattaches each stream byte-exactly over a
|
||||
// fresh conn. Negotiated, so either side may decline and get today's
|
||||
// behaviour (immediate teardown) unchanged.
|
||||
// conn drop parks the player instead of killing them, the hub hangs the
|
||||
// player socket, and the client reattaches byte-exactly over a fresh conn.
|
||||
// Negotiated, so either side may decline and get today's behaviour
|
||||
// (immediate teardown) unchanged.
|
||||
FlagStreamResume = 0x04
|
||||
|
||||
// Per-stream flow-control window bounds (bytes). The advertised window is the
|
||||
// receiver's promise of how much un-credited DATA it will buffer per stream.
|
||||
// Per-connection flow-control window bounds (bytes). The advertised window
|
||||
// is the receiver's promise of how much un-credited DATA it will buffer.
|
||||
DefaultStreamWindow = 256 * 1024
|
||||
MinStreamWindow = 32 * 1024
|
||||
MaxStreamWindow = 8 << 20
|
||||
|
||||
// DataChunkSize caps a single DATA frame's payload so no stream monopolizes
|
||||
// the shared worker connection for long.
|
||||
// DataChunkSize caps a single DATA frame's payload so one write cannot
|
||||
// occupy the link for a full 1-MiB frame.
|
||||
DataChunkSize = 32 * 1024
|
||||
)
|
||||
|
||||
@@ -211,14 +207,20 @@ type Mapping struct {
|
||||
|
||||
// Config is the client configuration (PROTOCOL.md §9.2).
|
||||
type Config struct {
|
||||
Server string `json:"server"`
|
||||
PSK string `json:"psk"`
|
||||
MaxConn int `json:"maxConn"`
|
||||
PingIntervalMs int `json:"pingIntervalMs"`
|
||||
StreamWindowBytes int `json:"streamWindowBytes"` // per-stream receive window; 0 = default
|
||||
Server string `json:"server"`
|
||||
PSK string `json:"psk"`
|
||||
// MaxTunnels is the max concurrent 1:1 worker connections (PROTOCOL.md §7.1).
|
||||
// 0 means the default. Clamped to [1, 4096].
|
||||
MaxTunnels int `json:"maxTunnels"`
|
||||
// MaxConn is the retired mux-era pool size. Ignored when loading a file:
|
||||
// honouring a value of 4 as a player cap would silently break existing
|
||||
// configs. Tests that construct a Config should set MaxTunnels instead.
|
||||
MaxConn int `json:"maxConn"`
|
||||
PingIntervalMs int `json:"pingIntervalMs"`
|
||||
StreamWindowBytes int `json:"streamWindowBytes"` // per-connection receive window; 0 = default
|
||||
// MaxBandwidth caps what the client sends to the hub, aggregated over every
|
||||
// stream on every worker conn — the direction that carries the game server's
|
||||
// output to the players, and the one a residential uplink runs out of first.
|
||||
// worker conn — the direction that carries the game server's output to the
|
||||
// players, and the one a residential uplink runs out of first.
|
||||
// Empty means no limit. See parseBandwidth for the accepted syntax.
|
||||
MaxBandwidth string `json:"maxBandwidth"`
|
||||
// StreamResume enables stream resumption (PROTOCOL.md §7.5). A pointer so an
|
||||
@@ -310,11 +312,16 @@ func LoadConfig(path string) (*Config, error) {
|
||||
if c.PSK == "" {
|
||||
return nil, fmt.Errorf("psk is required")
|
||||
}
|
||||
if c.MaxConn < 1 {
|
||||
c.MaxConn = 1
|
||||
if c.MaxConn != 0 && c.MaxTunnels == 0 {
|
||||
// Old mux pool size. Must not become the player cap: a previously-working
|
||||
// maxConn: 4 would admit only four players.
|
||||
fmt.Fprintf(os.Stderr, "redapricot-client: maxConn is ignored (it was the mux pool size); use maxTunnels (default %d)\n", DefaultMaxTunnels)
|
||||
}
|
||||
if c.MaxConn > 8 {
|
||||
c.MaxConn = 8
|
||||
if c.MaxTunnels < 1 {
|
||||
c.MaxTunnels = DefaultMaxTunnels
|
||||
}
|
||||
if c.MaxTunnels > MaxMaxTunnels {
|
||||
c.MaxTunnels = MaxMaxTunnels
|
||||
}
|
||||
if c.PingIntervalMs <= 0 {
|
||||
c.PingIntervalMs = DefaultPingIntervalMs
|
||||
|
||||
+19
-47
@@ -40,24 +40,25 @@ func stalledHub(t *testing.T) string {
|
||||
return ln.Addr().String()
|
||||
}
|
||||
|
||||
// TestAllocateDoesNotWedgePoolOnStalledHub is the regression guard for the
|
||||
// worst failure mode found in the stability audit: Allocate used to dial while
|
||||
// holding the pool mutex, and the handshake read had no deadline. One
|
||||
// TestDialDoesNotWedgeOnStalledHub is the regression guard for the worst
|
||||
// failure mode found in the stability audit: Dial used to share a single
|
||||
// in-flight handshake, and the handshake read had no deadline. One
|
||||
// unresponsive hub therefore parked every present and future allocation
|
||||
// forever, so no player could be served again until the process restarted.
|
||||
func TestAllocateDoesNotWedgePoolOnStalledHub(t *testing.T) {
|
||||
// forever. 1:1 dials independently, but each must still fail on its own
|
||||
// HandshakeTimeout rather than block the other.
|
||||
func TestDialDoesNotWedgeOnStalledHub(t *testing.T) {
|
||||
c := New(&Config{
|
||||
Server: stalledHub(t),
|
||||
PSK: "pool-test",
|
||||
MaxConn: 8,
|
||||
MaxTunnels: 8,
|
||||
PingIntervalMs: 20000,
|
||||
Mappings: []Mapping{{Pattern: "mc.local", Destination: "127.0.0.1:1"}},
|
||||
})
|
||||
|
||||
done := make(chan error, 2)
|
||||
go func() { _, _, err := c.pool.Allocate(); done <- err }()
|
||||
go func() { _, err := c.pool.Dial(); done <- err }()
|
||||
time.Sleep(200 * time.Millisecond) // let the first caller get into the dial
|
||||
go func() { _, _, err := c.pool.Allocate(); done <- err }()
|
||||
go func() { _, err := c.pool.Dial(); done <- err }()
|
||||
|
||||
// Both must give up on their own; neither may be stuck behind the other.
|
||||
limit := time.After(HandshakeTimeout + 15*time.Second)
|
||||
@@ -65,50 +66,21 @@ func TestAllocateDoesNotWedgePoolOnStalledHub(t *testing.T) {
|
||||
select {
|
||||
case err := <-done:
|
||||
if err == nil {
|
||||
t.Fatal("Allocate succeeded against a hub that never answers")
|
||||
t.Fatal("Dial succeeded against a hub that never answers")
|
||||
}
|
||||
case <-limit:
|
||||
t.Fatalf("Allocate #%d never returned: the pool is wedged again", i+1)
|
||||
t.Fatalf("Dial #%d never returned: a stalled hub wedged the other caller", i+1)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestAllocateSpreadsAcrossConns guards the allocation rule: the pool must fan
|
||||
// out to maxConn before stacking streams, so a single worker connection is
|
||||
// never the shared point of failure for every player. Seven players used to all
|
||||
// land on one conn, which meant one dead TCP connection dropped everybody.
|
||||
func TestAllocateSpreadsAcrossConns(t *testing.T) {
|
||||
p := &WorkerPool{maxConn: 4}
|
||||
p.cond = sync.NewCond(&p.mu)
|
||||
|
||||
newConn := func() *WorkerConn {
|
||||
return &WorkerConn{pool: p, streams: make(map[int]*Stream), nextSid: 1, done: make(chan struct{})}
|
||||
}
|
||||
p.conns = []*WorkerConn{newConn()}
|
||||
if !p.conns[0].registerStream(1, &Stream{}) {
|
||||
t.Fatal("registerStream refused on a live conn")
|
||||
}
|
||||
|
||||
// One conn holding a stream, pool below maxConn: growth is warranted.
|
||||
_, bestCount := p.leastLoadedLocked()
|
||||
if bestCount < StreamsBeforeGrowing {
|
||||
t.Fatalf("a conn with %d stream(s) should trigger growth", bestCount)
|
||||
}
|
||||
|
||||
// Once the pool is at maxConn, growth stops and streams stack on the
|
||||
// least-loaded conn instead.
|
||||
for len(p.conns) < p.maxConn {
|
||||
p.conns = append(p.conns, newConn())
|
||||
}
|
||||
best, bestCount := p.leastLoadedLocked()
|
||||
if bestCount != 0 {
|
||||
t.Fatalf("expected an empty conn to be least-loaded, got %d streams", bestCount)
|
||||
}
|
||||
p.maybeGrowLocked(bestCount)
|
||||
if p.dialing != 0 {
|
||||
t.Fatalf("pool dialed past maxConn=%d", p.maxConn)
|
||||
}
|
||||
if best == nil {
|
||||
t.Fatal("no conn selected")
|
||||
// TestDialRespectsMaxTunnels pins the concurrency cap: once live+dialing
|
||||
// equals maxTunnels, further Dial calls fail immediately rather than stacking.
|
||||
func TestDialRespectsMaxTunnels(t *testing.T) {
|
||||
p := newWorkerPool(&Client{}, 2)
|
||||
p.conns[&WorkerConn{id: 1}] = struct{}{}
|
||||
p.conns[&WorkerConn{id: 2}] = struct{}{}
|
||||
if _, err := p.Dial(); err != errTooManyTunnels {
|
||||
t.Fatalf("Dial at cap: got %v, want %v", err, errTooManyTunnels)
|
||||
}
|
||||
}
|
||||
|
||||
+41
-41
@@ -10,11 +10,10 @@ import (
|
||||
|
||||
// Stream resumption (PROTOCOL.md §7.5).
|
||||
//
|
||||
// A worker conn carries many players but is only the middle leg of each: when
|
||||
// it dies, both terminal sockets are usually still perfectly healthy. Tearing
|
||||
// the streams down therefore throws away working connections because a
|
||||
// replaceable transport failed — one conntrack expiry disconnects everyone on
|
||||
// that conn.
|
||||
// A worker conn is only the middle leg of the player it carries: when it dies,
|
||||
// both terminal sockets are usually still perfectly healthy. Tearing the
|
||||
// tunnel down therefore throws away working connections because a replaceable
|
||||
// transport failed.
|
||||
//
|
||||
// Instead the stream parks: the destination socket stays open, the hub hangs the
|
||||
// player socket, and the client reattaches over a fresh conn. Correctness rests
|
||||
@@ -104,7 +103,7 @@ func (s *Stream) resumeLoop(grace time.Duration) {
|
||||
return
|
||||
}
|
||||
// errResumeRaced falls through to the retry below. The hub has this
|
||||
// stream bound to a conn that is not ours — a half-open conn whose
|
||||
// player bound to a conn that is not ours — a half-open conn whose
|
||||
// death the hub has not yet learned, or a bind left behind by a racing
|
||||
// attempt on a now-dead conn. There is no other live attempt: park is
|
||||
// the only resumeLoop starter and it refuses to double-start. Retrying
|
||||
@@ -131,10 +130,10 @@ func (s *Stream) resumeLoop(grace time.Duration) {
|
||||
s.teardown(false)
|
||||
}
|
||||
|
||||
// tryResume performs one reattach attempt: find a live conn, claim a stream id
|
||||
// on it, send RESUME, and replay from wherever the hub says it got to.
|
||||
// tryResume performs one reattach attempt: dial a fresh worker conn, send
|
||||
// RESUME, and replay from wherever the hub says it got to.
|
||||
func (s *Stream) tryResume() error {
|
||||
wc, sid, err := s.allocateForResume()
|
||||
wc, err := s.allocateForResume()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -147,10 +146,10 @@ func (s *Stream) tryResume() error {
|
||||
s.resumeWait = wait
|
||||
s.mu.Unlock()
|
||||
|
||||
msg := wire.NewWriter().U8(MuxResume).VarInt(sid).Bytes(cid).
|
||||
msg := wire.NewWriter().U8(MuxResume).Bytes(cid).
|
||||
I64(accepted).I64(delivered).Out()
|
||||
if err := wc.fc.WriteFrame(msg); err != nil {
|
||||
s.abandonAttempt(wc, sid)
|
||||
s.abandonAttempt(wc)
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -158,47 +157,51 @@ func (s *Stream) tryResume() error {
|
||||
select {
|
||||
case res = <-wait:
|
||||
case <-s.done:
|
||||
// Torn down while waiting. teardown only deregisters the leg the stream
|
||||
// Torn down while waiting. teardown only detaches the conn the stream
|
||||
// was bound to, which is not this one, so the claim made above has to be
|
||||
// withdrawn here or it stays in the new conn's table forever.
|
||||
s.abandonAttempt(wc, sid)
|
||||
// withdrawn here or the new conn stays bound forever.
|
||||
s.abandonAttempt(wc)
|
||||
return errResumeRefused
|
||||
case <-time.After(ResumeAckTimeout):
|
||||
s.abandonAttempt(wc, sid)
|
||||
s.abandonAttempt(wc)
|
||||
return errResumeTimeout
|
||||
}
|
||||
if res.err != nil {
|
||||
s.abandonAttempt(wc, sid)
|
||||
s.abandonAttempt(wc)
|
||||
return res.err
|
||||
}
|
||||
return s.completeResume(wc, sid, res)
|
||||
return s.completeResume(wc, res)
|
||||
}
|
||||
|
||||
// allocateForResume picks a live conn that will honour a reattach.
|
||||
func (s *Stream) allocateForResume() (*WorkerConn, int, error) {
|
||||
for attempt := 0; attempt < allocateAttempts; attempt++ {
|
||||
wc, sid, err := s.client.pool.Allocate()
|
||||
// allocateForResume dials a dedicated worker conn that will honour a reattach.
|
||||
// 1:1: this must never land on someone else's tunnel.
|
||||
func (s *Stream) allocateForResume() (*WorkerConn, error) {
|
||||
for attempt := 0; attempt < dialAttempts; attempt++ {
|
||||
wc, err := s.client.pool.Dial()
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
return nil, err
|
||||
}
|
||||
// Re-checked per conn, not assumed from the dead one: this may be a
|
||||
// different or restarted hub. Sending RESUME to a hub that does not know
|
||||
// the frame type would hang the player for the rest of the grace waiting
|
||||
// for an answer that is never coming.
|
||||
if !wc.resume {
|
||||
return nil, 0, errResumeNoResume
|
||||
_ = wc.fc.Close()
|
||||
return nil, errResumeNoResume
|
||||
}
|
||||
if wc.registerStream(sid, s) {
|
||||
return wc, sid, nil
|
||||
if wc.attach(s) {
|
||||
return wc, nil
|
||||
}
|
||||
_ = wc.fc.Close()
|
||||
}
|
||||
return nil, 0, errResumeRefused
|
||||
return nil, errResumeRefused
|
||||
}
|
||||
|
||||
// abandonAttempt withdraws a failed attempt from the conn it was made on, so a
|
||||
// retry can never leave two RESUMEs outstanding for one stream.
|
||||
func (s *Stream) abandonAttempt(wc *WorkerConn, sid int) {
|
||||
wc.removeStream(sid)
|
||||
// abandonAttempt withdraws a failed attempt from the conn it was made on and
|
||||
// closes that conn — 1:1, it exists only for this attempt.
|
||||
func (s *Stream) abandonAttempt(wc *WorkerConn) {
|
||||
wc.detach()
|
||||
_ = wc.fc.Close()
|
||||
s.mu.Lock()
|
||||
s.resumeWait = nil
|
||||
s.mu.Unlock()
|
||||
@@ -206,7 +209,7 @@ func (s *Stream) abandonAttempt(wc *WorkerConn, sid int) {
|
||||
|
||||
// completeResume rebinds the stream to its new conn and replays what the hub is
|
||||
// missing, holding sendMu throughout so live traffic cannot overtake the replay.
|
||||
func (s *Stream) completeResume(wc *WorkerConn, sid int, res resumeResult) error {
|
||||
func (s *Stream) completeResume(wc *WorkerConn, res resumeResult) error {
|
||||
s.sendMu.Lock()
|
||||
// Delivery is a strictly stronger fact than credit — the hub only credits what
|
||||
// it has delivered — so the reported offset can be adopted wholesale. Doing so
|
||||
@@ -218,7 +221,7 @@ func (s *Stream) completeResume(wc *WorkerConn, sid int, res resumeResult) error
|
||||
replay := s.un.from(res.accepted)
|
||||
if replay == nil {
|
||||
s.sendMu.Unlock()
|
||||
s.abandonAttempt(wc, sid)
|
||||
s.abandonAttempt(wc)
|
||||
return errResumeTooOld
|
||||
}
|
||||
// Three offsets, three jobs, and conflating any two of them breaks something
|
||||
@@ -235,10 +238,8 @@ func (s *Stream) completeResume(wc *WorkerConn, sid int, res resumeResult) error
|
||||
outstanding := s.un.length()
|
||||
replayed := s.un.end() - res.accepted
|
||||
|
||||
// Publish the new binding before any frame goes out on it, and as one value:
|
||||
// stream ids restart at 1 per conn, so a half-updated pair would address a
|
||||
// different player's stream.
|
||||
s.leg.Store(&leg{wc: wc, sid: sid})
|
||||
// Publish the new binding before any frame goes out on it.
|
||||
s.wc.Store(wc)
|
||||
|
||||
s.mu.Lock()
|
||||
// Restated, not patched. The window is a delta ledger and the outage tore a
|
||||
@@ -251,7 +252,6 @@ func (s *Stream) completeResume(wc *WorkerConn, sid int, res resumeResult) error
|
||||
// Symmetrically, our own pending credit is discarded rather than flushed:
|
||||
// the delivered offset we reported already tells the hub everything those
|
||||
// deltas would have, and sending both would grant the same bytes twice.
|
||||
// Counting resumes from this baseline.
|
||||
s.consumed = 0
|
||||
if len(res.cid) == CIDLen {
|
||||
s.cid = res.cid // fresh capability, so a CID is never reusable twice
|
||||
@@ -267,9 +267,9 @@ func (s *Stream) completeResume(wc *WorkerConn, sid int, res resumeResult) error
|
||||
if n > s.client.chunk {
|
||||
n = s.client.chunk
|
||||
}
|
||||
if err := wc.sendData(sid, replay[:n]); err != nil {
|
||||
if err := wc.sendData(replay[:n]); err != nil {
|
||||
// The conn died mid-replay. The stream is still resumable, but not
|
||||
// from this leg — put it back in the parked state before returning
|
||||
// from this conn — put it back in the parked state before returning
|
||||
// so the conn's teardown takes park()'s already-branch instead of
|
||||
// starting a second resumeLoop. The loop we came from keeps
|
||||
// retrying with the fresh CID, which the hub re-parked alongside
|
||||
@@ -300,11 +300,11 @@ func (s *Stream) completeResume(wc *WorkerConn, sid int, res resumeResult) error
|
||||
// A destination that closed while we were parked owed the hub a FIN that had
|
||||
// nowhere to go at the time.
|
||||
if owedFin {
|
||||
wc.sendFin(sid)
|
||||
wc.sendFin()
|
||||
s.teardown(false)
|
||||
return nil
|
||||
}
|
||||
log.Printf("stream conn%d/sid%d resumed (%d bytes replayed, %d outstanding)", wc.id, sid, replayed, outstanding)
|
||||
log.Printf("stream %s resumed (%d bytes replayed, %d outstanding)", s.name(), replayed, outstanding)
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
+16
-18
@@ -57,11 +57,9 @@ type connStats struct {
|
||||
framesOut atomic.Int64
|
||||
writeErrs atomic.Int64
|
||||
|
||||
// Round-trip time of the mux heartbeat. The probe already carries a
|
||||
// Round-trip time of the worker heartbeat. The probe already carries a
|
||||
// timestamp that the peer echoes and both sides currently throw away, so
|
||||
// this measures tunnel latency for no added cost — and it is the best signal
|
||||
// available for head-of-line blocking, where one stream's backlog delays
|
||||
// every other stream sharing the connection.
|
||||
// this measures tunnel latency for no added cost.
|
||||
mu sync.Mutex
|
||||
rttLast time.Duration
|
||||
rttMin time.Duration
|
||||
@@ -137,22 +135,19 @@ func (c *Client) StatsLine() string {
|
||||
|
||||
live, parked := 0, 0
|
||||
for _, wc := range conns {
|
||||
wc.mu.Lock()
|
||||
streams := make([]*Stream, 0, len(wc.streams))
|
||||
for _, s := range wc.streams {
|
||||
streams = append(streams, s)
|
||||
}
|
||||
wc.mu.Unlock()
|
||||
live += len(streams)
|
||||
for _, s := range streams {
|
||||
s.mu.Lock()
|
||||
if s.parked {
|
||||
st := wc.getStream()
|
||||
bound := 0
|
||||
if st != nil {
|
||||
bound = 1
|
||||
live++
|
||||
st.mu.Lock()
|
||||
if st.parked {
|
||||
parked++
|
||||
}
|
||||
s.mu.Unlock()
|
||||
st.mu.Unlock()
|
||||
}
|
||||
|
||||
fmt.Fprintf(&b, " | conn%d streams=%d", wc.id, len(streams))
|
||||
fmt.Fprintf(&b, " | conn%d bound=%d", wc.id, bound)
|
||||
if cs := wc.stats; cs != nil {
|
||||
_, mn, avg, mx := cs.rtt()
|
||||
fmt.Fprintf(&b, " frames=%d/%d rtt=%s/%s/%s",
|
||||
@@ -162,7 +157,7 @@ func (c *Client) StatsLine() string {
|
||||
}
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(&b, " | streams=%d parked=%d", live, parked)
|
||||
fmt.Fprintf(&b, " | tunnels=%d parked=%d", live, parked)
|
||||
return b.String()
|
||||
}
|
||||
|
||||
@@ -189,7 +184,10 @@ func (s *Stream) logSummary() {
|
||||
|
||||
func (p *WorkerPool) snapshot() []*WorkerConn {
|
||||
p.mu.Lock()
|
||||
conns := append([]*WorkerConn(nil), p.conns...)
|
||||
conns := make([]*WorkerConn, 0, len(p.conns))
|
||||
for wc := range p.conns {
|
||||
conns = append(conns, wc)
|
||||
}
|
||||
p.mu.Unlock()
|
||||
sort.Slice(conns, func(i, j int) bool { return conns[i].id < conns[j].id })
|
||||
return conns
|
||||
|
||||
@@ -15,9 +15,9 @@ import (
|
||||
const MaxFrame = 1 << 20
|
||||
|
||||
// WriteTimeout bounds a single frame write. A peer that stops reading must not
|
||||
// be able to park every stream on the connection inside WriteFrame forever: the
|
||||
// write mutex is held for the whole socket write, so one stalled write would
|
||||
// otherwise wedge the entire multiplexed connection.
|
||||
// be able to park this connection inside WriteFrame forever: the write mutex
|
||||
// is held for the whole socket write, so one stalled write would otherwise
|
||||
// wedge liveness (PING/PONG) and WND/FIN on this tunnel.
|
||||
const WriteTimeout = 30 * time.Second
|
||||
|
||||
var (
|
||||
|
||||
+172
-291
@@ -12,166 +12,87 @@ import (
|
||||
"github.com/iceBear67/redapricot/client/wire"
|
||||
)
|
||||
|
||||
// errPoolClosed is returned by Allocate after Close: the pool is shutting down
|
||||
// errPoolClosed is returned by Dial after Close: the set is shutting down
|
||||
// and must not start new dials, so a caller (handleControlRequest,
|
||||
// allocateForResume) gives up rather than wait on a cond no one will satisfy.
|
||||
var errPoolClosed = errors.New("worker pool closed")
|
||||
// allocateForResume) gives up rather than waiting on a hub that will never
|
||||
// be used.
|
||||
var errPoolClosed = errors.New("worker set closed")
|
||||
|
||||
// StreamsBeforeGrowing is how many streams a worker conn may carry before the
|
||||
// pool starts opening another one. Set to 1 so the pool fans out to maxConn
|
||||
// under load *before* stacking streams: concentrating every player on a single
|
||||
// TCP connection makes that connection a shared point of failure, which is
|
||||
// exactly how a whole server's worth of players used to drop at once.
|
||||
const StreamsBeforeGrowing = 1
|
||||
// errTooManyTunnels is returned when live+in-flight worker conns already
|
||||
// equal maxTunnels. The ControlRequest is dropped; the hub closes the player
|
||||
// when pendingTimeoutMs fires.
|
||||
var errTooManyTunnels = errors.New("maxTunnels reached")
|
||||
|
||||
// allocateAttempts bounds how many times a caller retries Allocate when the
|
||||
// conn it was handed dies before the stream could be registered on it. The race
|
||||
// is narrow and each retry picks a different conn, so a small bound is enough;
|
||||
// an unbounded loop would spin against a hub that is refusing every connection.
|
||||
const allocateAttempts = 3
|
||||
// dialAttempts bounds how many times a caller retries Dial when the conn it
|
||||
// was handed dies before the stream could be attached to it. The race is
|
||||
// narrow; an unbounded loop would spin against a hub that is refusing every
|
||||
// connection.
|
||||
const dialAttempts = 3
|
||||
|
||||
// WorkerPool manages up to maxConn worker connections and allocates streams
|
||||
// using the least-loaded strategy (PROTOCOL.md §7.1).
|
||||
// WorkerPool tracks live 1:1 worker connections up to maxTunnels
|
||||
// (PROTOCOL.md §7.1). There is no least-loaded placement and no sharing:
|
||||
// every player gets its own TCP connection.
|
||||
type WorkerPool struct {
|
||||
client *Client
|
||||
maxConn int
|
||||
client *Client
|
||||
maxTunnels int
|
||||
|
||||
connSeq atomic.Int64 // conn ids, for log correlation
|
||||
|
||||
mu sync.Mutex
|
||||
cond *sync.Cond
|
||||
conns []*WorkerConn
|
||||
dialing int // dials currently in flight (foreground + background)
|
||||
dialGen uint64
|
||||
dialErr error // most recent dial failure
|
||||
closed bool // closeAll ran; no new conns may join the pool
|
||||
conns map[*WorkerConn]struct{}
|
||||
dialing int // dials currently in flight
|
||||
closed bool // closeAll ran; no new conns may join
|
||||
}
|
||||
|
||||
func newWorkerPool(c *Client, maxConn int) *WorkerPool {
|
||||
p := &WorkerPool{client: c, maxConn: maxConn}
|
||||
p.cond = sync.NewCond(&p.mu)
|
||||
return p
|
||||
func newWorkerPool(c *Client, maxTunnels int) *WorkerPool {
|
||||
return &WorkerPool{
|
||||
client: c,
|
||||
maxTunnels: maxTunnels,
|
||||
conns: make(map[*WorkerConn]struct{}),
|
||||
}
|
||||
}
|
||||
|
||||
// Allocate returns a worker conn and a fresh stream id to place a new stream on.
|
||||
// Dial opens a dedicated worker conn for one player.
|
||||
//
|
||||
// A dial is never performed while holding p.mu: session establishment involves
|
||||
// network I/O, and holding the pool lock across it would park every other
|
||||
// player behind one unresponsive hub. When the pool is empty exactly one caller
|
||||
// dials and the rest wait on the condition variable; when the pool is merely
|
||||
// below maxConn, growth happens in the background and the caller is served
|
||||
// immediately by an existing conn.
|
||||
func (p *WorkerPool) Allocate() (*WorkerConn, int, error) {
|
||||
// network I/O, and holding the lock across it would park every other player
|
||||
// behind one unresponsive hub. Each caller dials independently; unlike the
|
||||
// old mux pool there is no shared conn to wait for.
|
||||
func (p *WorkerPool) Dial() (*WorkerConn, error) {
|
||||
p.mu.Lock()
|
||||
for {
|
||||
if p.closed {
|
||||
// Close won. No new conn may join the pool, so no stream may be
|
||||
// placed; waiting on the cond could only be satisfied by a dial we
|
||||
// must not start.
|
||||
p.mu.Unlock()
|
||||
return nil, 0, errPoolClosed
|
||||
}
|
||||
best, bestCount := p.leastLoadedLocked()
|
||||
if best != nil {
|
||||
p.maybeGrowLocked(bestCount)
|
||||
p.mu.Unlock()
|
||||
return best, best.newSid(), nil
|
||||
}
|
||||
if p.dialing > 0 {
|
||||
// Someone is already dialing the first conn; wait for it rather
|
||||
// than piling up redundant connections.
|
||||
gen := p.dialGen
|
||||
p.cond.Wait()
|
||||
if len(p.conns) == 0 && p.dialGen != gen && p.dialErr != nil {
|
||||
err := p.dialErr
|
||||
p.mu.Unlock()
|
||||
return nil, 0, err
|
||||
}
|
||||
continue
|
||||
}
|
||||
p.dialing++
|
||||
p.mu.Unlock()
|
||||
wc, err := p.dialWorker()
|
||||
p.mu.Lock()
|
||||
p.dialing--
|
||||
p.dialGen++
|
||||
p.dialErr = err
|
||||
if err != nil {
|
||||
p.cond.Broadcast()
|
||||
p.mu.Unlock()
|
||||
return nil, 0, err
|
||||
}
|
||||
if p.closed {
|
||||
// Close raced this dial: the conn must not enter the pool. Closing
|
||||
// it here, under p.mu, is a raw socket close — fine, and it makes
|
||||
// the shutdown atomic with the pool state.
|
||||
p.cond.Broadcast()
|
||||
p.mu.Unlock()
|
||||
_ = wc.fc.Close()
|
||||
return nil, 0, errPoolClosed
|
||||
}
|
||||
p.conns = append(p.conns, wc)
|
||||
p.cond.Broadcast()
|
||||
}
|
||||
}
|
||||
|
||||
// leastLoadedLocked returns the worker conn carrying the fewest streams.
|
||||
func (p *WorkerPool) leastLoadedLocked() (*WorkerConn, int) {
|
||||
var best *WorkerConn
|
||||
bestCount := 0
|
||||
for _, wc := range p.conns {
|
||||
n := wc.streamCount()
|
||||
if best == nil || n < bestCount {
|
||||
best = wc
|
||||
bestCount = n
|
||||
}
|
||||
}
|
||||
return best, bestCount
|
||||
}
|
||||
|
||||
// maybeGrowLocked opens one more worker conn in the background when the pool is
|
||||
// below maxConn and the least-loaded conn is already carrying streams. The
|
||||
// caller does not wait for it: it keeps using the conn it already has, and the
|
||||
// new one picks up subsequent players.
|
||||
func (p *WorkerPool) maybeGrowLocked(bestCount int) {
|
||||
if bestCount < StreamsBeforeGrowing {
|
||||
return
|
||||
}
|
||||
if p.closed {
|
||||
return // shutdown; do not start dials nobody will join the pool
|
||||
p.mu.Unlock()
|
||||
return nil, errPoolClosed
|
||||
}
|
||||
if len(p.conns)+p.dialing >= p.maxConn {
|
||||
if bestCount > SaturationThreshold {
|
||||
log.Printf("worker pool at maxConn=%d with %d streams on the least-loaded conn", p.maxConn, bestCount)
|
||||
}
|
||||
return
|
||||
if len(p.conns)+p.dialing >= p.maxTunnels {
|
||||
p.mu.Unlock()
|
||||
return nil, errTooManyTunnels
|
||||
}
|
||||
p.dialing++
|
||||
go func() {
|
||||
wc, err := p.dialWorker()
|
||||
var surplus *WorkerConn
|
||||
p.mu.Lock()
|
||||
p.dialing--
|
||||
p.dialGen++
|
||||
p.dialErr = err
|
||||
switch {
|
||||
case err != nil:
|
||||
case p.closed:
|
||||
surplus = wc // Close raced this background dial
|
||||
case len(p.conns) < p.maxConn:
|
||||
p.conns = append(p.conns, wc)
|
||||
default:
|
||||
surplus = wc // raced with another dial
|
||||
}
|
||||
p.cond.Broadcast()
|
||||
p.mu.Unlock()
|
||||
|
||||
wc, err := p.dialWorker()
|
||||
|
||||
p.mu.Lock()
|
||||
p.dialing--
|
||||
if err != nil {
|
||||
p.mu.Unlock()
|
||||
if err != nil {
|
||||
log.Printf("worker pool: background dial failed: %v", err)
|
||||
}
|
||||
if surplus != nil {
|
||||
_ = surplus.fc.Close()
|
||||
}
|
||||
}()
|
||||
return nil, err
|
||||
}
|
||||
if p.closed {
|
||||
// Close raced this dial: the conn must not enter the set.
|
||||
p.mu.Unlock()
|
||||
_ = wc.fc.Close()
|
||||
return nil, errPoolClosed
|
||||
}
|
||||
if len(p.conns) >= p.maxTunnels {
|
||||
p.mu.Unlock()
|
||||
_ = wc.fc.Close()
|
||||
return nil, errTooManyTunnels
|
||||
}
|
||||
p.conns[wc] = struct{}{}
|
||||
p.mu.Unlock()
|
||||
return wc, nil
|
||||
}
|
||||
|
||||
// dialWorker establishes one worker conn. It must be called without p.mu held.
|
||||
@@ -188,8 +109,6 @@ func (p *WorkerPool) dialWorker() (*WorkerConn, error) {
|
||||
resume: sess.resume,
|
||||
grace: p.client.resumeGrace(sess.hubGrace),
|
||||
id: int(p.connSeq.Add(1)),
|
||||
streams: make(map[int]*Stream),
|
||||
nextSid: 1,
|
||||
done: make(chan struct{}),
|
||||
}
|
||||
if p.client.statsOn() {
|
||||
@@ -200,7 +119,7 @@ func (p *WorkerPool) dialWorker() (*WorkerConn, error) {
|
||||
if sess.heartbeat {
|
||||
go wc.heartbeatLoop(p.client.cfg.pingInterval(), p.client.cfg.heartbeatTimeout())
|
||||
} else {
|
||||
log.Printf("worker conn: hub does not support the mux heartbeat; " +
|
||||
log.Printf("worker conn: hub does not support the worker heartbeat; " +
|
||||
"a silently dropped path will only be caught by TCP keepalive")
|
||||
}
|
||||
log.Printf("opened worker conn (send window %d, recv window %d, heartbeat %v, resume %v)",
|
||||
@@ -217,41 +136,32 @@ func (p *WorkerPool) count() int {
|
||||
func (p *WorkerPool) remove(wc *WorkerConn) {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
for i, c := range p.conns {
|
||||
if c == wc {
|
||||
p.conns = append(p.conns[:i], p.conns[i+1:]...)
|
||||
// A waiter parked on an empty pool must re-evaluate: it may now
|
||||
// need to dial rather than keep waiting for this conn.
|
||||
p.cond.Broadcast()
|
||||
return
|
||||
}
|
||||
}
|
||||
delete(p.conns, wc)
|
||||
}
|
||||
|
||||
func (p *WorkerPool) closeAll() {
|
||||
p.mu.Lock()
|
||||
p.closed = true
|
||||
conns := append([]*WorkerConn(nil), p.conns...)
|
||||
// Wake waiters parked in Allocate: the closed flag they re-check is the
|
||||
// only thing that can release them now that no conn will ever join.
|
||||
p.cond.Broadcast()
|
||||
conns := make([]*WorkerConn, 0, len(p.conns))
|
||||
for wc := range p.conns {
|
||||
conns = append(conns, wc)
|
||||
}
|
||||
p.mu.Unlock()
|
||||
for _, wc := range conns {
|
||||
_ = wc.fc.Close()
|
||||
}
|
||||
}
|
||||
|
||||
// WorkerConn is one multiplexed worker connection to the hub.
|
||||
//
|
||||
// Only genuinely per-connection state lives here. Client-wide values (the
|
||||
// shaper, the advertised receive window, the DATA chunk cap) belong to Client:
|
||||
// reading them through a connection pointer would make every such read a
|
||||
// re-parenting hazard once a stream can migrate between conns.
|
||||
// WorkerConn is one 1:1 worker connection to the hub: it carries exactly one
|
||||
// player. Only genuinely per-connection state lives here. Client-wide values
|
||||
// (the shaper, the advertised receive window, the DATA chunk cap) belong to
|
||||
// Client: reading them through a connection pointer would make every such
|
||||
// read a re-parenting hazard once a stream can migrate between conns.
|
||||
type WorkerConn struct {
|
||||
pool *WorkerPool
|
||||
fc *wire.FramedConn
|
||||
|
||||
sendWndInit int // hub's advertised per-stream receive window (our send budget)
|
||||
sendWndInit int // hub's advertised receive window (our send budget)
|
||||
|
||||
// resume is whether this conn negotiated stream resumption, and grace how
|
||||
// long a stream parked from it may keep trying to reattach. Both are
|
||||
@@ -266,17 +176,16 @@ type WorkerConn struct {
|
||||
id int // for log correlation only
|
||||
stats *connStats // nil unless diagnostics are enabled
|
||||
|
||||
mu sync.Mutex
|
||||
streams map[int]*Stream
|
||||
nextSid int
|
||||
closed bool // readLoop has exited; registerStream must refuse
|
||||
mu sync.Mutex
|
||||
stream *Stream
|
||||
closed bool // readLoop has exited; attach must refuse
|
||||
}
|
||||
|
||||
// heartbeatLoop proves the worker conn is still carrying frames end to end. TCP
|
||||
// alone cannot tell us: a middlebox that drops an established flow (conntrack
|
||||
// expiry, firewall state loss) sends no FIN or RST, so the read loop would park
|
||||
// forever, the dead conn would stay in the pool, and every player routed to it
|
||||
// would silently fail until the process restarted.
|
||||
// forever and the player on this conn would silently fail until the process
|
||||
// restarted.
|
||||
func (wc *WorkerConn) heartbeatLoop(interval, timeout time.Duration) {
|
||||
ticker := time.NewTicker(interval)
|
||||
defer ticker.Stop()
|
||||
@@ -286,12 +195,11 @@ func (wc *WorkerConn) heartbeatLoop(interval, timeout time.Duration) {
|
||||
return
|
||||
case <-ticker.C:
|
||||
if silent := time.Since(time.UnixMilli(wc.lastPong.Load())); silent > timeout {
|
||||
log.Printf("worker conn silent for %s; dropping it and its %d stream(s)",
|
||||
silent.Round(time.Second), wc.streamCount())
|
||||
log.Printf("worker conn silent for %s; dropping it", silent.Round(time.Second))
|
||||
_ = wc.fc.Close() // readLoop unblocks and tears everything down
|
||||
return
|
||||
}
|
||||
msg := wire.NewWriter().U8(MuxPing).VarInt(MuxCtlSid).I64(time.Now().UnixMilli()).Out()
|
||||
msg := wire.NewWriter().U8(MuxPing).I64(time.Now().UnixMilli()).Out()
|
||||
if err := wc.fc.WriteFrame(msg); err != nil {
|
||||
return
|
||||
}
|
||||
@@ -299,56 +207,39 @@ func (wc *WorkerConn) heartbeatLoop(interval, timeout time.Duration) {
|
||||
}
|
||||
}
|
||||
|
||||
func (wc *WorkerConn) streamCount() int {
|
||||
wc.mu.Lock()
|
||||
defer wc.mu.Unlock()
|
||||
return len(wc.streams)
|
||||
}
|
||||
|
||||
func (wc *WorkerConn) newSid() int {
|
||||
wc.mu.Lock()
|
||||
defer wc.mu.Unlock()
|
||||
sid := wc.nextSid
|
||||
wc.nextSid++
|
||||
return sid
|
||||
}
|
||||
|
||||
// registerStream publishes a stream in the conn's table, or reports false if
|
||||
// the conn has already died.
|
||||
// attach publishes a stream on this conn, or reports false if the conn has
|
||||
// already died (or is already bound — which would be a caller bug).
|
||||
//
|
||||
// The check is not advisory. Allocate hands out a (conn, sid) pair under the
|
||||
// pool lock, and the conn's readLoop can exit before the caller gets here — it
|
||||
// has then already swapped the stream map, so a blind insert would land in a map
|
||||
// nothing iterates and the stream would never be torn down. That normally hides
|
||||
// behind a failing SYN, but not on a half-open conn whose readLoop died on a
|
||||
// framing error while the socket is still writable. Callers must re-Allocate.
|
||||
func (wc *WorkerConn) registerStream(sid int, st *Stream) bool {
|
||||
// The check is not advisory. Dial hands out a conn, and the conn's readLoop
|
||||
// can exit before the caller gets here. A blind store would land on a conn
|
||||
// nothing iterates and the stream would never be torn down.
|
||||
func (wc *WorkerConn) attach(st *Stream) bool {
|
||||
wc.mu.Lock()
|
||||
defer wc.mu.Unlock()
|
||||
if wc.closed {
|
||||
if wc.closed || wc.stream != nil {
|
||||
return false
|
||||
}
|
||||
wc.streams[sid] = st
|
||||
wc.stream = st
|
||||
return true
|
||||
}
|
||||
|
||||
func (wc *WorkerConn) getStream(sid int) *Stream {
|
||||
func (wc *WorkerConn) getStream() *Stream {
|
||||
wc.mu.Lock()
|
||||
defer wc.mu.Unlock()
|
||||
return wc.streams[sid]
|
||||
return wc.stream
|
||||
}
|
||||
|
||||
func (wc *WorkerConn) removeStream(sid int) *Stream {
|
||||
func (wc *WorkerConn) detach() *Stream {
|
||||
wc.mu.Lock()
|
||||
defer wc.mu.Unlock()
|
||||
st := wc.streams[sid]
|
||||
delete(wc.streams, sid)
|
||||
st := wc.stream
|
||||
wc.stream = nil
|
||||
return st
|
||||
}
|
||||
|
||||
// readLoop dispatches inbound mux frames. It must never block on a stream's
|
||||
// destination: DATA is only enqueued (the per-stream writeLoop does the actual
|
||||
// destination writes), so one slow destination cannot stall other streams.
|
||||
// readLoop dispatches inbound tunnel frames. DATA is only enqueued (the
|
||||
// stream's writeLoop does the actual destination writes), so a stalled
|
||||
// destination cannot stall liveness / WND / FIN dispatch on this conn.
|
||||
func (wc *WorkerConn) readLoop() {
|
||||
for {
|
||||
payload, err := wc.fc.ReadFrame()
|
||||
@@ -363,24 +254,20 @@ func (wc *WorkerConn) readLoop() {
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
sid, err := r.VarInt()
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
switch ftype {
|
||||
case MuxData:
|
||||
if st := wc.getStream(sid); st != nil {
|
||||
if st := wc.getStream(); st != nil {
|
||||
st.deliverFromHub(r.Remaining())
|
||||
}
|
||||
case MuxWnd:
|
||||
if delta, err := r.VarInt(); err == nil && delta > 0 {
|
||||
if st := wc.getStream(sid); st != nil {
|
||||
if st := wc.getStream(); st != nil {
|
||||
st.grantSendWnd(delta)
|
||||
}
|
||||
}
|
||||
case MuxFin:
|
||||
// Graceful: drain what is already queued to the destination first.
|
||||
if st := wc.removeStream(sid); st != nil {
|
||||
if st := wc.detach(); st != nil {
|
||||
st.gracefulFin()
|
||||
}
|
||||
case MuxRst:
|
||||
@@ -389,7 +276,7 @@ func (wc *WorkerConn) readLoop() {
|
||||
if b, err := r.U8(); err == nil {
|
||||
reason = int(b)
|
||||
}
|
||||
if st := wc.removeStream(sid); st != nil {
|
||||
if st := wc.detach(); st != nil {
|
||||
st.onRst(reason)
|
||||
}
|
||||
case MuxResumeAck:
|
||||
@@ -399,12 +286,12 @@ func (wc *WorkerConn) readLoop() {
|
||||
if aerr != nil || derr != nil || cerr != nil {
|
||||
continue
|
||||
}
|
||||
if st := wc.getStream(sid); st != nil {
|
||||
if st := wc.getStream(); st != nil {
|
||||
st.deliverResume(resumeResult{accepted: accepted, delivered: delivered, cid: cid})
|
||||
}
|
||||
case MuxPing:
|
||||
nonce, _ := r.I64()
|
||||
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxPong).VarInt(MuxCtlSid).I64(nonce).Out())
|
||||
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxPong).I64(nonce).Out())
|
||||
case MuxPong:
|
||||
now := time.Now()
|
||||
wc.lastPong.Store(now.UnixMilli())
|
||||
@@ -416,46 +303,42 @@ func (wc *WorkerConn) readLoop() {
|
||||
}
|
||||
}
|
||||
default:
|
||||
log.Printf("worker: unknown mux type %d", ftype)
|
||||
log.Printf("worker: unknown frame type %d", ftype)
|
||||
}
|
||||
}
|
||||
// Connection lost: tear down all streams and drop from pool.
|
||||
// Connection lost: tear down the bound stream and drop from the set.
|
||||
close(wc.done)
|
||||
wc.pool.remove(wc)
|
||||
wc.mu.Lock()
|
||||
// Marked before the map is swapped, under the same lock, so a concurrent
|
||||
// registerStream either lands in the map we are about to drain or is refused.
|
||||
// Marked before the pointer is cleared, under the same lock, so a concurrent
|
||||
// attach either lands in the field we are about to drain or is refused.
|
||||
wc.closed = true
|
||||
streams := make([]*Stream, 0, len(wc.streams))
|
||||
for _, st := range wc.streams {
|
||||
streams = append(streams, st)
|
||||
}
|
||||
wc.streams = make(map[int]*Stream)
|
||||
st := wc.stream
|
||||
wc.stream = nil
|
||||
wc.mu.Unlock()
|
||||
if st == nil {
|
||||
return
|
||||
}
|
||||
// Only the tunnel leg died. Where the session negotiated resumption the
|
||||
// destination sockets are kept open and each stream reattaches over a fresh
|
||||
// destination socket is kept open and the stream reattaches over a fresh
|
||||
// conn (§7.5); otherwise this is the old, unconditional teardown.
|
||||
// During Close there is no reattach to come: a stream that parked now would
|
||||
// hold its destination socket open past shutdown, so teardown instead.
|
||||
if wc.pool.client.closing.Load() {
|
||||
for _, st := range streams {
|
||||
st.teardown(false)
|
||||
}
|
||||
st.teardown(false)
|
||||
return
|
||||
}
|
||||
for _, st := range streams {
|
||||
if !st.park(wc.grace) {
|
||||
st.teardown(false)
|
||||
}
|
||||
if !st.park(wc.grace) {
|
||||
st.teardown(false)
|
||||
}
|
||||
}
|
||||
|
||||
func (wc *WorkerConn) sendSyn(sid int, cid []byte) error {
|
||||
return wc.fc.WriteFrame(wire.NewWriter().U8(MuxSyn).VarInt(sid).Bytes(cid).Out())
|
||||
func (wc *WorkerConn) sendSyn(cid []byte) error {
|
||||
return wc.fc.WriteFrame(wire.NewWriter().U8(MuxSyn).Bytes(cid).Out())
|
||||
}
|
||||
|
||||
func (wc *WorkerConn) sendData(sid int, data []byte) error {
|
||||
err := wc.fc.WriteFrame(wire.NewWriter().U8(MuxData).VarInt(sid).Bytes(data).Out())
|
||||
func (wc *WorkerConn) sendData(data []byte) error {
|
||||
err := wc.fc.WriteFrame(wire.NewWriter().U8(MuxData).Bytes(data).Out())
|
||||
if wc.stats != nil {
|
||||
wc.stats.framesOut.Add(1)
|
||||
if err != nil {
|
||||
@@ -465,42 +348,27 @@ func (wc *WorkerConn) sendData(sid int, data []byte) error {
|
||||
return err
|
||||
}
|
||||
|
||||
func (wc *WorkerConn) sendFin(sid int) {
|
||||
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxFin).VarInt(sid).Out())
|
||||
func (wc *WorkerConn) sendFin() {
|
||||
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxFin).Out())
|
||||
}
|
||||
|
||||
func (wc *WorkerConn) sendRst(sid int) {
|
||||
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxRst).VarInt(sid).Out())
|
||||
func (wc *WorkerConn) sendRst() {
|
||||
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxRst).Out())
|
||||
}
|
||||
|
||||
func (wc *WorkerConn) sendWndUpdate(sid, delta int) {
|
||||
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxWnd).VarInt(sid).VarInt(delta).Out())
|
||||
func (wc *WorkerConn) sendWndUpdate(delta int) {
|
||||
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxWnd).VarInt(delta).Out())
|
||||
}
|
||||
|
||||
// leg binds a stream to one worker conn. The two fields are only meaningful
|
||||
// together: stream ids are per-conn and restart at 1, so conn A's sid 3 and
|
||||
// conn B's sid 3 belong to different players. A torn read across the two would
|
||||
// credit, reset or FIN a stranger's stream, so the pair is swapped as one
|
||||
// immutable value rather than as two fields.
|
||||
type leg struct {
|
||||
wc *WorkerConn
|
||||
sid int
|
||||
}
|
||||
|
||||
// String formats one (conn, sid) snapshot for log correlation. Stream ids
|
||||
// restart at 1 per conn, so a bare sid cannot be traced across a reattach —
|
||||
// the conn id is what ties the log lines together.
|
||||
func (lg *leg) String() string { return fmt.Sprintf("conn%d/sid%d", lg.wc.id, lg.sid) }
|
||||
|
||||
// Stream bridges one player (via the hub) to one destination connection.
|
||||
//
|
||||
// Data from the hub is queued and written to the destination by a dedicated
|
||||
// writeLoop goroutine. The queue is bounded by the advertised receive window —
|
||||
// the hub never sends more un-credited bytes, so overflow is a protocol
|
||||
// violation and resets the stream.
|
||||
// violation and resets the tunnel.
|
||||
type Stream struct {
|
||||
client *Client
|
||||
leg atomic.Pointer[leg]
|
||||
client *Client
|
||||
wc atomic.Pointer[WorkerConn]
|
||||
mapping Mapping
|
||||
srcIP string
|
||||
srcPort int
|
||||
@@ -527,8 +395,8 @@ type Stream struct {
|
||||
// never interleave with live ones.
|
||||
//
|
||||
// Deliberately not s.mu: deliverFromHub takes s.mu from the worker readLoop,
|
||||
// and holding s.mu across a WriteFrame would stall frame dispatch for every
|
||||
// other stream on the connection.
|
||||
// and holding s.mu across a WriteFrame would stall frame dispatch (WND/FIN
|
||||
// /heartbeat replies) on this conn.
|
||||
sendMu sync.Mutex
|
||||
un unackedBuf // guarded by sendMu
|
||||
|
||||
@@ -578,10 +446,10 @@ type qentry struct {
|
||||
fromHub bool
|
||||
}
|
||||
|
||||
func newStream(c *Client, wc *WorkerConn, sid int, cid []byte, m Mapping, ip string, port int) *Stream {
|
||||
func newStream(c *Client, wc *WorkerConn, cid []byte, m Mapping, ip string, port int) *Stream {
|
||||
s := &Stream{client: c, cid: cid, mapping: m, srcIP: ip, srcPort: port,
|
||||
resumable: wc.resume, sendWnd: wc.sendWndInit, done: make(chan struct{})}
|
||||
s.leg.Store(&leg{wc: wc, sid: sid})
|
||||
s.wc.Store(wc)
|
||||
if c.statsOn() {
|
||||
s.stats = &streamStats{opened: time.Now()}
|
||||
}
|
||||
@@ -592,20 +460,28 @@ func newStream(c *Client, wc *WorkerConn, sid int, cid []byte, m Mapping, ip str
|
||||
return s
|
||||
}
|
||||
|
||||
// conn returns the stream's current binding. Every caller must take exactly one
|
||||
// snapshot and use both fields from it; re-loading mid-operation reintroduces
|
||||
// the torn-pair hazard the leg exists to prevent.
|
||||
func (s *Stream) conn() *leg { return s.leg.Load() }
|
||||
// conn returns the stream's current worker conn. May be the original or a
|
||||
// reattach; callers that send must take one snapshot and use it for the
|
||||
// whole operation so a concurrent rebind cannot split a write across conns.
|
||||
func (s *Stream) conn() *WorkerConn { return s.wc.Load() }
|
||||
|
||||
func (s *Stream) name() string {
|
||||
if wc := s.conn(); wc != nil {
|
||||
return fmt.Sprintf("conn%d", wc.id)
|
||||
}
|
||||
return "conn?"
|
||||
}
|
||||
|
||||
// run dials the destination, optionally writes the PROXY v2 header, then pumps
|
||||
// destination -> hub (respecting the stream send window when negotiated).
|
||||
// destination -> hub (respecting the send window when negotiated).
|
||||
func (s *Stream) run() {
|
||||
dest, err := net.DialTimeout("tcp", s.mapping.Destination, 10*time.Second)
|
||||
if err != nil {
|
||||
lg := s.conn()
|
||||
log.Printf("stream %s: dial %s failed: %v", lg, s.mapping.Destination, err)
|
||||
lg.wc.removeStream(lg.sid)
|
||||
lg.wc.sendRst(lg.sid)
|
||||
log.Printf("stream %s: dial %s failed: %v", s.name(), s.mapping.Destination, err)
|
||||
if wc := s.conn(); wc != nil {
|
||||
wc.detach()
|
||||
wc.sendRst()
|
||||
}
|
||||
s.teardown(false)
|
||||
return
|
||||
}
|
||||
@@ -616,7 +492,7 @@ func (s *Stream) run() {
|
||||
if s.mapping.ProxyProtocol {
|
||||
if hdr := s.buildProxyHeader(dest); hdr != nil {
|
||||
if _, err := dest.Write(hdr); err != nil {
|
||||
log.Printf("stream %s: proxy header write: %v", s.conn(), err)
|
||||
log.Printf("stream %s: proxy header write: %v", s.name(), err)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -663,7 +539,7 @@ func (s *Stream) run() {
|
||||
}
|
||||
|
||||
// sendToHub forwards destination bytes to the hub in bounded DATA frames,
|
||||
// honoring both the stream send window and the client-wide bandwidth cap.
|
||||
// honoring both the send window and the client-wide bandwidth cap.
|
||||
// Returns false once the stream closed or the worker conn failed.
|
||||
func (s *Stream) sendToHub(data []byte) bool {
|
||||
for len(data) > 0 {
|
||||
@@ -674,7 +550,7 @@ func (s *Stream) sendToHub(data []byte) bool {
|
||||
// Credit first, bandwidth second. The reverse order would charge the
|
||||
// budget for bytes still parked on an exhausted window, so the client
|
||||
// would throttle itself below the configured rate. Holding credit while
|
||||
// waiting for tokens is free — credit is per-stream, and the hub returns
|
||||
// waiting for tokens is free — credit is per-tunnel, and the hub returns
|
||||
// it as it drains data to the player, independent of our pacing.
|
||||
if !s.acquireSendWnd(n) {
|
||||
return false
|
||||
@@ -714,9 +590,6 @@ func (s *Stream) sendToHub(data []byte) bool {
|
||||
// one taken before the attempt. That also makes a failure survivable: while the
|
||||
// stream can still be resumed the bytes are already safe, and the reattach
|
||||
// replays them from wherever the hub says it got to.
|
||||
//
|
||||
// Note the old code let a failed write drop the rest of the chunk on the floor —
|
||||
// the caller's slice advance sat after the error return.
|
||||
func (s *Stream) emit(chunk []byte) bool {
|
||||
s.sendMu.Lock()
|
||||
if s.resumable {
|
||||
@@ -725,8 +598,13 @@ func (s *Stream) emit(chunk []byte) bool {
|
||||
s.un.advance(s.ackedOffset.Load())
|
||||
s.un.append(chunk)
|
||||
}
|
||||
lg := s.conn()
|
||||
err := lg.wc.sendData(lg.sid, chunk)
|
||||
wc := s.conn()
|
||||
var err error
|
||||
if wc != nil {
|
||||
err = wc.sendData(chunk)
|
||||
} else {
|
||||
err = errPoolClosed
|
||||
}
|
||||
s.sendMu.Unlock()
|
||||
|
||||
if err == nil {
|
||||
@@ -792,7 +670,7 @@ func (s *Stream) writeLoop() {
|
||||
|
||||
// deliverFromHub enqueues hub bytes for the destination. Called from the worker
|
||||
// readLoop; it never blocks — a peer that exceeds the advertised window is a
|
||||
// protocol violator and gets the stream reset.
|
||||
// protocol violator and gets the tunnel reset.
|
||||
func (s *Stream) deliverFromHub(data []byte) {
|
||||
if s.vel != nil {
|
||||
s.vel.ObserveC2S(data) // observation only; bytes still forwarded verbatim
|
||||
@@ -804,10 +682,11 @@ func (s *Stream) deliverFromHub(data []byte) {
|
||||
}
|
||||
if s.qBytes+len(data) > s.client.streamWnd {
|
||||
s.mu.Unlock()
|
||||
lg := s.conn()
|
||||
log.Printf("stream %s: peer exceeded flow-control window; resetting", lg)
|
||||
lg.wc.removeStream(lg.sid)
|
||||
lg.wc.sendRst(lg.sid)
|
||||
log.Printf("stream %s: peer exceeded flow-control window; resetting", s.name())
|
||||
if wc := s.conn(); wc != nil {
|
||||
wc.detach()
|
||||
wc.sendRst()
|
||||
}
|
||||
s.teardown(false)
|
||||
return
|
||||
}
|
||||
@@ -891,8 +770,9 @@ func (s *Stream) credit(n int) {
|
||||
delta := s.consumed
|
||||
s.consumed = 0
|
||||
s.mu.Unlock()
|
||||
lg := s.conn()
|
||||
lg.wc.sendWndUpdate(lg.sid, delta)
|
||||
if wc := s.conn(); wc != nil {
|
||||
wc.sendWndUpdate(delta)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Stream) buildProxyHeader(dest net.Conn) []byte {
|
||||
@@ -957,10 +837,11 @@ func (s *Stream) teardown(notifyHub bool) {
|
||||
if dest != nil {
|
||||
_ = dest.Close()
|
||||
}
|
||||
lg := s.conn()
|
||||
lg.wc.removeStream(lg.sid)
|
||||
if notifyHub {
|
||||
lg.wc.sendFin(lg.sid)
|
||||
if wc := s.conn(); wc != nil {
|
||||
wc.detach()
|
||||
if notifyHub {
|
||||
wc.sendFin()
|
||||
}
|
||||
}
|
||||
s.logSummary()
|
||||
}
|
||||
|
||||
+74
-89
@@ -15,9 +15,9 @@ on the wire, read [PROTOCOL.md](../PROTOCOL.md).
|
||||
┌──────────┐ Intent 17, magic 0x01 │ │ │
|
||||
│ Client │ ◀──────── control session ──────│ • pattern reg │ │
|
||||
│ (Go) │ ────────────────────────────────│ • CID table │ │
|
||||
│ │ Intent 17, magic 0x02 │ • mux demux │ │
|
||||
│ │ Intent 17, magic 0x02 │ • 1:1 workers │ │
|
||||
│ │ ═════════ worker conns ═════════│ │ │
|
||||
└──────────┘ multiplexed player streams └───────────────┘ │
|
||||
└──────────┘ one TCP conn per player └───────────────┘ │
|
||||
│ │
|
||||
▼ MC bytes (+ optional HAProxy v2) │
|
||||
┌───────────────┐ │
|
||||
@@ -68,15 +68,15 @@ Player Hub Client Destinatio
|
||||
│ │ pause player socket,
|
||||
│ │ buffer bytes, mint CID
|
||||
│ │─ ControlRequest(CID, pattern, ip:port) ─▶
|
||||
│ │ allocate worker+stream
|
||||
│ │◀──────── SYN(streamId, CID) ────────────│
|
||||
│ │ dial dedicated worker
|
||||
│ │◀──────────── SYN(CID) ──────────────────│
|
||||
│ │ takePending(CID) → bind dial destination,
|
||||
│ │ forward buffered bytes write HAProxy v2 hdr
|
||||
│ │─ DATA(streamId, handshake…) ───▶ ── handshake ──▶│
|
||||
│ resume ─────────────────────────────│ bridge stream ⇄ dest
|
||||
│ │─ DATA(handshake…) ─────────────▶ ── handshake ──▶│
|
||||
│ resume ─────────────────────────────│ bridge conn ⇄ dest
|
||||
│══════════════ player bytes ══ DATA ══▶│════ DATA ═══▶ dest.write │
|
||||
│◀═══════════ dest bytes ═══ DATA ══════│◀═══ DATA ════ dest.read │
|
||||
│ player closes ──────────────────────│─ FIN(streamId) ────────▶ close dest │
|
||||
│ player closes ──────────────────────│─ FIN ──────────────────▶ close dest │
|
||||
```
|
||||
|
||||
Key points:
|
||||
@@ -87,6 +87,11 @@ Key points:
|
||||
player's hostname — in `ControlRequest`, so the client can look it straight up
|
||||
in its own route table. Invalid patterns are rejected at registration with a
|
||||
non-zero `RegisterAck` status.
|
||||
* A **per-IP limiter** runs first: a token bucket (`playerRatePerSec` /
|
||||
`playerBurst`) and a concurrent-socket cap (`maxPlayersPerIp`). It applies
|
||||
only to player intents — never Intent 17 — and unmatched hostnames still
|
||||
consume a token, so a hostname scan is not a free flood. A refusal closes
|
||||
the socket before CID minting or pause.
|
||||
* The hub **pauses** the player socket the instant it matches, so no player
|
||||
bytes are lost while the takeover is arranged; the buffered handshake is
|
||||
forwarded **verbatim**, so the real server sees exactly what the player sent
|
||||
@@ -94,54 +99,38 @@ Key points:
|
||||
* **CID** is 16 random bytes minted by the hub and delivered only over the
|
||||
encrypted control session, so only the intended client learns it. Any worker
|
||||
connection presenting the correct CID is allowed to take over — that secrecy
|
||||
is what binds a worker stream to the right pending player without any explicit
|
||||
is what binds a worker conn to the right pending player without any explicit
|
||||
client identity.
|
||||
* Disconnects are symmetric: player-close → hub sends `FIN` → client closes the
|
||||
destination; destination-close → client sends `FIN` → hub closes the player.
|
||||
|
||||
## 3. Multiplexing (worker connections)
|
||||
## 3. Worker connections (1:1)
|
||||
|
||||
A worker connection is one encrypted TCP link carrying many **streams**. The
|
||||
frame is intentionally tiny (PROTOCOL.md §7):
|
||||
A worker connection is one encrypted TCP link carrying **exactly one player**.
|
||||
The frame is intentionally tiny (PROTOCOL.md §7):
|
||||
|
||||
```
|
||||
[plaintext VarInt length][ FrameType u8 | StreamID VarInt | Data… ] (payload encrypted)
|
||||
[plaintext VarInt length][ FrameType u8 | Data… ] (payload encrypted)
|
||||
```
|
||||
|
||||
Only the client opens streams (`SYN`), so stream-id allocation is a simple
|
||||
per-connection counter with no coordination.
|
||||
There is no stream id. The client dials a fresh worker for each
|
||||
`ControlRequest` and binds it with `SYN(CID)` after `SessionReady`. A second
|
||||
bind on the same conn is a protocol violation.
|
||||
|
||||
### 3.1 Pool & allocation
|
||||
### 3.1 Allocation
|
||||
|
||||
The client keeps 1…`maxConn` worker connections and places each new stream on
|
||||
the **least-loaded** one. The pool grows **breadth-first**: it dials out to
|
||||
`maxConn` before stacking streams, so that no single TCP connection ever becomes
|
||||
the shared point of failure for every player on the tunnel (PROTOCOL.md §7.1):
|
||||
The client holds at most `maxTunnels` live worker connections (default 256). A
|
||||
`ControlRequest` either gets its own TCP connection or is dropped — the hub
|
||||
then closes the player when `pendingTimeoutMs` fires.
|
||||
|
||||
```
|
||||
pick least-loaded conn; use it
|
||||
if leastLoaded.streams >= 1 and pool.size + dialsInFlight < maxConn:
|
||||
dial another worker conn in the background # the stream just placed does not wait
|
||||
```
|
||||
A dial is **never performed while holding the live-set lock** — session
|
||||
establishment is network I/O, and one unresponsive hub must not park every
|
||||
other player behind it. Each caller dials independently; callers are not
|
||||
serialized behind a single in-flight handshake.
|
||||
|
||||
The new connection becomes the least-loaded one and picks up subsequent streams.
|
||||
Once the pool is at `maxConn`, streams stack on the least-loaded connection;
|
||||
going past 8 active streams there is logged as pool saturation but is not an
|
||||
error.
|
||||
|
||||
Two properties of the dialing path matter as much as the placement rule:
|
||||
|
||||
* A dial is **never performed while holding the pool lock** — session
|
||||
establishment is network I/O, and one unresponsive hub must not park every
|
||||
other player behind it.
|
||||
* Only when the pool is *empty* does a caller dial synchronously, and then
|
||||
exactly one caller dials while the others wait on its result, so a burst of
|
||||
arrivals cannot open a burst of redundant connections.
|
||||
|
||||
The e2e test `TestConcurrentStreamsUseMultipleConns` drives 20 simultaneous
|
||||
streams with `maxConn=4` and confirms they spread over more than one connection
|
||||
without exceeding the cap; `TestAllocateDoesNotWedgePoolOnStalledHub` covers the
|
||||
stalled-dial path.
|
||||
The e2e test `TestEachPlayerGetsOwnWorker` drives concurrent players and
|
||||
confirms one worker conn each, without exceeding `maxTunnels`;
|
||||
`TestDialDoesNotWedgeOnStalledHub` covers the stalled-dial path.
|
||||
|
||||
## 4. Encryption
|
||||
|
||||
@@ -166,31 +155,28 @@ stalled-dial path.
|
||||
non-blocking; crypto is CPU-cheap. This trades multi-core scaling for
|
||||
simplicity and correctness.
|
||||
* **Client:** goroutine-per-concern. One goroutine reads each connection
|
||||
(control or worker); `WriteFrame` is mutex-serialized so many stream goroutines
|
||||
can share a worker connection safely. Each stream has two goroutines: `run`
|
||||
pumps destination → hub, and `writeLoop` is the only writer to the
|
||||
destination, draining a per-stream queue fed by the worker readLoop. The
|
||||
readLoop itself never writes to a destination, so a stalled destination can
|
||||
never block frame dispatch for other streams.
|
||||
(control or worker); `WriteFrame` is mutex-serialized. Each tunnel has two
|
||||
goroutines: `run` pumps destination → hub, and `writeLoop` is the only writer
|
||||
to the destination, draining a queue fed by the worker readLoop. The
|
||||
readLoop itself never writes to a destination, so a stalled destination cannot
|
||||
stall WND / FIN / heartbeat dispatch on that conn.
|
||||
|
||||
## 6. Back-pressure & flow control
|
||||
|
||||
Three mechanisms operate at different granularities:
|
||||
|
||||
* **Per-stream credit windows** (PROTOCOL.md §7.3; the windows are exchanged
|
||||
at session establishment): each stream direction has an independent byte
|
||||
* **Per-connection credit windows** (PROTOCOL.md §7.3; the windows are exchanged
|
||||
at session establishment): each tunnel direction has an independent byte
|
||||
budget equal to the receiver's advertised window (default 256 KiB). A sender
|
||||
that exhausts a
|
||||
stream's window pauses *only that stream's source* — the hub pauses the one
|
||||
player socket, the client parks the one destination-reader goroutine. Credit
|
||||
is granted back (`WND` frames, batched at half-window) as bytes are actually
|
||||
written to the terminal socket. The result: a slow player or slow destination
|
||||
jams its own stream at a bounded buffer size and nothing else. This is what
|
||||
eliminates head-of-line blocking between streams.
|
||||
* **Aggregate TCP back-pressure** on each worker connection: when the shared
|
||||
socket itself is congested (total bandwidth, not one stream), the hub parks
|
||||
all sending players until it drains, and the client's `WriteFrame` blocks.
|
||||
This is fair — when the pipe is genuinely full, everyone should slow down.
|
||||
that exhausts a window pauses *only that player's source* — the hub pauses the
|
||||
one player socket, the client parks the one destination-reader goroutine.
|
||||
Credit is granted back (`WND` frames, batched at half-window) as bytes are
|
||||
actually written to the terminal socket. The result: a slow player or slow
|
||||
destination jams its own tunnel at a bounded buffer size and nothing else.
|
||||
* **TCP back-pressure** on each worker connection: when that socket is
|
||||
congested, the hub parks that one player until it drains, and the client's
|
||||
`WriteFrame` blocks. Because the conn is 1:1, TCP HOL cannot stall another
|
||||
player.
|
||||
* **Client egress shaping** (optional, `maxBandwidth`; `client/shaper.go`): a
|
||||
rate cap on everything the client sends to the hub, across all worker conns.
|
||||
|
||||
@@ -198,18 +184,18 @@ The first two mechanisms have no time dimension. A credit window bounds how many
|
||||
bytes are *in flight*, and TCP back-pressure only reacts once the pipe is already
|
||||
full — which on a residential uplink is too late. One player loading chunks fills
|
||||
the line, the standing queue grows to seconds, and every other player's keepalive
|
||||
times out. Nothing in §7.3 prevents that: each stream is individually
|
||||
times out. Nothing in §7.3 prevents that: each tunnel is individually
|
||||
well-behaved, and collectively they still overrun the link.
|
||||
|
||||
The shaper closes that gap with a token bucket for the rate and start-time fair
|
||||
queueing for the split. A global virtual clock advances with each grant; every
|
||||
stream remembers where its last request finished, and a new request is stamped
|
||||
`max(stream.vfinish, vclock)`. Lowest stamp wins. A stream that keeps sending
|
||||
pushes its own stamp further out and yields; a stream returning from idle is
|
||||
tunnel remembers where its last request finished, and a new request is stamped
|
||||
`max(tunnel.vfinish, vclock)`. Lowest stamp wins. A tunnel that keeps sending
|
||||
pushes its own stamp further out and yields; a tunnel returning from idle is
|
||||
clamped back to the clock, so it cannot bank credit for time it did not use, but
|
||||
is not penalised for the idleness either. A stream sending a few hundred bytes
|
||||
is not penalised for the idleness either. A tunnel sending a few hundred bytes
|
||||
gets a nearer stamp than one sending a full chunk, so keepalives and chat overtake
|
||||
bulk terrain data for free. One stream alone still gets the entire rate.
|
||||
bulk terrain data for free. One tunnel alone still gets the entire rate.
|
||||
|
||||
Two details keep bursts cheap. The bucket banks 200 ms of transmission, so a
|
||||
player joining spends it at once instead of paying for the cap in visible
|
||||
@@ -222,7 +208,7 @@ This is entirely client-local: nothing about it appears on the wire, and the hub
|
||||
is unaware. Only DATA is shaped — delaying a `FIN`, `WND` or `PONG` would cause
|
||||
the false-death detection §7.4 exists to avoid.
|
||||
|
||||
The window also bounds memory: a stream can hold at most one window of
|
||||
The window also bounds memory: a tunnel can hold at most one window of
|
||||
undelivered data per direction (the client's pre-connect handshake buffer is
|
||||
covered by the same bound). With stream resumption enabled (§7) the *sender*
|
||||
holds a second window — the bytes it has sent but the peer has not yet credited,
|
||||
@@ -230,44 +216,42 @@ kept so they can be retransmitted after an outage. That is not a new bound so
|
||||
much as the existing one made symmetric: the region is exactly what flow control
|
||||
already declared outstanding, which is why resumption needs no cap of its own.
|
||||
|
||||
Per-stream flow control is mandatory: the hub rejects a session whose Rekey
|
||||
Per-connection flow control is mandatory: the hub rejects a session whose Rekey
|
||||
lacks the STREAM_FC flag, and the client rejects a hub that does not echo it —
|
||||
peers that predate the mechanism cannot connect at all.
|
||||
|
||||
What remains (by design) is TCP-level head-of-line blocking: a lost packet on
|
||||
a worker connection stalls all its streams for one retransmit. That is inherent
|
||||
to mux-over-TCP; the connection pool is the mitigation, and a datagram
|
||||
transport (QUIC) would be the escape hatch if it ever matters.
|
||||
TCP-level head-of-line blocking is now confined to one player: a lost packet
|
||||
stalls only that player's tunnel for one retransmit. The cost is one handshake
|
||||
and one NAT mapping per player instead of per pool slot.
|
||||
|
||||
## 7. Failure & recovery
|
||||
|
||||
* **Control session drop:** the client retries immediately, then backs off to a
|
||||
10s cap, and re-registers all patterns. Existing worker connections and their
|
||||
live streams are unaffected — they ride worker conns, which a control-session
|
||||
players are unaffected — they ride worker conns, which a control-session
|
||||
close never touches. The hub meanwhile keeps that session's routes as
|
||||
*orphaned* for `registrationGraceMs` (PROTOCOL.md §5.2) and **holds** players
|
||||
arriving on them instead of refusing them, replaying the control request once
|
||||
a client re-registers the pattern. Without that, the reconnect window is one
|
||||
in which every new player is told there is no such server.
|
||||
* **Worker connection drop:** the connection leaves the pool either way. What
|
||||
happens to its streams depends on whether STREAM_RESUME was negotiated:
|
||||
* *without it* — every stream is torn down (destinations closed) and the hub
|
||||
closes the corresponding player sockets, as it always did;
|
||||
* *with it* — the streams are **hung** instead (PROTOCOL.md §7.5). The
|
||||
destination sockets stay open, the hub pauses the player sockets and holds
|
||||
them for its grace period, and the client reattaches each stream over a
|
||||
freshly dialed connection, replaying byte-exactly from the offset the peer
|
||||
reports. Players see a stall rather than a disconnect.
|
||||
* **Worker connection drop:** the connection leaves the live set either way. What
|
||||
happens to its player depends on whether STREAM_RESUME was negotiated:
|
||||
* *without it* — the tunnel is torn down (destination closed) and the hub
|
||||
closes the player socket, as it always did;
|
||||
* *with it* — the player is **hung** instead (PROTOCOL.md §7.5). The
|
||||
destination socket stays open, the hub pauses the player socket and holds
|
||||
it for its grace period, and the client reattaches over a freshly dialed
|
||||
connection, replaying byte-exactly from the offset the peer reports. The
|
||||
player sees a stall rather than a disconnect.
|
||||
* **Pending timeout:** if no worker takes over a matched player within
|
||||
`pendingTimeoutMs`, the hub drops the pending entry and closes the player.
|
||||
* **Bad PSK / bad timestamp / bad magic:** the hub closes the TCP connection;
|
||||
the client's session establishment fails fast.
|
||||
|
||||
Resumption is worth the machinery because a worker connection is only the
|
||||
*middle* leg of every stream it carries. When it dies both terminal sockets are
|
||||
*middle* leg of the player it carries. When it dies both terminal sockets are
|
||||
usually still healthy, so the old behaviour discarded working connections
|
||||
because a replaceable transport failed — one conntrack expiry disconnected every
|
||||
player sharing that connection. It also has to be byte-exact rather than
|
||||
because a replaceable transport failed. It also has to be byte-exact rather than
|
||||
best-effort: bytes handed to a dying socket are lost with no notification and the
|
||||
frame cipher cannot be resynchronized, so an approximate reattach would splice
|
||||
the tunneled protocol mid-packet, which is worse than a clean close.
|
||||
@@ -279,13 +263,14 @@ wait on the control reconnect backoff.
|
||||
## 8. Known limitations
|
||||
|
||||
1. No AEAD — payload integrity/authenticity is not cryptographically guaranteed.
|
||||
2. TCP-level head-of-line blocking within a worker connection (lost packets;
|
||||
see §6) — per-stream flow control removes the application-level variant only.
|
||||
2. One handshake and one NAT mapping per player (the cost of dropping mux).
|
||||
3. Single-event-loop hub (see §5) bounds throughput to one core.
|
||||
4. `Intent 18` is reserved but only stubbed (the hub logs and closes).
|
||||
4. `Intent 18` is reserved for a plaintext status probe.
|
||||
5. Pattern ownership is last-writer-wins; two clients registering the identical
|
||||
pattern string will silently reassign it. Overlapping-but-distinct regexes are
|
||||
both kept, and when several match one hostname the winner is unspecified.
|
||||
6. The player IP limiter matches exact addresses. A `/64` of IPv6 clients looks
|
||||
like many independent IPs; aggregation is a later, isolated change.
|
||||
|
||||
These are deliberate scope choices for a connectivity-focused P2P tool, not
|
||||
oversights; each is a small, well-isolated change away from being hardened.
|
||||
|
||||
@@ -14,12 +14,12 @@ import (
|
||||
// startClientWithBandwidth is startClient with an egress cap, for the shaping
|
||||
// tests. Only the client→hub direction is shaped, which in this harness is the
|
||||
// leg carrying the mock destination's echo back to the player.
|
||||
func startClientWithBandwidth(t *testing.T, hubAddr, psk string, maxConn int, bandwidth string, mappings []client.Mapping) *client.Client {
|
||||
func startClientWithBandwidth(t *testing.T, hubAddr, psk string, maxTunnels int, bandwidth string, mappings []client.Mapping) *client.Client {
|
||||
t.Helper()
|
||||
cfg := &client.Config{
|
||||
Server: hubAddr,
|
||||
PSK: psk,
|
||||
MaxConn: maxConn,
|
||||
MaxTunnels: maxTunnels,
|
||||
PingIntervalMs: 20000,
|
||||
MaxBandwidth: bandwidth,
|
||||
Mappings: mappings,
|
||||
@@ -104,9 +104,9 @@ func TestCappedBandwidthDoesNotStarveLightStreams(t *testing.T) {
|
||||
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
|
||||
startHub(t, port, psk)
|
||||
dest := newMockDest(t, modeEcho)
|
||||
// maxConn=1 forces every stream onto one worker conn, so nothing but the
|
||||
// shaper is deciding who gets the link.
|
||||
startClientWithBandwidth(t, hubAddr, psk, 1, "1MB/s", []client.Mapping{
|
||||
// Each player has its own worker conn; the shaper is still the only thing
|
||||
// splitting the shared uplink budget.
|
||||
startClientWithBandwidth(t, hubAddr, psk, 4, "1MB/s", []client.Mapping{
|
||||
{Pattern: "mc.local", Destination: dest.addr},
|
||||
})
|
||||
|
||||
|
||||
+17
-20
@@ -13,19 +13,19 @@ import (
|
||||
|
||||
// startClient builds and starts an in-process client against the hub, with the
|
||||
// given mappings, returning the running client.
|
||||
func startClient(t *testing.T, hubAddr, psk string, maxConn int, mappings []client.Mapping) *client.Client {
|
||||
func startClient(t *testing.T, hubAddr, psk string, maxTunnels int, mappings []client.Mapping) *client.Client {
|
||||
t.Helper()
|
||||
return startClientWithPing(t, hubAddr, psk, maxConn, 20000, mappings)
|
||||
return startClientWithPing(t, hubAddr, psk, maxTunnels, 20000, mappings)
|
||||
}
|
||||
|
||||
// startClientWithPing is startClient with an explicit heartbeat interval, for
|
||||
// tests that need liveness detection to trigger quickly.
|
||||
func startClientWithPing(t *testing.T, hubAddr, psk string, maxConn, pingMs int, mappings []client.Mapping) *client.Client {
|
||||
func startClientWithPing(t *testing.T, hubAddr, psk string, maxTunnels, pingMs int, mappings []client.Mapping) *client.Client {
|
||||
t.Helper()
|
||||
return startClientCfg(t, &client.Config{
|
||||
Server: hubAddr,
|
||||
PSK: psk,
|
||||
MaxConn: maxConn,
|
||||
MaxTunnels: maxTunnels,
|
||||
PingIntervalMs: pingMs,
|
||||
Mappings: mappings,
|
||||
})
|
||||
@@ -107,7 +107,7 @@ func TestRegexPatternMatch(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestLargeTransfer pushes a multi-megabyte payload both ways to exercise mux
|
||||
// TestLargeTransfer pushes a multi-megabyte payload both ways to exercise
|
||||
// framing and back-pressure.
|
||||
func TestLargeTransfer(t *testing.T) {
|
||||
const psk = "e2e-large"
|
||||
@@ -147,18 +147,17 @@ func TestLargeTransfer(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestConcurrentStreamsUseMultipleConns confirms the least-loaded allocator
|
||||
// grows the pool breadth-first: concurrent streams spread over several worker
|
||||
// connections rather than stacking on one, without ever exceeding maxConn.
|
||||
func TestConcurrentStreamsUseMultipleConns(t *testing.T) {
|
||||
// TestEachPlayerGetsOwnWorker confirms the 1:1 rule: N concurrent players
|
||||
// produce N worker connections, and the maxTunnels cap is honoured.
|
||||
func TestEachPlayerGetsOwnWorker(t *testing.T) {
|
||||
const psk = "e2e-concurrent"
|
||||
const n = 20
|
||||
const maxConn = 4
|
||||
const n = 8
|
||||
const maxTunnels = 16
|
||||
port := freePort(t)
|
||||
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
|
||||
startHub(t, port, psk)
|
||||
dest := newMockDest(t, modeEcho)
|
||||
c := startClient(t, hubAddr, psk, maxConn, []client.Mapping{
|
||||
c := startClient(t, hubAddr, psk, maxTunnels, []client.Mapping{
|
||||
{Pattern: "mc.local", Destination: dest.addr},
|
||||
})
|
||||
|
||||
@@ -168,8 +167,6 @@ func TestConcurrentStreamsUseMultipleConns(t *testing.T) {
|
||||
_ = pc.Close()
|
||||
}
|
||||
}()
|
||||
// Establish streams sequentially so allocation is deterministic; keep them
|
||||
// all open to hold streams active.
|
||||
for i := 0; i < n; i++ {
|
||||
pc := dialPlayer(t, hubAddr, "mc.local")
|
||||
playerEcho(t, pc, []byte(fmt.Sprintf("hello-%d", i)))
|
||||
@@ -177,13 +174,13 @@ func TestConcurrentStreamsUseMultipleConns(t *testing.T) {
|
||||
}
|
||||
|
||||
got := c.WorkerConnCount()
|
||||
if got < 2 {
|
||||
t.Fatalf("expected >=2 worker conns for %d concurrent streams, got %d", n, got)
|
||||
if got != n {
|
||||
t.Fatalf("expected %d worker conns for %d players, got %d", n, n, got)
|
||||
}
|
||||
if got > maxConn {
|
||||
t.Fatalf("worker conns %d exceed maxConn %d", got, maxConn)
|
||||
if got > maxTunnels {
|
||||
t.Fatalf("worker conns %d exceed maxTunnels %d", got, maxTunnels)
|
||||
}
|
||||
t.Logf("%d concurrent streams spread over %d worker conn(s)", n, got)
|
||||
t.Logf("%d players on %d worker conn(s)", n, got)
|
||||
}
|
||||
|
||||
// TestProxyProtocol checks that the client prepends a correct HAProxy v2 header
|
||||
@@ -278,7 +275,7 @@ func TestBadPSK(t *testing.T) {
|
||||
cfg := &client.Config{
|
||||
Server: hubAddr,
|
||||
PSK: "totally-wrong",
|
||||
MaxConn: 2,
|
||||
MaxTunnels: 2,
|
||||
PingIntervalMs: 20000,
|
||||
Mappings: []client.Mapping{{Pattern: "mc.local", Destination: dest.addr}},
|
||||
}
|
||||
|
||||
@@ -0,0 +1,174 @@
|
||||
package e2e
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/iceBear67/redapricot/client"
|
||||
)
|
||||
|
||||
// expectClosedSoon fails unless conn is closed (or reset) within d. A player
|
||||
// that entered pending stays open until pendingTimeoutMs, so a fast EOF is
|
||||
// how we tell the limiter dropped the socket before match-side bookkeeping.
|
||||
func expectClosedSoon(t *testing.T, conn net.Conn, d time.Duration) {
|
||||
t.Helper()
|
||||
_ = conn.SetReadDeadline(time.Now().Add(d))
|
||||
n, err := conn.Read(make([]byte, 16))
|
||||
if err == nil {
|
||||
t.Fatalf("expected the hub to close the player, read %d bytes", n)
|
||||
}
|
||||
}
|
||||
|
||||
// TestPlayerBurstDropsExtraHandshakes: more arrivals than playerBurst from the
|
||||
// same IP are closed after the handshake and never become pending.
|
||||
func TestPlayerBurstDropsExtraHandshakes(t *testing.T) {
|
||||
const psk = "e2e-rate-burst"
|
||||
port := freePort(t)
|
||||
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
|
||||
startHubCfg(t, port, psk, map[string]any{
|
||||
"playerRatePerSec": 1,
|
||||
"playerBurst": 2,
|
||||
"maxPlayersPerIp": 64,
|
||||
})
|
||||
dest := newMockDest(t, modeEcho)
|
||||
startClient(t, hubAddr, psk, 8, []client.Mapping{
|
||||
{Pattern: "mc.local", Destination: dest.addr},
|
||||
})
|
||||
|
||||
kept := make([]net.Conn, 0, 2)
|
||||
defer func() {
|
||||
for _, c := range kept {
|
||||
_ = c.Close()
|
||||
}
|
||||
}()
|
||||
for i := 0; i < 2; i++ {
|
||||
pc := dialPlayer(t, hubAddr, "mc.local")
|
||||
playerEcho(t, pc, []byte(fmt.Sprintf("ok-%d", i)))
|
||||
kept = append(kept, pc)
|
||||
}
|
||||
|
||||
// Tokens spent, first two still held: extras must die well inside pendingTimeout.
|
||||
for i := 0; i < 2; i++ {
|
||||
extra := dialPlayer(t, hubAddr, "mc.local")
|
||||
expectClosedSoon(t, extra, 1500*time.Millisecond)
|
||||
_ = extra.Close()
|
||||
}
|
||||
|
||||
// The admitted players are unaffected.
|
||||
playerEcho(t, kept[0], []byte("still-here"))
|
||||
}
|
||||
|
||||
// TestMaxPlayersPerIpCapsConcurrentSockets: a second player from the same IP
|
||||
// is refused while the first is still open, and admitted again after it closes.
|
||||
func TestMaxPlayersPerIpCapsConcurrentSockets(t *testing.T) {
|
||||
const psk = "e2e-rate-conc"
|
||||
port := freePort(t)
|
||||
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
|
||||
startHubCfg(t, port, psk, map[string]any{
|
||||
"playerRatePerSec": 0,
|
||||
"playerBurst": 16,
|
||||
"maxPlayersPerIp": 1,
|
||||
})
|
||||
dest := newMockDest(t, modeEcho)
|
||||
startClient(t, hubAddr, psk, 4, []client.Mapping{
|
||||
{Pattern: "mc.local", Destination: dest.addr},
|
||||
})
|
||||
|
||||
first := dialPlayer(t, hubAddr, "mc.local")
|
||||
defer first.Close()
|
||||
playerEcho(t, first, []byte("first"))
|
||||
|
||||
second := dialPlayer(t, hubAddr, "mc.local")
|
||||
expectClosedSoon(t, second, 1500*time.Millisecond)
|
||||
_ = second.Close()
|
||||
|
||||
playerEcho(t, first, []byte("still-first"))
|
||||
_ = first.Close()
|
||||
// The closeHandler runs on the hub event loop; give it a beat to release.
|
||||
time.Sleep(200 * time.Millisecond)
|
||||
|
||||
third := dialPlayer(t, hubAddr, "mc.local")
|
||||
defer third.Close()
|
||||
playerEcho(t, third, []byte("after-release"))
|
||||
}
|
||||
|
||||
// TestUnmatchedHostConsumesRateBudget: a hostname scan is not a free flood.
|
||||
// Two unmatched handshakes spend the burst, so a later matching player is
|
||||
// also dropped.
|
||||
func TestUnmatchedHostConsumesRateBudget(t *testing.T) {
|
||||
const psk = "e2e-rate-scan"
|
||||
port := freePort(t)
|
||||
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
|
||||
startHubCfg(t, port, psk, map[string]any{
|
||||
"playerRatePerSec": 1,
|
||||
"playerBurst": 2,
|
||||
"maxPlayersPerIp": 64,
|
||||
})
|
||||
dest := newMockDest(t, modeEcho)
|
||||
startClient(t, hubAddr, psk, 4, []client.Mapping{
|
||||
{Pattern: "mc.local", Destination: dest.addr},
|
||||
})
|
||||
|
||||
for i := 0; i < 2; i++ {
|
||||
miss := dialPlayer(t, hubAddr, "no.such.host")
|
||||
expectClosedSoon(t, miss, 1500*time.Millisecond)
|
||||
_ = miss.Close()
|
||||
}
|
||||
|
||||
matched := dialPlayer(t, hubAddr, "mc.local")
|
||||
expectClosedSoon(t, matched, 1500*time.Millisecond)
|
||||
_ = matched.Close()
|
||||
}
|
||||
|
||||
// TestIntent17IgnoresPlayerLimiter: the control session and worker conns are
|
||||
// Intent 17, so a limiter tight enough to refuse a second player must not
|
||||
// prevent the client from connecting or taking over the first player.
|
||||
func TestIntent17IgnoresPlayerLimiter(t *testing.T) {
|
||||
const psk = "e2e-rate-intent17"
|
||||
port := freePort(t)
|
||||
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
|
||||
startHubCfg(t, port, psk, map[string]any{
|
||||
"playerRatePerSec": 1,
|
||||
"playerBurst": 1,
|
||||
"maxPlayersPerIp": 1,
|
||||
})
|
||||
dest := newMockDest(t, modeEcho)
|
||||
startClient(t, hubAddr, psk, 4, []client.Mapping{
|
||||
{Pattern: "mc.local", Destination: dest.addr},
|
||||
})
|
||||
|
||||
pc := dialPlayer(t, hubAddr, "mc.local")
|
||||
defer pc.Close()
|
||||
playerEcho(t, pc, []byte("intent17-ok"))
|
||||
}
|
||||
|
||||
// TestPlayerLimiterOffSwitch: both knobs at 0 restore phase-1 behaviour —
|
||||
// many players from 127.0.0.1 all get through.
|
||||
func TestPlayerLimiterOffSwitch(t *testing.T) {
|
||||
const psk = "e2e-rate-off"
|
||||
const n = 6
|
||||
port := freePort(t)
|
||||
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
|
||||
startHubCfg(t, port, psk, map[string]any{
|
||||
"playerRatePerSec": 0,
|
||||
"maxPlayersPerIp": 0,
|
||||
})
|
||||
dest := newMockDest(t, modeEcho)
|
||||
startClient(t, hubAddr, psk, n, []client.Mapping{
|
||||
{Pattern: "mc.local", Destination: dest.addr},
|
||||
})
|
||||
|
||||
conns := make([]net.Conn, 0, n)
|
||||
defer func() {
|
||||
for _, c := range conns {
|
||||
_ = c.Close()
|
||||
}
|
||||
}()
|
||||
for i := 0; i < n; i++ {
|
||||
pc := dialPlayer(t, hubAddr, "mc.local")
|
||||
playerEcho(t, pc, []byte(fmt.Sprintf("off-%d", i)))
|
||||
conns = append(conns, pc)
|
||||
}
|
||||
}
|
||||
+6
-7
@@ -131,10 +131,9 @@ func TestResumePreservesByteStream(t *testing.T) {
|
||||
}
|
||||
|
||||
// TestResumeWithConcurrentStreams covers the failure the single-stream test
|
||||
// cannot reach: stream ids restart at 1 on every conn, so after a reattach two
|
||||
// players can hold the same id on different conns. A binding that updates the
|
||||
// conn and the id separately will credit or reset the wrong player's stream, and
|
||||
// that only shows up when a second stream is there to be corrupted.
|
||||
// cannot reach: two independent tunnels resume at once. A torn rebind that
|
||||
// writes a frame onto the wrong conn would credit or reset the other player's
|
||||
// tunnel, and that only shows up when a second player is there to be corrupted.
|
||||
func TestResumeWithConcurrentStreams(t *testing.T) {
|
||||
const psk = "e2e-resume-multi"
|
||||
hubPort := freePort(t)
|
||||
@@ -143,7 +142,7 @@ func TestResumeWithConcurrentStreams(t *testing.T) {
|
||||
dest := newMockDest(t, modeEcho)
|
||||
|
||||
relay := newBlackholeRelay(t, hubAddr)
|
||||
startClientWithPing(t, relay.addr, psk, 2, 400, []client.Mapping{
|
||||
startClientWithPing(t, relay.addr, psk, 4, 400, []client.Mapping{
|
||||
{Pattern: "mc.local", Destination: dest.addr},
|
||||
})
|
||||
|
||||
@@ -197,7 +196,7 @@ func TestResumeDisabledClosesImmediately(t *testing.T) {
|
||||
startClientCfg(t, &client.Config{
|
||||
Server: relay.addr,
|
||||
PSK: psk,
|
||||
MaxConn: 1,
|
||||
MaxTunnels: 1,
|
||||
PingIntervalMs: 400,
|
||||
StreamResume: &off,
|
||||
Mappings: []client.Mapping{{Pattern: "mc.local", Destination: dest.addr}},
|
||||
@@ -237,7 +236,7 @@ func TestResumeGraceExpiryClosesPlayer(t *testing.T) {
|
||||
startClientCfg(t, &client.Config{
|
||||
Server: relay.addr,
|
||||
PSK: psk,
|
||||
MaxConn: 1,
|
||||
MaxTunnels: 1,
|
||||
PingIntervalMs: 400,
|
||||
ResumeGraceMs: 2000,
|
||||
Mappings: []client.Mapping{{Pattern: "mc.local", Destination: dest.addr}},
|
||||
|
||||
@@ -27,17 +27,15 @@ func echoRounds(t *testing.T, hubAddr string, rounds, size int) {
|
||||
}
|
||||
|
||||
// TestSlowPlayerDoesNotStallOthers: a player that stops reading while megabytes
|
||||
// are echoed back to it must not stall another stream on the same worker
|
||||
// connection (maxConn=1 forces sharing). Before per-stream flow control, the
|
||||
// hub paused the whole worker socket once that player's write queue filled,
|
||||
// freezing every other stream's downstream data.
|
||||
// are echoed back to it must not stall another player. Flow control and the
|
||||
// 1:1 worker model keep a jammed tunnel from taking anyone else with it.
|
||||
func TestSlowPlayerDoesNotStallOthers(t *testing.T) {
|
||||
const psk = "e2e-slowplayer"
|
||||
port := freePort(t)
|
||||
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
|
||||
startHub(t, port, psk)
|
||||
dest := newMockDest(t, modeEcho)
|
||||
startClient(t, hubAddr, psk, 1, []client.Mapping{
|
||||
startClient(t, hubAddr, psk, 2, []client.Mapping{
|
||||
{Pattern: "mc.local", Destination: dest.addr},
|
||||
})
|
||||
|
||||
@@ -52,7 +50,7 @@ func TestSlowPlayerDoesNotStallOthers(t *testing.T) {
|
||||
// Let the slow stream jam: its flow-control window fills and stays full.
|
||||
time.Sleep(1 * time.Second)
|
||||
|
||||
// The fast player shares the single worker conn and must still round-trip.
|
||||
// The fast player has its own worker conn and must still round-trip.
|
||||
echoRounds(t, hubAddr, 10, 8*1024)
|
||||
}
|
||||
|
||||
@@ -67,7 +65,7 @@ func TestSlowDestinationDoesNotStallOthers(t *testing.T) {
|
||||
startHub(t, port, psk)
|
||||
dest := newMockDest(t, modeEcho)
|
||||
hole := newMockDest(t, modeBlackhole)
|
||||
startClient(t, hubAddr, psk, 1, []client.Mapping{
|
||||
startClient(t, hubAddr, psk, 2, []client.Mapping{
|
||||
{Pattern: "mc.local", Destination: dest.addr},
|
||||
{Pattern: "hole.local", Destination: hole.addr},
|
||||
})
|
||||
|
||||
@@ -9,5 +9,8 @@
|
||||
"resumeGraceMs": 20000,
|
||||
"maxParkedStreams": 256,
|
||||
"statsIntervalMs": 0,
|
||||
"registrationGraceMs": 15000
|
||||
"registrationGraceMs": 15000,
|
||||
"playerRatePerSec": 8,
|
||||
"playerBurst": 16,
|
||||
"maxPlayersPerIp": 64
|
||||
}
|
||||
|
||||
@@ -19,7 +19,10 @@ public record Config(
|
||||
int maxParkedStreams,
|
||||
long maxParkedBytes,
|
||||
long statsIntervalMs,
|
||||
long registrationGraceMs
|
||||
long registrationGraceMs,
|
||||
double playerRatePerSec,
|
||||
int playerBurst,
|
||||
int maxPlayersPerIp
|
||||
) {
|
||||
public static Config load(Path file) throws Exception {
|
||||
JsonObject json = new JsonObject(Files.readString(file));
|
||||
@@ -68,6 +71,14 @@ public record Config(
|
||||
// (its backoff caps at 10s) without holding a player so long
|
||||
// that they give up anyway; 0 disables and restores the old
|
||||
// behaviour of dropping routes the moment a session closes.
|
||||
json.getLong("registrationGraceMs", 15_000L));
|
||||
json.getLong("registrationGraceMs", 15_000L),
|
||||
// Player-only (Intent ∉ {17, 18}). 0 turns that mechanism off.
|
||||
// Unmatched hostnames still consume a token — otherwise a
|
||||
// hostname scan is a free flood. Intent 17 is never admitted
|
||||
// through the limiter: every worker comes from the client's
|
||||
// one IP.
|
||||
Math.max(0, json.getDouble("playerRatePerSec", 8.0)),
|
||||
Math.max(1, json.getInteger("playerBurst", 16)),
|
||||
Math.max(0, json.getInteger("maxPlayersPerIp", 64)));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -33,6 +33,7 @@ public final class Hub {
|
||||
public final Config config;
|
||||
public final byte[] pskBytes;
|
||||
public final String pskAddress;
|
||||
public final IpRateLimiter limiter;
|
||||
|
||||
private final Map<String, Registration> patterns = new ConcurrentHashMap<>();
|
||||
private final Map<String, PendingPlayer> pending = new ConcurrentHashMap<>();
|
||||
@@ -77,6 +78,25 @@ public final class Hub {
|
||||
this.config = config;
|
||||
this.pskBytes = config.psk().getBytes(StandardCharsets.UTF_8);
|
||||
this.pskAddress = Crypto.pskAddress(config.psk());
|
||||
this.limiter = new IpRateLimiter(
|
||||
config.playerRatePerSec(), config.playerBurst(), config.maxPlayersPerIp());
|
||||
// Unit tests construct a Hub with a null Vertx (no event loop).
|
||||
if (limiter.enabled() && vertx != null) {
|
||||
vertx.setPeriodic(IpRateLimiter.SWEEP_MS, id -> limiter.sweep(System.currentTimeMillis()));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Admit one player socket from {@code ip}. {@code null} means allowed;
|
||||
* the caller must {@link #releasePlayer} on every close path (pending
|
||||
* timeout, unmatched host, player FIN, park eviction).
|
||||
*/
|
||||
public IpRateLimiter.Deny admitPlayer(String ip) {
|
||||
return limiter.admit(ip, System.currentTimeMillis());
|
||||
}
|
||||
|
||||
public void releasePlayer(String ip) {
|
||||
limiter.release(ip, System.currentTimeMillis());
|
||||
}
|
||||
|
||||
// ---- pattern registry ----
|
||||
@@ -297,6 +317,10 @@ public final class Hub {
|
||||
if (streams.remove(st.cidHex) == null) return;
|
||||
if (st.parked) unpark(st);
|
||||
st.unacked.clear();
|
||||
// The player's closeHandler was replaced at SYN/RESUME bind, so the
|
||||
// HubConnection cleanup never runs. This is the live/parked release
|
||||
// path; pending/unmatched still go through HubConnection.closeCleanup.
|
||||
releasePlayer(st.playerIp);
|
||||
}
|
||||
|
||||
/** Look up a stream by the CID a client presented, live or parked. */
|
||||
|
||||
@@ -280,6 +280,19 @@ public final class HubConnection {
|
||||
// ---- player connection ----
|
||||
|
||||
private void handlePlayer(String address) {
|
||||
// Before match / CID / pause: unmatched hostnames still consume a token,
|
||||
// otherwise a hostname scan is a free flood. Intent 17 never reaches
|
||||
// this method (PROTOCOL.md §9.1).
|
||||
String ip = socket.remoteAddress() != null ? socket.remoteAddress().host() : "0.0.0.0";
|
||||
if (hub.admitPlayer(ip) != null) {
|
||||
socket.close();
|
||||
return;
|
||||
}
|
||||
// Release on every close of this socket: pending timeout, unmatched
|
||||
// host, player FIN, park eviction. Later cleanups wrap this, they
|
||||
// must not replace it.
|
||||
closeCleanup = () -> hub.releasePlayer(ip);
|
||||
|
||||
String host = Hub.normalizeAddress(address);
|
||||
Hub.Match matched = hub.match(address);
|
||||
if (matched == null) {
|
||||
@@ -293,14 +306,16 @@ public final class HubConnection {
|
||||
String pattern = matched.pattern();
|
||||
byte[] cid = hub.newCid();
|
||||
String cidHex = Hex.encode(cid);
|
||||
String ip = socket.remoteAddress() != null ? socket.remoteAddress().host() : "0.0.0.0";
|
||||
int port = socket.remoteAddress() != null ? socket.remoteAddress().port() : 0;
|
||||
|
||||
socket.pause();
|
||||
Buffer buffered = hs.copy(); // handshake + any pipelined bytes, forwarded verbatim
|
||||
|
||||
PendingPlayer p = new PendingPlayer(cid, cidHex, socket, buffered, pattern, ip, port);
|
||||
closeCleanup = () -> hub.removePending(cidHex);
|
||||
closeCleanup = () -> {
|
||||
hub.removePending(cidHex);
|
||||
hub.releasePlayer(ip);
|
||||
};
|
||||
|
||||
if (session == null) {
|
||||
// The route is orphaned: its client's control session has closed and
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
package io.icybear.redapricot;
|
||||
|
||||
import org.apache.logging.log4j.LogManager;
|
||||
import org.apache.logging.log4j.Logger;
|
||||
|
||||
import java.util.Iterator;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* Per-IP admission for <em>player</em> connections only (PROTOCOL.md §9.1).
|
||||
*
|
||||
* <p>Intent 17 (control session + every worker conn) is never admitted through
|
||||
* here. Those sockets all come from the client's one address; limiting them
|
||||
* would be the hub throttling its own client. e2e shares {@code 127.0.0.1}
|
||||
* between players and the client for the same reason.
|
||||
*
|
||||
* <p>Event-loop confined: no locking. A {@code 0} rate or concurrent cap turns
|
||||
* that mechanism off; both {@code 0} makes {@link #admit} a no-op.
|
||||
*/
|
||||
public final class IpRateLimiter {
|
||||
private static final Logger LOG = LogManager.getLogger("redapricot.limit");
|
||||
|
||||
/** Idle buckets older than this are dropped so a one-shot flood cannot leak. */
|
||||
static final long SWEEP_MS = 60_000;
|
||||
private static final long DENY_LOG_INTERVAL_MS = 2_000;
|
||||
|
||||
public enum Deny { RATE, CONCURRENT }
|
||||
|
||||
private final double ratePerSec; // 0 = token bucket off
|
||||
private final double burst;
|
||||
private final int maxConcurrent; // 0 = concurrent cap off
|
||||
private final Map<String, Bucket> buckets = new LinkedHashMap<>();
|
||||
|
||||
static final class Bucket {
|
||||
double tokens;
|
||||
long lastRefillMs;
|
||||
int concurrent;
|
||||
long lastActivityMs;
|
||||
long lastDenyLogMs;
|
||||
int deniesSinceLog;
|
||||
}
|
||||
|
||||
public IpRateLimiter(double ratePerSec, double burst, int maxConcurrent) {
|
||||
this.ratePerSec = Math.max(0, ratePerSec);
|
||||
this.burst = Math.max(1, burst);
|
||||
this.maxConcurrent = Math.max(0, maxConcurrent);
|
||||
}
|
||||
|
||||
public boolean enabled() {
|
||||
return ratePerSec > 0 || maxConcurrent > 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* Consume one player admission for {@code ip}. {@code null} means allowed
|
||||
* and the caller <em>must</em> {@link #release} when the socket closes.
|
||||
*/
|
||||
public Deny admit(String ip, long nowMs) {
|
||||
if (!enabled()) return null;
|
||||
Bucket b = bucket(ip, nowMs);
|
||||
b.lastActivityMs = nowMs;
|
||||
refill(b, nowMs);
|
||||
if (ratePerSec > 0 && b.tokens < 1.0) {
|
||||
noteDeny(ip, b, nowMs, Deny.RATE);
|
||||
return Deny.RATE;
|
||||
}
|
||||
if (maxConcurrent > 0 && b.concurrent >= maxConcurrent) {
|
||||
noteDeny(ip, b, nowMs, Deny.CONCURRENT);
|
||||
return Deny.CONCURRENT;
|
||||
}
|
||||
if (ratePerSec > 0) b.tokens -= 1.0;
|
||||
b.concurrent++;
|
||||
return null;
|
||||
}
|
||||
|
||||
public void release(String ip, long nowMs) {
|
||||
Bucket b = buckets.get(ip);
|
||||
if (b == null) return;
|
||||
if (b.concurrent > 0) b.concurrent--;
|
||||
b.lastActivityMs = nowMs;
|
||||
}
|
||||
|
||||
/** Drop idle empty buckets. Safe to call on a timer. */
|
||||
public void sweep(long nowMs) {
|
||||
Iterator<Map.Entry<String, Bucket>> it = buckets.entrySet().iterator();
|
||||
while (it.hasNext()) {
|
||||
Bucket b = it.next().getValue();
|
||||
if (b.concurrent == 0 && nowMs - b.lastActivityMs >= SWEEP_MS) {
|
||||
it.remove();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Visible for tests. */
|
||||
int bucketCount() {
|
||||
return buckets.size();
|
||||
}
|
||||
|
||||
/** Visible for tests. */
|
||||
int concurrent(String ip) {
|
||||
Bucket b = buckets.get(ip);
|
||||
return b == null ? 0 : b.concurrent;
|
||||
}
|
||||
|
||||
private Bucket bucket(String ip, long nowMs) {
|
||||
Bucket b = buckets.get(ip);
|
||||
if (b != null) return b;
|
||||
b = new Bucket();
|
||||
b.tokens = burst;
|
||||
b.lastRefillMs = nowMs;
|
||||
b.lastActivityMs = nowMs;
|
||||
buckets.put(ip, b);
|
||||
return b;
|
||||
}
|
||||
|
||||
private void refill(Bucket b, long nowMs) {
|
||||
if (ratePerSec <= 0) return;
|
||||
double elapsed = (nowMs - b.lastRefillMs) / 1000.0;
|
||||
if (elapsed <= 0) return;
|
||||
b.tokens = Math.min(burst, b.tokens + elapsed * ratePerSec);
|
||||
b.lastRefillMs = nowMs;
|
||||
}
|
||||
|
||||
private void noteDeny(String ip, Bucket b, long nowMs, Deny why) {
|
||||
b.deniesSinceLog++;
|
||||
if (b.lastDenyLogMs != 0 && nowMs - b.lastDenyLogMs < DENY_LOG_INTERVAL_MS) {
|
||||
return;
|
||||
}
|
||||
LOG.warn("dropping player from {}: {} ({} similar since last log)",
|
||||
ip, why == Deny.RATE ? "rate" : "maxPlayersPerIp", b.deniesSinceLog);
|
||||
b.lastDenyLogMs = nowMs;
|
||||
b.deniesSinceLog = 0;
|
||||
}
|
||||
}
|
||||
@@ -4,10 +4,10 @@ import io.vertx.core.buffer.Buffer;
|
||||
import io.vertx.core.net.NetSocket;
|
||||
|
||||
/**
|
||||
* One tunneled player: the player socket plus the flow-control state of the mux
|
||||
* stream carrying it (PROTOCOL.md §7.3).
|
||||
* One tunneled player: the player socket plus the flow-control state of the
|
||||
* worker conn carrying it (PROTOCOL.md §7.3).
|
||||
*
|
||||
* <p>This is deliberately <em>not</em> owned by {@link WorkerConn}. A stream's
|
||||
* <p>This is deliberately <em>not</em> owned by {@link WorkerConn}. A tunnel's
|
||||
* identity is the player, not the connection it happens to ride: the worker conn
|
||||
* is a replaceable transport, and state that dies with it cannot be recovered
|
||||
* when it drops.
|
||||
@@ -25,9 +25,8 @@ public final class PlayerStream {
|
||||
final String playerIp;
|
||||
final int playerPort;
|
||||
|
||||
/** The conn currently carrying this stream, and its id there. */
|
||||
/** The conn currently carrying this player. */
|
||||
WorkerConn worker;
|
||||
int sid;
|
||||
|
||||
/** Budget for player -> client DATA. */
|
||||
int sendWnd;
|
||||
@@ -39,11 +38,11 @@ public final class PlayerStream {
|
||||
// Pause reasons. Vert.x pause() is a flag rather than a counter, so a socket
|
||||
// can be paused for several reasons at once and must only be resumed once
|
||||
// none of them hold — see WorkerConn#maybeResumePlayer.
|
||||
boolean pausedForWindow; // this stream's send window is exhausted
|
||||
boolean pausedForAggregate; // the shared worker socket's write queue is full
|
||||
boolean pausedForWindow; // this tunnel's send window is exhausted
|
||||
boolean pausedForAggregate; // the worker socket's write queue is full
|
||||
boolean parked; // the worker conn died; hanging until a reattach
|
||||
|
||||
/** Whether the conn carrying this stream negotiated resumption (§7.5). */
|
||||
/** Whether the conn carrying this player negotiated resumption (§7.5). */
|
||||
boolean resumable;
|
||||
|
||||
// Resumption bookkeeping (§7.5). Three distinct offsets, and conflating them
|
||||
@@ -64,7 +63,7 @@ public final class PlayerStream {
|
||||
long graceDeadline;
|
||||
long timerId = -1;
|
||||
|
||||
PlayerStream(PendingPlayer p, WorkerConn worker, int sid, int sendWnd) {
|
||||
PlayerStream(PendingPlayer p, WorkerConn worker, int sendWnd) {
|
||||
this.cid = p.getCid();
|
||||
this.cidHex = p.getCidHex();
|
||||
this.player = p.getSocket();
|
||||
@@ -72,7 +71,6 @@ public final class PlayerStream {
|
||||
this.playerIp = p.getPlayerIp();
|
||||
this.playerPort = p.getPlayerPort();
|
||||
this.worker = worker;
|
||||
this.sid = sid;
|
||||
this.sendWnd = sendWnd;
|
||||
}
|
||||
|
||||
@@ -81,7 +79,7 @@ public final class PlayerStream {
|
||||
return !pausedForWindow && !pausedForAggregate && !parked;
|
||||
}
|
||||
|
||||
/** Roughly how much this stream holds while parked, for the hub-wide cap. */
|
||||
/** Roughly how much this tunnel holds while parked, for the hub-wide cap. */
|
||||
int parkedBytes() {
|
||||
return unacked.length() + (pendingUp != null ? pendingUp.length() : 0);
|
||||
}
|
||||
|
||||
@@ -26,22 +26,19 @@ public final class Protocol {
|
||||
public static final int REGISTER_OK = 0x00;
|
||||
public static final int REGISTER_ERR_PATTERN = 0x01; // pattern is not a valid regular expression
|
||||
|
||||
// Worker-conn mux frame types
|
||||
// Worker-conn frame types (one player per conn; no stream id).
|
||||
public static final int MUX_SYN = 0x00;
|
||||
public static final int MUX_DATA = 0x01;
|
||||
public static final int MUX_FIN = 0x02;
|
||||
public static final int MUX_RST = 0x03;
|
||||
public static final int MUX_WND = 0x04; // per-stream flow-control credit grant
|
||||
public static final int MUX_PING = 0x05; // liveness probe, StreamID 0
|
||||
public static final int MUX_WND = 0x04; // per-connection flow-control credit grant
|
||||
public static final int MUX_PING = 0x05; // liveness probe
|
||||
public static final int MUX_PONG = 0x06; // liveness reply, echoes the nonce
|
||||
/** Reattach a parked stream to this conn: CID + the client's accepted offset (§7.5). */
|
||||
/** Reattach a parked player to this conn: CID + the client's accepted offset (§7.5). */
|
||||
public static final int MUX_RESUME = 0x07;
|
||||
/** Hub's answer to RESUME: its accepted offset plus a freshly minted CID. */
|
||||
public static final int MUX_RESUME_ACK = 0x08;
|
||||
|
||||
/** Reserved stream id for connection-scoped mux frames (PING/PONG). Streams start at 1. */
|
||||
public static final int MUX_CTL_SID = 0;
|
||||
|
||||
// RST reason codes (optional trailing byte; absence means "unspecified").
|
||||
// Distinguishing them matters for resume: "unknown stream" is terminal,
|
||||
// "already bound" means a racing attempt won and this one must not tear down.
|
||||
@@ -55,18 +52,18 @@ public final class Protocol {
|
||||
// Session-establishment feature flags (trailing VarInt on the Rekey message,
|
||||
// echoed after the SessionReady type byte when accepted).
|
||||
public static final int FLAG_STREAM_FC = 0x01;
|
||||
/** Mux-level PING/PONG on worker conns, so a silently dropped path is detected. */
|
||||
/** Connection-level PING/PONG on worker conns, so a silently dropped path is detected. */
|
||||
public static final int FLAG_WORKER_HEARTBEAT = 0x02;
|
||||
/**
|
||||
* Stream resumption (§7.5): on a worker-conn drop the hub hangs the player
|
||||
* socket instead of closing it, and the client reattaches the stream
|
||||
* socket instead of closing it, and the client reattaches that player
|
||||
* byte-exactly over a fresh conn. When accepted, the hub appends its resume
|
||||
* grace period to SessionReady so the client can bound its own retry budget
|
||||
* against it.
|
||||
*/
|
||||
public static final int FLAG_STREAM_RESUME = 0x04;
|
||||
|
||||
// Per-stream flow-control window bounds (bytes).
|
||||
// Per-connection flow-control window bounds (bytes).
|
||||
public static final int DEFAULT_STREAM_WINDOW = 256 * 1024;
|
||||
public static final int MIN_STREAM_WINDOW = 32 * 1024;
|
||||
public static final int MAX_STREAM_WINDOW = 8 << 20;
|
||||
|
||||
@@ -8,29 +8,26 @@ import lombok.RequiredArgsConstructor;
|
||||
import org.apache.logging.log4j.LogManager;
|
||||
import org.apache.logging.log4j.Logger;
|
||||
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* An authenticated worker connection (Magic 0x02). Multiplexes many player
|
||||
* streams; the client opens streams via SYN(CID) to take over pending players.
|
||||
* An authenticated worker connection (Magic 0x02). Carries exactly one player:
|
||||
* the TCP connection is the tunnel (PROTOCOL.md §7). The client binds it with
|
||||
* SYN(CID) (or RESUME) after SessionReady.
|
||||
*
|
||||
* <p>Every stream has a credit window in both directions (PROTOCOL.md §7.3),
|
||||
* so one slow player only ever stalls its own stream — the shared worker
|
||||
* socket is never paused because of a single stream.
|
||||
* <p>The connection has a credit window in both directions (PROTOCOL.md §7.3),
|
||||
* so a slow player only ever stalls itself.
|
||||
*/
|
||||
@RequiredArgsConstructor
|
||||
public final class WorkerConn {
|
||||
private static final Logger LOG = LogManager.getLogger("redapricot.worker");
|
||||
|
||||
/** Cap on a single DATA frame so no stream monopolizes the shared link for long. */
|
||||
/** Cap on a single DATA frame so one write cannot occupy the link for long. */
|
||||
private static final int CHUNK = 32 * 1024;
|
||||
|
||||
private final Hub hub;
|
||||
private final EncryptedFrames frames;
|
||||
private final String id;
|
||||
private final int sendWndInit; // client's advertised per-stream receive window (our send budget)
|
||||
private final int recvWndInit; // our advertised per-stream receive window (basis for credit grants)
|
||||
private final int sendWndInit; // client's advertised receive window (our send budget)
|
||||
private final int recvWndInit; // our advertised receive window (basis for credit grants)
|
||||
/**
|
||||
* Whether this conn negotiated stream resumption. Gates the park path: a
|
||||
* client that will never reattach is better served by an immediate close than
|
||||
@@ -38,46 +35,52 @@ public final class WorkerConn {
|
||||
*/
|
||||
private final boolean resume;
|
||||
|
||||
private final Map<Integer, PlayerStream> streams = new HashMap<>();
|
||||
/** The one player bound to this conn, or null until SYN/RESUME. */
|
||||
private PlayerStream stream;
|
||||
|
||||
private boolean workerDrainArmed = false; // whether the worker socket's single drainHandler is set
|
||||
private boolean workerDrainArmed = false; // whether the worker socket's drainHandler is set
|
||||
|
||||
public void onFrame(byte[] payload) {
|
||||
ProtoReader r = new ProtoReader(payload);
|
||||
int type = r.readUByte();
|
||||
int sid = r.readVarInt();
|
||||
switch (type) {
|
||||
case Protocol.MUX_SYN -> handleSyn(sid, r.readBytes(Protocol.CID_LEN));
|
||||
case Protocol.MUX_DATA -> handleData(sid, r.readBuffer(r.remaining()));
|
||||
case Protocol.MUX_WND -> handleWnd(sid, r.readVarInt());
|
||||
case Protocol.MUX_SYN -> handleSyn(r.readBytes(Protocol.CID_LEN));
|
||||
case Protocol.MUX_DATA -> handleData(r.readBuffer(r.remaining()));
|
||||
case Protocol.MUX_WND -> handleWnd(r.readVarInt());
|
||||
case Protocol.MUX_RESUME ->
|
||||
handleResume(sid, r.readBytes(Protocol.CID_LEN), r.readI64(), r.readI64());
|
||||
case Protocol.MUX_FIN, Protocol.MUX_RST -> closeStream(sid);
|
||||
handleResume(r.readBytes(Protocol.CID_LEN), r.readI64(), r.readI64());
|
||||
case Protocol.MUX_FIN, Protocol.MUX_RST -> closeBound();
|
||||
case Protocol.MUX_PING -> sendPong(r.readI64());
|
||||
case Protocol.MUX_PONG -> { /* liveness only; arrival is what matters */ }
|
||||
case Protocol.FRAME_ERROR -> LOG.warn("worker {} error frame", id);
|
||||
default -> LOG.warn("worker {} unknown mux type {}", id, type);
|
||||
default -> LOG.warn("worker {} unknown frame type {}", id, type);
|
||||
}
|
||||
}
|
||||
|
||||
private void handleSyn(int sid, byte[] cid) {
|
||||
private void handleSyn(byte[] cid) {
|
||||
if (stream != null) {
|
||||
LOG.warn("worker {} SYN on an already-bound conn; closing", id);
|
||||
sendRst(Protocol.RST_ALREADY_BOUND);
|
||||
frames.close();
|
||||
return;
|
||||
}
|
||||
PendingPlayer p = hub.takePending(cid);
|
||||
if (p == null) {
|
||||
LOG.warn("worker {} SYN for unknown CID", id);
|
||||
sendRst(sid, Protocol.RST_UNKNOWN_STREAM);
|
||||
sendRst(Protocol.RST_UNKNOWN_STREAM);
|
||||
return;
|
||||
}
|
||||
PlayerStream st = new PlayerStream(p, this, sid, sendWndInit);
|
||||
PlayerStream st = new PlayerStream(p, this, sendWndInit);
|
||||
st.resumable = resume;
|
||||
streams.put(sid, st);
|
||||
stream = st;
|
||||
hub.addStream(st);
|
||||
|
||||
// From now on the player socket belongs to this stream. The handlers are
|
||||
// From now on the player socket belongs to this tunnel. The handlers are
|
||||
// installed once and route through the hub, which dispatches to whichever
|
||||
// conn currently carries the stream.
|
||||
// conn currently carries the player.
|
||||
//
|
||||
// They must not call this conn's methods directly: a lambda defined here
|
||||
// captures `this`, so after the stream moves to another conn it would keep
|
||||
// captures `this`, so after the player moves to another conn it would keep
|
||||
// writing into the dead one's transport, where sends are silently dropped
|
||||
// and the player goes mute with nothing logged. Re-installing handlers on
|
||||
// every reattach would be the other option, but a Vert.x socket resumed
|
||||
@@ -91,17 +94,17 @@ public final class WorkerConn {
|
||||
sendUpstream(st, p.getBuffered());
|
||||
maybeResumePlayer(st);
|
||||
checkAggregate(st);
|
||||
LOG.info("worker {} stream {} bound to {}", id, sid, st.pattern);
|
||||
LOG.info("worker {} bound to {}", id, st.pattern);
|
||||
}
|
||||
|
||||
/** Player bytes arrived on a stream this conn currently carries. */
|
||||
/** Player bytes arrived on the player this conn currently carries. */
|
||||
void playerData(PlayerStream st, Buffer buf) {
|
||||
sendUpstream(st, buf);
|
||||
checkAggregate(st);
|
||||
}
|
||||
|
||||
/**
|
||||
* Send player bytes to the client, chunked and clipped to the stream window;
|
||||
* Send player bytes to the client, chunked and clipped to the send window;
|
||||
* the overflow is parked in {@code pendingUp} and the player socket paused
|
||||
* until the client grants more credit.
|
||||
*/
|
||||
@@ -120,7 +123,7 @@ public final class WorkerConn {
|
||||
}
|
||||
}
|
||||
|
||||
/** Send from {@code buf[off..]} within the stream window, chunked; returns the new offset. */
|
||||
/** Send from {@code buf[off..]} within the send window, chunked; returns the new offset. */
|
||||
private int drainUpstream(PlayerStream st, Buffer buf, int off) {
|
||||
while (off < buf.length() && st.sendWnd > 0) {
|
||||
int n = Math.min(Math.min(CHUNK, st.sendWnd), buf.length() - off);
|
||||
@@ -134,16 +137,16 @@ public final class WorkerConn {
|
||||
st.unacked.append(chunk);
|
||||
}
|
||||
st.sentOffset += n;
|
||||
sendData(st.sid, chunk);
|
||||
sendData(chunk);
|
||||
st.sendWnd -= n;
|
||||
off += n;
|
||||
}
|
||||
return off;
|
||||
}
|
||||
|
||||
/** The client granted {@code delta} more bytes of credit on a stream. */
|
||||
private void handleWnd(int sid, int delta) {
|
||||
PlayerStream st = streams.get(sid);
|
||||
/** The client granted {@code delta} more bytes of credit. */
|
||||
private void handleWnd(int delta) {
|
||||
PlayerStream st = stream;
|
||||
if (st == null || delta <= 0) return;
|
||||
// The running sum doubles as the acked offset: the client grants credit
|
||||
// exactly as bytes reach the destination socket, so a credited byte can
|
||||
@@ -160,13 +163,13 @@ public final class WorkerConn {
|
||||
checkAggregate(st);
|
||||
}
|
||||
|
||||
private void handleData(int sid, Buffer data) {
|
||||
PlayerStream st = streams.get(sid);
|
||||
private void handleData(Buffer data) {
|
||||
PlayerStream st = stream;
|
||||
if (st == null) return;
|
||||
// Never pause the shared socket: the client bounds what it sends per
|
||||
// stream to our advertised window, so a slow player only piles up a
|
||||
// bounded amount in its own write queue; credit is granted back as the
|
||||
// write completes (i.e. the bytes reached the player socket).
|
||||
// Never pause the worker socket: the client bounds what it sends to our
|
||||
// advertised window, so a slow player only piles up a bounded amount in
|
||||
// its own write queue; credit is granted back as the write completes
|
||||
// (i.e. the bytes reached the player socket).
|
||||
int len = data.length();
|
||||
// Accepted the moment the bytes are taken off the wire, not when the write
|
||||
// completes. Completion is asynchronous and suppressed once the connection
|
||||
@@ -176,7 +179,7 @@ public final class WorkerConn {
|
||||
st.acceptedOffset += len;
|
||||
st.player.write(data).onComplete(ar -> {
|
||||
if (ar.failed()) return;
|
||||
// Both counters advance even if this conn has since died or the stream
|
||||
// Both counters advance even if this conn has since died or the player
|
||||
// has moved on. Discarding them would destroy up to half a window of
|
||||
// credit per outage, and — worse — leave the delivered offset that a
|
||||
// reattach restates the window from permanently short.
|
||||
@@ -186,28 +189,34 @@ public final class WorkerConn {
|
||||
if (st.credited * 2 >= recvWndInit) {
|
||||
int delta = st.credited;
|
||||
st.credited = 0;
|
||||
sendWnd(st.sid, delta);
|
||||
sendWnd(delta);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Reattach a parked stream to this connection (§7.5).
|
||||
* Reattach a parked player to this connection (§7.5).
|
||||
*
|
||||
* <p>Runs to completion in one event-loop turn — rebind, acknowledge, replay —
|
||||
* so the hub's single-threaded model makes the ordering race-free by
|
||||
* construction, with no interleaving of live and replayed bytes to reason
|
||||
* about.
|
||||
*/
|
||||
private void handleResume(int sid, byte[] cid, long clientAccepted, long clientDelivered) {
|
||||
private void handleResume(byte[] cid, long clientAccepted, long clientDelivered) {
|
||||
if (stream != null) {
|
||||
LOG.warn("worker {} RESUME on an already-bound conn; closing", id);
|
||||
sendRst(Protocol.RST_ALREADY_BOUND);
|
||||
frames.close();
|
||||
return;
|
||||
}
|
||||
PlayerStream st = hub.takeParked(cid);
|
||||
if (st == null) {
|
||||
// Tell a stream we have never heard of apart from one that is still
|
||||
// Tell a player we have never heard of apart from one that is still
|
||||
// bound: the first is terminal for the client, the second means a
|
||||
// racing attempt won and this one must leave the player alone.
|
||||
boolean bound = hub.streamByCid(cid) != null;
|
||||
LOG.warn("worker {} RESUME for {} CID", id, bound ? "still-bound" : "unknown");
|
||||
sendRst(sid, bound ? Protocol.RST_ALREADY_BOUND : Protocol.RST_UNKNOWN_STREAM);
|
||||
sendRst(bound ? Protocol.RST_ALREADY_BOUND : Protocol.RST_UNKNOWN_STREAM);
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -222,16 +231,15 @@ public final class WorkerConn {
|
||||
if (replay == null) {
|
||||
LOG.warn("worker {} RESUME at offset {} outside the retained region [{}, {}]; closing player",
|
||||
id, clientAccepted, st.unacked.base(), st.unacked.end());
|
||||
sendRst(sid, Protocol.RST_UNKNOWN_STREAM);
|
||||
sendRst(Protocol.RST_UNKNOWN_STREAM);
|
||||
hub.removeStream(st);
|
||||
st.player.close();
|
||||
return;
|
||||
}
|
||||
|
||||
st.worker = this;
|
||||
st.sid = sid;
|
||||
st.resumable = resume;
|
||||
streams.put(sid, st);
|
||||
stream = st;
|
||||
|
||||
// Restate the window rather than patching it. Three offsets, three jobs:
|
||||
// the replay above is measured from what the client *accepted*, the window
|
||||
@@ -253,7 +261,7 @@ public final class WorkerConn {
|
||||
byte[] newCid = hub.newCid();
|
||||
hub.rekeyStream(st, newCid);
|
||||
frames.send(new ProtoWriter()
|
||||
.u8(Protocol.MUX_RESUME_ACK).varInt(sid)
|
||||
.u8(Protocol.MUX_RESUME_ACK)
|
||||
.i64(st.acceptedOffset)
|
||||
.i64(st.deliveredOffset)
|
||||
.bytes(newCid)
|
||||
@@ -264,7 +272,7 @@ public final class WorkerConn {
|
||||
// drainUpstream would do.
|
||||
for (int off = 0; off < replay.length(); off += CHUNK) {
|
||||
int end = Math.min(off + CHUNK, replay.length());
|
||||
sendData(sid, replay.getBytes(off, end));
|
||||
sendData(replay.getBytes(off, end));
|
||||
}
|
||||
|
||||
if (st.pendingUp != null) {
|
||||
@@ -275,12 +283,13 @@ public final class WorkerConn {
|
||||
if (st.pendingUp == null) st.pausedForWindow = false;
|
||||
maybeResumePlayer(st);
|
||||
checkAggregate(st);
|
||||
LOG.info("worker {} stream {} resumed ({} bytes replayed, {} outstanding)",
|
||||
id, sid, replay.length(), outstanding);
|
||||
LOG.info("worker {} resumed ({} bytes replayed, {} outstanding)",
|
||||
id, replay.length(), outstanding);
|
||||
}
|
||||
|
||||
private void closeStream(int sid) {
|
||||
PlayerStream st = streams.remove(sid);
|
||||
private void closeBound() {
|
||||
PlayerStream st = stream;
|
||||
stream = null;
|
||||
if (st != null) {
|
||||
st.worker = null;
|
||||
hub.removeStream(st);
|
||||
@@ -288,7 +297,7 @@ public final class WorkerConn {
|
||||
}
|
||||
}
|
||||
|
||||
/** Park the player if the shared worker socket's write queue is congested. */
|
||||
/** Park the player if the worker socket's write queue is congested. */
|
||||
private void checkAggregate(PlayerStream st) {
|
||||
if (!st.pausedForAggregate && frames.writeQueueFull()) {
|
||||
st.pausedForAggregate = true;
|
||||
@@ -297,17 +306,16 @@ public final class WorkerConn {
|
||||
}
|
||||
}
|
||||
|
||||
/** Register (once) the shared worker socket's single drain handler; on drain, wake parked players. */
|
||||
/** Register (once) the worker socket's drain handler; on drain, wake the player. */
|
||||
private void armWorkerDrain() {
|
||||
if (workerDrainArmed) return;
|
||||
workerDrainArmed = true;
|
||||
frames.socket().drainHandler(v -> {
|
||||
workerDrainArmed = false;
|
||||
for (PlayerStream st : streams.values()) {
|
||||
if (!st.pausedForAggregate) continue;
|
||||
st.pausedForAggregate = false;
|
||||
maybeResumePlayer(st);
|
||||
}
|
||||
PlayerStream st = stream;
|
||||
if (st == null || !st.pausedForAggregate) return;
|
||||
st.pausedForAggregate = false;
|
||||
maybeResumePlayer(st);
|
||||
});
|
||||
}
|
||||
|
||||
@@ -322,53 +330,56 @@ public final class WorkerConn {
|
||||
if (st.shouldFlow()) st.player.resume();
|
||||
}
|
||||
|
||||
/** The player side of a stream this conn carries vanished: unbind it and FIN the client. */
|
||||
/** The player side of the tunnel this conn carries vanished: unbind it and FIN the client. */
|
||||
void playerGone(PlayerStream st) {
|
||||
boolean wasLive = streams.remove(st.sid) == st;
|
||||
boolean wasLive = stream == st;
|
||||
if (wasLive) stream = null;
|
||||
st.worker = null;
|
||||
if (wasLive) sendFin(st.sid);
|
||||
if (wasLive) sendFin();
|
||||
}
|
||||
|
||||
private void sendData(int sid, byte[] data) {
|
||||
frames.send(new ProtoWriter().u8(Protocol.MUX_DATA).varInt(sid).bytes(data).toBytes());
|
||||
private void sendData(byte[] data) {
|
||||
frames.send(new ProtoWriter().u8(Protocol.MUX_DATA).bytes(data).toBytes());
|
||||
}
|
||||
|
||||
private void sendFin(int sid) {
|
||||
frames.send(new ProtoWriter().u8(Protocol.MUX_FIN).varInt(sid).toBytes());
|
||||
private void sendFin() {
|
||||
frames.send(new ProtoWriter().u8(Protocol.MUX_FIN).toBytes());
|
||||
}
|
||||
|
||||
/** The reason is a trailing byte, optional on the wire; peers that predate it send none. */
|
||||
private void sendRst(int sid, int reason) {
|
||||
frames.send(new ProtoWriter().u8(Protocol.MUX_RST).varInt(sid).u8(reason).toBytes());
|
||||
private void sendRst(int reason) {
|
||||
frames.send(new ProtoWriter().u8(Protocol.MUX_RST).u8(reason).toBytes());
|
||||
}
|
||||
|
||||
private void sendWnd(int sid, int delta) {
|
||||
frames.send(new ProtoWriter().u8(Protocol.MUX_WND).varInt(sid).varInt(delta).toBytes());
|
||||
private void sendWnd(int delta) {
|
||||
frames.send(new ProtoWriter().u8(Protocol.MUX_WND).varInt(delta).toBytes());
|
||||
}
|
||||
|
||||
/** Answer the client's liveness probe, echoing its nonce. */
|
||||
private void sendPong(long nonce) {
|
||||
frames.send(new ProtoWriter().u8(Protocol.MUX_PONG).varInt(Protocol.MUX_CTL_SID).i64(nonce).toBytes());
|
||||
frames.send(new ProtoWriter().u8(Protocol.MUX_PONG).i64(nonce).toBytes());
|
||||
}
|
||||
|
||||
/**
|
||||
* Only the tunnel leg died. Where the session negotiated resumption the
|
||||
* player sockets are hung rather than closed, and wait for the client to
|
||||
* reattach their streams over a fresh conn (§7.5); otherwise this is the old,
|
||||
* player socket is hung rather than closed, and waits for the client to
|
||||
* reattach over a fresh conn (§7.5); otherwise this is the old,
|
||||
* unconditional close.
|
||||
*/
|
||||
public void onClose() {
|
||||
int parked = 0;
|
||||
for (PlayerStream st : streams.values()) {
|
||||
st.worker = null;
|
||||
if (hub.park(st)) {
|
||||
parked++;
|
||||
} else {
|
||||
hub.removeStream(st);
|
||||
st.player.close();
|
||||
}
|
||||
PlayerStream st = stream;
|
||||
stream = null;
|
||||
if (st == null) {
|
||||
LOG.info("worker {} closed (unbound)", id);
|
||||
return;
|
||||
}
|
||||
st.worker = null;
|
||||
if (hub.park(st)) {
|
||||
LOG.info("worker {} closed (player hung for reattach)", id);
|
||||
} else {
|
||||
hub.removeStream(st);
|
||||
st.player.close();
|
||||
LOG.info("worker {} closed (player dropped)", id);
|
||||
}
|
||||
LOG.info("worker {} closed ({} of {} player(s) hung for reattach)", id, parked, streams.size());
|
||||
streams.clear();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -79,7 +79,8 @@ class CryptoCodecTest {
|
||||
private static Hub testHub() {
|
||||
return new Hub(null, new Config("0.0.0.0", 25565, "test-psk", 30_000L, 10_000L,
|
||||
Protocol.DEFAULT_STREAM_WINDOW, 90_000L,
|
||||
true, 20_000L, 256, 256L * 2 * Protocol.DEFAULT_STREAM_WINDOW, 0L, 15_000L));
|
||||
true, 20_000L, 256, 256L * 2 * Protocol.DEFAULT_STREAM_WINDOW, 0L, 15_000L,
|
||||
0, 1, 0));
|
||||
}
|
||||
|
||||
private static ControlSession testSession(Hub hub, String id) {
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
package io.icybear.redapricot;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertNotNull;
|
||||
import static org.junit.jupiter.api.Assertions.assertNull;
|
||||
|
||||
class IpRateLimiterTest {
|
||||
|
||||
@Test
|
||||
void burstThenRefill() {
|
||||
IpRateLimiter lim = new IpRateLimiter(2, 2, 0);
|
||||
long t = 1_000;
|
||||
assertNull(lim.admit("1.1.1.1", t));
|
||||
assertNull(lim.admit("1.1.1.1", t));
|
||||
assertEquals(IpRateLimiter.Deny.RATE, lim.admit("1.1.1.1", t));
|
||||
// 500 ms at 2/s = 1 token.
|
||||
assertNull(lim.admit("1.1.1.1", t + 500));
|
||||
assertEquals(IpRateLimiter.Deny.RATE, lim.admit("1.1.1.1", t + 500));
|
||||
}
|
||||
|
||||
@Test
|
||||
void concurrentCapIndependentOfRate() {
|
||||
IpRateLimiter lim = new IpRateLimiter(0, 16, 1);
|
||||
long t = 1_000;
|
||||
assertNull(lim.admit("10.0.0.1", t));
|
||||
assertEquals(1, lim.concurrent("10.0.0.1"));
|
||||
assertEquals(IpRateLimiter.Deny.CONCURRENT, lim.admit("10.0.0.1", t));
|
||||
lim.release("10.0.0.1", t);
|
||||
assertEquals(0, lim.concurrent("10.0.0.1"));
|
||||
assertNull(lim.admit("10.0.0.1", t));
|
||||
}
|
||||
|
||||
@Test
|
||||
void ipsAreIndependent() {
|
||||
IpRateLimiter lim = new IpRateLimiter(1, 1, 1);
|
||||
long t = 1_000;
|
||||
assertNull(lim.admit("a", t));
|
||||
assertNull(lim.admit("b", t));
|
||||
assertEquals(IpRateLimiter.Deny.RATE, lim.admit("a", t));
|
||||
assertEquals(IpRateLimiter.Deny.RATE, lim.admit("b", t));
|
||||
}
|
||||
|
||||
@Test
|
||||
void bothOffIsNoOp() {
|
||||
IpRateLimiter lim = new IpRateLimiter(0, 16, 0);
|
||||
long t = 1_000;
|
||||
for (int i = 0; i < 100; i++) {
|
||||
assertNull(lim.admit("1.2.3.4", t));
|
||||
}
|
||||
assertEquals(0, lim.bucketCount());
|
||||
}
|
||||
|
||||
@Test
|
||||
void sweepDropsIdleEmptyBuckets() {
|
||||
IpRateLimiter lim = new IpRateLimiter(8, 8, 64);
|
||||
long t = 1_000;
|
||||
assertNull(lim.admit("9.9.9.9", t));
|
||||
lim.release("9.9.9.9", t);
|
||||
assertEquals(1, lim.bucketCount());
|
||||
lim.sweep(t + IpRateLimiter.SWEEP_MS - 1);
|
||||
assertEquals(1, lim.bucketCount());
|
||||
lim.sweep(t + IpRateLimiter.SWEEP_MS);
|
||||
assertEquals(0, lim.bucketCount());
|
||||
}
|
||||
|
||||
@Test
|
||||
void sweepKeepsLiveBuckets() {
|
||||
IpRateLimiter lim = new IpRateLimiter(8, 8, 64);
|
||||
long t = 1_000;
|
||||
assertNull(lim.admit("9.9.9.9", t));
|
||||
lim.sweep(t + IpRateLimiter.SWEEP_MS * 2);
|
||||
assertEquals(1, lim.bucketCount());
|
||||
assertEquals(1, lim.concurrent("9.9.9.9"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void denyDoesNotConsumeASlot() {
|
||||
IpRateLimiter lim = new IpRateLimiter(1, 1, 8);
|
||||
long t = 1_000;
|
||||
assertNull(lim.admit("x", t));
|
||||
assertNotNull(lim.admit("x", t));
|
||||
assertEquals(1, lim.concurrent("x"));
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user