break: replace muxed workers with 1:1 tunnels
Worker frames are now FrameType + payload; there is no stream id. Each player gets its own worker conn. maxTunnels (default 256) caps concurrent tunnels. The old maxConn pool size is ignored so existing configs do not silently admit only a handful of players. Resume, per-direction windows, the control session, and the DATA-only shaper stay. A dropped worker still hangs that one player and reattaches over a fresh conn. Add a hub-side per-IP limiter for player intents only (default 8/s, burst 16, 64 concurrent). Unmatched hostnames consume a token; Intent 17 is never counted. 0 disables each knob.
This commit is contained in:
iceBear67
+102
-91
@@ -15,7 +15,7 @@ There are three roles:
|
||||
| **Player** | any | An ordinary Minecraft client connecting through the hub. |
|
||||
|
||||
```
|
||||
Player ──MC──▶ Hub(server) ══WorkerConn(mux)══▶ Client ──MC──▶ Destination
|
||||
Player ──MC──▶ Hub(server) ══WorkerConn(1:1)══▶ Client ──MC──▶ Destination
|
||||
▲ registers patterns / receives control requests │
|
||||
└────────────── Control Session ────────────────────┘
|
||||
```
|
||||
@@ -154,18 +154,18 @@ RandLen : VarInt # 8 ≤ RandLen ≤ 64
|
||||
Rand : Bytes[RandLen] # cryptographically random
|
||||
Timestamp : I64 # client's epoch milliseconds
|
||||
Flags : VarInt # feature flags; bit 0x01 (STREAM_FC) MUST be set
|
||||
RecvWindow: VarInt # client's per-stream receive window, bytes (§7.3)
|
||||
RecvWindow: VarInt # client's per-connection receive window, bytes (§7.3)
|
||||
```
|
||||
|
||||
`Flags` is a bitfield of features.
|
||||
|
||||
| Bit | Name | Meaning |
|
||||
|-----|------|---------|
|
||||
| `0x01` | STREAM_FC | **Per-stream flow control** (§7.3). Mandatory. |
|
||||
| `0x02` | WORKER_HEARTBEAT | Mux-level `PING`/`PONG` on worker conns (§7.4). Optional. |
|
||||
| `0x01` | STREAM_FC | **Per-connection flow control** (§7.3). Mandatory. |
|
||||
| `0x02` | WORKER_HEARTBEAT | Connection-level `PING`/`PONG` on worker conns (§7.4). Optional. |
|
||||
| `0x04` | STREAM_RESUME | **Stream resumption** (§7.5): a worker-conn drop hangs the player rather than closing it. Optional. |
|
||||
|
||||
STREAM_FC is mandatory: `RecvWindow` advertises the client's per-stream receive
|
||||
STREAM_FC is mandatory: `RecvWindow` advertises the client's per-connection receive
|
||||
window in bytes and must be positive. The hub closes the connection if the flag
|
||||
is missing, `RecvWindow` is absent or non-positive, or the fields are malformed.
|
||||
|
||||
@@ -195,7 +195,7 @@ SessionReady : payload = [ 0x00, Flags: VarInt, RecvWindow: VarInt,
|
||||
```
|
||||
|
||||
The hub echoes the accepted flags (STREAM_FC set) followed by its own
|
||||
per-stream receive window. A client must reject a SessionReady without the
|
||||
per-connection receive window. A client must reject a SessionReady without the
|
||||
STREAM_FC flag or without a positive window (an unsupported hub).
|
||||
|
||||
`ResumeGraceMs` is present only when the hub accepts STREAM_RESUME, and states
|
||||
@@ -314,35 +314,36 @@ Msg : String
|
||||
|
||||
Purely informational; the receiver logs it.
|
||||
|
||||
## 7. Worker conn & multiplexing
|
||||
## 7. Worker conn
|
||||
|
||||
A **Worker Conn** (`Magic == 0x02`) carries player↔destination traffic for many
|
||||
players over one TCP connection using a minimal stream multiplexer. The unit of
|
||||
work is a **stream**. Stream ids are assigned by the **client** (the only side
|
||||
that opens streams), unique per worker conn, starting at 1 and increasing.
|
||||
A **Worker Conn** (`Magic == 0x02`) carries player↔destination traffic for
|
||||
**exactly one player**. The TCP connection *is* the tunnel: there is no stream
|
||||
id and no multiplexer. The client dials a fresh worker conn for each
|
||||
`ControlRequest` (and for each resumption attempt).
|
||||
|
||||
Each encrypted frame on a worker conn carries one **mux frame**:
|
||||
Each encrypted frame on a worker conn carries one **tunnel frame**:
|
||||
|
||||
```
|
||||
FrameType : u8
|
||||
StreamID : VarInt
|
||||
Data : Bytes[...] # remainder of the frame payload
|
||||
```
|
||||
|
||||
| FrameType | Name | Direction | Data |
|
||||
|-----------|------|-----------|------|
|
||||
| `0x00` | SYN | C → S | `CID: Bytes[16]` — open a stream to take over the pending player identified by CID. |
|
||||
| `0x01` | DATA | both | raw tunneled bytes for the stream. |
|
||||
| `0x02` | FIN | both | *(empty)* — graceful close of the stream (both directions). This is the "disconnect" the hub sends when the player leaves. |
|
||||
| `0x00` | SYN | C → S | `CID: Bytes[16]` — take over the pending player identified by CID. |
|
||||
| `0x01` | DATA | both | raw tunneled bytes. |
|
||||
| `0x02` | FIN | both | *(empty)* — graceful close (both directions). This is the "disconnect" the hub sends when the player leaves. |
|
||||
| `0x03` | RST | both | *(optional 1 byte reason)* — abnormal close (e.g. CID unknown/expired, destination dial failed). |
|
||||
| `0x04` | WND | both | `Delta: VarInt` — flow-control credit grant (§7.3). |
|
||||
| `0x05` | PING | both | `Nonce: I64` — liveness probe on reserved StreamID `0` (§7.4). |
|
||||
| `0x05` | PING | both | `Nonce: I64` — liveness probe (§7.4). |
|
||||
| `0x06` | PONG | both | `Nonce: I64` — echoes the probe's nonce (§7.4). |
|
||||
| `0x07` | RESUME | C → S | `CID: Bytes[16]`, `Accepted: I64`, `Delivered: I64` — reattach a hung stream to this conn (§7.5). |
|
||||
| `0x07` | RESUME | C → S | `CID: Bytes[16]`, `Accepted: I64`, `Delivered: I64` — reattach a hung player to this conn (§7.5). |
|
||||
| `0x08` | RESUME_ACK | S → C | `Accepted: I64`, `Delivered: I64`, `NewCID: Bytes[16]` — the reattach succeeded (§7.5). |
|
||||
|
||||
There is no explicit SYN-ACK: success is implied by the hub forwarding the
|
||||
buffered Handshake as the stream's first `DATA`; failure is an `RST`.
|
||||
The first business frame after `SessionReady` must be `SYN` or `RESUME`. A
|
||||
second bind on an already-bound conn is a protocol violation: the hub replies
|
||||
`RST` and closes. There is no explicit SYN-ACK: success is implied by the hub
|
||||
forwarding the buffered Handshake as the first `DATA`; failure is an `RST`.
|
||||
|
||||
`RST` reason codes. The byte remains optional — a peer that predates it sends
|
||||
none, and a receiver must tolerate its absence — but distinguishing the reasons
|
||||
@@ -358,25 +359,17 @@ took it" call for opposite responses.
|
||||
| `0x04` | FLOW_CONTROL | The peer exceeded its advertised window. |
|
||||
| `0x05` | DIAL_FAILED | The client could not reach the destination. |
|
||||
|
||||
### 7.1 Stream allocation (client side)
|
||||
### 7.1 Tunnel allocation (client side)
|
||||
|
||||
The client keeps a pool of `1 ≤ N ≤ max_conn` worker conns (`max_conn`
|
||||
configurable, `1..8`). The pool grows **breadth-first**: spreading streams over
|
||||
several connections keeps any single TCP connection from becoming the shared
|
||||
point of failure for every player on the tunnel. To place a new stream:
|
||||
The client dials one worker conn per player, up to a configurable
|
||||
`maxTunnels` cap (default 256, clamped to `[1, 4096]`). There is no pool and no
|
||||
least-loaded placement: a `ControlRequest` either gets its own TCP connection or
|
||||
is dropped (the hub then closes the player when `pendingTimeoutMs` fires).
|
||||
|
||||
1. Pick the worker conn with the **fewest active streams**, and use it.
|
||||
2. If that conn already carries at least one stream and
|
||||
`poolSize + dialsInFlight < max_conn`, dial another worker conn **in the
|
||||
background**. The stream just placed is not delayed by that dial; the new
|
||||
conn becomes the least-loaded one and picks up subsequent streams.
|
||||
3. Once the pool is at `max_conn`, streams stack on the least-loaded conn.
|
||||
Exceeding `8` active streams there is logged as pool saturation.
|
||||
|
||||
Only when the pool is *empty* does a caller dial synchronously, and then exactly
|
||||
one caller dials while the others wait for its result. A dial is never performed
|
||||
while holding the pool lock: session establishment is network I/O, and one
|
||||
unresponsive hub must not be able to block unrelated players.
|
||||
A dial is never performed while holding the live-set lock: session establishment
|
||||
is network I/O, and one unresponsive hub must not be able to block unrelated
|
||||
players. Each dial is independent and bounded by the session-establishment
|
||||
deadline (§7.4); callers are not serialized behind a single in-flight handshake.
|
||||
|
||||
### 7.2 End-to-end player flow
|
||||
|
||||
@@ -388,45 +381,43 @@ unresponsive hub must not be able to block unrelated players.
|
||||
control session. If no SYN arrives within `pendingTimeoutMs` (default 10000)
|
||||
the pending entry is dropped and the player socket closed.
|
||||
3. The client receives `ControlRequest`, looks up the destination for `Pattern`,
|
||||
allocates a worker conn + `StreamID`, and sends `SYN(StreamID, CID)`. In
|
||||
parallel it dials the destination and (if configured) writes a HAProxy v2
|
||||
header (§8) carrying `PlayerIP:PlayerPort`.
|
||||
dials a dedicated worker conn, and sends `SYN(CID)`. In parallel it dials the
|
||||
destination and (if configured) writes a HAProxy v2 header (§8) carrying
|
||||
`PlayerIP:PlayerPort`.
|
||||
4. The hub matches `CID` to the pending player, binds
|
||||
`(workerConn, StreamID) ↔ playerSocket`, forwards the buffered bytes as
|
||||
`DATA`, and resumes the player socket. Subsequent player bytes become `DATA`
|
||||
frames; `DATA` frames from the client are written to the player socket. If
|
||||
`CID` is unknown/expired the hub replies `RST`.
|
||||
5. When the player disconnects the hub sends `FIN` on the stream; the client
|
||||
closes the destination. When the destination closes, the client sends `FIN`;
|
||||
the hub closes the player socket. `RST` is treated the same way (hard close).
|
||||
`workerConn ↔ playerSocket`, forwards the buffered bytes as `DATA`, and
|
||||
resumes the player socket. Subsequent player bytes become `DATA` frames;
|
||||
`DATA` frames from the client are written to the player socket. If `CID` is
|
||||
unknown/expired the hub replies `RST`.
|
||||
5. When the player disconnects the hub sends `FIN`; the client closes the
|
||||
destination. When the destination closes, the client sends `FIN`; the hub
|
||||
closes the player socket. `RST` is treated the same way (hard close).
|
||||
|
||||
Data on a worker conn is subject to that TCP connection's back-pressure for
|
||||
its **aggregate** bandwidth; *per-stream* fairness is governed by the credit
|
||||
windows of §7.3.
|
||||
Data on a worker conn is subject to that TCP connection's back-pressure. Credit
|
||||
windows of §7.3 bound how many bytes may be in flight on that one tunnel.
|
||||
|
||||
### 7.3 Per-stream flow control
|
||||
### 7.3 Per-connection flow control
|
||||
|
||||
Every stream carries an independent credit window per direction:
|
||||
Every worker conn carries an independent credit window per direction:
|
||||
|
||||
* Each side advertised its **receive window** W (bytes) at session setup. A
|
||||
sender may have at most W un-credited DATA bytes outstanding per stream; the
|
||||
initial budget is W, spent as DATA is sent (`Data` length only — SYN/FIN/RST
|
||||
frames are free) starting with the very first DATA on the stream (including
|
||||
the hub's forwarded handshake).
|
||||
sender may have at most W un-credited DATA bytes outstanding; the initial
|
||||
budget is W, spent as DATA is sent (`Data` length only — SYN/FIN/RST frames
|
||||
are free) starting with the very first DATA (including the hub's forwarded
|
||||
handshake).
|
||||
* The receiver returns credit with `WND(Delta)` once bytes are **delivered to
|
||||
the terminal socket** (written to the player / destination connection), not
|
||||
when they are merely buffered. Receivers should batch grants (the reference
|
||||
implementations send one `WND` per W/2 bytes consumed).
|
||||
* A sender whose window is exhausted pauses reading **that stream's source
|
||||
socket only**; the shared worker conn is never paused because of a single
|
||||
stream. A receiver that observes more than W un-credited bytes on a stream
|
||||
may reset it (`RST`) as a protocol violation.
|
||||
* A sender whose window is exhausted pauses reading **that player's source
|
||||
socket only**. A receiver that observes more than W un-credited bytes may
|
||||
reset the tunnel (`RST`) as a protocol violation.
|
||||
* Senders should also cap individual DATA payloads (the reference
|
||||
implementations use 32 KiB) so one stream cannot monopolize the link for a
|
||||
full 1-MiB frame.
|
||||
implementations use 32 KiB) so one write cannot occupy the link for a full
|
||||
1-MiB frame.
|
||||
|
||||
Both windows may differ (each side enforces the one its peer advertised).
|
||||
`Delta` must be positive; a `WND` for an unknown stream id is ignored.
|
||||
`Delta` must be positive; a `WND` on an unbound worker conn is ignored.
|
||||
|
||||
### 7.4 Liveness
|
||||
|
||||
@@ -444,10 +435,9 @@ Every established session is therefore covered by a heartbeat:
|
||||
hub answers `Pong`. If no `Pong` arrives for `3 × pingIntervalMs`, the client
|
||||
closes the session, which triggers its normal reconnect with backoff.
|
||||
* **Worker conns** — when WORKER_HEARTBEAT was negotiated, the same exchange
|
||||
runs as mux `PING`/`PONG` frames on the reserved StreamID `0`. On timeout the
|
||||
client closes the conn and drops it from the pool. Its streams are reset,
|
||||
unless STREAM_RESUME was negotiated, in which case they are hung and reattached
|
||||
over a fresh conn instead (§7.5).
|
||||
runs as connection-level `PING`/`PONG` frames. On timeout the client closes
|
||||
the conn. The player is reset, unless STREAM_RESUME was negotiated, in which
|
||||
case it is hung and reattached over a fresh conn instead (§7.5).
|
||||
* **Hub side** — an established redapricot session that receives no frame for
|
||||
`sessionIdleTimeoutMs` (default 90000, `0` disables) is closed. Player
|
||||
connections are never subject to this.
|
||||
@@ -457,26 +447,24 @@ that has become unreachable at the IP layer.
|
||||
|
||||
Session establishment (§4) is bounded by a single deadline covering the dial,
|
||||
the `Rekey` write and the `SessionReady` read, and every frame write is bounded
|
||||
too — a peer that stops reading must not be able to park a whole multiplexed
|
||||
connection inside one write.
|
||||
too — a peer that stops reading must not be able to park a write forever.
|
||||
|
||||
### 7.5 Stream resumption (STREAM_RESUME)
|
||||
|
||||
A worker conn is only the middle leg of every stream it carries. When it dies,
|
||||
A worker conn is only the middle leg of the player it carries. When it dies,
|
||||
both terminal sockets — the player's and the destination's — are usually still
|
||||
healthy, so resetting the streams discards working connections because a
|
||||
replaceable transport failed. One conntrack expiry disconnects every player on
|
||||
that conn.
|
||||
healthy, so resetting the tunnel discards working connections because a
|
||||
replaceable transport failed.
|
||||
|
||||
With STREAM_RESUME negotiated, a worker-conn drop instead **hangs** each stream:
|
||||
With STREAM_RESUME negotiated, a worker-conn drop instead **hangs** the player:
|
||||
|
||||
* the hub pauses the player socket, keeps its state, and holds it for
|
||||
`ResumeGraceMs` from the moment of the *first* hang (an absolute deadline, so a
|
||||
flapping client cannot extend it indefinitely);
|
||||
* the client keeps the destination socket open and reattaches the stream over a
|
||||
fresh worker conn by sending `RESUME` with the stream's CID;
|
||||
* the client keeps the destination socket open and reattaches over a fresh
|
||||
worker conn by sending `RESUME` with the player's CID;
|
||||
* the hub answers `RESUME_ACK`, or `RST(UNKNOWN_STREAM)` if it holds no such
|
||||
stream — which is also what a client gets from a hub that has restarted.
|
||||
player — which is also what a client gets from a hub that has restarted.
|
||||
|
||||
**Resumption is byte-exact, and must be.** Frames handed to a dying socket are
|
||||
lost with no notification, and the frame cipher cannot be resynchronized, so each
|
||||
@@ -512,13 +500,13 @@ Two rules deserve emphasis, because the obvious simplifications are wrong:
|
||||
credit can ever come back.
|
||||
|
||||
`RESUME_ACK` carries a freshly minted `NewCID`, which replaces the old one. A CID
|
||||
therefore stays single-use even though a stream may be reattached many times, so
|
||||
therefore stays single-use even though a player may be reattached many times, so
|
||||
a leaked CID grants no more than the outage in which it was observed.
|
||||
|
||||
Resumption is **hub-instance-affine**: a CID means nothing to a second hub behind
|
||||
an L4 load balancer, which answers `RST(UNKNOWN_STREAM)` and lets the client tear
|
||||
down at once. Because the grace period holds player sockets and their buffers, a
|
||||
hub bounds the number of hung streams and the bytes they retain, dropping the
|
||||
hub bounds the number of hung players and the bytes they retain, dropping the
|
||||
oldest first when either cap is reached.
|
||||
|
||||
## 8. HAProxy protocol v2 (optional)
|
||||
@@ -555,12 +543,15 @@ big-endian.
|
||||
"resumeGraceMs": 20000,
|
||||
"maxParkedStreams": 256,
|
||||
"statsIntervalMs": 0,
|
||||
"registrationGraceMs": 15000
|
||||
"registrationGraceMs": 15000,
|
||||
"playerRatePerSec": 8,
|
||||
"playerBurst": 16,
|
||||
"maxPlayersPerIp": 64
|
||||
}
|
||||
```
|
||||
|
||||
`streamWindowBytes` (optional, default 262144, clamped to [32768, 8388608]) is
|
||||
the hub's advertised per-stream receive window (§7.3).
|
||||
the hub's advertised per-connection receive window (§7.3).
|
||||
|
||||
`sessionIdleTimeoutMs` (optional, default 90000) closes an established control
|
||||
session or worker conn that has gone silent for that long (§7.4). It must stay
|
||||
@@ -587,13 +578,27 @@ session's routes are kept as **orphaned** rather than deleted (§5.2). `0`
|
||||
disables it, restoring the behaviour of dropping routes the instant a session
|
||||
closes.
|
||||
|
||||
`playerRatePerSec` (optional, default 8) and `playerBurst` (optional, default
|
||||
16) are a per-IP token bucket applied **only to player connections** (any
|
||||
handshake whose `Intent` is not 17 or 18). One arrival consumes one token;
|
||||
without a token the socket is closed after the handshake, before `match`, CID
|
||||
minting, or pause. Unmatched hostnames still consume a token — otherwise a
|
||||
hostname scan is a free flood. Intent 17 is never admitted through the
|
||||
limiter: every worker conn comes from the client's one address, and limiting
|
||||
those would be the hub throttling its own client. `playerRatePerSec: 0` turns
|
||||
the bucket off.
|
||||
|
||||
`maxPlayersPerIp` (optional, default 64) caps concurrent player sockets from
|
||||
one address (pending + live + parked). `0` disables the cap. Addresses are
|
||||
matched exactly; IPv6 `/64` aggregation is out of scope.
|
||||
|
||||
### 9.2 Client — JSON
|
||||
|
||||
```json
|
||||
{
|
||||
"server": "127.0.0.1:25565",
|
||||
"psk": "change-me",
|
||||
"maxConn": 4,
|
||||
"maxTunnels": 256,
|
||||
"pingIntervalMs": 20000,
|
||||
"streamWindowBytes": 262144,
|
||||
"maxBandwidth": "20mbps",
|
||||
@@ -606,8 +611,14 @@ closes.
|
||||
}
|
||||
```
|
||||
|
||||
`maxTunnels` (optional, default 256, clamped to [1, 4096]) is how many
|
||||
concurrent 1:1 worker connections the client will hold. A `ControlRequest`
|
||||
arriving at the cap is dropped. The older `maxConn` key (the mux-era pool size,
|
||||
clamped 1–8) is ignored if present: treating it as a player cap would silently
|
||||
limit a previously-working config to a handful of players.
|
||||
|
||||
`streamWindowBytes` (optional, default 262144, clamped to [32768, 8388608]) is
|
||||
the client's advertised per-stream receive window (§7.3).
|
||||
the client's advertised per-connection receive window (§7.3).
|
||||
|
||||
`streamResume` (optional, default true) offers STREAM_RESUME (§7.5). With it
|
||||
false the client never offers the flag, never retains a byte for retransmission,
|
||||
@@ -626,7 +637,7 @@ time per conn. These distinguish a slow backend from a saturated uplink from a
|
||||
bad path, which throughput alone cannot.
|
||||
|
||||
`maxBandwidth` (optional, default unlimited) caps the aggregate rate at which the
|
||||
client sends `DATA` to the hub, shared fairly across streams. Accepts `"20mbps"`
|
||||
client sends `DATA` to the hub, shared fairly across tunnels. Accepts `"20mbps"`
|
||||
(decimal bit units), `"2MB/s"` (binary byte units), or a bare number of bytes per
|
||||
second; the minimum is 8192 B/s. **This is a purely local policy and has no
|
||||
effect on the wire format** — a shaped client is indistinguishable from a slow
|
||||
@@ -664,17 +675,17 @@ not appear on the tunnel wire, and the exchange is invisible to the player.
|
||||
| pattern matching | case-insensitive, whole-string regex; first match wins |
|
||||
| CID length | 16 bytes |
|
||||
| max frame payload | 1 MiB |
|
||||
| pool growth | breadth-first: grow to `max_conn` before stacking |
|
||||
| saturation threshold (logged) | active streams `> 8` at `max_conn` |
|
||||
| max worker conns | `max_conn ∈ [1,8]` |
|
||||
| feature flag: per-stream flow control | `0x01` (mandatory) |
|
||||
| worker framing | `FrameType` + payload; no stream id |
|
||||
| max concurrent worker conns | `maxTunnels`, default 256, clamped `[1, 4096]` |
|
||||
| player IP rate / burst / concurrent | `8 /s`, burst `16`, `maxPlayersPerIp` `64` (Intent ∉ {17, 18} only) |
|
||||
| feature flag: per-connection flow control | `0x01` (mandatory) |
|
||||
| feature flag: worker heartbeat | `0x02` (negotiated) |
|
||||
| heartbeat timeout | `3 × pingIntervalMs` |
|
||||
| hub session idle timeout | 90000 ms (`0` disables) |
|
||||
| stream window default / bounds | 256 KiB, clamped to [32 KiB, 8 MiB] |
|
||||
| connection window default / bounds | 256 KiB, clamped to [32 KiB, 8 MiB] |
|
||||
| feature flag: stream resumption | 0x04 (negotiated) |
|
||||
| mux RESUME / RESUME_ACK | 0x07 / 0x08 |
|
||||
| RESUME / RESUME_ACK | 0x07 / 0x08 |
|
||||
| resume grace: hub / client default | 20000 ms / 15000 ms (hub value advertised) |
|
||||
| retained region per stream per direction | bounded by the stream window |
|
||||
| retained region per tunnel per direction | bounded by the connection window |
|
||||
| WND grant batching (reference) | one grant per window/2 consumed |
|
||||
| DATA chunk cap (reference) | 32 KiB |
|
||||
|
||||
Reference in New Issue
Block a user