cloud/redapricot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

break: replace muxed workers with 1:1 tunnels

Browse Source 
Worker frames are now FrameType + payload; there is no stream id.
Each player gets its own worker conn. maxTunnels (default 256)
caps concurrent tunnels. The old maxConn pool size is ignored so
existing configs do not silently admit only a handful of players.

Resume, per-direction windows, the control session, and the
DATA-only shaper stay. A dropped worker still hangs that one
player and reattaches over a fresh conn.

Add a hub-side per-IP limiter for player intents only (default
8/s, burst 16, 64 concurrent). Unmatched hostnames consume a
token; Intent 17 is never counted. 0 disables each knob.
This commit is contained in:
iceBear67
2026-08-15 18:32:51 +08:00
parent da17140583
commit 4df2560331
27 changed files with 1174 additions and 856 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+42 -39
View File
@@ -13,9 +13,9 @@ field, so a vanilla Minecraft client needs no modification.
> made reachable from the outside.
```
Player ──MC──▶ Hub (Java) ══ Worker Conn (mux) ══▶ Client (Go) ──MC──▶ Real MC server
vanilla client public IP encrypted, pooled behind NAT (localhost)
│ ▲
Player ──MC──▶ Hub (Java) ══ Worker Conn (1:1) ══▶ Client (Go) ──MC──▶ Real MC server
vanilla client public IP one encrypted TCP behind NAT (localhost)
│ per player ▲
└────────────── Control Session ─────────┘
(pattern registration + control requests)
```
@@ -40,13 +40,11 @@ one or more hostname patterns — each a **regular expression**. When a player
connects to the hub with a hostname that matches a registered pattern (and any
normal `Intent`), the hub assigns a random **CID**, buffers
the player's bytes, and asks the client (via the control session) to take over.
The client picks a **worker connection** — a multiplexed, encrypted TCP link
that carries many players as lightweight *streams* — opens a stream for that CID,
dials the real destination (optionally announcing the player's real IP with the
**HAProxy v2** protocol), and bridges the two ends. Worker connections are
pooled: the client uses up to `maxConn` of them and always places a new stream
on the least-loaded one, growing the pool to `maxConn` before stacking streams
so no single TCP connection carries every player.
The client dials a fresh **worker connection** — one encrypted TCP link per
player — announces the CID with `SYN`, dials the real destination (optionally
announcing the player's real IP with the **HAProxy v2** protocol), and bridges
the two ends. `maxTunnels` is the only player cap; the retired `maxConn` field
is ignored so an old config does not silently admit only four players.
## Repository layout
@@ -114,7 +112,7 @@ cp client/config.example.json client.json
# {
# "server": "hub.example.com:25565",
# "psk": "same-as-the-hub",
# "maxConn": 4,
# "maxTunnels": 256,
# "mappings": [
# { "pattern": "mc\\.example\\.com", "destination": "127.0.0.1:25566", "proxyProtocol": true }
# ]
@@ -175,6 +173,9 @@ secrets. The base image and build flags live in `.ko.yaml`.
| `maxParkedStreams` | `256` | Cap on simultaneously hung players; `maxParkedBytes` (default `maxParkedStreams × 2 × streamWindowBytes`) caps what they retain. Past either, the oldest are dropped. |
| `statsIntervalMs` | `0` (off) | Log a periodic line with live/hung stream counts, retained bytes and pattern count. |
| `registrationGraceMs` | `15000` | Keep a closed control session's routes as *orphaned* for this long, holding players that arrive on them instead of refusing them, and replaying their requests once the client re-registers. `0` disables it. |
| `playerRatePerSec` | `8` | Per-IP token-bucket rate for **player** connections only (Intent ∉ {17, 18}). Unmatched hostnames still consume a token. `0` disables. |
| `playerBurst` | `16` | Token-bucket depth for `playerRatePerSec`. |
| `maxPlayersPerIp` | `64` | Concurrent player sockets (pending + live + parked) from one IP. `0` disables. Intent 17 is never counted. |
### Client (`client/config.example.json`)
@@ -182,7 +183,7 @@ secrets. The base image and build flags live in `.ko.yaml`.
|------------------|--------------------|---------|
| `server` | *(required)* | Hub `host:port`. |
| `psk` | *(required)* | Shared secret; must match the hub. |
| `maxConn` | `1` (clamped 1–8) | Max worker connections in the pool. |
| `maxTunnels` | `256` (clamped 1–4096) | Max concurrent player tunnels. The retired `maxConn` field is ignored. |
| `pingIntervalMs` | `20000` (min 1000) | Heartbeat interval for the control session and every worker conn. A session with no reply for `3×` this is dropped and re-established. |
| `maxBandwidth` | *(unlimited)* | Caps what the client uploads to the hub, summed over every player — the direction carrying the game server's output, and the one a home uplink runs out of first. `"20mbps"`, `"512kbps"`, `"2MB/s"`, or a bare number of bytes/sec. **Bit units are decimal (`20mbps` = 20,000,000 bit/s); byte units are binary (`2MB/s` = 2 MiB/s).** Set it slightly below your real upload speed — framing and TCP/IP overhead are not counted. The budget is shared fairly across players, so one person loading chunks cannot time the others out. |
| `streamWindowBytes` | `262144` | Advertised per-stream receive window, clamped to [32 KiB, 8 MiB]. |
@@ -217,20 +218,23 @@ go test ./e2e/... -v
The e2e suite covers: a full player round-trip with verbatim handshake
forwarding and case-insensitive matching, regex wildcard pattern routing,
multi-megabyte transfers, concurrent
streams spreading across multiple worker connections, HAProxy v2 source-address
multi-megabyte transfers, concurrent players each on their own worker
connection, HAProxy v2 source-address
propagation, Velocity modern-forwarding interception (signed player-info
handoff to a mock Paper backend), player- and destination-initiated disconnect propagation, wrong-PSK
rejection, dropping of unmatched hostnames, stream isolation under a slow
player and under a slow destination (no head-of-line blocking), rejection
of pre-flow-control peers, and stream resumption — a tunnel hard-reset
mid-transfer with the player connection held open, asserting the byte stream
neither gains nor loses a byte, across concurrent streams, plus grace expiry and
the resume-disabled path, and control-outage handling — a player arriving while
the client's control session is down is held and then served once it
re-registers, with the grace-disabled and grace-expired paths pinned too. The Go and Java crypto layers are
independently pinned to the same SHA3-224 test vector so they cannot silently
drift apart.
handoff to a mock Paper backend), player- and destination-initiated disconnect
propagation, wrong-PSK rejection, dropping of unmatched hostnames, isolation
under a slow player and under a slow destination (no head-of-line blocking),
rejection of pre-flow-control peers, and stream resumption — a tunnel
hard-reset mid-transfer with the player connection held open, asserting the
byte stream neither gains nor loses a byte, across concurrent players, plus
grace expiry and the resume-disabled path, and control-outage handling — a
player arriving while the client's control session is down is held and then
served once it re-registers, with the grace-disabled and grace-expired paths
pinned too, and the per-IP player limiter — burst overflow and
`maxPlayersPerIp` drop extras before they become pending, Intent 17 is never
counted, and both knobs at `0` restore the unlimited path. The Go and Java
crypto layers are independently pinned to the same SHA3-224 test vector so they
cannot silently drift apart.
## Design notes & limitations
@@ -238,29 +242,28 @@ drift apart.
ChaCha20-encrypted (no AEAD tag) to minimize overhead. This protects against
casual sniffing, not a determined active attacker (see the note at the top of
[PROTOCOL.md](PROTOCOL.md) and [docs/architecture.md](docs/architecture.md) §8).
* **Per-stream flow control.** Each stream has credit-based windows in both
directions (windows exchanged at session setup, default 256 KiB), so a slow
player or slow destination jams only its own stream at a bounded buffer — no
application-level head-of-line blocking between streams. What remains is
TCP-level HOL (packet loss stalls a whole worker connection briefly);
raising `maxConn` spreads that.
* **Per-tunnel flow control.** Each worker connection has credit-based windows
in both directions (windows exchanged at session setup, default 256 KiB), so
a slow player or slow destination jams only its own tunnel at a bounded
buffer. There is no application-level head-of-line blocking: each player
owns a TCP connection.
* **Liveness is explicit.** Every session heartbeats, every socket write is
bounded, and session establishment has a deadline. A path that dies silently —
no `FIN`, no `RST`, as when a NAT or firewall forgets an established flow — is
detected within `3 × pingIntervalMs`, the dead connection is dropped from the
pool, and service is restored without operator action. TCP keepalive is on as
a second line of defence.
detected within `3 × pingIntervalMs`, the dead connection is dropped, and
service is restored without operator action. TCP keepalive is on as a
second line of defence.
* **A control-session reconnect no longer refuses new players.** While a client
is reconnecting the hub has no route for it, so arriving players used to be
told there is no such server. Those routes are now held briefly and the
players with them, then served once the client re-registers.
* **A dropped tunnel no longer drops the players.** A worker connection is only
the middle leg of the streams it carries; when it dies both terminal sockets
are usually still healthy. The hub now hangs those players while the client
reattaches their streams over a fresh connection, replaying byte-exactly from
the middle leg of the player it carries; when it dies both terminal sockets
are usually still healthy. The hub now hangs that player while the client
reattaches the tunnel over a fresh connection, replaying byte-exactly from
the offset the peer reports, so a conntrack expiry costs a stall rather than
disconnecting everyone on that connection. Negotiated, and `streamResume:
false` on either side restores the old behaviour.
a disconnect. Negotiated, and `streamResume: false` on either side restores
the old behaviour.
* **Single hub event loop.** The hub deploys one Vert.x verticle, so all state
is confined to one event loop (no locking). Throughput is bounded by one core;
ample for hundreds of players, not designed for tens of thousands.