impl connection recovery

README.md

@@ -169,6 +169,12 @@ secrets. The base image and build flags live in `.ko.yaml`.
 | `timestampWindowMs` | `30000`          | Allowed clock skew for a client's rekey timestamp. |
 | `pendingTimeoutMs`  | `10000`          | How long a matched player waits for a worker to take over. |
 | `sessionIdleTimeoutMs` | `90000`       | Close an established control/worker session that receives no frame for this long. Must exceed the client's `pingIntervalMs`; `0` disables. Player connections are unaffected. |
+| `streamWindowBytes` | `262144`         | Advertised per-stream receive window, clamped to [32 KiB, 8 MiB]. |
+| `streamResume`      | `true`           | Hang a player when its worker connection drops, so the client can reattach the stream instead of the player being disconnected. `false` restores the previous behaviour exactly and retains nothing. |
+| `resumeGraceMs`     | `20000`          | How long a hung player is held. Advertised to clients, which clamp their own retry budget below it. Must exceed the client's grace by at least one dial. |
+| `maxParkedStreams`  | `256`            | Cap on simultaneously hung players; `maxParkedBytes` (default `maxParkedStreams × 2 × streamWindowBytes`) caps what they retain. Past either, the oldest are dropped. |
+| `statsIntervalMs`   | `0` (off)        | Log a periodic line with live/hung stream counts, retained bytes and pattern count. |
+| `registrationGraceMs` | `15000`        | Keep a closed control session's routes as *orphaned* for this long, holding players that arrive on them instead of refusing them, and replaying their requests once the client re-registers. `0` disables it. |
 
 ### Client (`client/config.example.json`)
 
@@ -178,6 +184,11 @@ secrets. The base image and build flags live in `.ko.yaml`.
 | `psk`            | *(required)*       | Shared secret; must match the hub. |
 | `maxConn`        | `1` (clamped 1–8)  | Max worker connections in the pool. |
 | `pingIntervalMs` | `20000` (min 1000) | Heartbeat interval for the control session and every worker conn. A session with no reply for `3×` this is dropped and re-established. |
+| `maxBandwidth`   | *(unlimited)*      | Caps what the client uploads to the hub, summed over every player — the direction carrying the game server's output, and the one a home uplink runs out of first. `"20mbps"`, `"512kbps"`, `"2MB/s"`, or a bare number of bytes/sec. **Bit units are decimal (`20mbps` = 20,000,000 bit/s); byte units are binary (`2MB/s` = 2 MiB/s).** Set it slightly below your real upload speed — framing and TCP/IP overhead are not counted. The budget is shared fairly across players, so one person loading chunks cannot time the others out. |
+| `streamWindowBytes` | `262144`        | Advertised per-stream receive window, clamped to [32 KiB, 8 MiB]. |
+| `streamResume`   | `true`             | Reattach streams over a fresh connection when a worker connection drops, instead of disconnecting those players. `false` restores the previous behaviour exactly: nothing is retained and the send path is unchanged. |
+| `resumeGraceMs`  | `15000` (min 2000) | How long a stream keeps trying to reattach, clamped below the hub's advertised grace. Sized against the *backend*: a hung player stops answering the game server's KeepAlive, and vanilla disconnects a silent client at 30s, so a longer grace only resumes sessions the backend then kicks. |
+| `statsIntervalMs` | `0` (off)         | Log a periodic diagnostics line, plus a summary per stream at close: bytes each way, how long the stream was blocked on the flow-control window versus the bandwidth cap, receive-queue high-water mark, and heartbeat round-trip time per connection. Those distinguish a slow backend from a saturated uplink from a bad path, which throughput alone cannot. |
 | `mappings[]`     | *(≥1 required)*    | Route table (below). |
 | `mappings[].pattern`       | —        | Regex matched against the whole player hostname, case-insensitively. Escape dots (`mc\.example\.com`); `.` is a wildcard. |
 | `mappings[].destination`   | —        | Real server `host:port` to forward to. |
@@ -211,8 +222,13 @@ streams spreading across multiple worker connections, HAProxy v2 source-address
 propagation, Velocity modern-forwarding interception (signed player-info
 handoff to a mock Paper backend), player- and destination-initiated disconnect propagation, wrong-PSK
 rejection, dropping of unmatched hostnames, stream isolation under a slow
-player and under a slow destination (no head-of-line blocking), and rejection
-of pre-flow-control peers. The Go and Java crypto layers are
+player and under a slow destination (no head-of-line blocking), rejection
+of pre-flow-control peers, and stream resumption — a tunnel hard-reset
+mid-transfer with the player connection held open, asserting the byte stream
+neither gains nor loses a byte, across concurrent streams, plus grace expiry and
+the resume-disabled path, and control-outage handling — a player arriving while
+the client's control session is down is held and then served once it
+re-registers, with the grace-disabled and grace-expired paths pinned too. The Go and Java crypto layers are
 independently pinned to the same SHA3-224 test vector so they cannot silently
 drift apart.
 
@@ -234,6 +250,17 @@ drift apart.
   detected within `3 × pingIntervalMs`, the dead connection is dropped from the
   pool, and service is restored without operator action. TCP keepalive is on as
   a second line of defence.
+* **A control-session reconnect no longer refuses new players.** While a client
+  is reconnecting the hub has no route for it, so arriving players used to be
+  told there is no such server. Those routes are now held briefly and the
+  players with them, then served once the client re-registers.
+* **A dropped tunnel no longer drops the players.** A worker connection is only
+  the middle leg of the streams it carries; when it dies both terminal sockets
+  are usually still healthy. The hub now hangs those players while the client
+  reattaches their streams over a fresh connection, replaying byte-exactly from
+  the offset the peer reports, so a conntrack expiry costs a stall rather than
+  disconnecting everyone on that connection. Negotiated, and `streamResume:
+  false` on either side restores the old behaviour.
 * **Single hub event loop.** The hub deploys one Vert.x verticle, so all state
   is confined to one event loop (no locking). Throughput is bounded by one core;
   ample for hundreds of players, not designed for tens of thousands.