impl connection recovery
This commit is contained in:
iceBear67
+185
-6
@@ -150,6 +150,7 @@ RecvWindow: VarInt # client's per-stream receive window, bytes (§7.3)
|
||||
|-----|------|---------|
|
||||
| `0x01` | STREAM_FC | **Per-stream flow control** (§7.3). Mandatory. |
|
||||
| `0x02` | WORKER_HEARTBEAT | Mux-level `PING`/`PONG` on worker conns (§7.4). Optional. |
|
||||
| `0x04` | STREAM_RESUME | **Stream resumption** (§7.5): a worker-conn drop hangs the player rather than closing it. Optional. |
|
||||
|
||||
STREAM_FC is mandatory: `RecvWindow` advertises the client's per-stream receive
|
||||
window in bytes and must be positive. The hub closes the connection if the flag
|
||||
@@ -176,13 +177,22 @@ frame (both directions) is Phase B, counters reset to 0.
|
||||
The hub then sends one Phase-B frame to confirm success:
|
||||
|
||||
```
|
||||
SessionReady : payload = [ 0x00, Flags: VarInt, RecvWindow: VarInt ]
|
||||
SessionReady : payload = [ 0x00, Flags: VarInt, RecvWindow: VarInt,
|
||||
ResumeGraceMs: VarInt ] # only when STREAM_RESUME is set
|
||||
```
|
||||
|
||||
The hub echoes the accepted flags (STREAM_FC set) followed by its own
|
||||
per-stream receive window. A client must reject a SessionReady without the
|
||||
STREAM_FC flag or without a positive window (an unsupported hub).
|
||||
|
||||
`ResumeGraceMs` is present only when the hub accepts STREAM_RESUME, and states
|
||||
how long it will hang a player waiting for that player's stream to be reattached
|
||||
(§7.5). The client clamps its own retry budget below this value. Advertising it
|
||||
rather than assuming matching configuration is deliberate: the client must always
|
||||
give up first, and if the hub instead dropped a hung player while the client was
|
||||
still reattaching, the failure would be a silent hang rather than an error. A hub
|
||||
that sets the flag but omits the field is treated as not supporting resumption.
|
||||
|
||||
A hub that rejects the session simply closes the TCP connection (optionally
|
||||
after a Phase-B `Error` frame, §6). After `SessionReady`:
|
||||
|
||||
@@ -246,6 +256,40 @@ Because the pattern is a regex, a literal dot must be escaped (`mc\.example\.com
|
||||
an unescaped `.` is the regex "any character" wildcard. A pattern that fails to
|
||||
compile is rejected at `Register` time with `RegisterAck` status `1`.
|
||||
|
||||
### 5.2 Orphaned routes (control-session outage)
|
||||
|
||||
When a control session closes, its registrations are **not** deleted straight
|
||||
away. They are marked *orphaned* and kept for `registrationGraceMs`.
|
||||
|
||||
This costs nothing on the wire — it is entirely hub-side behaviour — but it
|
||||
closes a gap that is otherwise very visible. A client whose control session dies
|
||||
reconnects with backoff, and until it re-registers the hub has no route for it,
|
||||
so every player arriving in that window is told there is no such server. The
|
||||
players already tunneled are unaffected, since they ride worker conns, which a
|
||||
control-session close never touches.
|
||||
|
||||
While a route is orphaned:
|
||||
|
||||
* a player matching it is **held** — paused, with its handshake buffered exactly
|
||||
as for a normal pending player — and no `ControlRequest` is sent, because there
|
||||
is no session to send it to;
|
||||
* a player that was already pending when the session closed is moved into the
|
||||
same held state rather than being dropped;
|
||||
* when any client registers that pattern again, the hub delivers the
|
||||
`ControlRequest` it never sent and the player proceeds normally. The held
|
||||
player's deadline switches from the registration grace to `pendingTimeoutMs`
|
||||
at that point, since it is now waiting for a worker rather than for a route.
|
||||
|
||||
If the grace expires with no client having re-registered, the route and every
|
||||
player held on it are dropped. `registrationGraceMs: 0` disables the mechanism
|
||||
and restores the immediate-drop behaviour.
|
||||
|
||||
Note the hub cannot distinguish "this client is reconnecting" from "this client
|
||||
is gone for good" — that is what the grace period is a bet on. It is bounded on
|
||||
the client side too: the reference client retries immediately on a control-session
|
||||
drop and caps its backoff at 10s, so the bet is usually settled in well under a
|
||||
second.
|
||||
|
||||
## 6. Error frame (any redapricot connection)
|
||||
|
||||
At any time either side may send, then close:
|
||||
@@ -281,10 +325,26 @@ Data : Bytes[...] # remainder of the frame payload
|
||||
| `0x04` | WND | both | `Delta: VarInt` — flow-control credit grant (§7.3). |
|
||||
| `0x05` | PING | both | `Nonce: I64` — liveness probe on reserved StreamID `0` (§7.4). |
|
||||
| `0x06` | PONG | both | `Nonce: I64` — echoes the probe's nonce (§7.4). |
|
||||
| `0x07` | RESUME | C → S | `CID: Bytes[16]`, `Accepted: I64`, `Delivered: I64` — reattach a hung stream to this conn (§7.5). |
|
||||
| `0x08` | RESUME_ACK | S → C | `Accepted: I64`, `Delivered: I64`, `NewCID: Bytes[16]` — the reattach succeeded (§7.5). |
|
||||
|
||||
There is no explicit SYN-ACK: success is implied by the hub forwarding the
|
||||
buffered Handshake as the stream's first `DATA`; failure is an `RST`.
|
||||
|
||||
`RST` reason codes. The byte remains optional — a peer that predates it sends
|
||||
none, and a receiver must tolerate its absence — but distinguishing the reasons
|
||||
matters for resumption, where "this stream is gone" and "someone else already
|
||||
took it" call for opposite responses.
|
||||
|
||||
| Code | Name | Meaning |
|
||||
|------|------|---------|
|
||||
| `0x00` | UNSPECIFIED | No reason given (also the meaning of an absent byte). |
|
||||
| `0x01` | UNKNOWN_STREAM | CID unknown or expired, or the hub restarted. Terminal: stop retrying. |
|
||||
| `0x02` | ALREADY_BOUND | Another reattach already claimed this stream. Do **not** tear down. |
|
||||
| `0x03` | RESUME_ABANDONED | The peer gave up reattaching. |
|
||||
| `0x04` | FLOW_CONTROL | The peer exceeded its advertised window. |
|
||||
| `0x05` | DIAL_FAILED | The client could not reach the destination. |
|
||||
|
||||
### 7.1 Stream allocation (client side)
|
||||
|
||||
The client keeps a pool of `1 ≤ N ≤ max_conn` worker conns (`max_conn`
|
||||
@@ -372,8 +432,9 @@ Every established session is therefore covered by a heartbeat:
|
||||
closes the session, which triggers its normal reconnect with backoff.
|
||||
* **Worker conns** — when WORKER_HEARTBEAT was negotiated, the same exchange
|
||||
runs as mux `PING`/`PONG` frames on the reserved StreamID `0`. On timeout the
|
||||
client closes the conn; its streams are reset and it is dropped from the pool,
|
||||
so the next player gets a freshly dialed connection.
|
||||
client closes the conn and drops it from the pool. Its streams are reset,
|
||||
unless STREAM_RESUME was negotiated, in which case they are hung and reattached
|
||||
over a fresh conn instead (§7.5).
|
||||
* **Hub side** — an established redapricot session that receives no frame for
|
||||
`sessionIdleTimeoutMs` (default 90000, `0` disables) is closed. Player
|
||||
connections are never subject to this.
|
||||
@@ -381,11 +442,72 @@ Every established session is therefore covered by a heartbeat:
|
||||
Both ends also enable TCP keepalive, which catches the narrower case of a peer
|
||||
that has become unreachable at the IP layer.
|
||||
|
||||
Session establishment (§3.2) is bounded by a single deadline covering the dial,
|
||||
Session establishment (§4) is bounded by a single deadline covering the dial,
|
||||
the `Rekey` write and the `SessionReady` read, and every frame write is bounded
|
||||
too — a peer that stops reading must not be able to park a whole multiplexed
|
||||
connection inside one write.
|
||||
|
||||
### 7.5 Stream resumption (STREAM_RESUME)
|
||||
|
||||
A worker conn is only the middle leg of every stream it carries. When it dies,
|
||||
both terminal sockets — the player's and the destination's — are usually still
|
||||
healthy, so resetting the streams discards working connections because a
|
||||
replaceable transport failed. One conntrack expiry disconnects every player on
|
||||
that conn.
|
||||
|
||||
With STREAM_RESUME negotiated, a worker-conn drop instead **hangs** each stream:
|
||||
|
||||
* the hub pauses the player socket, keeps its state, and holds it for
|
||||
`ResumeGraceMs` from the moment of the *first* hang (an absolute deadline, so a
|
||||
flapping client cannot extend it indefinitely);
|
||||
* the client keeps the destination socket open and reattaches the stream over a
|
||||
fresh worker conn by sending `RESUME` with the stream's CID;
|
||||
* the hub answers `RESUME_ACK`, or `RST(UNKNOWN_STREAM)` if it holds no such
|
||||
stream — which is also what a client gets from a hub that has restarted.
|
||||
|
||||
**Resumption is byte-exact, and must be.** Frames handed to a dying socket are
|
||||
lost with no notification, and the frame cipher cannot be resynchronized, so each
|
||||
side replays whatever the other did not receive. Splicing the stream even one
|
||||
byte off corrupts the tunneled protocol.
|
||||
|
||||
Three offsets are tracked per direction, and they are not interchangeable:
|
||||
|
||||
| Offset | Meaning | Used for |
|
||||
|--------|---------|----------|
|
||||
| `Sent` | bytes handed to the wire | the end of the retained region |
|
||||
| `Accepted` | bytes taken off the wire toward the terminal socket | **where to replay from** |
|
||||
| `Delivered` | bytes actually written to the terminal socket | **how to restate the window** |
|
||||
|
||||
Each side retains the bytes between what the peer has credited and what it has
|
||||
sent. This costs no new bound: flow control (§7.3) already caps outstanding bytes
|
||||
at one window, so the retained region *is* the outstanding region.
|
||||
|
||||
On reattach both sides replay `[peer's Accepted, Sent)` and set
|
||||
`SendWindow = W − (Sent − peer's Delivered)`, then discard their own pending
|
||||
credit — the exchanged `Delivered` already carries everything those deltas would
|
||||
have, so emitting both would grant the same bytes twice.
|
||||
|
||||
Two rules deserve emphasis, because the obvious simplifications are wrong:
|
||||
|
||||
* *Accepted*, not *Delivered*, is the replay point. Delivery is signalled
|
||||
asynchronously on both sides and stops being reported exactly when a connection
|
||||
dies; replaying from it would re-send bytes the peer already has.
|
||||
* *Delivered*, not *credited*, sizes the window. Credit travels as deltas, and
|
||||
the grants in flight when the connection died are gone for good. A window
|
||||
derived from them is permanently short — and if a full window was outstanding
|
||||
at the drop, permanently zero, which deadlocks: nothing can be sent, so no
|
||||
credit can ever come back.
|
||||
|
||||
`RESUME_ACK` carries a freshly minted `NewCID`, which replaces the old one. A CID
|
||||
therefore stays single-use even though a stream may be reattached many times, so
|
||||
a leaked CID grants no more than the outage in which it was observed.
|
||||
|
||||
Resumption is **hub-instance-affine**: a CID means nothing to a second hub behind
|
||||
an L4 load balancer, which answers `RST(UNKNOWN_STREAM)` and lets the client tear
|
||||
down at once. Because the grace period holds player sockets and their buffers, a
|
||||
hub bounds the number of hung streams and the bytes they retain, dropping the
|
||||
oldest first when either cap is reached.
|
||||
|
||||
## 8. HAProxy protocol v2 (optional)
|
||||
|
||||
When a mapping has `proxyProtocol: true`, the client prepends a PROXY v2 header
|
||||
@@ -415,7 +537,12 @@ big-endian.
|
||||
"timestampWindowMs": 30000,
|
||||
"pendingTimeoutMs": 10000,
|
||||
"streamWindowBytes": 262144,
|
||||
"sessionIdleTimeoutMs": 90000
|
||||
"sessionIdleTimeoutMs": 90000,
|
||||
"streamResume": true,
|
||||
"resumeGraceMs": 20000,
|
||||
"maxParkedStreams": 256,
|
||||
"statsIntervalMs": 0,
|
||||
"registrationGraceMs": 15000
|
||||
}
|
||||
```
|
||||
|
||||
@@ -426,6 +553,27 @@ the hub's advertised per-stream receive window (§7.3).
|
||||
session or worker conn that has gone silent for that long (§7.4). It must stay
|
||||
comfortably above the client's `pingIntervalMs`; `0` disables the watchdog.
|
||||
|
||||
`streamResume` (optional, default true) offers STREAM_RESUME (§7.5). With it
|
||||
false the hub never echoes the flag and behaves exactly as a hub that predates
|
||||
the feature, allocating no retained regions.
|
||||
|
||||
`resumeGraceMs` (optional, default 20000) is how long a hung player is held, and
|
||||
is advertised in `SessionReady`. It must exceed the client's own grace by at
|
||||
least one dial, which is why the client clamps itself against the advertised
|
||||
value rather than its own configuration.
|
||||
|
||||
`maxParkedStreams` (optional, default 256) and `maxParkedBytes` (default
|
||||
`maxParkedStreams × 2 × streamWindowBytes`) bound what hung players may cost;
|
||||
past either the oldest are dropped.
|
||||
|
||||
`statsIntervalMs` (optional, default 0 = off) logs a periodic line with live and
|
||||
parked stream counts, retained bytes, and the pattern count.
|
||||
|
||||
`registrationGraceMs` (optional, default 15000) is how long a closed control
|
||||
session's routes are kept as **orphaned** rather than deleted (§5.2). `0`
|
||||
disables it, restoring the behaviour of dropping routes the instant a session
|
||||
closes.
|
||||
|
||||
### 9.2 Client — JSON
|
||||
|
||||
```json
|
||||
@@ -435,6 +583,10 @@ comfortably above the client's `pingIntervalMs`; `0` disables the watchdog.
|
||||
"maxConn": 4,
|
||||
"pingIntervalMs": 20000,
|
||||
"streamWindowBytes": 262144,
|
||||
"maxBandwidth": "20mbps",
|
||||
"streamResume": true,
|
||||
"resumeGraceMs": 15000,
|
||||
"statsIntervalMs": 0,
|
||||
"mappings": [
|
||||
{ "pattern": "mc\\.example\\.com", "destination": "127.0.0.1:25566", "proxyProtocol": true }
|
||||
]
|
||||
@@ -444,6 +596,30 @@ comfortably above the client's `pingIntervalMs`; `0` disables the watchdog.
|
||||
`streamWindowBytes` (optional, default 262144, clamped to [32768, 8388608]) is
|
||||
the client's advertised per-stream receive window (§7.3).
|
||||
|
||||
`streamResume` (optional, default true) offers STREAM_RESUME (§7.5). With it
|
||||
false the client never offers the flag, never retains a byte for retransmission,
|
||||
and behaves exactly as a client that predates the feature.
|
||||
|
||||
`resumeGraceMs` (optional, default 15000, minimum 2000) is how long a hung stream
|
||||
keeps trying to reattach, clamped below the hub's advertised grace. The default
|
||||
is chosen against the *backend*, not the tunnel: a hung player stops answering
|
||||
the game server's KeepAlive, and vanilla disconnects a silent client at 30s, so a
|
||||
longer grace would only resume sessions the backend then kicks.
|
||||
|
||||
`statsIntervalMs` (optional, default 0 = off) logs a periodic diagnostics line
|
||||
and a per-stream summary at close, reporting how long each stream spent blocked
|
||||
on the flow-control window versus the bandwidth cap, and the heartbeat round-trip
|
||||
time per conn. These distinguish a slow backend from a saturated uplink from a
|
||||
bad path, which throughput alone cannot.
|
||||
|
||||
`maxBandwidth` (optional, default unlimited) caps the aggregate rate at which the
|
||||
client sends `DATA` to the hub, shared fairly across streams. Accepts `"20mbps"`
|
||||
(decimal bit units), `"2MB/s"` (binary byte units), or a bare number of bytes per
|
||||
second; the minimum is 8192 B/s. **This is a purely local policy and has no
|
||||
effect on the wire format** — a shaped client is indistinguishable from a slow
|
||||
one, and the hub needs no support for it. Only `DATA` is paced; control frames
|
||||
are never delayed.
|
||||
|
||||
Each `pattern` is a regular expression (§5.1) matched against the whole
|
||||
normalized player hostname, case-insensitively. Escape literal dots (`mc\.example\.com`,
|
||||
which is `mc\\.example\\.com` in JSON); an unescaped `.` matches any character.
|
||||
@@ -483,6 +659,9 @@ not appear on the tunnel wire, and the exchange is invisible to the player.
|
||||
| heartbeat timeout | `3 × pingIntervalMs` |
|
||||
| hub session idle timeout | 90000 ms (`0` disables) |
|
||||
| stream window default / bounds | 256 KiB, clamped to [32 KiB, 8 MiB] |
|
||||
| feature flag: stream resumption | 0x04 (negotiated) |
|
||||
| mux RESUME / RESUME_ACK | 0x07 / 0x08 |
|
||||
| resume grace: hub / client default | 20000 ms / 15000 ms (hub value advertised) |
|
||||
| retained region per stream per direction | bounded by the stream window |
|
||||
| WND grant batching (reference) | one grant per window/2 consumed |
|
||||
| DATA chunk cap (reference) | 32 KiB |
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user