Files
redapricot/PROTOCOL.md
T
iceBear67 da17140583 fix: harden resume/shutdown paths, tighten Intent-18 and PSK handshake handling
Client (Go) — resume correctness
- C1: completeResume now re-parks the stream when replay fails mid-conn-loss.
  parked was cleared before the replay loop, so the dying conn's teardown
  would start a second resumeLoop and the two loops could strand the stream
  with neither alive. Resume stats are counted only after the replay lands.
- C2: RST(ALREADY_BOUND) is retryable instead of terminating the loop. With
  C1 fixed there is never a genuine second attempt, so "already bound" means
  the hub still holds the stream on a half-open conn; the retry waits out
  that bind (bounded by the grace deadline, teardown on expiry) instead of
  returning and leaving the destination socket hung forever.

Client (Go) — shutdown semantics
- C3: Close() sets a closing flag and cancels an internal context; dialSession
  takes a ctx (DialContext + AfterFunc so shutdown aborts in-flight
  handshakes); the worker pool refuses new conns after closeAll (Allocate,
  background growth, cond waiters); serveControl's reconnect loop is gated by
  closing so Close works even when the caller's Start context is not
  cancelled; conn-loss teardown closes streams outright during shutdown
  instead of parking them for a reattach that will never come.

Client (Go) — hygiene
- C4: pingInterval() clamps at the single point a duration is derived, so a
  hand-built Config with PingIntervalMs <= 0 can no longer panic
  time.NewTicker (added DefaultPingIntervalMs).
- E6: shaperStall is sampled right after shaper.Acquire, before the socket
  write, so a hub that is not reading is no longer charged to the bandwidth
  cap in the stats.
- E7: stream log lines now carry conn%d/sid%d (leg.String()), making streams
  traceable across reattaches.
- P5: mirror constants IntentReserved/RegisterOk/RegisterErrPattern added;
  RegisterAck dispatch logs rejection reasons via the named codes.

Hub (Java) + PROTOCOL.md
- P3: Intent 18 replies with a Minecraft status-response packet
  ([Len: VarInt][0x00][JSON: String]) and closes (socket.end, so the write
  always lands) instead of closing silently; documented in PROTOCOL.md §2.
- P4: PSK address check is strict equality with the lowercase hex address;
  an uppercase/case-folded variant is now rejected per PROTOCOL.md §2.
- P7: PROTOCOL.md §5 SessionReady row lists its real fields
  (Flags/RecvWindow/ResumeGraceMs) instead of "(none)".

Verified: go vet, go test -race ./client/..., gradle test, full e2e suite
(twice), resume e2e 3x, plus live probes of the hub with the real client
codec (Intent-18 status reply, strict-lowercase PSK acceptance/rejection).%
2026-08-15 17:47:39 +08:00

681 lines
32 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# redapricot (红杏) wire protocol
redapricot is a central-hub P2P tunnel that speaks (an extension of) the
Minecraft Java Edition protocol. This document is the normative wire spec that
the Java **server** (the *hub*) and the Go **client** both implement. It is
self-contained: everything needed to write an interoperable implementation is
here.
There are three roles:
| Role | Language | Description |
|------------|----------|-------------|
| **Server** | Java | The central hub. Accepts every inbound TCP connection (players *and* clients) on one port. |
| **Client** | Go | Registers routing patterns with the hub and forwards player traffic to real destinations. |
| **Player** | any | An ordinary Minecraft client connecting through the hub. |
```
Player ──MC──▶ Hub(server) ══WorkerConn(mux)══▶ Client ──MC──▶ Destination
▲ registers patterns / receives control requests │
└────────────── Control Session ────────────────────┘
```
Security is intentionally lightweight: the goal is connectivity, not
confidentiality against a determined attacker. The single shared secret is the
**PSK** (pre-shared key), a UTF-8 passphrase configured on the hub and every
client.
---
## 1. Primitive data types
These follow the Minecraft protocol exactly.
| Type | Encoding |
|-----------|----------|
| `VarInt` | LEB128, 7 data bits per byte, high bit = continuation, little-endian groups, two's-complement, max 5 bytes. |
| `String` | `VarInt` byte-length of the UTF-8 encoding, followed by the UTF-8 bytes. |
| `U16` | unsigned 16-bit, **big-endian**. |
| `I64` | signed 64-bit, **big-endian**. |
| `Bytes[N]`| exactly N raw bytes, no length prefix. |
| `u8` | single unsigned byte. |
## 2. Minecraft packet framing (plaintext)
Every connection begins as an ordinary Minecraft connection. An uncompressed
Minecraft packet is:
```
[Length: VarInt][PacketID: VarInt][Data...] Length = len(PacketID)+len(Data)
```
redapricot **never** enables Minecraft compression on the hub link. Player
traffic that is compressed end-to-end (negotiated between the player and the
real destination) is irrelevant — the hub forwards raw bytes and never inspects
anything past the Handshake.
### 2.1 Handshake
The first packet on every connection is the Handshake (packet id `0x00`,
Handshaking state):
```
ProtocolVersion : VarInt
ServerAddress : String (≤ 255)
ServerPort : U16
Intent : VarInt
```
The hub reads exactly one Handshake packet and dispatches on `Intent`:
| Intent | Meaning |
|---------------|---------|
| `17` | redapricot session establishment (control session *or* worker conn). |
| `18` | Reserved for redapricot management/status. Never matched against patterns. The reference hub replies with a status line and closes. |
| anything else | **Player** connection. `ServerAddress` is matched against the registered **regex** PATTERNs (§5.1). |
For `Intent == 18` the hub replies with a Minecraft status-response packet and
closes — the same shape a player receives for a status query (Intent 1), so the
port can be probed with ordinary tooling:
```
[Len: VarInt][PacketID 0x00][JSON: String]
```
The JSON is a minimal status payload, e.g.
`{"description":{"text":"redapricot hub"},"version":{"name":"redapricot","protocol":767},"players":{"max":0,"online":0}}`.
It is the one reply the hub sends in plaintext: Intent 18 never negotiates
encryption or any other redapricot state.
For `Intent == 17` the hub additionally requires
`ServerAddress == lowercase_hex(SHA3-224(PSK))` — a 56-character hex string.
This is the first (cheap) proof that the peer knows the PSK. A mismatch closes
the connection.
For player connections the hub normalizes `ServerAddress` before matching:
lower-cased, and any trailing `.` or Forge/FML `\0`-suffix (`host\0FML\0`)
stripped to the bare hostname. The resulting hostname is then tested against the
registered regex patterns (§5.1).
## 3. Encryption
Immediately **after** the `Intent == 17` Handshake, the connection switches to
an encrypted, self-delimiting frame stream. redapricot uses **ChaCha20**
(RFC 8439, 32-bit block counter, 96-bit nonce) as a raw stream cipher applied to
frame payloads (no Poly1305 tag — padding/space overhead is minimized, matching
the design goal).
Each direction is an independent ChaCha20 keystream. Keys are derived from a
"phase key" `PK` (raw bytes) as:
```
keyC2S = SHA3-256(PK ‖ 0x01) # client → server
keyS2C = SHA3-256(PK ‖ 0x02) # server → client
nonce = 0x00 × 12 # both directions
counter starts at 0 # both directions
```
Using distinct keys per direction avoids a two-time-pad while keeping the nonce
trivially fixed. Each side keeps two ChaCha20 instances (one encrypt, one
decrypt) and feeds bytes through them incrementally; the keystream position is
maintained across writes.
There are two phases:
* **Phase A** — `PK = PSK` (the configured passphrase, UTF-8 bytes).
* **Phase B** — `PK = REKEY` (see §4), used for the remainder of the connection.
### 3.1 Encrypted frames
Once encryption is on, the connection speaks length-prefixed **frames**:
```
[Length: VarInt] # PLAINTEXT (not encrypted)
[Payload: Bytes[Length]] # ciphertext (ChaCha20)
```
Only the payload is encrypted; the `Length` prefix is sent in the clear. The
cipher is a continuous per-direction keystream: each frame's payload advances
the keystream by exactly `Length` bytes (the length prefix consumes no
keystream). This keeps framing trivial — a reader reads a plaintext VarInt, then
decrypts exactly that many following bytes as one unit — and lets the cipher
phase switch (§4) happen cleanly on a frame boundary without ever decrypting a
later frame's bytes with the wrong key. Max payload length is `1 MiB`; larger
closes the connection.
## 4. Session establishment (Intent 17)
The first frame is sent by the peer that opened the connection (client → server)
and is encrypted with **Phase A**. Its payload is the **Rekey** message:
```
Magic : u8 # 0x01 = control session, 0x02 = worker conn
RandLen : VarInt # 8 ≤ RandLen ≤ 64
Rand : Bytes[RandLen] # cryptographically random
Timestamp : I64 # client's epoch milliseconds
Flags : VarInt # feature flags; bit 0x01 (STREAM_FC) MUST be set
RecvWindow: VarInt # client's per-stream receive window, bytes (§7.3)
```
`Flags` is a bitfield of features.
| Bit | Name | Meaning |
|-----|------|---------|
| `0x01` | STREAM_FC | **Per-stream flow control** (§7.3). Mandatory. |
| `0x02` | WORKER_HEARTBEAT | Mux-level `PING`/`PONG` on worker conns (§7.4). Optional. |
| `0x04` | STREAM_RESUME | **Stream resumption** (§7.5): a worker-conn drop hangs the player rather than closing it. Optional. |
STREAM_FC is mandatory: `RecvWindow` advertises the client's per-stream receive
window in bytes and must be positive. The hub closes the connection if the flag
is missing, `RecvWindow` is absent or non-positive, or the fields are malformed.
Optional bits are **negotiated**: the hub echoes in `SessionReady` only those it
accepts, and the client enables a feature only when its bit comes back. A hub
that does not know WORKER_HEARTBEAT simply omits the bit and the client falls
back to TCP keepalive alone.
The hub:
1. Decrypts frame 1 with Phase A.
2. Rejects (closes) if `|now Timestamp| > timestampWindowMs` (default 30000),
or if `RandLen` is out of range.
3. Computes `REKEY = Rand ‖ Timestamp` (the 8 timestamp bytes big-endian
appended to Rand — the `Magic` byte is **not** included).
4. Switches **both** its ciphers to Phase B keys derived from `REKEY`.
The client, after sending frame 1 with Phase A, likewise switches both its
ciphers to Phase B. In practice **only frame 1 uses Phase A**; every later
frame (both directions) is Phase B, counters reset to 0.
The hub then sends one Phase-B frame to confirm success:
```
SessionReady : payload = [ 0x00, Flags: VarInt, RecvWindow: VarInt,
ResumeGraceMs: VarInt ] # only when STREAM_RESUME is set
```
The hub echoes the accepted flags (STREAM_FC set) followed by its own
per-stream receive window. A client must reject a SessionReady without the
STREAM_FC flag or without a positive window (an unsupported hub).
`ResumeGraceMs` is present only when the hub accepts STREAM_RESUME, and states
how long it will hang a player waiting for that player's stream to be reattached
(§7.5). The client clamps its own retry budget below this value. Advertising it
rather than assuming matching configuration is deliberate: the client must always
give up first, and if the hub instead dropped a hung player while the client was
still reattaching, the failure would be a silent hang rather than an error. A hub
that sets the flag but omits the field is treated as not supporting resumption.
A hub that rejects the session simply closes the TCP connection (optionally
after a Phase-B `Error` frame, §6). After `SessionReady`:
* `Magic == 0x01` → the connection is a **Control Session** (§5).
* `Magic == 0x02` → the connection is a **Worker Conn** (§7).
## 5. Control session messages
After `SessionReady`, a control session exchanges **control messages**, one per
encrypted frame. Frame payload:
```
Type : u8
... : type-specific fields
```
| Type | Name | Direction | Fields |
|--------|----------------|-----------|--------|
| `0x00` | SessionReady | S → C | `Flags: VarInt`, `RecvWindow: VarInt`, `ResumeGraceMs: VarInt` (only when the hub accepted `STREAM_RESUME`) — the confirmation frame from §4 |
| `0x01` | Register | C → S | `Pattern: String` |
| `0x02` | Unregister | C → S | `Pattern: String` |
| `0x03` | RegisterAck | S → C | `Pattern: String`, `Status: u8` (0 = ok, 1 = invalid pattern) |
| `0x04` | ControlRequest | S → C | `CID: Bytes[16]`, `Pattern: String`, `PlayerIP: String`, `PlayerPort: U16` |
| `0x05` | Ping | C → S | `Nonce: I64` |
| `0x06` | Pong | S → C | `Nonce: I64` |
* **Register / Unregister**: the client may (un)register a PATTERN at any time.
A PATTERN is a **regular expression** (§5.1) and is stored **verbatim** — the
exact string is the registry key (never normalized, so regex metacharacters are
preserved). Re-registering the identical pattern string reassigns it to the
newest session (last writer wins); `Unregister` only removes it if the
requesting session still owns it.
* **RegisterAck**: acknowledges a `Register`. `Status` is `0` on success, or `1`
if the pattern is not a valid regular expression (in which case nothing is
registered). The `Pattern` echoes the string that was registered.
* **ControlRequest**: emitted by the hub when a player Handshake matches a
PATTERN this session registered. `Pattern` is the **registered pattern string
that matched** (echoed verbatim), *not* the player's hostname — so the client
can look the pattern up in its own route table. `CID` is 16
cryptographically-random bytes generated by the hub, unique to that pending
player. `PlayerIP`/`PlayerPort` are the player's source address (used for
HAProxy v2).
* **Ping/Pong**: optional keepalive so idle control sessions survive NAT
timeouts. The client pings periodically; the hub echoes the nonce.
### 5.1 Pattern matching
A registered PATTERN is a **regular expression** (the reference hub uses
`java.util.regex`). Matching is:
* **Case-insensitive** — patterns are compiled with a case-insensitive flag, and
the player hostname is lower-cased during normalization (§2.1).
* **Whole-string (anchored)** — the pattern must match the *entire* normalized
hostname, as if wrapped in `^…$`. `mc\.example\.com` matches `mc.example.com`
but not `mc.example.com.evil` or `sub.mc.example.com`.
* **First match wins** — the hostname is tested against every registered pattern;
the first that matches routes the player. If several patterns overlap, which
one wins is unspecified.
Because the pattern is a regex, a literal dot must be escaped (`mc\.example\.com`);
an unescaped `.` is the regex "any character" wildcard. A pattern that fails to
compile is rejected at `Register` time with `RegisterAck` status `1`.
### 5.2 Orphaned routes (control-session outage)
When a control session closes, its registrations are **not** deleted straight
away. They are marked *orphaned* and kept for `registrationGraceMs`.
This costs nothing on the wire — it is entirely hub-side behaviour — but it
closes a gap that is otherwise very visible. A client whose control session dies
reconnects with backoff, and until it re-registers the hub has no route for it,
so every player arriving in that window is told there is no such server. The
players already tunneled are unaffected, since they ride worker conns, which a
control-session close never touches.
While a route is orphaned:
* a player matching it is **held** — paused, with its handshake buffered exactly
as for a normal pending player — and no `ControlRequest` is sent, because there
is no session to send it to;
* a player that was already pending when the session closed is moved into the
same held state rather than being dropped;
* when any client registers that pattern again, the hub delivers the
`ControlRequest` it never sent and the player proceeds normally. The held
player's deadline switches from the registration grace to `pendingTimeoutMs`
at that point, since it is now waiting for a worker rather than for a route.
If the grace expires with no client having re-registered, the route and every
player held on it are dropped. `registrationGraceMs: 0` disables the mechanism
and restores the immediate-drop behaviour.
Note the hub cannot distinguish "this client is reconnecting" from "this client
is gone for good" — that is what the grace period is a bet on. It is bounded on
the client side too: the reference client retries immediately on a control-session
drop and caps its backoff at 10s, so the bet is usually settled in well under a
second.
## 6. Error frame (any redapricot connection)
At any time either side may send, then close:
```
Type : u8 = 0x7F
Msg : String
```
Purely informational; the receiver logs it.
## 7. Worker conn & multiplexing
A **Worker Conn** (`Magic == 0x02`) carries player↔destination traffic for many
players over one TCP connection using a minimal stream multiplexer. The unit of
work is a **stream**. Stream ids are assigned by the **client** (the only side
that opens streams), unique per worker conn, starting at 1 and increasing.
Each encrypted frame on a worker conn carries one **mux frame**:
```
FrameType : u8
StreamID : VarInt
Data : Bytes[...] # remainder of the frame payload
```
| FrameType | Name | Direction | Data |
|-----------|------|-----------|------|
| `0x00` | SYN | C → S | `CID: Bytes[16]` — open a stream to take over the pending player identified by CID. |
| `0x01` | DATA | both | raw tunneled bytes for the stream. |
| `0x02` | FIN | both | *(empty)* — graceful close of the stream (both directions). This is the "disconnect" the hub sends when the player leaves. |
| `0x03` | RST | both | *(optional 1 byte reason)* — abnormal close (e.g. CID unknown/expired, destination dial failed). |
| `0x04` | WND | both | `Delta: VarInt` — flow-control credit grant (§7.3). |
| `0x05` | PING | both | `Nonce: I64` — liveness probe on reserved StreamID `0` (§7.4). |
| `0x06` | PONG | both | `Nonce: I64` — echoes the probe's nonce (§7.4). |
| `0x07` | RESUME | C → S | `CID: Bytes[16]`, `Accepted: I64`, `Delivered: I64` — reattach a hung stream to this conn (§7.5). |
| `0x08` | RESUME_ACK | S → C | `Accepted: I64`, `Delivered: I64`, `NewCID: Bytes[16]` — the reattach succeeded (§7.5). |
There is no explicit SYN-ACK: success is implied by the hub forwarding the
buffered Handshake as the stream's first `DATA`; failure is an `RST`.
`RST` reason codes. The byte remains optional — a peer that predates it sends
none, and a receiver must tolerate its absence — but distinguishing the reasons
matters for resumption, where "this stream is gone" and "someone else already
took it" call for opposite responses.
| Code | Name | Meaning |
|------|------|---------|
| `0x00` | UNSPECIFIED | No reason given (also the meaning of an absent byte). |
| `0x01` | UNKNOWN_STREAM | CID unknown or expired, or the hub restarted. Terminal: stop retrying. |
| `0x02` | ALREADY_BOUND | Another reattach already claimed this stream. Do **not** tear down. |
| `0x03` | RESUME_ABANDONED | The peer gave up reattaching. |
| `0x04` | FLOW_CONTROL | The peer exceeded its advertised window. |
| `0x05` | DIAL_FAILED | The client could not reach the destination. |
### 7.1 Stream allocation (client side)
The client keeps a pool of `1 ≤ N ≤ max_conn` worker conns (`max_conn`
configurable, `1..8`). The pool grows **breadth-first**: spreading streams over
several connections keeps any single TCP connection from becoming the shared
point of failure for every player on the tunnel. To place a new stream:
1. Pick the worker conn with the **fewest active streams**, and use it.
2. If that conn already carries at least one stream and
`poolSize + dialsInFlight < max_conn`, dial another worker conn **in the
background**. The stream just placed is not delayed by that dial; the new
conn becomes the least-loaded one and picks up subsequent streams.
3. Once the pool is at `max_conn`, streams stack on the least-loaded conn.
Exceeding `8` active streams there is logged as pool saturation.
Only when the pool is *empty* does a caller dial synchronously, and then exactly
one caller dials while the others wait for its result. A dial is never performed
while holding the pool lock: session establishment is network I/O, and one
unresponsive hub must not be able to block unrelated players.
### 7.2 End-to-end player flow
1. Player connects to the hub and sends a Handshake with a matching
`ServerAddress` and `Intent ∉ {17,18}`.
2. Hub normalizes the address, finds the registering control session, generates
`CID`, **pauses** the player socket, buffers everything read so far (the raw
Handshake plus any pipelined bytes), and sends `ControlRequest` on the
control session. If no SYN arrives within `pendingTimeoutMs` (default 10000)
the pending entry is dropped and the player socket closed.
3. The client receives `ControlRequest`, looks up the destination for `Pattern`,
allocates a worker conn + `StreamID`, and sends `SYN(StreamID, CID)`. In
parallel it dials the destination and (if configured) writes a HAProxy v2
header (§8) carrying `PlayerIP:PlayerPort`.
4. The hub matches `CID` to the pending player, binds
`(workerConn, StreamID) ↔ playerSocket`, forwards the buffered bytes as
`DATA`, and resumes the player socket. Subsequent player bytes become `DATA`
frames; `DATA` frames from the client are written to the player socket. If
`CID` is unknown/expired the hub replies `RST`.
5. When the player disconnects the hub sends `FIN` on the stream; the client
closes the destination. When the destination closes, the client sends `FIN`;
the hub closes the player socket. `RST` is treated the same way (hard close).
Data on a worker conn is subject to that TCP connection's back-pressure for
its **aggregate** bandwidth; *per-stream* fairness is governed by the credit
windows of §7.3.
### 7.3 Per-stream flow control
Every stream carries an independent credit window per direction:
* Each side advertised its **receive window** W (bytes) at session setup. A
sender may have at most W un-credited DATA bytes outstanding per stream; the
initial budget is W, spent as DATA is sent (`Data` length only — SYN/FIN/RST
frames are free) starting with the very first DATA on the stream (including
the hub's forwarded handshake).
* The receiver returns credit with `WND(Delta)` once bytes are **delivered to
the terminal socket** (written to the player / destination connection), not
when they are merely buffered. Receivers should batch grants (the reference
implementations send one `WND` per W/2 bytes consumed).
* A sender whose window is exhausted pauses reading **that stream's source
socket only**; the shared worker conn is never paused because of a single
stream. A receiver that observes more than W un-credited bytes on a stream
may reset it (`RST`) as a protocol violation.
* Senders should also cap individual DATA payloads (the reference
implementations use 32 KiB) so one stream cannot monopolize the link for a
full 1-MiB frame.
Both windows may differ (each side enforces the one its peer advertised).
`Delta` must be positive; a `WND` for an unknown stream id is ignored.
### 7.4 Liveness
TCP alone cannot tell a healthy idle connection from a dead one. When a stateful
middlebox forgets an established flow — conntrack expiry, a firewall reload, a
cloud load balancer's idle timeout — it sends neither `FIN` nor `RST`. Both ends
keep a socket that will never again carry a byte, and a reader parked on it waits
forever. Without an application-level probe the client cannot notice: its worker
conn stays in the pool, the hub keeps routing players to a control session nobody
reads, and service does not return until the client process is restarted.
Every established session is therefore covered by a heartbeat:
* **Control session** — the client sends `Ping` every `pingIntervalMs` and the
hub answers `Pong`. If no `Pong` arrives for `3 × pingIntervalMs`, the client
closes the session, which triggers its normal reconnect with backoff.
* **Worker conns** — when WORKER_HEARTBEAT was negotiated, the same exchange
runs as mux `PING`/`PONG` frames on the reserved StreamID `0`. On timeout the
client closes the conn and drops it from the pool. Its streams are reset,
unless STREAM_RESUME was negotiated, in which case they are hung and reattached
over a fresh conn instead (§7.5).
* **Hub side** — an established redapricot session that receives no frame for
`sessionIdleTimeoutMs` (default 90000, `0` disables) is closed. Player
connections are never subject to this.
Both ends also enable TCP keepalive, which catches the narrower case of a peer
that has become unreachable at the IP layer.
Session establishment (§4) is bounded by a single deadline covering the dial,
the `Rekey` write and the `SessionReady` read, and every frame write is bounded
too — a peer that stops reading must not be able to park a whole multiplexed
connection inside one write.
### 7.5 Stream resumption (STREAM_RESUME)
A worker conn is only the middle leg of every stream it carries. When it dies,
both terminal sockets — the player's and the destination's — are usually still
healthy, so resetting the streams discards working connections because a
replaceable transport failed. One conntrack expiry disconnects every player on
that conn.
With STREAM_RESUME negotiated, a worker-conn drop instead **hangs** each stream:
* the hub pauses the player socket, keeps its state, and holds it for
`ResumeGraceMs` from the moment of the *first* hang (an absolute deadline, so a
flapping client cannot extend it indefinitely);
* the client keeps the destination socket open and reattaches the stream over a
fresh worker conn by sending `RESUME` with the stream's CID;
* the hub answers `RESUME_ACK`, or `RST(UNKNOWN_STREAM)` if it holds no such
stream — which is also what a client gets from a hub that has restarted.
**Resumption is byte-exact, and must be.** Frames handed to a dying socket are
lost with no notification, and the frame cipher cannot be resynchronized, so each
side replays whatever the other did not receive. Splicing the stream even one
byte off corrupts the tunneled protocol.
Three offsets are tracked per direction, and they are not interchangeable:
| Offset | Meaning | Used for |
|--------|---------|----------|
| `Sent` | bytes handed to the wire | the end of the retained region |
| `Accepted` | bytes taken off the wire toward the terminal socket | **where to replay from** |
| `Delivered` | bytes actually written to the terminal socket | **how to restate the window** |
Each side retains the bytes between what the peer has credited and what it has
sent. This costs no new bound: flow control (§7.3) already caps outstanding bytes
at one window, so the retained region *is* the outstanding region.
On reattach both sides replay `[peer's Accepted, Sent)` and set
`SendWindow = W (Sent peer's Delivered)`, then discard their own pending
credit — the exchanged `Delivered` already carries everything those deltas would
have, so emitting both would grant the same bytes twice.
Two rules deserve emphasis, because the obvious simplifications are wrong:
* *Accepted*, not *Delivered*, is the replay point. Delivery is signalled
asynchronously on both sides and stops being reported exactly when a connection
dies; replaying from it would re-send bytes the peer already has.
* *Delivered*, not *credited*, sizes the window. Credit travels as deltas, and
the grants in flight when the connection died are gone for good. A window
derived from them is permanently short — and if a full window was outstanding
at the drop, permanently zero, which deadlocks: nothing can be sent, so no
credit can ever come back.
`RESUME_ACK` carries a freshly minted `NewCID`, which replaces the old one. A CID
therefore stays single-use even though a stream may be reattached many times, so
a leaked CID grants no more than the outage in which it was observed.
Resumption is **hub-instance-affine**: a CID means nothing to a second hub behind
an L4 load balancer, which answers `RST(UNKNOWN_STREAM)` and lets the client tear
down at once. Because the grace period holds player sockets and their buffers, a
hub bounds the number of hung streams and the bytes they retain, dropping the
oldest first when either cap is reached.
## 8. HAProxy protocol v2 (optional)
When a mapping has `proxyProtocol: true`, the client prepends a PROXY v2 header
to the destination connection *before* any tunneled bytes, so the real server
sees the player's true source address.
```
Signature : 0D 0A 0D 0A 00 0D 0A 51 55 49 54 0A (12 bytes)
VerCmd : 0x21 (v2, PROXY command)
FamProto : 0x11 (TCP/IPv4) | 0x21 (TCP/IPv6)
Len : U16 (length of the address block)
Addresses : IPv4 → srcAddr[4] dstAddr[4] srcPort[2] dstPort[2] (12 bytes)
IPv6 → srcAddr[16] dstAddr[16] srcPort[2] dstPort[2] (36 bytes)
```
`src` is the player; `dst` is the destination the client dialed. Ports are
big-endian.
## 9. Configuration
### 9.1 Hub (server) — JSON
```json
{
"listen": "0.0.0.0:25565",
"psk": "change-me",
"timestampWindowMs": 30000,
"pendingTimeoutMs": 10000,
"streamWindowBytes": 262144,
"sessionIdleTimeoutMs": 90000,
"streamResume": true,
"resumeGraceMs": 20000,
"maxParkedStreams": 256,
"statsIntervalMs": 0,
"registrationGraceMs": 15000
}
```
`streamWindowBytes` (optional, default 262144, clamped to [32768, 8388608]) is
the hub's advertised per-stream receive window (§7.3).
`sessionIdleTimeoutMs` (optional, default 90000) closes an established control
session or worker conn that has gone silent for that long (§7.4). It must stay
comfortably above the client's `pingIntervalMs`; `0` disables the watchdog.
`streamResume` (optional, default true) offers STREAM_RESUME (§7.5). With it
false the hub never echoes the flag and behaves exactly as a hub that predates
the feature, allocating no retained regions.
`resumeGraceMs` (optional, default 20000) is how long a hung player is held, and
is advertised in `SessionReady`. It must exceed the client's own grace by at
least one dial, which is why the client clamps itself against the advertised
value rather than its own configuration.
`maxParkedStreams` (optional, default 256) and `maxParkedBytes` (default
`maxParkedStreams × 2 × streamWindowBytes`) bound what hung players may cost;
past either the oldest are dropped.
`statsIntervalMs` (optional, default 0 = off) logs a periodic line with live and
parked stream counts, retained bytes, and the pattern count.
`registrationGraceMs` (optional, default 15000) is how long a closed control
session's routes are kept as **orphaned** rather than deleted (§5.2). `0`
disables it, restoring the behaviour of dropping routes the instant a session
closes.
### 9.2 Client — JSON
```json
{
"server": "127.0.0.1:25565",
"psk": "change-me",
"maxConn": 4,
"pingIntervalMs": 20000,
"streamWindowBytes": 262144,
"maxBandwidth": "20mbps",
"streamResume": true,
"resumeGraceMs": 15000,
"statsIntervalMs": 0,
"mappings": [
{ "pattern": "mc\\.example\\.com", "destination": "127.0.0.1:25566", "proxyProtocol": true }
]
}
```
`streamWindowBytes` (optional, default 262144, clamped to [32768, 8388608]) is
the client's advertised per-stream receive window (§7.3).
`streamResume` (optional, default true) offers STREAM_RESUME (§7.5). With it
false the client never offers the flag, never retains a byte for retransmission,
and behaves exactly as a client that predates the feature.
`resumeGraceMs` (optional, default 15000, minimum 2000) is how long a hung stream
keeps trying to reattach, clamped below the hub's advertised grace. The default
is chosen against the *backend*, not the tunnel: a hung player stops answering
the game server's KeepAlive, and vanilla disconnects a silent client at 30s, so a
longer grace would only resume sessions the backend then kicks.
`statsIntervalMs` (optional, default 0 = off) logs a periodic diagnostics line
and a per-stream summary at close, reporting how long each stream spent blocked
on the flow-control window versus the bandwidth cap, and the heartbeat round-trip
time per conn. These distinguish a slow backend from a saturated uplink from a
bad path, which throughput alone cannot.
`maxBandwidth` (optional, default unlimited) caps the aggregate rate at which the
client sends `DATA` to the hub, shared fairly across streams. Accepts `"20mbps"`
(decimal bit units), `"2MB/s"` (binary byte units), or a bare number of bytes per
second; the minimum is 8192 B/s. **This is a purely local policy and has no
effect on the wire format** — a shaped client is indistinguishable from a slow
one, and the hub needs no support for it. Only `DATA` is paced; control frames
are never delayed.
Each `pattern` is a regular expression (§5.1) matched against the whole
normalized player hostname, case-insensitively. Escape literal dots (`mc\.example\.com`,
which is `mc\\.example\\.com` in JSON); an unescaped `.` matches any character.
Use ordinary regex to route wildcards, e.g. `.*\.example\.com` for every
subdomain or `(alpha|beta)\.mc\.net` for a fixed set.
`velocitySecret` (optional, per mapping) makes the client speak Velocity
"modern forwarding" towards that destination: during the Minecraft login phase
it swallows the backend's `velocity:player_info` Login Plugin Request and
answers with an HMAC-SHA256-signed payload carrying the player's real IP,
username and UUID (the UUID claimed in Login Start, or the offline-mode UUID
for protocols that carry none). Set it to the backend's
`proxies.velocity.secret`. This is purely client↔destination behavior — it does
not appear on the tunnel wire, and the exchange is invisible to the player.
## 10. Constants summary
| Name | Value |
|------|-------|
| redapricot Handshake intent | `17` |
| reserved management intent | `18` |
| Handshake address for Intent 17 | `hex(SHA3-224(PSK))` |
| cipher | ChaCha20 (RFC 8439), 12-byte zero nonce, per-direction key, payload-only |
| frame length prefix | plaintext VarInt |
| key derivation | `SHA3-256(PK ‖ 0x01)` c→s, `SHA3-256(PK ‖ 0x02)` s→c |
| rekey material | `Rand ‖ Timestamp(I64 BE)` |
| Magic: control / worker | `0x01` / `0x02` |
| RegisterAck status: ok / invalid pattern | `0x00` / `0x01` |
| pattern matching | case-insensitive, whole-string regex; first match wins |
| CID length | 16 bytes |
| max frame payload | 1 MiB |
| pool growth | breadth-first: grow to `max_conn` before stacking |
| saturation threshold (logged) | active streams `> 8` at `max_conn` |
| max worker conns | `max_conn ∈ [1,8]` |
| feature flag: per-stream flow control | `0x01` (mandatory) |
| feature flag: worker heartbeat | `0x02` (negotiated) |
| heartbeat timeout | `3 × pingIntervalMs` |
| hub session idle timeout | 90000 ms (`0` disables) |
| stream window default / bounds | 256 KiB, clamped to [32 KiB, 8 MiB] |
| feature flag: stream resumption | 0x04 (negotiated) |
| mux RESUME / RESUME_ACK | 0x07 / 0x08 |
| resume grace: hub / client default | 20000 ms / 15000 ms (hub value advertised) |
| retained region per stream per direction | bounded by the stream window |
| WND grant batching (reference) | one grant per window/2 consumed |
| DATA chunk cap (reference) | 32 KiB |