221 lines
13 KiB
Markdown
221 lines
13 KiB
Markdown
# redapricot architecture
|
|
|
|
This document explains *how* redapricot is built and *why*. For the exact bytes
|
|
on the wire, read [PROTOCOL.md](../PROTOCOL.md).
|
|
|
|
## 1. Roles and topology
|
|
|
|
```
|
|
┌───────────────────────── public internet ─────────────────────────┐
|
|
│ │
|
|
┌──────────┐ MC handshake (Intent 2/…) ┌───────────────┐ │
|
|
│ Player │ ───────────────────────────────▶│ │ │
|
|
└──────────┘ raw Minecraft bytes │ Hub │ │
|
|
│ (Java/Vert.x)│ │
|
|
┌──────────┐ Intent 17, magic 0x01 │ │ │
|
|
│ Client │ ◀──────── control session ──────│ • pattern reg │ │
|
|
│ (Go) │ ────────────────────────────────│ • CID table │ │
|
|
│ │ Intent 17, magic 0x02 │ • mux demux │ │
|
|
│ │ ═════════ worker conns ═════════│ │ │
|
|
└──────────┘ multiplexed player streams └───────────────┘ │
|
|
│ │
|
|
▼ MC bytes (+ optional HAProxy v2) │
|
|
┌───────────────┐ │
|
|
│ Real MC server│ (behind NAT, next to the client) │
|
|
└───────────────┘ │
|
|
```
|
|
|
|
Everything reaches the hub on **one TCP port**. The hub distinguishes three
|
|
kinds of inbound connection purely from the first Minecraft **Handshake**:
|
|
|
|
| Handshake `Intent` | Handled as |
|
|
|--------------------|------------|
|
|
| `17` + magic `0x01` | a **control session** from a client |
|
|
| `17` + magic `0x02` | a **worker connection** from a client |
|
|
| `18` | reserved (management/status) — never treated as a player |
|
|
| anything else | a **player** to be pattern-matched and tunneled |
|
|
|
|
Because players use ordinary intents (`1` status, `2` login, `3` transfer),
|
|
**vanilla clients need no changes**.
|
|
|
|
## 2. Connection lifecycle
|
|
|
|
### 2.1 Client establishes a control session
|
|
|
|
```
|
|
Client Hub
|
|
│ TCP connect │
|
|
│─ Handshake(Intent=17, addr=hex(SHA3-224(PSK))) ─▶ verify addr == expected
|
|
│ │
|
|
│ (both derive Phase-A keys = ChaCha20(SHA3-256(PSK ‖ dir)))
|
|
│─ Frame#1 [magic=0x01, rand, ts] ──────▶ check |now-ts| ≤ window
|
|
│ (both switch to Phase-B keys = ChaCha20(SHA3-256(rand‖ts ‖ dir)))
|
|
│◀──────────── Frame [SessionReady] ─────│
|
|
│─ Register("mc\.example\.com") ────────▶ patterns["mc\.example\.com"] = (regex, session)
|
|
│◀──────────── RegisterAck ──────────────│
|
|
│ ... periodic Ping/Pong ... │
|
|
```
|
|
|
|
Only frame #1 is encrypted with the PSK-derived key; a fresh random `rand‖ts`
|
|
becomes the per-connection key for everything after, so two connections never
|
|
share a keystream beyond that first frame.
|
|
|
|
### 2.2 A player arrives and is tunneled
|
|
|
|
```
|
|
Player Hub Client Destination
|
|
│─ Handshake(addr="mc.example.com", Intent=2)─▶ normalize + regex-match
|
|
│ │ pause player socket,
|
|
│ │ buffer bytes, mint CID
|
|
│ │─ ControlRequest(CID, pattern, ip:port) ─▶
|
|
│ │ allocate worker+stream
|
|
│ │◀──────── SYN(streamId, CID) ────────────│
|
|
│ │ takePending(CID) → bind dial destination,
|
|
│ │ forward buffered bytes write HAProxy v2 hdr
|
|
│ │─ DATA(streamId, handshake…) ───▶ ── handshake ──▶│
|
|
│ resume ─────────────────────────────│ bridge stream ⇄ dest
|
|
│══════════════ player bytes ══ DATA ══▶│════ DATA ═══▶ dest.write │
|
|
│◀═══════════ dest bytes ═══ DATA ══════│◀═══ DATA ════ dest.read │
|
|
│ player closes ──────────────────────│─ FIN(streamId) ────────▶ close dest │
|
|
```
|
|
|
|
Key points:
|
|
|
|
* **Patterns are regexes.** Each registered pattern is a case-insensitive
|
|
regular expression, matched against the *whole* normalized hostname (anchored,
|
|
first match wins). The hub echoes the **matched pattern string** — not the
|
|
player's hostname — in `ControlRequest`, so the client can look it straight up
|
|
in its own route table. Invalid patterns are rejected at registration with a
|
|
non-zero `RegisterAck` status.
|
|
* The hub **pauses** the player socket the instant it matches, so no player
|
|
bytes are lost while the takeover is arranged; the buffered handshake is
|
|
forwarded **verbatim**, so the real server sees exactly what the player sent
|
|
(including the original hostname — used for virtual-host routing there).
|
|
* **CID** is 16 random bytes minted by the hub and delivered only over the
|
|
encrypted control session, so only the intended client learns it. Any worker
|
|
connection presenting the correct CID is allowed to take over — that secrecy
|
|
is what binds a worker stream to the right pending player without any explicit
|
|
client identity.
|
|
* Disconnects are symmetric: player-close → hub sends `FIN` → client closes the
|
|
destination; destination-close → client sends `FIN` → hub closes the player.
|
|
|
|
## 3. Multiplexing (worker connections)
|
|
|
|
A worker connection is one encrypted TCP link carrying many **streams**. The
|
|
frame is intentionally tiny (PROTOCOL.md §7):
|
|
|
|
```
|
|
[plaintext VarInt length][ FrameType u8 | StreamID VarInt | Data… ] (payload encrypted)
|
|
```
|
|
|
|
Only the client opens streams (`SYN`), so stream-id allocation is a simple
|
|
per-connection counter with no coordination.
|
|
|
|
### 3.1 Pool & allocation
|
|
|
|
The client keeps 1…`maxConn` worker connections and places each new stream on
|
|
the **least-loaded** one. It opens an additional connection only when the
|
|
least-loaded connection is *saturated* (more than 8 active streams) and the pool
|
|
is below `maxConn`:
|
|
|
|
```
|
|
pick least-loaded conn
|
|
if leastLoaded.streams > 8 and pool.size < maxConn:
|
|
dial a new worker conn and use it
|
|
else:
|
|
use leastLoaded
|
|
```
|
|
|
|
The e2e test `TestConcurrentStreamsUseMultipleConns` drives 20 simultaneous
|
|
streams with `maxConn=4` and observes them deterministically spread over 3
|
|
connections (9 + 9 + 2), confirming the algorithm.
|
|
|
|
## 4. Encryption
|
|
|
|
* **Cipher:** ChaCha20 (RFC 8439) as a raw stream cipher over frame *payloads*.
|
|
The length prefix is plaintext, which makes the cipher **phase switch** at
|
|
rekey trivial (a reader always knows exactly how many ciphertext bytes belong
|
|
to the current frame and never decrypts the next frame with the wrong key).
|
|
* **Keys:** `SHA3-256(phaseKey ‖ 0x01)` for client→server and
|
|
`SHA3-256(phaseKey ‖ 0x02)` for server→client. Distinct per-direction keys
|
|
with a fixed zero nonce avoid a two-time pad without nonce management.
|
|
* **Interop:** Java's JCE `ChaCha20` and Go's `x/crypto/chacha20` produce byte-
|
|
identical keystreams (including across partial-block, arbitrarily-split
|
|
writes), and both `crypto/sha3` implementations agree — verified directly and
|
|
pinned by unit tests on both sides against a shared SHA3-224 vector.
|
|
|
|
## 5. Threading model
|
|
|
|
* **Hub:** a single Vert.x verticle instance. All accepted connections are
|
|
handled on that verticle's one event loop, so the pattern registry, CID table,
|
|
and per-connection state are touched by a single thread — no locks on the hot
|
|
path (concurrent maps are used only defensively). Every socket operation is
|
|
non-blocking; crypto is CPU-cheap. This trades multi-core scaling for
|
|
simplicity and correctness.
|
|
* **Client:** goroutine-per-concern. One goroutine reads each connection
|
|
(control or worker); `WriteFrame` is mutex-serialized so many stream goroutines
|
|
can share a worker connection safely. Each stream has two goroutines: `run`
|
|
pumps destination → hub, and `writeLoop` is the only writer to the
|
|
destination, draining a per-stream queue fed by the worker readLoop. The
|
|
readLoop itself never writes to a destination, so a stalled destination can
|
|
never block frame dispatch for other streams.
|
|
|
|
## 6. Back-pressure & flow control
|
|
|
|
Two mechanisms operate at different granularities:
|
|
|
|
* **Per-stream credit windows** (PROTOCOL.md §7.3; the windows are exchanged
|
|
at session establishment): each stream direction has an independent byte
|
|
budget equal to the receiver's advertised window (default 256 KiB). A sender
|
|
that exhausts a
|
|
stream's window pauses *only that stream's source* — the hub pauses the one
|
|
player socket, the client parks the one destination-reader goroutine. Credit
|
|
is granted back (`WND` frames, batched at half-window) as bytes are actually
|
|
written to the terminal socket. The result: a slow player or slow destination
|
|
jams its own stream at a bounded buffer size and nothing else. This is what
|
|
eliminates head-of-line blocking between streams.
|
|
* **Aggregate TCP back-pressure** on each worker connection: when the shared
|
|
socket itself is congested (total bandwidth, not one stream), the hub parks
|
|
all sending players until it drains, and the client's `WriteFrame` blocks.
|
|
This is fair — when the pipe is genuinely full, everyone should slow down.
|
|
|
|
The window also bounds memory: a stream can hold at most one window of
|
|
undelivered data per direction (the client's pre-connect handshake buffer is
|
|
covered by the same bound).
|
|
|
|
Per-stream flow control is mandatory: the hub rejects a session whose Rekey
|
|
lacks the STREAM_FC flag, and the client rejects a hub that does not echo it —
|
|
peers that predate the mechanism cannot connect at all.
|
|
|
|
What remains (by design) is TCP-level head-of-line blocking: a lost packet on
|
|
a worker connection stalls all its streams for one retransmit. That is inherent
|
|
to mux-over-TCP; the connection pool is the mitigation, and a datagram
|
|
transport (QUIC) would be the escape hatch if it ever matters.
|
|
|
|
## 7. Failure & recovery
|
|
|
|
* **Control session drop:** the client reconnects with capped exponential
|
|
backoff and re-registers all patterns. Existing worker connections and their
|
|
live streams are unaffected.
|
|
* **Worker connection drop:** every stream on it is torn down (destinations
|
|
closed); the hub closes the corresponding player sockets; the client removes
|
|
the connection from the pool and will dial a fresh one on the next allocation.
|
|
* **Pending timeout:** if no worker takes over a matched player within
|
|
`pendingTimeoutMs`, the hub drops the pending entry and closes the player.
|
|
* **Bad PSK / bad timestamp / bad magic:** the hub closes the TCP connection;
|
|
the client's session establishment fails fast.
|
|
|
|
## 8. Known limitations
|
|
|
|
1. No AEAD — payload integrity/authenticity is not cryptographically guaranteed.
|
|
2. TCP-level head-of-line blocking within a worker connection (lost packets;
|
|
see §6) — per-stream flow control removes the application-level variant only.
|
|
3. Single-event-loop hub (see §5) bounds throughput to one core.
|
|
4. `Intent 18` is reserved but only stubbed (the hub logs and closes).
|
|
5. Pattern ownership is last-writer-wins; two clients registering the identical
|
|
pattern string will silently reassign it. Overlapping-but-distinct regexes are
|
|
both kept, and when several match one hostname the winner is unspecified.
|
|
|
|
These are deliberate scope choices for a connectivity-focused P2P tool, not
|
|
oversights; each is a small, well-isolated change away from being hardened.
|