Compare commits

...
4 Commits
Author SHA1 Message Date
iceBear67 4df2560331 break: replace muxed workers with 1:1 tunnels
Worker frames are now FrameType + payload; there is no stream id.
Each player gets its own worker conn. maxTunnels (default 256)
caps concurrent tunnels. The old maxConn pool size is ignored so
existing configs do not silently admit only a handful of players.

Resume, per-direction windows, the control session, and the
DATA-only shaper stay. A dropped worker still hangs that one
player and reattaches over a fresh conn.

Add a hub-side per-IP limiter for player intents only (default
8/s, burst 16, 64 concurrent). Unmatched hostnames consume a
token; Intent 17 is never counted. 0 disables each knob.
2026-08-15 18:32:51 +08:00
iceBear67 da17140583 fix: harden resume/shutdown paths, tighten Intent-18 and PSK handshake handling
Client (Go) — resume correctness
- C1: completeResume now re-parks the stream when replay fails mid-conn-loss.
  parked was cleared before the replay loop, so the dying conn's teardown
  would start a second resumeLoop and the two loops could strand the stream
  with neither alive. Resume stats are counted only after the replay lands.
- C2: RST(ALREADY_BOUND) is retryable instead of terminating the loop. With
  C1 fixed there is never a genuine second attempt, so "already bound" means
  the hub still holds the stream on a half-open conn; the retry waits out
  that bind (bounded by the grace deadline, teardown on expiry) instead of
  returning and leaving the destination socket hung forever.

Client (Go) — shutdown semantics
- C3: Close() sets a closing flag and cancels an internal context; dialSession
  takes a ctx (DialContext + AfterFunc so shutdown aborts in-flight
  handshakes); the worker pool refuses new conns after closeAll (Allocate,
  background growth, cond waiters); serveControl's reconnect loop is gated by
  closing so Close works even when the caller's Start context is not
  cancelled; conn-loss teardown closes streams outright during shutdown
  instead of parking them for a reattach that will never come.

Client (Go) — hygiene
- C4: pingInterval() clamps at the single point a duration is derived, so a
  hand-built Config with PingIntervalMs <= 0 can no longer panic
  time.NewTicker (added DefaultPingIntervalMs).
- E6: shaperStall is sampled right after shaper.Acquire, before the socket
  write, so a hub that is not reading is no longer charged to the bandwidth
  cap in the stats.
- E7: stream log lines now carry conn%d/sid%d (leg.String()), making streams
  traceable across reattaches.
- P5: mirror constants IntentReserved/RegisterOk/RegisterErrPattern added;
  RegisterAck dispatch logs rejection reasons via the named codes.

Hub (Java) + PROTOCOL.md
- P3: Intent 18 replies with a Minecraft status-response packet
  ([Len: VarInt][0x00][JSON: String]) and closes (socket.end, so the write
  always lands) instead of closing silently; documented in PROTOCOL.md §2.
- P4: PSK address check is strict equality with the lowercase hex address;
  an uppercase/case-folded variant is now rejected per PROTOCOL.md §2.
- P7: PROTOCOL.md §5 SessionReady row lists its real fields
  (Flags/RecvWindow/ResumeGraceMs) instead of "(none)".

Verified: go vet, go test -race ./client/..., gradle test, full e2e suite
(twice), resume e2e 3x, plus live probes of the hub with the real client
codec (Intent-18 status reply, strict-lowercase PSK acceptance/rejection).%
2026-08-15 17:47:39 +08:00
iceBear67 7bd84af48d impl connection recovery 2026-08-15 17:31:35 +08:00
iceBear67 e63a34d53a fix 2026-07-25 16:33:28 +08:00
43 changed files with 6686 additions and 551 deletions
+286
View File
@@ -0,0 +1,286 @@
# redapricot 审计报告
审计日期:2026-08-15
审计方式:5 个并行探索子代理深读全部源码(~8200 行)与测试,交叉核对两端实现与
PROTOCOL.md;随后对全部高风险发现逐一人工复核(含 velocity 模块与 Velocity 官方
源码 `dev/4.0.0``PlayerDataForwarding.java` 逐字节比对)。
---
## 0. 总体评价
**设计质量很高,且文档与实现高度一致。** 亮点包括:单事件循环 hub 让并发模型归零;
「持锁不 dial」的池设计;resume 三偏移量(Sent/Accepted/Delivered)的区分在两端实现
和注释中都正确且互相印证;流控窗口同时充当重传缓冲区上限(「保留区无需自带上限」
这一论证成立,依赖链完整);off-switch(`streamResume:false``statsIntervalMs:0`
确实只留一个分支;写超时、心跳、会话建立 deadline 等「liveness 显式化」哲学落实到位。
**主要风险集中在三处**:客户端 resume 状态的并发管理(存在静默数据损坏路径)、
`Close()` 与关闭语义(库用场景泄漏)、测试覆盖(6 个协议行为零测试、Java 侧除纯
函数外零单测、e2e 注册就绪用固定 sleep)。
---
## 1. 协议实现审计
### 1.1 线级一致性:逐字节核对,全部一致 ✓
握手布局、Rekey 帧(`magic‖randLen‖rand‖ts‖flags‖window`REKEY=rand‖ts 不含
magic)、SessionReady、密钥派生(`SHA3-256(PK‖0x01/0x02)`、零 nonce、counter 0)、
控制消息、mux 帧(含 RESUME/RESUME_ACK 字段顺序)、流控语义(只 DATA 计窗口、WND
半窗批量、32KiB chunk)、resume 三偏移量语义、全部常量、配置默认值与钳制——两端实现
与 PROTOCOL.md 全部一致,无任何严重不一致。
**velocity 模块已经官方源码确证无误**`client/velocity.go` 与 PaperMC/Velocity
`dev/4.0.0``PlayerDataForwarding.java` 比对):
- payload 布局 `VarInt(version) ‖ String(ip) ‖ UUID[16] ‖ String(name) ‖ VarInt(properties)`
与官方 `writeVarInt + writeString + writeUuid + writeString + writeProperties` 完全一致;
- 版本协商(1.19.3+ 时 `requested≥4` 发 4、否则发 1)与官方 `findForwardingVersion`
逐分支一致,包括 v4 lazy-session 不带 key 段、v2/v3 只在 1.191.19.2 且客户端有
key 时使用(redapricot 无 key 故正确回落 v1)。
### 1.2 轻微偏差(不影响互通,但建议修)
| # | 问题 | 位置 | 说明 |
|---|------|------|------|
| P1 | **pattern 注册前被归一化,违反 §5 "verbatim" 约定** | `client.go:49-52``config.go:314-321` | 客户端注册的是 `NormalizeAddress(pattern)` 后的字符串。hub 端确实原样存储,但大小写敏感的 regex 会被破坏:以 `\.` 结尾的 pattern 经 `TrimRight(".")` 变成孤立 `\` 导致编译失败;`\Q…\E` 被小写化为 `\q…\e` 同样编译失败。应原样注册、仅匹配时归一化 |
| P2 | `streamWindowBytes: 0` 语义两端不同 | `Config.java:36-37` vs `config.go:71-75` | Go 视为"未设置→262144"Java 钳到最小 32768。规范未定义 0,属规范留白 |
| P3 | Intent 18 规范与实现不符 | `PROTOCOL.md:73-75` vs `HubConnection.java:96-98` | 规范称"回复 status line 后关闭",实现只 log+close |
| P4 | PSK address 比较宽松 | `HubConnection.java:113` | 规范要求小写精确匹配,实现用 `equalsIgnoreCase` |
| P5 | Go 侧缺镜像常量 | `client/config.go` | `INTENT_RESERVED``REGISTER_OK/ERR` 未定义,违反 CLAUDE.md「常量镜像」不变式 |
| P6 | RST 原因码半实现 | `worker.go:430-432``WorkerConn.java:55` | 客户端 `sendRst` 从不带原因字节(`RST_DIAL_FAILED/FLOW_CONTROL/RESUME_ABANDONED` 是死常量);hub 收到 RST 也不解析原因字节。协议标称"both directions"的对称性未实现 |
| P7 | 规范文档错误 | `PROTOCOL.md:214` | §5 表 SessionReady 标 "(none)",与 §4flags+window+grace)矛盾;§7.5 未提 `RST(ALREADY_BOUND)` 分支 |
---
## 2. 设计审计
**做得对且值得保留的设计**(均已核实,非泛泛而谈):
1. **池不在持锁时 dial**——`Allocate``maybeGrowLocked` 都把 dial 放到释放锁之后
/后台 goroutine,注释与代码一致;空池时单 dialer + `cond.Wait` + `dialGen` 代际
账本正确处理失败信号。
2. **流经 hub 路由而非闭包捕获 conn**——`Hub.onPlayerData``st.worker` 转发,
reattach 无需重装 handler,规避了死 conn 静默写缺陷。
3. **resume 账本**——重放点取 peer 的 accepted、窗口按 delivered 重述、丢弃自身在途
credit、`Delivered ≤ Accepted` 隐含不变式,两端对称且都有注释钉住;`UnackedBytes`
的摊还 O(1) advance、`from()` 越界返回 nil 触发终局 teardown(不静默截断)都正确。
4. **hub 侧 parked 上限**——`maxParkedStreams/maxParkedBytes` 以插入序 LinkedHashMap
淘汰最旧,`parkedBytes()` 计数与 `removeStream` 的扣减逻辑经核对自洽(写法绕,见 §4.6)。
5. **grace 协商而非配置假设**——hub 在 SessionReady 公布 `resumeGraceMs`,客户端钳制
在它之下,"客户端必须先放弃"从运维约定变成协议约束。
6. **shaper**——token bucket + start-time fair queuevclock 不因 size 前移是刻意的;
只整形 DATA、控制帧永不延迟;关停不卡流量;`NewShaper` 返回 nil 的短路设计让限速
关闭时热路径零分支。
**设计层面的顾虑**(非错误,属取舍):
- **单事件循环 hub**:一块网卡 + 1 MiB 帧 + 32KiB chunk,单核吞吐上限明确(文档已
承认)。但 **ChaCha20 的 `Cipher.update` 每次帧调用都过 JCE**,帧头 VarInt 解析 +
Buffer 切片每帧多次分配,实际吞吐可能在数百 Mbps 量级——建议做一次基准确认。
- **TCP-level HOL** 是 mux-over-TCP 固有(文档已承认),池是缓解而非解决。
- **无 AEAD**(文档已承认),ChaCha20 无完整性保护意味着主动攻击者可以翻转密文。
---
## 3. 逻辑漏洞(按严重度排序)
### 🔴 C1. 双 `resumeLoop` 竞态 → 静默数据损坏(客户端,最严重)
触发链(`client/resume.go``client/worker.go`):
1. `tryResume` 在新建 conn B 上 `registerStream` 并发 RESUME
2. `completeResume` 重放中途 `wc.sendData` 失败(B 刚注册即死,`resume.go:271-273`
——注意 `s.parked = false` 在重放**之前**已复位(`resume.go:253`);
3. B 的 readLoop 退出 → 对 B 上每条流调 `park()``worker.go:401-408`)→
`already := s.parked`**false** → 再起**第二个** `resumeLoop``resume.go:80`);
4. 旧 loop 仍在重试(该错误非终局,退避 500ms 后继续)。两个 loop 并发 `tryResume`
各自 `registerStream` 到不同 conn、各自把 `s.resumeWait` 覆盖成自己的 channel、
发两个并发 RESUME。
后果取决于时序,两种都是坏的:RESUME_ACK 被"非赢家"loop 消费 → leg 绑定到 hub 不
认识的 (conn, sid),上游(dest→player)数据被 hub 静默丢弃——**玩家 mute,无任何
日志**;或 RST 到达时 `resumeWait` 为 nil → 误 teardown 一个 hub 已恢复的流。
**修复**`completeResume` 失败时保持 `parked=true`(不让第二个 loop 启动),让
`park()``already` 分支兜底。
### 🔴 C2. 半开连接 + `RST(ALREADY_BOUND)` → 流永久悬挂(客户端)
场景:客户端心跳超时(60s)判定 worker conn 死亡并关闭,但 hub 侧该 conn 仍在
90s idle 窗口内(`WorkerConn.java` 侧流未 park)→ 客户端新 conn 发 RESUME → hub
`takeParked` 为 null、`streamByCid` 非空 → `RST(ALREADY_BOUND)`
`WorkerConn.java:207-210`)→ 客户端 `onRst` 映射为 `errResumeRaced``resumeLoop`
**无条件退出**`resume.go:104-108`),不 teardown 不重试。此后该流 `parked=true`
永远挂着、destination socket 永久泄漏,直到进程退出。
**修复**`errResumeRaced` 改为可重试(C1 修复后不再有真并发 loopALREADY_BOUND 只
意味着 hub 旧 conn 尚存),grace 耗尽后正常 teardown。
### 🟠 C3. `Close()` 不干净 + `dialSession` 无视 ctx(客户端)
- `Close()``client.go:413-421`)无 shutdown 标志。关闭 pool conns 后,readLoop 把流
park → `resumeLoop` 通过 `pool.Allocate()`(空池)**同步拨新 TCP 连接**继续重挂,
最长持续一个 grace(15s);
- `dialSession``client.go:100-210`)只用 `HandshakeTimeout` 硬 deadline,不读 ctx。
`Close()` 与在途重连/拨号并发时,新连接建立后无人关闭——`serveControl` 永久阻塞在
`ReadFrame` 上,goroutine + socket 泄漏。
**修复**`atomic.Bool closing``park` 时已 closing 直接 teardown`dialSession`
`net.Dialer.DialContext` + `context.AfterFunc` 在 ctx 取消时关连接。
### 🟠 C4. 手建 Config `PingIntervalMs<=0` → `time.NewTicker(0)` panic(客户端)
`New()``client.go:38-68`)只钳 window、解析带宽,**不验证 PingIntervalMs**`LoadConfig`
才钳到 ≥1000`config.go:292-298`)。注释明确说 Config 可被直接构造(测试用),此时
`time.NewTicker(0)` 直接崩进程(`client.go:342``worker.go:247`)。
**修复**`New()` 里 clamp 或显式报错。
### 🟡 服务端:无同等严重的逻辑漏洞
逐一复核了 `Hub.removeSession/expireOrphans/replayAwaiting`orphan 语义、deadline
切换、`rearm` 的 timer 与 `cancelTimer`)、`WorkerConn.handleResume`(含 replay 不
重复计窗、`st.credited=0`、CID 重铸)、`EnforceParkedCaps``parkedBytes` 先减后加再
`removeStream` 统一扣减——自洽但绕)、`EncryptedFrames.pump`(明文长度前缀/加密
负载/rekey 切换在帧边界无歧义)。**未发现逻辑错误**。健壮性/一致性缺口:
- **hub 不校验客户端发送窗口**(`WorkerConn.handleData` 只信任客户端)。协议 §7.3 明确
允许对超窗 RST,但 hub 不查——持 PSK 的恶意/损坏客户端可让 hub 端每流缓冲区无界
增长。建议按协议补校验。
- **`ProtoWriter.u8/u16` 静默截断**`ProtoWriter.java:11,16-19`):越界值截断为
"合法外观"的错误字节,无日志;与 reader 侧严格校验不对称。
- **停机无优雅 drain**`Main.java:40` 的 shutdown hook 不等待 `vertx.close()` 完成
(JVM halt 掐断异步关闭);无协议层下线通知;挂起玩家/流被硬切。
- **Vert.x 5 日志路由存疑**(中高置信):`Main.java:16` 设的是 Vert.x 4 的
`vertx.logger-delegate-factory-class-name`Vert.x 5 核心日志已走 SLF4J,且 classpath
上无 slf4j-api/log4j-slf4j2-impl。
---
## 4. 错误处理缺陷
| # | 问题 | 位置 | 后果 |
|---|------|------|------|
| E1 | **`registerAll` 半失败静默** | `client.go:254-263` | Register 写失败只 log 并 return`connectControl` 仍成功、`Start` 仍返回成功,但 hub 上一个 pattern 都没注册——玩家全被拒,且日志误导 |
| E2 | **RegisterAck status=1 不处理** | `client.go:314-318` | 非法 regex 只打日志,mapping 保留,之后永远收不到 ControlRequest |
| E3 | **WND/FIN/RST 写失败全吞** | `worker.go:429-435``worker.go:373` | WND 丢失 → hub 停发该流 → 玩家卡死,全程无日志 |
| E4 | **解析错误大量 `_ =`** | `client.go:325-327``worker.go:364-368` | 坏帧静默吞掉(hub 可信,可接受) |
| E5 | **`New()` 带宽解析失败静默关限速** | `client.go:58-59` | 运维以为限速生效,实际已关闭 |
| E6 | **`shaperStall` 归因污染** | `worker.go:637-648` | `Acquire``emit`(含最长 30s 的 socket 写)之间时间全记入"带宽上限";hub 停读的拥堵被误报成限速,与 stats 想区分的三病因矛盾。修法:elapsed 在 `Acquire` 返回后立即取 |
| E7 | **日志不可关联** | 多处 `stream %d` | sid 是 per-conn 的且 reattach 后会变,不带 conn id,排障无法把日志对到同一条流 |
| E8 | **RST 原因映射过宽** | `resume.go:309-312` | `RST_FLOW_CONTROL` 这类协议违规终局被当可重试,徒增延迟 |
| E9 | **resume 尝试越过 deadline 最多 10s** | `resume.go:152-168` | `time.After(ResumeAckTimeout)` 不随 deadline 缩短,玩家侧最坏失败延迟被放大到 grace+10s |
| E10 | **resumeGrace 与 hubGrace 取 min 后丢失下限** | `resume.go:38-44` | 误配 500ms grace 的 hub 让客户端预算 < 一次拨号所需,仅可观测性问题 |
---
## 5. 复杂度分析
| 热点 | 位置 | 分析 |
|------|------|------|
| `Stream` 类型 | `worker.go:451-518` | 25+ 字段、4 种同步原语(`s.mu`/`sendMu`/原子量/chan),resume 状态(parked/resumeWait/leg/cid/三 offset)与流量状态混居。**C1 正是"parked 复位"与"loop 存活"两状态未拆开导致的** |
| `completeResume` | `resume.go:203-288` | 全项目最复杂单函数:锁序(sendMu→s.mu)、三偏移换算、重放、窗口重述、欠账 FIN。正确但难审 |
| `WorkerPool.Allocate` | `worker.go:54-92` | cond 等待 + `dialGen`/`dialErr` 代际账本,失败路径无单测 |
| 接收方向账本 | `worker.go` 多处 | `q`/`qBytes`/`acceptedOffset`/`deliveredOffset`/`consumed` 由 writeLoop/deliverFromHub/credit/completeResume 四处维护 |
| shaper `headLocked` | `shaper.go:241-252` | 每 grant 线性扫描 O(n²);n 为流数时无感,但每 DATA 块一次 `make(chan)` 分配是热点 |
| `wire.Reader.VarInt` | `mc.go:55-61` | 每次 `bytes.NewReader` 分配,velocity/帧解析热路径高频调用 |
| velocity 状态机 | `velocity.go` | 两方向缓冲 + 5 个布尔,逻辑正确但可合并为三值枚举 |
| 测试 harness | `e2e/*` | 三层 helper 金字塔 + 3 个文件各自复制"hub+relay+dest+client"接线 |
---
## 6. 测试覆盖审计
**覆盖良好的**:黑盒路径恢复、控制断线三种路径、resume 字节精确/并发/禁用/grace 到期、
带宽整形与公平、慢流隔离、velocity 单测 13 场景 + e2e、PROXY v2、池广度优先、错误 PSK
拒绝。e2e 拉起**真实 Java hub 子进程**,架构扎实。
**零测试的协议行为**(6 个,e2e + 单测均未断言):
1. **RST 原因码**——协议定义了 6 个,两端只实际发射 UNKNOWN_STREAM/ALREADY_BOUND
ALREADY_BOUND 竞争路径(C1/C2 的暴露面)无任何测试;
2. **WND 窗口违规**——hub 不校验客户端窗口、客户端超窗检测无测试;
3. **pending 超时**——"已匹配但 SYN 永不来的玩家被关"无测试;
4. **hub idle watchdog**——`sessionIdleTimeoutMs` 无测试;
5. **CID 单次使用**——二次 SYN、resume 后旧 CID 失效无测试;
6. **1 MiB 帧上限**——两侧都无超长帧测试。
**结构性缺口**
- **Java 侧只有 1 个测试文件**(`CryptoCodecTest.java`,9 个纯函数用例)。hub 状态机、
`HubConnection` 握手/rekey 校验、`EncryptedFrames``WorkerConn` 流控/resume 零单测。
根因之一:`testHub()``vertx=null``CryptoCodecTest.java:79-83`),`Hub.rearm`
`vertx.setTimer` 会 NPE——**测试缝本身封死了 timer 路径**。
- **跨语言 pinning 只有 SHA3-224 一处**ChaCha20 字节一致性无 Java 侧单测、无
RFC 8439 KAT`deriveKey` 无绝对向量,全靠 e2e 兜底。
- **e2e 注册就绪用固定 200ms sleep**`e2e_test.go:47`)——全 e2e 最大的 flaky 源。
`Start` 只保证 Register 帧**写出**,不等待 RegisterAck。
- **flaky 前五**`TestShaperSharesFairlyBetweenStreams`35% 容差)、
`TestBlackholedPathRecovers`45s 轮询)、`TestResumePreservesByteStream`/
`TestResumeWithConcurrentStreams``TestControlOutageHangsArrivingPlayer`
- **velocity 无真实 Paper golden vector**——测试只证明自洽(本次审计已确认字节布局与
官方源码一致,建议补 golden 测试防回归)。
---
## 7. 重构建议(按性价比排序)
**P0 — 正确性(先做)**
1. **修 C1 双 resumeLoop 竞态**`completeResume` 失败保持 `parked=true`
`deliverResume` 校验 loop 归属。补 ALREADY_BOUND 竞争路径的 e2e。
2. **修 C2 半开悬挂**`errResumeRaced` 改为在 grace 剩余内重试。
3. **修 C3/C4**`Close()` 加 shutdown 标志、`dialSession` 尊重 ctx、`New()` 校验
`PingIntervalMs`
**P1 — 健壮性与可观测性**
4. 客户端 `registerAll` 失败要传导;RegisterAck status=1 至少移除/标记该 mapping。
5. hub 补客户端发送窗口校验(§7.3 允许的 RST);`ProtoWriter``u8/u16` 范围校验。
6. 修正 `shaperStall` 归因(elapsed 提前取);流日志统一带 `connID/sid`
7. pattern 原样注册(P1);`streamWindowBytes:0` 语义统一并写进规范(P2)。
**P2 — 测试**
8. Java 侧补:ChaCha20 RFC 8439 KAT + 分块跨块一致性、`EncryptedFrames` rekey、
`Hub` 状态机(timer 可注入)。
9. 客户端加 ready 信号(RegisterAck 挂钩到可等待 channel),替换 200ms sleep;补
6 个零覆盖协议行为的测试。
10. velocity 补官方字节 golden vector。
**P3 — 结构**
11. `completeResume` 拆成小函数 + 状态机图注释;接收方向账本封装为 `recvLedger`
12. harness 引入单一 `newTunnelHarness(t, opts…)` 组装点。
13. 热路径去分配:`wire.Reader.VarInt` 改游标读取;shaper chan 池化(低优先级)。
14. `Main` 的 shutdown hook 等待 `vertx.close()` 完成;核实 Vert.x 5 日志绑定。
---
## 8. 结论
协议实现是**干净且两端一致的**——这是本审计最重要的结论,逐字节核对未发现任何线级
不兼容。设计文档质量上乘,注释大量引用"它防的是什么故障",可维护性极好。
需要优先处理的是客户端 resume 路径的两个并发缺陷(C1 静默数据损坏、C2 悬挂泄漏)
和关闭语义(C3),其次是测试缺口。这两类问题恰好互相印证:C1 和 C2 都发生在
"连接死亡/重建"的交叉时序里,而这类时序恰恰是当前 e2e 覆盖最薄的地方。
---
## 附:修复状态跟踪
| 编号 | 内容 | 状态 |
|------|------|------|
| C1 | 双 resumeLoop 竞态 | ✅ 已修(`completeResume` 重放失败时重新置 `parked=true`,统计计数移至重放成功之后) |
| C2 | 半开 + RST(ALREADY_BOUND) 悬挂 | ✅ 已修(`resumeLoop``errResumeRaced` 改为可重试,由 grace 截止时间兜底) |
| C3 | Close()/dialSession ctx | ✅ 已修(`Client.closing` 标志 + 内部 ctx/cancel`dialSession(ctx)``DialContext` + `AfterFunc` 关闭;`WorkerPool.closed` 标志门控 Allocate/后台拨号;serveControl 重连被 closing 门控) |
| C4 | NewTicker(0) panic | ✅ 已修(`pingInterval()` 单点钳制,新增 `DefaultPingIntervalMs` |
| E6 | shaperStall 归因污染 | ✅ 已修(`elapsed()` 移至 `Acquire` 返回后、socket 写之前采样) |
| E7 | 流日志缺 conn id | ✅ 已修(`leg.String()` = `conn%d/sid%d`,替换全部 5 处流日志) |
| P3 | Intent 18 status line | ✅ 已修(hub 回复 Minecraft status 包 `[Len][0x00][JSON]` 后关闭;已用真实 codec 探针验证) |
| P4 | PSK address 严格小写 | ✅ 已修(`equalsIgnoreCase` → 严格 `equals`;大写变体探针验证被拒) |
| P5 | Go 镜像常量 | ✅ 已修(`IntentReserved=18``RegisterOk=0x00``RegisterErrPattern=0x01`RegisterAck 处理用命名常量) |
| P7 | PROTOCOL.md §5 SessionReady 表 | ✅ 已修(补充 `Flags/RecvWindow/ResumeGraceMs` 字段;§2 补充 Intent 18 status line 格式) |
验证:`go test ./client/... -count=1``go test -race ./client/...``gradle -p server test`
`go test ./e2e/... -count=1`(含 resume/blackhole 套件)全部通过;Intent 18 与严格小写
PSK 已用真实客户端 codec 对真实 hub 探针验证。
+166
View File
@@ -0,0 +1,166 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## What this is
redapricot is a central-hub P2P tunnel that speaks an *extension* of the Minecraft Java
Edition protocol. A **hub** (Java/Vert.x, one public TCP port) forwards players to a
**client** (Go) sitting next to a real Minecraft server behind NAT. Everything — players
and clients alike — arrives on the same port; the hub tells them apart by the handshake
`Intent` field (`17` = redapricot, `18` reserved, anything else = player).
Two implementations of one wire protocol live in this repo, so **most changes are
cross-language**. `PROTOCOL.md` is the normative spec; `docs/architecture.md` explains the
design rationale.
## Build
`scripts/build.sh` builds both sides (hub via `installDist`, client into `bin/`). It
auto-discovers a JDK/Gradle under `~/.sdkman/candidates/`.
**Gradle needs an explicit `JAVA_HOME` in this environment** — it is not inherited:
```bash
JAVA_HOME=$HOME/.sdkman/candidates/java/current gradle -p server installDist
JAVA_HOME=$HOME/.sdkman/candidates/java/current gradle -p server test
JAVA_HOME=$HOME/.sdkman/candidates/java/current gradle -p server shadowJar # fat jar
go build ./... # Go side
```
## Test
```bash
./scripts/e2e.sh # build both, Go unit tests, then e2e
go test ./client/... -count=1 # Go unit tests (codec, crypto, pool, shaper, velocity)
go test ./e2e/... -count=1 -v -timeout 300s
go test ./e2e/ -run TestRoundTrip -v # one e2e test
go test ./client/ -run TestShaperEnforcesRate -v
```
The e2e suite spawns the **real Java hub as a subprocess** from
`server/build/install/redapricot-server` (via `-cp lib/* io.icybear.redapricot.Main`) and
runs the Go client in-process against a mock destination. It fails fast if the hub has not
been installed, so **rebuild the hub after touching Java** or e2e silently tests stale
bytecode. `resolveJava` skips the suite when no JDK is found (`JAVA_HOME`, SDKMAN, `PATH`).
Timing-sensitive suites (`e2e/bandwidth_test.go`, `client/shaper_test.go`,
`e2e/slowstream_test.go`, `e2e/resilience_test.go`, `e2e/resume_test.go`) assert on rates
and recovery deadlines; give them slack rather than tightening thresholds. The resume tests
are load-sensitive — a busy machine (a gradle daemon in the background) shifts the races
they exercise, so run them a few times rather than trusting a single pass.
## Architecture
### Connection kinds (all on one port)
| First handshake | Becomes |
|---|---|
| `Intent 17`, rekey magic `0x01` | **control session** — pattern registration, `ControlRequest`, Ping/Pong |
| `Intent 17`, rekey magic `0x02` | **worker conn** — one player, 1:1 |
| any other intent | **player** — hostname regex-matched, then tunneled |
Session establishment (both kinds): plaintext handshake whose `Server Address` is
`hex(SHA3-224(PSK))` → one frame encrypted with PSK-derived keys carrying
`magic ‖ rand ‖ timestamp ‖ flags ‖ window`**both sides switch ciphers** to keys derived
from `rand‖ts` → hub replies `SessionReady` echoing accepted flags and its window.
### Player handoff
The hub pauses and buffers a matched player socket, mints a random 16-byte **CID**, and
sends `ControlRequest(CID, matchedPattern, ip, port)` down the control session. The client
dials a dedicated worker conn, `SYN`s it with that CID, dials the destination, and the hub
binds the pending player to that conn. The CID's secrecy (it only ever travels encrypted
over the control session) is what authorizes the takeover — there is no other client
identity check. The hub echoes **the matched pattern string, not the player's hostname**,
so the client can look it up directly in its route table; the buffered handshake is
forwarded verbatim so the backend sees the original hostname.
### Code map
- `server/src/main/java/io/icybear/redapricot/``HubConnection` (per-socket state machine:
handshake parse → dispatch → rekey → control/worker/player), `Hub` (pattern registry, CID
table, pending players), `ControlSession`, `WorkerConn` (1:1 tunnel + flow-control state),
`net/EncryptedFrames`, `crypto/Crypto`, `util/` (VarInt, ProtoReader/Writer, Hex).
- `client/``client.go` (control session, reconnect, dispatch), `worker.go` (1:1
dial, `WorkerConn`, `Stream` and its two goroutines), `shaper.go` (egress rate cap),
`velocity.go` (Velocity modern-forwarding interception), `proxyproto.go` (HAProxy v2),
`config.go` (config + all protocol constants), `wire/` (VarInt/MC codec, SHA3+ChaCha20,
`FramedConn`).
- `cmd/redapricot-client/` — binary entrypoint; `e2e/` — integration harness.
### Threading
- **Hub:** a single Vert.x verticle instance, so the pattern registry, CID table, and all
per-connection state are touched by one event loop and need no locking. Never introduce a
blocking call there.
- **Client:** one goroutine reads each connection; `WriteFrame` is mutex-serialized. Each
tunnel has exactly two goroutines — `run` (destination → hub) and `writeLoop` (the *only*
writer to the destination, draining a queue fed by the worker readLoop). The readLoop must
never write to a destination: WND is granted only after the dest write completes, and a
stalled backend must not stall heartbeat / FIN dispatch on that conn.
## Invariants to preserve when editing
- **Constants are mirrored** in `client/config.go` and `server/.../Protocol.java`. Changing
one without the other is a silent protocol break; update `PROTOCOL.md` too.
- **The frame length prefix is plaintext, the payload is encrypted.** This is deliberate: a
reader always knows how many ciphertext bytes belong to the current frame, which makes the
rekey cipher switch unambiguous. Do not encrypt the length.
- **Per-direction keys, fixed zero nonce** (`SHA3-256(phaseKey ‖ 0x01)` c2s,
`‖ 0x02` s2c). Both directions must never share a keystream. Go `x/crypto/chacha20` and
Java JCE `ChaCha20` are byte-identical here, and unit tests on both sides pin the same
SHA3-224 vector — keep that pinning if you touch crypto.
- **Per-connection flow control is mandatory.** The hub rejects a session whose rekey lacks
`FLAG_STREAM_FC`; the client rejects a hub that does not echo it. Credit is granted back
(`WND`) only as bytes are actually written to the terminal socket, batched at half-window.
- **One worker conn carries one player.** There is no stream id. The first business frame
after `SessionReady` is `SYN` or `RESUME`; a second bind on the same conn is a protocol
violation. `PING`/`PONG` are connection-scoped frames.
- **Only DATA is shaped** by `client/shaper.go`. Delaying `FIN`, `WND`, or `PONG` would trip
the very liveness detection the heartbeat exists for. The shaper is client-local and
invisible on the wire.
- **Each player gets its own worker dial**, up to `maxTunnels` (default 256). A dial is
never performed while holding the live-set lock: session establishment is network I/O,
and one unresponsive hub must not block unrelated players.
- **The hub IP limiter is player-only.** Intent 17 (control + workers) is never
admitted through it — those sockets share the client's one address. Unmatched
player hostnames still consume a token. `0` turns each knob off.
- **Liveness is explicit everywhere:** every session heartbeats (drop after
`3 × pingIntervalMs`), every socket write is bounded, establishment has a deadline. A
silently blackholed path (NAT forgetting a flow, no FIN/RST) must recover without operator
action — `TestBlackholedPathRecovers` guards this.
- **Stream resumption is byte-exact or it is nothing** (`PROTOCOL.md §7.5`,
`client/resume.go`, `WorkerConn.handleResume`). A worker-conn drop hangs the player and
reattaches over a fresh conn. Three offsets are tracked per direction and are
*not* interchangeable: replay from the peer's **accepted** offset, restate the window from
its **delivered** offset, and never size the window from **credited** — the grants in
flight when the conn died are gone for good, and a window derived from them can be
permanently zero, which deadlocks. `TestResumePreservesByteStream` guards this.
- **The retained region needs no cap of its own.** Flow control already bounds outstanding
bytes to one window, so the retransmit buffer *is* the outstanding region. Anything that
lets a sender exceed its window silently makes it unbounded.
- **Hub-side socket handlers must route through `st.worker`, never through a captured
conn.** A lambda installed in `handleSyn` closes over `this`; after a reattach it would
write into the dead conn's transport, where sends are dropped silently and the player goes
mute with nothing logged.
- **A closed control session orphans its routes rather than deleting them**
(`PROTOCOL.md §5.2`, `Hub.removeSession`). Players arriving during the client's
reconnect are held and replayed once it re-registers, instead of being told there
is no such server. Two traps: `Hub.match` must surface the orphaned state rather
than hand back a dead `ControlSession` (`EncryptedFrames.send` drops silently on a
closed transport, so the player would hang with no request ever sent), and a held
player's deadline is the registration grace, not `pendingTimeoutMs` — whichever is
shorter fires first and closes the socket.
- **The off switches must reach the hot path, not just the wire.** `streamResume: false`
allocates no retained region and `statsIntervalMs: 0` allocates no counters, so both cost
one predictable branch. `TestResumeDisabledAllocatesNothing` guards the first.
## Conventions
- Comments here explain *why*, often citing the failure they prevent, and reference
`PROTOCOL.md §N`. Match that when adding code on either side.
- Wire-visible behaviour changes need: both implementations, `PROTOCOL.md`, an e2e test, and
usually a note in `docs/architecture.md` and the README config tables.
- Java uses Lombok (freefair plugin) and Log4j2; JUL is routed through Log4j2 both via
`applicationDefaultJvmArgs` and programmatically in `Main` for the `java -jar` path.
+338 -64
View File
@@ -15,7 +15,7 @@ There are three roles:
| **Player** | any | An ordinary Minecraft client connecting through the hub. | | **Player** | any | An ordinary Minecraft client connecting through the hub. |
``` ```
Player ──MC──▶ Hub(server) ══WorkerConn(mux)══▶ Client ──MC──▶ Destination Player ──MC──▶ Hub(server) ══WorkerConn(1:1)══▶ Client ──MC──▶ Destination
▲ registers patterns / receives control requests │ ▲ registers patterns / receives control requests │
└────────────── Control Session ────────────────────┘ └────────────── Control Session ────────────────────┘
``` ```
@@ -74,6 +74,19 @@ The hub reads exactly one Handshake packet and dispatches on `Intent`:
| `18` | Reserved for redapricot management/status. Never matched against patterns. The reference hub replies with a status line and closes. | | `18` | Reserved for redapricot management/status. Never matched against patterns. The reference hub replies with a status line and closes. |
| anything else | **Player** connection. `ServerAddress` is matched against the registered **regex** PATTERNs (§5.1). | | anything else | **Player** connection. `ServerAddress` is matched against the registered **regex** PATTERNs (§5.1). |
For `Intent == 18` the hub replies with a Minecraft status-response packet and
closes — the same shape a player receives for a status query (Intent 1), so the
port can be probed with ordinary tooling:
```
[Len: VarInt][PacketID 0x00][JSON: String]
```
The JSON is a minimal status payload, e.g.
`{"description":{"text":"redapricot hub"},"version":{"name":"redapricot","protocol":767},"players":{"max":0,"online":0}}`.
It is the one reply the hub sends in plaintext: Intent 18 never negotiates
encryption or any other redapricot state.
For `Intent == 17` the hub additionally requires For `Intent == 17` the hub additionally requires
`ServerAddress == lowercase_hex(SHA3-224(PSK))` — a 56-character hex string. `ServerAddress == lowercase_hex(SHA3-224(PSK))` — a 56-character hex string.
This is the first (cheap) proof that the peer knows the PSK. A mismatch closes This is the first (cheap) proof that the peer knows the PSK. A mismatch closes
@@ -141,14 +154,25 @@ RandLen : VarInt # 8 ≤ RandLen ≤ 64
Rand : Bytes[RandLen] # cryptographically random Rand : Bytes[RandLen] # cryptographically random
Timestamp : I64 # client's epoch milliseconds Timestamp : I64 # client's epoch milliseconds
Flags : VarInt # feature flags; bit 0x01 (STREAM_FC) MUST be set Flags : VarInt # feature flags; bit 0x01 (STREAM_FC) MUST be set
RecvWindow: VarInt # client's per-stream receive window, bytes (§7.3) RecvWindow: VarInt # client's per-connection receive window, bytes (§7.3)
``` ```
`Flags` is a bitfield of features. Bit `0x01` (STREAM_FC) declares `Flags` is a bitfield of features.
**per-stream flow control** (§7.3) and is mandatory: `RecvWindow` advertises
the client's per-stream receive window in bytes and must be positive. The hub | Bit | Name | Meaning |
closes the connection if the flag is missing, `RecvWindow` is absent or |-----|------|---------|
non-positive, or the fields are malformed. | `0x01` | STREAM_FC | **Per-connection flow control** (§7.3). Mandatory. |
| `0x02` | WORKER_HEARTBEAT | Connection-level `PING`/`PONG` on worker conns (§7.4). Optional. |
| `0x04` | STREAM_RESUME | **Stream resumption** (§7.5): a worker-conn drop hangs the player rather than closing it. Optional. |
STREAM_FC is mandatory: `RecvWindow` advertises the client's per-connection receive
window in bytes and must be positive. The hub closes the connection if the flag
is missing, `RecvWindow` is absent or non-positive, or the fields are malformed.
Optional bits are **negotiated**: the hub echoes in `SessionReady` only those it
accepts, and the client enables a feature only when its bit comes back. A hub
that does not know WORKER_HEARTBEAT simply omits the bit and the client falls
back to TCP keepalive alone.
The hub: The hub:
@@ -166,13 +190,22 @@ frame (both directions) is Phase B, counters reset to 0.
The hub then sends one Phase-B frame to confirm success: The hub then sends one Phase-B frame to confirm success:
``` ```
SessionReady : payload = [ 0x00, Flags: VarInt, RecvWindow: VarInt ] SessionReady : payload = [ 0x00, Flags: VarInt, RecvWindow: VarInt,
ResumeGraceMs: VarInt ] # only when STREAM_RESUME is set
``` ```
The hub echoes the accepted flags (STREAM_FC set) followed by its own The hub echoes the accepted flags (STREAM_FC set) followed by its own
per-stream receive window. A client must reject a SessionReady without the per-connection receive window. A client must reject a SessionReady without the
STREAM_FC flag or without a positive window (an unsupported hub). STREAM_FC flag or without a positive window (an unsupported hub).
`ResumeGraceMs` is present only when the hub accepts STREAM_RESUME, and states
how long it will hang a player waiting for that player's stream to be reattached
(§7.5). The client clamps its own retry budget below this value. Advertising it
rather than assuming matching configuration is deliberate: the client must always
give up first, and if the hub instead dropped a hung player while the client was
still reattaching, the failure would be a silent hang rather than an error. A hub
that sets the flag but omits the field is treated as not supporting resumption.
A hub that rejects the session simply closes the TCP connection (optionally A hub that rejects the session simply closes the TCP connection (optionally
after a Phase-B `Error` frame, §6). After `SessionReady`: after a Phase-B `Error` frame, §6). After `SessionReady`:
@@ -191,7 +224,7 @@ Type : u8
| Type | Name | Direction | Fields | | Type | Name | Direction | Fields |
|--------|----------------|-----------|--------| |--------|----------------|-----------|--------|
| `0x00` | SessionReady | S → C | *(none)* — the confirmation frame from §4 | | `0x00` | SessionReady | S → C | `Flags: VarInt`, `RecvWindow: VarInt`, `ResumeGraceMs: VarInt` (only when the hub accepted `STREAM_RESUME`) — the confirmation frame from §4 |
| `0x01` | Register | C → S | `Pattern: String` | | `0x01` | Register | C → S | `Pattern: String` |
| `0x02` | Unregister | C → S | `Pattern: String` | | `0x02` | Unregister | C → S | `Pattern: String` |
| `0x03` | RegisterAck | S → C | `Pattern: String`, `Status: u8` (0 = ok, 1 = invalid pattern) | | `0x03` | RegisterAck | S → C | `Pattern: String`, `Status: u8` (0 = ok, 1 = invalid pattern) |
@@ -236,6 +269,40 @@ Because the pattern is a regex, a literal dot must be escaped (`mc\.example\.com
an unescaped `.` is the regex "any character" wildcard. A pattern that fails to an unescaped `.` is the regex "any character" wildcard. A pattern that fails to
compile is rejected at `Register` time with `RegisterAck` status `1`. compile is rejected at `Register` time with `RegisterAck` status `1`.
### 5.2 Orphaned routes (control-session outage)
When a control session closes, its registrations are **not** deleted straight
away. They are marked *orphaned* and kept for `registrationGraceMs`.
This costs nothing on the wire — it is entirely hub-side behaviour — but it
closes a gap that is otherwise very visible. A client whose control session dies
reconnects with backoff, and until it re-registers the hub has no route for it,
so every player arriving in that window is told there is no such server. The
players already tunneled are unaffected, since they ride worker conns, which a
control-session close never touches.
While a route is orphaned:
* a player matching it is **held** — paused, with its handshake buffered exactly
as for a normal pending player — and no `ControlRequest` is sent, because there
is no session to send it to;
* a player that was already pending when the session closed is moved into the
same held state rather than being dropped;
* when any client registers that pattern again, the hub delivers the
`ControlRequest` it never sent and the player proceeds normally. The held
player's deadline switches from the registration grace to `pendingTimeoutMs`
at that point, since it is now waiting for a worker rather than for a route.
If the grace expires with no client having re-registered, the route and every
player held on it are dropped. `registrationGraceMs: 0` disables the mechanism
and restores the immediate-drop behaviour.
Note the hub cannot distinguish "this client is reconnecting" from "this client
is gone for good" — that is what the grace period is a bet on. It is bounded on
the client side too: the reference client retries immediately on a control-session
drop and caps its backoff at 10s, so the bet is usually settled in well under a
second.
## 6. Error frame (any redapricot connection) ## 6. Error frame (any redapricot connection)
At any time either side may send, then close: At any time either side may send, then close:
@@ -247,41 +314,62 @@ Msg : String
Purely informational; the receiver logs it. Purely informational; the receiver logs it.
## 7. Worker conn & multiplexing ## 7. Worker conn
A **Worker Conn** (`Magic == 0x02`) carries player↔destination traffic for many A **Worker Conn** (`Magic == 0x02`) carries player↔destination traffic for
players over one TCP connection using a minimal stream multiplexer. The unit of **exactly one player**. The TCP connection *is* the tunnel: there is no stream
work is a **stream**. Stream ids are assigned by the **client** (the only side id and no multiplexer. The client dials a fresh worker conn for each
that opens streams), unique per worker conn, starting at 1 and increasing. `ControlRequest` (and for each resumption attempt).
Each encrypted frame on a worker conn carries one **mux frame**: Each encrypted frame on a worker conn carries one **tunnel frame**:
``` ```
FrameType : u8 FrameType : u8
StreamID : VarInt
Data : Bytes[...] # remainder of the frame payload Data : Bytes[...] # remainder of the frame payload
``` ```
| FrameType | Name | Direction | Data | | FrameType | Name | Direction | Data |
|-----------|------|-----------|------| |-----------|------|-----------|------|
| `0x00` | SYN | C → S | `CID: Bytes[16]` open a stream to take over the pending player identified by CID. | | `0x00` | SYN | C → S | `CID: Bytes[16]` — take over the pending player identified by CID. |
| `0x01` | DATA | both | raw tunneled bytes for the stream. | | `0x01` | DATA | both | raw tunneled bytes. |
| `0x02` | FIN | both | *(empty)* — graceful close of the stream (both directions). This is the "disconnect" the hub sends when the player leaves. | | `0x02` | FIN | both | *(empty)* — graceful close (both directions). This is the "disconnect" the hub sends when the player leaves. |
| `0x03` | RST | both | *(optional 1 byte reason)* — abnormal close (e.g. CID unknown/expired, destination dial failed). | | `0x03` | RST | both | *(optional 1 byte reason)* — abnormal close (e.g. CID unknown/expired, destination dial failed). |
| `0x04` | WND | both | `Delta: VarInt` — flow-control credit grant (§7.3). | | `0x04` | WND | both | `Delta: VarInt` — flow-control credit grant (§7.3). |
| `0x05` | PING | both | `Nonce: I64` — liveness probe (§7.4). |
| `0x06` | PONG | both | `Nonce: I64` — echoes the probe's nonce (§7.4). |
| `0x07` | RESUME | C → S | `CID: Bytes[16]`, `Accepted: I64`, `Delivered: I64` — reattach a hung player to this conn (§7.5). |
| `0x08` | RESUME_ACK | S → C | `Accepted: I64`, `Delivered: I64`, `NewCID: Bytes[16]` — the reattach succeeded (§7.5). |
There is no explicit SYN-ACK: success is implied by the hub forwarding the The first business frame after `SessionReady` must be `SYN` or `RESUME`. A
buffered Handshake as the stream's first `DATA`; failure is an `RST`. second bind on an already-bound conn is a protocol violation: the hub replies
`RST` and closes. There is no explicit SYN-ACK: success is implied by the hub
forwarding the buffered Handshake as the first `DATA`; failure is an `RST`.
### 7.1 Stream allocation (client side) `RST` reason codes. The byte remains optional — a peer that predates it sends
none, and a receiver must tolerate its absence — but distinguishing the reasons
matters for resumption, where "this stream is gone" and "someone else already
took it" call for opposite responses.
The client keeps a pool of `1 ≤ N ≤ max_conn` worker conns (`max_conn` | Code | Name | Meaning |
configurable, `1..8`). To place a new stream: |------|------|---------|
| `0x00` | UNSPECIFIED | No reason given (also the meaning of an absent byte). |
| `0x01` | UNKNOWN_STREAM | CID unknown or expired, or the hub restarted. Terminal: stop retrying. |
| `0x02` | ALREADY_BOUND | Another reattach already claimed this stream. Do **not** tear down. |
| `0x03` | RESUME_ABANDONED | The peer gave up reattaching. |
| `0x04` | FLOW_CONTROL | The peer exceeded its advertised window. |
| `0x05` | DIAL_FAILED | The client could not reach the destination. |
1. Pick the worker conn with the **fewest active streams**. ### 7.1 Tunnel allocation (client side)
2. If that minimum conn is **saturated** (active streams `> 8`) **and**
`poolSize < max_conn`, dial a new worker conn and use it instead. The client dials one worker conn per player, up to a configurable
3. Otherwise use the least-loaded conn (even if it exceeds 8 at `max_conn`). `maxTunnels` cap (default 256, clamped to `[1, 4096]`). There is no pool and no
least-loaded placement: a `ControlRequest` either gets its own TCP connection or
is dropped (the hub then closes the player when `pendingTimeoutMs` fires).
A dial is never performed while holding the live-set lock: session establishment
is network I/O, and one unresponsive hub must not be able to block unrelated
players. Each dial is independent and bounded by the session-establishment
deadline (§7.4); callers are not serialized behind a single in-flight handshake.
### 7.2 End-to-end player flow ### 7.2 End-to-end player flow
@@ -293,45 +381,133 @@ configurable, `1..8`). To place a new stream:
control session. If no SYN arrives within `pendingTimeoutMs` (default 10000) control session. If no SYN arrives within `pendingTimeoutMs` (default 10000)
the pending entry is dropped and the player socket closed. the pending entry is dropped and the player socket closed.
3. The client receives `ControlRequest`, looks up the destination for `Pattern`, 3. The client receives `ControlRequest`, looks up the destination for `Pattern`,
allocates a worker conn + `StreamID`, and sends `SYN(StreamID, CID)`. In dials a dedicated worker conn, and sends `SYN(CID)`. In parallel it dials the
parallel it dials the destination and (if configured) writes a HAProxy v2 destination and (if configured) writes a HAProxy v2 header (§8) carrying
header (§8) carrying `PlayerIP:PlayerPort`. `PlayerIP:PlayerPort`.
4. The hub matches `CID` to the pending player, binds 4. The hub matches `CID` to the pending player, binds
`(workerConn, StreamID) ↔ playerSocket`, forwards the buffered bytes as `workerConn ↔ playerSocket`, forwards the buffered bytes as `DATA`, and
`DATA`, and resumes the player socket. Subsequent player bytes become `DATA` resumes the player socket. Subsequent player bytes become `DATA` frames;
frames; `DATA` frames from the client are written to the player socket. If `DATA` frames from the client are written to the player socket. If `CID` is
`CID` is unknown/expired the hub replies `RST`. unknown/expired the hub replies `RST`.
5. When the player disconnects the hub sends `FIN` on the stream; the client 5. When the player disconnects the hub sends `FIN`; the client closes the
closes the destination. When the destination closes, the client sends `FIN`; destination. When the destination closes, the client sends `FIN`; the hub
the hub closes the player socket. `RST` is treated the same way (hard close). closes the player socket. `RST` is treated the same way (hard close).
Data on a worker conn is subject to that TCP connection's back-pressure for Data on a worker conn is subject to that TCP connection's back-pressure. Credit
its **aggregate** bandwidth; *per-stream* fairness is governed by the credit windows of §7.3 bound how many bytes may be in flight on that one tunnel.
windows of §7.3.
### 7.3 Per-stream flow control ### 7.3 Per-connection flow control
Every stream carries an independent credit window per direction: Every worker conn carries an independent credit window per direction:
* Each side advertised its **receive window** W (bytes) at session setup. A * Each side advertised its **receive window** W (bytes) at session setup. A
sender may have at most W un-credited DATA bytes outstanding per stream; the sender may have at most W un-credited DATA bytes outstanding; the initial
initial budget is W, spent as DATA is sent (`Data` length only — SYN/FIN/RST budget is W, spent as DATA is sent (`Data` length only — SYN/FIN/RST frames
frames are free) starting with the very first DATA on the stream (including are free) starting with the very first DATA (including the hub's forwarded
the hub's forwarded handshake). handshake).
* The receiver returns credit with `WND(Delta)` once bytes are **delivered to * The receiver returns credit with `WND(Delta)` once bytes are **delivered to
the terminal socket** (written to the player / destination connection), not the terminal socket** (written to the player / destination connection), not
when they are merely buffered. Receivers should batch grants (the reference when they are merely buffered. Receivers should batch grants (the reference
implementations send one `WND` per W/2 bytes consumed). implementations send one `WND` per W/2 bytes consumed).
* A sender whose window is exhausted pauses reading **that stream's source * A sender whose window is exhausted pauses reading **that player's source
socket only**; the shared worker conn is never paused because of a single socket only**. A receiver that observes more than W un-credited bytes may
stream. A receiver that observes more than W un-credited bytes on a stream reset the tunnel (`RST`) as a protocol violation.
may reset it (`RST`) as a protocol violation.
* Senders should also cap individual DATA payloads (the reference * Senders should also cap individual DATA payloads (the reference
implementations use 32 KiB) so one stream cannot monopolize the link for a implementations use 32 KiB) so one write cannot occupy the link for a full
full 1-MiB frame. 1-MiB frame.
Both windows may differ (each side enforces the one its peer advertised). Both windows may differ (each side enforces the one its peer advertised).
`Delta` must be positive; a `WND` for an unknown stream id is ignored. `Delta` must be positive; a `WND` on an unbound worker conn is ignored.
### 7.4 Liveness
TCP alone cannot tell a healthy idle connection from a dead one. When a stateful
middlebox forgets an established flow — conntrack expiry, a firewall reload, a
cloud load balancer's idle timeout — it sends neither `FIN` nor `RST`. Both ends
keep a socket that will never again carry a byte, and a reader parked on it waits
forever. Without an application-level probe the client cannot notice: its worker
conn stays in the pool, the hub keeps routing players to a control session nobody
reads, and service does not return until the client process is restarted.
Every established session is therefore covered by a heartbeat:
* **Control session** — the client sends `Ping` every `pingIntervalMs` and the
hub answers `Pong`. If no `Pong` arrives for `3 × pingIntervalMs`, the client
closes the session, which triggers its normal reconnect with backoff.
* **Worker conns** — when WORKER_HEARTBEAT was negotiated, the same exchange
runs as connection-level `PING`/`PONG` frames. On timeout the client closes
the conn. The player is reset, unless STREAM_RESUME was negotiated, in which
case it is hung and reattached over a fresh conn instead (§7.5).
* **Hub side** — an established redapricot session that receives no frame for
`sessionIdleTimeoutMs` (default 90000, `0` disables) is closed. Player
connections are never subject to this.
Both ends also enable TCP keepalive, which catches the narrower case of a peer
that has become unreachable at the IP layer.
Session establishment (§4) is bounded by a single deadline covering the dial,
the `Rekey` write and the `SessionReady` read, and every frame write is bounded
too — a peer that stops reading must not be able to park a write forever.
### 7.5 Stream resumption (STREAM_RESUME)
A worker conn is only the middle leg of the player it carries. When it dies,
both terminal sockets — the player's and the destination's — are usually still
healthy, so resetting the tunnel discards working connections because a
replaceable transport failed.
With STREAM_RESUME negotiated, a worker-conn drop instead **hangs** the player:
* the hub pauses the player socket, keeps its state, and holds it for
`ResumeGraceMs` from the moment of the *first* hang (an absolute deadline, so a
flapping client cannot extend it indefinitely);
* the client keeps the destination socket open and reattaches over a fresh
worker conn by sending `RESUME` with the player's CID;
* the hub answers `RESUME_ACK`, or `RST(UNKNOWN_STREAM)` if it holds no such
player — which is also what a client gets from a hub that has restarted.
**Resumption is byte-exact, and must be.** Frames handed to a dying socket are
lost with no notification, and the frame cipher cannot be resynchronized, so each
side replays whatever the other did not receive. Splicing the stream even one
byte off corrupts the tunneled protocol.
Three offsets are tracked per direction, and they are not interchangeable:
| Offset | Meaning | Used for |
|--------|---------|----------|
| `Sent` | bytes handed to the wire | the end of the retained region |
| `Accepted` | bytes taken off the wire toward the terminal socket | **where to replay from** |
| `Delivered` | bytes actually written to the terminal socket | **how to restate the window** |
Each side retains the bytes between what the peer has credited and what it has
sent. This costs no new bound: flow control (§7.3) already caps outstanding bytes
at one window, so the retained region *is* the outstanding region.
On reattach both sides replay `[peer's Accepted, Sent)` and set
`SendWindow = W (Sent peer's Delivered)`, then discard their own pending
credit — the exchanged `Delivered` already carries everything those deltas would
have, so emitting both would grant the same bytes twice.
Two rules deserve emphasis, because the obvious simplifications are wrong:
* *Accepted*, not *Delivered*, is the replay point. Delivery is signalled
asynchronously on both sides and stops being reported exactly when a connection
dies; replaying from it would re-send bytes the peer already has.
* *Delivered*, not *credited*, sizes the window. Credit travels as deltas, and
the grants in flight when the connection died are gone for good. A window
derived from them is permanently short — and if a full window was outstanding
at the drop, permanently zero, which deadlocks: nothing can be sent, so no
credit can ever come back.
`RESUME_ACK` carries a freshly minted `NewCID`, which replaces the old one. A CID
therefore stays single-use even though a player may be reattached many times, so
a leaked CID grants no more than the outage in which it was observed.
Resumption is **hub-instance-affine**: a CID means nothing to a second hub behind
an L4 load balancer, which answers `RST(UNKNOWN_STREAM)` and lets the client tear
down at once. Because the grace period holds player sockets and their buffers, a
hub bounds the number of hung players and the bytes they retain, dropping the
oldest first when either cap is reached.
## 8. HAProxy protocol v2 (optional) ## 8. HAProxy protocol v2 (optional)
@@ -361,12 +537,60 @@ big-endian.
"psk": "change-me", "psk": "change-me",
"timestampWindowMs": 30000, "timestampWindowMs": 30000,
"pendingTimeoutMs": 10000, "pendingTimeoutMs": 10000,
"streamWindowBytes": 262144 "streamWindowBytes": 262144,
"sessionIdleTimeoutMs": 90000,
"streamResume": true,
"resumeGraceMs": 20000,
"maxParkedStreams": 256,
"statsIntervalMs": 0,
"registrationGraceMs": 15000,
"playerRatePerSec": 8,
"playerBurst": 16,
"maxPlayersPerIp": 64
} }
``` ```
`streamWindowBytes` (optional, default 262144, clamped to [32768, 8388608]) is `streamWindowBytes` (optional, default 262144, clamped to [32768, 8388608]) is
the hub's advertised per-stream receive window (§7.3). the hub's advertised per-connection receive window (§7.3).
`sessionIdleTimeoutMs` (optional, default 90000) closes an established control
session or worker conn that has gone silent for that long (§7.4). It must stay
comfortably above the client's `pingIntervalMs`; `0` disables the watchdog.
`streamResume` (optional, default true) offers STREAM_RESUME (§7.5). With it
false the hub never echoes the flag and behaves exactly as a hub that predates
the feature, allocating no retained regions.
`resumeGraceMs` (optional, default 20000) is how long a hung player is held, and
is advertised in `SessionReady`. It must exceed the client's own grace by at
least one dial, which is why the client clamps itself against the advertised
value rather than its own configuration.
`maxParkedStreams` (optional, default 256) and `maxParkedBytes` (default
`maxParkedStreams × 2 × streamWindowBytes`) bound what hung players may cost;
past either the oldest are dropped.
`statsIntervalMs` (optional, default 0 = off) logs a periodic line with live and
parked stream counts, retained bytes, and the pattern count.
`registrationGraceMs` (optional, default 15000) is how long a closed control
session's routes are kept as **orphaned** rather than deleted (§5.2). `0`
disables it, restoring the behaviour of dropping routes the instant a session
closes.
`playerRatePerSec` (optional, default 8) and `playerBurst` (optional, default
16) are a per-IP token bucket applied **only to player connections** (any
handshake whose `Intent` is not 17 or 18). One arrival consumes one token;
without a token the socket is closed after the handshake, before `match`, CID
minting, or pause. Unmatched hostnames still consume a token — otherwise a
hostname scan is a free flood. Intent 17 is never admitted through the
limiter: every worker conn comes from the client's one address, and limiting
those would be the hub throttling its own client. `playerRatePerSec: 0` turns
the bucket off.
`maxPlayersPerIp` (optional, default 64) caps concurrent player sockets from
one address (pending + live + parked). `0` disables the cap. Addresses are
matched exactly; IPv6 `/64` aggregation is out of scope.
### 9.2 Client — JSON ### 9.2 Client — JSON
@@ -374,17 +598,51 @@ the hub's advertised per-stream receive window (§7.3).
{ {
"server": "127.0.0.1:25565", "server": "127.0.0.1:25565",
"psk": "change-me", "psk": "change-me",
"maxConn": 4, "maxTunnels": 256,
"pingIntervalMs": 20000, "pingIntervalMs": 20000,
"streamWindowBytes": 262144, "streamWindowBytes": 262144,
"maxBandwidth": "20mbps",
"streamResume": true,
"resumeGraceMs": 15000,
"statsIntervalMs": 0,
"mappings": [ "mappings": [
{ "pattern": "mc\\.example\\.com", "destination": "127.0.0.1:25566", "proxyProtocol": true } { "pattern": "mc\\.example\\.com", "destination": "127.0.0.1:25566", "proxyProtocol": true }
] ]
} }
``` ```
`maxTunnels` (optional, default 256, clamped to [1, 4096]) is how many
concurrent 1:1 worker connections the client will hold. A `ControlRequest`
arriving at the cap is dropped. The older `maxConn` key (the mux-era pool size,
clamped 18) is ignored if present: treating it as a player cap would silently
limit a previously-working config to a handful of players.
`streamWindowBytes` (optional, default 262144, clamped to [32768, 8388608]) is `streamWindowBytes` (optional, default 262144, clamped to [32768, 8388608]) is
the client's advertised per-stream receive window (§7.3). the client's advertised per-connection receive window (§7.3).
`streamResume` (optional, default true) offers STREAM_RESUME (§7.5). With it
false the client never offers the flag, never retains a byte for retransmission,
and behaves exactly as a client that predates the feature.
`resumeGraceMs` (optional, default 15000, minimum 2000) is how long a hung stream
keeps trying to reattach, clamped below the hub's advertised grace. The default
is chosen against the *backend*, not the tunnel: a hung player stops answering
the game server's KeepAlive, and vanilla disconnects a silent client at 30s, so a
longer grace would only resume sessions the backend then kicks.
`statsIntervalMs` (optional, default 0 = off) logs a periodic diagnostics line
and a per-stream summary at close, reporting how long each stream spent blocked
on the flow-control window versus the bandwidth cap, and the heartbeat round-trip
time per conn. These distinguish a slow backend from a saturated uplink from a
bad path, which throughput alone cannot.
`maxBandwidth` (optional, default unlimited) caps the aggregate rate at which the
client sends `DATA` to the hub, shared fairly across tunnels. Accepts `"20mbps"`
(decimal bit units), `"2MB/s"` (binary byte units), or a bare number of bytes per
second; the minimum is 8192 B/s. **This is a purely local policy and has no
effect on the wire format** — a shaped client is indistinguishable from a slow
one, and the hub needs no support for it. Only `DATA` is paced; control frames
are never delayed.
Each `pattern` is a regular expression (§5.1) matched against the whole Each `pattern` is a regular expression (§5.1) matched against the whole
normalized player hostname, case-insensitively. Escape literal dots (`mc\.example\.com`, normalized player hostname, case-insensitively. Escape literal dots (`mc\.example\.com`,
@@ -392,6 +650,15 @@ which is `mc\\.example\\.com` in JSON); an unescaped `.` matches any character.
Use ordinary regex to route wildcards, e.g. `.*\.example\.com` for every Use ordinary regex to route wildcards, e.g. `.*\.example\.com` for every
subdomain or `(alpha|beta)\.mc\.net` for a fixed set. subdomain or `(alpha|beta)\.mc\.net` for a fixed set.
`velocitySecret` (optional, per mapping) makes the client speak Velocity
"modern forwarding" towards that destination: during the Minecraft login phase
it swallows the backend's `velocity:player_info` Login Plugin Request and
answers with an HMAC-SHA256-signed payload carrying the player's real IP,
username and UUID (the UUID claimed in Login Start, or the offline-mode UUID
for protocols that carry none). Set it to the backend's
`proxies.velocity.secret`. This is purely client↔destination behavior — it does
not appear on the tunnel wire, and the exchange is invisible to the player.
## 10. Constants summary ## 10. Constants summary
| Name | Value | | Name | Value |
@@ -408,10 +675,17 @@ subdomain or `(alpha|beta)\.mc\.net` for a fixed set.
| pattern matching | case-insensitive, whole-string regex; first match wins | | pattern matching | case-insensitive, whole-string regex; first match wins |
| CID length | 16 bytes | | CID length | 16 bytes |
| max frame payload | 1 MiB | | max frame payload | 1 MiB |
| saturation threshold | active streams `> 8` | | worker framing | `FrameType` + payload; no stream id |
| max worker conns | `max_conn ∈ [1,8]` | | max concurrent worker conns | `maxTunnels`, default 256, clamped `[1, 4096]` |
| feature flag: per-stream flow control | `0x01` | | player IP rate / burst / concurrent | `8 /s`, burst `16`, `maxPlayersPerIp` `64` (Intent ∉ {17, 18} only) |
| stream window default / bounds | 256 KiB, clamped to [32 KiB, 8 MiB] | | feature flag: per-connection flow control | `0x01` (mandatory) |
| feature flag: worker heartbeat | `0x02` (negotiated) |
| heartbeat timeout | `3 × pingIntervalMs` |
| hub session idle timeout | 90000 ms (`0` disables) |
| connection window default / bounds | 256 KiB, clamped to [32 KiB, 8 MiB] |
| feature flag: stream resumption | 0x04 (negotiated) |
| RESUME / RESUME_ACK | 0x07 / 0x08 |
| resume grace: hub / client default | 20000 ms / 15000 ms (hub value advertised) |
| retained region per tunnel per direction | bounded by the connection window |
| WND grant batching (reference) | one grant per window/2 consumed | | WND grant batching (reference) | one grant per window/2 consumed |
| DATA chunk cap (reference) | 32 KiB | | DATA chunk cap (reference) | 32 KiB |
```
+70 -26
View File
@@ -13,9 +13,9 @@ field, so a vanilla Minecraft client needs no modification.
> made reachable from the outside. > made reachable from the outside.
``` ```
Player ──MC──▶ Hub (Java) ══ Worker Conn (mux) ══▶ Client (Go) ──MC──▶ Real MC server Player ──MC──▶ Hub (Java) ══ Worker Conn (1:1) ══▶ Client (Go) ──MC──▶ Real MC server
vanilla client public IP encrypted, pooled behind NAT (localhost) vanilla client public IP one encrypted TCP behind NAT (localhost)
per player
└────────────── Control Session ─────────┘ └────────────── Control Session ─────────┘
(pattern registration + control requests) (pattern registration + control requests)
``` ```
@@ -40,12 +40,11 @@ one or more hostname patterns — each a **regular expression**. When a player
connects to the hub with a hostname that matches a registered pattern (and any connects to the hub with a hostname that matches a registered pattern (and any
normal `Intent`), the hub assigns a random **CID**, buffers normal `Intent`), the hub assigns a random **CID**, buffers
the player's bytes, and asks the client (via the control session) to take over. the player's bytes, and asks the client (via the control session) to take over.
The client picks a **worker connection**a multiplexed, encrypted TCP link The client dials a fresh **worker connection**one encrypted TCP link per
that carries many players as lightweight *streams* — opens a stream for that CID, player — announces the CID with `SYN`, dials the real destination (optionally
dials the real destination (optionally announcing the player's real IP with the announcing the player's real IP with the **HAProxy v2** protocol), and bridges
**HAProxy v2** protocol), and bridges the two ends. Worker connections are the two ends. `maxTunnels` is the only player cap; the retired `maxConn` field
pooled: the client uses up to `maxConn` of them and always places a new stream is ignored so an old config does not silently admit only four players.
on the least-loaded one.
## Repository layout ## Repository layout
@@ -113,7 +112,7 @@ cp client/config.example.json client.json
# { # {
# "server": "hub.example.com:25565", # "server": "hub.example.com:25565",
# "psk": "same-as-the-hub", # "psk": "same-as-the-hub",
# "maxConn": 4, # "maxTunnels": 256,
# "mappings": [ # "mappings": [
# { "pattern": "mc\\.example\\.com", "destination": "127.0.0.1:25566", "proxyProtocol": true } # { "pattern": "mc\\.example\\.com", "destination": "127.0.0.1:25566", "proxyProtocol": true }
# ] # ]
@@ -126,6 +125,10 @@ just add the hub's IP with that hostname), then join `mc.example.com` in
Minecraft. The hub matches the hostname against the registered regex patterns Minecraft. The hub matches the hostname against the registered regex patterns
and tunnels you to `127.0.0.1:25566` behind the client. With `proxyProtocol: true`, the real server sees your true IP and tunnels you to `127.0.0.1:25566` behind the client. With `proxyProtocol: true`, the real server sees your true IP
(enable `proxy-protocol` / a compatible front-end on that server to consume it). (enable `proxy-protocol` / a compatible front-end on that server to consume it).
For a Paper backend, setting `velocitySecret` instead is usually nicer: the
client answers the backend's Velocity modern-forwarding login query, so the
server sees your real IP, username and UUID without any front-end — configure
the backend with `proxies.velocity.enabled: true` and the same secret.
## Container image (client) ## Container image (client)
@@ -163,6 +166,16 @@ secrets. The base image and build flags live in `.ko.yaml`.
| `psk` | *(required)* | Shared secret; must match every client. | | `psk` | *(required)* | Shared secret; must match every client. |
| `timestampWindowMs` | `30000` | Allowed clock skew for a client's rekey timestamp. | | `timestampWindowMs` | `30000` | Allowed clock skew for a client's rekey timestamp. |
| `pendingTimeoutMs` | `10000` | How long a matched player waits for a worker to take over. | | `pendingTimeoutMs` | `10000` | How long a matched player waits for a worker to take over. |
| `sessionIdleTimeoutMs` | `90000` | Close an established control/worker session that receives no frame for this long. Must exceed the client's `pingIntervalMs`; `0` disables. Player connections are unaffected. |
| `streamWindowBytes` | `262144` | Advertised per-stream receive window, clamped to [32 KiB, 8 MiB]. |
| `streamResume` | `true` | Hang a player when its worker connection drops, so the client can reattach the stream instead of the player being disconnected. `false` restores the previous behaviour exactly and retains nothing. |
| `resumeGraceMs` | `20000` | How long a hung player is held. Advertised to clients, which clamp their own retry budget below it. Must exceed the client's grace by at least one dial. |
| `maxParkedStreams` | `256` | Cap on simultaneously hung players; `maxParkedBytes` (default `maxParkedStreams × 2 × streamWindowBytes`) caps what they retain. Past either, the oldest are dropped. |
| `statsIntervalMs` | `0` (off) | Log a periodic line with live/hung stream counts, retained bytes and pattern count. |
| `registrationGraceMs` | `15000` | Keep a closed control session's routes as *orphaned* for this long, holding players that arrive on them instead of refusing them, and replaying their requests once the client re-registers. `0` disables it. |
| `playerRatePerSec` | `8` | Per-IP token-bucket rate for **player** connections only (Intent ∉ {17, 18}). Unmatched hostnames still consume a token. `0` disables. |
| `playerBurst` | `16` | Token-bucket depth for `playerRatePerSec`. |
| `maxPlayersPerIp` | `64` | Concurrent player sockets (pending + live + parked) from one IP. `0` disables. Intent 17 is never counted. |
### Client (`client/config.example.json`) ### Client (`client/config.example.json`)
@@ -170,12 +183,18 @@ secrets. The base image and build flags live in `.ko.yaml`.
|------------------|--------------------|---------| |------------------|--------------------|---------|
| `server` | *(required)* | Hub `host:port`. | | `server` | *(required)* | Hub `host:port`. |
| `psk` | *(required)* | Shared secret; must match the hub. | | `psk` | *(required)* | Shared secret; must match the hub. |
| `maxConn` | `1` (clamped 18) | Max worker connections in the pool. | | `maxTunnels` | `256` (clamped 14096) | Max concurrent player tunnels. The retired `maxConn` field is ignored. |
| `pingIntervalMs` | `20000` | Control-session keepalive interval. | | `pingIntervalMs` | `20000` (min 1000) | Heartbeat interval for the control session and every worker conn. A session with no reply for `3×` this is dropped and re-established. |
| `maxBandwidth` | *(unlimited)* | Caps what the client uploads to the hub, summed over every player — the direction carrying the game server's output, and the one a home uplink runs out of first. `"20mbps"`, `"512kbps"`, `"2MB/s"`, or a bare number of bytes/sec. **Bit units are decimal (`20mbps` = 20,000,000 bit/s); byte units are binary (`2MB/s` = 2 MiB/s).** Set it slightly below your real upload speed — framing and TCP/IP overhead are not counted. The budget is shared fairly across players, so one person loading chunks cannot time the others out. |
| `streamWindowBytes` | `262144` | Advertised per-stream receive window, clamped to [32 KiB, 8 MiB]. |
| `streamResume` | `true` | Reattach streams over a fresh connection when a worker connection drops, instead of disconnecting those players. `false` restores the previous behaviour exactly: nothing is retained and the send path is unchanged. |
| `resumeGraceMs` | `15000` (min 2000) | How long a stream keeps trying to reattach, clamped below the hub's advertised grace. Sized against the *backend*: a hung player stops answering the game server's KeepAlive, and vanilla disconnects a silent client at 30s, so a longer grace only resumes sessions the backend then kicks. |
| `statsIntervalMs` | `0` (off) | Log a periodic diagnostics line, plus a summary per stream at close: bytes each way, how long the stream was blocked on the flow-control window versus the bandwidth cap, receive-queue high-water mark, and heartbeat round-trip time per connection. Those distinguish a slow backend from a saturated uplink from a bad path, which throughput alone cannot. |
| `mappings[]` | *(≥1 required)* | Route table (below). | | `mappings[]` | *(≥1 required)* | Route table (below). |
| `mappings[].pattern` | — | Regex matched against the whole player hostname, case-insensitively. Escape dots (`mc\.example\.com`); `.` is a wildcard. | | `mappings[].pattern` | — | Regex matched against the whole player hostname, case-insensitively. Escape dots (`mc\.example\.com`); `.` is a wildcard. |
| `mappings[].destination` | — | Real server `host:port` to forward to. | | `mappings[].destination` | — | Real server `host:port` to forward to. |
| `mappings[].proxyProtocol` | `false` | Prepend a HAProxy v2 header carrying the player's IP. | | `mappings[].proxyProtocol` | `false` | Prepend a HAProxy v2 header carrying the player's IP. |
| `mappings[].velocitySecret` | *(off)* | Answer the destination's [Velocity modern forwarding](https://docs.papermc.io/velocity/player-information-forwarding/) login query with this secret, forwarding the player's real IP, username and UUID. Match it to the backend's `proxies.velocity.secret` (Paper). The forwarded profile carries no skin properties — the tunnel performs no Mojang authentication. |
## Testing ## Testing
@@ -199,14 +218,23 @@ go test ./e2e/... -v
The e2e suite covers: a full player round-trip with verbatim handshake The e2e suite covers: a full player round-trip with verbatim handshake
forwarding and case-insensitive matching, regex wildcard pattern routing, forwarding and case-insensitive matching, regex wildcard pattern routing,
multi-megabyte transfers, concurrent multi-megabyte transfers, concurrent players each on their own worker
streams spreading across multiple worker connections, HAProxy v2 source-address connection, HAProxy v2 source-address
propagation, player- and destination-initiated disconnect propagation, wrong-PSK propagation, Velocity modern-forwarding interception (signed player-info
rejection, dropping of unmatched hostnames, stream isolation under a slow handoff to a mock Paper backend), player- and destination-initiated disconnect
player and under a slow destination (no head-of-line blocking), and rejection propagation, wrong-PSK rejection, dropping of unmatched hostnames, isolation
of pre-flow-control peers. The Go and Java crypto layers are under a slow player and under a slow destination (no head-of-line blocking),
independently pinned to the same SHA3-224 test vector so they cannot silently rejection of pre-flow-control peers, and stream resumption — a tunnel
drift apart. hard-reset mid-transfer with the player connection held open, asserting the
byte stream neither gains nor loses a byte, across concurrent players, plus
grace expiry and the resume-disabled path, and control-outage handling — a
player arriving while the client's control session is down is held and then
served once it re-registers, with the grace-disabled and grace-expired paths
pinned too, and the per-IP player limiter — burst overflow and
`maxPlayersPerIp` drop extras before they become pending, Intent 17 is never
counted, and both knobs at `0` restore the unlimited path. The Go and Java
crypto layers are independently pinned to the same SHA3-224 test vector so they
cannot silently drift apart.
## Design notes & limitations ## Design notes & limitations
@@ -214,12 +242,28 @@ drift apart.
ChaCha20-encrypted (no AEAD tag) to minimize overhead. This protects against ChaCha20-encrypted (no AEAD tag) to minimize overhead. This protects against
casual sniffing, not a determined active attacker (see the note at the top of casual sniffing, not a determined active attacker (see the note at the top of
[PROTOCOL.md](PROTOCOL.md) and [docs/architecture.md](docs/architecture.md) §8). [PROTOCOL.md](PROTOCOL.md) and [docs/architecture.md](docs/architecture.md) §8).
* **Per-stream flow control.** Each stream has credit-based windows in both * **Per-tunnel flow control.** Each worker connection has credit-based windows
directions (windows exchanged at session setup, default 256 KiB), so a slow in both directions (windows exchanged at session setup, default 256 KiB), so
player or slow destination jams only its own stream at a bounded buffer — no a slow player or slow destination jams only its own tunnel at a bounded
application-level head-of-line blocking between streams. What remains is buffer. There is no application-level head-of-line blocking: each player
TCP-level HOL (packet loss stalls a whole worker connection briefly); owns a TCP connection.
raising `maxConn` spreads that. * **Liveness is explicit.** Every session heartbeats, every socket write is
bounded, and session establishment has a deadline. A path that dies silently —
no `FIN`, no `RST`, as when a NAT or firewall forgets an established flow — is
detected within `3 × pingIntervalMs`, the dead connection is dropped, and
service is restored without operator action. TCP keepalive is on as a
second line of defence.
* **A control-session reconnect no longer refuses new players.** While a client
is reconnecting the hub has no route for it, so arriving players used to be
told there is no such server. Those routes are now held briefly and the
players with them, then served once the client re-registers.
* **A dropped tunnel no longer drops the players.** A worker connection is only
the middle leg of the player it carries; when it dies both terminal sockets
are usually still healthy. The hub now hangs that player while the client
reattaches the tunnel over a fresh connection, replaying byte-exactly from
the offset the peer reports, so a conntrack expiry costs a stall rather than
a disconnect. Negotiated, and `streamResume: false` on either side restores
the old behaviour.
* **Single hub event loop.** The hub deploys one Vert.x verticle, so all state * **Single hub event loop.** The hub deploys one Vert.x verticle, so all state
is confined to one event loop (no locking). Throughput is bounded by one core; is confined to one event loop (no locking). Throughput is bounded by one core;
ample for hundreds of players, not designed for tens of thousands. ample for hundreds of players, not designed for tens of thousands.
+244 -52
View File
@@ -8,13 +8,14 @@ import (
"log" "log"
"net" "net"
"sync" "sync"
"sync/atomic"
"time" "time"
"github.com/iceBear67/redapricot/client/wire" "github.com/iceBear67/redapricot/client/wire"
) )
// Client is a redapricot client: it holds a control session with the hub and a // Client is a redapricot client: it holds a control session with the hub and
// pool of worker connections used to serve player streams. // dials one worker connection per player.
type Client struct { type Client struct {
cfg *Config cfg *Config
pskBytes []byte pskBytes []byte
@@ -24,7 +25,21 @@ type Client struct {
mappings map[string]Mapping // normalized pattern -> mapping mappings map[string]Mapping // normalized pattern -> mapping
pool *WorkerPool pool *WorkerPool
streamWnd int // our advertised per-stream receive window (bytes) // ctx/cancel own every worker dial: Close cancels it so an in-flight dial
// aborts instead of holding a goroutine for the whole handshake timeout.
// The control path uses the caller's context from Start, which is the same
// shutdown signal by convention.
ctx context.Context
cancel context.CancelFunc
// closing is set by Close; a conn-loss teardown checks it and closes
// streams outright rather than parking them for a reattach that is never
// coming. Dial also consults it through the pool's own flag.
closing atomic.Bool
streamWnd int // our advertised per-connection receive window (bytes)
shaper *Shaper // caps aggregate egress to the hub; nil when unlimited
chunk int // DATA payload cap; shrinks below DataChunkSize at low rates
mu sync.Mutex mu sync.Mutex
ctrl *wire.FramedConn ctrl *wire.FramedConn
@@ -32,11 +47,14 @@ type Client struct {
// New builds a client from config. // New builds a client from config.
func New(cfg *Config) *Client { func New(cfg *Config) *Client {
ctx, cancel := context.WithCancel(context.Background())
c := &Client{ c := &Client{
cfg: cfg, cfg: cfg,
pskBytes: []byte(cfg.PSK), pskBytes: []byte(cfg.PSK),
pskAddr: wire.PSKAddress([]byte(cfg.PSK)), pskAddr: wire.PSKAddress([]byte(cfg.PSK)),
mappings: make(map[string]Mapping), mappings: make(map[string]Mapping),
ctx: ctx,
cancel: cancel,
} }
if _, portStr, err := net.SplitHostPort(cfg.Server); err == nil { if _, portStr, err := net.SplitHostPort(cfg.Server); err == nil {
if p, err := net.LookupPort("tcp", portStr); err == nil { if p, err := net.LookupPort("tcp", portStr); err == nil {
@@ -47,10 +65,33 @@ func New(cfg *Config) *Client {
c.mappings[NormalizeAddress(m.Pattern)] = m c.mappings[NormalizeAddress(m.Pattern)] = m
} }
c.streamWnd = clampWindow(cfg.StreamWindowBytes) c.streamWnd = clampWindow(cfg.StreamWindowBytes)
c.pool = newWorkerPool(c, cfg.MaxConn) // Parsed here rather than in LoadConfig because a Config may also be built
// directly (tests). LoadConfig has already rejected a malformed value on the
// file path, so a failure here can only come from a hand-built Config.
bps, err := parseBandwidth(cfg.MaxBandwidth)
if err != nil {
log.Printf("client: %v; continuing without a bandwidth limit", err)
}
c.shaper = NewShaper(bps)
c.chunk = c.shaper.chunkSize()
if c.shaper != nil {
log.Printf("egress shaped to %d B/s (burst %d B, chunk %d B)",
bps, int64(c.shaper.burst), c.shaper.chunk)
}
c.pool = newWorkerPool(c, clampMaxTunnels(cfg.MaxTunnels))
return c return c
} }
func clampMaxTunnels(n int) int {
if n < 1 {
return DefaultMaxTunnels
}
if n > MaxMaxTunnels {
return MaxMaxTunnels
}
return n
}
func clampWindow(w int) int { func clampWindow(w int) int {
if w <= 0 { if w <= 0 {
return DefaultStreamWindow return DefaultStreamWindow
@@ -64,17 +105,45 @@ func clampWindow(w int) int {
return w return w
} }
// session is an established redapricot session: the frame transport plus what
// was negotiated during establishment.
type session struct {
fc *wire.FramedConn
peerWnd int // hub's advertised per-connection receive window
heartbeat bool // hub accepted connection-level PING/PONG on worker conns
resume bool // hub accepted stream resumption (§7.5)
// hubGrace is how long the hub will hang a parked player, as advertised in
// SessionReady. Zero when resumption was not negotiated.
hubGrace time.Duration
}
// dialSession opens a TCP connection, performs the Intent-17 handshake, the // dialSession opens a TCP connection, performs the Intent-17 handshake, the
// Phase-A rekey, and reads SessionReady, returning an established frame conn // Phase-A rekey, and reads SessionReady. Per-connection flow control is mandatory:
// and the hub's advertised per-stream receive window. Per-stream flow control // a hub that does not echo the STREAM_FC flag is rejected.
// is mandatory: a hub that does not echo the STREAM_FC flag is rejected. //
func (c *Client) dialSession(magic byte) (fc *wire.FramedConn, peerWnd int, err error) { // The whole exchange is bounded by HandshakeTimeout. A hub that accepts the
conn, err := net.DialTimeout("tcp", c.cfg.Server, 10*time.Second) // socket but never answers (wedged event loop, a load balancer accepting on its
// behalf) must fail fast rather than park the caller forever.
//
// ctx bounds the dial: the control path passes the caller's context so a
// shutdown mid-handshake aborts the attempt, and worker dials pass the client's
// own context so Close cancels in-flight dials. After the dial, a cancelled
// ctx keeps aborting by closing the conn underneath the deadline-bounded
// handshake I/O.
func (c *Client) dialSession(ctx context.Context, magic byte) (sess *session, err error) {
d := &net.Dialer{Timeout: HandshakeTimeout}
conn, err := d.DialContext(ctx, "tcp", c.cfg.Server)
if err != nil { if err != nil {
return nil, 0, err return nil, err
} }
// When ctx ends (shutdown), close the conn so the handshake below fails
// immediately instead of waiting out its deadline.
stop := context.AfterFunc(ctx, func() { _ = conn.Close() })
defer stop()
if tcp, ok := conn.(*net.TCPConn); ok { if tcp, ok := conn.(*net.TCPConn); ok {
_ = tcp.SetNoDelay(true) _ = tcp.SetNoDelay(true)
_ = tcp.SetKeepAlive(true)
_ = tcp.SetKeepAlivePeriod(TCPKeepAlivePeriod)
} }
ok := false ok := false
defer func() { defer func() {
@@ -82,30 +151,37 @@ func (c *Client) dialSession(magic byte) (fc *wire.FramedConn, peerWnd int, err
_ = conn.Close() _ = conn.Close()
} }
}() }()
if err := conn.SetDeadline(time.Now().Add(HandshakeTimeout)); err != nil {
return nil, err
}
// 1. plaintext Minecraft Handshake, Intent 17, address = hex(SHA3-224(PSK)). // 1. plaintext Minecraft Handshake, Intent 17, address = hex(SHA3-224(PSK)).
hs := wire.BuildHandshake(ProtocolVersion, c.pskAddr, c.serverPort, IntentRedapricot) hs := wire.BuildHandshake(ProtocolVersion, c.pskAddr, c.serverPort, IntentRedapricot)
if _, err := conn.Write(hs); err != nil { if _, err := conn.Write(hs); err != nil {
return nil, 0, err return nil, err
} }
// 2. Phase-A ciphers derived from the PSK. // 2. Phase-A ciphers derived from the PSK.
fc = wire.NewFramedConn(conn, fc := wire.NewFramedConn(conn,
wire.CipherFor(c.pskBytes, wire.DirS2C), // in: server -> client wire.CipherFor(c.pskBytes, wire.DirS2C), // in: server -> client
wire.CipherFor(c.pskBytes, wire.DirC2S), // out: client -> server wire.CipherFor(c.pskBytes, wire.DirC2S), // out: client -> server
) )
// 3. Rekey frame (Phase A), including the mandatory feature flags and our // 3. Rekey frame (Phase A), including the mandatory feature flags and our
// per-stream receive window. // per-connection receive window.
rnd := make([]byte, 16) rnd := make([]byte, 16)
if _, err := crand.Read(rnd); err != nil { if _, err := crand.Read(rnd); err != nil {
return nil, 0, err return nil, err
} }
ts := time.Now().UnixMilli() ts := time.Now().UnixMilli()
offered := FlagStreamFC | FlagWorkerHeartbeat
if c.cfg.resumeEnabled() {
offered |= FlagStreamResume
}
rekeyMsg := wire.NewWriter().U8(magic).VarInt(len(rnd)).Bytes(rnd).I64(ts). rekeyMsg := wire.NewWriter().U8(magic).VarInt(len(rnd)).Bytes(rnd).I64(ts).
VarInt(FlagStreamFC).VarInt(c.streamWnd).Out() VarInt(offered).VarInt(c.streamWnd).Out()
if err := fc.WriteFrame(rekeyMsg); err != nil { if err := fc.WriteFrame(rekeyMsg); err != nil {
return nil, 0, err return nil, err
} }
// 4. Switch to Phase-B ciphers: REKEY = Rand || Timestamp(I64 BE). // 4. Switch to Phase-B ciphers: REKEY = Rand || Timestamp(I64 BE).
@@ -120,49 +196,97 @@ func (c *Client) dialSession(magic byte) (fc *wire.FramedConn, peerWnd int, err
) )
// 5. SessionReady: the type byte followed by the hub's accepted flags and // 5. SessionReady: the type byte followed by the hub's accepted flags and
// its per-stream receive window. Both are required. // its per-connection receive window. Both are required.
payload, err := fc.ReadFrame() payload, err := fc.ReadFrame()
if err != nil { if err != nil {
return nil, 0, err return nil, err
} }
if len(payload) < 1 || payload[0] != CtlSessionReady { if len(payload) < 1 || payload[0] != CtlSessionReady {
return nil, 0, fmt.Errorf("expected SessionReady, got %v", payload) return nil, fmt.Errorf("expected SessionReady, got %v", payload)
} }
r := wire.NewReader(payload[1:]) r := wire.NewReader(payload[1:])
flags, ferr := r.VarInt() flags, ferr := r.VarInt()
hubWnd, werr := r.VarInt() hubWnd, werr := r.VarInt()
if ferr != nil || werr != nil || flags&FlagStreamFC == 0 || hubWnd <= 0 { if ferr != nil || werr != nil || flags&FlagStreamFC == 0 || hubWnd <= 0 {
return nil, 0, fmt.Errorf("hub did not accept per-stream flow control (unsupported hub version?)") return nil, fmt.Errorf("hub did not accept per-connection flow control (unsupported hub version?)")
} }
if hubWnd > MaxStreamWindow { if hubWnd > MaxStreamWindow {
hubWnd = MaxStreamWindow hubWnd = MaxStreamWindow
} }
// Resumption is negotiated per connection, and the hub's grace period rides
// along when it accepts. Our own grace is clamped strictly under the hub's:
// the client must always give up first, or the hub drops a hanging player
// while we are still mid-reattach. A hub that accepts the flag but omits the
// grace is treated as not supporting it at all rather than guessed at.
resume := flags&FlagStreamResume != 0
var hubGrace time.Duration
if resume {
graceMs, gerr := r.VarInt()
if gerr != nil || graceMs <= 0 {
log.Printf("hub accepted stream resume without advertising a grace period; disabling resume")
resume = false
} else {
hubGrace = time.Duration(graceMs) * time.Millisecond
}
}
// The session is live: drop the establishment deadline. From here on
// liveness is the heartbeat's job (and WriteFrame bounds each write).
if err := conn.SetDeadline(time.Time{}); err != nil {
return nil, err
}
ok = true ok = true
return fc, hubWnd, nil return &session{
fc: fc,
peerWnd: hubWnd,
heartbeat: flags&FlagWorkerHeartbeat != 0,
resume: resume,
hubGrace: hubGrace,
}, nil
} }
// statsOn reports whether performance diagnostics are enabled. When off, no
// counter struct is ever allocated and the instrumentation is a single branch.
func (c *Client) statsOn() bool { return c.cfg.StatsIntervalMs > 0 }
// Start establishes the control session and registers all patterns. It returns // Start establishes the control session and registers all patterns. It returns
// once the initial connection succeeds; subsequent drops are handled in the // once the initial connection succeeds; subsequent drops are handled in the
// background with reconnect. // background with reconnect.
func (c *Client) Start(ctx context.Context) error { func (c *Client) Start(ctx context.Context) error {
return c.connectControl(ctx) if err := c.connectControl(ctx); err != nil {
return err
}
if c.statsOn() {
go c.statsLoop(ctx.Done())
}
return nil
} }
func (c *Client) connectControl(ctx context.Context) error { func (c *Client) connectControl(ctx context.Context) error {
fc, _, err := c.dialSession(MagicControl) sess, err := c.dialSession(ctx, MagicControl)
if err != nil { if err != nil {
return fmt.Errorf("control connect: %w", err) return fmt.Errorf("control connect: %w", err)
} }
c.registerAll(fc) ctrl := &ctrlSession{fc: sess.fc}
ctrl.lastPong.Store(time.Now().UnixMilli())
c.registerAll(sess.fc)
c.mu.Lock() c.mu.Lock()
c.ctrl = fc c.ctrl = sess.fc
c.mu.Unlock() c.mu.Unlock()
log.Printf("control session established with %s", c.cfg.Server) log.Printf("control session established with %s", c.cfg.Server)
go c.serveControl(ctx, fc) go c.serveControl(ctx, ctrl)
go c.pingLoop(ctx, fc) go c.pingLoop(ctx, ctrl)
return nil return nil
} }
// ctrlSession tracks liveness for one control connection. A control session
// whose path dies silently must be detected, otherwise the hub keeps routing
// players to a session the client will never read from and nobody can connect.
type ctrlSession struct {
fc *wire.FramedConn
lastPong atomic.Int64 // unix ms of the most recent Pong
}
func (c *Client) registerAll(fc *wire.FramedConn) { func (c *Client) registerAll(fc *wire.FramedConn) {
for pattern := range c.mappings { for pattern := range c.mappings {
msg := wire.NewWriter().U8(CtlRegister).String(pattern).Out() msg := wire.NewWriter().U8(CtlRegister).String(pattern).Out()
@@ -174,33 +298,54 @@ func (c *Client) registerAll(fc *wire.FramedConn) {
} }
} }
func (c *Client) serveControl(ctx context.Context, fc *wire.FramedConn) { func (c *Client) serveControl(ctx context.Context, ctrl *ctrlSession) {
for { for {
payload, err := fc.ReadFrame() payload, err := ctrl.fc.ReadFrame()
if err != nil { if err != nil {
break break
} }
c.dispatchControl(payload) c.dispatchControl(ctrl, payload)
} }
_ = fc.Close() _ = ctrl.fc.Close()
if ctx.Err() != nil { if ctx.Err() != nil || c.closing.Load() {
return return
} }
// Reconnect with backoff. // Reconnect with backoff, but try immediately first. While the control
for backoff := 500 * time.Millisecond; ctx.Err() == nil; backoff *= 2 { // session is down the hub has no live route for this client, so every
if backoff > 10*time.Second { // millisecond of delay is a player arriving to be told there is no such
backoff = 10 * time.Second // server — and a session usually dies to a transient blip that the very next
// dial would have survived. Sleeping first spent that window unconditionally.
//
// The wait is on ctx rather than time.Sleep so shutdown is not held up by a
// backoff that has grown to the cap. Close() is authoritative on its own:
// it gates the loop directly because a caller may shut the client down by
// calling Close() without cancelling the context it passed to Start.
for backoff := time.Duration(0); ctx.Err() == nil && !c.closing.Load(); {
if backoff > 0 {
select {
case <-ctx.Done():
return
case <-time.After(backoff):
}
}
if c.closing.Load() {
return
} }
time.Sleep(backoff)
if err := c.connectControl(ctx); err == nil { if err := c.connectControl(ctx); err == nil {
return return
} else { } else {
log.Printf("control reconnect failed: %v", err) log.Printf("control reconnect failed: %v", err)
} }
switch {
case backoff == 0:
backoff = 500 * time.Millisecond
case backoff < maxControlBackoff:
backoff = min(backoff*2, maxControlBackoff)
}
} }
} }
func (c *Client) dispatchControl(payload []byte) { func (c *Client) dispatchControl(ctrl *ctrlSession, payload []byte) {
r := wire.NewReader(payload) r := wire.NewReader(payload)
t, err := r.U8() t, err := r.U8()
if err != nil { if err != nil {
@@ -212,7 +357,14 @@ func (c *Client) dispatchControl(payload []byte) {
case CtlRegisterAck: case CtlRegisterAck:
pattern, _ := r.String() pattern, _ := r.String()
status, _ := r.U8() status, _ := r.U8()
log.Printf("register ack %q status=%d", pattern, status) switch status {
case RegisterOk:
log.Printf("pattern %q registered", pattern)
case RegisterErrPattern:
log.Printf("pattern %q rejected: not a valid regular expression", pattern)
default:
log.Printf("pattern %q rejected: status=%d", pattern, status)
}
case CtlControlRequest: case CtlControlRequest:
cid, err := r.Bytes(CIDLen) cid, err := r.Bytes(CIDLen)
if err != nil { if err != nil {
@@ -223,30 +375,41 @@ func (c *Client) dispatchControl(payload []byte) {
port, _ := r.U16() port, _ := r.U16()
go c.handleControlRequest(cid, pattern, ip, int(port)) go c.handleControlRequest(cid, pattern, ip, int(port))
case CtlPong: case CtlPong:
// ignore ctrl.lastPong.Store(time.Now().UnixMilli())
default: default:
log.Printf("control: unknown message type %d", t) log.Printf("control: unknown message type %d", t)
} }
} }
func (c *Client) pingLoop(ctx context.Context, fc *wire.FramedConn) { // pingLoop keeps the control session alive and, crucially, verifies that the
ticker := time.NewTicker(time.Duration(c.cfg.PingIntervalMs) * time.Millisecond) // hub is still answering. A path that dies silently (no FIN/RST) would
// otherwise leave the read loop parked forever: the client would believe it is
// still registered while the hub routes players into the void.
func (c *Client) pingLoop(ctx context.Context, ctrl *ctrlSession) {
ticker := time.NewTicker(c.cfg.pingInterval())
defer ticker.Stop() defer ticker.Stop()
timeout := c.cfg.heartbeatTimeout()
for { for {
select { select {
case <-ctx.Done(): case <-ctx.Done():
return return
case <-ticker.C: case <-ticker.C:
last := time.UnixMilli(ctrl.lastPong.Load())
if time.Since(last) > timeout {
log.Printf("control session silent for %s; dropping it to force a reconnect", time.Since(last).Round(time.Second))
_ = ctrl.fc.Close() // unblocks serveControl, which reconnects
return
}
msg := wire.NewWriter().U8(CtlPing).I64(time.Now().UnixMilli()).Out() msg := wire.NewWriter().U8(CtlPing).I64(time.Now().UnixMilli()).Out()
if err := fc.WriteFrame(msg); err != nil { if err := ctrl.fc.WriteFrame(msg); err != nil {
return return
} }
} }
} }
} }
// handleControlRequest reacts to a matched player: allocate a worker stream, // handleControlRequest reacts to a matched player: dial a dedicated worker
// SYN it, and bridge it to the mapped destination. // conn, SYN it, and bridge it to the mapped destination.
func (c *Client) handleControlRequest(cid []byte, pattern, ip string, port int) { func (c *Client) handleControlRequest(cid []byte, pattern, ip string, port int) {
mapping, ok := c.mappings[NormalizeAddress(pattern)] mapping, ok := c.mappings[NormalizeAddress(pattern)]
if !ok { if !ok {
@@ -254,24 +417,52 @@ func (c *Client) handleControlRequest(cid []byte, pattern, ip string, port int)
return return
} }
log.Printf("player %s:%d joined via pattern %q -> %s", ip, port, pattern, mapping.Destination) log.Printf("player %s:%d joined via pattern %q -> %s", ip, port, pattern, mapping.Destination)
wc, sid, err := c.pool.Allocate()
if err != nil { // Dial and attach must agree on a live conn: Dial hands out a conn that can
log.Printf("worker allocate failed: %v", err) // die before we attach on it, which would strand the stream on a conn
// nothing iterates. attach reports that, and we simply dial another.
var st *Stream
var wc *WorkerConn
for attempt := 0; attempt < dialAttempts; attempt++ {
var err error
wc, err = c.pool.Dial()
if err != nil {
log.Printf("worker dial failed: %v", err)
return
}
st = newStream(c, wc, cid, mapping, ip, port)
// Attach before SYN so inbound DATA can never race ahead of the binding,
// and start the pumps before the (bounded) SYN write so a failed or slow
// SYN cannot strand a stream that nothing would ever tear down.
if wc.attach(st) {
break
}
_ = wc.fc.Close()
st = nil
}
if st == nil {
log.Printf("worker dial failed: no live conn after %d attempts", dialAttempts)
return return
} }
st := newStream(wc, sid, cid, mapping, ip, port)
wc.registerStream(sid, st)
wc.sendSyn(sid, cid)
go st.writeLoop() go st.writeLoop()
go st.run() go st.run()
if err := wc.sendSyn(cid); err != nil {
log.Printf("stream %s: SYN failed: %v", st.name(), err)
st.teardown(false)
}
} }
// WorkerConnCount reports the current number of open worker connections // WorkerConnCount reports the current number of open worker connections
// (exposed for tests/observability). // (exposed for tests/observability).
func (c *Client) WorkerConnCount() int { return c.pool.count() } func (c *Client) WorkerConnCount() int { return c.pool.count() }
// Close tears down the control session and all worker connections. // Close tears down the control session and all worker connections. Idempotent:
// a second call (or a Close racing a reconnect) only re-closes what is still
// open.
func (c *Client) Close() { func (c *Client) Close() {
c.closing.Store(true)
c.cancel() // aborts in-flight worker dials
c.mu.Lock() c.mu.Lock()
fc := c.ctrl fc := c.ctrl
c.mu.Unlock() c.mu.Unlock()
@@ -279,4 +470,5 @@ func (c *Client) Close() {
_ = fc.Close() _ = fc.Close()
} }
c.pool.closeAll() c.pool.closeAll()
c.shaper.Stop()
} }
+5 -1
View File
@@ -1,9 +1,13 @@
{ {
"server": "hub.example.com:25565", "server": "hub.example.com:25565",
"psk": "change-me-to-a-long-random-passphrase", "psk": "change-me-to-a-long-random-passphrase",
"maxConn": 4, "maxTunnels": 256,
"pingIntervalMs": 20000, "pingIntervalMs": 20000,
"streamWindowBytes": 262144, "streamWindowBytes": 262144,
"maxBandwidth": "",
"streamResume": true,
"resumeGraceMs": 15000,
"statsIntervalMs": 0,
"mappings": [ "mappings": [
{ {
"pattern": "mc\\.example\\.com", "pattern": "mc\\.example\\.com",
+263 -17
View File
@@ -4,13 +4,20 @@ import (
"encoding/json" "encoding/json"
"fmt" "fmt"
"os" "os"
"strconv"
"strings" "strings"
"time"
) )
// Protocol constants (mirror of the Java Protocol class; see PROTOCOL.md). // Protocol constants (mirror of the Java Protocol class; see PROTOCOL.md).
const ( const (
IntentRedapricot = 17 IntentRedapricot = 17
ProtocolVersion = 767 // arbitrary; the hub ignores it // IntentReserved is the handshake intent reserved for redapricot
// management/status (mirror of Protocol.INTENT_RESERVED). The hub never
// pattern-matches it; it replies with a Minecraft status line and closes,
// so an operator can probe the port without joining the protocol.
IntentReserved = 18
ProtocolVersion = 767 // arbitrary; the hub ignores it
MagicControl = 0x01 MagicControl = 0x01
MagicWorker = 0x02 MagicWorker = 0x02
@@ -25,45 +32,268 @@ const (
CtlPing = 0x05 CtlPing = 0x05
CtlPong = 0x06 CtlPong = 0x06
// RegisterAck status codes (mirror of Protocol.REGISTER_OK/_ERR_PATTERN).
RegisterOk = 0x00
RegisterErrPattern = 0x01 // pattern is not a valid regular expression
MuxSyn = 0x00 MuxSyn = 0x00
MuxData = 0x01 MuxData = 0x01
MuxFin = 0x02 MuxFin = 0x02
MuxRst = 0x03 MuxRst = 0x03
MuxWnd = 0x04 MuxWnd = 0x04
MuxPing = 0x05
MuxPong = 0x06
// MuxResume reattaches a parked player to this conn (CID + our accepted
// offset); MuxResumeAck carries the hub's accepted offset and a fresh CID.
MuxResume = 0x07
MuxResumeAck = 0x08
// RST reason codes (optional trailing byte; absence means "unspecified").
// Distinguishing them matters for resume: an unknown stream is terminal,
// while "already bound" means the hub has the player on another conn — the
// reattach retries until that bind dies and the hub re-parks the player.
RstUnspecified = 0x00
RstUnknownStream = 0x01
RstAlreadyBound = 0x02
RstResumeAbandoned = 0x03
RstFlowControl = 0x04
RstDialFailed = 0x05
FrameError = 0x7F FrameError = 0x7F
SaturationThreshold = 8 // DefaultMaxTunnels / MaxMaxTunnels bound concurrent 1:1 worker conns.
// The old mux-era maxConn cap of 8 would silently become "8 players".
DefaultMaxTunnels = 256
MaxMaxTunnels = 4096
// Session-establishment feature flags (trailing VarInt on the Rekey message). // Session-establishment feature flags (trailing VarInt on the Rekey message).
FlagStreamFC = 0x01 FlagStreamFC = 0x01
// FlagWorkerHeartbeat enables connection-level PING/PONG on worker conns.
// Without it a worker conn whose path is silently blackholed (NAT/conntrack
// drop, firewall) is never detected: the read loop parks forever and that
// player is stuck until the client restarts.
FlagWorkerHeartbeat = 0x02
// FlagStreamResume enables stream resumption (PROTOCOL.md §7.5): a worker
// conn drop parks the player instead of killing them, the hub hangs the
// player socket, and the client reattaches byte-exactly over a fresh conn.
// Negotiated, so either side may decline and get today's behaviour
// (immediate teardown) unchanged.
FlagStreamResume = 0x04
// Per-stream flow-control window bounds (bytes). The advertised window is the // Per-connection flow-control window bounds (bytes). The advertised window
// receiver's promise of how much un-credited DATA it will buffer per stream. // is the receiver's promise of how much un-credited DATA it will buffer.
DefaultStreamWindow = 256 * 1024 DefaultStreamWindow = 256 * 1024
MinStreamWindow = 32 * 1024 MinStreamWindow = 32 * 1024
MaxStreamWindow = 8 << 20 MaxStreamWindow = 8 << 20
// DataChunkSize caps a single DATA frame's payload so no stream monopolizes // DataChunkSize caps a single DATA frame's payload so one write cannot
// the shared worker connection for long. // occupy the link for a full 1-MiB frame.
DataChunkSize = 32 * 1024 DataChunkSize = 32 * 1024
) )
// Egress bandwidth shaping (see shaper.go and docs/architecture.md §6). These
// are entirely client-local: nothing here appears on the wire.
const (
// MinBandwidth floors a configured cap. Below this the tunnel cannot carry a
// Minecraft session at all, so such a value is a unit typo ("20bps" for
// "20mbps") and is rejected rather than silently clamped.
MinBandwidth = 8 * 1024
// ShaperBurstSeconds is how much transmission time the token bucket banks
// while idle. Big enough to absorb a chunk-load spike; small enough that
// releasing it cannot overrun the physical uplink and rebuild the standing
// queue the cap exists to prevent.
ShaperBurstSeconds = 0.2
// MinShaperBurst must exceed DataChunkSize: a request larger than the bucket
// could never be afforded and would park forever.
MinShaperBurst = 64 * 1024
MaxShaperBurst = 4 << 20
// ShaperSliceSeconds bounds how long one stream holds the link before the
// scheduler can switch, by sizing the send chunk to that much transmission
// time. Above ~13 Mbps this yields DataChunkSize and nothing changes.
ShaperSliceSeconds = 0.02
MinShaperChunk = 4 * 1024
)
// Timeouts. Every tunnel socket is covered by one of these: without them a
// silently dropped path (no FIN/RST) leaves the client parked forever.
const (
// HandshakeTimeout bounds session establishment end to end — the TCP dial,
// the Rekey write and the SessionReady read. A hub that accepts the socket
// but never answers must not park the caller (and, for the pool, every other
// player behind it) indefinitely.
HandshakeTimeout = 15 * time.Second
// TCPKeepAlivePeriod asks the kernel to probe idle tunnel sockets, so a peer
// that becomes unreachable is detected even when no frames are in flight.
TCPKeepAlivePeriod = 30 * time.Second
// MissedHeartbeats is how many ping intervals may pass with no reply before
// a session is declared dead and dropped.
MissedHeartbeats = 3
// DefaultPingIntervalMs is used when a config carries no (or a
// non-positive) interval. LoadConfig applies the same default.
DefaultPingIntervalMs = 20000
// MinPingIntervalMs floors the configured ping interval so the derived
// heartbeat timeout can never be short enough to cause spurious drops.
// Applied wherever the interval is read, not just on the file path: a
// hand-built Config (tests) carrying 0 would otherwise panic
// time.NewTicker at the call site.
MinPingIntervalMs = 1000
// DefaultResumeGraceMs is how long a parked stream keeps trying to reattach
// before giving up and closing the destination.
//
// Chosen against the backend, not the tunnel: a hung player stops answering
// the game server's KeepAlive, and vanilla disconnects a silent client at
// 30s. A longer grace would resume sessions the backend then kicks anyway.
DefaultResumeGraceMs = 15000
// MinResumeGraceMs floors the grace so it can always fit at least one dial;
// a grace shorter than HandshakeTimeout could never complete an attempt.
MinResumeGraceMs = 2000
// ResumeRetryDelay paces reattach attempts after a failure. Short, because
// the player is hanging for the whole grace period.
ResumeRetryDelay = 500 * time.Millisecond
// maxControlBackoff caps the control-session reconnect delay. The hub holds
// this client's routes only for its own registration grace, so a backoff that
// grew past that would strand players it is hanging on our behalf.
maxControlBackoff = 10 * time.Second
// ResumeAckTimeout bounds the wait for RESUME_ACK on a conn that completed
// its handshake but then went quiet, so a wedged hub does not consume the
// entire grace budget in one attempt.
ResumeAckTimeout = 10 * time.Second
)
// heartbeatTimeout is how long a session may go without a reply before it is
// considered dead, derived from the configured ping interval.
func (c *Config) heartbeatTimeout() time.Duration {
return c.pingInterval() * MissedHeartbeats
}
// pingInterval is the configured heartbeat period, clamped at the single point
// where a duration is derived. LoadConfig also clamps on the file path; this
// covers Configs built directly (tests), where a 0 or negative PingIntervalMs
// would panic time.NewTicker — a panic, not a log line, because the interval
// feeds the heartbeat timeout too.
func (c *Config) pingInterval() time.Duration {
ms := c.PingIntervalMs
if ms <= 0 {
ms = DefaultPingIntervalMs
}
if ms < MinPingIntervalMs {
ms = MinPingIntervalMs
}
return time.Duration(ms) * time.Millisecond
}
// Mapping routes a registered pattern to a real destination. // Mapping routes a registered pattern to a real destination.
type Mapping struct { type Mapping struct {
Pattern string `json:"pattern"` Pattern string `json:"pattern"`
Destination string `json:"destination"` Destination string `json:"destination"`
ProxyProtocol bool `json:"proxyProtocol"` ProxyProtocol bool `json:"proxyProtocol"`
// VelocitySecret, when non-empty, answers the destination's Velocity
// modern-forwarding login query (velocity:player_info) with this secret,
// forwarding the player's real IP, username and UUID (see velocity.go).
VelocitySecret string `json:"velocitySecret"`
} }
// Config is the client configuration (PROTOCOL.md §9.2). // Config is the client configuration (PROTOCOL.md §9.2).
type Config struct { type Config struct {
Server string `json:"server"` Server string `json:"server"`
PSK string `json:"psk"` PSK string `json:"psk"`
MaxConn int `json:"maxConn"` // MaxTunnels is the max concurrent 1:1 worker connections (PROTOCOL.md §7.1).
PingIntervalMs int `json:"pingIntervalMs"` // 0 means the default. Clamped to [1, 4096].
StreamWindowBytes int `json:"streamWindowBytes"` // per-stream receive window; 0 = default MaxTunnels int `json:"maxTunnels"`
Mappings []Mapping `json:"mappings"` // MaxConn is the retired mux-era pool size. Ignored when loading a file:
// honouring a value of 4 as a player cap would silently break existing
// configs. Tests that construct a Config should set MaxTunnels instead.
MaxConn int `json:"maxConn"`
PingIntervalMs int `json:"pingIntervalMs"`
StreamWindowBytes int `json:"streamWindowBytes"` // per-connection receive window; 0 = default
// MaxBandwidth caps what the client sends to the hub, aggregated over every
// worker conn — the direction that carries the game server's output to the
// players, and the one a residential uplink runs out of first.
// Empty means no limit. See parseBandwidth for the accepted syntax.
MaxBandwidth string `json:"maxBandwidth"`
// StreamResume enables stream resumption (PROTOCOL.md §7.5). A pointer so an
// absent key means "on" while an explicit false disables it: with it off the
// client never offers the flag, allocates no retransmit buffers, and behaves
// exactly as a pre-resume client.
StreamResume *bool `json:"streamResume"`
// ResumeGraceMs bounds how long a parked stream keeps trying to reattach.
// Clamped below the hub's advertised grace so the client always gives up
// first and the hub is never left holding a player nobody will claim.
ResumeGraceMs int `json:"resumeGraceMs"`
// StatsIntervalMs enables the periodic performance summary; 0 (the default)
// disables it and costs nothing.
StatsIntervalMs int `json:"statsIntervalMs"`
Mappings []Mapping `json:"mappings"`
}
// resumeEnabled reports whether stream resumption is configured on.
func (c *Config) resumeEnabled() bool { return c.StreamResume == nil || *c.StreamResume }
// resumeGrace is how long a parked stream may keep trying to reattach.
func (c *Config) resumeGrace() time.Duration {
ms := c.ResumeGraceMs
if ms <= 0 {
ms = DefaultResumeGraceMs
}
if ms < MinResumeGraceMs {
ms = MinResumeGraceMs
}
return time.Duration(ms) * time.Millisecond
}
// bandwidthUnits maps a rate suffix to its value in bytes per second. Bit units
// are decimal because that is what ISPs quote; byte units are binary to match
// streamWindowBytes. Ordered longest-suffix-first so "kbps" is not read as
// "bps", nor "gb/s" as "b/s".
var bandwidthUnits = []struct {
suffix string
mul float64
}{
{"gbps", 1e9 / 8}, {"gbit", 1e9 / 8},
{"mbps", 1e6 / 8}, {"mbit", 1e6 / 8},
{"kbps", 1e3 / 8}, {"kbit", 1e3 / 8},
{"gb/s", 1 << 30}, {"mb/s", 1 << 20}, {"kb/s", 1 << 10},
{"bps", 1.0 / 8},
{"b/s", 1},
}
// parseBandwidth converts a human-readable rate to bytes per second. The empty
// string means "no limit" and yields 0.
//
// "20mbps" 20 megabits/s = 2500000 B/s
// "512kbps" 512 kilobits/s = 64000 B/s
// "2MB/s" 2 mebibytes/s = 2097152 B/s
// "1500000" a bare number is already bytes per second
func parseBandwidth(s string) (int64, error) {
s = strings.TrimSpace(s)
if s == "" {
return 0, nil
}
lower := strings.ToLower(s)
num, mul := lower, 1.0
for _, u := range bandwidthUnits {
if strings.HasSuffix(lower, u.suffix) {
num, mul = strings.TrimSpace(lower[:len(lower)-len(u.suffix)]), u.mul
break
}
}
v, err := strconv.ParseFloat(num, 64)
if err != nil || v <= 0 {
return 0, fmt.Errorf(`maxBandwidth: cannot read %q as a rate (try "20mbps", "2MB/s", or bytes per second)`, s)
}
return int64(v * mul), nil
} }
// LoadConfig reads and validates a JSON config file. // LoadConfig reads and validates a JSON config file.
@@ -82,14 +312,30 @@ func LoadConfig(path string) (*Config, error) {
if c.PSK == "" { if c.PSK == "" {
return nil, fmt.Errorf("psk is required") return nil, fmt.Errorf("psk is required")
} }
if c.MaxConn < 1 { if c.MaxConn != 0 && c.MaxTunnels == 0 {
c.MaxConn = 1 // Old mux pool size. Must not become the player cap: a previously-working
// maxConn: 4 would admit only four players.
fmt.Fprintf(os.Stderr, "redapricot-client: maxConn is ignored (it was the mux pool size); use maxTunnels (default %d)\n", DefaultMaxTunnels)
} }
if c.MaxConn > 8 { if c.MaxTunnels < 1 {
c.MaxConn = 8 c.MaxTunnels = DefaultMaxTunnels
}
if c.MaxTunnels > MaxMaxTunnels {
c.MaxTunnels = MaxMaxTunnels
} }
if c.PingIntervalMs <= 0 { if c.PingIntervalMs <= 0 {
c.PingIntervalMs = 20000 c.PingIntervalMs = DefaultPingIntervalMs
}
if c.PingIntervalMs < MinPingIntervalMs {
c.PingIntervalMs = MinPingIntervalMs
}
// Parsed here only to fail fast on a bad value; New does the real conversion.
bps, err := parseBandwidth(c.MaxBandwidth)
if err != nil {
return nil, err
}
if bps > 0 && bps < MinBandwidth {
return nil, fmt.Errorf("maxBandwidth %q is only %d B/s; the minimum is %d B/s", c.MaxBandwidth, bps, MinBandwidth)
} }
if len(c.Mappings) == 0 { if len(c.Mappings) == 0 {
return nil, fmt.Errorf("at least one mapping is required") return nil, fmt.Errorf("at least one mapping is required")
+86
View File
@@ -0,0 +1,86 @@
package client
import (
"net"
"sync"
"testing"
"time"
)
// stalledHub accepts connections and then says nothing: it never answers the
// Rekey frame with SessionReady, and never closes. This models a hub with a
// wedged event loop, or a load balancer accepting on behalf of a dead backend.
func stalledHub(t *testing.T) string {
t.Helper()
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
var mu sync.Mutex
var held []net.Conn
t.Cleanup(func() {
_ = ln.Close()
mu.Lock()
for _, c := range held {
_ = c.Close()
}
mu.Unlock()
})
go func() {
for {
c, err := ln.Accept()
if err != nil {
return
}
mu.Lock()
held = append(held, c)
mu.Unlock()
}
}()
return ln.Addr().String()
}
// TestDialDoesNotWedgeOnStalledHub is the regression guard for the worst
// failure mode found in the stability audit: Dial used to share a single
// in-flight handshake, and the handshake read had no deadline. One
// unresponsive hub therefore parked every present and future allocation
// forever. 1:1 dials independently, but each must still fail on its own
// HandshakeTimeout rather than block the other.
func TestDialDoesNotWedgeOnStalledHub(t *testing.T) {
c := New(&Config{
Server: stalledHub(t),
PSK: "pool-test",
MaxTunnels: 8,
PingIntervalMs: 20000,
Mappings: []Mapping{{Pattern: "mc.local", Destination: "127.0.0.1:1"}},
})
done := make(chan error, 2)
go func() { _, err := c.pool.Dial(); done <- err }()
time.Sleep(200 * time.Millisecond) // let the first caller get into the dial
go func() { _, err := c.pool.Dial(); done <- err }()
// Both must give up on their own; neither may be stuck behind the other.
limit := time.After(HandshakeTimeout + 15*time.Second)
for i := 0; i < 2; i++ {
select {
case err := <-done:
if err == nil {
t.Fatal("Dial succeeded against a hub that never answers")
}
case <-limit:
t.Fatalf("Dial #%d never returned: a stalled hub wedged the other caller", i+1)
}
}
}
// TestDialRespectsMaxTunnels pins the concurrency cap: once live+dialing
// equals maxTunnels, further Dial calls fail immediately rather than stacking.
func TestDialRespectsMaxTunnels(t *testing.T) {
p := newWorkerPool(&Client{}, 2)
p.conns[&WorkerConn{id: 1}] = struct{}{}
p.conns[&WorkerConn{id: 2}] = struct{}{}
if _, err := p.Dial(); err != errTooManyTunnels {
t.Fatalf("Dial at cap: got %v, want %v", err, errTooManyTunnels)
}
}
+353
View File
@@ -0,0 +1,353 @@
package client
import (
"errors"
"log"
"time"
"github.com/iceBear67/redapricot/client/wire"
)
// Stream resumption (PROTOCOL.md §7.5).
//
// A worker conn is only the middle leg of the player it carries: when it dies,
// both terminal sockets are usually still perfectly healthy. Tearing the
// tunnel down therefore throws away working connections because a replaceable
// transport failed.
//
// Instead the stream parks: the destination socket stays open, the hub hangs the
// player socket, and the client reattaches over a fresh conn. Correctness rests
// on retransmission being byte-exact. Bytes handed to a dying socket are lost
// with no notification, and the frame cipher cannot be resynchronized, so each
// side replays from the offset the other reports it accepted.
var (
errResumeUnknown = errors.New("hub does not know this stream")
errResumeRaced = errors.New("hub has this stream bound to another conn")
errResumeRefused = errors.New("hub refused the reattach")
errResumeTimeout = errors.New("no RESUME_ACK from the hub")
errResumeConnLost = errors.New("the conn carrying the reattach died")
errResumeTooOld = errors.New("hub accepted past what we still hold")
errResumeNoResume = errors.New("hub does not support stream resumption")
)
// resumeGrace is how long a stream parked from a conn may keep trying, clamped
// under what the hub advertised. The client must always give up first: a hub
// that drops the player while we are still reattaching would leave us pumping a
// destination nobody is reading.
func (c *Client) resumeGrace(hubGrace time.Duration) time.Duration {
grace := c.cfg.resumeGrace()
if hubGrace > 0 && hubGrace < grace {
grace = hubGrace
}
return grace
}
// park suspends a stream whose worker conn died instead of destroying it, and
// starts trying to reattach. Reports false when the stream cannot be parked, in
// which case the caller tears it down as before.
//
// A stream already closing (finPending) is not parked: the hub has said the
// player is gone, so there is nothing left to preserve.
func (s *Stream) park(grace time.Duration) bool {
if !s.resumable {
return false
}
s.mu.Lock()
if s.closed || s.finPending {
s.mu.Unlock()
return false
}
already := s.parked
s.parked = true
if s.stats != nil && !already {
s.parkedAt = time.Now()
}
s.mu.Unlock()
if already {
// A reattach was already in flight and had registered this stream on the
// conn that just died — which is how it got here at all. That attempt
// still owns the stream, so reporting failure would have the caller tear
// down a player that is mid-recovery. Fail its wait immediately rather
// than let it sit out the ack timeout: the grace budget is small, and
// spending ten seconds of it waiting on a socket that is already gone is
// the difference between reattaching and dropping the player.
s.deliverResume(resumeResult{err: errResumeConnLost})
return true
}
go s.resumeLoop(grace)
return true
}
// resumeLoop reattaches the stream, retrying until it succeeds or the grace
// period runs out.
//
// Attempts are started right up to the deadline rather than reserving a whole
// dial's worth of budget for the last one. Reserving it would be self-defeating
// — the grace and HandshakeTimeout are the same order of magnitude, so the
// reservation can consume the entire budget and leave no attempt at all — and
// overshooting is safe: an attempt that lands after the hub has dropped the
// player is answered with RST(unknown stream) and tears down cleanly.
func (s *Stream) resumeLoop(grace time.Duration) {
deadline := time.Now().Add(grace)
for {
if s.isClosed() {
return
}
if !time.Now().Before(deadline) {
break
}
err := s.tryResume()
if err == nil {
return
}
// errResumeRaced falls through to the retry below. The hub has this
// player bound to a conn that is not ours — a half-open conn whose
// death the hub has not yet learned, or a bind left behind by a racing
// attempt on a now-dead conn. There is no other live attempt: park is
// the only resumeLoop starter and it refuses to double-start. Retrying
// is safe precisely because nothing else owns the stream — the foreign
// bind dies with its conn, the hub re-parks, and a later RESUME lands.
// The grace deadline bounds the loop and expiry tears the stream down,
// so a hub that never re-parks cannot hang us forever.
if errors.Is(err, errResumeUnknown) || errors.Is(err, errResumeTooOld) {
// Terminal: the hub has no state for this stream (it restarted, the
// grace expired, or a load balancer sent us to a different instance).
log.Printf("stream resume abandoned: %v", err)
s.teardown(false)
return
}
select {
case <-s.done:
return
case <-time.After(ResumeRetryDelay):
}
}
log.Printf("stream resume gave up after %s; closing destination", grace)
// No FIN: the only conns we could send it on are the ones that just failed
// us. The hub drops the hanging player when its own grace expires.
s.teardown(false)
}
// tryResume performs one reattach attempt: dial a fresh worker conn, send
// RESUME, and replay from wherever the hub says it got to.
func (s *Stream) tryResume() error {
wc, err := s.allocateForResume()
if err != nil {
return err
}
s.mu.Lock()
cid := s.cid
accepted := s.acceptedOffset
delivered := s.deliveredOffset
wait := make(chan resumeResult, 1)
s.resumeWait = wait
s.mu.Unlock()
msg := wire.NewWriter().U8(MuxResume).Bytes(cid).
I64(accepted).I64(delivered).Out()
if err := wc.fc.WriteFrame(msg); err != nil {
s.abandonAttempt(wc)
return err
}
var res resumeResult
select {
case res = <-wait:
case <-s.done:
// Torn down while waiting. teardown only detaches the conn the stream
// was bound to, which is not this one, so the claim made above has to be
// withdrawn here or the new conn stays bound forever.
s.abandonAttempt(wc)
return errResumeRefused
case <-time.After(ResumeAckTimeout):
s.abandonAttempt(wc)
return errResumeTimeout
}
if res.err != nil {
s.abandonAttempt(wc)
return res.err
}
return s.completeResume(wc, res)
}
// allocateForResume dials a dedicated worker conn that will honour a reattach.
// 1:1: this must never land on someone else's tunnel.
func (s *Stream) allocateForResume() (*WorkerConn, error) {
for attempt := 0; attempt < dialAttempts; attempt++ {
wc, err := s.client.pool.Dial()
if err != nil {
return nil, err
}
// Re-checked per conn, not assumed from the dead one: this may be a
// different or restarted hub. Sending RESUME to a hub that does not know
// the frame type would hang the player for the rest of the grace waiting
// for an answer that is never coming.
if !wc.resume {
_ = wc.fc.Close()
return nil, errResumeNoResume
}
if wc.attach(s) {
return wc, nil
}
_ = wc.fc.Close()
}
return nil, errResumeRefused
}
// abandonAttempt withdraws a failed attempt from the conn it was made on and
// closes that conn — 1:1, it exists only for this attempt.
func (s *Stream) abandonAttempt(wc *WorkerConn) {
wc.detach()
_ = wc.fc.Close()
s.mu.Lock()
s.resumeWait = nil
s.mu.Unlock()
}
// completeResume rebinds the stream to its new conn and replays what the hub is
// missing, holding sendMu throughout so live traffic cannot overtake the replay.
func (s *Stream) completeResume(wc *WorkerConn, res resumeResult) error {
s.sendMu.Lock()
// Delivery is a strictly stronger fact than credit — the hub only credits what
// it has delivered — so the reported offset can be adopted wholesale. Doing so
// also repairs the ledger: the grants destroyed by the outage are exactly the
// gap between the two, and without this the retained region would carry that
// dead prefix for the rest of the stream's life.
s.ackedOffset.Store(res.delivered)
s.un.advance(res.delivered)
replay := s.un.from(res.accepted)
if replay == nil {
s.sendMu.Unlock()
s.abandonAttempt(wc)
return errResumeTooOld
}
// Three offsets, three jobs, and conflating any two of them breaks something
// different.
//
// What to replay is measured from what the hub *accepted* — the bytes it
// never received. What the window should be is measured from what it
// *delivered*, because the window is a promise about undelivered bytes.
// It cannot be measured from what it *credited*: credit arrives as deltas,
// and the grants in flight when the connection died are gone for good, so a
// window derived from them would be permanently short — and, when a full
// window was outstanding at the drop, permanently zero. That is a deadlock,
// not a slowdown: no credit can arrive because nothing can be sent.
outstanding := s.un.length()
replayed := s.un.end() - res.accepted
// Publish the new binding before any frame goes out on it.
s.wc.Store(wc)
s.mu.Lock()
// Restated, not patched. The window is a delta ledger and the outage tore a
// hole in it; deriving it afresh from the delivered offset closes the hole
// exactly, whatever was lost.
s.sendWnd = wc.sendWndInit - int(outstanding)
if s.sendWnd < 0 {
s.sendWnd = 0
}
// Symmetrically, our own pending credit is discarded rather than flushed:
// the delivered offset we reported already tells the hub everything those
// deltas would have, and sending both would grant the same bytes twice.
s.consumed = 0
if len(res.cid) == CIDLen {
s.cid = res.cid // fresh capability, so a CID is never reusable twice
}
s.parked = false
s.resumeWait = nil
owedFin := s.finToHub
s.cond.Broadcast() // release acquireSendWnd and any parked writer
s.mu.Unlock()
for len(replay) > 0 {
n := len(replay)
if n > s.client.chunk {
n = s.client.chunk
}
if err := wc.sendData(replay[:n]); err != nil {
// The conn died mid-replay. The stream is still resumable, but not
// from this conn — put it back in the parked state before returning
// so the conn's teardown takes park()'s already-branch instead of
// starting a second resumeLoop. The loop we came from keeps
// retrying with the fresh CID, which the hub re-parked alongside
// the stream when this conn died. Without this re-park the flag
// cleared above would let two loops race one stream, and a racing
// RST would then strand it with neither loop alive.
s.mu.Lock()
s.parked = true
s.mu.Unlock()
s.sendMu.Unlock()
return err
}
replay = replay[n:]
}
s.sendMu.Unlock()
// Counted only once the replay actually landed: a failed reattach above
// returns before this, so a conn dying mid-replay does not inflate the
// resume count with an attempt that never completed.
if s.stats != nil {
s.mu.Lock()
s.stats.resumes++
s.stats.hung += time.Since(s.parkedAt)
s.stats.replayBytes += replayed
s.mu.Unlock()
}
// A destination that closed while we were parked owed the hub a FIN that had
// nowhere to go at the time.
if owedFin {
wc.sendFin()
s.teardown(false)
return nil
}
log.Printf("stream %s resumed (%d bytes replayed, %d outstanding)", s.name(), replayed, outstanding)
return nil
}
// deliverResume hands an answer to a reattach that is waiting for one. Reports
// false when no attempt was in flight, so the caller can treat the frame as it
// would on any live stream.
func (s *Stream) deliverResume(res resumeResult) bool {
s.mu.Lock()
ch := s.resumeWait
s.resumeWait = nil
s.mu.Unlock()
if ch == nil {
return false
}
ch <- res // buffered, and read at most once per attempt
return true
}
// onRst applies an RST, using the reason to tell a stream that is genuinely gone
// from one that a racing reattach has taken over.
func (s *Stream) onRst(reason int) {
err := errResumeRefused
switch reason {
case RstUnknownStream:
err = errResumeUnknown
case RstAlreadyBound:
err = errResumeRaced
}
if s.deliverResume(resumeResult{err: err}) {
return
}
s.teardown(false)
}
// noteFinWhileParked records a FIN the stream owes the hub but cannot send,
// because the only conn it has is the one that just died. Reports false when the
// stream is not parked and the caller should send it normally.
func (s *Stream) noteFinWhileParked() bool {
s.mu.Lock()
defer s.mu.Unlock()
if !s.parked {
return false
}
s.finToHub = true
return true
}
+104
View File
@@ -0,0 +1,104 @@
package client
import (
"testing"
)
// The retained region is the one real cost stream resumption adds to the send
// path: a chunk has to survive past the frame write, so it is copied. These
// pin both halves of that claim — that the copy is the only cost, and that
// disabling the feature removes it entirely rather than merely shrinking it.
func benchStream(resumable bool) *Stream {
s := &Stream{resumable: resumable}
return s
}
// BenchmarkRetainChunk measures what emit adds over a bare frame write: the
// trim-and-append into the retained region. Compare the two variants; the delta
// is the per-byte copy the feature costs.
func BenchmarkRetainChunk(b *testing.B) {
chunk := make([]byte, DataChunkSize)
window := int64(DefaultStreamWindow)
b.Run("resume-on", func(b *testing.B) {
s := benchStream(true)
b.SetBytes(int64(len(chunk)))
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
// Model the steady state: credit trails one window behind, so the
// buffer trims about as fast as it grows and stays bounded.
acked := s.un.end() - window
if acked < 0 {
acked = 0
}
s.un.advance(acked)
s.un.append(chunk)
}
if got := int64(s.un.length()); got > window+int64(len(chunk)) {
b.Fatalf("retained region grew past one window: %d", got)
}
})
b.Run("resume-off", func(b *testing.B) {
s := benchStream(false)
b.SetBytes(int64(len(chunk)))
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
if s.resumable {
s.un.advance(0)
s.un.append(chunk)
}
}
})
}
// TestResumeDisabledAllocatesNothing pins the off switch at the level that
// matters. It is easy for a feature flag to stop the wire behaviour while
// leaving the bookkeeping running, which would keep the memory cost and the
// per-byte copy for a user who explicitly turned it off — a partial revert that
// nobody would notice.
func TestResumeDisabledAllocatesNothing(t *testing.T) {
chunk := make([]byte, DataChunkSize)
s := benchStream(false)
allocs := testing.AllocsPerRun(1000, func() {
if s.resumable {
s.un.advance(0)
s.un.append(chunk)
}
})
if allocs != 0 {
t.Fatalf("resume disabled still allocated %.1f times per send", allocs)
}
if s.un.buf != nil {
t.Fatalf("resume disabled still allocated a retained region of %d bytes", cap(s.un.buf))
}
}
// TestRetainedRegionStaysWithinWindow is the memory bound the design rests on:
// flow control already caps outstanding bytes at one window, so the retained
// region needs no cap of its own. If that ever stopped holding, a busy stream
// would grow without limit and the hub would be the first to notice.
func TestRetainedRegionStaysWithinWindow(t *testing.T) {
const window = DefaultStreamWindow
chunk := make([]byte, DataChunkSize)
var u unackedBuf
for i := 0; i < 5000; i++ {
// A sender may never have more than one window outstanding, which is
// exactly what acquireSendWnd enforces before emit is ever reached.
if u.length()+len(chunk) > window {
u.advance(u.base() + int64(len(chunk)))
}
u.append(chunk)
if u.length() > window {
t.Fatalf("round %d: retained %d bytes for a %d-byte window", i, u.length(), window)
}
}
if cap(u.buf) > 4*window {
t.Fatalf("backing array grew to %d for a %d-byte window", cap(u.buf), window)
}
}
+268
View File
@@ -0,0 +1,268 @@
package client
import (
"sync"
"time"
)
// minShaperWait floors the dispatcher's sleep so floating-point dust in the
// token arithmetic cannot spin it.
const minShaperWait = time.Millisecond
// Shaper caps the aggregate rate at which the client writes DATA to the hub and
// divides that budget across streams.
//
// The credit windows of PROTOCOL.md §7.3 bound how many bytes may be *in flight*
// per stream; they say nothing about bytes per *second*. That is the gap this
// fills. On a residential uplink one player loading chunks will otherwise
// saturate the line and push every other player's keepalive past its timeout.
//
// Two mechanisms are layered:
//
// - A token bucket sets the long-run rate and the size of the burst that may
// be spent after an idle period.
// - Start-time fair queueing decides who spends those tokens. A global virtual
// clock advances with each grant; every stream remembers the virtual time at
// which its last request finished. A request is stamped
// max(share.vfinish, vclock) and the lowest stamp is served first, so a
// stream that keeps sending pushes its own stamp further out and yields to
// quieter streams. The clamp to vclock is what keeps bursts cheap: a stream
// returning from idle is pulled back to the head of the clock, so it cannot
// hoard credit while it was idle, but it is not punished for the idleness
// either. One stream alone gets the whole rate.
//
// A nil *Shaper means "no limit"; every method short-circuits, so call sites do
// not branch.
type Shaper struct {
rate float64 // bytes per second
burst float64 // token bucket capacity, bytes
chunk int // how much a caller should request at a time
mu sync.Mutex
tokens float64
last time.Time
vclock float64 // virtual time, in bytes of service granted
waiting []*shaperReq // unordered; the dispatcher scans for the lowest vstart
wake chan struct{} // cap 1, non-blocking: nudges the dispatcher
done chan struct{}
once sync.Once
}
// shaperShare is one stream's position in the fair queue. It lives on the
// Stream and dies with it; a fresh share starts at zero and is clamped up to
// the current virtual clock on its first request.
type shaperShare struct{ vfinish float64 }
// shaperReq is one pending Acquire. granted and membership in Shaper.waiting
// are both guarded by Shaper.mu.
type shaperReq struct {
n int
vstart float64
grant chan struct{}
granted bool
}
// NewShaper builds a shaper for the given rate. A non-positive rate returns nil,
// which every method treats as "unlimited".
func NewShaper(bytesPerSec int64) *Shaper {
if bytesPerSec <= 0 {
return nil
}
rate := float64(bytesPerSec)
burst := rate * ShaperBurstSeconds
// The floor is a correctness constraint, not a preference: a request larger
// than the bucket could never be afforded and would park forever.
if burst < MinShaperBurst {
burst = MinShaperBurst
}
if burst > MaxShaperBurst {
burst = MaxShaperBurst
}
chunk := int(rate * ShaperSliceSeconds)
if chunk < MinShaperChunk {
chunk = MinShaperChunk
}
if chunk > DataChunkSize {
chunk = DataChunkSize
}
sh := &Shaper{
rate: rate,
burst: burst,
chunk: chunk,
tokens: burst,
last: time.Now(),
wake: make(chan struct{}, 1),
done: make(chan struct{}),
}
go sh.dispatch()
return sh
}
// chunkSize is how many bytes a sender should offer per request. It is sized to
// ShaperSliceSeconds of transmission so no stream holds the link for long before
// the scheduler can switch: at 1 Mbps a full 32 KiB chunk takes ~256 ms, which is
// enough dead air to drag other players towards a keepalive timeout.
func (sh *Shaper) chunkSize() int {
if sh == nil {
return DataChunkSize
}
return sh.chunk
}
// Acquire blocks until n bytes of bandwidth budget are available for the stream
// owning share. It returns false only when cancel fires first, in which case
// nothing was charged.
//
// cancel is the stream's done channel: a stream torn down while parked here must
// not keep a goroutine (and its Stream) alive waiting for tokens it will never
// use.
func (sh *Shaper) Acquire(share *shaperShare, n int, cancel <-chan struct{}) bool {
if sh == nil || n <= 0 {
return true
}
req := &shaperReq{n: n, grant: make(chan struct{})}
sh.mu.Lock()
// Stamp the request and reserve this stream's slot in virtual time up front,
// so a stream cannot queue many requests at the same cheap stamp.
req.vstart = share.vfinish
if req.vstart < sh.vclock {
req.vstart = sh.vclock
}
share.vfinish = req.vstart + float64(n)
sh.waiting = append(sh.waiting, req)
sh.mu.Unlock()
sh.nudge()
select {
case <-req.grant:
return true
case <-sh.done:
// Shaping stopped: let live traffic through rather than stalling it.
sh.mu.Lock()
sh.removeLocked(req)
sh.mu.Unlock()
return true
case <-cancel:
sh.mu.Lock()
granted := req.granted
if !granted {
sh.removeLocked(req)
}
sh.mu.Unlock()
return granted
}
}
// Stop shuts the dispatcher down and releases everyone parked in Acquire.
func (sh *Shaper) Stop() {
if sh == nil {
return
}
sh.once.Do(func() { close(sh.done) })
}
// dispatch is the single goroutine that hands out tokens. It sleeps exactly as
// long as the next waiter needs rather than polling on a fixed tick, so an idle
// shaper costs nothing.
func (sh *Shaper) dispatch() {
for {
wait := sh.grantReady()
var tick <-chan time.Time
var timer *time.Timer
if wait > 0 {
timer = time.NewTimer(wait)
tick = timer.C
}
select {
case <-tick:
case <-sh.wake:
case <-sh.done:
if timer != nil {
timer.Stop()
}
return
}
if timer != nil {
timer.Stop()
}
}
}
// grantReady refills the bucket and grants every waiter it can afford, lowest
// virtual start time first. It returns how long until the next waiter becomes
// affordable, or 0 when nothing is pending.
func (sh *Shaper) grantReady() time.Duration {
sh.mu.Lock()
defer sh.mu.Unlock()
now := time.Now()
if elapsed := now.Sub(sh.last); elapsed > 0 {
sh.tokens += sh.rate * elapsed.Seconds()
if sh.tokens > sh.burst {
sh.tokens = sh.burst
}
sh.last = now
}
for {
req := sh.headLocked()
if req == nil {
return 0
}
// Callers stay under chunkSize, which NewShaper keeps below the bucket.
// Should a future caller not, wait for a full bucket rather than for a
// token count that can never be reached, and let the balance go negative:
// the debt is repaid by the next refill, so the long-run rate still holds.
need := min(float64(req.n), sh.burst)
if need > sh.tokens {
wait := time.Duration((need - sh.tokens) / sh.rate * float64(time.Second))
if wait < minShaperWait {
wait = minShaperWait
}
return wait
}
sh.tokens -= float64(req.n)
// The clock follows the request being served, never runs ahead of it.
if req.vstart > sh.vclock {
sh.vclock = req.vstart
}
req.granted = true
sh.removeLocked(req)
close(req.grant)
}
}
// headLocked returns the pending request with the lowest virtual start time.
// A linear scan is deliberate: the queue holds at most one entry per live
// stream (tens, not thousands), so a heap would cost more in complexity than it
// saves in comparisons.
func (sh *Shaper) headLocked() *shaperReq {
var best *shaperReq
for _, w := range sh.waiting {
if best == nil || w.vstart < best.vstart {
best = w
}
}
return best
}
func (sh *Shaper) removeLocked(req *shaperReq) {
for i, w := range sh.waiting {
if w == req {
sh.waiting = append(sh.waiting[:i], sh.waiting[i+1:]...)
return
}
}
}
func (sh *Shaper) nudge() {
select {
case sh.wake <- struct{}{}:
default:
}
}
+231
View File
@@ -0,0 +1,231 @@
package client
import (
"sync"
"sync/atomic"
"testing"
"time"
)
func TestParseBandwidth(t *testing.T) {
cases := []struct {
in string
want int64
}{
{"", 0},
{"20mbps", 2_500_000},
{"20Mbps", 2_500_000},
{"20 mbps", 2_500_000},
{"1.5mbit", 187_500},
{"512kbps", 64_000},
{"1gbps", 125_000_000},
{"2MB/s", 2 << 20},
{"500kb/s", 500 << 10},
{"1GB/s", 1 << 30},
{"8bps", 1},
{"4096b/s", 4096},
{"1500000", 1_500_000}, // bare number is already bytes/sec
}
for _, c := range cases {
got, err := parseBandwidth(c.in)
if err != nil {
t.Errorf("parseBandwidth(%q): unexpected error %v", c.in, err)
continue
}
if got != c.want {
t.Errorf("parseBandwidth(%q) = %d, want %d", c.in, got, c.want)
}
}
for _, bad := range []string{"fast", "20megabits", "-5mbps", "0", "0mbps", "mbps", "20 mb ps"} {
if _, err := parseBandwidth(bad); err == nil {
t.Errorf("parseBandwidth(%q): expected an error", bad)
}
}
}
// A nil shaper is the "unlimited" case and must be safe on every path, because
// call sites deliberately do not branch on it.
func TestNilShaperIsUnlimited(t *testing.T) {
var sh *Shaper
if sh = NewShaper(0); sh != nil {
t.Fatal("NewShaper(0) should return nil")
}
if got := sh.chunkSize(); got != DataChunkSize {
t.Errorf("nil chunkSize = %d, want %d", got, DataChunkSize)
}
if !sh.Acquire(&shaperShare{}, 1<<20, nil) {
t.Error("nil Acquire should always succeed")
}
sh.Stop() // must not panic
}
// The virtual-time bookkeeping is what makes the shaper fair, so assert it
// directly. The rate is high enough that tokens never bind, leaving only the
// stamping under test — no timing, no flakiness.
func TestShaperIdleStreamCannotHoardCredit(t *testing.T) {
sh := NewShaper(1 << 30)
defer sh.Stop()
var heavy, light shaperShare
for i := 0; i < 10; i++ {
if !sh.Acquire(&heavy, 1000, nil) {
t.Fatal("acquire failed")
}
}
if heavy.vfinish != 10000 {
t.Errorf("heavy.vfinish = %v, want 10000", heavy.vfinish)
}
sh.mu.Lock()
vclock := sh.vclock
sh.mu.Unlock()
if vclock != 9000 {
t.Errorf("vclock = %v, want 9000 (the stamp of the last request served)", vclock)
}
// light was idle for all of it. Its stale vfinish of 0 must be clamped up to
// the current clock: it may not bank the virtual time it never spent, which
// is what would let it starve heavy on return.
if !sh.Acquire(&light, 1000, nil) {
t.Fatal("acquire failed")
}
if light.vfinish != vclock+1000 {
t.Errorf("light.vfinish = %v, want %v (clamped to the clock, not 1000)", light.vfinish, vclock+1000)
}
}
func TestShaperEnforcesRate(t *testing.T) {
const rate = 1 << 20 // 1 MiB/s
sh := NewShaper(rate)
defer sh.Stop()
var share shaperShare
const total = 512 << 10
const chunk = 8 << 10
start := time.Now()
for sent := 0; sent < total; sent += chunk {
if !sh.Acquire(&share, chunk, nil) {
t.Fatal("acquire failed")
}
}
elapsed := time.Since(start)
// The bucket starts full, so the burst is free and only the remainder is
// paced: (512 KiB - 200 KiB) / 1 MiB/s ≈ 300 ms. Bounds are wide on purpose.
if elapsed < 200*time.Millisecond {
t.Errorf("sent %d bytes at %d B/s in only %v; the cap is not being enforced", total, rate, elapsed)
}
if elapsed > time.Second {
t.Errorf("took %v, far longer than the ~300ms the rate implies", elapsed)
}
}
// An idle stream must be able to spend the banked burst at once, otherwise a
// player joining pays for the cap in visible chunk-loading latency.
func TestShaperAllowsBurst(t *testing.T) {
sh := NewShaper(1 << 20)
defer sh.Stop()
var share shaperShare
start := time.Now()
for i := 0; i < 6; i++ {
if !sh.Acquire(&share, 32<<10, nil) { // 192 KiB, inside the 200 KiB bucket
t.Fatal("acquire failed")
}
}
if elapsed := time.Since(start); elapsed > 100*time.Millisecond {
t.Errorf("burst of 192 KiB took %v; the bucket should have covered it instantly", elapsed)
}
}
// The point of the whole exercise: a stream that never stops asking must not
// crowd another one out.
func TestShaperSharesFairlyBetweenStreams(t *testing.T) {
sh := NewShaper(1 << 20)
defer sh.Stop()
stop := make(chan struct{})
var counts [2]atomic.Int64
var wg sync.WaitGroup
for i := range counts {
wg.Add(1)
go func(i int) {
defer wg.Done()
var share shaperShare
for {
if !sh.Acquire(&share, 4<<10, stop) {
return
}
counts[i].Add(4 << 10)
}
}(i)
}
// Spend the token bucket before measuring, the way the idle-credit test
// raises the rate so tokens never bind: isolate the property under test.
//
// While the bucket has tokens there is no queue to arbitrate — every request
// is granted the moment it arrives, and being the stream that is owed service
// only helps when both are enqueued at the same instant. The burst is
// therefore first-come-first-served by construction, and at 0.2s of
// transmission it is a quarter of a 600ms window, enough to swamp the result:
// a run where one goroutine happened to win the bucket landed at 520192 vs
// 315392, which is exactly "one took the whole burst, then the two split the
// remainder evenly".
//
// Fairness here is a steady-state property, and that is what matters in
// practice — the bucket is empty whenever the link is actually busy.
time.Sleep(250 * time.Millisecond)
counts[0].Store(0)
counts[1].Store(0)
time.Sleep(600 * time.Millisecond)
close(stop)
wg.Wait()
a, b := counts[0].Load(), counts[1].Load()
if a == 0 || b == 0 {
t.Fatalf("one stream was starved entirely: %d vs %d", a, b)
}
lo, hi := min(a, b), max(a, b)
t.Logf("steady-state split: %d vs %d bytes (%.3fx)", a, b, float64(hi)/float64(lo))
if float64(hi) > 1.35*float64(lo) {
t.Errorf("unfair split: %d vs %d bytes (>35%% apart)", a, b)
}
}
// A stream torn down while parked must release immediately and leave no trace
// in the queue, or its goroutine (and the Stream it closes over) leaks.
func TestShaperAcquireCancels(t *testing.T) {
sh := NewShaper(MinBandwidth) // 8 KiB/s: a parked request would wait seconds
defer sh.Stop()
var share shaperShare
if !sh.Acquire(&share, MinShaperBurst, nil) { // drain the bucket
t.Fatal("acquire failed")
}
cancel := make(chan struct{})
result := make(chan bool, 1)
go func() { result <- sh.Acquire(&share, 32<<10, cancel) }()
time.Sleep(50 * time.Millisecond)
close(cancel)
select {
case ok := <-result:
if ok {
t.Error("Acquire returned true after cancellation")
}
case <-time.After(time.Second):
t.Fatal("Acquire did not return after its cancel channel closed")
}
sh.mu.Lock()
n := len(sh.waiting)
sh.mu.Unlock()
if n != 0 {
t.Errorf("%d cancelled request(s) left in the queue", n)
}
}
+221
View File
@@ -0,0 +1,221 @@
package client
import (
"fmt"
"log"
"sort"
"strings"
"sync"
"sync/atomic"
"time"
)
// Performance diagnostics.
//
// The question an operator actually has is "why is this tunnel slow?", and
// nothing here could answer it before. A stream that is not moving bytes is
// blocked on exactly one of three things:
//
// - the flow-control window — the peer is not draining to its terminal
// socket, so the bottleneck is past the tunnel (a struggling game server, a
// player on a bad link);
// - the shaper — the configured bandwidth cap is the binding constraint, and
// raising it is the fix;
// - the peer socket itself — bytes move, but slowly, which points at the path
// rather than at either end.
//
// Those three are indistinguishable from throughput alone and call for
// completely different responses, so they are counted apart.
//
// Cost. Both structs are nil unless statsIntervalMs is set, so the default is a
// single predictable branch per event and no allocation at all. When enabled,
// counters sit under locks the code already holds; only the frame counters use
// atomics, because the read loop must never queue behind a send. A clock is read
// only when a goroutine is about to block, never per chunk — if nothing stalls,
// nothing is timed.
// streamStats accumulates one stream's lifetime. Guarded by Stream.mu.
type streamStats struct {
opened time.Time
bytesUp int64 // destination -> hub
bytesDown int64 // hub -> destination
windowStall time.Duration // blocked with no send credit
shaperStall time.Duration // blocked on the bandwidth cap
qPeak int // high-water mark of the receive queue
resumes int
hung time.Duration // total time parked awaiting a reattach
replayBytes int64
}
// connStats accumulates one worker conn's lifetime.
type connStats struct {
opened time.Time
framesIn atomic.Int64
framesOut atomic.Int64
writeErrs atomic.Int64
// Round-trip time of the worker heartbeat. The probe already carries a
// timestamp that the peer echoes and both sides currently throw away, so
// this measures tunnel latency for no added cost.
mu sync.Mutex
rttLast time.Duration
rttMin time.Duration
rttMax time.Duration
rttSum time.Duration
rttN int64
}
func (cs *connStats) observeRTT(d time.Duration) {
if cs == nil || d < 0 {
return // a nonce we cannot read as one of our own timestamps
}
cs.mu.Lock()
defer cs.mu.Unlock()
cs.rttLast = d
if cs.rttN == 0 || d < cs.rttMin {
cs.rttMin = d
}
if d > cs.rttMax {
cs.rttMax = d
}
cs.rttSum += d
cs.rttN++
}
func (cs *connStats) rtt() (last, min, avg, max time.Duration) {
cs.mu.Lock()
defer cs.mu.Unlock()
if cs.rttN == 0 {
return 0, 0, 0, 0
}
return cs.rttLast, cs.rttMin, cs.rttSum / time.Duration(cs.rttN), cs.rttMax
}
// stallClock times a block without charging the path that does not block: the
// clock is read only once a wait is actually about to happen.
type stallClock struct{ start time.Time }
func (t *stallClock) begin(on bool) {
if on && t.start.IsZero() {
t.start = time.Now()
}
}
func (t *stallClock) elapsed() time.Duration {
if t.start.IsZero() {
return 0
}
return time.Since(t.start)
}
// statsLoop prints one aggregate line per interval. Never started when
// statsIntervalMs is 0, which is the default.
func (c *Client) statsLoop(stop <-chan struct{}) {
ticker := time.NewTicker(time.Duration(c.cfg.StatsIntervalMs) * time.Millisecond)
defer ticker.Stop()
for {
select {
case <-stop:
return
case <-ticker.C:
log.Print(c.StatsLine())
}
}
}
// StatsLine renders the current pool and per-conn state as one greppable line.
// Exported so tests and embedders can sample it without waiting for the ticker.
func (c *Client) StatsLine() string {
conns := c.pool.snapshot()
var b strings.Builder
fmt.Fprintf(&b, "stats conns=%d", len(conns))
live, parked := 0, 0
for _, wc := range conns {
st := wc.getStream()
bound := 0
if st != nil {
bound = 1
live++
st.mu.Lock()
if st.parked {
parked++
}
st.mu.Unlock()
}
fmt.Fprintf(&b, " | conn%d bound=%d", wc.id, bound)
if cs := wc.stats; cs != nil {
_, mn, avg, mx := cs.rtt()
fmt.Fprintf(&b, " frames=%d/%d rtt=%s/%s/%s",
cs.framesIn.Load(), cs.framesOut.Load(), round(mn), round(avg), round(mx))
if n := cs.writeErrs.Load(); n > 0 {
fmt.Fprintf(&b, " writeErrs=%d", n)
}
}
}
fmt.Fprintf(&b, " | tunnels=%d parked=%d", live, parked)
return b.String()
}
// logSummary reports a stream's lifetime as it closes. This is the artifact that
// answers a specific complaint after the fact, once the periodic line has
// scrolled away.
func (s *Stream) logSummary() {
s.mu.Lock()
st := s.stats
if st == nil {
s.mu.Unlock()
return
}
line := fmt.Sprintf("stream closed after %s: up=%s down=%s stalled(window=%s shaper=%s) qPeak=%s",
round(time.Since(st.opened)), bytesHuman(st.bytesUp), bytesHuman(st.bytesDown),
round(st.windowStall), round(st.shaperStall), bytesHuman(int64(st.qPeak)))
if st.resumes > 0 {
line += fmt.Sprintf(" resumes=%d hung=%s replayed=%s",
st.resumes, round(st.hung), bytesHuman(st.replayBytes))
}
s.mu.Unlock()
log.Print(line)
}
func (p *WorkerPool) snapshot() []*WorkerConn {
p.mu.Lock()
conns := make([]*WorkerConn, 0, len(p.conns))
for wc := range p.conns {
conns = append(conns, wc)
}
p.mu.Unlock()
sort.Slice(conns, func(i, j int) bool { return conns[i].id < conns[j].id })
return conns
}
// round trims a duration to something readable in a log line.
func round(d time.Duration) time.Duration {
switch {
case d <= 0:
return 0
case d < time.Millisecond:
return d.Round(time.Microsecond)
case d < time.Second:
return d.Round(time.Millisecond)
default:
return d.Round(10 * time.Millisecond)
}
}
func bytesHuman(n int64) string {
const unit = 1024
if n < unit {
return fmt.Sprintf("%dB", n)
}
div, exp := int64(unit), 0
for v := n / unit; v >= unit; v /= unit {
div *= unit
exp++
}
return fmt.Sprintf("%.1f%ciB", float64(n)/float64(div), "KMGT"[exp])
}
+64
View File
@@ -0,0 +1,64 @@
package client
// unackedBuf holds the bytes a stream has sent but the peer has not yet
// credited — exactly the region a reattach may have to retransmit
// (PROTOCOL.md §7.5).
//
// It needs no cap of its own: credit is only granted as bytes reach the peer's
// terminal socket, so flow control already bounds the outstanding region to one
// window. That is what makes byte-exact resumption affordable at all.
//
// A read offset rather than a copy-down on every trim. Credit arrives once per
// half-window, and copying the live remainder each time would add a second
// per-byte copy to the whole send path; compacting only once the dead prefix
// dominates makes it amortized O(1).
type unackedBuf struct {
buf []byte
head int // bytes at the front already credited, awaiting reclamation
baseOff int64 // stream offset of buf[head]
}
// length is how many bytes are still outstanding.
func (u *unackedBuf) length() int { return len(u.buf) - u.head }
// base is the offset of the first byte still held.
func (u *unackedBuf) base() int64 { return u.baseOff }
// end is the offset one past the last byte sent.
func (u *unackedBuf) end() int64 { return u.baseOff + int64(u.length()) }
func (u *unackedBuf) append(p []byte) { u.buf = append(u.buf, p...) }
// advance drops everything the peer has credited up to off.
func (u *unackedBuf) advance(off int64) {
drop := int(off - u.baseOff)
if drop <= 0 {
return
}
if n := u.length(); drop > n {
drop = n // only reachable from a peer crediting bytes it was never sent
}
u.head += drop
u.baseOff += int64(drop)
switch {
case u.head == len(u.buf):
u.buf, u.head = u.buf[:0], 0 // fully drained: restart at the front
case u.head > len(u.buf)/2:
u.buf = append(u.buf[:0], u.buf[u.head:]...)
u.head = 0
}
}
// from returns the outstanding bytes at and after off, or nil when off falls
// outside what is still held — which means the peer reported an offset we can no
// longer satisfy, and the stream cannot be resumed.
func (u *unackedBuf) from(off int64) []byte {
skip := off - u.baseOff
if skip < 0 || skip > int64(u.length()) {
return nil
}
return u.buf[u.head+int(skip):]
}
// reset releases the buffer once a stream can no longer be resumed.
func (u *unackedBuf) reset() { u.buf, u.head = nil, 0 }
+112
View File
@@ -0,0 +1,112 @@
package client
import (
"bytes"
"testing"
)
// The retained region is what a reattach replays from, so an off-by-one here is
// not a dropped byte but a spliced stream: the peer resumes mid-packet and the
// session dies in a way no round-trip test would attribute to this code.
func TestUnackedTracksOffsets(t *testing.T) {
var u unackedBuf
u.append([]byte("hello"))
u.append([]byte("world"))
if got := u.length(); got != 10 {
t.Fatalf("length = %d, want 10", got)
}
if got := u.end(); got != 10 {
t.Fatalf("end = %d, want 10", got)
}
if got := u.from(0); !bytes.Equal(got, []byte("helloworld")) {
t.Fatalf("from(0) = %q", got)
}
// A reattach replays from wherever the peer got to, which lands anywhere —
// including the middle of a chunk boundary.
if got := u.from(3); !bytes.Equal(got, []byte("loworld")) {
t.Fatalf("from(3) = %q", got)
}
if got := u.from(10); len(got) != 0 {
t.Fatalf("from(end) = %q, want empty", got)
}
}
func TestUnackedAdvanceDropsCreditedBytes(t *testing.T) {
var u unackedBuf
u.append([]byte("abcdefghij"))
u.advance(4)
if got := u.length(); got != 6 {
t.Fatalf("length after advance = %d, want 6", got)
}
if got := u.end(); got != 10 {
t.Fatalf("end must not move when bytes are dropped: got %d, want 10", got)
}
if got := u.from(4); !bytes.Equal(got, []byte("efghij")) {
t.Fatalf("from(4) = %q", got)
}
// Below the retained region: the peer named an offset we can no longer
// satisfy, which must be reported rather than silently clamped — replaying
// the wrong range is worse than refusing to replay.
if got := u.from(3); got != nil {
t.Fatalf("from(3) below base = %q, want nil", got)
}
if got := u.from(11); got != nil {
t.Fatalf("from(11) past end = %q, want nil", got)
}
}
// Interleaving appends and advances is the steady-state pattern: credit arrives
// every half window while the sender keeps writing. The buffer must stay exact
// across the compaction that eventually triggers.
func TestUnackedSurvivesInterleavedAppendAndAdvance(t *testing.T) {
var u unackedBuf
var sent []byte
var acked int64
for i := 0; i < 200; i++ {
chunk := bytes.Repeat([]byte{byte(i)}, 97)
sent = append(sent, chunk...)
// emit's order: reclaim what has been credited so far, then retain the
// new chunk. The base therefore trails the credit that arrived since.
base := acked
u.advance(base)
u.append(chunk)
if got, want := u.end(), int64(len(sent)); got != want {
t.Fatalf("round %d: end = %d, want %d", i, got, want)
}
if got, want := u.length(), len(sent)-int(base); got != want {
t.Fatalf("round %d: length = %d, want %d", i, got, want)
}
if got, want := u.from(base), sent[base:]; !bytes.Equal(got, want) {
t.Fatalf("round %d: retained region diverges from what was sent", i)
}
// The peer can only ever credit bytes it has actually received.
if i%3 == 0 {
if acked += 61; acked > int64(len(sent)) {
acked = int64(len(sent))
}
}
}
}
// Compaction reuses the backing array, so a stream that runs for hours must not
// grow one: this is a full window per stream, on both sides.
func TestUnackedReclaimsBackingArray(t *testing.T) {
var u unackedBuf
chunk := bytes.Repeat([]byte{7}, 4096)
for i := 0; i < 500; i++ {
u.advance(u.end()) // fully credited every round
u.append(chunk)
}
if u.length() != len(chunk) {
t.Fatalf("length = %d, want %d", u.length(), len(chunk))
}
if cap(u.buf) > 8*len(chunk) {
t.Fatalf("backing array grew to %d bytes for a %d-byte window", cap(u.buf), len(chunk))
}
}
+372
View File
@@ -0,0 +1,372 @@
package client
import (
"crypto/hmac"
"crypto/md5"
"crypto/sha256"
"errors"
"sync"
"sync/atomic"
"github.com/iceBear67/redapricot/client/wire"
)
// Velocity "modern forwarding" support.
//
// A Paper backend configured with Velocity modern player-info forwarding runs
// in offline mode and instead trusts a signed login payload from its proxy:
// during the login phase it sends a Login Plugin Request on channel
// "velocity:player_info" and expects a Login Plugin Response whose data is an
// HMAC-SHA256 signature followed by the player's real address, UUID, username
// and profile properties.
//
// redapricot is a transparent tunnel, so that request would reach the vanilla
// player, who cannot answer it and gets kicked. When a mapping sets
// "velocitySecret", the stream answers on the player's behalf: it observes the
// player's Handshake and Login Start to learn the protocol version, username
// and UUID, swallows the backend's velocity:player_info request instead of
// forwarding it, and injects the signed response. Everything else — and
// everything after the exchange — is forwarded verbatim. On any traffic that
// does not look like a vanilla login (status pings, parse errors, oversized
// packets) the stream fails open into pure passthrough.
//
// The forwarded profile carries no properties (skin/cape textures): the tunnel
// never talks to Mojang, exactly like an offline-mode proxy.
const (
velocityChannel = "velocity:player_info"
// Forwarding payload versions (Velocity's VelocityConstants). We never use
// versions 2/3 (WITH_KEY): they exist only for 1.191.19.2 chat signing,
// and version 1 remains acceptable to every backend.
velocityVersionDefault = 1
velocityVersionLazySession = 4
// Minecraft protocol versions at which the Login Start layout changes.
protocol1_19 = 759 // + optional signature key
protocol1_19_1 = 760 // + optional profile UUID (after the key)
protocol1_19_3 = 761 // key removed, optional UUID stays
protocol1_20_2 = 764 // UUID mandatory
// Handshake intents that enter the login phase.
intentLogin = 2
intentTransfer = 3
// Login-phase packet ids (stable across protocol versions).
loginC2SPluginResponse = 0x02
loginS2CDisconnect = 0x00
loginS2CEncryptionRequest = 0x01
loginS2CSuccess = 0x02
loginS2CSetCompression = 0x03
loginS2CPluginRequest = 0x04
// Sniff-buffer caps. Login-phase packets are small; anything larger means
// this is not the exchange we are looking for.
maxC2SSniff = 8 << 10
maxS2CSniff = 64 << 10
)
var errVelocitySniff = errors.New("velocity: connection does not follow the vanilla login flow")
// velocityForwarder is the per-stream login interceptor. ObserveC2S is called
// from the worker read loop, ProcessS2C from the stream's destination-read
// goroutine; the mutex orders them, and passthrough short-circuits both once
// interception is over.
type velocityForwarder struct {
passthrough atomic.Bool // fully transparent, buffers empty: skip the mutex
mu sync.Mutex
secret []byte
srcIP string
// player -> server observation
c2sBuf []byte
c2sDone bool
handshakeParsed bool
protocol int
loginStartSeen bool
username string
uuid [16]byte
// server -> player interception; done means the s2c side (and with it the
// whole interceptor) is finished.
s2cBuf []byte
done bool
}
func newVelocityForwarder(secret, srcIP string) *velocityForwarder {
return &velocityForwarder{secret: []byte(secret), srcIP: srcIP}
}
// Passthrough reports that interception is over and both directions may skip
// the forwarder entirely.
func (v *velocityForwarder) Passthrough() bool { return v.passthrough.Load() }
// abortLocked gives up on interception: the stream becomes pure passthrough.
// s2cBuf is deliberately kept — ProcessS2C flushes it to the player.
func (v *velocityForwarder) abortLocked() {
v.done = true
v.c2sDone = true
v.c2sBuf = nil
if len(v.s2cBuf) == 0 {
v.passthrough.Store(true)
}
}
// ObserveC2S watches player->server bytes (already being forwarded verbatim by
// the caller) until the Handshake and Login Start have been parsed.
func (v *velocityForwarder) ObserveC2S(data []byte) {
if v.passthrough.Load() {
return
}
v.mu.Lock()
defer v.mu.Unlock()
if v.c2sDone {
return
}
v.c2sBuf = append(v.c2sBuf, data...)
for !v.c2sDone {
_, body, rest, ok, err := nextPacket(v.c2sBuf, maxC2SSniff)
if err != nil {
v.abortLocked()
return
}
if !ok {
if len(v.c2sBuf) > maxC2SSniff {
v.abortLocked()
}
return
}
v.c2sBuf = rest
if err := v.observeC2SPacket(body); err != nil {
v.abortLocked()
return
}
}
v.c2sBuf = nil
}
// observeC2SPacket handles one player packet: first the Handshake, then Login
// Start. Any deviation from the vanilla login flow is an error (→ fail open).
func (v *velocityForwarder) observeC2SPacket(body []byte) error {
r := wire.NewReader(body)
id, err := r.VarInt()
if err != nil || id != 0x00 { // Handshake and Login Start are both 0x00
return errVelocitySniff
}
if !v.handshakeParsed {
proto, perr := r.VarInt()
_, aerr := r.String() // address
_, poerr := r.U16() // port
intent, ierr := r.VarInt()
if perr != nil || aerr != nil || poerr != nil || ierr != nil {
return errVelocitySniff
}
if intent != intentLogin && intent != intentTransfer {
return errVelocitySniff // status ping etc.: nothing to intercept
}
v.protocol = proto
v.handshakeParsed = true
return nil
}
return v.parseLoginStart(r)
}
func (v *velocityForwarder) parseLoginStart(r *wire.Reader) error {
name, err := r.String()
if err != nil || len(name) == 0 || len(name) > 16 {
return errVelocitySniff
}
if v.protocol >= protocol1_19 && v.protocol < protocol1_19_3 {
// Optional chat-signing key: expiry + public key + signature.
hasKey, err := r.U8()
if err != nil {
return errVelocitySniff
}
if hasKey != 0 {
if _, err := r.I64(); err != nil {
return errVelocitySniff
}
for i := 0; i < 2; i++ {
n, err := r.VarInt()
if err != nil {
return errVelocitySniff
}
if _, err := r.Bytes(n); err != nil {
return errVelocitySniff
}
}
}
}
haveUUID := false
switch {
case v.protocol >= protocol1_20_2:
haveUUID = true
case v.protocol >= protocol1_19_1:
flag, err := r.U8()
if err != nil {
return errVelocitySniff
}
haveUUID = flag != 0
}
if haveUUID {
b, err := r.Bytes(16)
if err != nil {
return errVelocitySniff
}
copy(v.uuid[:], b)
} else {
v.uuid = offlineUUID(name)
}
v.username = name
v.loginStartSeen = true
v.c2sDone = true
return nil
}
// ProcessS2C consumes one chunk of server->player bytes. It returns the bytes
// to forward to the player and, once the velocity query has been answered, the
// Login Plugin Response to inject towards the server. Complete packets are
// forwarded as they parse; a trailing partial packet stays buffered until the
// next chunk.
func (v *velocityForwarder) ProcessS2C(data []byte) (forward, inject []byte) {
v.mu.Lock()
defer v.mu.Unlock()
if v.done {
// Interception ended from the c2s side while bytes sat buffered here.
if len(v.s2cBuf) > 0 {
forward = append(v.s2cBuf, data...)
v.s2cBuf = nil
v.passthrough.Store(true)
return forward, nil
}
v.passthrough.Store(true)
return data, nil
}
v.s2cBuf = append(v.s2cBuf, data...)
loop:
for {
raw, body, rest, ok, err := nextPacket(v.s2cBuf, maxS2CSniff)
if err != nil || (!ok && len(v.s2cBuf) > maxS2CSniff) {
v.abortLocked() // unconsumed bytes are flushed below
break
}
if !ok {
break // partial packet: wait for the next chunk
}
r := wire.NewReader(body)
id, err := r.VarInt()
if err != nil {
v.abortLocked()
break
}
switch id {
case loginS2CPluginRequest:
msgID, merr := r.VarInt()
channel, cerr := r.String()
if merr != nil || cerr != nil {
v.abortLocked()
break loop
}
if channel == velocityChannel {
if !v.loginStartSeen {
// Cannot answer without a parsed Login Start; let the
// request through — the backend will kick the player with
// its own clear message.
v.abortLocked()
break loop
}
inject = v.buildResponseLocked(msgID, r.Remaining())
v.s2cBuf = rest // swallow the request: the player never sees it
v.done = true
break loop
}
// Another plugin channel (e.g. a mod handshake): the player
// answers it itself; forward and keep watching.
case loginS2CDisconnect, loginS2CEncryptionRequest, loginS2CSuccess, loginS2CSetCompression:
// Login phase is over (or turning encrypted/compressed) and no
// velocity query showed up: stop watching.
v.done = true
default:
// Cookie Request (0x05, 1.20.5+) or future packets: forward.
}
v.s2cBuf = rest
forward = append(forward, raw...)
if v.done {
break
}
}
if v.done {
forward = append(forward, v.s2cBuf...)
v.s2cBuf = nil
v.c2sBuf = nil
v.c2sDone = true
v.passthrough.Store(true)
}
return forward, inject
}
// buildResponseLocked crafts the serverbound Login Plugin Response carrying the
// signed forwarding payload (mirrors Velocity's createForwardingData).
func (v *velocityForwarder) buildResponseLocked(msgID int, reqData []byte) []byte {
// The request data is the backend's maximum supported forwarding version
// (absent on very old backends → 1).
requested := velocityVersionDefault
if len(reqData) > 0 {
if n, err := wire.NewReader(reqData).VarInt(); err == nil {
requested = n
}
}
version := velocityVersionDefault
if requested >= velocityVersionLazySession && v.protocol >= protocol1_19_3 {
version = velocityVersionLazySession
}
payload := wire.NewWriter().
VarInt(version).
String(v.srcIP).
Bytes(v.uuid[:]). // UUID = 16 raw bytes (two big-endian longs)
String(v.username).
VarInt(0). // profile properties
Out()
mac := hmac.New(sha256.New, v.secret)
mac.Write(payload)
body := wire.NewWriter().
VarInt(loginC2SPluginResponse).
VarInt(msgID).
U8(1). // successful
Bytes(mac.Sum(nil)).
Bytes(payload).
Out()
return append(wire.AppendVarInt(nil, len(body)), body...)
}
// nextPacket splits one length-prefixed Minecraft packet off buf. raw includes
// the length header, body is the packet payload, rest what follows. ok is
// false while the packet is still incomplete; err reports a malformed or
// oversized length header.
func nextPacket(buf []byte, max int) (raw, body, rest []byte, ok bool, err error) {
r := wire.NewReader(buf)
n, verr := r.VarInt()
if verr != nil {
if len(buf) >= wire.VarIntMaxBytes {
return nil, nil, buf, false, verr
}
return nil, nil, buf, false, nil // header not complete yet
}
if n <= 0 || n > max {
return nil, nil, buf, false, errVelocitySniff
}
hdr := len(buf) - len(r.Remaining())
if len(buf) < hdr+n {
return nil, nil, buf, false, nil
}
return buf[:hdr+n], buf[hdr : hdr+n], buf[hdr+n:], true, nil
}
// offlineUUID derives the offline-mode UUID for a username, identical to
// Java's UUID.nameUUIDFromBytes("OfflinePlayer:" + name): a v3 (MD5) UUID.
func offlineUUID(name string) [16]byte {
sum := md5.Sum([]byte("OfflinePlayer:" + name))
sum[6] = sum[6]&0x0f | 0x30 // version 3
sum[8] = sum[8]&0x3f | 0x80 // IETF variant
return sum
}
+348
View File
@@ -0,0 +1,348 @@
package client
import (
"bytes"
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"testing"
"github.com/iceBear67/redapricot/client/wire"
)
// ---- packet builders (player/backend side) ----
func mkPacket(body []byte) []byte {
return append(wire.AppendVarInt(nil, len(body)), body...)
}
func mkLoginStart(proto int, name string, uuid []byte, keyData bool) []byte {
w := wire.NewWriter().VarInt(0x00).String(name)
if proto >= protocol1_19 && proto < protocol1_19_3 {
if keyData {
w.U8(1).I64(1234567890)
pub := bytes.Repeat([]byte{0xAA}, 33)
sig := bytes.Repeat([]byte{0xBB}, 17)
w.VarInt(len(pub)).Bytes(pub).VarInt(len(sig)).Bytes(sig)
} else {
w.U8(0)
}
}
switch {
case proto >= protocol1_20_2:
w.Bytes(uuid)
case proto >= protocol1_19_1:
if uuid != nil {
w.U8(1).Bytes(uuid)
} else {
w.U8(0)
}
}
return mkPacket(w.Out())
}
func mkPluginRequest(msgID int, channel string, data []byte) []byte {
body := wire.NewWriter().VarInt(loginS2CPluginRequest).VarInt(msgID).String(channel).Bytes(data).Out()
return mkPacket(body)
}
// feedLogin drives a full player login prologue through ObserveC2S in n-byte
// chunks.
func feedLogin(v *velocityForwarder, proto int, intent int, loginStart []byte, chunk int) {
stream := wire.BuildHandshake(proto, "mc.example.com", 25565, intent)
if loginStart != nil {
stream = append(stream, loginStart...)
}
for len(stream) > 0 {
n := chunk
if n > len(stream) {
n = len(stream)
}
v.ObserveC2S(stream[:n])
stream = stream[n:]
}
}
// parseResponse validates the injected Login Plugin Response and returns the
// echoed message id and the signed forwarding payload.
func parseResponse(t *testing.T, secret string, inject []byte) (msgID int, payload *wire.Reader) {
t.Helper()
r := wire.NewReader(inject)
plen, err := r.VarInt()
if err != nil || plen != len(r.Remaining()) {
t.Fatalf("bad response length prefix: %v (declared %d, have %d)", err, plen, len(r.Remaining()))
}
id, _ := r.VarInt()
if id != loginC2SPluginResponse {
t.Fatalf("response packet id = %#x, want 0x02", id)
}
msgID, _ = r.VarInt()
ok, _ := r.U8()
if ok != 1 {
t.Fatalf("response not marked successful")
}
sig, err := r.Bytes(32)
if err != nil {
t.Fatalf("response missing signature: %v", err)
}
data := r.Remaining()
mac := hmac.New(sha256.New, []byte(secret))
mac.Write(data)
if !hmac.Equal(sig, mac.Sum(nil)) {
t.Fatalf("forwarding payload signature does not verify")
}
return msgID, wire.NewReader(data)
}
func assertPayload(t *testing.T, r *wire.Reader, version int, ip, name, uuidHex string) {
t.Helper()
gotVer, _ := r.VarInt()
if gotVer != version {
t.Fatalf("forwarding version = %d, want %d", gotVer, version)
}
gotIP, _ := r.String()
if gotIP != ip {
t.Fatalf("forwarded address = %q, want %q", gotIP, ip)
}
gotUUID, err := r.Bytes(16)
if err != nil {
t.Fatalf("payload missing uuid: %v", err)
}
if hex.EncodeToString(gotUUID) != uuidHex {
t.Fatalf("forwarded uuid = %x, want %s", gotUUID, uuidHex)
}
gotName, _ := r.String()
if gotName != name {
t.Fatalf("forwarded username = %q, want %q", gotName, name)
}
props, err := r.VarInt()
if err != nil || props != 0 {
t.Fatalf("forwarded properties = %d (%v), want 0", props, err)
}
if len(r.Remaining()) != 0 {
t.Fatalf("trailing bytes in forwarding payload: %x", r.Remaining())
}
}
// ---- tests ----
const testSecret = "unit-secret"
func TestVelocityInterceptModern(t *testing.T) {
uuid, _ := hex.DecodeString("00112233445566778899aabbccddeeff")
v := newVelocityForwarder(testSecret, "203.0.113.7")
feedLogin(v, 767, intentLogin, mkLoginStart(767, "icybear", uuid, false), 1)
// Backend query, requesting up to forwarding version 4, fed byte by byte:
// nothing may reach the player, and the response appears with the last byte.
req := mkPluginRequest(99, velocityChannel, []byte{0x04})
var inject []byte
for i, b := range req {
fwd, inj := v.ProcessS2C([]byte{b})
if len(fwd) != 0 {
t.Fatalf("byte %d: request leaked to the player: %x", i, fwd)
}
if inj != nil {
inject = inj
}
}
if inject == nil {
t.Fatalf("no response was injected")
}
msgID, payload := parseResponse(t, testSecret, inject)
if msgID != 99 {
t.Fatalf("echoed message id = %d, want 99", msgID)
}
assertPayload(t, payload, velocityVersionLazySession, "203.0.113.7", "icybear",
"00112233445566778899aabbccddeeff")
if !v.Passthrough() {
t.Fatalf("interceptor should be passthrough after answering")
}
garbage := []byte{0xde, 0xad, 0xbe, 0xef}
if fwd, inj := v.ProcessS2C(garbage); !bytes.Equal(fwd, garbage) || inj != nil {
t.Fatalf("post-login bytes not passed through verbatim")
}
}
func TestVelocityNegativeMessageID(t *testing.T) {
// Paper picks the message id with ThreadLocalRandom.nextInt(): it is
// negative half the time and must be echoed bit-exactly.
uuid, _ := hex.DecodeString("00112233445566778899aabbccddeeff")
v := newVelocityForwarder(testSecret, "198.51.100.1")
feedLogin(v, 767, intentLogin, mkLoginStart(767, "neg", uuid, false), 64)
_, inject := v.ProcessS2C(mkPluginRequest(-123456, velocityChannel, []byte{0x04}))
if inject == nil {
t.Fatalf("no response was injected")
}
msgID, _ := parseResponse(t, testSecret, inject)
if !bytes.Equal(wire.AppendVarInt(nil, msgID), wire.AppendVarInt(nil, -123456)) {
t.Fatalf("negative message id not echoed bit-exactly (got %d)", msgID)
}
}
func TestVelocityOfflineUUIDAndV1(t *testing.T) {
// 1.18.2 player: no UUID in Login Start -> Java's offline UUID; an old
// backend requesting version 1 gets version 1.
v := newVelocityForwarder(testSecret, "192.0.2.9")
feedLogin(v, 758, intentLogin, mkLoginStart(758, "Notch", nil, false), 3)
_, inject := v.ProcessS2C(mkPluginRequest(7, velocityChannel, []byte{0x01}))
if inject == nil {
t.Fatalf("no response was injected")
}
_, payload := parseResponse(t, testSecret, inject)
// UUID.nameUUIDFromBytes("OfflinePlayer:Notch".getBytes(UTF_8)).
assertPayload(t, payload, velocityVersionDefault, "192.0.2.9", "Notch",
"b50ad385829d3141a2167e7d7539ba7f")
}
func TestVelocityVersionGating(t *testing.T) {
// A modern backend (requests 4) behind a pre-1.19.3 player must get v1.
v := newVelocityForwarder(testSecret, "192.0.2.9")
feedLogin(v, 758, intentLogin, mkLoginStart(758, "Old", nil, false), 5)
_, inject := v.ProcessS2C(mkPluginRequest(1, velocityChannel, []byte{0x04}))
_, payload := parseResponse(t, testSecret, inject)
ver, _ := payload.VarInt()
if ver != velocityVersionDefault {
t.Fatalf("version = %d, want 1 for a pre-1.19.3 player", ver)
}
// An empty request (very old backend) also means v1.
v2 := newVelocityForwarder(testSecret, "192.0.2.9")
feedLogin(v2, 767, intentLogin, mkLoginStart(767, "New", make([]byte, 16), false), 5)
_, inject2 := v2.ProcessS2C(mkPluginRequest(1, velocityChannel, nil))
_, payload2 := parseResponse(t, testSecret, inject2)
ver2, _ := payload2.VarInt()
if ver2 != velocityVersionDefault {
t.Fatalf("version = %d, want 1 for an empty version request", ver2)
}
}
func TestVelocityLoginStartVariants(t *testing.T) {
uuid, _ := hex.DecodeString("ffeeddccbbaa99887766554433221100")
cases := []struct {
name string
proto int
start []byte
uuidHex string
}{
{"1.19 with key, no uuid", 759, mkLoginStart(759, "Notch", nil, true),
"b50ad385829d3141a2167e7d7539ba7f"},
{"1.19.1 with key and uuid", 760, mkLoginStart(760, "Notch", uuid, true),
"ffeeddccbbaa99887766554433221100"},
{"1.19.3 optional uuid present", 761, mkLoginStart(761, "Notch", uuid, false),
"ffeeddccbbaa99887766554433221100"},
{"1.19.3 optional uuid absent", 761, mkLoginStart(761, "Notch", nil, false),
"b50ad385829d3141a2167e7d7539ba7f"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
v := newVelocityForwarder(testSecret, "192.0.2.1")
feedLogin(v, tc.proto, intentLogin, tc.start, 2)
_, inject := v.ProcessS2C(mkPluginRequest(3, velocityChannel, []byte{0x01}))
if inject == nil {
t.Fatalf("no response was injected")
}
_, payload := parseResponse(t, testSecret, inject)
assertPayload(t, payload, velocityVersionDefault, "192.0.2.1", "Notch", tc.uuidHex)
})
}
}
func TestVelocityStatusPassthrough(t *testing.T) {
v := newVelocityForwarder(testSecret, "192.0.2.1")
feedLogin(v, 767, 1 /* status */, nil, 100)
if !v.Passthrough() {
t.Fatalf("status intent should turn the stream transparent")
}
data := []byte("not a minecraft packet at all")
if fwd, inj := v.ProcessS2C(data); !bytes.Equal(fwd, data) || inj != nil {
t.Fatalf("status traffic must pass through untouched")
}
}
func TestVelocityOtherChannelForwarded(t *testing.T) {
uuid := make([]byte, 16)
v := newVelocityForwarder(testSecret, "192.0.2.1")
feedLogin(v, 767, intentLogin, mkLoginStart(767, "modded", uuid, false), 50)
// A modded-handshake query and the velocity query coalesced in one chunk:
// the first must reach the player, the second must not.
other := mkPluginRequest(1, "fml:loginwrapper", []byte{0x00, 0x01})
velo := mkPluginRequest(2, velocityChannel, []byte{0x04})
fwd, inject := v.ProcessS2C(append(append([]byte{}, other...), velo...))
if !bytes.Equal(fwd, other) {
t.Fatalf("non-velocity query not forwarded verbatim:\n got %x\nwant %x", fwd, other)
}
if inject == nil {
t.Fatalf("velocity query in the same chunk was not answered")
}
}
func TestVelocityNoQueryLoginSuccess(t *testing.T) {
// Backend without velocity forwarding: the first login packet ends
// interception and everything flows verbatim.
uuid := make([]byte, 16)
v := newVelocityForwarder(testSecret, "192.0.2.1")
feedLogin(v, 767, intentLogin, mkLoginStart(767, "plain", uuid, false), 50)
success := mkPacket(wire.NewWriter().VarInt(loginS2CSuccess).Bytes(uuid).String("plain").VarInt(0).Out())
tail := []byte("compressed gibberish after login")
fwd, inject := v.ProcessS2C(append(append([]byte{}, success...), tail...))
if inject != nil {
t.Fatalf("nothing should be injected without a velocity query")
}
want := append(append([]byte{}, success...), tail...)
if !bytes.Equal(fwd, want) {
t.Fatalf("login success not flushed verbatim")
}
if !v.Passthrough() {
t.Fatalf("interceptor should be passthrough after Login Success")
}
}
func TestVelocityQueryBeforeLoginStartFailsOpen(t *testing.T) {
// A query arriving before the Login Start was observed cannot be answered:
// it must reach the player unmodified (who will then be kicked by the
// backend with its own message).
v := newVelocityForwarder(testSecret, "192.0.2.1")
feedLogin(v, 767, intentLogin, nil, 50) // handshake only
req := mkPluginRequest(5, velocityChannel, []byte{0x04})
fwd, inject := v.ProcessS2C(req)
if inject != nil {
t.Fatalf("must not answer without a Login Start")
}
if !bytes.Equal(fwd, req) {
t.Fatalf("unanswerable query not passed through")
}
if !v.Passthrough() {
t.Fatalf("interceptor should fail open")
}
}
func TestVelocityOversizedC2SFailsOpen(t *testing.T) {
v := newVelocityForwarder(testSecret, "192.0.2.1")
// A declared c2s packet length beyond the sniff cap aborts interception.
v.ObserveC2S(wire.AppendVarInt(nil, maxC2SSniff+1))
if !v.Passthrough() {
t.Fatalf("oversized login packet should turn the stream transparent")
}
}
func TestVelocityC2SAbortFlushesBufferedS2C(t *testing.T) {
v := newVelocityForwarder(testSecret, "192.0.2.1")
feedLogin(v, 767, intentLogin, nil, 50) // handshake only; login pending
req := mkPluginRequest(5, velocityChannel, []byte{0x04})
half := len(req) / 2
if fwd, _ := v.ProcessS2C(req[:half]); len(fwd) != 0 {
t.Fatalf("partial packet must stay buffered")
}
// The c2s side now aborts (e.g. unparseable player bytes) while s2c bytes
// sit buffered: they must not be lost.
v.ObserveC2S(wire.AppendVarInt(nil, maxC2SSniff+1))
fwd, inject := v.ProcessS2C(req[half:])
if inject != nil {
t.Fatalf("aborted interceptor must not inject")
}
if !bytes.Equal(fwd, req) {
t.Fatalf("buffered s2c bytes lost on abort:\n got %x\nwant %x", fwd, req)
}
}
+32 -3
View File
@@ -6,6 +6,7 @@ import (
"io" "io"
"net" "net"
"sync" "sync"
"time"
"golang.org/x/crypto/chacha20" "golang.org/x/crypto/chacha20"
) )
@@ -13,7 +14,19 @@ import (
// MaxFrame is the maximum decrypted frame payload size (1 MiB). // MaxFrame is the maximum decrypted frame payload size (1 MiB).
const MaxFrame = 1 << 20 const MaxFrame = 1 << 20
var errFrameTooBig = errors.New("wire: frame exceeds max size") // WriteTimeout bounds a single frame write. A peer that stops reading must not
// be able to park this connection inside WriteFrame forever: the write mutex
// is held for the whole socket write, so one stalled write would otherwise
// wedge liveness (PING/PONG) and WND/FIN on this tunnel.
const WriteTimeout = 30 * time.Second
var (
errFrameTooBig = errors.New("wire: frame exceeds max size")
// ErrBroken is returned once a write has failed. The ChaCha20 keystream has
// already advanced (and the socket may hold a partial frame), so the
// connection can never be resynchronized and is closed for good.
ErrBroken = errors.New("wire: connection is broken")
)
// FramedConn is the encrypted, length-prefixed frame transport (PROTOCOL.md §3.1). // FramedConn is the encrypted, length-prefixed frame transport (PROTOCOL.md §3.1).
// The VarInt length prefix is plaintext; the payload is ChaCha20-encrypted with a // The VarInt length prefix is plaintext; the payload is ChaCha20-encrypted with a
@@ -24,7 +37,9 @@ type FramedConn struct {
r *bufio.Reader r *bufio.Reader
in *chacha20.Cipher in *chacha20.Cipher
out *chacha20.Cipher out *chacha20.Cipher
wmu sync.Mutex
wmu sync.Mutex
broken bool
} }
func NewFramedConn(conn net.Conn, in, out *chacha20.Cipher) *FramedConn { func NewFramedConn(conn net.Conn, in, out *chacha20.Cipher) *FramedConn {
@@ -62,15 +77,29 @@ func (f *FramedConn) ReadFrame() ([]byte, error) {
} }
// WriteFrame encrypts and sends one frame payload. Safe for concurrent callers. // WriteFrame encrypts and sends one frame payload. Safe for concurrent callers.
//
// The write is bounded by WriteTimeout. On any write error the connection is
// marked broken and closed, which unblocks the reader so the owner can tear the
// session down instead of leaving every stream parked on the write mutex.
func (f *FramedConn) WriteFrame(payload []byte) error { func (f *FramedConn) WriteFrame(payload []byte) error {
f.wmu.Lock() f.wmu.Lock()
defer f.wmu.Unlock() defer f.wmu.Unlock()
if f.broken {
return ErrBroken
}
ct := make([]byte, len(payload)) ct := make([]byte, len(payload))
f.out.XORKeyStream(ct, payload) f.out.XORKeyStream(ct, payload)
out := AppendVarInt(make([]byte, 0, VarIntMaxBytes+len(ct)), len(ct)) out := AppendVarInt(make([]byte, 0, VarIntMaxBytes+len(ct)), len(ct))
out = append(out, ct...) out = append(out, ct...)
_ = f.conn.SetWriteDeadline(time.Now().Add(WriteTimeout))
_, err := f.conn.Write(out) _, err := f.conn.Write(out)
return err if err != nil {
f.broken = true
_ = f.conn.Close()
return err
}
_ = f.conn.SetWriteDeadline(time.Time{})
return nil
} }
func (f *FramedConn) Close() error { return f.conn.Close() } func (f *FramedConn) Close() error { return f.conn.Close() }
+533 -140
View File
@@ -1,75 +1,129 @@
package client package client
import ( import (
"errors"
"fmt"
"log" "log"
"net" "net"
"sync" "sync"
"sync/atomic"
"time" "time"
"github.com/iceBear67/redapricot/client/wire" "github.com/iceBear67/redapricot/client/wire"
) )
// WorkerPool manages up to maxConn worker connections and allocates streams // errPoolClosed is returned by Dial after Close: the set is shutting down
// using the least-loaded strategy (PROTOCOL.md §7.1). // and must not start new dials, so a caller (handleControlRequest,
// allocateForResume) gives up rather than waiting on a hub that will never
// be used.
var errPoolClosed = errors.New("worker set closed")
// errTooManyTunnels is returned when live+in-flight worker conns already
// equal maxTunnels. The ControlRequest is dropped; the hub closes the player
// when pendingTimeoutMs fires.
var errTooManyTunnels = errors.New("maxTunnels reached")
// dialAttempts bounds how many times a caller retries Dial when the conn it
// was handed dies before the stream could be attached to it. The race is
// narrow; an unbounded loop would spin against a hub that is refusing every
// connection.
const dialAttempts = 3
// WorkerPool tracks live 1:1 worker connections up to maxTunnels
// (PROTOCOL.md §7.1). There is no least-loaded placement and no sharing:
// every player gets its own TCP connection.
type WorkerPool struct { type WorkerPool struct {
client *Client client *Client
maxConn int maxTunnels int
mu sync.Mutex connSeq atomic.Int64 // conn ids, for log correlation
conns []*WorkerConn
mu sync.Mutex
conns map[*WorkerConn]struct{}
dialing int // dials currently in flight
closed bool // closeAll ran; no new conns may join
} }
func newWorkerPool(c *Client, maxConn int) *WorkerPool { func newWorkerPool(c *Client, maxTunnels int) *WorkerPool {
return &WorkerPool{client: c, maxConn: maxConn} return &WorkerPool{
client: c,
maxTunnels: maxTunnels,
conns: make(map[*WorkerConn]struct{}),
}
} }
// Allocate returns a worker conn and a fresh stream id to place a new stream on. // Dial opens a dedicated worker conn for one player.
func (p *WorkerPool) Allocate() (*WorkerConn, int, error) { //
// A dial is never performed while holding p.mu: session establishment involves
// network I/O, and holding the lock across it would park every other player
// behind one unresponsive hub. Each caller dials independently; unlike the
// old mux pool there is no shared conn to wait for.
func (p *WorkerPool) Dial() (*WorkerConn, error) {
p.mu.Lock() p.mu.Lock()
defer p.mu.Unlock() if p.closed {
p.mu.Unlock()
var best *WorkerConn return nil, errPoolClosed
bestCount := 0
for _, wc := range p.conns {
n := wc.streamCount()
if best == nil || n < bestCount {
best = wc
bestCount = n
}
} }
if len(p.conns)+p.dialing >= p.maxTunnels {
needNew := best == nil || (bestCount > SaturationThreshold && len(p.conns) < p.maxConn) p.mu.Unlock()
if needNew { return nil, errTooManyTunnels
wc, err := p.dialWorker()
if err != nil {
if best == nil {
return nil, 0, err
}
log.Printf("worker dial failed, reusing existing conn: %v", err)
} else {
p.conns = append(p.conns, wc)
best = wc
}
} }
return best, best.newSid(), nil p.dialing++
p.mu.Unlock()
wc, err := p.dialWorker()
p.mu.Lock()
p.dialing--
if err != nil {
p.mu.Unlock()
return nil, err
}
if p.closed {
// Close raced this dial: the conn must not enter the set.
p.mu.Unlock()
_ = wc.fc.Close()
return nil, errPoolClosed
}
if len(p.conns) >= p.maxTunnels {
p.mu.Unlock()
_ = wc.fc.Close()
return nil, errTooManyTunnels
}
p.conns[wc] = struct{}{}
p.mu.Unlock()
return wc, nil
} }
// dialWorker establishes one worker conn. It must be called without p.mu held.
// The dial runs on the client's context so Close aborts it mid-handshake.
func (p *WorkerPool) dialWorker() (*WorkerConn, error) { func (p *WorkerPool) dialWorker() (*WorkerConn, error) {
fc, peerWnd, err := p.client.dialSession(MagicWorker) sess, err := p.client.dialSession(p.client.ctx, MagicWorker)
if err != nil { if err != nil {
return nil, err return nil, err
} }
wc := &WorkerConn{ wc := &WorkerConn{
pool: p, pool: p,
fc: fc, fc: sess.fc,
sendWndInit: peerWnd, sendWndInit: sess.peerWnd,
recvWndInit: p.client.streamWnd, resume: sess.resume,
streams: make(map[int]*Stream), grace: p.client.resumeGrace(sess.hubGrace),
nextSid: 1, id: int(p.connSeq.Add(1)),
done: make(chan struct{}),
} }
if p.client.statsOn() {
wc.stats = &connStats{opened: time.Now()}
}
wc.lastPong.Store(time.Now().UnixMilli())
go wc.readLoop() go wc.readLoop()
log.Printf("opened worker conn (#%d in pool, send window %d, recv window %d)", if sess.heartbeat {
len(p.conns)+1, wc.sendWndInit, wc.recvWndInit) go wc.heartbeatLoop(p.client.cfg.pingInterval(), p.client.cfg.heartbeatTimeout())
} else {
log.Printf("worker conn: hub does not support the worker heartbeat; " +
"a silently dropped path will only be caught by TCP keepalive")
}
log.Printf("opened worker conn (send window %d, recv window %d, heartbeat %v, resume %v)",
wc.sendWndInit, p.client.streamWnd, sess.heartbeat, wc.resume)
return wc, nil return wc, nil
} }
@@ -82,144 +136,228 @@ func (p *WorkerPool) count() int {
func (p *WorkerPool) remove(wc *WorkerConn) { func (p *WorkerPool) remove(wc *WorkerConn) {
p.mu.Lock() p.mu.Lock()
defer p.mu.Unlock() defer p.mu.Unlock()
for i, c := range p.conns { delete(p.conns, wc)
if c == wc {
p.conns = append(p.conns[:i], p.conns[i+1:]...)
return
}
}
} }
func (p *WorkerPool) closeAll() { func (p *WorkerPool) closeAll() {
p.mu.Lock() p.mu.Lock()
conns := append([]*WorkerConn(nil), p.conns...) p.closed = true
conns := make([]*WorkerConn, 0, len(p.conns))
for wc := range p.conns {
conns = append(conns, wc)
}
p.mu.Unlock() p.mu.Unlock()
for _, wc := range conns { for _, wc := range conns {
_ = wc.fc.Close() _ = wc.fc.Close()
} }
} }
// WorkerConn is one multiplexed worker connection to the hub. // WorkerConn is one 1:1 worker connection to the hub: it carries exactly one
// player. Only genuinely per-connection state lives here. Client-wide values
// (the shaper, the advertised receive window, the DATA chunk cap) belong to
// Client: reading them through a connection pointer would make every such
// read a re-parenting hazard once a stream can migrate between conns.
type WorkerConn struct { type WorkerConn struct {
pool *WorkerPool pool *WorkerPool
fc *wire.FramedConn fc *wire.FramedConn
sendWndInit int // hub's advertised per-stream receive window (our send budget) sendWndInit int // hub's advertised receive window (our send budget)
recvWndInit int // our advertised per-stream receive window (bounds each recv queue)
mu sync.Mutex // resume is whether this conn negotiated stream resumption, and grace how
streams map[int]*Stream // long a stream parked from it may keep trying to reattach. Both are
nextSid int // per-conn: a reattach may land on a different (or restarted) hub, so the
// flag must be re-checked on the conn that will carry the RESUME.
resume bool
grace time.Duration
done chan struct{} // closed when readLoop exits
lastPong atomic.Int64 // unix ms of the most recent PONG
id int // for log correlation only
stats *connStats // nil unless diagnostics are enabled
mu sync.Mutex
stream *Stream
closed bool // readLoop has exited; attach must refuse
} }
func (wc *WorkerConn) streamCount() int { // heartbeatLoop proves the worker conn is still carrying frames end to end. TCP
// alone cannot tell us: a middlebox that drops an established flow (conntrack
// expiry, firewall state loss) sends no FIN or RST, so the read loop would park
// forever and the player on this conn would silently fail until the process
// restarted.
func (wc *WorkerConn) heartbeatLoop(interval, timeout time.Duration) {
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-wc.done:
return
case <-ticker.C:
if silent := time.Since(time.UnixMilli(wc.lastPong.Load())); silent > timeout {
log.Printf("worker conn silent for %s; dropping it", silent.Round(time.Second))
_ = wc.fc.Close() // readLoop unblocks and tears everything down
return
}
msg := wire.NewWriter().U8(MuxPing).I64(time.Now().UnixMilli()).Out()
if err := wc.fc.WriteFrame(msg); err != nil {
return
}
}
}
}
// attach publishes a stream on this conn, or reports false if the conn has
// already died (or is already bound — which would be a caller bug).
//
// The check is not advisory. Dial hands out a conn, and the conn's readLoop
// can exit before the caller gets here. A blind store would land on a conn
// nothing iterates and the stream would never be torn down.
func (wc *WorkerConn) attach(st *Stream) bool {
wc.mu.Lock() wc.mu.Lock()
defer wc.mu.Unlock() defer wc.mu.Unlock()
return len(wc.streams) if wc.closed || wc.stream != nil {
return false
}
wc.stream = st
return true
} }
func (wc *WorkerConn) newSid() int { func (wc *WorkerConn) getStream() *Stream {
wc.mu.Lock() wc.mu.Lock()
defer wc.mu.Unlock() defer wc.mu.Unlock()
sid := wc.nextSid return wc.stream
wc.nextSid++
return sid
} }
func (wc *WorkerConn) registerStream(sid int, st *Stream) { func (wc *WorkerConn) detach() *Stream {
wc.mu.Lock()
wc.streams[sid] = st
wc.mu.Unlock()
}
func (wc *WorkerConn) getStream(sid int) *Stream {
wc.mu.Lock() wc.mu.Lock()
defer wc.mu.Unlock() defer wc.mu.Unlock()
return wc.streams[sid] st := wc.stream
} wc.stream = nil
func (wc *WorkerConn) removeStream(sid int) *Stream {
wc.mu.Lock()
defer wc.mu.Unlock()
st := wc.streams[sid]
delete(wc.streams, sid)
return st return st
} }
// readLoop dispatches inbound mux frames. It must never block on a stream's // readLoop dispatches inbound tunnel frames. DATA is only enqueued (the
// destination: DATA is only enqueued (the per-stream writeLoop does the actual // stream's writeLoop does the actual destination writes), so a stalled
// destination writes), so one slow destination cannot stall other streams. // destination cannot stall liveness / WND / FIN dispatch on this conn.
func (wc *WorkerConn) readLoop() { func (wc *WorkerConn) readLoop() {
for { for {
payload, err := wc.fc.ReadFrame() payload, err := wc.fc.ReadFrame()
if err != nil { if err != nil {
break break
} }
if wc.stats != nil {
wc.stats.framesIn.Add(1)
}
r := wire.NewReader(payload) r := wire.NewReader(payload)
ftype, err := r.U8() ftype, err := r.U8()
if err != nil { if err != nil {
continue continue
} }
sid, err := r.VarInt()
if err != nil {
continue
}
switch ftype { switch ftype {
case MuxData: case MuxData:
if st := wc.getStream(sid); st != nil { if st := wc.getStream(); st != nil {
st.deliverFromHub(r.Remaining()) st.deliverFromHub(r.Remaining())
} }
case MuxWnd: case MuxWnd:
if delta, err := r.VarInt(); err == nil && delta > 0 { if delta, err := r.VarInt(); err == nil && delta > 0 {
if st := wc.getStream(sid); st != nil { if st := wc.getStream(); st != nil {
st.grantSendWnd(delta) st.grantSendWnd(delta)
} }
} }
case MuxFin: case MuxFin:
// Graceful: drain what is already queued to the destination first. // Graceful: drain what is already queued to the destination first.
if st := wc.removeStream(sid); st != nil { if st := wc.detach(); st != nil {
st.gracefulFin() st.gracefulFin()
} }
case MuxRst: case MuxRst:
if st := wc.removeStream(sid); st != nil { // The reason is an optional trailing byte; older peers send none.
st.teardown(false) reason := RstUnspecified
if b, err := r.U8(); err == nil {
reason = int(b)
}
if st := wc.detach(); st != nil {
st.onRst(reason)
}
case MuxResumeAck:
accepted, aerr := r.I64()
delivered, derr := r.I64()
cid, cerr := r.Bytes(CIDLen)
if aerr != nil || derr != nil || cerr != nil {
continue
}
if st := wc.getStream(); st != nil {
st.deliverResume(resumeResult{accepted: accepted, delivered: delivered, cid: cid})
}
case MuxPing:
nonce, _ := r.I64()
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxPong).I64(nonce).Out())
case MuxPong:
now := time.Now()
wc.lastPong.Store(now.UnixMilli())
// The probe's nonce is the timestamp we sent, echoed back, so the
// round trip is free to measure and nobody was reading it.
if wc.stats != nil {
if sent, err := r.I64(); err == nil {
wc.stats.observeRTT(now.Sub(time.UnixMilli(sent)))
}
} }
default: default:
log.Printf("worker: unknown mux type %d", ftype) log.Printf("worker: unknown frame type %d", ftype)
} }
} }
// Connection lost: tear down all streams and drop from pool. // Connection lost: tear down the bound stream and drop from the set.
close(wc.done)
wc.pool.remove(wc) wc.pool.remove(wc)
wc.mu.Lock() wc.mu.Lock()
streams := make([]*Stream, 0, len(wc.streams)) // Marked before the pointer is cleared, under the same lock, so a concurrent
for _, st := range wc.streams { // attach either lands in the field we are about to drain or is refused.
streams = append(streams, st) wc.closed = true
} st := wc.stream
wc.streams = make(map[int]*Stream) wc.stream = nil
wc.mu.Unlock() wc.mu.Unlock()
for _, st := range streams { if st == nil {
return
}
// Only the tunnel leg died. Where the session negotiated resumption the
// destination socket is kept open and the stream reattaches over a fresh
// conn (§7.5); otherwise this is the old, unconditional teardown.
// During Close there is no reattach to come: a stream that parked now would
// hold its destination socket open past shutdown, so teardown instead.
if wc.pool.client.closing.Load() {
st.teardown(false)
return
}
if !st.park(wc.grace) {
st.teardown(false) st.teardown(false)
} }
} }
func (wc *WorkerConn) sendSyn(sid int, cid []byte) { func (wc *WorkerConn) sendSyn(cid []byte) error {
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxSyn).VarInt(sid).Bytes(cid).Out()) return wc.fc.WriteFrame(wire.NewWriter().U8(MuxSyn).Bytes(cid).Out())
} }
func (wc *WorkerConn) sendData(sid int, data []byte) error { func (wc *WorkerConn) sendData(data []byte) error {
return wc.fc.WriteFrame(wire.NewWriter().U8(MuxData).VarInt(sid).Bytes(data).Out()) err := wc.fc.WriteFrame(wire.NewWriter().U8(MuxData).Bytes(data).Out())
if wc.stats != nil {
wc.stats.framesOut.Add(1)
if err != nil {
wc.stats.writeErrs.Add(1)
}
}
return err
} }
func (wc *WorkerConn) sendFin(sid int) { func (wc *WorkerConn) sendFin() {
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxFin).VarInt(sid).Out()) _ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxFin).Out())
} }
func (wc *WorkerConn) sendRst(sid int) { func (wc *WorkerConn) sendRst() {
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxRst).VarInt(sid).Out()) _ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxRst).Out())
} }
func (wc *WorkerConn) sendWndUpdate(sid, delta int) { func (wc *WorkerConn) sendWndUpdate(delta int) {
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxWnd).VarInt(sid).VarInt(delta).Out()) _ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxWnd).VarInt(delta).Out())
} }
// Stream bridges one player (via the hub) to one destination connection. // Stream bridges one player (via the hub) to one destination connection.
@@ -227,41 +365,123 @@ func (wc *WorkerConn) sendWndUpdate(sid, delta int) {
// Data from the hub is queued and written to the destination by a dedicated // Data from the hub is queued and written to the destination by a dedicated
// writeLoop goroutine. The queue is bounded by the advertised receive window — // writeLoop goroutine. The queue is bounded by the advertised receive window —
// the hub never sends more un-credited bytes, so overflow is a protocol // the hub never sends more un-credited bytes, so overflow is a protocol
// violation and resets the stream. // violation and resets the tunnel.
type Stream struct { type Stream struct {
wc *WorkerConn client *Client
sid int wc atomic.Pointer[WorkerConn]
cid []byte
mapping Mapping mapping Mapping
srcIP string srcIP string
srcPort int srcPort int
vel *velocityForwarder // non-nil when the mapping sets velocitySecret
share shaperShare // this stream's position in the egress fair queue
done chan struct{} // closed on teardown; unparks a shaper wait
// resumable is fixed at creation from the conn's negotiated flag. With it
// false none of the bookkeeping below runs and no buffer is ever allocated,
// so a client with resumption disabled pays exactly what it used to.
resumable bool
stats *streamStats // nil unless diagnostics are enabled
// ackedOffset is the running sum of WND deltas received. Credit is granted
// only as bytes reach the peer's terminal socket, so it is a sound lower
// bound on what has been delivered. Atomic because the worker readLoop
// advances it and must never block behind the send path.
ackedOffset atomic.Int64
// sendMu serializes the send path — buffer the chunk, advance the offset,
// write the frame — against a reattach's retransmit, so replayed bytes can
// never interleave with live ones.
//
// Deliberately not s.mu: deliverFromHub takes s.mu from the worker readLoop,
// and holding s.mu across a WriteFrame would stall frame dispatch (WND/FIN
// /heartbeat replies) on this conn.
sendMu sync.Mutex
un unackedBuf // guarded by sendMu
mu sync.Mutex mu sync.Mutex
cond *sync.Cond cond *sync.Cond
cid []byte // takeover capability; re-minted by the hub on each resume
dest net.Conn dest net.Conn
connected bool connected bool
closed bool closed bool
finPending bool // hub sent FIN; close the destination once the queue drains parked bool // worker conn died; awaiting reattach on a fresh one
q [][]byte // hub -> destination, waiting for writeLoop parkedAt time.Time // when the current hang began; diagnostics only
qBytes int finPending bool // hub sent FIN; close the destination once the queue drains
sendWnd int // flow control: budget for destination -> hub DATA finToHub bool // destination closed while parked; FIN owed once reattached
consumed int // flow control: drained bytes not yet credited back to the hub q []qentry // hub/local -> destination, waiting for writeLoop
qBytes int // hub bytes only: bounds the peer against its window
// acceptedOffset counts hub bytes enqueued toward the destination. This, not
// "bytes written", is what a reattach reports: acceptance is synchronous and
// stable at park time, whereas delivery is signalled asynchronously and goes
// silent exactly when the connection dies — which would under-report and make
// the hub retransmit bytes the player already has.
acceptedOffset int64
// deliveredOffset counts hub bytes actually written to the destination.
// Distinct from acceptedOffset and needed for a different job: a reattach
// replays from what the peer *accepted*, but sizes the window from what it
// *delivered*, because the window is a promise about undelivered bytes.
deliveredOffset int64
sendWnd int // flow control: budget for destination -> hub DATA
consumed int // flow control: drained bytes not yet credited back to the hub
resumeWait chan resumeResult
} }
func newStream(wc *WorkerConn, sid int, cid []byte, m Mapping, ip string, port int) *Stream { // resumeResult is the hub's answer to a RESUME: how far it got in both senses,
s := &Stream{wc: wc, sid: sid, cid: cid, mapping: m, srcIP: ip, srcPort: port, sendWnd: wc.sendWndInit} // plus a fresh CID — or the reason the reattach was refused.
type resumeResult struct {
accepted int64
delivered int64
cid []byte
err error
}
// qentry is one queued write towards the destination. Only hub-originated
// entries take part in flow control; locally injected bytes (the velocity
// login response) are neither counted against the hub's window nor credited
// back when drained.
type qentry struct {
data []byte
fromHub bool
}
func newStream(c *Client, wc *WorkerConn, cid []byte, m Mapping, ip string, port int) *Stream {
s := &Stream{client: c, cid: cid, mapping: m, srcIP: ip, srcPort: port,
resumable: wc.resume, sendWnd: wc.sendWndInit, done: make(chan struct{})}
s.wc.Store(wc)
if c.statsOn() {
s.stats = &streamStats{opened: time.Now()}
}
if m.VelocitySecret != "" {
s.vel = newVelocityForwarder(m.VelocitySecret, ip)
}
s.cond = sync.NewCond(&s.mu) s.cond = sync.NewCond(&s.mu)
return s return s
} }
// conn returns the stream's current worker conn. May be the original or a
// reattach; callers that send must take one snapshot and use it for the
// whole operation so a concurrent rebind cannot split a write across conns.
func (s *Stream) conn() *WorkerConn { return s.wc.Load() }
func (s *Stream) name() string {
if wc := s.conn(); wc != nil {
return fmt.Sprintf("conn%d", wc.id)
}
return "conn?"
}
// run dials the destination, optionally writes the PROXY v2 header, then pumps // run dials the destination, optionally writes the PROXY v2 header, then pumps
// destination -> hub (respecting the stream send window when negotiated). // destination -> hub (respecting the send window when negotiated).
func (s *Stream) run() { func (s *Stream) run() {
dest, err := net.DialTimeout("tcp", s.mapping.Destination, 10*time.Second) dest, err := net.DialTimeout("tcp", s.mapping.Destination, 10*time.Second)
if err != nil { if err != nil {
log.Printf("stream %d: dial %s failed: %v", s.sid, s.mapping.Destination, err) log.Printf("stream %s: dial %s failed: %v", s.name(), s.mapping.Destination, err)
s.wc.removeStream(s.sid) if wc := s.conn(); wc != nil {
s.wc.sendRst(s.sid) wc.detach()
wc.sendRst()
}
s.teardown(false) s.teardown(false)
return return
} }
@@ -272,7 +492,7 @@ func (s *Stream) run() {
if s.mapping.ProxyProtocol { if s.mapping.ProxyProtocol {
if hdr := s.buildProxyHeader(dest); hdr != nil { if hdr := s.buildProxyHeader(dest); hdr != nil {
if _, err := dest.Write(hdr); err != nil { if _, err := dest.Write(hdr); err != nil {
log.Printf("stream %d: proxy header write: %v", s.sid, err) log.Printf("stream %s: proxy header write: %v", s.name(), err)
} }
} }
} }
@@ -285,6 +505,12 @@ func (s *Stream) run() {
} }
s.dest = dest s.dest = dest
s.connected = true s.connected = true
if s.finPending {
// The hub FIN'd while we were still dialing, so gracefulFin could not
// arm the drain deadline (there was no destination yet). Arm it now,
// otherwise writeLoop can block on an unresponsive destination forever.
_ = dest.SetWriteDeadline(time.Now().Add(finDrainTimeout))
}
s.cond.Broadcast() // wake writeLoop: queued hub bytes can flow now s.cond.Broadcast() // wake writeLoop: queued hub bytes can flow now
s.mu.Unlock() s.mu.Unlock()
@@ -293,10 +519,15 @@ func (s *Stream) run() {
for { for {
n, err := dest.Read(buf) n, err := dest.Read(buf)
if n > 0 { if n > 0 {
if !s.acquireSendWnd(n) { chunk := buf[:n]
break if s.vel != nil && !s.vel.Passthrough() {
fwd, inject := s.vel.ProcessS2C(chunk)
if len(inject) > 0 {
s.injectToDest(inject)
}
chunk = fwd
} }
if werr := s.wc.sendData(s.sid, buf[:n]); werr != nil { if !s.sendToHub(chunk) {
break break
} }
} }
@@ -307,6 +538,94 @@ func (s *Stream) run() {
s.teardown(true) s.teardown(true)
} }
// sendToHub forwards destination bytes to the hub in bounded DATA frames,
// honoring both the send window and the client-wide bandwidth cap.
// Returns false once the stream closed or the worker conn failed.
func (s *Stream) sendToHub(data []byte) bool {
for len(data) > 0 {
n := len(data)
if n > s.client.chunk {
n = s.client.chunk
}
// Credit first, bandwidth second. The reverse order would charge the
// budget for bytes still parked on an exhausted window, so the client
// would throttle itself below the configured rate. Holding credit while
// waiting for tokens is free — credit is per-tunnel, and the hub returns
// it as it drains data to the player, independent of our pacing.
if !s.acquireSendWnd(n) {
return false
}
var shaperStall stallClock
shaperStall.begin(s.stats != nil)
if !s.client.shaper.Acquire(&s.share, n, s.done) {
return false
}
// Sampled here, before the socket write: emit's write to the hub can
// block when the hub is not reading, and charging that time to the
// bandwidth cap would blame maxBandwidth for a hub that is not draining.
waited := shaperStall.elapsed()
if !s.emit(data[:n]) {
return false
}
if s.stats != nil {
s.mu.Lock()
// Separated from the window stall on purpose: this one says the
// configured cap is the binding constraint, and raising maxBandwidth
// is the fix. The window stall says the opposite.
s.stats.shaperStall += waited
s.stats.bytesUp += int64(n)
s.mu.Unlock()
}
data = data[n:]
}
return true
}
// emit records a chunk for possible retransmission and writes it to the current
// worker conn. Returns false once the stream is finished with.
//
// The record is taken first and unconditionally. A write that fails on a dying
// connection has already spent window and may have put part of the frame on the
// wire, so the only trustworthy account of what the peer still owes us is the
// one taken before the attempt. That also makes a failure survivable: while the
// stream can still be resumed the bytes are already safe, and the reattach
// replays them from wherever the hub says it got to.
func (s *Stream) emit(chunk []byte) bool {
s.sendMu.Lock()
if s.resumable {
// Reclaim what the hub has credited before growing the buffer, so the
// outstanding region stays bounded by one window.
s.un.advance(s.ackedOffset.Load())
s.un.append(chunk)
}
wc := s.conn()
var err error
if wc != nil {
err = wc.sendData(chunk)
} else {
err = errPoolClosed
}
s.sendMu.Unlock()
if err == nil {
return true
}
if !s.resumable {
return false
}
// The conn is gone but the stream is not: readLoop parks it and a reattach
// replays the buffer. Keep pumping the destination — acquireSendWnd stops us
// once a full window is outstanding, so nothing is lost and nothing grows
// without bound.
return !s.isClosed()
}
func (s *Stream) isClosed() bool {
s.mu.Lock()
defer s.mu.Unlock()
return s.closed
}
// writeLoop is the only writer to the destination. It drains the receive queue, // writeLoop is the only writer to the destination. It drains the receive queue,
// credits the hub as bytes land on the destination socket, and performs the // credits the hub as bytes land on the destination socket, and performs the
// deferred graceful close when a FIN arrived with data still queued. // deferred graceful close when a FIN arrived with data still queued.
@@ -325,39 +644,76 @@ func (s *Stream) writeLoop() {
s.teardown(false) s.teardown(false)
return return
} }
data := s.q[0] e := s.q[0]
s.q = s.q[1:] s.q = s.q[1:]
s.qBytes -= len(data) if e.fromHub {
s.qBytes -= len(e.data)
}
dest := s.dest dest := s.dest
s.mu.Unlock() s.mu.Unlock()
if _, err := dest.Write(data); err != nil { if _, err := dest.Write(e.data); err != nil {
s.teardown(true) s.teardown(true)
return return
} }
s.credit(len(data)) if e.fromHub {
s.mu.Lock()
s.deliveredOffset += int64(len(e.data))
if s.stats != nil {
s.stats.bytesDown += int64(len(e.data))
}
s.mu.Unlock()
s.credit(len(e.data))
}
} }
} }
// deliverFromHub enqueues hub bytes for the destination. Called from the worker // deliverFromHub enqueues hub bytes for the destination. Called from the worker
// readLoop; it never blocks — a peer that exceeds the advertised window is a // readLoop; it never blocks — a peer that exceeds the advertised window is a
// protocol violator and gets the stream reset. // protocol violator and gets the tunnel reset.
func (s *Stream) deliverFromHub(data []byte) { func (s *Stream) deliverFromHub(data []byte) {
if s.vel != nil {
s.vel.ObserveC2S(data) // observation only; bytes still forwarded verbatim
}
s.mu.Lock() s.mu.Lock()
if s.closed || s.finPending { if s.closed || s.finPending {
s.mu.Unlock() s.mu.Unlock()
return return
} }
if s.qBytes+len(data) > s.wc.recvWndInit { if s.qBytes+len(data) > s.client.streamWnd {
s.mu.Unlock() s.mu.Unlock()
log.Printf("stream %d: peer exceeded flow-control window; resetting", s.sid) log.Printf("stream %s: peer exceeded flow-control window; resetting", s.name())
s.wc.removeStream(s.sid) if wc := s.conn(); wc != nil {
s.wc.sendRst(s.sid) wc.detach()
wc.sendRst()
}
s.teardown(false) s.teardown(false)
return return
} }
s.q = append(s.q, data) s.q = append(s.q, qentry{data: data, fromHub: true})
s.qBytes += len(data) s.qBytes += len(data)
// Accepted, not delivered: from here the bytes are ours to write, and the
// only way we fail to is by destroying the stream — which also ends any
// prospect of resuming it. That makes this a sound reattach coordinate.
s.acceptedOffset += int64(len(data))
if s.stats != nil && s.qBytes > s.stats.qPeak {
// How close the receive queue came to the advertised window: near it
// means the destination is the slow party.
s.stats.qPeak = s.qBytes
}
s.cond.Broadcast()
s.mu.Unlock()
}
// injectToDest queues locally generated bytes (the velocity login response)
// for the destination, outside flow-control accounting.
func (s *Stream) injectToDest(data []byte) {
s.mu.Lock()
if s.closed || s.finPending {
s.mu.Unlock()
return
}
s.q = append(s.q, qentry{data: data})
s.cond.Broadcast() s.cond.Broadcast()
s.mu.Unlock() s.mu.Unlock()
} }
@@ -367,9 +723,18 @@ func (s *Stream) deliverFromHub(data []byte) {
func (s *Stream) acquireSendWnd(n int) bool { func (s *Stream) acquireSendWnd(n int) bool {
s.mu.Lock() s.mu.Lock()
defer s.mu.Unlock() defer s.mu.Unlock()
// Timed only when it actually blocks, so a stream that never runs out of
// credit never reads the clock. A large windowStall is the signal that the
// peer is not draining to its terminal socket — the bottleneck is past the
// tunnel, not in it.
var stall stallClock
for !s.closed && s.sendWnd < n { for !s.closed && s.sendWnd < n {
stall.begin(s.stats != nil)
s.cond.Wait() s.cond.Wait()
} }
if s.stats != nil {
s.stats.windowStall += stall.elapsed()
}
if s.closed { if s.closed {
return false return false
} }
@@ -378,6 +743,11 @@ func (s *Stream) acquireSendWnd(n int) bool {
} }
func (s *Stream) grantSendWnd(delta int) { func (s *Stream) grantSendWnd(delta int) {
// The running sum doubles as the acked offset: the hub grants credit exactly
// as bytes reach the player socket, so a byte that has been credited can
// never need retransmitting. Advanced without a lock so the worker readLoop
// never blocks behind a send in progress.
s.ackedOffset.Add(int64(delta))
s.mu.Lock() s.mu.Lock()
s.sendWnd += delta s.sendWnd += delta
s.cond.Broadcast() s.cond.Broadcast()
@@ -386,17 +756,23 @@ func (s *Stream) grantSendWnd(delta int) {
// credit accounts bytes drained to the destination and grants the hub more // credit accounts bytes drained to the destination and grants the hub more
// window once half of our receive window has been consumed. // window once half of our receive window has been consumed.
// While parked the grant is only withheld, never dropped: consumed keeps
// accumulating and a reattach flushes it on the new conn. Resetting it would
// destroy up to half a window of credit per outage, and after a few flaps the
// stream would throttle to a crawl.
func (s *Stream) credit(n int) { func (s *Stream) credit(n int) {
s.mu.Lock() s.mu.Lock()
s.consumed += n s.consumed += n
if s.closed || s.consumed*2 < s.wc.recvWndInit { if s.closed || s.parked || s.consumed*2 < s.client.streamWnd {
s.mu.Unlock() s.mu.Unlock()
return return
} }
delta := s.consumed delta := s.consumed
s.consumed = 0 s.consumed = 0
s.mu.Unlock() s.mu.Unlock()
s.wc.sendWndUpdate(s.sid, delta) if wc := s.conn(); wc != nil {
wc.sendWndUpdate(delta)
}
} }
func (s *Stream) buildProxyHeader(dest net.Conn) []byte { func (s *Stream) buildProxyHeader(dest net.Conn) []byte {
@@ -434,6 +810,13 @@ func (s *Stream) gracefulFin() {
// teardown closes the stream immediately; notifyHub sends a FIN when true. // teardown closes the stream immediately; notifyHub sends a FIN when true.
// Idempotent; wakes every goroutine parked on the stream. // Idempotent; wakes every goroutine parked on the stream.
func (s *Stream) teardown(notifyHub bool) { func (s *Stream) teardown(notifyHub bool) {
// A parked stream owes the hub a FIN it cannot send: the only conn it has is
// the one that just failed. Keep it alive so the reattach can deliver it and
// the player gets a clean disconnect, rather than hanging until the hub's
// grace expires.
if notifyHub && s.noteFinWhileParked() {
return
}
s.mu.Lock() s.mu.Lock()
if s.closed { if s.closed {
s.mu.Unlock() s.mu.Unlock()
@@ -441,14 +824,24 @@ func (s *Stream) teardown(notifyHub bool) {
} }
s.closed = true s.closed = true
dest := s.dest dest := s.dest
close(s.done) // guarded by the idempotence check above, so exactly once
s.cond.Broadcast() s.cond.Broadcast()
s.mu.Unlock() s.mu.Unlock()
// The stream can no longer be resumed, so the retransmit buffer is dead
// weight — up to a full window of it per stream.
s.sendMu.Lock()
s.un.reset()
s.sendMu.Unlock()
if dest != nil { if dest != nil {
_ = dest.Close() _ = dest.Close()
} }
s.wc.removeStream(s.sid) if wc := s.conn(); wc != nil {
if notifyHub { wc.detach()
s.wc.sendFin(s.sid) if notifyHub {
wc.sendFin()
}
} }
s.logSummary()
} }
+121 -65
View File
@@ -15,9 +15,9 @@ on the wire, read [PROTOCOL.md](../PROTOCOL.md).
┌──────────┐ Intent 17, magic 0x01 │ │ │ ┌──────────┐ Intent 17, magic 0x01 │ │ │
│ Client │ ◀──────── control session ──────│ • pattern reg │ │ │ Client │ ◀──────── control session ──────│ • pattern reg │ │
│ (Go) │ ────────────────────────────────│ • CID table │ │ │ (Go) │ ────────────────────────────────│ • CID table │ │
│ │ Intent 17, magic 0x02 │ • mux demux │ │ │ │ Intent 17, magic 0x02 │ • 1:1 workers │ │
│ │ ═════════ worker conns ═════════│ │ │ │ │ ═════════ worker conns ═════════│ │ │
└──────────┘ multiplexed player streams └───────────────┘ │ └──────────┘ one TCP conn per player └───────────────┘ │
│ │ │ │
▼ MC bytes (+ optional HAProxy v2) │ ▼ MC bytes (+ optional HAProxy v2) │
┌───────────────┐ │ ┌───────────────┐ │
@@ -68,15 +68,15 @@ Player Hub Client Destinatio
│ │ pause player socket, │ │ pause player socket,
│ │ buffer bytes, mint CID │ │ buffer bytes, mint CID
│ │─ ControlRequest(CID, pattern, ip:port) ─▶ │ │─ ControlRequest(CID, pattern, ip:port) ─▶
│ │ allocate worker+stream │ │ dial dedicated worker
│ │◀──────── SYN(streamId, CID) ────────────│ │ │◀──────────── SYN(CID) ──────────────────│
│ │ takePending(CID) → bind dial destination, │ │ takePending(CID) → bind dial destination,
│ │ forward buffered bytes write HAProxy v2 hdr │ │ forward buffered bytes write HAProxy v2 hdr
│ │─ DATA(streamId, handshake…) ───▶ ── handshake ──▶│ │ │─ DATA(handshake…) ─────────────▶ ── handshake ──▶│
│ resume ─────────────────────────────│ bridge stream ⇄ dest │ resume ─────────────────────────────│ bridge conn ⇄ dest
│══════════════ player bytes ══ DATA ══▶│════ DATA ═══▶ dest.write │ │══════════════ player bytes ══ DATA ══▶│════ DATA ═══▶ dest.write │
│◀═══════════ dest bytes ═══ DATA ══════│◀═══ DATA ════ dest.read │ │◀═══════════ dest bytes ═══ DATA ══════│◀═══ DATA ════ dest.read │
│ player closes ──────────────────────│─ FIN(streamId) ────────▶ close dest │ │ player closes ──────────────────────│─ FIN ──────────────────▶ close dest │
``` ```
Key points: Key points:
@@ -87,6 +87,11 @@ Key points:
player's hostname — in `ControlRequest`, so the client can look it straight up player's hostname — in `ControlRequest`, so the client can look it straight up
in its own route table. Invalid patterns are rejected at registration with a in its own route table. Invalid patterns are rejected at registration with a
non-zero `RegisterAck` status. non-zero `RegisterAck` status.
* A **per-IP limiter** runs first: a token bucket (`playerRatePerSec` /
`playerBurst`) and a concurrent-socket cap (`maxPlayersPerIp`). It applies
only to player intents — never Intent 17 — and unmatched hostnames still
consume a token, so a hostname scan is not a free flood. A refusal closes
the socket before CID minting or pause.
* The hub **pauses** the player socket the instant it matches, so no player * The hub **pauses** the player socket the instant it matches, so no player
bytes are lost while the takeover is arranged; the buffered handshake is bytes are lost while the takeover is arranged; the buffered handshake is
forwarded **verbatim**, so the real server sees exactly what the player sent forwarded **verbatim**, so the real server sees exactly what the player sent
@@ -94,41 +99,38 @@ Key points:
* **CID** is 16 random bytes minted by the hub and delivered only over the * **CID** is 16 random bytes minted by the hub and delivered only over the
encrypted control session, so only the intended client learns it. Any worker encrypted control session, so only the intended client learns it. Any worker
connection presenting the correct CID is allowed to take over — that secrecy connection presenting the correct CID is allowed to take over — that secrecy
is what binds a worker stream to the right pending player without any explicit is what binds a worker conn to the right pending player without any explicit
client identity. client identity.
* Disconnects are symmetric: player-close → hub sends `FIN` → client closes the * Disconnects are symmetric: player-close → hub sends `FIN` → client closes the
destination; destination-close → client sends `FIN` → hub closes the player. destination; destination-close → client sends `FIN` → hub closes the player.
## 3. Multiplexing (worker connections) ## 3. Worker connections (1:1)
A worker connection is one encrypted TCP link carrying many **streams**. The A worker connection is one encrypted TCP link carrying **exactly one player**.
frame is intentionally tiny (PROTOCOL.md §7): The frame is intentionally tiny (PROTOCOL.md §7):
``` ```
[plaintext VarInt length][ FrameType u8 | StreamID VarInt | Data… ] (payload encrypted) [plaintext VarInt length][ FrameType u8 | Data… ] (payload encrypted)
``` ```
Only the client opens streams (`SYN`), so stream-id allocation is a simple There is no stream id. The client dials a fresh worker for each
per-connection counter with no coordination. `ControlRequest` and binds it with `SYN(CID)` after `SessionReady`. A second
bind on the same conn is a protocol violation.
### 3.1 Pool & allocation ### 3.1 Allocation
The client keeps 1…`maxConn` worker connections and places each new stream on The client holds at most `maxTunnels` live worker connections (default 256). A
the **least-loaded** one. It opens an additional connection only when the `ControlRequest` either gets its own TCP connection or is dropped — the hub
least-loaded connection is *saturated* (more than 8 active streams) and the pool then closes the player when `pendingTimeoutMs` fires.
is below `maxConn`:
``` A dial is **never performed while holding the live-set lock** — session
pick least-loaded conn establishment is network I/O, and one unresponsive hub must not park every
if leastLoaded.streams > 8 and pool.size < maxConn: other player behind it. Each caller dials independently; callers are not
dial a new worker conn and use it serialized behind a single in-flight handshake.
else:
use leastLoaded
```
The e2e test `TestConcurrentStreamsUseMultipleConns` drives 20 simultaneous The e2e test `TestEachPlayerGetsOwnWorker` drives concurrent players and
streams with `maxConn=4` and observes them deterministically spread over 3 confirms one worker conn each, without exceeding `maxTunnels`;
connections (9 + 9 + 2), confirming the algorithm. `TestDialDoesNotWedgeOnStalledHub` covers the stalled-dial path.
## 4. Encryption ## 4. Encryption
@@ -153,68 +155,122 @@ connections (9 + 9 + 2), confirming the algorithm.
non-blocking; crypto is CPU-cheap. This trades multi-core scaling for non-blocking; crypto is CPU-cheap. This trades multi-core scaling for
simplicity and correctness. simplicity and correctness.
* **Client:** goroutine-per-concern. One goroutine reads each connection * **Client:** goroutine-per-concern. One goroutine reads each connection
(control or worker); `WriteFrame` is mutex-serialized so many stream goroutines (control or worker); `WriteFrame` is mutex-serialized. Each tunnel has two
can share a worker connection safely. Each stream has two goroutines: `run` goroutines: `run` pumps destination → hub, and `writeLoop` is the only writer
pumps destination → hub, and `writeLoop` is the only writer to the to the destination, draining a queue fed by the worker readLoop. The
destination, draining a per-stream queue fed by the worker readLoop. The readLoop itself never writes to a destination, so a stalled destination cannot
readLoop itself never writes to a destination, so a stalled destination can stall WND / FIN / heartbeat dispatch on that conn.
never block frame dispatch for other streams.
## 6. Back-pressure & flow control ## 6. Back-pressure & flow control
Two mechanisms operate at different granularities: Three mechanisms operate at different granularities:
* **Per-stream credit windows** (PROTOCOL.md §7.3; the windows are exchanged * **Per-connection credit windows** (PROTOCOL.md §7.3; the windows are exchanged
at session establishment): each stream direction has an independent byte at session establishment): each tunnel direction has an independent byte
budget equal to the receiver's advertised window (default 256 KiB). A sender budget equal to the receiver's advertised window (default 256 KiB). A sender
that exhausts a that exhausts a window pauses *only that player's source* — the hub pauses the
stream's window pauses *only that stream's source* — the hub pauses the one one player socket, the client parks the one destination-reader goroutine.
player socket, the client parks the one destination-reader goroutine. Credit Credit is granted back (`WND` frames, batched at half-window) as bytes are
is granted back (`WND` frames, batched at half-window) as bytes are actually actually written to the terminal socket. The result: a slow player or slow
written to the terminal socket. The result: a slow player or slow destination destination jams its own tunnel at a bounded buffer size and nothing else.
jams its own stream at a bounded buffer size and nothing else. This is what * **TCP back-pressure** on each worker connection: when that socket is
eliminates head-of-line blocking between streams. congested, the hub parks that one player until it drains, and the client's
* **Aggregate TCP back-pressure** on each worker connection: when the shared `WriteFrame` blocks. Because the conn is 1:1, TCP HOL cannot stall another
socket itself is congested (total bandwidth, not one stream), the hub parks player.
all sending players until it drains, and the client's `WriteFrame` blocks. * **Client egress shaping** (optional, `maxBandwidth`; `client/shaper.go`): a
This is fair — when the pipe is genuinely full, everyone should slow down. rate cap on everything the client sends to the hub, across all worker conns.
The window also bounds memory: a stream can hold at most one window of The first two mechanisms have no time dimension. A credit window bounds how many
bytes are *in flight*, and TCP back-pressure only reacts once the pipe is already
full — which on a residential uplink is too late. One player loading chunks fills
the line, the standing queue grows to seconds, and every other player's keepalive
times out. Nothing in §7.3 prevents that: each tunnel is individually
well-behaved, and collectively they still overrun the link.
The shaper closes that gap with a token bucket for the rate and start-time fair
queueing for the split. A global virtual clock advances with each grant; every
tunnel remembers where its last request finished, and a new request is stamped
`max(tunnel.vfinish, vclock)`. Lowest stamp wins. A tunnel that keeps sending
pushes its own stamp further out and yields; a tunnel returning from idle is
clamped back to the clock, so it cannot bank credit for time it did not use, but
is not penalised for the idleness either. A tunnel sending a few hundred bytes
gets a nearer stamp than one sending a full chunk, so keepalives and chat overtake
bulk terrain data for free. One tunnel alone still gets the entire rate.
Two details keep bursts cheap. The bucket banks 200 ms of transmission, so a
player joining spends it at once instead of paying for the cap in visible
chunk-loading latency. And the DATA chunk shrinks to ~20 ms of transmission when
the rate is low (floor 4 KiB), because a fixed 32 KiB chunk is a 256 ms slot at
1 Mbps — long enough dead air to drag the other players towards the very timeout
the cap exists to prevent. Above ~13 Mbps the chunk stays at the usual 32 KiB.
This is entirely client-local: nothing about it appears on the wire, and the hub
is unaware. Only DATA is shaped — delaying a `FIN`, `WND` or `PONG` would cause
the false-death detection §7.4 exists to avoid.
The window also bounds memory: a tunnel can hold at most one window of
undelivered data per direction (the client's pre-connect handshake buffer is undelivered data per direction (the client's pre-connect handshake buffer is
covered by the same bound). covered by the same bound). With stream resumption enabled (§7) the *sender*
holds a second window — the bytes it has sent but the peer has not yet credited,
kept so they can be retransmitted after an outage. That is not a new bound so
much as the existing one made symmetric: the region is exactly what flow control
already declared outstanding, which is why resumption needs no cap of its own.
Per-stream flow control is mandatory: the hub rejects a session whose Rekey Per-connection flow control is mandatory: the hub rejects a session whose Rekey
lacks the STREAM_FC flag, and the client rejects a hub that does not echo it — lacks the STREAM_FC flag, and the client rejects a hub that does not echo it —
peers that predate the mechanism cannot connect at all. peers that predate the mechanism cannot connect at all.
What remains (by design) is TCP-level head-of-line blocking: a lost packet on TCP-level head-of-line blocking is now confined to one player: a lost packet
a worker connection stalls all its streams for one retransmit. That is inherent stalls only that player's tunnel for one retransmit. The cost is one handshake
to mux-over-TCP; the connection pool is the mitigation, and a datagram and one NAT mapping per player instead of per pool slot.
transport (QUIC) would be the escape hatch if it ever matters.
## 7. Failure & recovery ## 7. Failure & recovery
* **Control session drop:** the client reconnects with capped exponential * **Control session drop:** the client retries immediately, then backs off to a
backoff and re-registers all patterns. Existing worker connections and their 10s cap, and re-registers all patterns. Existing worker connections and their
live streams are unaffected. players are unaffected — they ride worker conns, which a control-session
* **Worker connection drop:** every stream on it is torn down (destinations close never touches. The hub meanwhile keeps that session's routes as
closed); the hub closes the corresponding player sockets; the client removes *orphaned* for `registrationGraceMs` (PROTOCOL.md §5.2) and **holds** players
the connection from the pool and will dial a fresh one on the next allocation. arriving on them instead of refusing them, replaying the control request once
a client re-registers the pattern. Without that, the reconnect window is one
in which every new player is told there is no such server.
* **Worker connection drop:** the connection leaves the live set either way. What
happens to its player depends on whether STREAM_RESUME was negotiated:
* *without it* — the tunnel is torn down (destination closed) and the hub
closes the player socket, as it always did;
* *with it* — the player is **hung** instead (PROTOCOL.md §7.5). The
destination socket stays open, the hub pauses the player socket and holds
it for its grace period, and the client reattaches over a freshly dialed
connection, replaying byte-exactly from the offset the peer reports. The
player sees a stall rather than a disconnect.
* **Pending timeout:** if no worker takes over a matched player within * **Pending timeout:** if no worker takes over a matched player within
`pendingTimeoutMs`, the hub drops the pending entry and closes the player. `pendingTimeoutMs`, the hub drops the pending entry and closes the player.
* **Bad PSK / bad timestamp / bad magic:** the hub closes the TCP connection; * **Bad PSK / bad timestamp / bad magic:** the hub closes the TCP connection;
the client's session establishment fails fast. the client's session establishment fails fast.
Resumption is worth the machinery because a worker connection is only the
*middle* leg of the player it carries. When it dies both terminal sockets are
usually still healthy, so the old behaviour discarded working connections
because a replaceable transport failed. It also has to be byte-exact rather than
best-effort: bytes handed to a dying socket are lost with no notification and the
frame cipher cannot be resynchronized, so an approximate reattach would splice
the tunneled protocol mid-packet, which is worse than a clean close.
Notably, resumption does not depend on the control session. A blip usually kills
both, and the reattach path needs only a worker connection, so recovery does not
wait on the control reconnect backoff.
## 8. Known limitations ## 8. Known limitations
1. No AEAD — payload integrity/authenticity is not cryptographically guaranteed. 1. No AEAD — payload integrity/authenticity is not cryptographically guaranteed.
2. TCP-level head-of-line blocking within a worker connection (lost packets; 2. One handshake and one NAT mapping per player (the cost of dropping mux).
see §6) — per-stream flow control removes the application-level variant only.
3. Single-event-loop hub (see §5) bounds throughput to one core. 3. Single-event-loop hub (see §5) bounds throughput to one core.
4. `Intent 18` is reserved but only stubbed (the hub logs and closes). 4. `Intent 18` is reserved for a plaintext status probe.
5. Pattern ownership is last-writer-wins; two clients registering the identical 5. Pattern ownership is last-writer-wins; two clients registering the identical
pattern string will silently reassign it. Overlapping-but-distinct regexes are pattern string will silently reassign it. Overlapping-but-distinct regexes are
both kept, and when several match one hostname the winner is unspecified. both kept, and when several match one hostname the winner is unspecified.
6. The player IP limiter matches exact addresses. A `/64` of IPv6 clients looks
like many independent IPs; aggregation is a later, isolated change.
These are deliberate scope choices for a connectivity-focused P2P tool, not These are deliberate scope choices for a connectivity-focused P2P tool, not
oversights; each is a small, well-isolated change away from being hardened. oversights; each is a small, well-isolated change away from being hardened.
+147
View File
@@ -0,0 +1,147 @@
package e2e
import (
"context"
"fmt"
"io"
"net"
"testing"
"time"
"github.com/iceBear67/redapricot/client"
)
// startClientWithBandwidth is startClient with an egress cap, for the shaping
// tests. Only the client→hub direction is shaped, which in this harness is the
// leg carrying the mock destination's echo back to the player.
func startClientWithBandwidth(t *testing.T, hubAddr, psk string, maxTunnels int, bandwidth string, mappings []client.Mapping) *client.Client {
t.Helper()
cfg := &client.Config{
Server: hubAddr,
PSK: psk,
MaxTunnels: maxTunnels,
PingIntervalMs: 20000,
MaxBandwidth: bandwidth,
Mappings: mappings,
}
c := client.New(cfg)
ctx, cancel := context.WithCancel(context.Background())
t.Cleanup(func() {
cancel()
c.Close()
})
if err := c.Start(ctx); err != nil {
t.Fatalf("client start: %v", err)
}
time.Sleep(200 * time.Millisecond)
return c
}
// drain reads a connection until it fails, so a greedy player keeps pulling
// bytes instead of stalling on its own receive window.
func drain(conn net.Conn) {
go func() { _, _ = io.Copy(io.Discard, conn) }()
}
// TestBandwidthCapIsEnforced pushes a payload that cannot fit in the burst and
// asserts the transfer takes at least as long as the configured rate implies.
// This is the first timing assertion in the suite, so the bounds are wide: the
// unshaped path completes this in well under a second, making the lower bound
// an unambiguous signal rather than a tight measurement.
func TestBandwidthCapIsEnforced(t *testing.T) {
const psk = "e2e-bwcap"
port := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
startHub(t, port, psk)
dest := newMockDest(t, modeEcho)
startClientWithBandwidth(t, hubAddr, psk, 1, "1MB/s", []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr},
})
pc := dialPlayer(t, hubAddr, "mc.local")
defer pc.Close()
payload := make([]byte, 2*1024*1024)
for i := range payload {
payload[i] = byte(i*31 + 7)
}
start := time.Now()
writeErr := make(chan error, 1)
go func() {
_, err := pc.Write(payload)
writeErr <- err
}()
got := make([]byte, len(payload))
_ = pc.SetReadDeadline(time.Now().Add(60 * time.Second))
if _, err := io.ReadFull(pc, got); err != nil {
t.Fatalf("read echo: %v", err)
}
elapsed := time.Since(start)
if err := <-writeErr; err != nil {
t.Fatalf("write payload: %v", err)
}
// 2 MiB at 1 MiB/s, minus the 200 KiB the bucket has banked, is ~1.8 s.
const floor = 1200 * time.Millisecond
if elapsed < floor {
t.Errorf("2 MiB echoed back in %v under a 1MB/s cap; expected at least %v, so the cap is not taking effect", elapsed, floor)
}
if elapsed > 30*time.Second {
t.Errorf("transfer took %v, far beyond the ~1.8s the rate implies", elapsed)
}
}
// TestCappedBandwidthDoesNotStarveLightStreams is the point of the fair queue:
// with the link deliberately capped and three players saturating it, a fourth
// player exchanging small messages must keep round-tripping promptly. A plain
// FIFO token bucket would leave it queued behind the heavy players' backlog,
// which for a real Minecraft client means a keepalive timeout — the exact
// failure this feature exists to prevent.
func TestCappedBandwidthDoesNotStarveLightStreams(t *testing.T) {
const psk = "e2e-bwfair"
port := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
startHub(t, port, psk)
dest := newMockDest(t, modeEcho)
// Each player has its own worker conn; the shaper is still the only thing
// splitting the shared uplink budget.
startClientWithBandwidth(t, hubAddr, psk, 4, "1MB/s", []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr},
})
// Three greedy players: each writes megabytes and keeps draining the echo,
// so they compete for the cap for the whole test rather than parking on
// their own flow-control windows.
for i := 0; i < 3; i++ {
heavy := dialPlayer(t, hubAddr, "mc.local")
defer heavy.Close()
drain(heavy)
go func() { _, _ = heavy.Write(make([]byte, 8*1024*1024)) }()
}
time.Sleep(500 * time.Millisecond) // let them saturate the shaper
light := dialPlayer(t, hubAddr, "mc.local")
defer light.Close()
payload := make([]byte, 4*1024)
for i := range payload {
payload[i] = byte(i*13 + 5)
}
var worst time.Duration
for i := 0; i < 10; i++ {
start := time.Now()
playerEcho(t, light, payload)
if d := time.Since(start); d > worst {
worst = d
}
}
// Fair sharing puts a 4 KiB round at roughly 4 KiB / (1 MiB/s ÷ 4) ≈ 16 ms of
// link time. The bound is two orders of magnitude looser so only genuine
// starvation trips it.
const limit = 3 * time.Second
if worst > limit {
t.Errorf("slowest small round-trip took %v (limit %v); heavy streams are crowding out the light one", worst, limit)
}
}
+144
View File
@@ -0,0 +1,144 @@
package e2e
import (
"errors"
"fmt"
"io"
"net"
"testing"
"time"
"github.com/iceBear67/redapricot/client"
"github.com/iceBear67/redapricot/client/wire"
)
// A control session dying takes the client's routes with it. Players already
// tunneled are unaffected — they ride worker conns — but anyone *arriving*
// during the reconnect used to be told there is no such server, even though the
// tunnel was a second from being back.
//
// The hub now keeps those routes as orphaned for its registration grace and
// hangs arriving players on them, replaying the control request it never sent
// once a client re-registers the pattern.
// outageRelay starts a hub, an echoing destination, and a client reaching the
// hub only through a relay, so the tunnel can be cut without touching the
// players — who connect to the hub directly, as they would from the internet.
func outageRelay(t *testing.T, psk string, hubCfg map[string]any) (hubAddr string, relay *blackholeRelay) {
t.Helper()
hubPort := freePort(t)
hubAddr = fmt.Sprintf("127.0.0.1:%d", hubPort)
startHubCfg(t, hubPort, psk, hubCfg)
dest := newMockDest(t, modeEcho)
relay = newBlackholeRelay(t, hubAddr)
startClientWithPing(t, relay.addr, psk, 1, 400, []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr},
})
return hubAddr, relay
}
// TestControlOutageHangsArrivingPlayer is the point of the feature: a player
// that shows up while the client is reconnecting gets held and then served,
// rather than refused.
func TestControlOutageHangsArrivingPlayer(t *testing.T) {
const psk = "e2e-ctl-hang"
hubAddr, relay := outageRelay(t, psk, nil)
// Cut the tunnel and keep it cut, so the client cannot re-register.
relay.stop()
relay.dropAll()
// Let the hub see the close and orphan the route before the player arrives.
time.Sleep(500 * time.Millisecond)
pc := resumePlayer(t, hubAddr, "mc.local")
if _, err := pc.Write([]byte("held")); err != nil {
t.Fatalf("player write during outage: %v", err)
}
// Nothing can come back yet — but the socket must still be open. Before this
// change the hub had already closed it.
_ = pc.SetReadDeadline(time.Now().Add(700 * time.Millisecond))
if _, err := pc.Read(make([]byte, 1)); err == nil {
t.Fatal("player was served while the route was orphaned")
} else if !isTimeout(err) {
t.Fatalf("player was dropped during the control outage instead of being held: %v", err)
}
relay.restore()
// The client reconnects, re-registers, and the hub replays the request it
// held — so the bytes written during the outage arrive at the destination and
// echo back on the same connection.
echo := make([]byte, 4)
_ = pc.SetReadDeadline(time.Now().Add(30 * time.Second))
if _, err := io.ReadFull(pc, echo); err != nil {
t.Fatalf("held player never served after the route came back: %v", err)
}
if string(echo) != "held" {
t.Fatalf("echo = %q, want %q", echo, "held")
}
}
// TestControlOutageGraceDisabledClosesPlayer pins the off switch: with the grace
// at zero the hub must drop the route the instant its session closes, exactly as
// it did before, rather than hanging players for a client that may never return.
func TestControlOutageGraceDisabledClosesPlayer(t *testing.T) {
const psk = "e2e-ctl-nohang"
hubAddr, relay := outageRelay(t, psk, map[string]any{"registrationGraceMs": 0})
relay.stop()
relay.dropAll()
time.Sleep(500 * time.Millisecond)
// No route at all now, so the hub closes the connection during the handshake.
pc, err := net.DialTimeout("tcp", hubAddr, 5*time.Second)
if err != nil {
t.Fatalf("player dial: %v", err)
}
defer pc.Close()
if _, err := pc.Write(playerHandshake("mc.local")); err != nil {
t.Fatalf("player handshake: %v", err)
}
_ = pc.SetReadDeadline(time.Now().Add(10 * time.Second))
if _, err := pc.Read(make([]byte, 1)); err == nil || isTimeout(err) {
t.Fatalf("player was held with the registration grace disabled (err=%v)", err)
}
}
// TestControlOutageHangExpiresClosesPlayer covers the other end: a route that is
// never reclaimed must not hold its players forever.
func TestControlOutageHangExpiresClosesPlayer(t *testing.T) {
const psk = "e2e-ctl-expire"
hubAddr, relay := outageRelay(t, psk, map[string]any{"registrationGraceMs": 2000})
relay.stop()
relay.dropAll()
time.Sleep(500 * time.Millisecond)
pc := resumePlayer(t, hubAddr, "mc.local")
start := time.Now()
_ = pc.SetReadDeadline(time.Now().Add(20 * time.Second))
if _, err := pc.Read(make([]byte, 1)); err == nil || isTimeout(err) {
t.Fatalf("held player was never released after the grace expired (err=%v)", err)
}
if elapsed := time.Since(start); elapsed < 500*time.Millisecond {
t.Fatalf("player closed after %s, so it was refused rather than held", elapsed)
} else {
t.Logf("held player released after %s", elapsed.Round(100*time.Millisecond))
}
}
// isTimeout distinguishes "still held, nothing to read yet" from "the hub closed
// us" — which is the whole distinction these tests turn on.
func isTimeout(err error) bool {
var ne net.Error
if errors.As(err, &ne) {
return ne.Timeout()
}
return false
}
func playerHandshake(host string) []byte {
return wire.BuildHandshake(767, host, 25565, 2)
}
+31 -20
View File
@@ -13,15 +13,28 @@ import (
// startClient builds and starts an in-process client against the hub, with the // startClient builds and starts an in-process client against the hub, with the
// given mappings, returning the running client. // given mappings, returning the running client.
func startClient(t *testing.T, hubAddr, psk string, maxConn int, mappings []client.Mapping) *client.Client { func startClient(t *testing.T, hubAddr, psk string, maxTunnels int, mappings []client.Mapping) *client.Client {
t.Helper() t.Helper()
cfg := &client.Config{ return startClientWithPing(t, hubAddr, psk, maxTunnels, 20000, mappings)
}
// startClientWithPing is startClient with an explicit heartbeat interval, for
// tests that need liveness detection to trigger quickly.
func startClientWithPing(t *testing.T, hubAddr, psk string, maxTunnels, pingMs int, mappings []client.Mapping) *client.Client {
t.Helper()
return startClientCfg(t, &client.Config{
Server: hubAddr, Server: hubAddr,
PSK: psk, PSK: psk,
MaxConn: maxConn, MaxTunnels: maxTunnels,
PingIntervalMs: 20000, PingIntervalMs: pingMs,
Mappings: mappings, Mappings: mappings,
} })
}
// startClientCfg runs an in-process client from a fully-specified config, for
// tests that need a knob the shorthand helpers do not expose.
func startClientCfg(t *testing.T, cfg *client.Config) *client.Client {
t.Helper()
c := client.New(cfg) c := client.New(cfg)
ctx, cancel := context.WithCancel(context.Background()) ctx, cancel := context.WithCancel(context.Background())
t.Cleanup(func() { t.Cleanup(func() {
@@ -94,7 +107,7 @@ func TestRegexPatternMatch(t *testing.T) {
} }
} }
// TestLargeTransfer pushes a multi-megabyte payload both ways to exercise mux // TestLargeTransfer pushes a multi-megabyte payload both ways to exercise
// framing and back-pressure. // framing and back-pressure.
func TestLargeTransfer(t *testing.T) { func TestLargeTransfer(t *testing.T) {
const psk = "e2e-large" const psk = "e2e-large"
@@ -134,17 +147,17 @@ func TestLargeTransfer(t *testing.T) {
} }
} }
// TestConcurrentStreamsUseMultipleConns confirms the least-loaded allocator // TestEachPlayerGetsOwnWorker confirms the 1:1 rule: N concurrent players
// opens additional worker connections once streams saturate (>8). // produce N worker connections, and the maxTunnels cap is honoured.
func TestConcurrentStreamsUseMultipleConns(t *testing.T) { func TestEachPlayerGetsOwnWorker(t *testing.T) {
const psk = "e2e-concurrent" const psk = "e2e-concurrent"
const n = 20 const n = 8
const maxConn = 4 const maxTunnels = 16
port := freePort(t) port := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", port) hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
startHub(t, port, psk) startHub(t, port, psk)
dest := newMockDest(t, modeEcho) dest := newMockDest(t, modeEcho)
c := startClient(t, hubAddr, psk, maxConn, []client.Mapping{ c := startClient(t, hubAddr, psk, maxTunnels, []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr}, {Pattern: "mc.local", Destination: dest.addr},
}) })
@@ -154,8 +167,6 @@ func TestConcurrentStreamsUseMultipleConns(t *testing.T) {
_ = pc.Close() _ = pc.Close()
} }
}() }()
// Establish streams sequentially so allocation is deterministic; keep them
// all open to hold streams active.
for i := 0; i < n; i++ { for i := 0; i < n; i++ {
pc := dialPlayer(t, hubAddr, "mc.local") pc := dialPlayer(t, hubAddr, "mc.local")
playerEcho(t, pc, []byte(fmt.Sprintf("hello-%d", i))) playerEcho(t, pc, []byte(fmt.Sprintf("hello-%d", i)))
@@ -163,13 +174,13 @@ func TestConcurrentStreamsUseMultipleConns(t *testing.T) {
} }
got := c.WorkerConnCount() got := c.WorkerConnCount()
if got < 2 { if got != n {
t.Fatalf("expected >=2 worker conns for %d concurrent streams, got %d", n, got) t.Fatalf("expected %d worker conns for %d players, got %d", n, n, got)
} }
if got > maxConn { if got > maxTunnels {
t.Fatalf("worker conns %d exceed maxConn %d", got, maxConn) t.Fatalf("worker conns %d exceed maxTunnels %d", got, maxTunnels)
} }
t.Logf("%d concurrent streams spread over %d worker conn(s)", n, got) t.Logf("%d players on %d worker conn(s)", n, got)
} }
// TestProxyProtocol checks that the client prepends a correct HAProxy v2 header // TestProxyProtocol checks that the client prepends a correct HAProxy v2 header
@@ -264,7 +275,7 @@ func TestBadPSK(t *testing.T) {
cfg := &client.Config{ cfg := &client.Config{
Server: hubAddr, Server: hubAddr,
PSK: "totally-wrong", PSK: "totally-wrong",
MaxConn: 2, MaxTunnels: 2,
PingIntervalMs: 20000, PingIntervalMs: 20000,
Mappings: []client.Mapping{{Pattern: "mc.local", Destination: dest.addr}}, Mappings: []client.Mapping{{Pattern: "mc.local", Destination: dest.addr}},
} }
+22 -2
View File
@@ -4,6 +4,7 @@ import (
"bufio" "bufio"
"bytes" "bytes"
"encoding/binary" "encoding/binary"
"encoding/json"
"fmt" "fmt"
"io" "io"
"net" "net"
@@ -61,14 +62,33 @@ func freePort(t *testing.T) int {
// startHub launches the Java hub on the given port and blocks until it accepts // startHub launches the Java hub on the given port and blocks until it accepts
// connections. The process is killed on test cleanup. // connections. The process is killed on test cleanup.
func startHub(t *testing.T, port int, psk string) { func startHub(t *testing.T, port int, psk string) {
t.Helper()
startHubCfg(t, port, psk, nil)
}
// startHubCfg is startHub with extra config keys merged over the defaults, for
// tests that need to tune a hub-side knob.
func startHubCfg(t *testing.T, port int, psk string, extra map[string]any) {
t.Helper() t.Helper()
install := filepath.Join(repoRoot(), "server", "build", "install", "redapricot-server") install := filepath.Join(repoRoot(), "server", "build", "install", "redapricot-server")
if _, err := os.Stat(install); err != nil { if _, err := os.Stat(install); err != nil {
t.Fatalf("hub not built at %s (run scripts/build.sh first): %v", install, err) t.Fatalf("hub not built at %s (run scripts/build.sh first): %v", install, err)
} }
cfg := fmt.Sprintf(`{"listen":"127.0.0.1:%d","psk":%q,"timestampWindowMs":30000,"pendingTimeoutMs":5000}`, port, psk) settings := map[string]any{
"listen": fmt.Sprintf("127.0.0.1:%d", port),
"psk": psk,
"timestampWindowMs": 30000,
"pendingTimeoutMs": 5000,
}
for k, v := range extra {
settings[k] = v
}
cfg, err := json.Marshal(settings)
if err != nil {
t.Fatal(err)
}
cfgPath := filepath.Join(t.TempDir(), "hub.json") cfgPath := filepath.Join(t.TempDir(), "hub.json")
if err := os.WriteFile(cfgPath, []byte(cfg), 0o644); err != nil { if err := os.WriteFile(cfgPath, cfg, 0o644); err != nil {
t.Fatal(err) t.Fatal(err)
} }
+174
View File
@@ -0,0 +1,174 @@
package e2e
import (
"fmt"
"net"
"testing"
"time"
"github.com/iceBear67/redapricot/client"
)
// expectClosedSoon fails unless conn is closed (or reset) within d. A player
// that entered pending stays open until pendingTimeoutMs, so a fast EOF is
// how we tell the limiter dropped the socket before match-side bookkeeping.
func expectClosedSoon(t *testing.T, conn net.Conn, d time.Duration) {
t.Helper()
_ = conn.SetReadDeadline(time.Now().Add(d))
n, err := conn.Read(make([]byte, 16))
if err == nil {
t.Fatalf("expected the hub to close the player, read %d bytes", n)
}
}
// TestPlayerBurstDropsExtraHandshakes: more arrivals than playerBurst from the
// same IP are closed after the handshake and never become pending.
func TestPlayerBurstDropsExtraHandshakes(t *testing.T) {
const psk = "e2e-rate-burst"
port := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
startHubCfg(t, port, psk, map[string]any{
"playerRatePerSec": 1,
"playerBurst": 2,
"maxPlayersPerIp": 64,
})
dest := newMockDest(t, modeEcho)
startClient(t, hubAddr, psk, 8, []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr},
})
kept := make([]net.Conn, 0, 2)
defer func() {
for _, c := range kept {
_ = c.Close()
}
}()
for i := 0; i < 2; i++ {
pc := dialPlayer(t, hubAddr, "mc.local")
playerEcho(t, pc, []byte(fmt.Sprintf("ok-%d", i)))
kept = append(kept, pc)
}
// Tokens spent, first two still held: extras must die well inside pendingTimeout.
for i := 0; i < 2; i++ {
extra := dialPlayer(t, hubAddr, "mc.local")
expectClosedSoon(t, extra, 1500*time.Millisecond)
_ = extra.Close()
}
// The admitted players are unaffected.
playerEcho(t, kept[0], []byte("still-here"))
}
// TestMaxPlayersPerIpCapsConcurrentSockets: a second player from the same IP
// is refused while the first is still open, and admitted again after it closes.
func TestMaxPlayersPerIpCapsConcurrentSockets(t *testing.T) {
const psk = "e2e-rate-conc"
port := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
startHubCfg(t, port, psk, map[string]any{
"playerRatePerSec": 0,
"playerBurst": 16,
"maxPlayersPerIp": 1,
})
dest := newMockDest(t, modeEcho)
startClient(t, hubAddr, psk, 4, []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr},
})
first := dialPlayer(t, hubAddr, "mc.local")
defer first.Close()
playerEcho(t, first, []byte("first"))
second := dialPlayer(t, hubAddr, "mc.local")
expectClosedSoon(t, second, 1500*time.Millisecond)
_ = second.Close()
playerEcho(t, first, []byte("still-first"))
_ = first.Close()
// The closeHandler runs on the hub event loop; give it a beat to release.
time.Sleep(200 * time.Millisecond)
third := dialPlayer(t, hubAddr, "mc.local")
defer third.Close()
playerEcho(t, third, []byte("after-release"))
}
// TestUnmatchedHostConsumesRateBudget: a hostname scan is not a free flood.
// Two unmatched handshakes spend the burst, so a later matching player is
// also dropped.
func TestUnmatchedHostConsumesRateBudget(t *testing.T) {
const psk = "e2e-rate-scan"
port := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
startHubCfg(t, port, psk, map[string]any{
"playerRatePerSec": 1,
"playerBurst": 2,
"maxPlayersPerIp": 64,
})
dest := newMockDest(t, modeEcho)
startClient(t, hubAddr, psk, 4, []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr},
})
for i := 0; i < 2; i++ {
miss := dialPlayer(t, hubAddr, "no.such.host")
expectClosedSoon(t, miss, 1500*time.Millisecond)
_ = miss.Close()
}
matched := dialPlayer(t, hubAddr, "mc.local")
expectClosedSoon(t, matched, 1500*time.Millisecond)
_ = matched.Close()
}
// TestIntent17IgnoresPlayerLimiter: the control session and worker conns are
// Intent 17, so a limiter tight enough to refuse a second player must not
// prevent the client from connecting or taking over the first player.
func TestIntent17IgnoresPlayerLimiter(t *testing.T) {
const psk = "e2e-rate-intent17"
port := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
startHubCfg(t, port, psk, map[string]any{
"playerRatePerSec": 1,
"playerBurst": 1,
"maxPlayersPerIp": 1,
})
dest := newMockDest(t, modeEcho)
startClient(t, hubAddr, psk, 4, []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr},
})
pc := dialPlayer(t, hubAddr, "mc.local")
defer pc.Close()
playerEcho(t, pc, []byte("intent17-ok"))
}
// TestPlayerLimiterOffSwitch: both knobs at 0 restore phase-1 behaviour —
// many players from 127.0.0.1 all get through.
func TestPlayerLimiterOffSwitch(t *testing.T) {
const psk = "e2e-rate-off"
const n = 6
port := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
startHubCfg(t, port, psk, map[string]any{
"playerRatePerSec": 0,
"maxPlayersPerIp": 0,
})
dest := newMockDest(t, modeEcho)
startClient(t, hubAddr, psk, n, []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr},
})
conns := make([]net.Conn, 0, n)
defer func() {
for _, c := range conns {
_ = c.Close()
}
}()
for i := 0; i < n; i++ {
pc := dialPlayer(t, hubAddr, "mc.local")
playerEcho(t, pc, []byte(fmt.Sprintf("off-%d", i)))
conns = append(conns, pc)
}
}
+187
View File
@@ -0,0 +1,187 @@
package e2e
import (
"fmt"
"io"
"net"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/iceBear67/redapricot/client"
"github.com/iceBear67/redapricot/client/wire"
)
// blackholeRelay forwards TCP to the hub until it is switched off, after which
// already-established pairs silently stop carrying bytes while their sockets
// stay open. That is what a stateful middlebox looks like when it forgets a
// flow: conntrack expiry, a firewall reload, or a cloud LB idle timeout. No FIN
// and no RST ever reach either end, so nothing below the application layer can
// notice.
// Only flows that already existed when the switch is thrown go dark; new
// connections are carried normally, exactly as when a middlebox forgets
// established state but keeps forwarding fresh traffic.
type blackholeRelay struct {
addr string
backend string
gen atomic.Uint64
down atomic.Bool
mu sync.Mutex
held []net.Conn
}
func newBlackholeRelay(t *testing.T, backend string) *blackholeRelay {
t.Helper()
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("relay listen: %v", err)
}
r := &blackholeRelay{addr: ln.Addr().String(), backend: backend}
t.Cleanup(func() {
_ = ln.Close()
r.mu.Lock()
for _, c := range r.held {
_ = c.Close()
}
r.mu.Unlock()
})
go func() {
for {
cli, err := ln.Accept()
if err != nil {
return
}
go r.handle(cli)
}
}()
return r
}
func (r *blackholeRelay) handle(cli net.Conn) {
born := r.gen.Load()
if r.down.Load() {
_ = cli.Close()
return
}
up, err := net.Dial("tcp", r.backend)
if err != nil {
_ = cli.Close()
return
}
r.mu.Lock()
r.held = append(r.held, cli, up)
r.mu.Unlock()
pipe := func(dst, src net.Conn) {
buf := make([]byte, 32*1024)
for {
n, err := src.Read(buf)
if n > 0 && r.gen.Load() == born {
if _, werr := dst.Write(buf[:n]); werr != nil {
return
}
}
if err != nil {
return
}
}
}
go pipe(up, cli)
go pipe(cli, up)
}
// blackhole strands every currently-established pair. Later connections are
// unaffected.
func (r *blackholeRelay) blackhole() { r.gen.Add(1) }
// dropAll hard-resets every currently-established pair: a real FIN/RST reaches
// both ends immediately, as when a middlebox is restarted or a route flaps,
// rather than the silent stranding blackhole models. Later connections are
// carried normally.
//
// stop takes the relay out of service: new connections are refused rather than
// carried, so the tunnel stays down until restore is called. Models an outage
// the client cannot immediately reconnect through.
func (r *blackholeRelay) stop() { r.down.Store(true) }
// restore puts the relay back in service. Flows stranded before it are still
// dead — only fresh connections are carried, which is what a client gets after
// a middlebox or upstream link comes back.
func (r *blackholeRelay) restore() { r.down.Store(false) }
// This is the fast path into stream resumption: the client learns the conn is
// gone at once instead of waiting out a heartbeat timeout.
func (r *blackholeRelay) dropAll() {
r.gen.Add(1)
r.mu.Lock()
held := r.held
r.held = nil
r.mu.Unlock()
for _, c := range held {
_ = c.Close()
}
}
// TestBlackholedPathRecovers is the end-to-end regression guard for the
// stability bug this hardening was written for: with the tunnel's path silently
// dropped, the client used to notice nothing at all. Its read loops parked
// forever, the dead worker conn stayed in the pool, the hub kept routing players
// to a control session nobody was reading, and no player could connect again
// until the client process was restarted.
//
// The heartbeats must now detect the silence, drop both sessions, and let the
// existing reconnect path restore service on its own.
func TestBlackholedPathRecovers(t *testing.T) {
const psk = "e2e-blackhole"
hubPort := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", hubPort)
startHub(t, hubPort, psk)
dest := newMockDest(t, modeEcho)
// The client reaches the hub only through the relay; players connect to the
// hub directly, as they would from the internet.
relay := newBlackholeRelay(t, hubAddr)
const pingMs = 400 // heartbeat timeout is 3x this
c := startClientWithPing(t, relay.addr, psk, 4, pingMs, []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr},
})
play := func(what string) error {
pc, err := net.DialTimeout("tcp", hubAddr, 5*time.Second)
if err != nil {
return err
}
defer pc.Close()
if _, err := pc.Write(wire.BuildHandshake(767, "mc.local", 25565, 2)); err != nil {
return err
}
msg := []byte(what)
if _, err := pc.Write(msg); err != nil {
return err
}
got := make([]byte, len(msg))
_ = pc.SetReadDeadline(time.Now().Add(15 * time.Second))
_, err = io.ReadFull(pc, got)
return err
}
if err := play("before"); err != nil {
t.Fatalf("baseline round-trip failed: %v", err)
}
relay.blackhole()
t.Log("path blackholed: no FIN, no RST, sockets held open")
// Wait for the heartbeats to fire, the sessions to be dropped, and the
// control session to reconnect through a fresh relay pair.
deadline := time.Now().Add(45 * time.Second)
var lastErr error
for time.Now().Before(deadline) {
if lastErr = play("after"); lastErr == nil {
t.Logf("recovered on its own; worker conns now %d", c.WorkerConnCount())
return
}
time.Sleep(500 * time.Millisecond)
}
t.Fatalf("client never recovered from the blackholed path (last error: %v)", lastErr)
}
+269
View File
@@ -0,0 +1,269 @@
package e2e
import (
"bytes"
"fmt"
"io"
"math/rand"
"net"
"sync"
"testing"
"time"
"github.com/iceBear67/redapricot/client"
"github.com/iceBear67/redapricot/client/wire"
)
// resumePlayer opens a player connection and returns it, having sent only the
// handshake. The caller keeps it open across the outage — which is the whole
// point: before stream resumption this socket was closed by the hub the instant
// its worker conn died.
func resumePlayer(t *testing.T, hubAddr, host string) net.Conn {
t.Helper()
pc, err := net.DialTimeout("tcp", hubAddr, 5*time.Second)
if err != nil {
t.Fatalf("player dial: %v", err)
}
t.Cleanup(func() { _ = pc.Close() })
if _, err := pc.Write(wire.BuildHandshake(767, host, 25565, 2)); err != nil {
t.Fatalf("player handshake: %v", err)
}
return pc
}
// echoExchange streams payload through an echoing destination and verifies that
// what comes back is byte-for-byte identical, calling disrupt once `at` bytes
// have made the round trip.
//
// Comparing the whole stream rather than sampling is deliberate: a resumption
// bug does not corrupt bytes, it duplicates or skips a range, and only an exact
// comparison of the full sequence catches an off-by-one in the offsets.
func echoExchange(t *testing.T, pc net.Conn, payload []byte, at int, disrupt func()) {
t.Helper()
const chunk = 16 << 10
var wg sync.WaitGroup
wg.Add(1)
writeErr := make(chan error, 1)
go func() {
defer wg.Done()
for off := 0; off < len(payload); off += chunk {
end := min(off+chunk, len(payload))
_ = pc.SetWriteDeadline(time.Now().Add(60 * time.Second))
if _, err := pc.Write(payload[off:end]); err != nil {
writeErr <- fmt.Errorf("write at %d: %w", off, err)
return
}
}
writeErr <- nil
}()
got := make([]byte, len(payload))
read, fired := 0, false
for read < len(got) {
_ = pc.SetReadDeadline(time.Now().Add(60 * time.Second))
n, err := pc.Read(got[read:])
read += n
if !fired && read >= at {
fired = true
disrupt()
}
if err != nil {
t.Fatalf("player read failed after %d/%d bytes: %v", read, len(got), err)
}
}
wg.Wait()
if err := <-writeErr; err != nil {
t.Fatalf("player write: %v", err)
}
if !bytes.Equal(got, payload) {
// Report the first divergence: its offset says whether the stream gained
// or lost bytes, which is the difference between a retransmit that
// replayed too much and one that replayed too little.
for i := range got {
if got[i] != payload[i] {
t.Fatalf("echo diverges at byte %d of %d (sent %#x, got %#x)",
i, len(payload), payload[i], got[i])
}
}
}
}
func randomPayload(seed int64, n int) []byte {
p := make([]byte, n)
rand.New(rand.NewSource(seed)).Read(p)
return p
}
// TestResumePreservesByteStream is the correctness bar for stream resumption:
// a worker conn is hard-reset mid-transfer and the *same* player connection must
// keep working, with a byte stream that neither gains nor loses a single byte.
//
// Byte-exactness is the whole difficulty. Frames handed to a dying socket are
// lost with no notification and the cipher cannot be resynchronized, so each
// side has to replay from the offset the other reports it accepted. Getting that
// offset wrong by any amount splices the stream mid-Minecraft-packet, which a
// round-trip test that only checked "traffic flows again" would happily pass.
func TestResumePreservesByteStream(t *testing.T) {
const psk = "e2e-resume"
hubPort := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", hubPort)
startHub(t, hubPort, psk)
dest := newMockDest(t, modeEcho)
// Only the tunnel runs through the relay; the player talks to the hub
// directly, as it would from the internet. So the drop hits the middle leg
// while both terminal sockets stay healthy — exactly the case resumption is
// for.
relay := newBlackholeRelay(t, hubAddr)
c := startClientWithPing(t, relay.addr, psk, 1, 400, []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr},
})
pc := resumePlayer(t, hubAddr, "mc.local")
payload := randomPayload(1, 3<<20)
echoExchange(t, pc, payload, 512<<10, func() {
t.Log("hard-resetting the tunnel mid-transfer")
relay.dropAll()
})
t.Logf("stream survived the reset intact; worker conns now %d", c.WorkerConnCount())
}
// TestResumeWithConcurrentStreams covers the failure the single-stream test
// cannot reach: two independent tunnels resume at once. A torn rebind that
// writes a frame onto the wrong conn would credit or reset the other player's
// tunnel, and that only shows up when a second player is there to be corrupted.
func TestResumeWithConcurrentStreams(t *testing.T) {
const psk = "e2e-resume-multi"
hubPort := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", hubPort)
startHub(t, hubPort, psk)
dest := newMockDest(t, modeEcho)
relay := newBlackholeRelay(t, hubAddr)
startClientWithPing(t, relay.addr, psk, 4, 400, []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr},
})
const players = 3
conns := make([]net.Conn, players)
for i := range conns {
conns[i] = resumePlayer(t, hubAddr, "mc.local")
// Distinct payloads: if a reattach crosses two streams the bytes land on
// the wrong player, which an identical payload would hide.
if _, err := conns[i].Write([]byte(fmt.Sprintf("hello-%d", i))); err != nil {
t.Fatalf("player %d warmup write: %v", i, err)
}
echo := make([]byte, len("hello-0"))
_ = conns[i].SetReadDeadline(time.Now().Add(15 * time.Second))
if _, err := io.ReadFull(conns[i], echo); err != nil {
t.Fatalf("player %d warmup echo: %v", i, err)
}
}
var wg sync.WaitGroup
for i := range conns {
wg.Add(1)
go func(i int) {
defer wg.Done()
payload := randomPayload(int64(100+i), 768<<10)
// Only the first player triggers the reset; the others are mid-flight
// when it lands.
disrupt := func() {}
if i == 0 {
disrupt = relay.dropAll
}
echoExchange(t, conns[i], payload, 128<<10, disrupt)
}(i)
}
wg.Wait()
}
// TestResumeDisabledClosesImmediately pins the off switch. With resumption
// declined the hub must not hang the player waiting for a reattach that is never
// coming: the socket has to close as it did before the feature existed, so
// disabling it is a true revert rather than a slower failure.
func TestResumeDisabledClosesImmediately(t *testing.T) {
const psk = "e2e-resume-off"
hubPort := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", hubPort)
startHub(t, hubPort, psk)
dest := newMockDest(t, modeEcho)
relay := newBlackholeRelay(t, hubAddr)
off := false
startClientCfg(t, &client.Config{
Server: relay.addr,
PSK: psk,
MaxTunnels: 1,
PingIntervalMs: 400,
StreamResume: &off,
Mappings: []client.Mapping{{Pattern: "mc.local", Destination: dest.addr}},
})
pc := resumePlayer(t, hubAddr, "mc.local")
if _, err := pc.Write([]byte("ping")); err != nil {
t.Fatalf("warmup write: %v", err)
}
echo := make([]byte, 4)
_ = pc.SetReadDeadline(time.Now().Add(15 * time.Second))
if _, err := io.ReadFull(pc, echo); err != nil {
t.Fatalf("warmup echo: %v", err)
}
relay.dropAll()
// Well inside the hub's 20s grace: if the player is still open here, the hub
// parked a stream for a client that never opted in.
_ = pc.SetReadDeadline(time.Now().Add(10 * time.Second))
if _, err := pc.Read(make([]byte, 1)); err == nil {
t.Fatal("player socket stayed open after the tunnel dropped with resume disabled")
}
}
// TestResumeGraceExpiryClosesPlayer covers the other end of the lifetime: when
// the tunnel never comes back, a hung player must not hang forever. The hub
// drops it once its grace expires, without leaking the stream or its buffers.
func TestResumeGraceExpiryClosesPlayer(t *testing.T) {
const psk = "e2e-resume-grace"
hubPort := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", hubPort)
startHubCfg(t, hubPort, psk, map[string]any{"resumeGraceMs": 3000})
dest := newMockDest(t, modeEcho)
relay := newBlackholeRelay(t, hubAddr)
startClientCfg(t, &client.Config{
Server: relay.addr,
PSK: psk,
MaxTunnels: 1,
PingIntervalMs: 400,
ResumeGraceMs: 2000,
Mappings: []client.Mapping{{Pattern: "mc.local", Destination: dest.addr}},
})
pc := resumePlayer(t, hubAddr, "mc.local")
if _, err := pc.Write([]byte("ping")); err != nil {
t.Fatalf("warmup write: %v", err)
}
echo := make([]byte, 4)
_ = pc.SetReadDeadline(time.Now().Add(15 * time.Second))
if _, err := io.ReadFull(pc, echo); err != nil {
t.Fatalf("warmup echo: %v", err)
}
// Stop carrying the tunnel entirely, so every reattach attempt fails.
relay.stop()
relay.dropAll()
start := time.Now()
_ = pc.SetReadDeadline(time.Now().Add(30 * time.Second))
if _, err := pc.Read(make([]byte, 1)); err == nil {
t.Fatal("player socket never closed after the resume grace expired")
}
if elapsed := time.Since(start); elapsed < time.Second {
t.Fatalf("player closed after %s, before any reattach could be attempted", elapsed)
} else {
t.Logf("hung player released after %s", elapsed.Round(100*time.Millisecond))
}
}
+5 -7
View File
@@ -27,17 +27,15 @@ func echoRounds(t *testing.T, hubAddr string, rounds, size int) {
} }
// TestSlowPlayerDoesNotStallOthers: a player that stops reading while megabytes // TestSlowPlayerDoesNotStallOthers: a player that stops reading while megabytes
// are echoed back to it must not stall another stream on the same worker // are echoed back to it must not stall another player. Flow control and the
// connection (maxConn=1 forces sharing). Before per-stream flow control, the // 1:1 worker model keep a jammed tunnel from taking anyone else with it.
// hub paused the whole worker socket once that player's write queue filled,
// freezing every other stream's downstream data.
func TestSlowPlayerDoesNotStallOthers(t *testing.T) { func TestSlowPlayerDoesNotStallOthers(t *testing.T) {
const psk = "e2e-slowplayer" const psk = "e2e-slowplayer"
port := freePort(t) port := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", port) hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
startHub(t, port, psk) startHub(t, port, psk)
dest := newMockDest(t, modeEcho) dest := newMockDest(t, modeEcho)
startClient(t, hubAddr, psk, 1, []client.Mapping{ startClient(t, hubAddr, psk, 2, []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr}, {Pattern: "mc.local", Destination: dest.addr},
}) })
@@ -52,7 +50,7 @@ func TestSlowPlayerDoesNotStallOthers(t *testing.T) {
// Let the slow stream jam: its flow-control window fills and stays full. // Let the slow stream jam: its flow-control window fills and stays full.
time.Sleep(1 * time.Second) time.Sleep(1 * time.Second)
// The fast player shares the single worker conn and must still round-trip. // The fast player has its own worker conn and must still round-trip.
echoRounds(t, hubAddr, 10, 8*1024) echoRounds(t, hubAddr, 10, 8*1024)
} }
@@ -67,7 +65,7 @@ func TestSlowDestinationDoesNotStallOthers(t *testing.T) {
startHub(t, port, psk) startHub(t, port, psk)
dest := newMockDest(t, modeEcho) dest := newMockDest(t, modeEcho)
hole := newMockDest(t, modeBlackhole) hole := newMockDest(t, modeBlackhole)
startClient(t, hubAddr, psk, 1, []client.Mapping{ startClient(t, hubAddr, psk, 2, []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr}, {Pattern: "mc.local", Destination: dest.addr},
{Pattern: "hole.local", Destination: hole.addr}, {Pattern: "hole.local", Destination: hole.addr},
}) })
+219
View File
@@ -0,0 +1,219 @@
package e2e
import (
"bufio"
"bytes"
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"fmt"
"io"
"net"
"testing"
"time"
"github.com/iceBear67/redapricot/client"
"github.com/iceBear67/redapricot/client/wire"
)
// ---- mock velocity-forwarding destination ----
// veloEvent is what the mock backend saw in the (verified) forwarding payload.
type veloEvent struct {
version int
ip string
uuid []byte
name string
err error
}
type veloDest struct {
addr string
secret string
success []byte // the Login Success packet the backend sends after the exchange
events chan veloEvent
}
// newVeloDest starts a mock backend that requires Velocity modern forwarding:
// it reads the handshake and Login Start, sends the velocity:player_info
// query (with a negative message id, as Paper's random ids often are), verifies
// the HMAC-signed response, and finally sends a recognizable Login Success.
func newVeloDest(t *testing.T, secret string) *veloDest {
t.Helper()
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("velo dest listen: %v", err)
}
d := &veloDest{
addr: ln.Addr().String(),
secret: secret,
success: mcPacket(wire.NewWriter().
VarInt(0x02). // Login Success
Bytes(bytes.Repeat([]byte{0x42}, 16)).
String("e2ePlayer").
VarInt(0).
Out()),
events: make(chan veloEvent, 16),
}
t.Cleanup(func() { _ = ln.Close() })
go func() {
for {
conn, err := ln.Accept()
if err != nil {
return
}
go d.handle(conn)
}
}()
return d
}
const veloMsgID = -777
func (d *veloDest) handle(conn net.Conn) {
defer conn.Close()
ev := d.exchange(conn)
d.events <- ev
if ev.err == nil {
_, _ = conn.Write(d.success)
}
_, _ = io.Copy(io.Discard, conn) // hold the connection until the peer closes
}
func (d *veloDest) exchange(conn net.Conn) veloEvent {
br := bufio.NewReader(conn)
if _, err := readMCPacket(br); err != nil { // handshake
return veloEvent{err: fmt.Errorf("read handshake: %w", err)}
}
if _, err := readMCPacket(br); err != nil { // login start
return veloEvent{err: fmt.Errorf("read login start: %w", err)}
}
query := mcPacket(wire.NewWriter().
VarInt(0x04). // Login Plugin Request
VarInt(veloMsgID).
String("velocity:player_info").
U8(0x04). // max supported forwarding version
Out())
if _, err := conn.Write(query); err != nil {
return veloEvent{err: err}
}
resp, err := readMCPacket(br)
if err != nil {
return veloEvent{err: fmt.Errorf("read plugin response: %w", err)}
}
r := wire.NewReader(resp)
id, _ := r.VarInt()
if id != 0x02 {
return veloEvent{err: fmt.Errorf("expected Login Plugin Response, got packet %#x", id)}
}
msgID, _ := r.VarInt()
if !bytes.Equal(wire.AppendVarInt(nil, msgID), wire.AppendVarInt(nil, veloMsgID)) {
return veloEvent{err: fmt.Errorf("message id not echoed: got %d", msgID)}
}
okFlag, _ := r.U8()
if okFlag != 1 {
return veloEvent{err: fmt.Errorf("response marked unsuccessful")}
}
sig, err := r.Bytes(32)
if err != nil {
return veloEvent{err: fmt.Errorf("missing signature: %w", err)}
}
payload := r.Remaining()
mac := hmac.New(sha256.New, []byte(d.secret))
mac.Write(payload)
if !hmac.Equal(sig, mac.Sum(nil)) {
return veloEvent{err: fmt.Errorf("forwarding signature does not verify")}
}
pr := wire.NewReader(payload)
var ev veloEvent
ev.version, _ = pr.VarInt()
ev.ip, _ = pr.String()
ev.uuid, _ = pr.Bytes(16)
ev.name, err = pr.String()
if err != nil {
return veloEvent{err: fmt.Errorf("truncated payload: %w", err)}
}
if props, err := pr.VarInt(); err != nil || props != 0 || len(pr.Remaining()) != 0 {
return veloEvent{err: fmt.Errorf("unexpected properties/trailer in payload")}
}
return ev
}
func mcPacket(body []byte) []byte {
return append(wire.AppendVarInt(nil, len(body)), body...)
}
func readMCPacket(br *bufio.Reader) ([]byte, error) {
n, err := wire.ReadVarInt(br)
if err != nil {
return nil, err
}
if n <= 0 || n > 1<<20 {
return nil, fmt.Errorf("bad packet length %d", n)
}
pkt := make([]byte, n)
if _, err := io.ReadFull(br, pkt); err != nil {
return nil, err
}
return pkt, nil
}
// ---- test ----
// TestVelocityForwarding drives a full player login through the hub and a
// velocity-enabled mapping: the backend's velocity:player_info query must be
// answered by the client (never reaching the player), carrying the player's
// real IP, username and UUID, and the player's first bytes must be the
// backend's Login Success.
func TestVelocityForwarding(t *testing.T) {
const psk = "e2e-velocity"
const secret = "velo-forwarding-secret"
port := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
startHub(t, port, psk)
dest := newVeloDest(t, secret)
startClient(t, hubAddr, psk, 2, []client.Mapping{
{Pattern: `velo\.local`, Destination: dest.addr, VelocitySecret: secret},
})
pc := dialPlayer(t, hubAddr, "velo.local") // protocol 767, login intent
defer pc.Close()
uuid, _ := hex.DecodeString("00112233445566778899aabbccddeeff")
loginStart := mcPacket(wire.NewWriter().VarInt(0x00).String("e2ePlayer").Bytes(uuid).Out())
if _, err := pc.Write(loginStart); err != nil {
t.Fatalf("player login start: %v", err)
}
var ev veloEvent
select {
case ev = <-dest.events:
case <-time.After(10 * time.Second):
t.Fatalf("backend never completed the forwarding exchange")
}
if ev.err != nil {
t.Fatalf("backend rejected the forwarding exchange: %v", ev.err)
}
if ev.version != 4 {
t.Fatalf("forwarding version = %d, want 4 (lazy session)", ev.version)
}
if ev.ip != "127.0.0.1" {
t.Fatalf("forwarded IP = %q, want the player's real 127.0.0.1", ev.ip)
}
if ev.name != "e2ePlayer" || !bytes.Equal(ev.uuid, uuid) {
t.Fatalf("forwarded profile = %s/%x, want e2ePlayer/%x", ev.name, ev.uuid, uuid)
}
// The player must see the Login Success as its very first bytes — the
// velocity query must have been swallowed by the client.
got := make([]byte, len(dest.success))
_ = pc.SetReadDeadline(time.Now().Add(10 * time.Second))
if _, err := io.ReadFull(pc, got); err != nil {
t.Fatalf("player read login success: %v", err)
}
if !bytes.Equal(got, dest.success) {
t.Fatalf("player's first bytes are not the Login Success:\n got %x\nwant %x", got, dest.success)
}
}
+10 -1
View File
@@ -3,5 +3,14 @@
"psk": "change-me-to-a-long-random-passphrase", "psk": "change-me-to-a-long-random-passphrase",
"timestampWindowMs": 30000, "timestampWindowMs": 30000,
"pendingTimeoutMs": 10000, "pendingTimeoutMs": 10000,
"streamWindowBytes": 262144 "streamWindowBytes": 262144,
"sessionIdleTimeoutMs": 90000,
"streamResume": true,
"resumeGraceMs": 20000,
"maxParkedStreams": 256,
"statsIntervalMs": 0,
"registrationGraceMs": 15000,
"playerRatePerSec": 8,
"playerBurst": 16,
"maxPlayersPerIp": 64
} }
@@ -12,7 +12,17 @@ public record Config(
String psk, String psk,
long timestampWindowMs, long timestampWindowMs,
long pendingTimeoutMs, long pendingTimeoutMs,
int streamWindowBytes int streamWindowBytes,
long sessionIdleTimeoutMs,
boolean streamResume,
long resumeGraceMs,
int maxParkedStreams,
long maxParkedBytes,
long statsIntervalMs,
long registrationGraceMs,
double playerRatePerSec,
int playerBurst,
int maxPlayersPerIp
) { ) {
public static Config load(Path file) throws Exception { public static Config load(Path file) throws Exception {
JsonObject json = new JsonObject(Files.readString(file)); JsonObject json = new JsonObject(Files.readString(file));
@@ -29,12 +39,46 @@ public record Config(
int window = json.getInteger("streamWindowBytes", Protocol.DEFAULT_STREAM_WINDOW); int window = json.getInteger("streamWindowBytes", Protocol.DEFAULT_STREAM_WINDOW);
window = Math.max(Protocol.MIN_STREAM_WINDOW, Math.min(Protocol.MAX_STREAM_WINDOW, window)); window = Math.max(Protocol.MIN_STREAM_WINDOW, Math.min(Protocol.MAX_STREAM_WINDOW, window));
boolean resume = json.getBoolean("streamResume", Boolean.TRUE);
// A parked stream can hold up to one window of unacked bytes plus one of
// parked player bytes, for the whole grace period, and nothing else
// bounds how many streams park at once — so anyone able to kill worker
// conns is otherwise a cheap memory amplifier. The default admits ~256
// hanging players at the default window.
int maxParked = json.getInteger("maxParkedStreams", 256);
long maxParkedBytes = json.getLong("maxParkedBytes", (long) maxParked * 2 * window);
return new Config( return new Config(
host, host,
port, port,
psk, psk,
json.getLong("timestampWindowMs", 30_000L), json.getLong("timestampWindowMs", 30_000L),
json.getLong("pendingTimeoutMs", 10_000L), json.getLong("pendingTimeoutMs", 10_000L),
window); window,
// Comfortably above the client's default 20s ping interval;
// 0 disables the watchdog.
json.getLong("sessionIdleTimeoutMs", 90_000L),
resume,
// Must exceed the client's own grace by at least one dial, or the
// hub drops a player while its client is still mid-reattach. The
// value is advertised in SessionReady precisely so the client can
// clamp itself under it rather than rely on matching config.
json.getLong("resumeGraceMs", 20_000L),
maxParked,
maxParkedBytes,
json.getLong("statsIntervalMs", 0L),
// Long enough to cover a client's control-session reconnect
// (its backoff caps at 10s) without holding a player so long
// that they give up anyway; 0 disables and restores the old
// behaviour of dropping routes the moment a session closes.
json.getLong("registrationGraceMs", 15_000L),
// Player-only (Intent ∉ {17, 18}). 0 turns that mechanism off.
// Unmatched hostnames still consume a token — otherwise a
// hostname scan is a free flood. Intent 17 is never admitted
// through the limiter: every worker comes from the client's
// one IP.
Math.max(0, json.getDouble("playerRatePerSec", 8.0)),
Math.max(1, json.getInteger("playerBurst", 16)),
Math.max(0, json.getInteger("maxPlayersPerIp", 64)));
} }
} }
@@ -7,10 +7,16 @@ import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger; import org.apache.logging.log4j.Logger;
import java.nio.charset.StandardCharsets; import java.nio.charset.StandardCharsets;
import java.security.SecureRandom;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale; import java.util.Locale;
import java.util.Map; import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ThreadLocalRandom;
import java.util.regex.Pattern; import java.util.regex.Pattern;
import java.util.regex.PatternSyntaxException; import java.util.regex.PatternSyntaxException;
@@ -21,26 +27,76 @@ import java.util.regex.PatternSyntaxException;
*/ */
public final class Hub { public final class Hub {
private static final Logger LOG = LogManager.getLogger("redapricot.hub"); private static final Logger LOG = LogManager.getLogger("redapricot.hub");
private static final SecureRandom RNG = new SecureRandom();
public final Vertx vertx; public final Vertx vertx;
public final Config config; public final Config config;
public final byte[] pskBytes; public final byte[] pskBytes;
public final String pskAddress; public final String pskAddress;
public final IpRateLimiter limiter;
private final Map<String, Registration> patterns = new ConcurrentHashMap<>(); private final Map<String, Registration> patterns = new ConcurrentHashMap<>();
private final Map<String, PendingPlayer> pending = new ConcurrentHashMap<>(); private final Map<String, PendingPlayer> pending = new ConcurrentHashMap<>();
/** A compiled routing pattern and the control session that registered it. */ /**
private record Registration(Pattern regex, ControlSession session) {} * Every tunneled player, keyed by its current CID, whether live or parked.
* Keeping parked streams here rather than on the worker conn is the whole
* point: a stream's identity is the player, and state that dies with the
* connection cannot survive that connection dying. Insertion-ordered so the
* parked cap can evict the oldest first.
*/
private final Map<String, PlayerStream> streams = new LinkedHashMap<>();
private int parkedCount;
private long parkedBytes;
/** A successful match: the registered pattern that matched and its owning session. */ /**
public record Match(String pattern, ControlSession session) {} * A compiled routing pattern and the control session that registered it.
*
* <p>{@code session} is null while the registration is <b>orphaned</b> — its
* client's control session has closed but the route is held open until
* {@code orphanDeadline} in case the client reconnects. Players matching an
* orphaned route are hung rather than refused.
*/
private record Registration(Pattern regex, ControlSession session, long orphanDeadline) {
Registration(Pattern regex, ControlSession session) {
this(regex, session, 0);
}
boolean orphaned() {
return session == null;
}
}
/**
* A successful match: the registered pattern that matched and its owning
* session, which is null when the route is orphaned (§ control-outage hang).
*/
public record Match(String pattern, ControlSession session, long orphanDeadline) {}
public Hub(Vertx vertx, Config config) { public Hub(Vertx vertx, Config config) {
this.vertx = vertx; this.vertx = vertx;
this.config = config; this.config = config;
this.pskBytes = config.psk().getBytes(StandardCharsets.UTF_8); this.pskBytes = config.psk().getBytes(StandardCharsets.UTF_8);
this.pskAddress = Crypto.pskAddress(config.psk()); this.pskAddress = Crypto.pskAddress(config.psk());
this.limiter = new IpRateLimiter(
config.playerRatePerSec(), config.playerBurst(), config.maxPlayersPerIp());
// Unit tests construct a Hub with a null Vertx (no event loop).
if (limiter.enabled() && vertx != null) {
vertx.setPeriodic(IpRateLimiter.SWEEP_MS, id -> limiter.sweep(System.currentTimeMillis()));
}
}
/**
* Admit one player socket from {@code ip}. {@code null} means allowed;
* the caller must {@link #releasePlayer} on every close path (pending
* timeout, unmatched host, player FIN, park eviction).
*/
public IpRateLimiter.Deny admitPlayer(String ip) {
return limiter.admit(ip, System.currentTimeMillis());
}
public void releasePlayer(String ip) {
limiter.release(ip, System.currentTimeMillis());
} }
// ---- pattern registry ---- // ---- pattern registry ----
@@ -64,9 +120,29 @@ public final class Hub {
} }
patterns.put(pattern, new Registration(regex, session)); patterns.put(pattern, new Registration(regex, session));
LOG.info("registered pattern '{}' -> {}", pattern, session.id()); LOG.info("registered pattern '{}' -> {}", pattern, session.id());
replayAwaiting(pattern, session);
return Protocol.REGISTER_OK; return Protocol.REGISTER_OK;
} }
/**
* Deliver the control requests held while this pattern had no live session.
*
* <p>These players connected during the client's reconnect and were hung
* instead of refused; the request was never sent, so it is sent now. Each
* moves from "waiting for a route" to the ordinary "waiting for a worker",
* which means swapping its deadline over to {@code pendingTimeoutMs}.
*/
private void replayAwaiting(String pattern, ControlSession session) {
for (PendingPlayer p : pending.values()) {
if (!p.isAwaitingSession() || !p.getPattern().equals(pattern)) continue;
p.setAwaitingSession(false);
p.setOwner(session);
rearm(p, config.pendingTimeoutMs());
session.sendControlRequest(p.getCid(), p.getPattern(), p.getPlayerIp(), p.getPlayerPort());
LOG.info("replayed control request for hung player {} on session {}", p.getCidHex(), session.id());
}
}
public void unregister(String pattern, ControlSession session) { public void unregister(String pattern, ControlSession session) {
// Remove only if this session still owns the pattern (a newer session may have taken it). // Remove only if this session still owns the pattern (a newer session may have taken it).
patterns.computeIfPresent(pattern, (k, reg) -> reg.session() == session ? null : reg); patterns.computeIfPresent(pattern, (k, reg) -> reg.session() == session ? null : reg);
@@ -80,36 +156,140 @@ public final class Hub {
public Match match(String address) { public Match match(String address) {
String host = normalizeAddress(address); String host = normalizeAddress(address);
for (Map.Entry<String, Registration> e : patterns.entrySet()) { for (Map.Entry<String, Registration> e : patterns.entrySet()) {
if (e.getValue().regex().matcher(host).matches()) { Registration reg = e.getValue();
return new Match(e.getKey(), e.getValue().session()); if (reg.regex().matcher(host).matches()) {
return new Match(e.getKey(), reg.session(), reg.orphanDeadline());
} }
} }
return null; return null;
} }
/** Drop every pattern owned by a (closing) session, plus any players still pending for it. */ /**
* A control session closed. Its routes are kept as <b>orphaned</b> for
* {@code registrationGraceMs}, and the players waiting on them are hung
* rather than dropped.
*
* <p>Without this, a client's reconnect — half a second at best, ten at worst
* once its backoff has grown — is a window in which every arriving player is
* told there is no such server, even though the tunnel is seconds from being
* back. The players already tunneled are unaffected either way; they ride
* worker conns, which a control-session close never touches.
*
* <p>A grace of 0 restores the old behaviour exactly.
*/
public void removeSession(ControlSession session) { public void removeSession(ControlSession session) {
patterns.entrySet().removeIf(e -> e.getValue().session() == session); long grace = config.registrationGraceMs();
if (grace <= 0) {
patterns.entrySet().removeIf(e -> e.getValue().session() == session);
pending.values().removeIf(p -> {
if (p.getOwner() != session) return false;
cancelTimer(p);
p.getSocket().close();
LOG.info("dropping pending player {} (control session {} closed)", p.getCidHex(), session.id());
return true;
});
return;
}
long deadline = System.currentTimeMillis() + grace;
int orphaned = 0;
for (Map.Entry<String, Registration> e : patterns.entrySet()) {
Registration reg = e.getValue();
if (reg.session() != session) continue;
e.setValue(new Registration(reg.regex(), null, deadline));
orphaned++;
}
// A player that was already matched is in the same position: its request
// went to a session that will never answer, so it waits for the route to
// come back and is then replayed like any other.
int hung = 0;
for (PendingPlayer p : pending.values()) {
if (p.getOwner() != session) continue;
p.setOwner(null);
p.setAwaitingSession(true);
rearm(p, grace);
hung++;
}
if (orphaned > 0 || hung > 0) {
LOG.info("control session {} closed; holding {} route(s) and {} player(s) for {}ms",
session.id(), orphaned, hung, grace);
vertx.setTimer(grace, id -> expireOrphans());
}
}
/** Drop routes whose grace ran out, and the players still hung on them. */
private void expireOrphans() {
long now = System.currentTimeMillis();
Set<String> gone = new HashSet<>();
patterns.entrySet().removeIf(e -> {
Registration reg = e.getValue();
if (!reg.orphaned() || reg.orphanDeadline() > now) return false;
gone.add(e.getKey());
return true;
});
if (gone.isEmpty()) return;
LOG.info("dropping {} orphaned route(s) not reclaimed within the grace period", gone.size());
pending.values().removeIf(p -> { pending.values().removeIf(p -> {
if (p.getOwner() != session) return false; if (!p.isAwaitingSession() || !gone.contains(p.getPattern())) return false;
if (p.getTimerId() >= 0) vertx.cancelTimer(p.getTimerId()); cancelTimer(p);
p.getSocket().close(); p.getSocket().close();
LOG.info("dropping pending player {} (control session {} closed)", p.getCidHex(), session.id()); LOG.info("dropping hung player {} (route '{}' never came back)", p.getCidHex(), p.getPattern());
return true; return true;
}); });
} }
// ---- pending players ---- // ---- pending players ----
/**
* Mint a takeover capability. The CID is the only thing authorizing a client
* to claim a player (and, with resumption, to reclaim one), so it comes from
* a cryptographic source rather than ThreadLocalRandom — a predictable value
* would be a session-hijacking primitive.
*/
public byte[] newCid() { public byte[] newCid() {
byte[] cid = new byte[Protocol.CID_LEN]; byte[] cid = new byte[Protocol.CID_LEN];
ThreadLocalRandom.current().nextBytes(cid); RNG.nextBytes(cid);
return cid; return cid;
} }
public void addPending(PendingPlayer p) { public void addPending(PendingPlayer p) {
pending.put(p.getCidHex(), p); pending.put(p.getCidHex(), p);
p.setTimerId(vertx.setTimer(config.pendingTimeoutMs(), id -> { rearm(p, config.pendingTimeoutMs());
}
/**
* Hang a player whose route is orphaned: hold it until a client re-registers
* the pattern, at which point {@link #replayAwaiting} delivers the control
* request that was never sent.
*
* @param deadline wall-clock millis at which the route's grace runs out
*/
public void addAwaiting(PendingPlayer p, long deadline) {
p.setOwner(null);
p.setAwaitingSession(true);
pending.put(p.getCidHex(), p);
rearm(p, Math.max(1, deadline - System.currentTimeMillis()));
LOG.info("holding player {} for '{}': route is orphaned, waiting for its client",
p.getCidHex(), p.getPattern());
}
public PendingPlayer takePending(byte[] cid) {
String hex = Hex.encode(cid);
PendingPlayer p = pending.remove(hex);
if (p != null) cancelTimer(p);
return p;
}
public void removePending(String cidHex) {
PendingPlayer p = pending.remove(cidHex);
if (p != null) cancelTimer(p);
}
/** Replace a pending player's deadline, cancelling whatever it had. */
private void rearm(PendingPlayer p, long delayMs) {
cancelTimer(p);
p.setTimerId(vertx.setTimer(delayMs, id -> {
PendingPlayer removed = pending.remove(p.getCidHex()); PendingPlayer removed = pending.remove(p.getCidHex());
if (removed != null) { if (removed != null) {
LOG.warn("pending player {} timed out", p.getCidHex()); LOG.warn("pending player {} timed out", p.getCidHex());
@@ -118,16 +298,171 @@ public final class Hub {
})); }));
} }
public PendingPlayer takePending(byte[] cid) { private void cancelTimer(PendingPlayer p) {
String hex = Hex.encode(cid); if (p.getTimerId() >= 0) {
PendingPlayer p = pending.remove(hex); vertx.cancelTimer(p.getTimerId());
if (p != null && p.getTimerId() >= 0) vertx.cancelTimer(p.getTimerId()); p.setTimerId(-1);
return p; }
} }
public void removePending(String cidHex) { // ---- tunneled streams & resumption (§7.5) ----
PendingPlayer p = pending.remove(cidHex);
if (p != null && p.getTimerId() >= 0) vertx.cancelTimer(p.getTimerId()); /** Register a stream that has just been bound to a worker conn. */
public void addStream(PlayerStream st) {
streams.put(st.cidHex, st);
}
/** Forget a stream for good; its player socket is gone or going. */
public void removeStream(PlayerStream st) {
if (streams.remove(st.cidHex) == null) return;
if (st.parked) unpark(st);
st.unacked.clear();
// The player's closeHandler was replaced at SYN/RESUME bind, so the
// HubConnection cleanup never runs. This is the live/parked release
// path; pending/unmatched still go through HubConnection.closeCleanup.
releasePlayer(st.playerIp);
}
/** Look up a stream by the CID a client presented, live or parked. */
public PlayerStream streamByCid(byte[] cid) {
return streams.get(Hex.encode(cid));
}
/**
* Player bytes arrived: hand them to whichever conn currently carries the
* stream. Routing here rather than from the conn that installed the socket
* handler is what lets a stream change conns without rebinding handlers.
*/
public void onPlayerData(PlayerStream st, io.vertx.core.buffer.Buffer buf) {
WorkerConn w = st.worker;
if (w != null) {
w.playerData(st, buf);
return;
}
// Parked, so there is nowhere to send: hold the bytes in order and make
// sure the socket really is stopped. The player is paused on park, but a
// batch already in flight can still land here.
if (st.pendingUp == null) st.pendingUp = io.vertx.core.buffer.Buffer.buffer();
st.pendingUp.appendBuffer(buf);
st.player.pause();
}
/** The player hung up. Drop the stream everywhere and tell the client if it is still bound. */
public void onPlayerGone(PlayerStream st) {
WorkerConn w = st.worker;
removeStream(st);
if (w != null) w.playerGone(st);
}
/**
* Hang a player whose worker conn died, instead of closing it.
*
* <p>Only the tunnel leg failed — the player socket is still perfectly good —
* so it is paused and held until the client reattaches the stream over a
* fresh conn. The deadline is absolute and fixed at the first park: re-arming
* it on each park would let a flapping client hold a player forever.
*
* @return false if the stream cannot be parked and must be closed instead
*/
public boolean park(PlayerStream st) {
if (!st.resumable || st.parked) return false;
long now = System.currentTimeMillis();
if (st.graceDeadline == 0) st.graceDeadline = now + config.resumeGraceMs();
long remaining = st.graceDeadline - now;
if (remaining <= 0) return false;
st.parked = true;
st.worker = null;
// Explicitly, not as a side effect of the window filling: an idle stream
// has a wide-open window, so nothing else would stop the next keepalive
// walking into a send on a dead connection.
st.player.pause();
st.pausedForAggregate = false; // that conn's drain handler will never fire again
parkedCount++;
parkedBytes += st.parkedBytes();
st.timerId = vertx.setTimer(remaining, id -> {
LOG.info("parked player {} not reclaimed within grace; closing", st.cidHex);
removeStream(st);
st.player.close();
});
enforceParkedCaps(st);
return true;
}
/**
* Reclaim a parked stream. Returns null when no stream is parked under this
* CID; the caller distinguishes "never heard of it" from "still bound
* elsewhere" via {@link #streamByCid}.
*/
public PlayerStream takeParked(byte[] cid) {
PlayerStream st = streams.get(Hex.encode(cid));
if (st == null || !st.parked) return null;
unpark(st);
return st;
}
/** Re-key a stream to the freshly minted CID handed out in RESUME_ACK. */
public void rekeyStream(PlayerStream st, byte[] cid) {
streams.remove(st.cidHex);
st.cid = cid;
st.cidHex = Hex.encode(cid);
streams.put(st.cidHex, st);
}
private void unpark(PlayerStream st) {
if (!st.parked) return;
st.parked = false;
parkedCount--;
parkedBytes -= st.parkedBytes();
if (parkedBytes < 0) parkedBytes = 0;
if (st.timerId >= 0) {
vertx.cancelTimer(st.timerId);
st.timerId = -1;
}
}
/**
* Bound what hanging players may cost. Each holds up to a window of unsent
* bytes plus a window of parked ones for the whole grace period, and nothing
* else limits how many park at once — so anyone able to kill worker conns is
* otherwise a cheap memory amplifier. Oldest first, since they have the least
* grace left to be reclaimed in.
*/
private void enforceParkedCaps(PlayerStream keep) {
if (parkedCount <= config.maxParkedStreams() && parkedBytes <= config.maxParkedBytes()) return;
List<PlayerStream> evict = new ArrayList<>();
Iterator<PlayerStream> it = streams.values().iterator();
while (it.hasNext() && (parkedCount - evict.size() > config.maxParkedStreams()
|| parkedBytes > config.maxParkedBytes())) {
PlayerStream st = it.next();
if (!st.parked || st == keep) continue;
evict.add(st);
parkedBytes -= st.parkedBytes();
}
for (PlayerStream st : evict) {
LOG.warn("parked-stream cap reached; dropping hanging player {}", st.cidHex);
parkedBytes += st.parkedBytes(); // removeStream subtracts it again
removeStream(st);
st.player.close();
}
}
/** Live and parked stream counts, for the periodic stats line. */
public int streamCount() {
return streams.size();
}
public int parkedCount() {
return parkedCount;
}
public long parkedBytes() {
return parkedBytes;
}
public int patternCount() {
return patterns.size();
} }
// ---- helpers ---- // ---- helpers ----
@@ -103,17 +103,45 @@ public final class HubConnection {
if (intent == Protocol.INTENT_REDAPRICOT) { if (intent == Protocol.INTENT_REDAPRICOT) {
beginRedapricot(address, afterHandshake); beginRedapricot(address, afterHandshake);
} else if (intent == Protocol.INTENT_RESERVED) { } else if (intent == Protocol.INTENT_RESERVED) {
LOG.info("{} reserved intent 18; closing", id); LOG.info("{} reserved intent 18; replying with a status line and closing", id);
socket.close(); sendStatusLine();
} else { } else {
handlePlayer(address); handlePlayer(address);
} }
} }
// ---- Intent 18 (reserved: management/status) ----
/**
* Reply to an Intent-18 probe with a Minecraft status-response packet
* ({@code [Len: VarInt][0x00][JSON: String]}), the same shape a player gets
* for a status query (Intent 1), then close. This lets an operator probe
* the public port with ordinary tooling without joining the protocol, and
* it is the one reply the hub sends in plaintext — Intent 18 never
* negotiates encryption.
*/
private void sendStatusLine() {
String json = "{\"description\":{\"text\":\"redapricot hub\"},"
+ "\"version\":{\"name\":\"redapricot\",\"protocol\":767},"
+ "\"players\":{\"max\":0,\"online\":0}}";
ProtoWriter body = new ProtoWriter().u8(0x00).string(json);
byte[] bodyBytes = body.toBytes();
ProtoWriter pkt = new ProtoWriter().varInt(bodyBytes.length).bytes(bodyBytes);
// end() writes the reply and closes after it lands, so the packet is
// never cut short by the close racing the flush.
socket.end(Buffer.buffer(pkt.toBytes()));
}
// ---- redapricot session (Intent 17) ---- // ---- redapricot session (Intent 17) ----
private void beginRedapricot(String address, Buffer afterHandshake) { private void beginRedapricot(String address, Buffer afterHandshake) {
if (!address.equalsIgnoreCase(hub.pskAddress)) { // Strict equality, per PROTOCOL.md §2: the contract is exactly
// lowercase_hex(SHA3-224(PSK)), and the client always sends that. An
// uppercase or otherwise case-folded variant is not a valid session —
// accepting it would widen the acceptance surface beyond what the spec
// promises (and what the client ever produces), which is exactly the
// kind of "close enough" check that hides a wrong-PSK probe.
if (!address.equals(hub.pskAddress)) {
LOG.warn("{} bad PSK address; closing", id); LOG.warn("{} bad PSK address; closing", id);
socket.close(); socket.close();
return; return;
@@ -165,6 +193,13 @@ public final class HubConnection {
return; return;
} }
peerWindow = Math.min(peerWindow, Protocol.MAX_STREAM_WINDOW); peerWindow = Math.min(peerWindow, Protocol.MAX_STREAM_WINDOW);
// Heartbeat is optional: accept it only when the client offered it, so
// older clients keep working (they just lose silent-path detection).
boolean heartbeat = (flags & Protocol.FLAG_WORKER_HEARTBEAT) != 0;
// Resumption likewise. Parking a stream for a client that will never
// reattach is strictly worse than closing it — the player hangs for the
// whole grace instead of failing fast — so this bit gates the park path.
boolean resume = hub.config.streamResume() && (flags & Protocol.FLAG_STREAM_RESUME) != 0;
// REKEY = Rand || Timestamp(I64 big-endian). Magic is excluded. // REKEY = Rand || Timestamp(I64 big-endian). Magic is excluded.
byte[] rekey = new byte[randLen + 8]; byte[] rekey = new byte[randLen + 8];
@@ -178,32 +213,86 @@ public final class HubConnection {
frames.switchCiphers( frames.switchCiphers(
Crypto.decryptCipher(rekey, Crypto.DIR_C2S), Crypto.decryptCipher(rekey, Crypto.DIR_C2S),
Crypto.encryptCipher(rekey, Crypto.DIR_S2C)); Crypto.encryptCipher(rekey, Crypto.DIR_S2C));
// Echo the accepted flags plus our own receive window. // Echo the accepted flags plus our own receive window. When resumption is
frames.send(new ProtoWriter() // accepted, our grace period follows: the client clamps its own retry
// budget under it, which turns a cross-config invariant ("the hub must
// wait longer than the client retries") into a negotiated one that
// operator skew cannot break.
int accepted = Protocol.FLAG_STREAM_FC
| (heartbeat ? Protocol.FLAG_WORKER_HEARTBEAT : 0)
| (resume ? Protocol.FLAG_STREAM_RESUME : 0);
ProtoWriter ready = new ProtoWriter()
.u8(Protocol.CTL_SESSION_READY) .u8(Protocol.CTL_SESSION_READY)
.varInt(Protocol.FLAG_STREAM_FC) .varInt(accepted)
.varInt(hub.config.streamWindowBytes()) .varInt(hub.config.streamWindowBytes());
.toBytes()); if (resume) ready.varInt((int) hub.config.resumeGraceMs());
frames.send(ready.toBytes());
if (magic == Protocol.MAGIC_CONTROL) { if (magic == Protocol.MAGIC_CONTROL) {
ControlSession session = new ControlSession(hub, frames, id); ControlSession session = new ControlSession(hub, frames, id);
frames.setHandler(session::onFrame); frames.setHandler(session::onFrame);
closeCleanup = session::onClose; closeCleanup = session::onClose;
LOG.info("{} control session established", id); LOG.info("{} control session established (heartbeat {})", id, heartbeat);
} else if (magic == Protocol.MAGIC_WORKER) { } else if (magic == Protocol.MAGIC_WORKER) {
WorkerConn worker = new WorkerConn(hub, frames, id, peerWindow, hub.config.streamWindowBytes()); WorkerConn worker = new WorkerConn(hub, frames, id, peerWindow, hub.config.streamWindowBytes(), resume);
frames.setHandler(worker::onFrame); frames.setHandler(worker::onFrame);
closeCleanup = worker::onClose; closeCleanup = worker::onClose;
LOG.info("{} worker conn established (peer window {})", id, peerWindow); LOG.info("{} worker conn established (peer window {}, heartbeat {}, resume {})",
id, peerWindow, heartbeat, resume);
} else { } else {
LOG.warn("{} bad magic {}; closing", id, magic); LOG.warn("{} bad magic {}; closing", id, magic);
frames.close(); frames.close();
return;
} }
armIdleWatchdog();
}
/**
* Drop an established redapricot session that has gone silent. Clients ping
* both their control session and every worker conn, so silence means the
* path is dead — without this the hub would keep a zombie control session
* registered and keep routing players into it, and zombie worker conns would
* hold player sockets open forever. Player connections are never subject to
* this; only authenticated sessions are.
*/
private void armIdleWatchdog() {
long idleMs = hub.config.sessionIdleTimeoutMs();
if (idleMs <= 0) return;
long timer = hub.vertx.setPeriodic(idleMs / 2, tid -> {
if (frames.isClosed()) {
hub.vertx.cancelTimer(tid);
return;
}
long silent = System.currentTimeMillis() - frames.lastFrameAt();
if (silent > idleMs) {
LOG.warn("{} session silent for {}ms; closing", id, silent);
hub.vertx.cancelTimer(tid);
frames.close();
}
});
Runnable inner = closeCleanup;
closeCleanup = () -> {
hub.vertx.cancelTimer(timer);
inner.run();
};
} }
// ---- player connection ---- // ---- player connection ----
private void handlePlayer(String address) { private void handlePlayer(String address) {
// Before match / CID / pause: unmatched hostnames still consume a token,
// otherwise a hostname scan is a free flood. Intent 17 never reaches
// this method (PROTOCOL.md §9.1).
String ip = socket.remoteAddress() != null ? socket.remoteAddress().host() : "0.0.0.0";
if (hub.admitPlayer(ip) != null) {
socket.close();
return;
}
// Release on every close of this socket: pending timeout, unmatched
// host, player FIN, park eviction. Later cleanups wrap this, they
// must not replace it.
closeCleanup = () -> hub.releasePlayer(ip);
String host = Hub.normalizeAddress(address); String host = Hub.normalizeAddress(address);
Hub.Match matched = hub.match(address); Hub.Match matched = hub.match(address);
if (matched == null) { if (matched == null) {
@@ -217,18 +306,29 @@ public final class HubConnection {
String pattern = matched.pattern(); String pattern = matched.pattern();
byte[] cid = hub.newCid(); byte[] cid = hub.newCid();
String cidHex = Hex.encode(cid); String cidHex = Hex.encode(cid);
String ip = socket.remoteAddress() != null ? socket.remoteAddress().host() : "0.0.0.0";
int port = socket.remoteAddress() != null ? socket.remoteAddress().port() : 0; int port = socket.remoteAddress() != null ? socket.remoteAddress().port() : 0;
socket.pause(); socket.pause();
Buffer buffered = hs.copy(); // handshake + any pipelined bytes, forwarded verbatim Buffer buffered = hs.copy(); // handshake + any pipelined bytes, forwarded verbatim
PendingPlayer p = new PendingPlayer(cid, cidHex, socket, buffered, pattern, ip, port, session); PendingPlayer p = new PendingPlayer(cid, cidHex, socket, buffered, pattern, ip, port);
hub.addPending(p); closeCleanup = () -> {
closeCleanup = () -> hub.removePending(cidHex); hub.removePending(cidHex);
hub.releasePlayer(ip);
};
session.sendControlRequest(cid, pattern, ip, port); if (session == null) {
LOG.info("{} player {}:{} host '{}' matched pattern '{}' cid={}", // The route is orphaned: its client's control session has closed and
id, ip, port, host, pattern, cidHex); // has not come back yet. Hold the player rather than telling it there
// is no such server — the request is replayed the moment a client
// re-registers the pattern.
hub.addAwaiting(p, matched.orphanDeadline());
} else {
p.setOwner(session);
hub.addPending(p);
session.sendControlRequest(cid, pattern, ip, port);
}
LOG.info("{} player {}:{} host '{}' matched pattern '{}' cid={}{}",
id, ip, port, host, pattern, cidHex, session == null ? " (held: route orphaned)" : "");
} }
} }
@@ -24,6 +24,9 @@ public final class HubServer extends AbstractVerticle {
.setHost(config.host()) .setHost(config.host())
.setPort(config.port()) .setPort(config.port())
.setTcpNoDelay(true) .setTcpNoDelay(true)
// Probe idle sockets so a peer that becomes unreachable is
// eventually detected even when no frames are in flight.
.setTcpKeepAlive(true)
.setReuseAddress(true); .setReuseAddress(true);
NetServer server = vertx.createNetServer(opts); NetServer server = vertx.createNetServer(opts);
@@ -32,10 +35,27 @@ public final class HubServer extends AbstractVerticle {
if (ar.succeeded()) { if (ar.succeeded()) {
LOG.info("redapricot hub listening on {}:{}", config.host(), ar.result().actualPort()); LOG.info("redapricot hub listening on {}:{}", config.host(), ar.result().actualPort());
LOG.info("PSK handshake address: {}", hub.pskAddress); LOG.info("PSK handshake address: {}", hub.pskAddress);
armStats();
startPromise.complete(); startPromise.complete();
} else { } else {
startPromise.fail(ar.cause()); startPromise.fail(ar.cause());
} }
}); });
} }
/**
* Periodic one-line snapshot of what the hub is holding. Off unless
* statsIntervalMs is set, so it costs nothing by default.
*
* <p>Parked streams and the bytes they retain are the numbers worth watching:
* they are the memory stream resumption trades for keeping players connected,
* and the first place a resumption problem shows up as a trend.
*/
private void armStats() {
long interval = config.statsIntervalMs();
if (interval <= 0) return;
vertx.setPeriodic(interval, id -> LOG.info(
"stats streams={} parked={} parkedBytes={} patterns={}",
hub.streamCount(), hub.parkedCount(), hub.parkedBytes(), hub.patternCount()));
}
} }
@@ -0,0 +1,134 @@
package io.icybear.redapricot;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.Map;
/**
* Per-IP admission for <em>player</em> connections only (PROTOCOL.md §9.1).
*
* <p>Intent 17 (control session + every worker conn) is never admitted through
* here. Those sockets all come from the client's one address; limiting them
* would be the hub throttling its own client. e2e shares {@code 127.0.0.1}
* between players and the client for the same reason.
*
* <p>Event-loop confined: no locking. A {@code 0} rate or concurrent cap turns
* that mechanism off; both {@code 0} makes {@link #admit} a no-op.
*/
public final class IpRateLimiter {
private static final Logger LOG = LogManager.getLogger("redapricot.limit");
/** Idle buckets older than this are dropped so a one-shot flood cannot leak. */
static final long SWEEP_MS = 60_000;
private static final long DENY_LOG_INTERVAL_MS = 2_000;
public enum Deny { RATE, CONCURRENT }
private final double ratePerSec; // 0 = token bucket off
private final double burst;
private final int maxConcurrent; // 0 = concurrent cap off
private final Map<String, Bucket> buckets = new LinkedHashMap<>();
static final class Bucket {
double tokens;
long lastRefillMs;
int concurrent;
long lastActivityMs;
long lastDenyLogMs;
int deniesSinceLog;
}
public IpRateLimiter(double ratePerSec, double burst, int maxConcurrent) {
this.ratePerSec = Math.max(0, ratePerSec);
this.burst = Math.max(1, burst);
this.maxConcurrent = Math.max(0, maxConcurrent);
}
public boolean enabled() {
return ratePerSec > 0 || maxConcurrent > 0;
}
/**
* Consume one player admission for {@code ip}. {@code null} means allowed
* and the caller <em>must</em> {@link #release} when the socket closes.
*/
public Deny admit(String ip, long nowMs) {
if (!enabled()) return null;
Bucket b = bucket(ip, nowMs);
b.lastActivityMs = nowMs;
refill(b, nowMs);
if (ratePerSec > 0 && b.tokens < 1.0) {
noteDeny(ip, b, nowMs, Deny.RATE);
return Deny.RATE;
}
if (maxConcurrent > 0 && b.concurrent >= maxConcurrent) {
noteDeny(ip, b, nowMs, Deny.CONCURRENT);
return Deny.CONCURRENT;
}
if (ratePerSec > 0) b.tokens -= 1.0;
b.concurrent++;
return null;
}
public void release(String ip, long nowMs) {
Bucket b = buckets.get(ip);
if (b == null) return;
if (b.concurrent > 0) b.concurrent--;
b.lastActivityMs = nowMs;
}
/** Drop idle empty buckets. Safe to call on a timer. */
public void sweep(long nowMs) {
Iterator<Map.Entry<String, Bucket>> it = buckets.entrySet().iterator();
while (it.hasNext()) {
Bucket b = it.next().getValue();
if (b.concurrent == 0 && nowMs - b.lastActivityMs >= SWEEP_MS) {
it.remove();
}
}
}
/** Visible for tests. */
int bucketCount() {
return buckets.size();
}
/** Visible for tests. */
int concurrent(String ip) {
Bucket b = buckets.get(ip);
return b == null ? 0 : b.concurrent;
}
private Bucket bucket(String ip, long nowMs) {
Bucket b = buckets.get(ip);
if (b != null) return b;
b = new Bucket();
b.tokens = burst;
b.lastRefillMs = nowMs;
b.lastActivityMs = nowMs;
buckets.put(ip, b);
return b;
}
private void refill(Bucket b, long nowMs) {
if (ratePerSec <= 0) return;
double elapsed = (nowMs - b.lastRefillMs) / 1000.0;
if (elapsed <= 0) return;
b.tokens = Math.min(burst, b.tokens + elapsed * ratePerSec);
b.lastRefillMs = nowMs;
}
private void noteDeny(String ip, Bucket b, long nowMs, Deny why) {
b.deniesSinceLog++;
if (b.lastDenyLogMs != 0 && nowMs - b.lastDenyLogMs < DENY_LOG_INTERVAL_MS) {
return;
}
LOG.warn("dropping player from {}: {} ({} similar since last log)",
ip, why == Deny.RATE ? "rate" : "maxPlayersPerIp", b.deniesSinceLog);
b.lastDenyLogMs = nowMs;
b.deniesSinceLog = 0;
}
}
@@ -17,7 +17,22 @@ public final class PendingPlayer {
private final String pattern; private final String pattern;
private final String playerIp; private final String playerIp;
private final int playerPort; private final int playerPort;
private final ControlSession owner; // control session this player was routed to
/**
* Control session this player was routed to, or null while the route is
* orphaned. Mutable because a hung player is rebound to whichever session
* re-registers its pattern.
*/
@Setter
private ControlSession owner;
/**
* Hung waiting for a control session to come back, rather than waiting for a
* worker to claim it. No ControlRequest has been delivered yet, so this
* player is the hub's to replay once a route reappears.
*/
@Setter
private boolean awaitingSession;
@Setter @Setter
private long timerId = -1; private long timerId = -1;
@@ -0,0 +1,86 @@
package io.icybear.redapricot;
import io.vertx.core.buffer.Buffer;
import io.vertx.core.net.NetSocket;
/**
* One tunneled player: the player socket plus the flow-control state of the
* worker conn carrying it (PROTOCOL.md §7.3).
*
* <p>This is deliberately <em>not</em> owned by {@link WorkerConn}. A tunnel's
* identity is the player, not the connection it happens to ride: the worker conn
* is a replaceable transport, and state that dies with it cannot be recovered
* when it drops.
*
* <p>Confined to the hub's single event loop, so the mutable fields need no
* synchronization.
*/
public final class PlayerStream {
/** Capability that authorized the takeover; also the resume key. Re-minted on each reattach. */
byte[] cid;
String cidHex;
final NetSocket player;
/** Registered pattern that matched, echoed to the client. */
final String pattern;
final String playerIp;
final int playerPort;
/** The conn currently carrying this player. */
WorkerConn worker;
/** Budget for player -> client DATA. */
int sendWnd;
/** Player bytes awaiting send window; the player is paused while non-null. */
Buffer pendingUp;
/** client -> player bytes flushed but not yet granted back. */
int credited;
// Pause reasons. Vert.x pause() is a flag rather than a counter, so a socket
// can be paused for several reasons at once and must only be resumed once
// none of them hold — see WorkerConn#maybeResumePlayer.
boolean pausedForWindow; // this tunnel's send window is exhausted
boolean pausedForAggregate; // the worker socket's write queue is full
boolean parked; // the worker conn died; hanging until a reattach
/** Whether the conn carrying this player negotiated resumption (§7.5). */
boolean resumable;
// Resumption bookkeeping (§7.5). Three distinct offsets, and conflating them
// is the classic mistake: what to retransmit is measured from what the peer
// *accepted*, while the flow-control window is measured from what it
// *credited*. The gap between the two is credit still owed.
long sentOffset; // bytes handed to the wire
long ackedOffset; // running sum of WND deltas received
long acceptedOffset; // client -> player bytes taken off the wire
long deliveredOffset; // client -> player bytes actually written to the socket
final UnackedBytes unacked = new UnackedBytes();
/**
* Absolute wall-clock deadline for reattaching, fixed at the first park. Not
* re-armed on a later park: a flapping hub would otherwise keep extending it
* and hang the player indefinitely.
*/
long graceDeadline;
long timerId = -1;
PlayerStream(PendingPlayer p, WorkerConn worker, int sendWnd) {
this.cid = p.getCid();
this.cidHex = p.getCidHex();
this.player = p.getSocket();
this.pattern = p.getPattern();
this.playerIp = p.getPlayerIp();
this.playerPort = p.getPlayerPort();
this.worker = worker;
this.sendWnd = sendWnd;
}
/** Whether the player socket should be flowing right now. */
boolean shouldFlow() {
return !pausedForWindow && !pausedForAggregate && !parked;
}
/** Roughly how much this tunnel holds while parked, for the hub-wide cap. */
int parkedBytes() {
return unacked.length() + (pendingUp != null ? pendingUp.length() : 0);
}
}
@@ -26,18 +26,44 @@ public final class Protocol {
public static final int REGISTER_OK = 0x00; public static final int REGISTER_OK = 0x00;
public static final int REGISTER_ERR_PATTERN = 0x01; // pattern is not a valid regular expression public static final int REGISTER_ERR_PATTERN = 0x01; // pattern is not a valid regular expression
// Worker-conn mux frame types // Worker-conn frame types (one player per conn; no stream id).
public static final int MUX_SYN = 0x00; public static final int MUX_SYN = 0x00;
public static final int MUX_DATA = 0x01; public static final int MUX_DATA = 0x01;
public static final int MUX_FIN = 0x02; public static final int MUX_FIN = 0x02;
public static final int MUX_RST = 0x03; public static final int MUX_RST = 0x03;
public static final int MUX_WND = 0x04; // per-stream flow-control credit grant public static final int MUX_WND = 0x04; // per-connection flow-control credit grant
public static final int MUX_PING = 0x05; // liveness probe
public static final int MUX_PONG = 0x06; // liveness reply, echoes the nonce
/** Reattach a parked player to this conn: CID + the client's accepted offset (§7.5). */
public static final int MUX_RESUME = 0x07;
/** Hub's answer to RESUME: its accepted offset plus a freshly minted CID. */
public static final int MUX_RESUME_ACK = 0x08;
// RST reason codes (optional trailing byte; absence means "unspecified").
// Distinguishing them matters for resume: "unknown stream" is terminal,
// "already bound" means a racing attempt won and this one must not tear down.
public static final int RST_UNSPECIFIED = 0x00;
public static final int RST_UNKNOWN_STREAM = 0x01; // CID unknown, expired, or hub restarted
public static final int RST_ALREADY_BOUND = 0x02; // another RESUME won the race
public static final int RST_RESUME_ABANDONED = 0x03;
public static final int RST_FLOW_CONTROL = 0x04;
public static final int RST_DIAL_FAILED = 0x05;
// Session-establishment feature flags (trailing VarInt on the Rekey message, // Session-establishment feature flags (trailing VarInt on the Rekey message,
// echoed after the SessionReady type byte when accepted). // echoed after the SessionReady type byte when accepted).
public static final int FLAG_STREAM_FC = 0x01; public static final int FLAG_STREAM_FC = 0x01;
/** Connection-level PING/PONG on worker conns, so a silently dropped path is detected. */
public static final int FLAG_WORKER_HEARTBEAT = 0x02;
/**
* Stream resumption (§7.5): on a worker-conn drop the hub hangs the player
* socket instead of closing it, and the client reattaches that player
* byte-exactly over a fresh conn. When accepted, the hub appends its resume
* grace period to SessionReady so the client can bound its own retry budget
* against it.
*/
public static final int FLAG_STREAM_RESUME = 0x04;
// Per-stream flow-control window bounds (bytes). // Per-connection flow-control window bounds (bytes).
public static final int DEFAULT_STREAM_WINDOW = 256 * 1024; public static final int DEFAULT_STREAM_WINDOW = 256 * 1024;
public static final int MIN_STREAM_WINDOW = 32 * 1024; public static final int MIN_STREAM_WINDOW = 32 * 1024;
public static final int MAX_STREAM_WINDOW = 8 << 20; public static final int MAX_STREAM_WINDOW = 8 << 20;
@@ -0,0 +1,95 @@
package io.icybear.redapricot;
import io.vertx.core.buffer.Buffer;
import java.util.ArrayDeque;
import java.util.Deque;
/**
* The bytes a stream has sent but the client has not yet credited — exactly the
* region a reattach may have to retransmit (PROTOCOL.md §7.5).
*
* <p>It needs no cap of its own: credit is only granted as bytes reach the
* client's destination socket, so flow control already bounds the outstanding
* region to one window. That is what makes byte-exact resumption affordable.
*
* <p>A deque of the chunks already materialized by the send path, rather than one
* growing {@link Buffer}: appending to a Buffer reallocates and recopies as it
* grows, which would add a second per-byte copy to the whole upstream path. Here
* retention is free — the chunk was allocated to be sent anyway.
*/
final class UnackedBytes {
private final Deque<byte[]> chunks = new ArrayDeque<>();
private int head; // bytes of the first chunk already credited
private long base; // stream offset of the first live byte
private int length; // live bytes across all chunks
int length() {
return length;
}
long base() {
return base;
}
/** Offset one past the last byte handed to the wire. */
long end() {
return base + length;
}
void append(byte[] chunk) {
if (chunk.length == 0) return;
chunks.addLast(chunk);
length += chunk.length;
}
/** Drop everything the client has credited up to {@code off}. */
void advance(long off) {
long drop = off - base;
if (drop <= 0) return;
if (drop > length) drop = length; // only from a peer crediting bytes never sent
while (drop > 0) {
byte[] first = chunks.peekFirst();
int avail = first.length - head;
int take = (int) Math.min(drop, avail);
head += take;
base += take;
length -= take;
drop -= take;
if (head == first.length) {
chunks.removeFirst();
head = 0;
}
}
}
/**
* The outstanding bytes at and after {@code off}, or {@code null} when
* {@code off} falls outside what is still held — which means the peer named
* an offset we can no longer satisfy and the stream cannot be resumed.
*/
Buffer from(long off) {
long skip = off - base;
if (skip < 0 || skip > length) return null;
Buffer out = Buffer.buffer((int) (length - skip));
int start = head;
for (byte[] chunk : chunks) {
int avail = chunk.length - start;
if (skip >= avail) {
skip -= avail;
start = 0;
continue;
}
out.appendBytes(chunk, start + (int) skip, avail - (int) skip);
skip = 0;
start = 0;
}
return out;
}
void clear() {
chunks.clear();
head = 0;
length = 0;
}
}
@@ -4,109 +4,116 @@ import io.icybear.redapricot.net.EncryptedFrames;
import io.icybear.redapricot.util.ProtoReader; import io.icybear.redapricot.util.ProtoReader;
import io.icybear.redapricot.util.ProtoWriter; import io.icybear.redapricot.util.ProtoWriter;
import io.vertx.core.buffer.Buffer; import io.vertx.core.buffer.Buffer;
import io.vertx.core.net.NetSocket;
import lombok.RequiredArgsConstructor; import lombok.RequiredArgsConstructor;
import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger; import org.apache.logging.log4j.Logger;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
/** /**
* An authenticated worker connection (Magic 0x02). Multiplexes many player * An authenticated worker connection (Magic 0x02). Carries exactly one player:
* streams; the client opens streams via SYN(CID) to take over pending players. * the TCP connection is the tunnel (PROTOCOL.md §7). The client binds it with
* SYN(CID) (or RESUME) after SessionReady.
* *
* <p>Every stream has a credit window in both directions (PROTOCOL.md §7.3), * <p>The connection has a credit window in both directions (PROTOCOL.md §7.3),
* so one slow player only ever stalls its own stream — the shared worker * so a slow player only ever stalls itself.
* socket is never paused because of a single stream.
*/ */
@RequiredArgsConstructor @RequiredArgsConstructor
public final class WorkerConn { public final class WorkerConn {
private static final Logger LOG = LogManager.getLogger("redapricot.worker"); private static final Logger LOG = LogManager.getLogger("redapricot.worker");
/** Cap on a single DATA frame so no stream monopolizes the shared link for long. */ /** Cap on a single DATA frame so one write cannot occupy the link for long. */
private static final int CHUNK = 32 * 1024; private static final int CHUNK = 32 * 1024;
private final Hub hub; private final Hub hub;
private final EncryptedFrames frames; private final EncryptedFrames frames;
private final String id; private final String id;
private final int sendWndInit; // client's advertised per-stream receive window (our send budget) private final int sendWndInit; // client's advertised receive window (our send budget)
private final int recvWndInit; // our advertised per-stream receive window (basis for credit grants) private final int recvWndInit; // our advertised receive window (basis for credit grants)
/**
* Whether this conn negotiated stream resumption. Gates the park path: a
* client that will never reattach is better served by an immediate close than
* by a player left hanging for the whole grace period.
*/
private final boolean resume;
private final Map<Integer, StreamState> streams = new HashMap<>(); /** The one player bound to this conn, or null until SYN/RESUME. */
private PlayerStream stream;
// Aggregate backpressure for the single shared worker socket: players parked private boolean workerDrainArmed = false; // whether the worker socket's drainHandler is set
// until its write queue drains. Per-stream fairness is the credit windows'
// job; this only reacts to the whole pipe being congested.
private final Set<StreamState> upstreamPaused = new HashSet<>();
private boolean workerDrainArmed = false; // whether the worker socket's single drainHandler is set
/** Per-stream flow-control bookkeeping. */
private final class StreamState {
final NetSocket player;
int sendWnd = sendWndInit; // budget for player -> client DATA
Buffer pendingUp; // player bytes awaiting send window (player is paused meanwhile)
boolean pausedForWindow;
int credited; // client -> player bytes flushed but not yet granted back
StreamState(NetSocket player) {
this.player = player;
}
}
public void onFrame(byte[] payload) { public void onFrame(byte[] payload) {
ProtoReader r = new ProtoReader(payload); ProtoReader r = new ProtoReader(payload);
int type = r.readUByte(); int type = r.readUByte();
int sid = r.readVarInt();
switch (type) { switch (type) {
case Protocol.MUX_SYN -> handleSyn(sid, r.readBytes(Protocol.CID_LEN)); case Protocol.MUX_SYN -> handleSyn(r.readBytes(Protocol.CID_LEN));
case Protocol.MUX_DATA -> handleData(sid, r.readBuffer(r.remaining())); case Protocol.MUX_DATA -> handleData(r.readBuffer(r.remaining()));
case Protocol.MUX_WND -> handleWnd(sid, r.readVarInt()); case Protocol.MUX_WND -> handleWnd(r.readVarInt());
case Protocol.MUX_FIN, Protocol.MUX_RST -> closeStream(sid); case Protocol.MUX_RESUME ->
handleResume(r.readBytes(Protocol.CID_LEN), r.readI64(), r.readI64());
case Protocol.MUX_FIN, Protocol.MUX_RST -> closeBound();
case Protocol.MUX_PING -> sendPong(r.readI64());
case Protocol.MUX_PONG -> { /* liveness only; arrival is what matters */ }
case Protocol.FRAME_ERROR -> LOG.warn("worker {} error frame", id); case Protocol.FRAME_ERROR -> LOG.warn("worker {} error frame", id);
default -> LOG.warn("worker {} unknown mux type {}", id, type); default -> LOG.warn("worker {} unknown frame type {}", id, type);
} }
} }
private void handleSyn(int sid, byte[] cid) { private void handleSyn(byte[] cid) {
if (stream != null) {
LOG.warn("worker {} SYN on an already-bound conn; closing", id);
sendRst(Protocol.RST_ALREADY_BOUND);
frames.close();
return;
}
PendingPlayer p = hub.takePending(cid); PendingPlayer p = hub.takePending(cid);
if (p == null) { if (p == null) {
LOG.warn("worker {} SYN for unknown CID", id); LOG.warn("worker {} SYN for unknown CID", id);
sendRst(sid); sendRst(Protocol.RST_UNKNOWN_STREAM);
return; return;
} }
NetSocket player = p.getSocket(); PlayerStream st = new PlayerStream(p, this, sendWndInit);
StreamState st = new StreamState(player); st.resumable = resume;
streams.put(sid, st); stream = st;
hub.addStream(st);
// From now on the player socket belongs to this stream. // From now on the player socket belongs to this tunnel. The handlers are
player.handler(buf -> { // installed once and route through the hub, which dispatches to whichever
sendUpstream(sid, st, buf); // conn currently carries the player.
checkAggregate(st); //
}); // They must not call this conn's methods directly: a lambda defined here
player.closeHandler(v -> onPlayerGone(sid, st)); // captures `this`, so after the player moves to another conn it would keep
player.exceptionHandler(t -> onPlayerGone(sid, st)); // writing into the dead one's transport, where sends are silently dropped
// and the player goes mute with nothing logged. Re-installing handlers on
// every reattach would be the other option, but a Vert.x socket resumed
// while a stale handler is still attached loses bytes, so routing beats
// rebinding.
st.player.handler(buf -> hub.onPlayerData(st, buf));
st.player.closeHandler(v -> hub.onPlayerGone(st));
st.player.exceptionHandler(t -> hub.onPlayerGone(st));
// Forward the buffered handshake (and any pipelined bytes), then resume. // Forward the buffered handshake (and any pipelined bytes), then resume.
sendUpstream(sid, st, p.getBuffered()); sendUpstream(st, p.getBuffered());
if (!st.pausedForWindow) player.resume(); maybeResumePlayer(st);
checkAggregate(st);
LOG.info("worker {} bound to {}", id, st.pattern);
}
/** Player bytes arrived on the player this conn currently carries. */
void playerData(PlayerStream st, Buffer buf) {
sendUpstream(st, buf);
checkAggregate(st); checkAggregate(st);
LOG.info("worker {} stream {} bound to {}", id, sid, p.getPattern());
} }
/** /**
* Send player bytes to the client, chunked and clipped to the stream window; * Send player bytes to the client, chunked and clipped to the send window;
* the overflow is parked in {@code pendingUp} and the player socket paused * the overflow is parked in {@code pendingUp} and the player socket paused
* until the client grants more credit. * until the client grants more credit.
*/ */
private void sendUpstream(int sid, StreamState st, Buffer buf) { private void sendUpstream(PlayerStream st, Buffer buf) {
if (st.pendingUp != null) { // still waiting for window; keep ordering if (st.pendingUp != null) { // still waiting for window; keep ordering
st.pendingUp.appendBuffer(buf); st.pendingUp.appendBuffer(buf);
return; return;
} }
int off = drainUpstream(sid, st, buf, 0); int off = drainUpstream(st, buf, 0);
if (off < buf.length()) { if (off < buf.length()) {
st.pendingUp = buf.getBuffer(off, buf.length()); st.pendingUp = buf.getBuffer(off, buf.length());
if (!st.pausedForWindow) { if (!st.pausedForWindow) {
@@ -116,111 +123,263 @@ public final class WorkerConn {
} }
} }
/** Send from {@code buf[off..]} within the stream window, chunked; returns the new offset. */ /** Send from {@code buf[off..]} within the send window, chunked; returns the new offset. */
private int drainUpstream(int sid, StreamState st, Buffer buf, int off) { private int drainUpstream(PlayerStream st, Buffer buf, int off) {
while (off < buf.length() && st.sendWnd > 0) { while (off < buf.length() && st.sendWnd > 0) {
int n = Math.min(Math.min(CHUNK, st.sendWnd), buf.length() - off); int n = Math.min(Math.min(CHUNK, st.sendWnd), buf.length() - off);
sendData(sid, buf.getBytes(off, off + n)); byte[] chunk = buf.getBytes(off, off + n);
if (st.resumable) {
// Retain before sending. A frame written to a dying socket is
// lost with no notification, so the only trustworthy record of
// what the client still owes us is taken before the attempt.
// Retention is free here: the chunk was materialized to be sent.
st.unacked.advance(st.ackedOffset);
st.unacked.append(chunk);
}
st.sentOffset += n;
sendData(chunk);
st.sendWnd -= n; st.sendWnd -= n;
off += n; off += n;
} }
return off; return off;
} }
/** The client granted {@code delta} more bytes of credit on a stream. */ /** The client granted {@code delta} more bytes of credit. */
private void handleWnd(int sid, int delta) { private void handleWnd(int delta) {
StreamState st = streams.get(sid); PlayerStream st = stream;
if (st == null || delta <= 0) return; if (st == null || delta <= 0) return;
// The running sum doubles as the acked offset: the client grants credit
// exactly as bytes reach the destination socket, so a credited byte can
// never need retransmitting.
st.ackedOffset += delta;
st.sendWnd += delta; st.sendWnd += delta;
if (st.pendingUp != null) { if (st.pendingUp != null) {
Buffer pending = st.pendingUp; Buffer pending = st.pendingUp;
int off = drainUpstream(sid, st, pending, 0); int off = drainUpstream(st, pending, 0);
st.pendingUp = off >= pending.length() ? null : pending.getBuffer(off, pending.length()); st.pendingUp = off >= pending.length() ? null : pending.getBuffer(off, pending.length());
} }
if (st.pendingUp == null && st.pausedForWindow) { if (st.pendingUp == null) st.pausedForWindow = false;
st.pausedForWindow = false; maybeResumePlayer(st);
if (!upstreamPaused.contains(st)) st.player.resume();
}
checkAggregate(st); checkAggregate(st);
} }
private void handleData(int sid, Buffer data) { private void handleData(Buffer data) {
StreamState st = streams.get(sid); PlayerStream st = stream;
if (st == null) return; if (st == null) return;
// Never pause the shared socket: the client bounds what it sends per // Never pause the worker socket: the client bounds what it sends to our
// stream to our advertised window, so a slow player only piles up a // advertised window, so a slow player only piles up a bounded amount in
// bounded amount in its own write queue; credit is granted back as the // its own write queue; credit is granted back as the write completes
// write completes (i.e. the bytes reached the player socket). // (i.e. the bytes reached the player socket).
int len = data.length(); int len = data.length();
// Accepted the moment the bytes are taken off the wire, not when the write
// completes. Completion is asynchronous and suppressed once the connection
// closes — precisely when a reattach needs this number to be right — so
// reporting delivery would under-count and make the client replay bytes
// the player already has.
st.acceptedOffset += len;
st.player.write(data).onComplete(ar -> { st.player.write(data).onComplete(ar -> {
if (ar.failed() || frames.isClosed() || streams.get(sid) != st) return; if (ar.failed()) return;
// Both counters advance even if this conn has since died or the player
// has moved on. Discarding them would destroy up to half a window of
// credit per outage, and — worse — leave the delivered offset that a
// reattach restates the window from permanently short.
st.deliveredOffset += len;
st.credited += len; st.credited += len;
if (st.worker != this || frames.isClosed()) return;
if (st.credited * 2 >= recvWndInit) { if (st.credited * 2 >= recvWndInit) {
int delta = st.credited; int delta = st.credited;
st.credited = 0; st.credited = 0;
sendWnd(sid, delta); sendWnd(delta);
} }
}); });
} }
private void closeStream(int sid) { /**
StreamState st = streams.remove(sid); * Reattach a parked player to this connection (§7.5).
*
* <p>Runs to completion in one event-loop turn — rebind, acknowledge, replay —
* so the hub's single-threaded model makes the ordering race-free by
* construction, with no interleaving of live and replayed bytes to reason
* about.
*/
private void handleResume(byte[] cid, long clientAccepted, long clientDelivered) {
if (stream != null) {
LOG.warn("worker {} RESUME on an already-bound conn; closing", id);
sendRst(Protocol.RST_ALREADY_BOUND);
frames.close();
return;
}
PlayerStream st = hub.takeParked(cid);
if (st == null) {
// Tell a player we have never heard of apart from one that is still
// bound: the first is terminal for the client, the second means a
// racing attempt won and this one must leave the player alone.
boolean bound = hub.streamByCid(cid) != null;
LOG.warn("worker {} RESUME for {} CID", id, bound ? "still-bound" : "unknown");
sendRst(bound ? Protocol.RST_ALREADY_BOUND : Protocol.RST_UNKNOWN_STREAM);
return;
}
// Delivery is a strictly stronger fact than credit — the client only
// credits what it has delivered — so the reported offset can be adopted
// wholesale. Doing so also repairs the ledger: the grants destroyed by the
// outage are exactly the gap between the two, and without this the
// retained region would carry that dead prefix for the stream's whole life.
st.ackedOffset = Math.max(st.ackedOffset, clientDelivered);
st.unacked.advance(st.ackedOffset);
Buffer replay = st.unacked.from(clientAccepted);
if (replay == null) {
LOG.warn("worker {} RESUME at offset {} outside the retained region [{}, {}]; closing player",
id, clientAccepted, st.unacked.base(), st.unacked.end());
sendRst(Protocol.RST_UNKNOWN_STREAM);
hub.removeStream(st);
st.player.close();
return;
}
st.worker = this;
st.resumable = resume;
stream = st;
// Restate the window rather than patching it. Three offsets, three jobs:
// the replay above is measured from what the client *accepted*, the window
// from what it *delivered* (the window being a promise about undelivered
// bytes), and never from what it *credited* — credit travels as deltas, and
// the grants in flight when the connection died are gone for good, so a
// window derived from them stays permanently short. When a full window was
// outstanding at the drop that means permanently zero, which deadlocks:
// nothing can be sent, so no credit can ever come back.
int outstanding = st.unacked.length();
st.sendWnd = Math.max(0, sendWndInit - outstanding);
// Symmetrically, drop our own pending credit instead of flushing it: the
// delivered offset in the ack already carries everything those deltas
// would have, and sending both would grant the same bytes twice.
st.credited = 0;
// A fresh capability per reattach keeps a CID single-use, so a leaked one
// never grants more than the outage it was observed in.
byte[] newCid = hub.newCid();
hub.rekeyStream(st, newCid);
frames.send(new ProtoWriter()
.u8(Protocol.MUX_RESUME_ACK)
.i64(st.acceptedOffset)
.i64(st.deliveredOffset)
.bytes(newCid)
.toBytes());
// Replayed straight to the wire: it must not be re-charged against the
// window or re-appended to the retained region, both of which
// drainUpstream would do.
for (int off = 0; off < replay.length(); off += CHUNK) {
int end = Math.min(off + CHUNK, replay.length());
sendData(replay.getBytes(off, end));
}
if (st.pendingUp != null) {
Buffer pending = st.pendingUp;
int off = drainUpstream(st, pending, 0);
st.pendingUp = off >= pending.length() ? null : pending.getBuffer(off, pending.length());
}
if (st.pendingUp == null) st.pausedForWindow = false;
maybeResumePlayer(st);
checkAggregate(st);
LOG.info("worker {} resumed ({} bytes replayed, {} outstanding)",
id, replay.length(), outstanding);
}
private void closeBound() {
PlayerStream st = stream;
stream = null;
if (st != null) { if (st != null) {
upstreamPaused.remove(st); st.worker = null;
hub.removeStream(st);
st.player.close(); st.player.close();
} }
} }
/** Park the player if the shared worker socket's write queue is congested. */ /** Park the player if the worker socket's write queue is congested. */
private void checkAggregate(StreamState st) { private void checkAggregate(PlayerStream st) {
if (frames.writeQueueFull() && upstreamPaused.add(st)) { if (!st.pausedForAggregate && frames.writeQueueFull()) {
st.pausedForAggregate = true;
st.player.pause(); st.player.pause();
armWorkerDrain(); armWorkerDrain();
} }
} }
/** Register (once) the shared worker socket's single drain handler; on drain, wake parked players. */ /** Register (once) the worker socket's drain handler; on drain, wake the player. */
private void armWorkerDrain() { private void armWorkerDrain() {
if (workerDrainArmed) return; if (workerDrainArmed) return;
workerDrainArmed = true; workerDrainArmed = true;
frames.socket().drainHandler(v -> { frames.socket().drainHandler(v -> {
workerDrainArmed = false; workerDrainArmed = false;
if (upstreamPaused.isEmpty()) return; PlayerStream st = stream;
StreamState[] parked = upstreamPaused.toArray(new StreamState[0]); if (st == null || !st.pausedForAggregate) return;
upstreamPaused.clear(); st.pausedForAggregate = false;
for (StreamState st : parked) { maybeResumePlayer(st);
if (!st.pausedForWindow) st.player.resume();
}
}); });
} }
/** The player side of a stream vanished: drop it from every table, release any backpressure it held, and FIN the peer if still live. */ /**
private void onPlayerGone(int sid, StreamState st) { * Resume the player socket if no reason to hold it applies any more.
boolean wasLive = streams.remove(sid) == st; *
upstreamPaused.remove(st); * <p>The single arbitration point for every pause reason. Vert.x
if (wasLive) sendFin(sid); * {@code pause()} is a flag rather than a counter, so resuming while another
* reason still holds would let bytes through that we have nowhere to put.
*/
private void maybeResumePlayer(PlayerStream st) {
if (st.shouldFlow()) st.player.resume();
} }
private void sendData(int sid, byte[] data) { /** The player side of the tunnel this conn carries vanished: unbind it and FIN the client. */
frames.send(new ProtoWriter().u8(Protocol.MUX_DATA).varInt(sid).bytes(data).toBytes()); void playerGone(PlayerStream st) {
boolean wasLive = stream == st;
if (wasLive) stream = null;
st.worker = null;
if (wasLive) sendFin();
} }
private void sendFin(int sid) { private void sendData(byte[] data) {
frames.send(new ProtoWriter().u8(Protocol.MUX_FIN).varInt(sid).toBytes()); frames.send(new ProtoWriter().u8(Protocol.MUX_DATA).bytes(data).toBytes());
} }
private void sendRst(int sid) { private void sendFin() {
frames.send(new ProtoWriter().u8(Protocol.MUX_RST).varInt(sid).toBytes()); frames.send(new ProtoWriter().u8(Protocol.MUX_FIN).toBytes());
} }
private void sendWnd(int sid, int delta) { /** The reason is a trailing byte, optional on the wire; peers that predate it send none. */
frames.send(new ProtoWriter().u8(Protocol.MUX_WND).varInt(sid).varInt(delta).toBytes()); private void sendRst(int reason) {
frames.send(new ProtoWriter().u8(Protocol.MUX_RST).u8(reason).toBytes());
} }
private void sendWnd(int delta) {
frames.send(new ProtoWriter().u8(Protocol.MUX_WND).varInt(delta).toBytes());
}
/** Answer the client's liveness probe, echoing its nonce. */
private void sendPong(long nonce) {
frames.send(new ProtoWriter().u8(Protocol.MUX_PONG).i64(nonce).toBytes());
}
/**
* Only the tunnel leg died. Where the session negotiated resumption the
* player socket is hung rather than closed, and waits for the client to
* reattach over a fresh conn (§7.5); otherwise this is the old,
* unconditional close.
*/
public void onClose() { public void onClose() {
for (StreamState st : streams.values()) st.player.close(); PlayerStream st = stream;
streams.clear(); stream = null;
upstreamPaused.clear(); if (st == null) {
LOG.info("worker {} closed", id); LOG.info("worker {} closed (unbound)", id);
return;
}
st.worker = null;
if (hub.park(st)) {
LOG.info("worker {} closed (player hung for reattach)", id);
} else {
hub.removeStream(st);
st.player.close();
LOG.info("worker {} closed (player dropped)", id);
}
} }
} }
@@ -23,6 +23,7 @@ public final class EncryptedFrames {
private FrameHandler handler; private FrameHandler handler;
private Buffer buf = Buffer.buffer(); private Buffer buf = Buffer.buffer();
private boolean closed = false; private boolean closed = false;
private long lastFrameAt = System.currentTimeMillis();
public EncryptedFrames(NetSocket socket, Cipher in, Cipher out, FrameHandler handler) { public EncryptedFrames(NetSocket socket, Cipher in, Cipher out, FrameHandler handler) {
this.socket = socket; this.socket = socket;
@@ -71,6 +72,7 @@ public final class EncryptedFrames {
byte[] pt = in.update(ct); byte[] pt = in.update(ct);
if (pt == null) pt = new byte[0]; if (pt == null) pt = new byte[0];
buf = buf.getBuffer(hdr + payloadLen, buf.length()); buf = buf.getBuffer(hdr + payloadLen, buf.length());
lastFrameAt = System.currentTimeMillis();
FrameHandler h = handler; FrameHandler h = handler;
if (h != null) { if (h != null) {
try { try {
@@ -96,6 +98,9 @@ public final class EncryptedFrames {
public boolean writeQueueFull() { return socket.writeQueueFull(); } public boolean writeQueueFull() { return socket.writeQueueFull(); }
/** Wall-clock millis when the last complete frame was decoded; basis for idle detection. */
public long lastFrameAt() { return lastFrameAt; }
public void close() { public void close() {
if (closed) return; if (closed) return;
closed = true; closed = true;
@@ -77,7 +77,10 @@ class CryptoCodecTest {
/** A Hub whose event loop is never touched (register/match/normalize use no Vert.x state). */ /** A Hub whose event loop is never touched (register/match/normalize use no Vert.x state). */
private static Hub testHub() { private static Hub testHub() {
return new Hub(null, new Config("0.0.0.0", 25565, "test-psk", 30_000L, 10_000L, Protocol.DEFAULT_STREAM_WINDOW)); return new Hub(null, new Config("0.0.0.0", 25565, "test-psk", 30_000L, 10_000L,
Protocol.DEFAULT_STREAM_WINDOW, 90_000L,
true, 20_000L, 256, 256L * 2 * Protocol.DEFAULT_STREAM_WINDOW, 0L, 15_000L,
0, 1, 0));
} }
private static ControlSession testSession(Hub hub, String id) { private static ControlSession testSession(Hub hub, String id) {
@@ -0,0 +1,86 @@
package io.icybear.redapricot;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
class IpRateLimiterTest {
@Test
void burstThenRefill() {
IpRateLimiter lim = new IpRateLimiter(2, 2, 0);
long t = 1_000;
assertNull(lim.admit("1.1.1.1", t));
assertNull(lim.admit("1.1.1.1", t));
assertEquals(IpRateLimiter.Deny.RATE, lim.admit("1.1.1.1", t));
// 500 ms at 2/s = 1 token.
assertNull(lim.admit("1.1.1.1", t + 500));
assertEquals(IpRateLimiter.Deny.RATE, lim.admit("1.1.1.1", t + 500));
}
@Test
void concurrentCapIndependentOfRate() {
IpRateLimiter lim = new IpRateLimiter(0, 16, 1);
long t = 1_000;
assertNull(lim.admit("10.0.0.1", t));
assertEquals(1, lim.concurrent("10.0.0.1"));
assertEquals(IpRateLimiter.Deny.CONCURRENT, lim.admit("10.0.0.1", t));
lim.release("10.0.0.1", t);
assertEquals(0, lim.concurrent("10.0.0.1"));
assertNull(lim.admit("10.0.0.1", t));
}
@Test
void ipsAreIndependent() {
IpRateLimiter lim = new IpRateLimiter(1, 1, 1);
long t = 1_000;
assertNull(lim.admit("a", t));
assertNull(lim.admit("b", t));
assertEquals(IpRateLimiter.Deny.RATE, lim.admit("a", t));
assertEquals(IpRateLimiter.Deny.RATE, lim.admit("b", t));
}
@Test
void bothOffIsNoOp() {
IpRateLimiter lim = new IpRateLimiter(0, 16, 0);
long t = 1_000;
for (int i = 0; i < 100; i++) {
assertNull(lim.admit("1.2.3.4", t));
}
assertEquals(0, lim.bucketCount());
}
@Test
void sweepDropsIdleEmptyBuckets() {
IpRateLimiter lim = new IpRateLimiter(8, 8, 64);
long t = 1_000;
assertNull(lim.admit("9.9.9.9", t));
lim.release("9.9.9.9", t);
assertEquals(1, lim.bucketCount());
lim.sweep(t + IpRateLimiter.SWEEP_MS - 1);
assertEquals(1, lim.bucketCount());
lim.sweep(t + IpRateLimiter.SWEEP_MS);
assertEquals(0, lim.bucketCount());
}
@Test
void sweepKeepsLiveBuckets() {
IpRateLimiter lim = new IpRateLimiter(8, 8, 64);
long t = 1_000;
assertNull(lim.admit("9.9.9.9", t));
lim.sweep(t + IpRateLimiter.SWEEP_MS * 2);
assertEquals(1, lim.bucketCount());
assertEquals(1, lim.concurrent("9.9.9.9"));
}
@Test
void denyDoesNotConsumeASlot() {
IpRateLimiter lim = new IpRateLimiter(1, 1, 8);
long t = 1_000;
assertNull(lim.admit("x", t));
assertNotNull(lim.admit("x", t));
assertEquals(1, lim.concurrent("x"));
}
}