Compare commits
4
Commits
a41cb7965e
...
4df2560331
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4df2560331 | ||
|
|
da17140583 | ||
|
|
7bd84af48d | ||
|
|
e63a34d53a |
@@ -0,0 +1,286 @@
|
|||||||
|
# redapricot 审计报告
|
||||||
|
|
||||||
|
审计日期:2026-08-15
|
||||||
|
审计方式:5 个并行探索子代理深读全部源码(~8200 行)与测试,交叉核对两端实现与
|
||||||
|
PROTOCOL.md;随后对全部高风险发现逐一人工复核(含 velocity 模块与 Velocity 官方
|
||||||
|
源码 `dev/4.0.0` 的 `PlayerDataForwarding.java` 逐字节比对)。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 0. 总体评价
|
||||||
|
|
||||||
|
**设计质量很高,且文档与实现高度一致。** 亮点包括:单事件循环 hub 让并发模型归零;
|
||||||
|
「持锁不 dial」的池设计;resume 三偏移量(Sent/Accepted/Delivered)的区分在两端实现
|
||||||
|
和注释中都正确且互相印证;流控窗口同时充当重传缓冲区上限(「保留区无需自带上限」
|
||||||
|
这一论证成立,依赖链完整);off-switch(`streamResume:false`、`statsIntervalMs:0`)
|
||||||
|
确实只留一个分支;写超时、心跳、会话建立 deadline 等「liveness 显式化」哲学落实到位。
|
||||||
|
|
||||||
|
**主要风险集中在三处**:客户端 resume 状态的并发管理(存在静默数据损坏路径)、
|
||||||
|
`Close()` 与关闭语义(库用场景泄漏)、测试覆盖(6 个协议行为零测试、Java 侧除纯
|
||||||
|
函数外零单测、e2e 注册就绪用固定 sleep)。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. 协议实现审计
|
||||||
|
|
||||||
|
### 1.1 线级一致性:逐字节核对,全部一致 ✓
|
||||||
|
|
||||||
|
握手布局、Rekey 帧(`magic‖randLen‖rand‖ts‖flags‖window`,REKEY=rand‖ts 不含
|
||||||
|
magic)、SessionReady、密钥派生(`SHA3-256(PK‖0x01/0x02)`、零 nonce、counter 0)、
|
||||||
|
控制消息、mux 帧(含 RESUME/RESUME_ACK 字段顺序)、流控语义(只 DATA 计窗口、WND
|
||||||
|
半窗批量、32KiB chunk)、resume 三偏移量语义、全部常量、配置默认值与钳制——两端实现
|
||||||
|
与 PROTOCOL.md 全部一致,无任何严重不一致。
|
||||||
|
|
||||||
|
**velocity 模块已经官方源码确证无误**(`client/velocity.go` 与 PaperMC/Velocity
|
||||||
|
`dev/4.0.0` 的 `PlayerDataForwarding.java` 比对):
|
||||||
|
|
||||||
|
- payload 布局 `VarInt(version) ‖ String(ip) ‖ UUID[16] ‖ String(name) ‖ VarInt(properties)`
|
||||||
|
与官方 `writeVarInt + writeString + writeUuid + writeString + writeProperties` 完全一致;
|
||||||
|
- 版本协商(1.19.3+ 时 `requested≥4` 发 4、否则发 1)与官方 `findForwardingVersion`
|
||||||
|
逐分支一致,包括 v4 lazy-session 不带 key 段、v2/v3 只在 1.19–1.19.2 且客户端有
|
||||||
|
key 时使用(redapricot 无 key 故正确回落 v1)。
|
||||||
|
|
||||||
|
### 1.2 轻微偏差(不影响互通,但建议修)
|
||||||
|
|
||||||
|
| # | 问题 | 位置 | 说明 |
|
||||||
|
|---|------|------|------|
|
||||||
|
| P1 | **pattern 注册前被归一化,违反 §5 "verbatim" 约定** | `client.go:49-52`、`config.go:314-321` | 客户端注册的是 `NormalizeAddress(pattern)` 后的字符串。hub 端确实原样存储,但大小写敏感的 regex 会被破坏:以 `\.` 结尾的 pattern 经 `TrimRight(".")` 变成孤立 `\` 导致编译失败;`\Q…\E` 被小写化为 `\q…\e` 同样编译失败。应原样注册、仅匹配时归一化 |
|
||||||
|
| P2 | `streamWindowBytes: 0` 语义两端不同 | `Config.java:36-37` vs `config.go:71-75` | Go 视为"未设置→262144",Java 钳到最小 32768。规范未定义 0,属规范留白 |
|
||||||
|
| P3 | Intent 18 规范与实现不符 | `PROTOCOL.md:73-75` vs `HubConnection.java:96-98` | 规范称"回复 status line 后关闭",实现只 log+close |
|
||||||
|
| P4 | PSK address 比较宽松 | `HubConnection.java:113` | 规范要求小写精确匹配,实现用 `equalsIgnoreCase` |
|
||||||
|
| P5 | Go 侧缺镜像常量 | `client/config.go` | `INTENT_RESERVED`、`REGISTER_OK/ERR` 未定义,违反 CLAUDE.md「常量镜像」不变式 |
|
||||||
|
| P6 | RST 原因码半实现 | `worker.go:430-432`;`WorkerConn.java:55` | 客户端 `sendRst` 从不带原因字节(`RST_DIAL_FAILED/FLOW_CONTROL/RESUME_ABANDONED` 是死常量);hub 收到 RST 也不解析原因字节。协议标称"both directions"的对称性未实现 |
|
||||||
|
| P7 | 规范文档错误 | `PROTOCOL.md:214` | §5 表 SessionReady 标 "(none)",与 §4(flags+window+grace)矛盾;§7.5 未提 `RST(ALREADY_BOUND)` 分支 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. 设计审计
|
||||||
|
|
||||||
|
**做得对且值得保留的设计**(均已核实,非泛泛而谈):
|
||||||
|
|
||||||
|
1. **池不在持锁时 dial**——`Allocate` 与 `maybeGrowLocked` 都把 dial 放到释放锁之后
|
||||||
|
/后台 goroutine,注释与代码一致;空池时单 dialer + `cond.Wait` + `dialGen` 代际
|
||||||
|
账本正确处理失败信号。
|
||||||
|
2. **流经 hub 路由而非闭包捕获 conn**——`Hub.onPlayerData` 按 `st.worker` 转发,
|
||||||
|
reattach 无需重装 handler,规避了死 conn 静默写缺陷。
|
||||||
|
3. **resume 账本**——重放点取 peer 的 accepted、窗口按 delivered 重述、丢弃自身在途
|
||||||
|
credit、`Delivered ≤ Accepted` 隐含不变式,两端对称且都有注释钉住;`UnackedBytes`
|
||||||
|
的摊还 O(1) advance、`from()` 越界返回 nil 触发终局 teardown(不静默截断)都正确。
|
||||||
|
4. **hub 侧 parked 上限**——`maxParkedStreams/maxParkedBytes` 以插入序 LinkedHashMap
|
||||||
|
淘汰最旧,`parkedBytes()` 计数与 `removeStream` 的扣减逻辑经核对自洽(写法绕,见 §4.6)。
|
||||||
|
5. **grace 协商而非配置假设**——hub 在 SessionReady 公布 `resumeGraceMs`,客户端钳制
|
||||||
|
在它之下,"客户端必须先放弃"从运维约定变成协议约束。
|
||||||
|
6. **shaper**——token bucket + start-time fair queue,vclock 不因 size 前移是刻意的;
|
||||||
|
只整形 DATA、控制帧永不延迟;关停不卡流量;`NewShaper` 返回 nil 的短路设计让限速
|
||||||
|
关闭时热路径零分支。
|
||||||
|
|
||||||
|
**设计层面的顾虑**(非错误,属取舍):
|
||||||
|
|
||||||
|
- **单事件循环 hub**:一块网卡 + 1 MiB 帧 + 32KiB chunk,单核吞吐上限明确(文档已
|
||||||
|
承认)。但 **ChaCha20 的 `Cipher.update` 每次帧调用都过 JCE**,帧头 VarInt 解析 +
|
||||||
|
Buffer 切片每帧多次分配,实际吞吐可能在数百 Mbps 量级——建议做一次基准确认。
|
||||||
|
- **TCP-level HOL** 是 mux-over-TCP 固有(文档已承认),池是缓解而非解决。
|
||||||
|
- **无 AEAD**(文档已承认),ChaCha20 无完整性保护意味着主动攻击者可以翻转密文。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. 逻辑漏洞(按严重度排序)
|
||||||
|
|
||||||
|
### 🔴 C1. 双 `resumeLoop` 竞态 → 静默数据损坏(客户端,最严重)
|
||||||
|
|
||||||
|
触发链(`client/resume.go`、`client/worker.go`):
|
||||||
|
|
||||||
|
1. `tryResume` 在新建 conn B 上 `registerStream` 并发 RESUME;
|
||||||
|
2. `completeResume` 重放中途 `wc.sendData` 失败(B 刚注册即死,`resume.go:271-273`)
|
||||||
|
——注意 `s.parked = false` 在重放**之前**已复位(`resume.go:253`);
|
||||||
|
3. B 的 readLoop 退出 → 对 B 上每条流调 `park()`(`worker.go:401-408`)→
|
||||||
|
`already := s.parked` 为 **false** → 再起**第二个** `resumeLoop`(`resume.go:80`);
|
||||||
|
4. 旧 loop 仍在重试(该错误非终局,退避 500ms 后继续)。两个 loop 并发 `tryResume`:
|
||||||
|
各自 `registerStream` 到不同 conn、各自把 `s.resumeWait` 覆盖成自己的 channel、
|
||||||
|
发两个并发 RESUME。
|
||||||
|
|
||||||
|
后果取决于时序,两种都是坏的:RESUME_ACK 被"非赢家"loop 消费 → leg 绑定到 hub 不
|
||||||
|
认识的 (conn, sid),上游(dest→player)数据被 hub 静默丢弃——**玩家 mute,无任何
|
||||||
|
日志**;或 RST 到达时 `resumeWait` 为 nil → 误 teardown 一个 hub 已恢复的流。
|
||||||
|
|
||||||
|
**修复**:`completeResume` 失败时保持 `parked=true`(不让第二个 loop 启动),让
|
||||||
|
`park()` 的 `already` 分支兜底。
|
||||||
|
|
||||||
|
### 🔴 C2. 半开连接 + `RST(ALREADY_BOUND)` → 流永久悬挂(客户端)
|
||||||
|
|
||||||
|
场景:客户端心跳超时(60s)判定 worker conn 死亡并关闭,但 hub 侧该 conn 仍在
|
||||||
|
90s idle 窗口内(`WorkerConn.java` 侧流未 park)→ 客户端新 conn 发 RESUME → hub
|
||||||
|
`takeParked` 为 null、`streamByCid` 非空 → `RST(ALREADY_BOUND)`
|
||||||
|
(`WorkerConn.java:207-210`)→ 客户端 `onRst` 映射为 `errResumeRaced` → `resumeLoop`
|
||||||
|
**无条件退出**(`resume.go:104-108`),不 teardown 不重试。此后该流 `parked=true`
|
||||||
|
永远挂着、destination socket 永久泄漏,直到进程退出。
|
||||||
|
|
||||||
|
**修复**:`errResumeRaced` 改为可重试(C1 修复后不再有真并发 loop,ALREADY_BOUND 只
|
||||||
|
意味着 hub 旧 conn 尚存),grace 耗尽后正常 teardown。
|
||||||
|
|
||||||
|
### 🟠 C3. `Close()` 不干净 + `dialSession` 无视 ctx(客户端)
|
||||||
|
|
||||||
|
- `Close()`(`client.go:413-421`)无 shutdown 标志。关闭 pool conns 后,readLoop 把流
|
||||||
|
park → `resumeLoop` 通过 `pool.Allocate()`(空池)**同步拨新 TCP 连接**继续重挂,
|
||||||
|
最长持续一个 grace(15s);
|
||||||
|
- `dialSession`(`client.go:100-210`)只用 `HandshakeTimeout` 硬 deadline,不读 ctx。
|
||||||
|
`Close()` 与在途重连/拨号并发时,新连接建立后无人关闭——`serveControl` 永久阻塞在
|
||||||
|
`ReadFrame` 上,goroutine + socket 泄漏。
|
||||||
|
|
||||||
|
**修复**:`atomic.Bool closing`;`park` 时已 closing 直接 teardown;`dialSession` 用
|
||||||
|
`net.Dialer.DialContext` + `context.AfterFunc` 在 ctx 取消时关连接。
|
||||||
|
|
||||||
|
### 🟠 C4. 手建 Config `PingIntervalMs<=0` → `time.NewTicker(0)` panic(客户端)
|
||||||
|
|
||||||
|
`New()`(`client.go:38-68`)只钳 window、解析带宽,**不验证 PingIntervalMs**(`LoadConfig`
|
||||||
|
才钳到 ≥1000,`config.go:292-298`)。注释明确说 Config 可被直接构造(测试用),此时
|
||||||
|
`time.NewTicker(0)` 直接崩进程(`client.go:342`、`worker.go:247`)。
|
||||||
|
|
||||||
|
**修复**:`New()` 里 clamp 或显式报错。
|
||||||
|
|
||||||
|
### 🟡 服务端:无同等严重的逻辑漏洞
|
||||||
|
|
||||||
|
逐一复核了 `Hub.removeSession/expireOrphans/replayAwaiting`(orphan 语义、deadline
|
||||||
|
切换、`rearm` 的 timer 与 `cancelTimer`)、`WorkerConn.handleResume`(含 replay 不
|
||||||
|
重复计窗、`st.credited=0`、CID 重铸)、`EnforceParkedCaps`(`parkedBytes` 先减后加再
|
||||||
|
经 `removeStream` 统一扣减——自洽但绕)、`EncryptedFrames.pump`(明文长度前缀/加密
|
||||||
|
负载/rekey 切换在帧边界无歧义)。**未发现逻辑错误**。健壮性/一致性缺口:
|
||||||
|
|
||||||
|
- **hub 不校验客户端发送窗口**(`WorkerConn.handleData` 只信任客户端)。协议 §7.3 明确
|
||||||
|
允许对超窗 RST,但 hub 不查——持 PSK 的恶意/损坏客户端可让 hub 端每流缓冲区无界
|
||||||
|
增长。建议按协议补校验。
|
||||||
|
- **`ProtoWriter.u8/u16` 静默截断**(`ProtoWriter.java:11,16-19`):越界值截断为
|
||||||
|
"合法外观"的错误字节,无日志;与 reader 侧严格校验不对称。
|
||||||
|
- **停机无优雅 drain**:`Main.java:40` 的 shutdown hook 不等待 `vertx.close()` 完成
|
||||||
|
(JVM halt 掐断异步关闭);无协议层下线通知;挂起玩家/流被硬切。
|
||||||
|
- **Vert.x 5 日志路由存疑**(中高置信):`Main.java:16` 设的是 Vert.x 4 的
|
||||||
|
`vertx.logger-delegate-factory-class-name`,Vert.x 5 核心日志已走 SLF4J,且 classpath
|
||||||
|
上无 slf4j-api/log4j-slf4j2-impl。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. 错误处理缺陷
|
||||||
|
|
||||||
|
| # | 问题 | 位置 | 后果 |
|
||||||
|
|---|------|------|------|
|
||||||
|
| E1 | **`registerAll` 半失败静默** | `client.go:254-263` | Register 写失败只 log 并 return,`connectControl` 仍成功、`Start` 仍返回成功,但 hub 上一个 pattern 都没注册——玩家全被拒,且日志误导 |
|
||||||
|
| E2 | **RegisterAck status=1 不处理** | `client.go:314-318` | 非法 regex 只打日志,mapping 保留,之后永远收不到 ControlRequest |
|
||||||
|
| E3 | **WND/FIN/RST 写失败全吞** | `worker.go:429-435`、`worker.go:373` | WND 丢失 → hub 停发该流 → 玩家卡死,全程无日志 |
|
||||||
|
| E4 | **解析错误大量 `_ =`** | `client.go:325-327`、`worker.go:364-368` | 坏帧静默吞掉(hub 可信,可接受) |
|
||||||
|
| E5 | **`New()` 带宽解析失败静默关限速** | `client.go:58-59` | 运维以为限速生效,实际已关闭 |
|
||||||
|
| E6 | **`shaperStall` 归因污染** | `worker.go:637-648` | `Acquire`→`emit`(含最长 30s 的 socket 写)之间时间全记入"带宽上限";hub 停读的拥堵被误报成限速,与 stats 想区分的三病因矛盾。修法:elapsed 在 `Acquire` 返回后立即取 |
|
||||||
|
| E7 | **日志不可关联** | 多处 `stream %d` | sid 是 per-conn 的且 reattach 后会变,不带 conn id,排障无法把日志对到同一条流 |
|
||||||
|
| E8 | **RST 原因映射过宽** | `resume.go:309-312` | `RST_FLOW_CONTROL` 这类协议违规终局被当可重试,徒增延迟 |
|
||||||
|
| E9 | **resume 尝试越过 deadline 最多 10s** | `resume.go:152-168` | `time.After(ResumeAckTimeout)` 不随 deadline 缩短,玩家侧最坏失败延迟被放大到 grace+10s |
|
||||||
|
| E10 | **resumeGrace 与 hubGrace 取 min 后丢失下限** | `resume.go:38-44` | 误配 500ms grace 的 hub 让客户端预算 < 一次拨号所需,仅可观测性问题 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. 复杂度分析
|
||||||
|
|
||||||
|
| 热点 | 位置 | 分析 |
|
||||||
|
|------|------|------|
|
||||||
|
| `Stream` 类型 | `worker.go:451-518` | 25+ 字段、4 种同步原语(`s.mu`/`sendMu`/原子量/chan),resume 状态(parked/resumeWait/leg/cid/三 offset)与流量状态混居。**C1 正是"parked 复位"与"loop 存活"两状态未拆开导致的** |
|
||||||
|
| `completeResume` | `resume.go:203-288` | 全项目最复杂单函数:锁序(sendMu→s.mu)、三偏移换算、重放、窗口重述、欠账 FIN。正确但难审 |
|
||||||
|
| `WorkerPool.Allocate` | `worker.go:54-92` | cond 等待 + `dialGen`/`dialErr` 代际账本,失败路径无单测 |
|
||||||
|
| 接收方向账本 | `worker.go` 多处 | `q`/`qBytes`/`acceptedOffset`/`deliveredOffset`/`consumed` 由 writeLoop/deliverFromHub/credit/completeResume 四处维护 |
|
||||||
|
| shaper `headLocked` | `shaper.go:241-252` | 每 grant 线性扫描 O(n²);n 为流数时无感,但每 DATA 块一次 `make(chan)` 分配是热点 |
|
||||||
|
| `wire.Reader.VarInt` | `mc.go:55-61` | 每次 `bytes.NewReader` 分配,velocity/帧解析热路径高频调用 |
|
||||||
|
| velocity 状态机 | `velocity.go` | 两方向缓冲 + 5 个布尔,逻辑正确但可合并为三值枚举 |
|
||||||
|
| 测试 harness | `e2e/*` | 三层 helper 金字塔 + 3 个文件各自复制"hub+relay+dest+client"接线 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. 测试覆盖审计
|
||||||
|
|
||||||
|
**覆盖良好的**:黑盒路径恢复、控制断线三种路径、resume 字节精确/并发/禁用/grace 到期、
|
||||||
|
带宽整形与公平、慢流隔离、velocity 单测 13 场景 + e2e、PROXY v2、池广度优先、错误 PSK
|
||||||
|
拒绝。e2e 拉起**真实 Java hub 子进程**,架构扎实。
|
||||||
|
|
||||||
|
**零测试的协议行为**(6 个,e2e + 单测均未断言):
|
||||||
|
|
||||||
|
1. **RST 原因码**——协议定义了 6 个,两端只实际发射 UNKNOWN_STREAM/ALREADY_BOUND;
|
||||||
|
ALREADY_BOUND 竞争路径(C1/C2 的暴露面)无任何测试;
|
||||||
|
2. **WND 窗口违规**——hub 不校验客户端窗口、客户端超窗检测无测试;
|
||||||
|
3. **pending 超时**——"已匹配但 SYN 永不来的玩家被关"无测试;
|
||||||
|
4. **hub idle watchdog**——`sessionIdleTimeoutMs` 无测试;
|
||||||
|
5. **CID 单次使用**——二次 SYN、resume 后旧 CID 失效无测试;
|
||||||
|
6. **1 MiB 帧上限**——两侧都无超长帧测试。
|
||||||
|
|
||||||
|
**结构性缺口**:
|
||||||
|
|
||||||
|
- **Java 侧只有 1 个测试文件**(`CryptoCodecTest.java`,9 个纯函数用例)。hub 状态机、
|
||||||
|
`HubConnection` 握手/rekey 校验、`EncryptedFrames`、`WorkerConn` 流控/resume 零单测。
|
||||||
|
根因之一:`testHub()` 传 `vertx=null`(`CryptoCodecTest.java:79-83`),`Hub.rearm` 用
|
||||||
|
`vertx.setTimer` 会 NPE——**测试缝本身封死了 timer 路径**。
|
||||||
|
- **跨语言 pinning 只有 SHA3-224 一处**;ChaCha20 字节一致性无 Java 侧单测、无
|
||||||
|
RFC 8439 KAT,`deriveKey` 无绝对向量,全靠 e2e 兜底。
|
||||||
|
- **e2e 注册就绪用固定 200ms sleep**(`e2e_test.go:47`)——全 e2e 最大的 flaky 源。
|
||||||
|
`Start` 只保证 Register 帧**写出**,不等待 RegisterAck。
|
||||||
|
- **flaky 前五**:`TestShaperSharesFairlyBetweenStreams`(35% 容差)、
|
||||||
|
`TestBlackholedPathRecovers`(45s 轮询)、`TestResumePreservesByteStream`/
|
||||||
|
`TestResumeWithConcurrentStreams`、`TestControlOutageHangsArrivingPlayer`。
|
||||||
|
- **velocity 无真实 Paper golden vector**——测试只证明自洽(本次审计已确认字节布局与
|
||||||
|
官方源码一致,建议补 golden 测试防回归)。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. 重构建议(按性价比排序)
|
||||||
|
|
||||||
|
**P0 — 正确性(先做)**
|
||||||
|
1. **修 C1 双 resumeLoop 竞态**:`completeResume` 失败保持 `parked=true`;
|
||||||
|
`deliverResume` 校验 loop 归属。补 ALREADY_BOUND 竞争路径的 e2e。
|
||||||
|
2. **修 C2 半开悬挂**:`errResumeRaced` 改为在 grace 剩余内重试。
|
||||||
|
3. **修 C3/C4**:`Close()` 加 shutdown 标志、`dialSession` 尊重 ctx、`New()` 校验
|
||||||
|
`PingIntervalMs`。
|
||||||
|
|
||||||
|
**P1 — 健壮性与可观测性**
|
||||||
|
4. 客户端 `registerAll` 失败要传导;RegisterAck status=1 至少移除/标记该 mapping。
|
||||||
|
5. hub 补客户端发送窗口校验(§7.3 允许的 RST);`ProtoWriter` 加 `u8/u16` 范围校验。
|
||||||
|
6. 修正 `shaperStall` 归因(elapsed 提前取);流日志统一带 `connID/sid`。
|
||||||
|
7. pattern 原样注册(P1);`streamWindowBytes:0` 语义统一并写进规范(P2)。
|
||||||
|
|
||||||
|
**P2 — 测试**
|
||||||
|
8. Java 侧补:ChaCha20 RFC 8439 KAT + 分块跨块一致性、`EncryptedFrames` rekey、
|
||||||
|
`Hub` 状态机(timer 可注入)。
|
||||||
|
9. 客户端加 ready 信号(RegisterAck 挂钩到可等待 channel),替换 200ms sleep;补
|
||||||
|
6 个零覆盖协议行为的测试。
|
||||||
|
10. velocity 补官方字节 golden vector。
|
||||||
|
|
||||||
|
**P3 — 结构**
|
||||||
|
11. `completeResume` 拆成小函数 + 状态机图注释;接收方向账本封装为 `recvLedger`。
|
||||||
|
12. harness 引入单一 `newTunnelHarness(t, opts…)` 组装点。
|
||||||
|
13. 热路径去分配:`wire.Reader.VarInt` 改游标读取;shaper chan 池化(低优先级)。
|
||||||
|
14. `Main` 的 shutdown hook 等待 `vertx.close()` 完成;核实 Vert.x 5 日志绑定。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. 结论
|
||||||
|
|
||||||
|
协议实现是**干净且两端一致的**——这是本审计最重要的结论,逐字节核对未发现任何线级
|
||||||
|
不兼容。设计文档质量上乘,注释大量引用"它防的是什么故障",可维护性极好。
|
||||||
|
|
||||||
|
需要优先处理的是客户端 resume 路径的两个并发缺陷(C1 静默数据损坏、C2 悬挂泄漏)
|
||||||
|
和关闭语义(C3),其次是测试缺口。这两类问题恰好互相印证:C1 和 C2 都发生在
|
||||||
|
"连接死亡/重建"的交叉时序里,而这类时序恰恰是当前 e2e 覆盖最薄的地方。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 附:修复状态跟踪
|
||||||
|
|
||||||
|
| 编号 | 内容 | 状态 |
|
||||||
|
|------|------|------|
|
||||||
|
| C1 | 双 resumeLoop 竞态 | ✅ 已修(`completeResume` 重放失败时重新置 `parked=true`,统计计数移至重放成功之后) |
|
||||||
|
| C2 | 半开 + RST(ALREADY_BOUND) 悬挂 | ✅ 已修(`resumeLoop` 对 `errResumeRaced` 改为可重试,由 grace 截止时间兜底) |
|
||||||
|
| C3 | Close()/dialSession ctx | ✅ 已修(`Client.closing` 标志 + 内部 ctx/cancel;`dialSession(ctx)` 用 `DialContext` + `AfterFunc` 关闭;`WorkerPool.closed` 标志门控 Allocate/后台拨号;serveControl 重连被 closing 门控) |
|
||||||
|
| C4 | NewTicker(0) panic | ✅ 已修(`pingInterval()` 单点钳制,新增 `DefaultPingIntervalMs`) |
|
||||||
|
| E6 | shaperStall 归因污染 | ✅ 已修(`elapsed()` 移至 `Acquire` 返回后、socket 写之前采样) |
|
||||||
|
| E7 | 流日志缺 conn id | ✅ 已修(`leg.String()` = `conn%d/sid%d`,替换全部 5 处流日志) |
|
||||||
|
| P3 | Intent 18 status line | ✅ 已修(hub 回复 Minecraft status 包 `[Len][0x00][JSON]` 后关闭;已用真实 codec 探针验证) |
|
||||||
|
| P4 | PSK address 严格小写 | ✅ 已修(`equalsIgnoreCase` → 严格 `equals`;大写变体探针验证被拒) |
|
||||||
|
| P5 | Go 镜像常量 | ✅ 已修(`IntentReserved=18`、`RegisterOk=0x00`、`RegisterErrPattern=0x01`,RegisterAck 处理用命名常量) |
|
||||||
|
| P7 | PROTOCOL.md §5 SessionReady 表 | ✅ 已修(补充 `Flags/RecvWindow/ResumeGraceMs` 字段;§2 补充 Intent 18 status line 格式) |
|
||||||
|
|
||||||
|
验证:`go test ./client/... -count=1`、`go test -race ./client/...`、`gradle -p server test`、
|
||||||
|
`go test ./e2e/... -count=1`(含 resume/blackhole 套件)全部通过;Intent 18 与严格小写
|
||||||
|
PSK 已用真实客户端 codec 对真实 hub 探针验证。
|
||||||
@@ -0,0 +1,166 @@
|
|||||||
|
# CLAUDE.md
|
||||||
|
|
||||||
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||||
|
|
||||||
|
## What this is
|
||||||
|
|
||||||
|
redapricot is a central-hub P2P tunnel that speaks an *extension* of the Minecraft Java
|
||||||
|
Edition protocol. A **hub** (Java/Vert.x, one public TCP port) forwards players to a
|
||||||
|
**client** (Go) sitting next to a real Minecraft server behind NAT. Everything — players
|
||||||
|
and clients alike — arrives on the same port; the hub tells them apart by the handshake
|
||||||
|
`Intent` field (`17` = redapricot, `18` reserved, anything else = player).
|
||||||
|
|
||||||
|
Two implementations of one wire protocol live in this repo, so **most changes are
|
||||||
|
cross-language**. `PROTOCOL.md` is the normative spec; `docs/architecture.md` explains the
|
||||||
|
design rationale.
|
||||||
|
|
||||||
|
## Build
|
||||||
|
|
||||||
|
`scripts/build.sh` builds both sides (hub via `installDist`, client into `bin/`). It
|
||||||
|
auto-discovers a JDK/Gradle under `~/.sdkman/candidates/`.
|
||||||
|
|
||||||
|
**Gradle needs an explicit `JAVA_HOME` in this environment** — it is not inherited:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
JAVA_HOME=$HOME/.sdkman/candidates/java/current gradle -p server installDist
|
||||||
|
JAVA_HOME=$HOME/.sdkman/candidates/java/current gradle -p server test
|
||||||
|
JAVA_HOME=$HOME/.sdkman/candidates/java/current gradle -p server shadowJar # fat jar
|
||||||
|
go build ./... # Go side
|
||||||
|
```
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./scripts/e2e.sh # build both, Go unit tests, then e2e
|
||||||
|
go test ./client/... -count=1 # Go unit tests (codec, crypto, pool, shaper, velocity)
|
||||||
|
go test ./e2e/... -count=1 -v -timeout 300s
|
||||||
|
go test ./e2e/ -run TestRoundTrip -v # one e2e test
|
||||||
|
go test ./client/ -run TestShaperEnforcesRate -v
|
||||||
|
```
|
||||||
|
|
||||||
|
The e2e suite spawns the **real Java hub as a subprocess** from
|
||||||
|
`server/build/install/redapricot-server` (via `-cp lib/* io.icybear.redapricot.Main`) and
|
||||||
|
runs the Go client in-process against a mock destination. It fails fast if the hub has not
|
||||||
|
been installed, so **rebuild the hub after touching Java** or e2e silently tests stale
|
||||||
|
bytecode. `resolveJava` skips the suite when no JDK is found (`JAVA_HOME`, SDKMAN, `PATH`).
|
||||||
|
|
||||||
|
Timing-sensitive suites (`e2e/bandwidth_test.go`, `client/shaper_test.go`,
|
||||||
|
`e2e/slowstream_test.go`, `e2e/resilience_test.go`, `e2e/resume_test.go`) assert on rates
|
||||||
|
and recovery deadlines; give them slack rather than tightening thresholds. The resume tests
|
||||||
|
are load-sensitive — a busy machine (a gradle daemon in the background) shifts the races
|
||||||
|
they exercise, so run them a few times rather than trusting a single pass.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
### Connection kinds (all on one port)
|
||||||
|
|
||||||
|
| First handshake | Becomes |
|
||||||
|
|---|---|
|
||||||
|
| `Intent 17`, rekey magic `0x01` | **control session** — pattern registration, `ControlRequest`, Ping/Pong |
|
||||||
|
| `Intent 17`, rekey magic `0x02` | **worker conn** — one player, 1:1 |
|
||||||
|
| any other intent | **player** — hostname regex-matched, then tunneled |
|
||||||
|
|
||||||
|
Session establishment (both kinds): plaintext handshake whose `Server Address` is
|
||||||
|
`hex(SHA3-224(PSK))` → one frame encrypted with PSK-derived keys carrying
|
||||||
|
`magic ‖ rand ‖ timestamp ‖ flags ‖ window` → **both sides switch ciphers** to keys derived
|
||||||
|
from `rand‖ts` → hub replies `SessionReady` echoing accepted flags and its window.
|
||||||
|
|
||||||
|
### Player handoff
|
||||||
|
|
||||||
|
The hub pauses and buffers a matched player socket, mints a random 16-byte **CID**, and
|
||||||
|
sends `ControlRequest(CID, matchedPattern, ip, port)` down the control session. The client
|
||||||
|
dials a dedicated worker conn, `SYN`s it with that CID, dials the destination, and the hub
|
||||||
|
binds the pending player to that conn. The CID's secrecy (it only ever travels encrypted
|
||||||
|
over the control session) is what authorizes the takeover — there is no other client
|
||||||
|
identity check. The hub echoes **the matched pattern string, not the player's hostname**,
|
||||||
|
so the client can look it up directly in its route table; the buffered handshake is
|
||||||
|
forwarded verbatim so the backend sees the original hostname.
|
||||||
|
|
||||||
|
### Code map
|
||||||
|
|
||||||
|
- `server/src/main/java/io/icybear/redapricot/` — `HubConnection` (per-socket state machine:
|
||||||
|
handshake parse → dispatch → rekey → control/worker/player), `Hub` (pattern registry, CID
|
||||||
|
table, pending players), `ControlSession`, `WorkerConn` (1:1 tunnel + flow-control state),
|
||||||
|
`net/EncryptedFrames`, `crypto/Crypto`, `util/` (VarInt, ProtoReader/Writer, Hex).
|
||||||
|
- `client/` — `client.go` (control session, reconnect, dispatch), `worker.go` (1:1
|
||||||
|
dial, `WorkerConn`, `Stream` and its two goroutines), `shaper.go` (egress rate cap),
|
||||||
|
`velocity.go` (Velocity modern-forwarding interception), `proxyproto.go` (HAProxy v2),
|
||||||
|
`config.go` (config + all protocol constants), `wire/` (VarInt/MC codec, SHA3+ChaCha20,
|
||||||
|
`FramedConn`).
|
||||||
|
- `cmd/redapricot-client/` — binary entrypoint; `e2e/` — integration harness.
|
||||||
|
|
||||||
|
### Threading
|
||||||
|
|
||||||
|
- **Hub:** a single Vert.x verticle instance, so the pattern registry, CID table, and all
|
||||||
|
per-connection state are touched by one event loop and need no locking. Never introduce a
|
||||||
|
blocking call there.
|
||||||
|
- **Client:** one goroutine reads each connection; `WriteFrame` is mutex-serialized. Each
|
||||||
|
tunnel has exactly two goroutines — `run` (destination → hub) and `writeLoop` (the *only*
|
||||||
|
writer to the destination, draining a queue fed by the worker readLoop). The readLoop must
|
||||||
|
never write to a destination: WND is granted only after the dest write completes, and a
|
||||||
|
stalled backend must not stall heartbeat / FIN dispatch on that conn.
|
||||||
|
|
||||||
|
## Invariants to preserve when editing
|
||||||
|
|
||||||
|
- **Constants are mirrored** in `client/config.go` and `server/.../Protocol.java`. Changing
|
||||||
|
one without the other is a silent protocol break; update `PROTOCOL.md` too.
|
||||||
|
- **The frame length prefix is plaintext, the payload is encrypted.** This is deliberate: a
|
||||||
|
reader always knows how many ciphertext bytes belong to the current frame, which makes the
|
||||||
|
rekey cipher switch unambiguous. Do not encrypt the length.
|
||||||
|
- **Per-direction keys, fixed zero nonce** (`SHA3-256(phaseKey ‖ 0x01)` c2s,
|
||||||
|
`‖ 0x02` s2c). Both directions must never share a keystream. Go `x/crypto/chacha20` and
|
||||||
|
Java JCE `ChaCha20` are byte-identical here, and unit tests on both sides pin the same
|
||||||
|
SHA3-224 vector — keep that pinning if you touch crypto.
|
||||||
|
- **Per-connection flow control is mandatory.** The hub rejects a session whose rekey lacks
|
||||||
|
`FLAG_STREAM_FC`; the client rejects a hub that does not echo it. Credit is granted back
|
||||||
|
(`WND`) only as bytes are actually written to the terminal socket, batched at half-window.
|
||||||
|
- **One worker conn carries one player.** There is no stream id. The first business frame
|
||||||
|
after `SessionReady` is `SYN` or `RESUME`; a second bind on the same conn is a protocol
|
||||||
|
violation. `PING`/`PONG` are connection-scoped frames.
|
||||||
|
- **Only DATA is shaped** by `client/shaper.go`. Delaying `FIN`, `WND`, or `PONG` would trip
|
||||||
|
the very liveness detection the heartbeat exists for. The shaper is client-local and
|
||||||
|
invisible on the wire.
|
||||||
|
- **Each player gets its own worker dial**, up to `maxTunnels` (default 256). A dial is
|
||||||
|
never performed while holding the live-set lock: session establishment is network I/O,
|
||||||
|
and one unresponsive hub must not block unrelated players.
|
||||||
|
- **The hub IP limiter is player-only.** Intent 17 (control + workers) is never
|
||||||
|
admitted through it — those sockets share the client's one address. Unmatched
|
||||||
|
player hostnames still consume a token. `0` turns each knob off.
|
||||||
|
- **Liveness is explicit everywhere:** every session heartbeats (drop after
|
||||||
|
`3 × pingIntervalMs`), every socket write is bounded, establishment has a deadline. A
|
||||||
|
silently blackholed path (NAT forgetting a flow, no FIN/RST) must recover without operator
|
||||||
|
action — `TestBlackholedPathRecovers` guards this.
|
||||||
|
- **Stream resumption is byte-exact or it is nothing** (`PROTOCOL.md §7.5`,
|
||||||
|
`client/resume.go`, `WorkerConn.handleResume`). A worker-conn drop hangs the player and
|
||||||
|
reattaches over a fresh conn. Three offsets are tracked per direction and are
|
||||||
|
*not* interchangeable: replay from the peer's **accepted** offset, restate the window from
|
||||||
|
its **delivered** offset, and never size the window from **credited** — the grants in
|
||||||
|
flight when the conn died are gone for good, and a window derived from them can be
|
||||||
|
permanently zero, which deadlocks. `TestResumePreservesByteStream` guards this.
|
||||||
|
- **The retained region needs no cap of its own.** Flow control already bounds outstanding
|
||||||
|
bytes to one window, so the retransmit buffer *is* the outstanding region. Anything that
|
||||||
|
lets a sender exceed its window silently makes it unbounded.
|
||||||
|
- **Hub-side socket handlers must route through `st.worker`, never through a captured
|
||||||
|
conn.** A lambda installed in `handleSyn` closes over `this`; after a reattach it would
|
||||||
|
write into the dead conn's transport, where sends are dropped silently and the player goes
|
||||||
|
mute with nothing logged.
|
||||||
|
- **A closed control session orphans its routes rather than deleting them**
|
||||||
|
(`PROTOCOL.md §5.2`, `Hub.removeSession`). Players arriving during the client's
|
||||||
|
reconnect are held and replayed once it re-registers, instead of being told there
|
||||||
|
is no such server. Two traps: `Hub.match` must surface the orphaned state rather
|
||||||
|
than hand back a dead `ControlSession` (`EncryptedFrames.send` drops silently on a
|
||||||
|
closed transport, so the player would hang with no request ever sent), and a held
|
||||||
|
player's deadline is the registration grace, not `pendingTimeoutMs` — whichever is
|
||||||
|
shorter fires first and closes the socket.
|
||||||
|
- **The off switches must reach the hot path, not just the wire.** `streamResume: false`
|
||||||
|
allocates no retained region and `statsIntervalMs: 0` allocates no counters, so both cost
|
||||||
|
one predictable branch. `TestResumeDisabledAllocatesNothing` guards the first.
|
||||||
|
|
||||||
|
## Conventions
|
||||||
|
|
||||||
|
- Comments here explain *why*, often citing the failure they prevent, and reference
|
||||||
|
`PROTOCOL.md §N`. Match that when adding code on either side.
|
||||||
|
- Wire-visible behaviour changes need: both implementations, `PROTOCOL.md`, an e2e test, and
|
||||||
|
usually a note in `docs/architecture.md` and the README config tables.
|
||||||
|
- Java uses Lombok (freefair plugin) and Log4j2; JUL is routed through Log4j2 both via
|
||||||
|
`applicationDefaultJvmArgs` and programmatically in `Main` for the `java -jar` path.
|
||||||
+338
-64
@@ -15,7 +15,7 @@ There are three roles:
|
|||||||
| **Player** | any | An ordinary Minecraft client connecting through the hub. |
|
| **Player** | any | An ordinary Minecraft client connecting through the hub. |
|
||||||
|
|
||||||
```
|
```
|
||||||
Player ──MC──▶ Hub(server) ══WorkerConn(mux)══▶ Client ──MC──▶ Destination
|
Player ──MC──▶ Hub(server) ══WorkerConn(1:1)══▶ Client ──MC──▶ Destination
|
||||||
▲ registers patterns / receives control requests │
|
▲ registers patterns / receives control requests │
|
||||||
└────────────── Control Session ────────────────────┘
|
└────────────── Control Session ────────────────────┘
|
||||||
```
|
```
|
||||||
@@ -74,6 +74,19 @@ The hub reads exactly one Handshake packet and dispatches on `Intent`:
|
|||||||
| `18` | Reserved for redapricot management/status. Never matched against patterns. The reference hub replies with a status line and closes. |
|
| `18` | Reserved for redapricot management/status. Never matched against patterns. The reference hub replies with a status line and closes. |
|
||||||
| anything else | **Player** connection. `ServerAddress` is matched against the registered **regex** PATTERNs (§5.1). |
|
| anything else | **Player** connection. `ServerAddress` is matched against the registered **regex** PATTERNs (§5.1). |
|
||||||
|
|
||||||
|
For `Intent == 18` the hub replies with a Minecraft status-response packet and
|
||||||
|
closes — the same shape a player receives for a status query (Intent 1), so the
|
||||||
|
port can be probed with ordinary tooling:
|
||||||
|
|
||||||
|
```
|
||||||
|
[Len: VarInt][PacketID 0x00][JSON: String]
|
||||||
|
```
|
||||||
|
|
||||||
|
The JSON is a minimal status payload, e.g.
|
||||||
|
`{"description":{"text":"redapricot hub"},"version":{"name":"redapricot","protocol":767},"players":{"max":0,"online":0}}`.
|
||||||
|
It is the one reply the hub sends in plaintext: Intent 18 never negotiates
|
||||||
|
encryption or any other redapricot state.
|
||||||
|
|
||||||
For `Intent == 17` the hub additionally requires
|
For `Intent == 17` the hub additionally requires
|
||||||
`ServerAddress == lowercase_hex(SHA3-224(PSK))` — a 56-character hex string.
|
`ServerAddress == lowercase_hex(SHA3-224(PSK))` — a 56-character hex string.
|
||||||
This is the first (cheap) proof that the peer knows the PSK. A mismatch closes
|
This is the first (cheap) proof that the peer knows the PSK. A mismatch closes
|
||||||
@@ -141,14 +154,25 @@ RandLen : VarInt # 8 ≤ RandLen ≤ 64
|
|||||||
Rand : Bytes[RandLen] # cryptographically random
|
Rand : Bytes[RandLen] # cryptographically random
|
||||||
Timestamp : I64 # client's epoch milliseconds
|
Timestamp : I64 # client's epoch milliseconds
|
||||||
Flags : VarInt # feature flags; bit 0x01 (STREAM_FC) MUST be set
|
Flags : VarInt # feature flags; bit 0x01 (STREAM_FC) MUST be set
|
||||||
RecvWindow: VarInt # client's per-stream receive window, bytes (§7.3)
|
RecvWindow: VarInt # client's per-connection receive window, bytes (§7.3)
|
||||||
```
|
```
|
||||||
|
|
||||||
`Flags` is a bitfield of features. Bit `0x01` (STREAM_FC) declares
|
`Flags` is a bitfield of features.
|
||||||
**per-stream flow control** (§7.3) and is mandatory: `RecvWindow` advertises
|
|
||||||
the client's per-stream receive window in bytes and must be positive. The hub
|
| Bit | Name | Meaning |
|
||||||
closes the connection if the flag is missing, `RecvWindow` is absent or
|
|-----|------|---------|
|
||||||
non-positive, or the fields are malformed.
|
| `0x01` | STREAM_FC | **Per-connection flow control** (§7.3). Mandatory. |
|
||||||
|
| `0x02` | WORKER_HEARTBEAT | Connection-level `PING`/`PONG` on worker conns (§7.4). Optional. |
|
||||||
|
| `0x04` | STREAM_RESUME | **Stream resumption** (§7.5): a worker-conn drop hangs the player rather than closing it. Optional. |
|
||||||
|
|
||||||
|
STREAM_FC is mandatory: `RecvWindow` advertises the client's per-connection receive
|
||||||
|
window in bytes and must be positive. The hub closes the connection if the flag
|
||||||
|
is missing, `RecvWindow` is absent or non-positive, or the fields are malformed.
|
||||||
|
|
||||||
|
Optional bits are **negotiated**: the hub echoes in `SessionReady` only those it
|
||||||
|
accepts, and the client enables a feature only when its bit comes back. A hub
|
||||||
|
that does not know WORKER_HEARTBEAT simply omits the bit and the client falls
|
||||||
|
back to TCP keepalive alone.
|
||||||
|
|
||||||
The hub:
|
The hub:
|
||||||
|
|
||||||
@@ -166,13 +190,22 @@ frame (both directions) is Phase B, counters reset to 0.
|
|||||||
The hub then sends one Phase-B frame to confirm success:
|
The hub then sends one Phase-B frame to confirm success:
|
||||||
|
|
||||||
```
|
```
|
||||||
SessionReady : payload = [ 0x00, Flags: VarInt, RecvWindow: VarInt ]
|
SessionReady : payload = [ 0x00, Flags: VarInt, RecvWindow: VarInt,
|
||||||
|
ResumeGraceMs: VarInt ] # only when STREAM_RESUME is set
|
||||||
```
|
```
|
||||||
|
|
||||||
The hub echoes the accepted flags (STREAM_FC set) followed by its own
|
The hub echoes the accepted flags (STREAM_FC set) followed by its own
|
||||||
per-stream receive window. A client must reject a SessionReady without the
|
per-connection receive window. A client must reject a SessionReady without the
|
||||||
STREAM_FC flag or without a positive window (an unsupported hub).
|
STREAM_FC flag or without a positive window (an unsupported hub).
|
||||||
|
|
||||||
|
`ResumeGraceMs` is present only when the hub accepts STREAM_RESUME, and states
|
||||||
|
how long it will hang a player waiting for that player's stream to be reattached
|
||||||
|
(§7.5). The client clamps its own retry budget below this value. Advertising it
|
||||||
|
rather than assuming matching configuration is deliberate: the client must always
|
||||||
|
give up first, and if the hub instead dropped a hung player while the client was
|
||||||
|
still reattaching, the failure would be a silent hang rather than an error. A hub
|
||||||
|
that sets the flag but omits the field is treated as not supporting resumption.
|
||||||
|
|
||||||
A hub that rejects the session simply closes the TCP connection (optionally
|
A hub that rejects the session simply closes the TCP connection (optionally
|
||||||
after a Phase-B `Error` frame, §6). After `SessionReady`:
|
after a Phase-B `Error` frame, §6). After `SessionReady`:
|
||||||
|
|
||||||
@@ -191,7 +224,7 @@ Type : u8
|
|||||||
|
|
||||||
| Type | Name | Direction | Fields |
|
| Type | Name | Direction | Fields |
|
||||||
|--------|----------------|-----------|--------|
|
|--------|----------------|-----------|--------|
|
||||||
| `0x00` | SessionReady | S → C | *(none)* — the confirmation frame from §4 |
|
| `0x00` | SessionReady | S → C | `Flags: VarInt`, `RecvWindow: VarInt`, `ResumeGraceMs: VarInt` (only when the hub accepted `STREAM_RESUME`) — the confirmation frame from §4 |
|
||||||
| `0x01` | Register | C → S | `Pattern: String` |
|
| `0x01` | Register | C → S | `Pattern: String` |
|
||||||
| `0x02` | Unregister | C → S | `Pattern: String` |
|
| `0x02` | Unregister | C → S | `Pattern: String` |
|
||||||
| `0x03` | RegisterAck | S → C | `Pattern: String`, `Status: u8` (0 = ok, 1 = invalid pattern) |
|
| `0x03` | RegisterAck | S → C | `Pattern: String`, `Status: u8` (0 = ok, 1 = invalid pattern) |
|
||||||
@@ -236,6 +269,40 @@ Because the pattern is a regex, a literal dot must be escaped (`mc\.example\.com
|
|||||||
an unescaped `.` is the regex "any character" wildcard. A pattern that fails to
|
an unescaped `.` is the regex "any character" wildcard. A pattern that fails to
|
||||||
compile is rejected at `Register` time with `RegisterAck` status `1`.
|
compile is rejected at `Register` time with `RegisterAck` status `1`.
|
||||||
|
|
||||||
|
### 5.2 Orphaned routes (control-session outage)
|
||||||
|
|
||||||
|
When a control session closes, its registrations are **not** deleted straight
|
||||||
|
away. They are marked *orphaned* and kept for `registrationGraceMs`.
|
||||||
|
|
||||||
|
This costs nothing on the wire — it is entirely hub-side behaviour — but it
|
||||||
|
closes a gap that is otherwise very visible. A client whose control session dies
|
||||||
|
reconnects with backoff, and until it re-registers the hub has no route for it,
|
||||||
|
so every player arriving in that window is told there is no such server. The
|
||||||
|
players already tunneled are unaffected, since they ride worker conns, which a
|
||||||
|
control-session close never touches.
|
||||||
|
|
||||||
|
While a route is orphaned:
|
||||||
|
|
||||||
|
* a player matching it is **held** — paused, with its handshake buffered exactly
|
||||||
|
as for a normal pending player — and no `ControlRequest` is sent, because there
|
||||||
|
is no session to send it to;
|
||||||
|
* a player that was already pending when the session closed is moved into the
|
||||||
|
same held state rather than being dropped;
|
||||||
|
* when any client registers that pattern again, the hub delivers the
|
||||||
|
`ControlRequest` it never sent and the player proceeds normally. The held
|
||||||
|
player's deadline switches from the registration grace to `pendingTimeoutMs`
|
||||||
|
at that point, since it is now waiting for a worker rather than for a route.
|
||||||
|
|
||||||
|
If the grace expires with no client having re-registered, the route and every
|
||||||
|
player held on it are dropped. `registrationGraceMs: 0` disables the mechanism
|
||||||
|
and restores the immediate-drop behaviour.
|
||||||
|
|
||||||
|
Note the hub cannot distinguish "this client is reconnecting" from "this client
|
||||||
|
is gone for good" — that is what the grace period is a bet on. It is bounded on
|
||||||
|
the client side too: the reference client retries immediately on a control-session
|
||||||
|
drop and caps its backoff at 10s, so the bet is usually settled in well under a
|
||||||
|
second.
|
||||||
|
|
||||||
## 6. Error frame (any redapricot connection)
|
## 6. Error frame (any redapricot connection)
|
||||||
|
|
||||||
At any time either side may send, then close:
|
At any time either side may send, then close:
|
||||||
@@ -247,41 +314,62 @@ Msg : String
|
|||||||
|
|
||||||
Purely informational; the receiver logs it.
|
Purely informational; the receiver logs it.
|
||||||
|
|
||||||
## 7. Worker conn & multiplexing
|
## 7. Worker conn
|
||||||
|
|
||||||
A **Worker Conn** (`Magic == 0x02`) carries player↔destination traffic for many
|
A **Worker Conn** (`Magic == 0x02`) carries player↔destination traffic for
|
||||||
players over one TCP connection using a minimal stream multiplexer. The unit of
|
**exactly one player**. The TCP connection *is* the tunnel: there is no stream
|
||||||
work is a **stream**. Stream ids are assigned by the **client** (the only side
|
id and no multiplexer. The client dials a fresh worker conn for each
|
||||||
that opens streams), unique per worker conn, starting at 1 and increasing.
|
`ControlRequest` (and for each resumption attempt).
|
||||||
|
|
||||||
Each encrypted frame on a worker conn carries one **mux frame**:
|
Each encrypted frame on a worker conn carries one **tunnel frame**:
|
||||||
|
|
||||||
```
|
```
|
||||||
FrameType : u8
|
FrameType : u8
|
||||||
StreamID : VarInt
|
|
||||||
Data : Bytes[...] # remainder of the frame payload
|
Data : Bytes[...] # remainder of the frame payload
|
||||||
```
|
```
|
||||||
|
|
||||||
| FrameType | Name | Direction | Data |
|
| FrameType | Name | Direction | Data |
|
||||||
|-----------|------|-----------|------|
|
|-----------|------|-----------|------|
|
||||||
| `0x00` | SYN | C → S | `CID: Bytes[16]` — open a stream to take over the pending player identified by CID. |
|
| `0x00` | SYN | C → S | `CID: Bytes[16]` — take over the pending player identified by CID. |
|
||||||
| `0x01` | DATA | both | raw tunneled bytes for the stream. |
|
| `0x01` | DATA | both | raw tunneled bytes. |
|
||||||
| `0x02` | FIN | both | *(empty)* — graceful close of the stream (both directions). This is the "disconnect" the hub sends when the player leaves. |
|
| `0x02` | FIN | both | *(empty)* — graceful close (both directions). This is the "disconnect" the hub sends when the player leaves. |
|
||||||
| `0x03` | RST | both | *(optional 1 byte reason)* — abnormal close (e.g. CID unknown/expired, destination dial failed). |
|
| `0x03` | RST | both | *(optional 1 byte reason)* — abnormal close (e.g. CID unknown/expired, destination dial failed). |
|
||||||
| `0x04` | WND | both | `Delta: VarInt` — flow-control credit grant (§7.3). |
|
| `0x04` | WND | both | `Delta: VarInt` — flow-control credit grant (§7.3). |
|
||||||
|
| `0x05` | PING | both | `Nonce: I64` — liveness probe (§7.4). |
|
||||||
|
| `0x06` | PONG | both | `Nonce: I64` — echoes the probe's nonce (§7.4). |
|
||||||
|
| `0x07` | RESUME | C → S | `CID: Bytes[16]`, `Accepted: I64`, `Delivered: I64` — reattach a hung player to this conn (§7.5). |
|
||||||
|
| `0x08` | RESUME_ACK | S → C | `Accepted: I64`, `Delivered: I64`, `NewCID: Bytes[16]` — the reattach succeeded (§7.5). |
|
||||||
|
|
||||||
There is no explicit SYN-ACK: success is implied by the hub forwarding the
|
The first business frame after `SessionReady` must be `SYN` or `RESUME`. A
|
||||||
buffered Handshake as the stream's first `DATA`; failure is an `RST`.
|
second bind on an already-bound conn is a protocol violation: the hub replies
|
||||||
|
`RST` and closes. There is no explicit SYN-ACK: success is implied by the hub
|
||||||
|
forwarding the buffered Handshake as the first `DATA`; failure is an `RST`.
|
||||||
|
|
||||||
### 7.1 Stream allocation (client side)
|
`RST` reason codes. The byte remains optional — a peer that predates it sends
|
||||||
|
none, and a receiver must tolerate its absence — but distinguishing the reasons
|
||||||
|
matters for resumption, where "this stream is gone" and "someone else already
|
||||||
|
took it" call for opposite responses.
|
||||||
|
|
||||||
The client keeps a pool of `1 ≤ N ≤ max_conn` worker conns (`max_conn`
|
| Code | Name | Meaning |
|
||||||
configurable, `1..8`). To place a new stream:
|
|------|------|---------|
|
||||||
|
| `0x00` | UNSPECIFIED | No reason given (also the meaning of an absent byte). |
|
||||||
|
| `0x01` | UNKNOWN_STREAM | CID unknown or expired, or the hub restarted. Terminal: stop retrying. |
|
||||||
|
| `0x02` | ALREADY_BOUND | Another reattach already claimed this stream. Do **not** tear down. |
|
||||||
|
| `0x03` | RESUME_ABANDONED | The peer gave up reattaching. |
|
||||||
|
| `0x04` | FLOW_CONTROL | The peer exceeded its advertised window. |
|
||||||
|
| `0x05` | DIAL_FAILED | The client could not reach the destination. |
|
||||||
|
|
||||||
1. Pick the worker conn with the **fewest active streams**.
|
### 7.1 Tunnel allocation (client side)
|
||||||
2. If that minimum conn is **saturated** (active streams `> 8`) **and**
|
|
||||||
`poolSize < max_conn`, dial a new worker conn and use it instead.
|
The client dials one worker conn per player, up to a configurable
|
||||||
3. Otherwise use the least-loaded conn (even if it exceeds 8 at `max_conn`).
|
`maxTunnels` cap (default 256, clamped to `[1, 4096]`). There is no pool and no
|
||||||
|
least-loaded placement: a `ControlRequest` either gets its own TCP connection or
|
||||||
|
is dropped (the hub then closes the player when `pendingTimeoutMs` fires).
|
||||||
|
|
||||||
|
A dial is never performed while holding the live-set lock: session establishment
|
||||||
|
is network I/O, and one unresponsive hub must not be able to block unrelated
|
||||||
|
players. Each dial is independent and bounded by the session-establishment
|
||||||
|
deadline (§7.4); callers are not serialized behind a single in-flight handshake.
|
||||||
|
|
||||||
### 7.2 End-to-end player flow
|
### 7.2 End-to-end player flow
|
||||||
|
|
||||||
@@ -293,45 +381,133 @@ configurable, `1..8`). To place a new stream:
|
|||||||
control session. If no SYN arrives within `pendingTimeoutMs` (default 10000)
|
control session. If no SYN arrives within `pendingTimeoutMs` (default 10000)
|
||||||
the pending entry is dropped and the player socket closed.
|
the pending entry is dropped and the player socket closed.
|
||||||
3. The client receives `ControlRequest`, looks up the destination for `Pattern`,
|
3. The client receives `ControlRequest`, looks up the destination for `Pattern`,
|
||||||
allocates a worker conn + `StreamID`, and sends `SYN(StreamID, CID)`. In
|
dials a dedicated worker conn, and sends `SYN(CID)`. In parallel it dials the
|
||||||
parallel it dials the destination and (if configured) writes a HAProxy v2
|
destination and (if configured) writes a HAProxy v2 header (§8) carrying
|
||||||
header (§8) carrying `PlayerIP:PlayerPort`.
|
`PlayerIP:PlayerPort`.
|
||||||
4. The hub matches `CID` to the pending player, binds
|
4. The hub matches `CID` to the pending player, binds
|
||||||
`(workerConn, StreamID) ↔ playerSocket`, forwards the buffered bytes as
|
`workerConn ↔ playerSocket`, forwards the buffered bytes as `DATA`, and
|
||||||
`DATA`, and resumes the player socket. Subsequent player bytes become `DATA`
|
resumes the player socket. Subsequent player bytes become `DATA` frames;
|
||||||
frames; `DATA` frames from the client are written to the player socket. If
|
`DATA` frames from the client are written to the player socket. If `CID` is
|
||||||
`CID` is unknown/expired the hub replies `RST`.
|
unknown/expired the hub replies `RST`.
|
||||||
5. When the player disconnects the hub sends `FIN` on the stream; the client
|
5. When the player disconnects the hub sends `FIN`; the client closes the
|
||||||
closes the destination. When the destination closes, the client sends `FIN`;
|
destination. When the destination closes, the client sends `FIN`; the hub
|
||||||
the hub closes the player socket. `RST` is treated the same way (hard close).
|
closes the player socket. `RST` is treated the same way (hard close).
|
||||||
|
|
||||||
Data on a worker conn is subject to that TCP connection's back-pressure for
|
Data on a worker conn is subject to that TCP connection's back-pressure. Credit
|
||||||
its **aggregate** bandwidth; *per-stream* fairness is governed by the credit
|
windows of §7.3 bound how many bytes may be in flight on that one tunnel.
|
||||||
windows of §7.3.
|
|
||||||
|
|
||||||
### 7.3 Per-stream flow control
|
### 7.3 Per-connection flow control
|
||||||
|
|
||||||
Every stream carries an independent credit window per direction:
|
Every worker conn carries an independent credit window per direction:
|
||||||
|
|
||||||
* Each side advertised its **receive window** W (bytes) at session setup. A
|
* Each side advertised its **receive window** W (bytes) at session setup. A
|
||||||
sender may have at most W un-credited DATA bytes outstanding per stream; the
|
sender may have at most W un-credited DATA bytes outstanding; the initial
|
||||||
initial budget is W, spent as DATA is sent (`Data` length only — SYN/FIN/RST
|
budget is W, spent as DATA is sent (`Data` length only — SYN/FIN/RST frames
|
||||||
frames are free) starting with the very first DATA on the stream (including
|
are free) starting with the very first DATA (including the hub's forwarded
|
||||||
the hub's forwarded handshake).
|
handshake).
|
||||||
* The receiver returns credit with `WND(Delta)` once bytes are **delivered to
|
* The receiver returns credit with `WND(Delta)` once bytes are **delivered to
|
||||||
the terminal socket** (written to the player / destination connection), not
|
the terminal socket** (written to the player / destination connection), not
|
||||||
when they are merely buffered. Receivers should batch grants (the reference
|
when they are merely buffered. Receivers should batch grants (the reference
|
||||||
implementations send one `WND` per W/2 bytes consumed).
|
implementations send one `WND` per W/2 bytes consumed).
|
||||||
* A sender whose window is exhausted pauses reading **that stream's source
|
* A sender whose window is exhausted pauses reading **that player's source
|
||||||
socket only**; the shared worker conn is never paused because of a single
|
socket only**. A receiver that observes more than W un-credited bytes may
|
||||||
stream. A receiver that observes more than W un-credited bytes on a stream
|
reset the tunnel (`RST`) as a protocol violation.
|
||||||
may reset it (`RST`) as a protocol violation.
|
|
||||||
* Senders should also cap individual DATA payloads (the reference
|
* Senders should also cap individual DATA payloads (the reference
|
||||||
implementations use 32 KiB) so one stream cannot monopolize the link for a
|
implementations use 32 KiB) so one write cannot occupy the link for a full
|
||||||
full 1-MiB frame.
|
1-MiB frame.
|
||||||
|
|
||||||
Both windows may differ (each side enforces the one its peer advertised).
|
Both windows may differ (each side enforces the one its peer advertised).
|
||||||
`Delta` must be positive; a `WND` for an unknown stream id is ignored.
|
`Delta` must be positive; a `WND` on an unbound worker conn is ignored.
|
||||||
|
|
||||||
|
### 7.4 Liveness
|
||||||
|
|
||||||
|
TCP alone cannot tell a healthy idle connection from a dead one. When a stateful
|
||||||
|
middlebox forgets an established flow — conntrack expiry, a firewall reload, a
|
||||||
|
cloud load balancer's idle timeout — it sends neither `FIN` nor `RST`. Both ends
|
||||||
|
keep a socket that will never again carry a byte, and a reader parked on it waits
|
||||||
|
forever. Without an application-level probe the client cannot notice: its worker
|
||||||
|
conn stays in the pool, the hub keeps routing players to a control session nobody
|
||||||
|
reads, and service does not return until the client process is restarted.
|
||||||
|
|
||||||
|
Every established session is therefore covered by a heartbeat:
|
||||||
|
|
||||||
|
* **Control session** — the client sends `Ping` every `pingIntervalMs` and the
|
||||||
|
hub answers `Pong`. If no `Pong` arrives for `3 × pingIntervalMs`, the client
|
||||||
|
closes the session, which triggers its normal reconnect with backoff.
|
||||||
|
* **Worker conns** — when WORKER_HEARTBEAT was negotiated, the same exchange
|
||||||
|
runs as connection-level `PING`/`PONG` frames. On timeout the client closes
|
||||||
|
the conn. The player is reset, unless STREAM_RESUME was negotiated, in which
|
||||||
|
case it is hung and reattached over a fresh conn instead (§7.5).
|
||||||
|
* **Hub side** — an established redapricot session that receives no frame for
|
||||||
|
`sessionIdleTimeoutMs` (default 90000, `0` disables) is closed. Player
|
||||||
|
connections are never subject to this.
|
||||||
|
|
||||||
|
Both ends also enable TCP keepalive, which catches the narrower case of a peer
|
||||||
|
that has become unreachable at the IP layer.
|
||||||
|
|
||||||
|
Session establishment (§4) is bounded by a single deadline covering the dial,
|
||||||
|
the `Rekey` write and the `SessionReady` read, and every frame write is bounded
|
||||||
|
too — a peer that stops reading must not be able to park a write forever.
|
||||||
|
|
||||||
|
### 7.5 Stream resumption (STREAM_RESUME)
|
||||||
|
|
||||||
|
A worker conn is only the middle leg of the player it carries. When it dies,
|
||||||
|
both terminal sockets — the player's and the destination's — are usually still
|
||||||
|
healthy, so resetting the tunnel discards working connections because a
|
||||||
|
replaceable transport failed.
|
||||||
|
|
||||||
|
With STREAM_RESUME negotiated, a worker-conn drop instead **hangs** the player:
|
||||||
|
|
||||||
|
* the hub pauses the player socket, keeps its state, and holds it for
|
||||||
|
`ResumeGraceMs` from the moment of the *first* hang (an absolute deadline, so a
|
||||||
|
flapping client cannot extend it indefinitely);
|
||||||
|
* the client keeps the destination socket open and reattaches over a fresh
|
||||||
|
worker conn by sending `RESUME` with the player's CID;
|
||||||
|
* the hub answers `RESUME_ACK`, or `RST(UNKNOWN_STREAM)` if it holds no such
|
||||||
|
player — which is also what a client gets from a hub that has restarted.
|
||||||
|
|
||||||
|
**Resumption is byte-exact, and must be.** Frames handed to a dying socket are
|
||||||
|
lost with no notification, and the frame cipher cannot be resynchronized, so each
|
||||||
|
side replays whatever the other did not receive. Splicing the stream even one
|
||||||
|
byte off corrupts the tunneled protocol.
|
||||||
|
|
||||||
|
Three offsets are tracked per direction, and they are not interchangeable:
|
||||||
|
|
||||||
|
| Offset | Meaning | Used for |
|
||||||
|
|--------|---------|----------|
|
||||||
|
| `Sent` | bytes handed to the wire | the end of the retained region |
|
||||||
|
| `Accepted` | bytes taken off the wire toward the terminal socket | **where to replay from** |
|
||||||
|
| `Delivered` | bytes actually written to the terminal socket | **how to restate the window** |
|
||||||
|
|
||||||
|
Each side retains the bytes between what the peer has credited and what it has
|
||||||
|
sent. This costs no new bound: flow control (§7.3) already caps outstanding bytes
|
||||||
|
at one window, so the retained region *is* the outstanding region.
|
||||||
|
|
||||||
|
On reattach both sides replay `[peer's Accepted, Sent)` and set
|
||||||
|
`SendWindow = W − (Sent − peer's Delivered)`, then discard their own pending
|
||||||
|
credit — the exchanged `Delivered` already carries everything those deltas would
|
||||||
|
have, so emitting both would grant the same bytes twice.
|
||||||
|
|
||||||
|
Two rules deserve emphasis, because the obvious simplifications are wrong:
|
||||||
|
|
||||||
|
* *Accepted*, not *Delivered*, is the replay point. Delivery is signalled
|
||||||
|
asynchronously on both sides and stops being reported exactly when a connection
|
||||||
|
dies; replaying from it would re-send bytes the peer already has.
|
||||||
|
* *Delivered*, not *credited*, sizes the window. Credit travels as deltas, and
|
||||||
|
the grants in flight when the connection died are gone for good. A window
|
||||||
|
derived from them is permanently short — and if a full window was outstanding
|
||||||
|
at the drop, permanently zero, which deadlocks: nothing can be sent, so no
|
||||||
|
credit can ever come back.
|
||||||
|
|
||||||
|
`RESUME_ACK` carries a freshly minted `NewCID`, which replaces the old one. A CID
|
||||||
|
therefore stays single-use even though a player may be reattached many times, so
|
||||||
|
a leaked CID grants no more than the outage in which it was observed.
|
||||||
|
|
||||||
|
Resumption is **hub-instance-affine**: a CID means nothing to a second hub behind
|
||||||
|
an L4 load balancer, which answers `RST(UNKNOWN_STREAM)` and lets the client tear
|
||||||
|
down at once. Because the grace period holds player sockets and their buffers, a
|
||||||
|
hub bounds the number of hung players and the bytes they retain, dropping the
|
||||||
|
oldest first when either cap is reached.
|
||||||
|
|
||||||
## 8. HAProxy protocol v2 (optional)
|
## 8. HAProxy protocol v2 (optional)
|
||||||
|
|
||||||
@@ -361,12 +537,60 @@ big-endian.
|
|||||||
"psk": "change-me",
|
"psk": "change-me",
|
||||||
"timestampWindowMs": 30000,
|
"timestampWindowMs": 30000,
|
||||||
"pendingTimeoutMs": 10000,
|
"pendingTimeoutMs": 10000,
|
||||||
"streamWindowBytes": 262144
|
"streamWindowBytes": 262144,
|
||||||
|
"sessionIdleTimeoutMs": 90000,
|
||||||
|
"streamResume": true,
|
||||||
|
"resumeGraceMs": 20000,
|
||||||
|
"maxParkedStreams": 256,
|
||||||
|
"statsIntervalMs": 0,
|
||||||
|
"registrationGraceMs": 15000,
|
||||||
|
"playerRatePerSec": 8,
|
||||||
|
"playerBurst": 16,
|
||||||
|
"maxPlayersPerIp": 64
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
`streamWindowBytes` (optional, default 262144, clamped to [32768, 8388608]) is
|
`streamWindowBytes` (optional, default 262144, clamped to [32768, 8388608]) is
|
||||||
the hub's advertised per-stream receive window (§7.3).
|
the hub's advertised per-connection receive window (§7.3).
|
||||||
|
|
||||||
|
`sessionIdleTimeoutMs` (optional, default 90000) closes an established control
|
||||||
|
session or worker conn that has gone silent for that long (§7.4). It must stay
|
||||||
|
comfortably above the client's `pingIntervalMs`; `0` disables the watchdog.
|
||||||
|
|
||||||
|
`streamResume` (optional, default true) offers STREAM_RESUME (§7.5). With it
|
||||||
|
false the hub never echoes the flag and behaves exactly as a hub that predates
|
||||||
|
the feature, allocating no retained regions.
|
||||||
|
|
||||||
|
`resumeGraceMs` (optional, default 20000) is how long a hung player is held, and
|
||||||
|
is advertised in `SessionReady`. It must exceed the client's own grace by at
|
||||||
|
least one dial, which is why the client clamps itself against the advertised
|
||||||
|
value rather than its own configuration.
|
||||||
|
|
||||||
|
`maxParkedStreams` (optional, default 256) and `maxParkedBytes` (default
|
||||||
|
`maxParkedStreams × 2 × streamWindowBytes`) bound what hung players may cost;
|
||||||
|
past either the oldest are dropped.
|
||||||
|
|
||||||
|
`statsIntervalMs` (optional, default 0 = off) logs a periodic line with live and
|
||||||
|
parked stream counts, retained bytes, and the pattern count.
|
||||||
|
|
||||||
|
`registrationGraceMs` (optional, default 15000) is how long a closed control
|
||||||
|
session's routes are kept as **orphaned** rather than deleted (§5.2). `0`
|
||||||
|
disables it, restoring the behaviour of dropping routes the instant a session
|
||||||
|
closes.
|
||||||
|
|
||||||
|
`playerRatePerSec` (optional, default 8) and `playerBurst` (optional, default
|
||||||
|
16) are a per-IP token bucket applied **only to player connections** (any
|
||||||
|
handshake whose `Intent` is not 17 or 18). One arrival consumes one token;
|
||||||
|
without a token the socket is closed after the handshake, before `match`, CID
|
||||||
|
minting, or pause. Unmatched hostnames still consume a token — otherwise a
|
||||||
|
hostname scan is a free flood. Intent 17 is never admitted through the
|
||||||
|
limiter: every worker conn comes from the client's one address, and limiting
|
||||||
|
those would be the hub throttling its own client. `playerRatePerSec: 0` turns
|
||||||
|
the bucket off.
|
||||||
|
|
||||||
|
`maxPlayersPerIp` (optional, default 64) caps concurrent player sockets from
|
||||||
|
one address (pending + live + parked). `0` disables the cap. Addresses are
|
||||||
|
matched exactly; IPv6 `/64` aggregation is out of scope.
|
||||||
|
|
||||||
### 9.2 Client — JSON
|
### 9.2 Client — JSON
|
||||||
|
|
||||||
@@ -374,17 +598,51 @@ the hub's advertised per-stream receive window (§7.3).
|
|||||||
{
|
{
|
||||||
"server": "127.0.0.1:25565",
|
"server": "127.0.0.1:25565",
|
||||||
"psk": "change-me",
|
"psk": "change-me",
|
||||||
"maxConn": 4,
|
"maxTunnels": 256,
|
||||||
"pingIntervalMs": 20000,
|
"pingIntervalMs": 20000,
|
||||||
"streamWindowBytes": 262144,
|
"streamWindowBytes": 262144,
|
||||||
|
"maxBandwidth": "20mbps",
|
||||||
|
"streamResume": true,
|
||||||
|
"resumeGraceMs": 15000,
|
||||||
|
"statsIntervalMs": 0,
|
||||||
"mappings": [
|
"mappings": [
|
||||||
{ "pattern": "mc\\.example\\.com", "destination": "127.0.0.1:25566", "proxyProtocol": true }
|
{ "pattern": "mc\\.example\\.com", "destination": "127.0.0.1:25566", "proxyProtocol": true }
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
`maxTunnels` (optional, default 256, clamped to [1, 4096]) is how many
|
||||||
|
concurrent 1:1 worker connections the client will hold. A `ControlRequest`
|
||||||
|
arriving at the cap is dropped. The older `maxConn` key (the mux-era pool size,
|
||||||
|
clamped 1–8) is ignored if present: treating it as a player cap would silently
|
||||||
|
limit a previously-working config to a handful of players.
|
||||||
|
|
||||||
`streamWindowBytes` (optional, default 262144, clamped to [32768, 8388608]) is
|
`streamWindowBytes` (optional, default 262144, clamped to [32768, 8388608]) is
|
||||||
the client's advertised per-stream receive window (§7.3).
|
the client's advertised per-connection receive window (§7.3).
|
||||||
|
|
||||||
|
`streamResume` (optional, default true) offers STREAM_RESUME (§7.5). With it
|
||||||
|
false the client never offers the flag, never retains a byte for retransmission,
|
||||||
|
and behaves exactly as a client that predates the feature.
|
||||||
|
|
||||||
|
`resumeGraceMs` (optional, default 15000, minimum 2000) is how long a hung stream
|
||||||
|
keeps trying to reattach, clamped below the hub's advertised grace. The default
|
||||||
|
is chosen against the *backend*, not the tunnel: a hung player stops answering
|
||||||
|
the game server's KeepAlive, and vanilla disconnects a silent client at 30s, so a
|
||||||
|
longer grace would only resume sessions the backend then kicks.
|
||||||
|
|
||||||
|
`statsIntervalMs` (optional, default 0 = off) logs a periodic diagnostics line
|
||||||
|
and a per-stream summary at close, reporting how long each stream spent blocked
|
||||||
|
on the flow-control window versus the bandwidth cap, and the heartbeat round-trip
|
||||||
|
time per conn. These distinguish a slow backend from a saturated uplink from a
|
||||||
|
bad path, which throughput alone cannot.
|
||||||
|
|
||||||
|
`maxBandwidth` (optional, default unlimited) caps the aggregate rate at which the
|
||||||
|
client sends `DATA` to the hub, shared fairly across tunnels. Accepts `"20mbps"`
|
||||||
|
(decimal bit units), `"2MB/s"` (binary byte units), or a bare number of bytes per
|
||||||
|
second; the minimum is 8192 B/s. **This is a purely local policy and has no
|
||||||
|
effect on the wire format** — a shaped client is indistinguishable from a slow
|
||||||
|
one, and the hub needs no support for it. Only `DATA` is paced; control frames
|
||||||
|
are never delayed.
|
||||||
|
|
||||||
Each `pattern` is a regular expression (§5.1) matched against the whole
|
Each `pattern` is a regular expression (§5.1) matched against the whole
|
||||||
normalized player hostname, case-insensitively. Escape literal dots (`mc\.example\.com`,
|
normalized player hostname, case-insensitively. Escape literal dots (`mc\.example\.com`,
|
||||||
@@ -392,6 +650,15 @@ which is `mc\\.example\\.com` in JSON); an unescaped `.` matches any character.
|
|||||||
Use ordinary regex to route wildcards, e.g. `.*\.example\.com` for every
|
Use ordinary regex to route wildcards, e.g. `.*\.example\.com` for every
|
||||||
subdomain or `(alpha|beta)\.mc\.net` for a fixed set.
|
subdomain or `(alpha|beta)\.mc\.net` for a fixed set.
|
||||||
|
|
||||||
|
`velocitySecret` (optional, per mapping) makes the client speak Velocity
|
||||||
|
"modern forwarding" towards that destination: during the Minecraft login phase
|
||||||
|
it swallows the backend's `velocity:player_info` Login Plugin Request and
|
||||||
|
answers with an HMAC-SHA256-signed payload carrying the player's real IP,
|
||||||
|
username and UUID (the UUID claimed in Login Start, or the offline-mode UUID
|
||||||
|
for protocols that carry none). Set it to the backend's
|
||||||
|
`proxies.velocity.secret`. This is purely client↔destination behavior — it does
|
||||||
|
not appear on the tunnel wire, and the exchange is invisible to the player.
|
||||||
|
|
||||||
## 10. Constants summary
|
## 10. Constants summary
|
||||||
|
|
||||||
| Name | Value |
|
| Name | Value |
|
||||||
@@ -408,10 +675,17 @@ subdomain or `(alpha|beta)\.mc\.net` for a fixed set.
|
|||||||
| pattern matching | case-insensitive, whole-string regex; first match wins |
|
| pattern matching | case-insensitive, whole-string regex; first match wins |
|
||||||
| CID length | 16 bytes |
|
| CID length | 16 bytes |
|
||||||
| max frame payload | 1 MiB |
|
| max frame payload | 1 MiB |
|
||||||
| saturation threshold | active streams `> 8` |
|
| worker framing | `FrameType` + payload; no stream id |
|
||||||
| max worker conns | `max_conn ∈ [1,8]` |
|
| max concurrent worker conns | `maxTunnels`, default 256, clamped `[1, 4096]` |
|
||||||
| feature flag: per-stream flow control | `0x01` |
|
| player IP rate / burst / concurrent | `8 /s`, burst `16`, `maxPlayersPerIp` `64` (Intent ∉ {17, 18} only) |
|
||||||
| stream window default / bounds | 256 KiB, clamped to [32 KiB, 8 MiB] |
|
| feature flag: per-connection flow control | `0x01` (mandatory) |
|
||||||
|
| feature flag: worker heartbeat | `0x02` (negotiated) |
|
||||||
|
| heartbeat timeout | `3 × pingIntervalMs` |
|
||||||
|
| hub session idle timeout | 90000 ms (`0` disables) |
|
||||||
|
| connection window default / bounds | 256 KiB, clamped to [32 KiB, 8 MiB] |
|
||||||
|
| feature flag: stream resumption | 0x04 (negotiated) |
|
||||||
|
| RESUME / RESUME_ACK | 0x07 / 0x08 |
|
||||||
|
| resume grace: hub / client default | 20000 ms / 15000 ms (hub value advertised) |
|
||||||
|
| retained region per tunnel per direction | bounded by the connection window |
|
||||||
| WND grant batching (reference) | one grant per window/2 consumed |
|
| WND grant batching (reference) | one grant per window/2 consumed |
|
||||||
| DATA chunk cap (reference) | 32 KiB |
|
| DATA chunk cap (reference) | 32 KiB |
|
||||||
```
|
|
||||||
|
|||||||
@@ -13,9 +13,9 @@ field, so a vanilla Minecraft client needs no modification.
|
|||||||
> made reachable from the outside.
|
> made reachable from the outside.
|
||||||
|
|
||||||
```
|
```
|
||||||
Player ──MC──▶ Hub (Java) ══ Worker Conn (mux) ══▶ Client (Go) ──MC──▶ Real MC server
|
Player ──MC──▶ Hub (Java) ══ Worker Conn (1:1) ══▶ Client (Go) ──MC──▶ Real MC server
|
||||||
vanilla client public IP encrypted, pooled behind NAT (localhost)
|
vanilla client public IP one encrypted TCP behind NAT (localhost)
|
||||||
│ ▲
|
│ per player ▲
|
||||||
└────────────── Control Session ─────────┘
|
└────────────── Control Session ─────────┘
|
||||||
(pattern registration + control requests)
|
(pattern registration + control requests)
|
||||||
```
|
```
|
||||||
@@ -40,12 +40,11 @@ one or more hostname patterns — each a **regular expression**. When a player
|
|||||||
connects to the hub with a hostname that matches a registered pattern (and any
|
connects to the hub with a hostname that matches a registered pattern (and any
|
||||||
normal `Intent`), the hub assigns a random **CID**, buffers
|
normal `Intent`), the hub assigns a random **CID**, buffers
|
||||||
the player's bytes, and asks the client (via the control session) to take over.
|
the player's bytes, and asks the client (via the control session) to take over.
|
||||||
The client picks a **worker connection** — a multiplexed, encrypted TCP link
|
The client dials a fresh **worker connection** — one encrypted TCP link per
|
||||||
that carries many players as lightweight *streams* — opens a stream for that CID,
|
player — announces the CID with `SYN`, dials the real destination (optionally
|
||||||
dials the real destination (optionally announcing the player's real IP with the
|
announcing the player's real IP with the **HAProxy v2** protocol), and bridges
|
||||||
**HAProxy v2** protocol), and bridges the two ends. Worker connections are
|
the two ends. `maxTunnels` is the only player cap; the retired `maxConn` field
|
||||||
pooled: the client uses up to `maxConn` of them and always places a new stream
|
is ignored so an old config does not silently admit only four players.
|
||||||
on the least-loaded one.
|
|
||||||
|
|
||||||
## Repository layout
|
## Repository layout
|
||||||
|
|
||||||
@@ -113,7 +112,7 @@ cp client/config.example.json client.json
|
|||||||
# {
|
# {
|
||||||
# "server": "hub.example.com:25565",
|
# "server": "hub.example.com:25565",
|
||||||
# "psk": "same-as-the-hub",
|
# "psk": "same-as-the-hub",
|
||||||
# "maxConn": 4,
|
# "maxTunnels": 256,
|
||||||
# "mappings": [
|
# "mappings": [
|
||||||
# { "pattern": "mc\\.example\\.com", "destination": "127.0.0.1:25566", "proxyProtocol": true }
|
# { "pattern": "mc\\.example\\.com", "destination": "127.0.0.1:25566", "proxyProtocol": true }
|
||||||
# ]
|
# ]
|
||||||
@@ -126,6 +125,10 @@ just add the hub's IP with that hostname), then join `mc.example.com` in
|
|||||||
Minecraft. The hub matches the hostname against the registered regex patterns
|
Minecraft. The hub matches the hostname against the registered regex patterns
|
||||||
and tunnels you to `127.0.0.1:25566` behind the client. With `proxyProtocol: true`, the real server sees your true IP
|
and tunnels you to `127.0.0.1:25566` behind the client. With `proxyProtocol: true`, the real server sees your true IP
|
||||||
(enable `proxy-protocol` / a compatible front-end on that server to consume it).
|
(enable `proxy-protocol` / a compatible front-end on that server to consume it).
|
||||||
|
For a Paper backend, setting `velocitySecret` instead is usually nicer: the
|
||||||
|
client answers the backend's Velocity modern-forwarding login query, so the
|
||||||
|
server sees your real IP, username and UUID without any front-end — configure
|
||||||
|
the backend with `proxies.velocity.enabled: true` and the same secret.
|
||||||
|
|
||||||
## Container image (client)
|
## Container image (client)
|
||||||
|
|
||||||
@@ -163,6 +166,16 @@ secrets. The base image and build flags live in `.ko.yaml`.
|
|||||||
| `psk` | *(required)* | Shared secret; must match every client. |
|
| `psk` | *(required)* | Shared secret; must match every client. |
|
||||||
| `timestampWindowMs` | `30000` | Allowed clock skew for a client's rekey timestamp. |
|
| `timestampWindowMs` | `30000` | Allowed clock skew for a client's rekey timestamp. |
|
||||||
| `pendingTimeoutMs` | `10000` | How long a matched player waits for a worker to take over. |
|
| `pendingTimeoutMs` | `10000` | How long a matched player waits for a worker to take over. |
|
||||||
|
| `sessionIdleTimeoutMs` | `90000` | Close an established control/worker session that receives no frame for this long. Must exceed the client's `pingIntervalMs`; `0` disables. Player connections are unaffected. |
|
||||||
|
| `streamWindowBytes` | `262144` | Advertised per-stream receive window, clamped to [32 KiB, 8 MiB]. |
|
||||||
|
| `streamResume` | `true` | Hang a player when its worker connection drops, so the client can reattach the stream instead of the player being disconnected. `false` restores the previous behaviour exactly and retains nothing. |
|
||||||
|
| `resumeGraceMs` | `20000` | How long a hung player is held. Advertised to clients, which clamp their own retry budget below it. Must exceed the client's grace by at least one dial. |
|
||||||
|
| `maxParkedStreams` | `256` | Cap on simultaneously hung players; `maxParkedBytes` (default `maxParkedStreams × 2 × streamWindowBytes`) caps what they retain. Past either, the oldest are dropped. |
|
||||||
|
| `statsIntervalMs` | `0` (off) | Log a periodic line with live/hung stream counts, retained bytes and pattern count. |
|
||||||
|
| `registrationGraceMs` | `15000` | Keep a closed control session's routes as *orphaned* for this long, holding players that arrive on them instead of refusing them, and replaying their requests once the client re-registers. `0` disables it. |
|
||||||
|
| `playerRatePerSec` | `8` | Per-IP token-bucket rate for **player** connections only (Intent ∉ {17, 18}). Unmatched hostnames still consume a token. `0` disables. |
|
||||||
|
| `playerBurst` | `16` | Token-bucket depth for `playerRatePerSec`. |
|
||||||
|
| `maxPlayersPerIp` | `64` | Concurrent player sockets (pending + live + parked) from one IP. `0` disables. Intent 17 is never counted. |
|
||||||
|
|
||||||
### Client (`client/config.example.json`)
|
### Client (`client/config.example.json`)
|
||||||
|
|
||||||
@@ -170,12 +183,18 @@ secrets. The base image and build flags live in `.ko.yaml`.
|
|||||||
|------------------|--------------------|---------|
|
|------------------|--------------------|---------|
|
||||||
| `server` | *(required)* | Hub `host:port`. |
|
| `server` | *(required)* | Hub `host:port`. |
|
||||||
| `psk` | *(required)* | Shared secret; must match the hub. |
|
| `psk` | *(required)* | Shared secret; must match the hub. |
|
||||||
| `maxConn` | `1` (clamped 1–8) | Max worker connections in the pool. |
|
| `maxTunnels` | `256` (clamped 1–4096) | Max concurrent player tunnels. The retired `maxConn` field is ignored. |
|
||||||
| `pingIntervalMs` | `20000` | Control-session keepalive interval. |
|
| `pingIntervalMs` | `20000` (min 1000) | Heartbeat interval for the control session and every worker conn. A session with no reply for `3×` this is dropped and re-established. |
|
||||||
|
| `maxBandwidth` | *(unlimited)* | Caps what the client uploads to the hub, summed over every player — the direction carrying the game server's output, and the one a home uplink runs out of first. `"20mbps"`, `"512kbps"`, `"2MB/s"`, or a bare number of bytes/sec. **Bit units are decimal (`20mbps` = 20,000,000 bit/s); byte units are binary (`2MB/s` = 2 MiB/s).** Set it slightly below your real upload speed — framing and TCP/IP overhead are not counted. The budget is shared fairly across players, so one person loading chunks cannot time the others out. |
|
||||||
|
| `streamWindowBytes` | `262144` | Advertised per-stream receive window, clamped to [32 KiB, 8 MiB]. |
|
||||||
|
| `streamResume` | `true` | Reattach streams over a fresh connection when a worker connection drops, instead of disconnecting those players. `false` restores the previous behaviour exactly: nothing is retained and the send path is unchanged. |
|
||||||
|
| `resumeGraceMs` | `15000` (min 2000) | How long a stream keeps trying to reattach, clamped below the hub's advertised grace. Sized against the *backend*: a hung player stops answering the game server's KeepAlive, and vanilla disconnects a silent client at 30s, so a longer grace only resumes sessions the backend then kicks. |
|
||||||
|
| `statsIntervalMs` | `0` (off) | Log a periodic diagnostics line, plus a summary per stream at close: bytes each way, how long the stream was blocked on the flow-control window versus the bandwidth cap, receive-queue high-water mark, and heartbeat round-trip time per connection. Those distinguish a slow backend from a saturated uplink from a bad path, which throughput alone cannot. |
|
||||||
| `mappings[]` | *(≥1 required)* | Route table (below). |
|
| `mappings[]` | *(≥1 required)* | Route table (below). |
|
||||||
| `mappings[].pattern` | — | Regex matched against the whole player hostname, case-insensitively. Escape dots (`mc\.example\.com`); `.` is a wildcard. |
|
| `mappings[].pattern` | — | Regex matched against the whole player hostname, case-insensitively. Escape dots (`mc\.example\.com`); `.` is a wildcard. |
|
||||||
| `mappings[].destination` | — | Real server `host:port` to forward to. |
|
| `mappings[].destination` | — | Real server `host:port` to forward to. |
|
||||||
| `mappings[].proxyProtocol` | `false` | Prepend a HAProxy v2 header carrying the player's IP. |
|
| `mappings[].proxyProtocol` | `false` | Prepend a HAProxy v2 header carrying the player's IP. |
|
||||||
|
| `mappings[].velocitySecret` | *(off)* | Answer the destination's [Velocity modern forwarding](https://docs.papermc.io/velocity/player-information-forwarding/) login query with this secret, forwarding the player's real IP, username and UUID. Match it to the backend's `proxies.velocity.secret` (Paper). The forwarded profile carries no skin properties — the tunnel performs no Mojang authentication. |
|
||||||
|
|
||||||
## Testing
|
## Testing
|
||||||
|
|
||||||
@@ -199,14 +218,23 @@ go test ./e2e/... -v
|
|||||||
|
|
||||||
The e2e suite covers: a full player round-trip with verbatim handshake
|
The e2e suite covers: a full player round-trip with verbatim handshake
|
||||||
forwarding and case-insensitive matching, regex wildcard pattern routing,
|
forwarding and case-insensitive matching, regex wildcard pattern routing,
|
||||||
multi-megabyte transfers, concurrent
|
multi-megabyte transfers, concurrent players each on their own worker
|
||||||
streams spreading across multiple worker connections, HAProxy v2 source-address
|
connection, HAProxy v2 source-address
|
||||||
propagation, player- and destination-initiated disconnect propagation, wrong-PSK
|
propagation, Velocity modern-forwarding interception (signed player-info
|
||||||
rejection, dropping of unmatched hostnames, stream isolation under a slow
|
handoff to a mock Paper backend), player- and destination-initiated disconnect
|
||||||
player and under a slow destination (no head-of-line blocking), and rejection
|
propagation, wrong-PSK rejection, dropping of unmatched hostnames, isolation
|
||||||
of pre-flow-control peers. The Go and Java crypto layers are
|
under a slow player and under a slow destination (no head-of-line blocking),
|
||||||
independently pinned to the same SHA3-224 test vector so they cannot silently
|
rejection of pre-flow-control peers, and stream resumption — a tunnel
|
||||||
drift apart.
|
hard-reset mid-transfer with the player connection held open, asserting the
|
||||||
|
byte stream neither gains nor loses a byte, across concurrent players, plus
|
||||||
|
grace expiry and the resume-disabled path, and control-outage handling — a
|
||||||
|
player arriving while the client's control session is down is held and then
|
||||||
|
served once it re-registers, with the grace-disabled and grace-expired paths
|
||||||
|
pinned too, and the per-IP player limiter — burst overflow and
|
||||||
|
`maxPlayersPerIp` drop extras before they become pending, Intent 17 is never
|
||||||
|
counted, and both knobs at `0` restore the unlimited path. The Go and Java
|
||||||
|
crypto layers are independently pinned to the same SHA3-224 test vector so they
|
||||||
|
cannot silently drift apart.
|
||||||
|
|
||||||
## Design notes & limitations
|
## Design notes & limitations
|
||||||
|
|
||||||
@@ -214,12 +242,28 @@ drift apart.
|
|||||||
ChaCha20-encrypted (no AEAD tag) to minimize overhead. This protects against
|
ChaCha20-encrypted (no AEAD tag) to minimize overhead. This protects against
|
||||||
casual sniffing, not a determined active attacker (see the note at the top of
|
casual sniffing, not a determined active attacker (see the note at the top of
|
||||||
[PROTOCOL.md](PROTOCOL.md) and [docs/architecture.md](docs/architecture.md) §8).
|
[PROTOCOL.md](PROTOCOL.md) and [docs/architecture.md](docs/architecture.md) §8).
|
||||||
* **Per-stream flow control.** Each stream has credit-based windows in both
|
* **Per-tunnel flow control.** Each worker connection has credit-based windows
|
||||||
directions (windows exchanged at session setup, default 256 KiB), so a slow
|
in both directions (windows exchanged at session setup, default 256 KiB), so
|
||||||
player or slow destination jams only its own stream at a bounded buffer — no
|
a slow player or slow destination jams only its own tunnel at a bounded
|
||||||
application-level head-of-line blocking between streams. What remains is
|
buffer. There is no application-level head-of-line blocking: each player
|
||||||
TCP-level HOL (packet loss stalls a whole worker connection briefly);
|
owns a TCP connection.
|
||||||
raising `maxConn` spreads that.
|
* **Liveness is explicit.** Every session heartbeats, every socket write is
|
||||||
|
bounded, and session establishment has a deadline. A path that dies silently —
|
||||||
|
no `FIN`, no `RST`, as when a NAT or firewall forgets an established flow — is
|
||||||
|
detected within `3 × pingIntervalMs`, the dead connection is dropped, and
|
||||||
|
service is restored without operator action. TCP keepalive is on as a
|
||||||
|
second line of defence.
|
||||||
|
* **A control-session reconnect no longer refuses new players.** While a client
|
||||||
|
is reconnecting the hub has no route for it, so arriving players used to be
|
||||||
|
told there is no such server. Those routes are now held briefly and the
|
||||||
|
players with them, then served once the client re-registers.
|
||||||
|
* **A dropped tunnel no longer drops the players.** A worker connection is only
|
||||||
|
the middle leg of the player it carries; when it dies both terminal sockets
|
||||||
|
are usually still healthy. The hub now hangs that player while the client
|
||||||
|
reattaches the tunnel over a fresh connection, replaying byte-exactly from
|
||||||
|
the offset the peer reports, so a conntrack expiry costs a stall rather than
|
||||||
|
a disconnect. Negotiated, and `streamResume: false` on either side restores
|
||||||
|
the old behaviour.
|
||||||
* **Single hub event loop.** The hub deploys one Vert.x verticle, so all state
|
* **Single hub event loop.** The hub deploys one Vert.x verticle, so all state
|
||||||
is confined to one event loop (no locking). Throughput is bounded by one core;
|
is confined to one event loop (no locking). Throughput is bounded by one core;
|
||||||
ample for hundreds of players, not designed for tens of thousands.
|
ample for hundreds of players, not designed for tens of thousands.
|
||||||
|
|||||||
+244
-52
@@ -8,13 +8,14 @@ import (
|
|||||||
"log"
|
"log"
|
||||||
"net"
|
"net"
|
||||||
"sync"
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/iceBear67/redapricot/client/wire"
|
"github.com/iceBear67/redapricot/client/wire"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Client is a redapricot client: it holds a control session with the hub and a
|
// Client is a redapricot client: it holds a control session with the hub and
|
||||||
// pool of worker connections used to serve player streams.
|
// dials one worker connection per player.
|
||||||
type Client struct {
|
type Client struct {
|
||||||
cfg *Config
|
cfg *Config
|
||||||
pskBytes []byte
|
pskBytes []byte
|
||||||
@@ -24,7 +25,21 @@ type Client struct {
|
|||||||
mappings map[string]Mapping // normalized pattern -> mapping
|
mappings map[string]Mapping // normalized pattern -> mapping
|
||||||
pool *WorkerPool
|
pool *WorkerPool
|
||||||
|
|
||||||
streamWnd int // our advertised per-stream receive window (bytes)
|
// ctx/cancel own every worker dial: Close cancels it so an in-flight dial
|
||||||
|
// aborts instead of holding a goroutine for the whole handshake timeout.
|
||||||
|
// The control path uses the caller's context from Start, which is the same
|
||||||
|
// shutdown signal by convention.
|
||||||
|
ctx context.Context
|
||||||
|
cancel context.CancelFunc
|
||||||
|
|
||||||
|
// closing is set by Close; a conn-loss teardown checks it and closes
|
||||||
|
// streams outright rather than parking them for a reattach that is never
|
||||||
|
// coming. Dial also consults it through the pool's own flag.
|
||||||
|
closing atomic.Bool
|
||||||
|
|
||||||
|
streamWnd int // our advertised per-connection receive window (bytes)
|
||||||
|
shaper *Shaper // caps aggregate egress to the hub; nil when unlimited
|
||||||
|
chunk int // DATA payload cap; shrinks below DataChunkSize at low rates
|
||||||
|
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
ctrl *wire.FramedConn
|
ctrl *wire.FramedConn
|
||||||
@@ -32,11 +47,14 @@ type Client struct {
|
|||||||
|
|
||||||
// New builds a client from config.
|
// New builds a client from config.
|
||||||
func New(cfg *Config) *Client {
|
func New(cfg *Config) *Client {
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
c := &Client{
|
c := &Client{
|
||||||
cfg: cfg,
|
cfg: cfg,
|
||||||
pskBytes: []byte(cfg.PSK),
|
pskBytes: []byte(cfg.PSK),
|
||||||
pskAddr: wire.PSKAddress([]byte(cfg.PSK)),
|
pskAddr: wire.PSKAddress([]byte(cfg.PSK)),
|
||||||
mappings: make(map[string]Mapping),
|
mappings: make(map[string]Mapping),
|
||||||
|
ctx: ctx,
|
||||||
|
cancel: cancel,
|
||||||
}
|
}
|
||||||
if _, portStr, err := net.SplitHostPort(cfg.Server); err == nil {
|
if _, portStr, err := net.SplitHostPort(cfg.Server); err == nil {
|
||||||
if p, err := net.LookupPort("tcp", portStr); err == nil {
|
if p, err := net.LookupPort("tcp", portStr); err == nil {
|
||||||
@@ -47,10 +65,33 @@ func New(cfg *Config) *Client {
|
|||||||
c.mappings[NormalizeAddress(m.Pattern)] = m
|
c.mappings[NormalizeAddress(m.Pattern)] = m
|
||||||
}
|
}
|
||||||
c.streamWnd = clampWindow(cfg.StreamWindowBytes)
|
c.streamWnd = clampWindow(cfg.StreamWindowBytes)
|
||||||
c.pool = newWorkerPool(c, cfg.MaxConn)
|
// Parsed here rather than in LoadConfig because a Config may also be built
|
||||||
|
// directly (tests). LoadConfig has already rejected a malformed value on the
|
||||||
|
// file path, so a failure here can only come from a hand-built Config.
|
||||||
|
bps, err := parseBandwidth(cfg.MaxBandwidth)
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("client: %v; continuing without a bandwidth limit", err)
|
||||||
|
}
|
||||||
|
c.shaper = NewShaper(bps)
|
||||||
|
c.chunk = c.shaper.chunkSize()
|
||||||
|
if c.shaper != nil {
|
||||||
|
log.Printf("egress shaped to %d B/s (burst %d B, chunk %d B)",
|
||||||
|
bps, int64(c.shaper.burst), c.shaper.chunk)
|
||||||
|
}
|
||||||
|
c.pool = newWorkerPool(c, clampMaxTunnels(cfg.MaxTunnels))
|
||||||
return c
|
return c
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func clampMaxTunnels(n int) int {
|
||||||
|
if n < 1 {
|
||||||
|
return DefaultMaxTunnels
|
||||||
|
}
|
||||||
|
if n > MaxMaxTunnels {
|
||||||
|
return MaxMaxTunnels
|
||||||
|
}
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
|
||||||
func clampWindow(w int) int {
|
func clampWindow(w int) int {
|
||||||
if w <= 0 {
|
if w <= 0 {
|
||||||
return DefaultStreamWindow
|
return DefaultStreamWindow
|
||||||
@@ -64,17 +105,45 @@ func clampWindow(w int) int {
|
|||||||
return w
|
return w
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// session is an established redapricot session: the frame transport plus what
|
||||||
|
// was negotiated during establishment.
|
||||||
|
type session struct {
|
||||||
|
fc *wire.FramedConn
|
||||||
|
peerWnd int // hub's advertised per-connection receive window
|
||||||
|
heartbeat bool // hub accepted connection-level PING/PONG on worker conns
|
||||||
|
resume bool // hub accepted stream resumption (§7.5)
|
||||||
|
// hubGrace is how long the hub will hang a parked player, as advertised in
|
||||||
|
// SessionReady. Zero when resumption was not negotiated.
|
||||||
|
hubGrace time.Duration
|
||||||
|
}
|
||||||
|
|
||||||
// dialSession opens a TCP connection, performs the Intent-17 handshake, the
|
// dialSession opens a TCP connection, performs the Intent-17 handshake, the
|
||||||
// Phase-A rekey, and reads SessionReady, returning an established frame conn
|
// Phase-A rekey, and reads SessionReady. Per-connection flow control is mandatory:
|
||||||
// and the hub's advertised per-stream receive window. Per-stream flow control
|
// a hub that does not echo the STREAM_FC flag is rejected.
|
||||||
// is mandatory: a hub that does not echo the STREAM_FC flag is rejected.
|
//
|
||||||
func (c *Client) dialSession(magic byte) (fc *wire.FramedConn, peerWnd int, err error) {
|
// The whole exchange is bounded by HandshakeTimeout. A hub that accepts the
|
||||||
conn, err := net.DialTimeout("tcp", c.cfg.Server, 10*time.Second)
|
// socket but never answers (wedged event loop, a load balancer accepting on its
|
||||||
|
// behalf) must fail fast rather than park the caller forever.
|
||||||
|
//
|
||||||
|
// ctx bounds the dial: the control path passes the caller's context so a
|
||||||
|
// shutdown mid-handshake aborts the attempt, and worker dials pass the client's
|
||||||
|
// own context so Close cancels in-flight dials. After the dial, a cancelled
|
||||||
|
// ctx keeps aborting by closing the conn underneath the deadline-bounded
|
||||||
|
// handshake I/O.
|
||||||
|
func (c *Client) dialSession(ctx context.Context, magic byte) (sess *session, err error) {
|
||||||
|
d := &net.Dialer{Timeout: HandshakeTimeout}
|
||||||
|
conn, err := d.DialContext(ctx, "tcp", c.cfg.Server)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, 0, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
// When ctx ends (shutdown), close the conn so the handshake below fails
|
||||||
|
// immediately instead of waiting out its deadline.
|
||||||
|
stop := context.AfterFunc(ctx, func() { _ = conn.Close() })
|
||||||
|
defer stop()
|
||||||
if tcp, ok := conn.(*net.TCPConn); ok {
|
if tcp, ok := conn.(*net.TCPConn); ok {
|
||||||
_ = tcp.SetNoDelay(true)
|
_ = tcp.SetNoDelay(true)
|
||||||
|
_ = tcp.SetKeepAlive(true)
|
||||||
|
_ = tcp.SetKeepAlivePeriod(TCPKeepAlivePeriod)
|
||||||
}
|
}
|
||||||
ok := false
|
ok := false
|
||||||
defer func() {
|
defer func() {
|
||||||
@@ -82,30 +151,37 @@ func (c *Client) dialSession(magic byte) (fc *wire.FramedConn, peerWnd int, err
|
|||||||
_ = conn.Close()
|
_ = conn.Close()
|
||||||
}
|
}
|
||||||
}()
|
}()
|
||||||
|
if err := conn.SetDeadline(time.Now().Add(HandshakeTimeout)); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
// 1. plaintext Minecraft Handshake, Intent 17, address = hex(SHA3-224(PSK)).
|
// 1. plaintext Minecraft Handshake, Intent 17, address = hex(SHA3-224(PSK)).
|
||||||
hs := wire.BuildHandshake(ProtocolVersion, c.pskAddr, c.serverPort, IntentRedapricot)
|
hs := wire.BuildHandshake(ProtocolVersion, c.pskAddr, c.serverPort, IntentRedapricot)
|
||||||
if _, err := conn.Write(hs); err != nil {
|
if _, err := conn.Write(hs); err != nil {
|
||||||
return nil, 0, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
// 2. Phase-A ciphers derived from the PSK.
|
// 2. Phase-A ciphers derived from the PSK.
|
||||||
fc = wire.NewFramedConn(conn,
|
fc := wire.NewFramedConn(conn,
|
||||||
wire.CipherFor(c.pskBytes, wire.DirS2C), // in: server -> client
|
wire.CipherFor(c.pskBytes, wire.DirS2C), // in: server -> client
|
||||||
wire.CipherFor(c.pskBytes, wire.DirC2S), // out: client -> server
|
wire.CipherFor(c.pskBytes, wire.DirC2S), // out: client -> server
|
||||||
)
|
)
|
||||||
|
|
||||||
// 3. Rekey frame (Phase A), including the mandatory feature flags and our
|
// 3. Rekey frame (Phase A), including the mandatory feature flags and our
|
||||||
// per-stream receive window.
|
// per-connection receive window.
|
||||||
rnd := make([]byte, 16)
|
rnd := make([]byte, 16)
|
||||||
if _, err := crand.Read(rnd); err != nil {
|
if _, err := crand.Read(rnd); err != nil {
|
||||||
return nil, 0, err
|
return nil, err
|
||||||
}
|
}
|
||||||
ts := time.Now().UnixMilli()
|
ts := time.Now().UnixMilli()
|
||||||
|
offered := FlagStreamFC | FlagWorkerHeartbeat
|
||||||
|
if c.cfg.resumeEnabled() {
|
||||||
|
offered |= FlagStreamResume
|
||||||
|
}
|
||||||
rekeyMsg := wire.NewWriter().U8(magic).VarInt(len(rnd)).Bytes(rnd).I64(ts).
|
rekeyMsg := wire.NewWriter().U8(magic).VarInt(len(rnd)).Bytes(rnd).I64(ts).
|
||||||
VarInt(FlagStreamFC).VarInt(c.streamWnd).Out()
|
VarInt(offered).VarInt(c.streamWnd).Out()
|
||||||
if err := fc.WriteFrame(rekeyMsg); err != nil {
|
if err := fc.WriteFrame(rekeyMsg); err != nil {
|
||||||
return nil, 0, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
// 4. Switch to Phase-B ciphers: REKEY = Rand || Timestamp(I64 BE).
|
// 4. Switch to Phase-B ciphers: REKEY = Rand || Timestamp(I64 BE).
|
||||||
@@ -120,49 +196,97 @@ func (c *Client) dialSession(magic byte) (fc *wire.FramedConn, peerWnd int, err
|
|||||||
)
|
)
|
||||||
|
|
||||||
// 5. SessionReady: the type byte followed by the hub's accepted flags and
|
// 5. SessionReady: the type byte followed by the hub's accepted flags and
|
||||||
// its per-stream receive window. Both are required.
|
// its per-connection receive window. Both are required.
|
||||||
payload, err := fc.ReadFrame()
|
payload, err := fc.ReadFrame()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, 0, err
|
return nil, err
|
||||||
}
|
}
|
||||||
if len(payload) < 1 || payload[0] != CtlSessionReady {
|
if len(payload) < 1 || payload[0] != CtlSessionReady {
|
||||||
return nil, 0, fmt.Errorf("expected SessionReady, got %v", payload)
|
return nil, fmt.Errorf("expected SessionReady, got %v", payload)
|
||||||
}
|
}
|
||||||
r := wire.NewReader(payload[1:])
|
r := wire.NewReader(payload[1:])
|
||||||
flags, ferr := r.VarInt()
|
flags, ferr := r.VarInt()
|
||||||
hubWnd, werr := r.VarInt()
|
hubWnd, werr := r.VarInt()
|
||||||
if ferr != nil || werr != nil || flags&FlagStreamFC == 0 || hubWnd <= 0 {
|
if ferr != nil || werr != nil || flags&FlagStreamFC == 0 || hubWnd <= 0 {
|
||||||
return nil, 0, fmt.Errorf("hub did not accept per-stream flow control (unsupported hub version?)")
|
return nil, fmt.Errorf("hub did not accept per-connection flow control (unsupported hub version?)")
|
||||||
}
|
}
|
||||||
if hubWnd > MaxStreamWindow {
|
if hubWnd > MaxStreamWindow {
|
||||||
hubWnd = MaxStreamWindow
|
hubWnd = MaxStreamWindow
|
||||||
}
|
}
|
||||||
|
// Resumption is negotiated per connection, and the hub's grace period rides
|
||||||
|
// along when it accepts. Our own grace is clamped strictly under the hub's:
|
||||||
|
// the client must always give up first, or the hub drops a hanging player
|
||||||
|
// while we are still mid-reattach. A hub that accepts the flag but omits the
|
||||||
|
// grace is treated as not supporting it at all rather than guessed at.
|
||||||
|
resume := flags&FlagStreamResume != 0
|
||||||
|
var hubGrace time.Duration
|
||||||
|
if resume {
|
||||||
|
graceMs, gerr := r.VarInt()
|
||||||
|
if gerr != nil || graceMs <= 0 {
|
||||||
|
log.Printf("hub accepted stream resume without advertising a grace period; disabling resume")
|
||||||
|
resume = false
|
||||||
|
} else {
|
||||||
|
hubGrace = time.Duration(graceMs) * time.Millisecond
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The session is live: drop the establishment deadline. From here on
|
||||||
|
// liveness is the heartbeat's job (and WriteFrame bounds each write).
|
||||||
|
if err := conn.SetDeadline(time.Time{}); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
ok = true
|
ok = true
|
||||||
return fc, hubWnd, nil
|
return &session{
|
||||||
|
fc: fc,
|
||||||
|
peerWnd: hubWnd,
|
||||||
|
heartbeat: flags&FlagWorkerHeartbeat != 0,
|
||||||
|
resume: resume,
|
||||||
|
hubGrace: hubGrace,
|
||||||
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// statsOn reports whether performance diagnostics are enabled. When off, no
|
||||||
|
// counter struct is ever allocated and the instrumentation is a single branch.
|
||||||
|
func (c *Client) statsOn() bool { return c.cfg.StatsIntervalMs > 0 }
|
||||||
|
|
||||||
// Start establishes the control session and registers all patterns. It returns
|
// Start establishes the control session and registers all patterns. It returns
|
||||||
// once the initial connection succeeds; subsequent drops are handled in the
|
// once the initial connection succeeds; subsequent drops are handled in the
|
||||||
// background with reconnect.
|
// background with reconnect.
|
||||||
func (c *Client) Start(ctx context.Context) error {
|
func (c *Client) Start(ctx context.Context) error {
|
||||||
return c.connectControl(ctx)
|
if err := c.connectControl(ctx); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if c.statsOn() {
|
||||||
|
go c.statsLoop(ctx.Done())
|
||||||
|
}
|
||||||
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *Client) connectControl(ctx context.Context) error {
|
func (c *Client) connectControl(ctx context.Context) error {
|
||||||
fc, _, err := c.dialSession(MagicControl)
|
sess, err := c.dialSession(ctx, MagicControl)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("control connect: %w", err)
|
return fmt.Errorf("control connect: %w", err)
|
||||||
}
|
}
|
||||||
c.registerAll(fc)
|
ctrl := &ctrlSession{fc: sess.fc}
|
||||||
|
ctrl.lastPong.Store(time.Now().UnixMilli())
|
||||||
|
c.registerAll(sess.fc)
|
||||||
c.mu.Lock()
|
c.mu.Lock()
|
||||||
c.ctrl = fc
|
c.ctrl = sess.fc
|
||||||
c.mu.Unlock()
|
c.mu.Unlock()
|
||||||
log.Printf("control session established with %s", c.cfg.Server)
|
log.Printf("control session established with %s", c.cfg.Server)
|
||||||
go c.serveControl(ctx, fc)
|
go c.serveControl(ctx, ctrl)
|
||||||
go c.pingLoop(ctx, fc)
|
go c.pingLoop(ctx, ctrl)
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ctrlSession tracks liveness for one control connection. A control session
|
||||||
|
// whose path dies silently must be detected, otherwise the hub keeps routing
|
||||||
|
// players to a session the client will never read from and nobody can connect.
|
||||||
|
type ctrlSession struct {
|
||||||
|
fc *wire.FramedConn
|
||||||
|
lastPong atomic.Int64 // unix ms of the most recent Pong
|
||||||
|
}
|
||||||
|
|
||||||
func (c *Client) registerAll(fc *wire.FramedConn) {
|
func (c *Client) registerAll(fc *wire.FramedConn) {
|
||||||
for pattern := range c.mappings {
|
for pattern := range c.mappings {
|
||||||
msg := wire.NewWriter().U8(CtlRegister).String(pattern).Out()
|
msg := wire.NewWriter().U8(CtlRegister).String(pattern).Out()
|
||||||
@@ -174,33 +298,54 @@ func (c *Client) registerAll(fc *wire.FramedConn) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *Client) serveControl(ctx context.Context, fc *wire.FramedConn) {
|
func (c *Client) serveControl(ctx context.Context, ctrl *ctrlSession) {
|
||||||
for {
|
for {
|
||||||
payload, err := fc.ReadFrame()
|
payload, err := ctrl.fc.ReadFrame()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
c.dispatchControl(payload)
|
c.dispatchControl(ctrl, payload)
|
||||||
}
|
}
|
||||||
_ = fc.Close()
|
_ = ctrl.fc.Close()
|
||||||
if ctx.Err() != nil {
|
if ctx.Err() != nil || c.closing.Load() {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
// Reconnect with backoff.
|
// Reconnect with backoff, but try immediately first. While the control
|
||||||
for backoff := 500 * time.Millisecond; ctx.Err() == nil; backoff *= 2 {
|
// session is down the hub has no live route for this client, so every
|
||||||
if backoff > 10*time.Second {
|
// millisecond of delay is a player arriving to be told there is no such
|
||||||
backoff = 10 * time.Second
|
// server — and a session usually dies to a transient blip that the very next
|
||||||
|
// dial would have survived. Sleeping first spent that window unconditionally.
|
||||||
|
//
|
||||||
|
// The wait is on ctx rather than time.Sleep so shutdown is not held up by a
|
||||||
|
// backoff that has grown to the cap. Close() is authoritative on its own:
|
||||||
|
// it gates the loop directly because a caller may shut the client down by
|
||||||
|
// calling Close() without cancelling the context it passed to Start.
|
||||||
|
for backoff := time.Duration(0); ctx.Err() == nil && !c.closing.Load(); {
|
||||||
|
if backoff > 0 {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case <-time.After(backoff):
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if c.closing.Load() {
|
||||||
|
return
|
||||||
}
|
}
|
||||||
time.Sleep(backoff)
|
|
||||||
if err := c.connectControl(ctx); err == nil {
|
if err := c.connectControl(ctx); err == nil {
|
||||||
return
|
return
|
||||||
} else {
|
} else {
|
||||||
log.Printf("control reconnect failed: %v", err)
|
log.Printf("control reconnect failed: %v", err)
|
||||||
}
|
}
|
||||||
|
switch {
|
||||||
|
case backoff == 0:
|
||||||
|
backoff = 500 * time.Millisecond
|
||||||
|
case backoff < maxControlBackoff:
|
||||||
|
backoff = min(backoff*2, maxControlBackoff)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *Client) dispatchControl(payload []byte) {
|
func (c *Client) dispatchControl(ctrl *ctrlSession, payload []byte) {
|
||||||
r := wire.NewReader(payload)
|
r := wire.NewReader(payload)
|
||||||
t, err := r.U8()
|
t, err := r.U8()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -212,7 +357,14 @@ func (c *Client) dispatchControl(payload []byte) {
|
|||||||
case CtlRegisterAck:
|
case CtlRegisterAck:
|
||||||
pattern, _ := r.String()
|
pattern, _ := r.String()
|
||||||
status, _ := r.U8()
|
status, _ := r.U8()
|
||||||
log.Printf("register ack %q status=%d", pattern, status)
|
switch status {
|
||||||
|
case RegisterOk:
|
||||||
|
log.Printf("pattern %q registered", pattern)
|
||||||
|
case RegisterErrPattern:
|
||||||
|
log.Printf("pattern %q rejected: not a valid regular expression", pattern)
|
||||||
|
default:
|
||||||
|
log.Printf("pattern %q rejected: status=%d", pattern, status)
|
||||||
|
}
|
||||||
case CtlControlRequest:
|
case CtlControlRequest:
|
||||||
cid, err := r.Bytes(CIDLen)
|
cid, err := r.Bytes(CIDLen)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -223,30 +375,41 @@ func (c *Client) dispatchControl(payload []byte) {
|
|||||||
port, _ := r.U16()
|
port, _ := r.U16()
|
||||||
go c.handleControlRequest(cid, pattern, ip, int(port))
|
go c.handleControlRequest(cid, pattern, ip, int(port))
|
||||||
case CtlPong:
|
case CtlPong:
|
||||||
// ignore
|
ctrl.lastPong.Store(time.Now().UnixMilli())
|
||||||
default:
|
default:
|
||||||
log.Printf("control: unknown message type %d", t)
|
log.Printf("control: unknown message type %d", t)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *Client) pingLoop(ctx context.Context, fc *wire.FramedConn) {
|
// pingLoop keeps the control session alive and, crucially, verifies that the
|
||||||
ticker := time.NewTicker(time.Duration(c.cfg.PingIntervalMs) * time.Millisecond)
|
// hub is still answering. A path that dies silently (no FIN/RST) would
|
||||||
|
// otherwise leave the read loop parked forever: the client would believe it is
|
||||||
|
// still registered while the hub routes players into the void.
|
||||||
|
func (c *Client) pingLoop(ctx context.Context, ctrl *ctrlSession) {
|
||||||
|
ticker := time.NewTicker(c.cfg.pingInterval())
|
||||||
defer ticker.Stop()
|
defer ticker.Stop()
|
||||||
|
timeout := c.cfg.heartbeatTimeout()
|
||||||
for {
|
for {
|
||||||
select {
|
select {
|
||||||
case <-ctx.Done():
|
case <-ctx.Done():
|
||||||
return
|
return
|
||||||
case <-ticker.C:
|
case <-ticker.C:
|
||||||
|
last := time.UnixMilli(ctrl.lastPong.Load())
|
||||||
|
if time.Since(last) > timeout {
|
||||||
|
log.Printf("control session silent for %s; dropping it to force a reconnect", time.Since(last).Round(time.Second))
|
||||||
|
_ = ctrl.fc.Close() // unblocks serveControl, which reconnects
|
||||||
|
return
|
||||||
|
}
|
||||||
msg := wire.NewWriter().U8(CtlPing).I64(time.Now().UnixMilli()).Out()
|
msg := wire.NewWriter().U8(CtlPing).I64(time.Now().UnixMilli()).Out()
|
||||||
if err := fc.WriteFrame(msg); err != nil {
|
if err := ctrl.fc.WriteFrame(msg); err != nil {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// handleControlRequest reacts to a matched player: allocate a worker stream,
|
// handleControlRequest reacts to a matched player: dial a dedicated worker
|
||||||
// SYN it, and bridge it to the mapped destination.
|
// conn, SYN it, and bridge it to the mapped destination.
|
||||||
func (c *Client) handleControlRequest(cid []byte, pattern, ip string, port int) {
|
func (c *Client) handleControlRequest(cid []byte, pattern, ip string, port int) {
|
||||||
mapping, ok := c.mappings[NormalizeAddress(pattern)]
|
mapping, ok := c.mappings[NormalizeAddress(pattern)]
|
||||||
if !ok {
|
if !ok {
|
||||||
@@ -254,24 +417,52 @@ func (c *Client) handleControlRequest(cid []byte, pattern, ip string, port int)
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
log.Printf("player %s:%d joined via pattern %q -> %s", ip, port, pattern, mapping.Destination)
|
log.Printf("player %s:%d joined via pattern %q -> %s", ip, port, pattern, mapping.Destination)
|
||||||
wc, sid, err := c.pool.Allocate()
|
|
||||||
if err != nil {
|
// Dial and attach must agree on a live conn: Dial hands out a conn that can
|
||||||
log.Printf("worker allocate failed: %v", err)
|
// die before we attach on it, which would strand the stream on a conn
|
||||||
|
// nothing iterates. attach reports that, and we simply dial another.
|
||||||
|
var st *Stream
|
||||||
|
var wc *WorkerConn
|
||||||
|
for attempt := 0; attempt < dialAttempts; attempt++ {
|
||||||
|
var err error
|
||||||
|
wc, err = c.pool.Dial()
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("worker dial failed: %v", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
st = newStream(c, wc, cid, mapping, ip, port)
|
||||||
|
// Attach before SYN so inbound DATA can never race ahead of the binding,
|
||||||
|
// and start the pumps before the (bounded) SYN write so a failed or slow
|
||||||
|
// SYN cannot strand a stream that nothing would ever tear down.
|
||||||
|
if wc.attach(st) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
_ = wc.fc.Close()
|
||||||
|
st = nil
|
||||||
|
}
|
||||||
|
if st == nil {
|
||||||
|
log.Printf("worker dial failed: no live conn after %d attempts", dialAttempts)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
st := newStream(wc, sid, cid, mapping, ip, port)
|
|
||||||
wc.registerStream(sid, st)
|
|
||||||
wc.sendSyn(sid, cid)
|
|
||||||
go st.writeLoop()
|
go st.writeLoop()
|
||||||
go st.run()
|
go st.run()
|
||||||
|
if err := wc.sendSyn(cid); err != nil {
|
||||||
|
log.Printf("stream %s: SYN failed: %v", st.name(), err)
|
||||||
|
st.teardown(false)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// WorkerConnCount reports the current number of open worker connections
|
// WorkerConnCount reports the current number of open worker connections
|
||||||
// (exposed for tests/observability).
|
// (exposed for tests/observability).
|
||||||
func (c *Client) WorkerConnCount() int { return c.pool.count() }
|
func (c *Client) WorkerConnCount() int { return c.pool.count() }
|
||||||
|
|
||||||
// Close tears down the control session and all worker connections.
|
// Close tears down the control session and all worker connections. Idempotent:
|
||||||
|
// a second call (or a Close racing a reconnect) only re-closes what is still
|
||||||
|
// open.
|
||||||
func (c *Client) Close() {
|
func (c *Client) Close() {
|
||||||
|
c.closing.Store(true)
|
||||||
|
c.cancel() // aborts in-flight worker dials
|
||||||
c.mu.Lock()
|
c.mu.Lock()
|
||||||
fc := c.ctrl
|
fc := c.ctrl
|
||||||
c.mu.Unlock()
|
c.mu.Unlock()
|
||||||
@@ -279,4 +470,5 @@ func (c *Client) Close() {
|
|||||||
_ = fc.Close()
|
_ = fc.Close()
|
||||||
}
|
}
|
||||||
c.pool.closeAll()
|
c.pool.closeAll()
|
||||||
|
c.shaper.Stop()
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,9 +1,13 @@
|
|||||||
{
|
{
|
||||||
"server": "hub.example.com:25565",
|
"server": "hub.example.com:25565",
|
||||||
"psk": "change-me-to-a-long-random-passphrase",
|
"psk": "change-me-to-a-long-random-passphrase",
|
||||||
"maxConn": 4,
|
"maxTunnels": 256,
|
||||||
"pingIntervalMs": 20000,
|
"pingIntervalMs": 20000,
|
||||||
"streamWindowBytes": 262144,
|
"streamWindowBytes": 262144,
|
||||||
|
"maxBandwidth": "",
|
||||||
|
"streamResume": true,
|
||||||
|
"resumeGraceMs": 15000,
|
||||||
|
"statsIntervalMs": 0,
|
||||||
"mappings": [
|
"mappings": [
|
||||||
{
|
{
|
||||||
"pattern": "mc\\.example\\.com",
|
"pattern": "mc\\.example\\.com",
|
||||||
|
|||||||
+263
-17
@@ -4,13 +4,20 @@ import (
|
|||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Protocol constants (mirror of the Java Protocol class; see PROTOCOL.md).
|
// Protocol constants (mirror of the Java Protocol class; see PROTOCOL.md).
|
||||||
const (
|
const (
|
||||||
IntentRedapricot = 17
|
IntentRedapricot = 17
|
||||||
ProtocolVersion = 767 // arbitrary; the hub ignores it
|
// IntentReserved is the handshake intent reserved for redapricot
|
||||||
|
// management/status (mirror of Protocol.INTENT_RESERVED). The hub never
|
||||||
|
// pattern-matches it; it replies with a Minecraft status line and closes,
|
||||||
|
// so an operator can probe the port without joining the protocol.
|
||||||
|
IntentReserved = 18
|
||||||
|
ProtocolVersion = 767 // arbitrary; the hub ignores it
|
||||||
|
|
||||||
MagicControl = 0x01
|
MagicControl = 0x01
|
||||||
MagicWorker = 0x02
|
MagicWorker = 0x02
|
||||||
@@ -25,45 +32,268 @@ const (
|
|||||||
CtlPing = 0x05
|
CtlPing = 0x05
|
||||||
CtlPong = 0x06
|
CtlPong = 0x06
|
||||||
|
|
||||||
|
// RegisterAck status codes (mirror of Protocol.REGISTER_OK/_ERR_PATTERN).
|
||||||
|
RegisterOk = 0x00
|
||||||
|
RegisterErrPattern = 0x01 // pattern is not a valid regular expression
|
||||||
|
|
||||||
MuxSyn = 0x00
|
MuxSyn = 0x00
|
||||||
MuxData = 0x01
|
MuxData = 0x01
|
||||||
MuxFin = 0x02
|
MuxFin = 0x02
|
||||||
MuxRst = 0x03
|
MuxRst = 0x03
|
||||||
MuxWnd = 0x04
|
MuxWnd = 0x04
|
||||||
|
MuxPing = 0x05
|
||||||
|
MuxPong = 0x06
|
||||||
|
// MuxResume reattaches a parked player to this conn (CID + our accepted
|
||||||
|
// offset); MuxResumeAck carries the hub's accepted offset and a fresh CID.
|
||||||
|
MuxResume = 0x07
|
||||||
|
MuxResumeAck = 0x08
|
||||||
|
|
||||||
|
// RST reason codes (optional trailing byte; absence means "unspecified").
|
||||||
|
// Distinguishing them matters for resume: an unknown stream is terminal,
|
||||||
|
// while "already bound" means the hub has the player on another conn — the
|
||||||
|
// reattach retries until that bind dies and the hub re-parks the player.
|
||||||
|
RstUnspecified = 0x00
|
||||||
|
RstUnknownStream = 0x01
|
||||||
|
RstAlreadyBound = 0x02
|
||||||
|
RstResumeAbandoned = 0x03
|
||||||
|
RstFlowControl = 0x04
|
||||||
|
RstDialFailed = 0x05
|
||||||
|
|
||||||
FrameError = 0x7F
|
FrameError = 0x7F
|
||||||
|
|
||||||
SaturationThreshold = 8
|
// DefaultMaxTunnels / MaxMaxTunnels bound concurrent 1:1 worker conns.
|
||||||
|
// The old mux-era maxConn cap of 8 would silently become "8 players".
|
||||||
|
DefaultMaxTunnels = 256
|
||||||
|
MaxMaxTunnels = 4096
|
||||||
|
|
||||||
// Session-establishment feature flags (trailing VarInt on the Rekey message).
|
// Session-establishment feature flags (trailing VarInt on the Rekey message).
|
||||||
FlagStreamFC = 0x01
|
FlagStreamFC = 0x01
|
||||||
|
// FlagWorkerHeartbeat enables connection-level PING/PONG on worker conns.
|
||||||
|
// Without it a worker conn whose path is silently blackholed (NAT/conntrack
|
||||||
|
// drop, firewall) is never detected: the read loop parks forever and that
|
||||||
|
// player is stuck until the client restarts.
|
||||||
|
FlagWorkerHeartbeat = 0x02
|
||||||
|
// FlagStreamResume enables stream resumption (PROTOCOL.md §7.5): a worker
|
||||||
|
// conn drop parks the player instead of killing them, the hub hangs the
|
||||||
|
// player socket, and the client reattaches byte-exactly over a fresh conn.
|
||||||
|
// Negotiated, so either side may decline and get today's behaviour
|
||||||
|
// (immediate teardown) unchanged.
|
||||||
|
FlagStreamResume = 0x04
|
||||||
|
|
||||||
// Per-stream flow-control window bounds (bytes). The advertised window is the
|
// Per-connection flow-control window bounds (bytes). The advertised window
|
||||||
// receiver's promise of how much un-credited DATA it will buffer per stream.
|
// is the receiver's promise of how much un-credited DATA it will buffer.
|
||||||
DefaultStreamWindow = 256 * 1024
|
DefaultStreamWindow = 256 * 1024
|
||||||
MinStreamWindow = 32 * 1024
|
MinStreamWindow = 32 * 1024
|
||||||
MaxStreamWindow = 8 << 20
|
MaxStreamWindow = 8 << 20
|
||||||
|
|
||||||
// DataChunkSize caps a single DATA frame's payload so no stream monopolizes
|
// DataChunkSize caps a single DATA frame's payload so one write cannot
|
||||||
// the shared worker connection for long.
|
// occupy the link for a full 1-MiB frame.
|
||||||
DataChunkSize = 32 * 1024
|
DataChunkSize = 32 * 1024
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// Egress bandwidth shaping (see shaper.go and docs/architecture.md §6). These
|
||||||
|
// are entirely client-local: nothing here appears on the wire.
|
||||||
|
const (
|
||||||
|
// MinBandwidth floors a configured cap. Below this the tunnel cannot carry a
|
||||||
|
// Minecraft session at all, so such a value is a unit typo ("20bps" for
|
||||||
|
// "20mbps") and is rejected rather than silently clamped.
|
||||||
|
MinBandwidth = 8 * 1024
|
||||||
|
|
||||||
|
// ShaperBurstSeconds is how much transmission time the token bucket banks
|
||||||
|
// while idle. Big enough to absorb a chunk-load spike; small enough that
|
||||||
|
// releasing it cannot overrun the physical uplink and rebuild the standing
|
||||||
|
// queue the cap exists to prevent.
|
||||||
|
ShaperBurstSeconds = 0.2
|
||||||
|
|
||||||
|
// MinShaperBurst must exceed DataChunkSize: a request larger than the bucket
|
||||||
|
// could never be afforded and would park forever.
|
||||||
|
MinShaperBurst = 64 * 1024
|
||||||
|
MaxShaperBurst = 4 << 20
|
||||||
|
|
||||||
|
// ShaperSliceSeconds bounds how long one stream holds the link before the
|
||||||
|
// scheduler can switch, by sizing the send chunk to that much transmission
|
||||||
|
// time. Above ~13 Mbps this yields DataChunkSize and nothing changes.
|
||||||
|
ShaperSliceSeconds = 0.02
|
||||||
|
MinShaperChunk = 4 * 1024
|
||||||
|
)
|
||||||
|
|
||||||
|
// Timeouts. Every tunnel socket is covered by one of these: without them a
|
||||||
|
// silently dropped path (no FIN/RST) leaves the client parked forever.
|
||||||
|
const (
|
||||||
|
// HandshakeTimeout bounds session establishment end to end — the TCP dial,
|
||||||
|
// the Rekey write and the SessionReady read. A hub that accepts the socket
|
||||||
|
// but never answers must not park the caller (and, for the pool, every other
|
||||||
|
// player behind it) indefinitely.
|
||||||
|
HandshakeTimeout = 15 * time.Second
|
||||||
|
|
||||||
|
// TCPKeepAlivePeriod asks the kernel to probe idle tunnel sockets, so a peer
|
||||||
|
// that becomes unreachable is detected even when no frames are in flight.
|
||||||
|
TCPKeepAlivePeriod = 30 * time.Second
|
||||||
|
|
||||||
|
// MissedHeartbeats is how many ping intervals may pass with no reply before
|
||||||
|
// a session is declared dead and dropped.
|
||||||
|
MissedHeartbeats = 3
|
||||||
|
|
||||||
|
// DefaultPingIntervalMs is used when a config carries no (or a
|
||||||
|
// non-positive) interval. LoadConfig applies the same default.
|
||||||
|
DefaultPingIntervalMs = 20000
|
||||||
|
|
||||||
|
// MinPingIntervalMs floors the configured ping interval so the derived
|
||||||
|
// heartbeat timeout can never be short enough to cause spurious drops.
|
||||||
|
// Applied wherever the interval is read, not just on the file path: a
|
||||||
|
// hand-built Config (tests) carrying 0 would otherwise panic
|
||||||
|
// time.NewTicker at the call site.
|
||||||
|
MinPingIntervalMs = 1000
|
||||||
|
|
||||||
|
// DefaultResumeGraceMs is how long a parked stream keeps trying to reattach
|
||||||
|
// before giving up and closing the destination.
|
||||||
|
//
|
||||||
|
// Chosen against the backend, not the tunnel: a hung player stops answering
|
||||||
|
// the game server's KeepAlive, and vanilla disconnects a silent client at
|
||||||
|
// 30s. A longer grace would resume sessions the backend then kicks anyway.
|
||||||
|
DefaultResumeGraceMs = 15000
|
||||||
|
|
||||||
|
// MinResumeGraceMs floors the grace so it can always fit at least one dial;
|
||||||
|
// a grace shorter than HandshakeTimeout could never complete an attempt.
|
||||||
|
MinResumeGraceMs = 2000
|
||||||
|
|
||||||
|
// ResumeRetryDelay paces reattach attempts after a failure. Short, because
|
||||||
|
// the player is hanging for the whole grace period.
|
||||||
|
ResumeRetryDelay = 500 * time.Millisecond
|
||||||
|
|
||||||
|
// maxControlBackoff caps the control-session reconnect delay. The hub holds
|
||||||
|
// this client's routes only for its own registration grace, so a backoff that
|
||||||
|
// grew past that would strand players it is hanging on our behalf.
|
||||||
|
maxControlBackoff = 10 * time.Second
|
||||||
|
|
||||||
|
// ResumeAckTimeout bounds the wait for RESUME_ACK on a conn that completed
|
||||||
|
// its handshake but then went quiet, so a wedged hub does not consume the
|
||||||
|
// entire grace budget in one attempt.
|
||||||
|
ResumeAckTimeout = 10 * time.Second
|
||||||
|
)
|
||||||
|
|
||||||
|
// heartbeatTimeout is how long a session may go without a reply before it is
|
||||||
|
// considered dead, derived from the configured ping interval.
|
||||||
|
func (c *Config) heartbeatTimeout() time.Duration {
|
||||||
|
return c.pingInterval() * MissedHeartbeats
|
||||||
|
}
|
||||||
|
|
||||||
|
// pingInterval is the configured heartbeat period, clamped at the single point
|
||||||
|
// where a duration is derived. LoadConfig also clamps on the file path; this
|
||||||
|
// covers Configs built directly (tests), where a 0 or negative PingIntervalMs
|
||||||
|
// would panic time.NewTicker — a panic, not a log line, because the interval
|
||||||
|
// feeds the heartbeat timeout too.
|
||||||
|
func (c *Config) pingInterval() time.Duration {
|
||||||
|
ms := c.PingIntervalMs
|
||||||
|
if ms <= 0 {
|
||||||
|
ms = DefaultPingIntervalMs
|
||||||
|
}
|
||||||
|
if ms < MinPingIntervalMs {
|
||||||
|
ms = MinPingIntervalMs
|
||||||
|
}
|
||||||
|
return time.Duration(ms) * time.Millisecond
|
||||||
|
}
|
||||||
|
|
||||||
// Mapping routes a registered pattern to a real destination.
|
// Mapping routes a registered pattern to a real destination.
|
||||||
type Mapping struct {
|
type Mapping struct {
|
||||||
Pattern string `json:"pattern"`
|
Pattern string `json:"pattern"`
|
||||||
Destination string `json:"destination"`
|
Destination string `json:"destination"`
|
||||||
ProxyProtocol bool `json:"proxyProtocol"`
|
ProxyProtocol bool `json:"proxyProtocol"`
|
||||||
|
// VelocitySecret, when non-empty, answers the destination's Velocity
|
||||||
|
// modern-forwarding login query (velocity:player_info) with this secret,
|
||||||
|
// forwarding the player's real IP, username and UUID (see velocity.go).
|
||||||
|
VelocitySecret string `json:"velocitySecret"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// Config is the client configuration (PROTOCOL.md §9.2).
|
// Config is the client configuration (PROTOCOL.md §9.2).
|
||||||
type Config struct {
|
type Config struct {
|
||||||
Server string `json:"server"`
|
Server string `json:"server"`
|
||||||
PSK string `json:"psk"`
|
PSK string `json:"psk"`
|
||||||
MaxConn int `json:"maxConn"`
|
// MaxTunnels is the max concurrent 1:1 worker connections (PROTOCOL.md §7.1).
|
||||||
PingIntervalMs int `json:"pingIntervalMs"`
|
// 0 means the default. Clamped to [1, 4096].
|
||||||
StreamWindowBytes int `json:"streamWindowBytes"` // per-stream receive window; 0 = default
|
MaxTunnels int `json:"maxTunnels"`
|
||||||
Mappings []Mapping `json:"mappings"`
|
// MaxConn is the retired mux-era pool size. Ignored when loading a file:
|
||||||
|
// honouring a value of 4 as a player cap would silently break existing
|
||||||
|
// configs. Tests that construct a Config should set MaxTunnels instead.
|
||||||
|
MaxConn int `json:"maxConn"`
|
||||||
|
PingIntervalMs int `json:"pingIntervalMs"`
|
||||||
|
StreamWindowBytes int `json:"streamWindowBytes"` // per-connection receive window; 0 = default
|
||||||
|
// MaxBandwidth caps what the client sends to the hub, aggregated over every
|
||||||
|
// worker conn — the direction that carries the game server's output to the
|
||||||
|
// players, and the one a residential uplink runs out of first.
|
||||||
|
// Empty means no limit. See parseBandwidth for the accepted syntax.
|
||||||
|
MaxBandwidth string `json:"maxBandwidth"`
|
||||||
|
// StreamResume enables stream resumption (PROTOCOL.md §7.5). A pointer so an
|
||||||
|
// absent key means "on" while an explicit false disables it: with it off the
|
||||||
|
// client never offers the flag, allocates no retransmit buffers, and behaves
|
||||||
|
// exactly as a pre-resume client.
|
||||||
|
StreamResume *bool `json:"streamResume"`
|
||||||
|
// ResumeGraceMs bounds how long a parked stream keeps trying to reattach.
|
||||||
|
// Clamped below the hub's advertised grace so the client always gives up
|
||||||
|
// first and the hub is never left holding a player nobody will claim.
|
||||||
|
ResumeGraceMs int `json:"resumeGraceMs"`
|
||||||
|
// StatsIntervalMs enables the periodic performance summary; 0 (the default)
|
||||||
|
// disables it and costs nothing.
|
||||||
|
StatsIntervalMs int `json:"statsIntervalMs"`
|
||||||
|
Mappings []Mapping `json:"mappings"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// resumeEnabled reports whether stream resumption is configured on.
|
||||||
|
func (c *Config) resumeEnabled() bool { return c.StreamResume == nil || *c.StreamResume }
|
||||||
|
|
||||||
|
// resumeGrace is how long a parked stream may keep trying to reattach.
|
||||||
|
func (c *Config) resumeGrace() time.Duration {
|
||||||
|
ms := c.ResumeGraceMs
|
||||||
|
if ms <= 0 {
|
||||||
|
ms = DefaultResumeGraceMs
|
||||||
|
}
|
||||||
|
if ms < MinResumeGraceMs {
|
||||||
|
ms = MinResumeGraceMs
|
||||||
|
}
|
||||||
|
return time.Duration(ms) * time.Millisecond
|
||||||
|
}
|
||||||
|
|
||||||
|
// bandwidthUnits maps a rate suffix to its value in bytes per second. Bit units
|
||||||
|
// are decimal because that is what ISPs quote; byte units are binary to match
|
||||||
|
// streamWindowBytes. Ordered longest-suffix-first so "kbps" is not read as
|
||||||
|
// "bps", nor "gb/s" as "b/s".
|
||||||
|
var bandwidthUnits = []struct {
|
||||||
|
suffix string
|
||||||
|
mul float64
|
||||||
|
}{
|
||||||
|
{"gbps", 1e9 / 8}, {"gbit", 1e9 / 8},
|
||||||
|
{"mbps", 1e6 / 8}, {"mbit", 1e6 / 8},
|
||||||
|
{"kbps", 1e3 / 8}, {"kbit", 1e3 / 8},
|
||||||
|
{"gb/s", 1 << 30}, {"mb/s", 1 << 20}, {"kb/s", 1 << 10},
|
||||||
|
{"bps", 1.0 / 8},
|
||||||
|
{"b/s", 1},
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseBandwidth converts a human-readable rate to bytes per second. The empty
|
||||||
|
// string means "no limit" and yields 0.
|
||||||
|
//
|
||||||
|
// "20mbps" 20 megabits/s = 2500000 B/s
|
||||||
|
// "512kbps" 512 kilobits/s = 64000 B/s
|
||||||
|
// "2MB/s" 2 mebibytes/s = 2097152 B/s
|
||||||
|
// "1500000" a bare number is already bytes per second
|
||||||
|
func parseBandwidth(s string) (int64, error) {
|
||||||
|
s = strings.TrimSpace(s)
|
||||||
|
if s == "" {
|
||||||
|
return 0, nil
|
||||||
|
}
|
||||||
|
lower := strings.ToLower(s)
|
||||||
|
num, mul := lower, 1.0
|
||||||
|
for _, u := range bandwidthUnits {
|
||||||
|
if strings.HasSuffix(lower, u.suffix) {
|
||||||
|
num, mul = strings.TrimSpace(lower[:len(lower)-len(u.suffix)]), u.mul
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
v, err := strconv.ParseFloat(num, 64)
|
||||||
|
if err != nil || v <= 0 {
|
||||||
|
return 0, fmt.Errorf(`maxBandwidth: cannot read %q as a rate (try "20mbps", "2MB/s", or bytes per second)`, s)
|
||||||
|
}
|
||||||
|
return int64(v * mul), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// LoadConfig reads and validates a JSON config file.
|
// LoadConfig reads and validates a JSON config file.
|
||||||
@@ -82,14 +312,30 @@ func LoadConfig(path string) (*Config, error) {
|
|||||||
if c.PSK == "" {
|
if c.PSK == "" {
|
||||||
return nil, fmt.Errorf("psk is required")
|
return nil, fmt.Errorf("psk is required")
|
||||||
}
|
}
|
||||||
if c.MaxConn < 1 {
|
if c.MaxConn != 0 && c.MaxTunnels == 0 {
|
||||||
c.MaxConn = 1
|
// Old mux pool size. Must not become the player cap: a previously-working
|
||||||
|
// maxConn: 4 would admit only four players.
|
||||||
|
fmt.Fprintf(os.Stderr, "redapricot-client: maxConn is ignored (it was the mux pool size); use maxTunnels (default %d)\n", DefaultMaxTunnels)
|
||||||
}
|
}
|
||||||
if c.MaxConn > 8 {
|
if c.MaxTunnels < 1 {
|
||||||
c.MaxConn = 8
|
c.MaxTunnels = DefaultMaxTunnels
|
||||||
|
}
|
||||||
|
if c.MaxTunnels > MaxMaxTunnels {
|
||||||
|
c.MaxTunnels = MaxMaxTunnels
|
||||||
}
|
}
|
||||||
if c.PingIntervalMs <= 0 {
|
if c.PingIntervalMs <= 0 {
|
||||||
c.PingIntervalMs = 20000
|
c.PingIntervalMs = DefaultPingIntervalMs
|
||||||
|
}
|
||||||
|
if c.PingIntervalMs < MinPingIntervalMs {
|
||||||
|
c.PingIntervalMs = MinPingIntervalMs
|
||||||
|
}
|
||||||
|
// Parsed here only to fail fast on a bad value; New does the real conversion.
|
||||||
|
bps, err := parseBandwidth(c.MaxBandwidth)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if bps > 0 && bps < MinBandwidth {
|
||||||
|
return nil, fmt.Errorf("maxBandwidth %q is only %d B/s; the minimum is %d B/s", c.MaxBandwidth, bps, MinBandwidth)
|
||||||
}
|
}
|
||||||
if len(c.Mappings) == 0 {
|
if len(c.Mappings) == 0 {
|
||||||
return nil, fmt.Errorf("at least one mapping is required")
|
return nil, fmt.Errorf("at least one mapping is required")
|
||||||
|
|||||||
@@ -0,0 +1,86 @@
|
|||||||
|
package client
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// stalledHub accepts connections and then says nothing: it never answers the
|
||||||
|
// Rekey frame with SessionReady, and never closes. This models a hub with a
|
||||||
|
// wedged event loop, or a load balancer accepting on behalf of a dead backend.
|
||||||
|
func stalledHub(t *testing.T) string {
|
||||||
|
t.Helper()
|
||||||
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var mu sync.Mutex
|
||||||
|
var held []net.Conn
|
||||||
|
t.Cleanup(func() {
|
||||||
|
_ = ln.Close()
|
||||||
|
mu.Lock()
|
||||||
|
for _, c := range held {
|
||||||
|
_ = c.Close()
|
||||||
|
}
|
||||||
|
mu.Unlock()
|
||||||
|
})
|
||||||
|
go func() {
|
||||||
|
for {
|
||||||
|
c, err := ln.Accept()
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
mu.Lock()
|
||||||
|
held = append(held, c)
|
||||||
|
mu.Unlock()
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
return ln.Addr().String()
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestDialDoesNotWedgeOnStalledHub is the regression guard for the worst
|
||||||
|
// failure mode found in the stability audit: Dial used to share a single
|
||||||
|
// in-flight handshake, and the handshake read had no deadline. One
|
||||||
|
// unresponsive hub therefore parked every present and future allocation
|
||||||
|
// forever. 1:1 dials independently, but each must still fail on its own
|
||||||
|
// HandshakeTimeout rather than block the other.
|
||||||
|
func TestDialDoesNotWedgeOnStalledHub(t *testing.T) {
|
||||||
|
c := New(&Config{
|
||||||
|
Server: stalledHub(t),
|
||||||
|
PSK: "pool-test",
|
||||||
|
MaxTunnels: 8,
|
||||||
|
PingIntervalMs: 20000,
|
||||||
|
Mappings: []Mapping{{Pattern: "mc.local", Destination: "127.0.0.1:1"}},
|
||||||
|
})
|
||||||
|
|
||||||
|
done := make(chan error, 2)
|
||||||
|
go func() { _, err := c.pool.Dial(); done <- err }()
|
||||||
|
time.Sleep(200 * time.Millisecond) // let the first caller get into the dial
|
||||||
|
go func() { _, err := c.pool.Dial(); done <- err }()
|
||||||
|
|
||||||
|
// Both must give up on their own; neither may be stuck behind the other.
|
||||||
|
limit := time.After(HandshakeTimeout + 15*time.Second)
|
||||||
|
for i := 0; i < 2; i++ {
|
||||||
|
select {
|
||||||
|
case err := <-done:
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Dial succeeded against a hub that never answers")
|
||||||
|
}
|
||||||
|
case <-limit:
|
||||||
|
t.Fatalf("Dial #%d never returned: a stalled hub wedged the other caller", i+1)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestDialRespectsMaxTunnels pins the concurrency cap: once live+dialing
|
||||||
|
// equals maxTunnels, further Dial calls fail immediately rather than stacking.
|
||||||
|
func TestDialRespectsMaxTunnels(t *testing.T) {
|
||||||
|
p := newWorkerPool(&Client{}, 2)
|
||||||
|
p.conns[&WorkerConn{id: 1}] = struct{}{}
|
||||||
|
p.conns[&WorkerConn{id: 2}] = struct{}{}
|
||||||
|
if _, err := p.Dial(); err != errTooManyTunnels {
|
||||||
|
t.Fatalf("Dial at cap: got %v, want %v", err, errTooManyTunnels)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,353 @@
|
|||||||
|
package client
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"log"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/iceBear67/redapricot/client/wire"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Stream resumption (PROTOCOL.md §7.5).
|
||||||
|
//
|
||||||
|
// A worker conn is only the middle leg of the player it carries: when it dies,
|
||||||
|
// both terminal sockets are usually still perfectly healthy. Tearing the
|
||||||
|
// tunnel down therefore throws away working connections because a replaceable
|
||||||
|
// transport failed.
|
||||||
|
//
|
||||||
|
// Instead the stream parks: the destination socket stays open, the hub hangs the
|
||||||
|
// player socket, and the client reattaches over a fresh conn. Correctness rests
|
||||||
|
// on retransmission being byte-exact. Bytes handed to a dying socket are lost
|
||||||
|
// with no notification, and the frame cipher cannot be resynchronized, so each
|
||||||
|
// side replays from the offset the other reports it accepted.
|
||||||
|
|
||||||
|
var (
|
||||||
|
errResumeUnknown = errors.New("hub does not know this stream")
|
||||||
|
errResumeRaced = errors.New("hub has this stream bound to another conn")
|
||||||
|
errResumeRefused = errors.New("hub refused the reattach")
|
||||||
|
errResumeTimeout = errors.New("no RESUME_ACK from the hub")
|
||||||
|
errResumeConnLost = errors.New("the conn carrying the reattach died")
|
||||||
|
errResumeTooOld = errors.New("hub accepted past what we still hold")
|
||||||
|
errResumeNoResume = errors.New("hub does not support stream resumption")
|
||||||
|
)
|
||||||
|
|
||||||
|
// resumeGrace is how long a stream parked from a conn may keep trying, clamped
|
||||||
|
// under what the hub advertised. The client must always give up first: a hub
|
||||||
|
// that drops the player while we are still reattaching would leave us pumping a
|
||||||
|
// destination nobody is reading.
|
||||||
|
func (c *Client) resumeGrace(hubGrace time.Duration) time.Duration {
|
||||||
|
grace := c.cfg.resumeGrace()
|
||||||
|
if hubGrace > 0 && hubGrace < grace {
|
||||||
|
grace = hubGrace
|
||||||
|
}
|
||||||
|
return grace
|
||||||
|
}
|
||||||
|
|
||||||
|
// park suspends a stream whose worker conn died instead of destroying it, and
|
||||||
|
// starts trying to reattach. Reports false when the stream cannot be parked, in
|
||||||
|
// which case the caller tears it down as before.
|
||||||
|
//
|
||||||
|
// A stream already closing (finPending) is not parked: the hub has said the
|
||||||
|
// player is gone, so there is nothing left to preserve.
|
||||||
|
func (s *Stream) park(grace time.Duration) bool {
|
||||||
|
if !s.resumable {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
s.mu.Lock()
|
||||||
|
if s.closed || s.finPending {
|
||||||
|
s.mu.Unlock()
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
already := s.parked
|
||||||
|
s.parked = true
|
||||||
|
if s.stats != nil && !already {
|
||||||
|
s.parkedAt = time.Now()
|
||||||
|
}
|
||||||
|
s.mu.Unlock()
|
||||||
|
|
||||||
|
if already {
|
||||||
|
// A reattach was already in flight and had registered this stream on the
|
||||||
|
// conn that just died — which is how it got here at all. That attempt
|
||||||
|
// still owns the stream, so reporting failure would have the caller tear
|
||||||
|
// down a player that is mid-recovery. Fail its wait immediately rather
|
||||||
|
// than let it sit out the ack timeout: the grace budget is small, and
|
||||||
|
// spending ten seconds of it waiting on a socket that is already gone is
|
||||||
|
// the difference between reattaching and dropping the player.
|
||||||
|
s.deliverResume(resumeResult{err: errResumeConnLost})
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
go s.resumeLoop(grace)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// resumeLoop reattaches the stream, retrying until it succeeds or the grace
|
||||||
|
// period runs out.
|
||||||
|
//
|
||||||
|
// Attempts are started right up to the deadline rather than reserving a whole
|
||||||
|
// dial's worth of budget for the last one. Reserving it would be self-defeating
|
||||||
|
// — the grace and HandshakeTimeout are the same order of magnitude, so the
|
||||||
|
// reservation can consume the entire budget and leave no attempt at all — and
|
||||||
|
// overshooting is safe: an attempt that lands after the hub has dropped the
|
||||||
|
// player is answered with RST(unknown stream) and tears down cleanly.
|
||||||
|
func (s *Stream) resumeLoop(grace time.Duration) {
|
||||||
|
deadline := time.Now().Add(grace)
|
||||||
|
for {
|
||||||
|
if s.isClosed() {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !time.Now().Before(deadline) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
err := s.tryResume()
|
||||||
|
if err == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// errResumeRaced falls through to the retry below. The hub has this
|
||||||
|
// player bound to a conn that is not ours — a half-open conn whose
|
||||||
|
// death the hub has not yet learned, or a bind left behind by a racing
|
||||||
|
// attempt on a now-dead conn. There is no other live attempt: park is
|
||||||
|
// the only resumeLoop starter and it refuses to double-start. Retrying
|
||||||
|
// is safe precisely because nothing else owns the stream — the foreign
|
||||||
|
// bind dies with its conn, the hub re-parks, and a later RESUME lands.
|
||||||
|
// The grace deadline bounds the loop and expiry tears the stream down,
|
||||||
|
// so a hub that never re-parks cannot hang us forever.
|
||||||
|
if errors.Is(err, errResumeUnknown) || errors.Is(err, errResumeTooOld) {
|
||||||
|
// Terminal: the hub has no state for this stream (it restarted, the
|
||||||
|
// grace expired, or a load balancer sent us to a different instance).
|
||||||
|
log.Printf("stream resume abandoned: %v", err)
|
||||||
|
s.teardown(false)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-s.done:
|
||||||
|
return
|
||||||
|
case <-time.After(ResumeRetryDelay):
|
||||||
|
}
|
||||||
|
}
|
||||||
|
log.Printf("stream resume gave up after %s; closing destination", grace)
|
||||||
|
// No FIN: the only conns we could send it on are the ones that just failed
|
||||||
|
// us. The hub drops the hanging player when its own grace expires.
|
||||||
|
s.teardown(false)
|
||||||
|
}
|
||||||
|
|
||||||
|
// tryResume performs one reattach attempt: dial a fresh worker conn, send
|
||||||
|
// RESUME, and replay from wherever the hub says it got to.
|
||||||
|
func (s *Stream) tryResume() error {
|
||||||
|
wc, err := s.allocateForResume()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
s.mu.Lock()
|
||||||
|
cid := s.cid
|
||||||
|
accepted := s.acceptedOffset
|
||||||
|
delivered := s.deliveredOffset
|
||||||
|
wait := make(chan resumeResult, 1)
|
||||||
|
s.resumeWait = wait
|
||||||
|
s.mu.Unlock()
|
||||||
|
|
||||||
|
msg := wire.NewWriter().U8(MuxResume).Bytes(cid).
|
||||||
|
I64(accepted).I64(delivered).Out()
|
||||||
|
if err := wc.fc.WriteFrame(msg); err != nil {
|
||||||
|
s.abandonAttempt(wc)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
var res resumeResult
|
||||||
|
select {
|
||||||
|
case res = <-wait:
|
||||||
|
case <-s.done:
|
||||||
|
// Torn down while waiting. teardown only detaches the conn the stream
|
||||||
|
// was bound to, which is not this one, so the claim made above has to be
|
||||||
|
// withdrawn here or the new conn stays bound forever.
|
||||||
|
s.abandonAttempt(wc)
|
||||||
|
return errResumeRefused
|
||||||
|
case <-time.After(ResumeAckTimeout):
|
||||||
|
s.abandonAttempt(wc)
|
||||||
|
return errResumeTimeout
|
||||||
|
}
|
||||||
|
if res.err != nil {
|
||||||
|
s.abandonAttempt(wc)
|
||||||
|
return res.err
|
||||||
|
}
|
||||||
|
return s.completeResume(wc, res)
|
||||||
|
}
|
||||||
|
|
||||||
|
// allocateForResume dials a dedicated worker conn that will honour a reattach.
|
||||||
|
// 1:1: this must never land on someone else's tunnel.
|
||||||
|
func (s *Stream) allocateForResume() (*WorkerConn, error) {
|
||||||
|
for attempt := 0; attempt < dialAttempts; attempt++ {
|
||||||
|
wc, err := s.client.pool.Dial()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
// Re-checked per conn, not assumed from the dead one: this may be a
|
||||||
|
// different or restarted hub. Sending RESUME to a hub that does not know
|
||||||
|
// the frame type would hang the player for the rest of the grace waiting
|
||||||
|
// for an answer that is never coming.
|
||||||
|
if !wc.resume {
|
||||||
|
_ = wc.fc.Close()
|
||||||
|
return nil, errResumeNoResume
|
||||||
|
}
|
||||||
|
if wc.attach(s) {
|
||||||
|
return wc, nil
|
||||||
|
}
|
||||||
|
_ = wc.fc.Close()
|
||||||
|
}
|
||||||
|
return nil, errResumeRefused
|
||||||
|
}
|
||||||
|
|
||||||
|
// abandonAttempt withdraws a failed attempt from the conn it was made on and
|
||||||
|
// closes that conn — 1:1, it exists only for this attempt.
|
||||||
|
func (s *Stream) abandonAttempt(wc *WorkerConn) {
|
||||||
|
wc.detach()
|
||||||
|
_ = wc.fc.Close()
|
||||||
|
s.mu.Lock()
|
||||||
|
s.resumeWait = nil
|
||||||
|
s.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
// completeResume rebinds the stream to its new conn and replays what the hub is
|
||||||
|
// missing, holding sendMu throughout so live traffic cannot overtake the replay.
|
||||||
|
func (s *Stream) completeResume(wc *WorkerConn, res resumeResult) error {
|
||||||
|
s.sendMu.Lock()
|
||||||
|
// Delivery is a strictly stronger fact than credit — the hub only credits what
|
||||||
|
// it has delivered — so the reported offset can be adopted wholesale. Doing so
|
||||||
|
// also repairs the ledger: the grants destroyed by the outage are exactly the
|
||||||
|
// gap between the two, and without this the retained region would carry that
|
||||||
|
// dead prefix for the rest of the stream's life.
|
||||||
|
s.ackedOffset.Store(res.delivered)
|
||||||
|
s.un.advance(res.delivered)
|
||||||
|
replay := s.un.from(res.accepted)
|
||||||
|
if replay == nil {
|
||||||
|
s.sendMu.Unlock()
|
||||||
|
s.abandonAttempt(wc)
|
||||||
|
return errResumeTooOld
|
||||||
|
}
|
||||||
|
// Three offsets, three jobs, and conflating any two of them breaks something
|
||||||
|
// different.
|
||||||
|
//
|
||||||
|
// What to replay is measured from what the hub *accepted* — the bytes it
|
||||||
|
// never received. What the window should be is measured from what it
|
||||||
|
// *delivered*, because the window is a promise about undelivered bytes.
|
||||||
|
// It cannot be measured from what it *credited*: credit arrives as deltas,
|
||||||
|
// and the grants in flight when the connection died are gone for good, so a
|
||||||
|
// window derived from them would be permanently short — and, when a full
|
||||||
|
// window was outstanding at the drop, permanently zero. That is a deadlock,
|
||||||
|
// not a slowdown: no credit can arrive because nothing can be sent.
|
||||||
|
outstanding := s.un.length()
|
||||||
|
replayed := s.un.end() - res.accepted
|
||||||
|
|
||||||
|
// Publish the new binding before any frame goes out on it.
|
||||||
|
s.wc.Store(wc)
|
||||||
|
|
||||||
|
s.mu.Lock()
|
||||||
|
// Restated, not patched. The window is a delta ledger and the outage tore a
|
||||||
|
// hole in it; deriving it afresh from the delivered offset closes the hole
|
||||||
|
// exactly, whatever was lost.
|
||||||
|
s.sendWnd = wc.sendWndInit - int(outstanding)
|
||||||
|
if s.sendWnd < 0 {
|
||||||
|
s.sendWnd = 0
|
||||||
|
}
|
||||||
|
// Symmetrically, our own pending credit is discarded rather than flushed:
|
||||||
|
// the delivered offset we reported already tells the hub everything those
|
||||||
|
// deltas would have, and sending both would grant the same bytes twice.
|
||||||
|
s.consumed = 0
|
||||||
|
if len(res.cid) == CIDLen {
|
||||||
|
s.cid = res.cid // fresh capability, so a CID is never reusable twice
|
||||||
|
}
|
||||||
|
s.parked = false
|
||||||
|
s.resumeWait = nil
|
||||||
|
owedFin := s.finToHub
|
||||||
|
s.cond.Broadcast() // release acquireSendWnd and any parked writer
|
||||||
|
s.mu.Unlock()
|
||||||
|
|
||||||
|
for len(replay) > 0 {
|
||||||
|
n := len(replay)
|
||||||
|
if n > s.client.chunk {
|
||||||
|
n = s.client.chunk
|
||||||
|
}
|
||||||
|
if err := wc.sendData(replay[:n]); err != nil {
|
||||||
|
// The conn died mid-replay. The stream is still resumable, but not
|
||||||
|
// from this conn — put it back in the parked state before returning
|
||||||
|
// so the conn's teardown takes park()'s already-branch instead of
|
||||||
|
// starting a second resumeLoop. The loop we came from keeps
|
||||||
|
// retrying with the fresh CID, which the hub re-parked alongside
|
||||||
|
// the stream when this conn died. Without this re-park the flag
|
||||||
|
// cleared above would let two loops race one stream, and a racing
|
||||||
|
// RST would then strand it with neither loop alive.
|
||||||
|
s.mu.Lock()
|
||||||
|
s.parked = true
|
||||||
|
s.mu.Unlock()
|
||||||
|
s.sendMu.Unlock()
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
replay = replay[n:]
|
||||||
|
}
|
||||||
|
s.sendMu.Unlock()
|
||||||
|
|
||||||
|
// Counted only once the replay actually landed: a failed reattach above
|
||||||
|
// returns before this, so a conn dying mid-replay does not inflate the
|
||||||
|
// resume count with an attempt that never completed.
|
||||||
|
if s.stats != nil {
|
||||||
|
s.mu.Lock()
|
||||||
|
s.stats.resumes++
|
||||||
|
s.stats.hung += time.Since(s.parkedAt)
|
||||||
|
s.stats.replayBytes += replayed
|
||||||
|
s.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
// A destination that closed while we were parked owed the hub a FIN that had
|
||||||
|
// nowhere to go at the time.
|
||||||
|
if owedFin {
|
||||||
|
wc.sendFin()
|
||||||
|
s.teardown(false)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
log.Printf("stream %s resumed (%d bytes replayed, %d outstanding)", s.name(), replayed, outstanding)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// deliverResume hands an answer to a reattach that is waiting for one. Reports
|
||||||
|
// false when no attempt was in flight, so the caller can treat the frame as it
|
||||||
|
// would on any live stream.
|
||||||
|
func (s *Stream) deliverResume(res resumeResult) bool {
|
||||||
|
s.mu.Lock()
|
||||||
|
ch := s.resumeWait
|
||||||
|
s.resumeWait = nil
|
||||||
|
s.mu.Unlock()
|
||||||
|
if ch == nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
ch <- res // buffered, and read at most once per attempt
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// onRst applies an RST, using the reason to tell a stream that is genuinely gone
|
||||||
|
// from one that a racing reattach has taken over.
|
||||||
|
func (s *Stream) onRst(reason int) {
|
||||||
|
err := errResumeRefused
|
||||||
|
switch reason {
|
||||||
|
case RstUnknownStream:
|
||||||
|
err = errResumeUnknown
|
||||||
|
case RstAlreadyBound:
|
||||||
|
err = errResumeRaced
|
||||||
|
}
|
||||||
|
if s.deliverResume(resumeResult{err: err}) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.teardown(false)
|
||||||
|
}
|
||||||
|
|
||||||
|
// noteFinWhileParked records a FIN the stream owes the hub but cannot send,
|
||||||
|
// because the only conn it has is the one that just died. Reports false when the
|
||||||
|
// stream is not parked and the caller should send it normally.
|
||||||
|
func (s *Stream) noteFinWhileParked() bool {
|
||||||
|
s.mu.Lock()
|
||||||
|
defer s.mu.Unlock()
|
||||||
|
if !s.parked {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
s.finToHub = true
|
||||||
|
return true
|
||||||
|
}
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
package client
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The retained region is the one real cost stream resumption adds to the send
|
||||||
|
// path: a chunk has to survive past the frame write, so it is copied. These
|
||||||
|
// pin both halves of that claim — that the copy is the only cost, and that
|
||||||
|
// disabling the feature removes it entirely rather than merely shrinking it.
|
||||||
|
|
||||||
|
func benchStream(resumable bool) *Stream {
|
||||||
|
s := &Stream{resumable: resumable}
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
// BenchmarkRetainChunk measures what emit adds over a bare frame write: the
|
||||||
|
// trim-and-append into the retained region. Compare the two variants; the delta
|
||||||
|
// is the per-byte copy the feature costs.
|
||||||
|
func BenchmarkRetainChunk(b *testing.B) {
|
||||||
|
chunk := make([]byte, DataChunkSize)
|
||||||
|
window := int64(DefaultStreamWindow)
|
||||||
|
|
||||||
|
b.Run("resume-on", func(b *testing.B) {
|
||||||
|
s := benchStream(true)
|
||||||
|
b.SetBytes(int64(len(chunk)))
|
||||||
|
b.ReportAllocs()
|
||||||
|
b.ResetTimer()
|
||||||
|
for i := 0; i < b.N; i++ {
|
||||||
|
// Model the steady state: credit trails one window behind, so the
|
||||||
|
// buffer trims about as fast as it grows and stays bounded.
|
||||||
|
acked := s.un.end() - window
|
||||||
|
if acked < 0 {
|
||||||
|
acked = 0
|
||||||
|
}
|
||||||
|
s.un.advance(acked)
|
||||||
|
s.un.append(chunk)
|
||||||
|
}
|
||||||
|
if got := int64(s.un.length()); got > window+int64(len(chunk)) {
|
||||||
|
b.Fatalf("retained region grew past one window: %d", got)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
b.Run("resume-off", func(b *testing.B) {
|
||||||
|
s := benchStream(false)
|
||||||
|
b.SetBytes(int64(len(chunk)))
|
||||||
|
b.ReportAllocs()
|
||||||
|
b.ResetTimer()
|
||||||
|
for i := 0; i < b.N; i++ {
|
||||||
|
if s.resumable {
|
||||||
|
s.un.advance(0)
|
||||||
|
s.un.append(chunk)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestResumeDisabledAllocatesNothing pins the off switch at the level that
|
||||||
|
// matters. It is easy for a feature flag to stop the wire behaviour while
|
||||||
|
// leaving the bookkeeping running, which would keep the memory cost and the
|
||||||
|
// per-byte copy for a user who explicitly turned it off — a partial revert that
|
||||||
|
// nobody would notice.
|
||||||
|
func TestResumeDisabledAllocatesNothing(t *testing.T) {
|
||||||
|
chunk := make([]byte, DataChunkSize)
|
||||||
|
s := benchStream(false)
|
||||||
|
|
||||||
|
allocs := testing.AllocsPerRun(1000, func() {
|
||||||
|
if s.resumable {
|
||||||
|
s.un.advance(0)
|
||||||
|
s.un.append(chunk)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
if allocs != 0 {
|
||||||
|
t.Fatalf("resume disabled still allocated %.1f times per send", allocs)
|
||||||
|
}
|
||||||
|
if s.un.buf != nil {
|
||||||
|
t.Fatalf("resume disabled still allocated a retained region of %d bytes", cap(s.un.buf))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRetainedRegionStaysWithinWindow is the memory bound the design rests on:
|
||||||
|
// flow control already caps outstanding bytes at one window, so the retained
|
||||||
|
// region needs no cap of its own. If that ever stopped holding, a busy stream
|
||||||
|
// would grow without limit and the hub would be the first to notice.
|
||||||
|
func TestRetainedRegionStaysWithinWindow(t *testing.T) {
|
||||||
|
const window = DefaultStreamWindow
|
||||||
|
chunk := make([]byte, DataChunkSize)
|
||||||
|
var u unackedBuf
|
||||||
|
|
||||||
|
for i := 0; i < 5000; i++ {
|
||||||
|
// A sender may never have more than one window outstanding, which is
|
||||||
|
// exactly what acquireSendWnd enforces before emit is ever reached.
|
||||||
|
if u.length()+len(chunk) > window {
|
||||||
|
u.advance(u.base() + int64(len(chunk)))
|
||||||
|
}
|
||||||
|
u.append(chunk)
|
||||||
|
if u.length() > window {
|
||||||
|
t.Fatalf("round %d: retained %d bytes for a %d-byte window", i, u.length(), window)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if cap(u.buf) > 4*window {
|
||||||
|
t.Fatalf("backing array grew to %d for a %d-byte window", cap(u.buf), window)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,268 @@
|
|||||||
|
package client
|
||||||
|
|
||||||
|
import (
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// minShaperWait floors the dispatcher's sleep so floating-point dust in the
|
||||||
|
// token arithmetic cannot spin it.
|
||||||
|
const minShaperWait = time.Millisecond
|
||||||
|
|
||||||
|
// Shaper caps the aggregate rate at which the client writes DATA to the hub and
|
||||||
|
// divides that budget across streams.
|
||||||
|
//
|
||||||
|
// The credit windows of PROTOCOL.md §7.3 bound how many bytes may be *in flight*
|
||||||
|
// per stream; they say nothing about bytes per *second*. That is the gap this
|
||||||
|
// fills. On a residential uplink one player loading chunks will otherwise
|
||||||
|
// saturate the line and push every other player's keepalive past its timeout.
|
||||||
|
//
|
||||||
|
// Two mechanisms are layered:
|
||||||
|
//
|
||||||
|
// - A token bucket sets the long-run rate and the size of the burst that may
|
||||||
|
// be spent after an idle period.
|
||||||
|
// - Start-time fair queueing decides who spends those tokens. A global virtual
|
||||||
|
// clock advances with each grant; every stream remembers the virtual time at
|
||||||
|
// which its last request finished. A request is stamped
|
||||||
|
// max(share.vfinish, vclock) and the lowest stamp is served first, so a
|
||||||
|
// stream that keeps sending pushes its own stamp further out and yields to
|
||||||
|
// quieter streams. The clamp to vclock is what keeps bursts cheap: a stream
|
||||||
|
// returning from idle is pulled back to the head of the clock, so it cannot
|
||||||
|
// hoard credit while it was idle, but it is not punished for the idleness
|
||||||
|
// either. One stream alone gets the whole rate.
|
||||||
|
//
|
||||||
|
// A nil *Shaper means "no limit"; every method short-circuits, so call sites do
|
||||||
|
// not branch.
|
||||||
|
type Shaper struct {
|
||||||
|
rate float64 // bytes per second
|
||||||
|
burst float64 // token bucket capacity, bytes
|
||||||
|
chunk int // how much a caller should request at a time
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
tokens float64
|
||||||
|
last time.Time
|
||||||
|
vclock float64 // virtual time, in bytes of service granted
|
||||||
|
waiting []*shaperReq // unordered; the dispatcher scans for the lowest vstart
|
||||||
|
|
||||||
|
wake chan struct{} // cap 1, non-blocking: nudges the dispatcher
|
||||||
|
done chan struct{}
|
||||||
|
once sync.Once
|
||||||
|
}
|
||||||
|
|
||||||
|
// shaperShare is one stream's position in the fair queue. It lives on the
|
||||||
|
// Stream and dies with it; a fresh share starts at zero and is clamped up to
|
||||||
|
// the current virtual clock on its first request.
|
||||||
|
type shaperShare struct{ vfinish float64 }
|
||||||
|
|
||||||
|
// shaperReq is one pending Acquire. granted and membership in Shaper.waiting
|
||||||
|
// are both guarded by Shaper.mu.
|
||||||
|
type shaperReq struct {
|
||||||
|
n int
|
||||||
|
vstart float64
|
||||||
|
grant chan struct{}
|
||||||
|
granted bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewShaper builds a shaper for the given rate. A non-positive rate returns nil,
|
||||||
|
// which every method treats as "unlimited".
|
||||||
|
func NewShaper(bytesPerSec int64) *Shaper {
|
||||||
|
if bytesPerSec <= 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
rate := float64(bytesPerSec)
|
||||||
|
|
||||||
|
burst := rate * ShaperBurstSeconds
|
||||||
|
// The floor is a correctness constraint, not a preference: a request larger
|
||||||
|
// than the bucket could never be afforded and would park forever.
|
||||||
|
if burst < MinShaperBurst {
|
||||||
|
burst = MinShaperBurst
|
||||||
|
}
|
||||||
|
if burst > MaxShaperBurst {
|
||||||
|
burst = MaxShaperBurst
|
||||||
|
}
|
||||||
|
|
||||||
|
chunk := int(rate * ShaperSliceSeconds)
|
||||||
|
if chunk < MinShaperChunk {
|
||||||
|
chunk = MinShaperChunk
|
||||||
|
}
|
||||||
|
if chunk > DataChunkSize {
|
||||||
|
chunk = DataChunkSize
|
||||||
|
}
|
||||||
|
|
||||||
|
sh := &Shaper{
|
||||||
|
rate: rate,
|
||||||
|
burst: burst,
|
||||||
|
chunk: chunk,
|
||||||
|
tokens: burst,
|
||||||
|
last: time.Now(),
|
||||||
|
wake: make(chan struct{}, 1),
|
||||||
|
done: make(chan struct{}),
|
||||||
|
}
|
||||||
|
go sh.dispatch()
|
||||||
|
return sh
|
||||||
|
}
|
||||||
|
|
||||||
|
// chunkSize is how many bytes a sender should offer per request. It is sized to
|
||||||
|
// ShaperSliceSeconds of transmission so no stream holds the link for long before
|
||||||
|
// the scheduler can switch: at 1 Mbps a full 32 KiB chunk takes ~256 ms, which is
|
||||||
|
// enough dead air to drag other players towards a keepalive timeout.
|
||||||
|
func (sh *Shaper) chunkSize() int {
|
||||||
|
if sh == nil {
|
||||||
|
return DataChunkSize
|
||||||
|
}
|
||||||
|
return sh.chunk
|
||||||
|
}
|
||||||
|
|
||||||
|
// Acquire blocks until n bytes of bandwidth budget are available for the stream
|
||||||
|
// owning share. It returns false only when cancel fires first, in which case
|
||||||
|
// nothing was charged.
|
||||||
|
//
|
||||||
|
// cancel is the stream's done channel: a stream torn down while parked here must
|
||||||
|
// not keep a goroutine (and its Stream) alive waiting for tokens it will never
|
||||||
|
// use.
|
||||||
|
func (sh *Shaper) Acquire(share *shaperShare, n int, cancel <-chan struct{}) bool {
|
||||||
|
if sh == nil || n <= 0 {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
req := &shaperReq{n: n, grant: make(chan struct{})}
|
||||||
|
|
||||||
|
sh.mu.Lock()
|
||||||
|
// Stamp the request and reserve this stream's slot in virtual time up front,
|
||||||
|
// so a stream cannot queue many requests at the same cheap stamp.
|
||||||
|
req.vstart = share.vfinish
|
||||||
|
if req.vstart < sh.vclock {
|
||||||
|
req.vstart = sh.vclock
|
||||||
|
}
|
||||||
|
share.vfinish = req.vstart + float64(n)
|
||||||
|
sh.waiting = append(sh.waiting, req)
|
||||||
|
sh.mu.Unlock()
|
||||||
|
sh.nudge()
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-req.grant:
|
||||||
|
return true
|
||||||
|
case <-sh.done:
|
||||||
|
// Shaping stopped: let live traffic through rather than stalling it.
|
||||||
|
sh.mu.Lock()
|
||||||
|
sh.removeLocked(req)
|
||||||
|
sh.mu.Unlock()
|
||||||
|
return true
|
||||||
|
case <-cancel:
|
||||||
|
sh.mu.Lock()
|
||||||
|
granted := req.granted
|
||||||
|
if !granted {
|
||||||
|
sh.removeLocked(req)
|
||||||
|
}
|
||||||
|
sh.mu.Unlock()
|
||||||
|
return granted
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stop shuts the dispatcher down and releases everyone parked in Acquire.
|
||||||
|
func (sh *Shaper) Stop() {
|
||||||
|
if sh == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
sh.once.Do(func() { close(sh.done) })
|
||||||
|
}
|
||||||
|
|
||||||
|
// dispatch is the single goroutine that hands out tokens. It sleeps exactly as
|
||||||
|
// long as the next waiter needs rather than polling on a fixed tick, so an idle
|
||||||
|
// shaper costs nothing.
|
||||||
|
func (sh *Shaper) dispatch() {
|
||||||
|
for {
|
||||||
|
wait := sh.grantReady()
|
||||||
|
var tick <-chan time.Time
|
||||||
|
var timer *time.Timer
|
||||||
|
if wait > 0 {
|
||||||
|
timer = time.NewTimer(wait)
|
||||||
|
tick = timer.C
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-tick:
|
||||||
|
case <-sh.wake:
|
||||||
|
case <-sh.done:
|
||||||
|
if timer != nil {
|
||||||
|
timer.Stop()
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if timer != nil {
|
||||||
|
timer.Stop()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// grantReady refills the bucket and grants every waiter it can afford, lowest
|
||||||
|
// virtual start time first. It returns how long until the next waiter becomes
|
||||||
|
// affordable, or 0 when nothing is pending.
|
||||||
|
func (sh *Shaper) grantReady() time.Duration {
|
||||||
|
sh.mu.Lock()
|
||||||
|
defer sh.mu.Unlock()
|
||||||
|
|
||||||
|
now := time.Now()
|
||||||
|
if elapsed := now.Sub(sh.last); elapsed > 0 {
|
||||||
|
sh.tokens += sh.rate * elapsed.Seconds()
|
||||||
|
if sh.tokens > sh.burst {
|
||||||
|
sh.tokens = sh.burst
|
||||||
|
}
|
||||||
|
sh.last = now
|
||||||
|
}
|
||||||
|
|
||||||
|
for {
|
||||||
|
req := sh.headLocked()
|
||||||
|
if req == nil {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
// Callers stay under chunkSize, which NewShaper keeps below the bucket.
|
||||||
|
// Should a future caller not, wait for a full bucket rather than for a
|
||||||
|
// token count that can never be reached, and let the balance go negative:
|
||||||
|
// the debt is repaid by the next refill, so the long-run rate still holds.
|
||||||
|
need := min(float64(req.n), sh.burst)
|
||||||
|
if need > sh.tokens {
|
||||||
|
wait := time.Duration((need - sh.tokens) / sh.rate * float64(time.Second))
|
||||||
|
if wait < minShaperWait {
|
||||||
|
wait = minShaperWait
|
||||||
|
}
|
||||||
|
return wait
|
||||||
|
}
|
||||||
|
sh.tokens -= float64(req.n)
|
||||||
|
// The clock follows the request being served, never runs ahead of it.
|
||||||
|
if req.vstart > sh.vclock {
|
||||||
|
sh.vclock = req.vstart
|
||||||
|
}
|
||||||
|
req.granted = true
|
||||||
|
sh.removeLocked(req)
|
||||||
|
close(req.grant)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// headLocked returns the pending request with the lowest virtual start time.
|
||||||
|
// A linear scan is deliberate: the queue holds at most one entry per live
|
||||||
|
// stream (tens, not thousands), so a heap would cost more in complexity than it
|
||||||
|
// saves in comparisons.
|
||||||
|
func (sh *Shaper) headLocked() *shaperReq {
|
||||||
|
var best *shaperReq
|
||||||
|
for _, w := range sh.waiting {
|
||||||
|
if best == nil || w.vstart < best.vstart {
|
||||||
|
best = w
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return best
|
||||||
|
}
|
||||||
|
|
||||||
|
func (sh *Shaper) removeLocked(req *shaperReq) {
|
||||||
|
for i, w := range sh.waiting {
|
||||||
|
if w == req {
|
||||||
|
sh.waiting = append(sh.waiting[:i], sh.waiting[i+1:]...)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (sh *Shaper) nudge() {
|
||||||
|
select {
|
||||||
|
case sh.wake <- struct{}{}:
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,231 @@
|
|||||||
|
package client
|
||||||
|
|
||||||
|
import (
|
||||||
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestParseBandwidth(t *testing.T) {
|
||||||
|
cases := []struct {
|
||||||
|
in string
|
||||||
|
want int64
|
||||||
|
}{
|
||||||
|
{"", 0},
|
||||||
|
{"20mbps", 2_500_000},
|
||||||
|
{"20Mbps", 2_500_000},
|
||||||
|
{"20 mbps", 2_500_000},
|
||||||
|
{"1.5mbit", 187_500},
|
||||||
|
{"512kbps", 64_000},
|
||||||
|
{"1gbps", 125_000_000},
|
||||||
|
{"2MB/s", 2 << 20},
|
||||||
|
{"500kb/s", 500 << 10},
|
||||||
|
{"1GB/s", 1 << 30},
|
||||||
|
{"8bps", 1},
|
||||||
|
{"4096b/s", 4096},
|
||||||
|
{"1500000", 1_500_000}, // bare number is already bytes/sec
|
||||||
|
}
|
||||||
|
for _, c := range cases {
|
||||||
|
got, err := parseBandwidth(c.in)
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("parseBandwidth(%q): unexpected error %v", c.in, err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if got != c.want {
|
||||||
|
t.Errorf("parseBandwidth(%q) = %d, want %d", c.in, got, c.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, bad := range []string{"fast", "20megabits", "-5mbps", "0", "0mbps", "mbps", "20 mb ps"} {
|
||||||
|
if _, err := parseBandwidth(bad); err == nil {
|
||||||
|
t.Errorf("parseBandwidth(%q): expected an error", bad)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A nil shaper is the "unlimited" case and must be safe on every path, because
|
||||||
|
// call sites deliberately do not branch on it.
|
||||||
|
func TestNilShaperIsUnlimited(t *testing.T) {
|
||||||
|
var sh *Shaper
|
||||||
|
if sh = NewShaper(0); sh != nil {
|
||||||
|
t.Fatal("NewShaper(0) should return nil")
|
||||||
|
}
|
||||||
|
if got := sh.chunkSize(); got != DataChunkSize {
|
||||||
|
t.Errorf("nil chunkSize = %d, want %d", got, DataChunkSize)
|
||||||
|
}
|
||||||
|
if !sh.Acquire(&shaperShare{}, 1<<20, nil) {
|
||||||
|
t.Error("nil Acquire should always succeed")
|
||||||
|
}
|
||||||
|
sh.Stop() // must not panic
|
||||||
|
}
|
||||||
|
|
||||||
|
// The virtual-time bookkeeping is what makes the shaper fair, so assert it
|
||||||
|
// directly. The rate is high enough that tokens never bind, leaving only the
|
||||||
|
// stamping under test — no timing, no flakiness.
|
||||||
|
func TestShaperIdleStreamCannotHoardCredit(t *testing.T) {
|
||||||
|
sh := NewShaper(1 << 30)
|
||||||
|
defer sh.Stop()
|
||||||
|
|
||||||
|
var heavy, light shaperShare
|
||||||
|
for i := 0; i < 10; i++ {
|
||||||
|
if !sh.Acquire(&heavy, 1000, nil) {
|
||||||
|
t.Fatal("acquire failed")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if heavy.vfinish != 10000 {
|
||||||
|
t.Errorf("heavy.vfinish = %v, want 10000", heavy.vfinish)
|
||||||
|
}
|
||||||
|
sh.mu.Lock()
|
||||||
|
vclock := sh.vclock
|
||||||
|
sh.mu.Unlock()
|
||||||
|
if vclock != 9000 {
|
||||||
|
t.Errorf("vclock = %v, want 9000 (the stamp of the last request served)", vclock)
|
||||||
|
}
|
||||||
|
|
||||||
|
// light was idle for all of it. Its stale vfinish of 0 must be clamped up to
|
||||||
|
// the current clock: it may not bank the virtual time it never spent, which
|
||||||
|
// is what would let it starve heavy on return.
|
||||||
|
if !sh.Acquire(&light, 1000, nil) {
|
||||||
|
t.Fatal("acquire failed")
|
||||||
|
}
|
||||||
|
if light.vfinish != vclock+1000 {
|
||||||
|
t.Errorf("light.vfinish = %v, want %v (clamped to the clock, not 1000)", light.vfinish, vclock+1000)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestShaperEnforcesRate(t *testing.T) {
|
||||||
|
const rate = 1 << 20 // 1 MiB/s
|
||||||
|
sh := NewShaper(rate)
|
||||||
|
defer sh.Stop()
|
||||||
|
|
||||||
|
var share shaperShare
|
||||||
|
const total = 512 << 10
|
||||||
|
const chunk = 8 << 10
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
for sent := 0; sent < total; sent += chunk {
|
||||||
|
if !sh.Acquire(&share, chunk, nil) {
|
||||||
|
t.Fatal("acquire failed")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
elapsed := time.Since(start)
|
||||||
|
|
||||||
|
// The bucket starts full, so the burst is free and only the remainder is
|
||||||
|
// paced: (512 KiB - 200 KiB) / 1 MiB/s ≈ 300 ms. Bounds are wide on purpose.
|
||||||
|
if elapsed < 200*time.Millisecond {
|
||||||
|
t.Errorf("sent %d bytes at %d B/s in only %v; the cap is not being enforced", total, rate, elapsed)
|
||||||
|
}
|
||||||
|
if elapsed > time.Second {
|
||||||
|
t.Errorf("took %v, far longer than the ~300ms the rate implies", elapsed)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// An idle stream must be able to spend the banked burst at once, otherwise a
|
||||||
|
// player joining pays for the cap in visible chunk-loading latency.
|
||||||
|
func TestShaperAllowsBurst(t *testing.T) {
|
||||||
|
sh := NewShaper(1 << 20)
|
||||||
|
defer sh.Stop()
|
||||||
|
|
||||||
|
var share shaperShare
|
||||||
|
start := time.Now()
|
||||||
|
for i := 0; i < 6; i++ {
|
||||||
|
if !sh.Acquire(&share, 32<<10, nil) { // 192 KiB, inside the 200 KiB bucket
|
||||||
|
t.Fatal("acquire failed")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if elapsed := time.Since(start); elapsed > 100*time.Millisecond {
|
||||||
|
t.Errorf("burst of 192 KiB took %v; the bucket should have covered it instantly", elapsed)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The point of the whole exercise: a stream that never stops asking must not
|
||||||
|
// crowd another one out.
|
||||||
|
func TestShaperSharesFairlyBetweenStreams(t *testing.T) {
|
||||||
|
sh := NewShaper(1 << 20)
|
||||||
|
defer sh.Stop()
|
||||||
|
|
||||||
|
stop := make(chan struct{})
|
||||||
|
var counts [2]atomic.Int64
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for i := range counts {
|
||||||
|
wg.Add(1)
|
||||||
|
go func(i int) {
|
||||||
|
defer wg.Done()
|
||||||
|
var share shaperShare
|
||||||
|
for {
|
||||||
|
if !sh.Acquire(&share, 4<<10, stop) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
counts[i].Add(4 << 10)
|
||||||
|
}
|
||||||
|
}(i)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Spend the token bucket before measuring, the way the idle-credit test
|
||||||
|
// raises the rate so tokens never bind: isolate the property under test.
|
||||||
|
//
|
||||||
|
// While the bucket has tokens there is no queue to arbitrate — every request
|
||||||
|
// is granted the moment it arrives, and being the stream that is owed service
|
||||||
|
// only helps when both are enqueued at the same instant. The burst is
|
||||||
|
// therefore first-come-first-served by construction, and at 0.2s of
|
||||||
|
// transmission it is a quarter of a 600ms window, enough to swamp the result:
|
||||||
|
// a run where one goroutine happened to win the bucket landed at 520192 vs
|
||||||
|
// 315392, which is exactly "one took the whole burst, then the two split the
|
||||||
|
// remainder evenly".
|
||||||
|
//
|
||||||
|
// Fairness here is a steady-state property, and that is what matters in
|
||||||
|
// practice — the bucket is empty whenever the link is actually busy.
|
||||||
|
time.Sleep(250 * time.Millisecond)
|
||||||
|
counts[0].Store(0)
|
||||||
|
counts[1].Store(0)
|
||||||
|
|
||||||
|
time.Sleep(600 * time.Millisecond)
|
||||||
|
close(stop)
|
||||||
|
wg.Wait()
|
||||||
|
|
||||||
|
a, b := counts[0].Load(), counts[1].Load()
|
||||||
|
if a == 0 || b == 0 {
|
||||||
|
t.Fatalf("one stream was starved entirely: %d vs %d", a, b)
|
||||||
|
}
|
||||||
|
lo, hi := min(a, b), max(a, b)
|
||||||
|
t.Logf("steady-state split: %d vs %d bytes (%.3fx)", a, b, float64(hi)/float64(lo))
|
||||||
|
if float64(hi) > 1.35*float64(lo) {
|
||||||
|
t.Errorf("unfair split: %d vs %d bytes (>35%% apart)", a, b)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A stream torn down while parked must release immediately and leave no trace
|
||||||
|
// in the queue, or its goroutine (and the Stream it closes over) leaks.
|
||||||
|
func TestShaperAcquireCancels(t *testing.T) {
|
||||||
|
sh := NewShaper(MinBandwidth) // 8 KiB/s: a parked request would wait seconds
|
||||||
|
defer sh.Stop()
|
||||||
|
|
||||||
|
var share shaperShare
|
||||||
|
if !sh.Acquire(&share, MinShaperBurst, nil) { // drain the bucket
|
||||||
|
t.Fatal("acquire failed")
|
||||||
|
}
|
||||||
|
|
||||||
|
cancel := make(chan struct{})
|
||||||
|
result := make(chan bool, 1)
|
||||||
|
go func() { result <- sh.Acquire(&share, 32<<10, cancel) }()
|
||||||
|
|
||||||
|
time.Sleep(50 * time.Millisecond)
|
||||||
|
close(cancel)
|
||||||
|
|
||||||
|
select {
|
||||||
|
case ok := <-result:
|
||||||
|
if ok {
|
||||||
|
t.Error("Acquire returned true after cancellation")
|
||||||
|
}
|
||||||
|
case <-time.After(time.Second):
|
||||||
|
t.Fatal("Acquire did not return after its cancel channel closed")
|
||||||
|
}
|
||||||
|
|
||||||
|
sh.mu.Lock()
|
||||||
|
n := len(sh.waiting)
|
||||||
|
sh.mu.Unlock()
|
||||||
|
if n != 0 {
|
||||||
|
t.Errorf("%d cancelled request(s) left in the queue", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
+221
@@ -0,0 +1,221 @@
|
|||||||
|
package client
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"log"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Performance diagnostics.
|
||||||
|
//
|
||||||
|
// The question an operator actually has is "why is this tunnel slow?", and
|
||||||
|
// nothing here could answer it before. A stream that is not moving bytes is
|
||||||
|
// blocked on exactly one of three things:
|
||||||
|
//
|
||||||
|
// - the flow-control window — the peer is not draining to its terminal
|
||||||
|
// socket, so the bottleneck is past the tunnel (a struggling game server, a
|
||||||
|
// player on a bad link);
|
||||||
|
// - the shaper — the configured bandwidth cap is the binding constraint, and
|
||||||
|
// raising it is the fix;
|
||||||
|
// - the peer socket itself — bytes move, but slowly, which points at the path
|
||||||
|
// rather than at either end.
|
||||||
|
//
|
||||||
|
// Those three are indistinguishable from throughput alone and call for
|
||||||
|
// completely different responses, so they are counted apart.
|
||||||
|
//
|
||||||
|
// Cost. Both structs are nil unless statsIntervalMs is set, so the default is a
|
||||||
|
// single predictable branch per event and no allocation at all. When enabled,
|
||||||
|
// counters sit under locks the code already holds; only the frame counters use
|
||||||
|
// atomics, because the read loop must never queue behind a send. A clock is read
|
||||||
|
// only when a goroutine is about to block, never per chunk — if nothing stalls,
|
||||||
|
// nothing is timed.
|
||||||
|
|
||||||
|
// streamStats accumulates one stream's lifetime. Guarded by Stream.mu.
|
||||||
|
type streamStats struct {
|
||||||
|
opened time.Time
|
||||||
|
|
||||||
|
bytesUp int64 // destination -> hub
|
||||||
|
bytesDown int64 // hub -> destination
|
||||||
|
|
||||||
|
windowStall time.Duration // blocked with no send credit
|
||||||
|
shaperStall time.Duration // blocked on the bandwidth cap
|
||||||
|
qPeak int // high-water mark of the receive queue
|
||||||
|
|
||||||
|
resumes int
|
||||||
|
hung time.Duration // total time parked awaiting a reattach
|
||||||
|
replayBytes int64
|
||||||
|
}
|
||||||
|
|
||||||
|
// connStats accumulates one worker conn's lifetime.
|
||||||
|
type connStats struct {
|
||||||
|
opened time.Time
|
||||||
|
framesIn atomic.Int64
|
||||||
|
framesOut atomic.Int64
|
||||||
|
writeErrs atomic.Int64
|
||||||
|
|
||||||
|
// Round-trip time of the worker heartbeat. The probe already carries a
|
||||||
|
// timestamp that the peer echoes and both sides currently throw away, so
|
||||||
|
// this measures tunnel latency for no added cost.
|
||||||
|
mu sync.Mutex
|
||||||
|
rttLast time.Duration
|
||||||
|
rttMin time.Duration
|
||||||
|
rttMax time.Duration
|
||||||
|
rttSum time.Duration
|
||||||
|
rttN int64
|
||||||
|
}
|
||||||
|
|
||||||
|
func (cs *connStats) observeRTT(d time.Duration) {
|
||||||
|
if cs == nil || d < 0 {
|
||||||
|
return // a nonce we cannot read as one of our own timestamps
|
||||||
|
}
|
||||||
|
cs.mu.Lock()
|
||||||
|
defer cs.mu.Unlock()
|
||||||
|
cs.rttLast = d
|
||||||
|
if cs.rttN == 0 || d < cs.rttMin {
|
||||||
|
cs.rttMin = d
|
||||||
|
}
|
||||||
|
if d > cs.rttMax {
|
||||||
|
cs.rttMax = d
|
||||||
|
}
|
||||||
|
cs.rttSum += d
|
||||||
|
cs.rttN++
|
||||||
|
}
|
||||||
|
|
||||||
|
func (cs *connStats) rtt() (last, min, avg, max time.Duration) {
|
||||||
|
cs.mu.Lock()
|
||||||
|
defer cs.mu.Unlock()
|
||||||
|
if cs.rttN == 0 {
|
||||||
|
return 0, 0, 0, 0
|
||||||
|
}
|
||||||
|
return cs.rttLast, cs.rttMin, cs.rttSum / time.Duration(cs.rttN), cs.rttMax
|
||||||
|
}
|
||||||
|
|
||||||
|
// stallClock times a block without charging the path that does not block: the
|
||||||
|
// clock is read only once a wait is actually about to happen.
|
||||||
|
type stallClock struct{ start time.Time }
|
||||||
|
|
||||||
|
func (t *stallClock) begin(on bool) {
|
||||||
|
if on && t.start.IsZero() {
|
||||||
|
t.start = time.Now()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *stallClock) elapsed() time.Duration {
|
||||||
|
if t.start.IsZero() {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
return time.Since(t.start)
|
||||||
|
}
|
||||||
|
|
||||||
|
// statsLoop prints one aggregate line per interval. Never started when
|
||||||
|
// statsIntervalMs is 0, which is the default.
|
||||||
|
func (c *Client) statsLoop(stop <-chan struct{}) {
|
||||||
|
ticker := time.NewTicker(time.Duration(c.cfg.StatsIntervalMs) * time.Millisecond)
|
||||||
|
defer ticker.Stop()
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-stop:
|
||||||
|
return
|
||||||
|
case <-ticker.C:
|
||||||
|
log.Print(c.StatsLine())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// StatsLine renders the current pool and per-conn state as one greppable line.
|
||||||
|
// Exported so tests and embedders can sample it without waiting for the ticker.
|
||||||
|
func (c *Client) StatsLine() string {
|
||||||
|
conns := c.pool.snapshot()
|
||||||
|
var b strings.Builder
|
||||||
|
fmt.Fprintf(&b, "stats conns=%d", len(conns))
|
||||||
|
|
||||||
|
live, parked := 0, 0
|
||||||
|
for _, wc := range conns {
|
||||||
|
st := wc.getStream()
|
||||||
|
bound := 0
|
||||||
|
if st != nil {
|
||||||
|
bound = 1
|
||||||
|
live++
|
||||||
|
st.mu.Lock()
|
||||||
|
if st.parked {
|
||||||
|
parked++
|
||||||
|
}
|
||||||
|
st.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Fprintf(&b, " | conn%d bound=%d", wc.id, bound)
|
||||||
|
if cs := wc.stats; cs != nil {
|
||||||
|
_, mn, avg, mx := cs.rtt()
|
||||||
|
fmt.Fprintf(&b, " frames=%d/%d rtt=%s/%s/%s",
|
||||||
|
cs.framesIn.Load(), cs.framesOut.Load(), round(mn), round(avg), round(mx))
|
||||||
|
if n := cs.writeErrs.Load(); n > 0 {
|
||||||
|
fmt.Fprintf(&b, " writeErrs=%d", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
fmt.Fprintf(&b, " | tunnels=%d parked=%d", live, parked)
|
||||||
|
return b.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
// logSummary reports a stream's lifetime as it closes. This is the artifact that
|
||||||
|
// answers a specific complaint after the fact, once the periodic line has
|
||||||
|
// scrolled away.
|
||||||
|
func (s *Stream) logSummary() {
|
||||||
|
s.mu.Lock()
|
||||||
|
st := s.stats
|
||||||
|
if st == nil {
|
||||||
|
s.mu.Unlock()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
line := fmt.Sprintf("stream closed after %s: up=%s down=%s stalled(window=%s shaper=%s) qPeak=%s",
|
||||||
|
round(time.Since(st.opened)), bytesHuman(st.bytesUp), bytesHuman(st.bytesDown),
|
||||||
|
round(st.windowStall), round(st.shaperStall), bytesHuman(int64(st.qPeak)))
|
||||||
|
if st.resumes > 0 {
|
||||||
|
line += fmt.Sprintf(" resumes=%d hung=%s replayed=%s",
|
||||||
|
st.resumes, round(st.hung), bytesHuman(st.replayBytes))
|
||||||
|
}
|
||||||
|
s.mu.Unlock()
|
||||||
|
log.Print(line)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *WorkerPool) snapshot() []*WorkerConn {
|
||||||
|
p.mu.Lock()
|
||||||
|
conns := make([]*WorkerConn, 0, len(p.conns))
|
||||||
|
for wc := range p.conns {
|
||||||
|
conns = append(conns, wc)
|
||||||
|
}
|
||||||
|
p.mu.Unlock()
|
||||||
|
sort.Slice(conns, func(i, j int) bool { return conns[i].id < conns[j].id })
|
||||||
|
return conns
|
||||||
|
}
|
||||||
|
|
||||||
|
// round trims a duration to something readable in a log line.
|
||||||
|
func round(d time.Duration) time.Duration {
|
||||||
|
switch {
|
||||||
|
case d <= 0:
|
||||||
|
return 0
|
||||||
|
case d < time.Millisecond:
|
||||||
|
return d.Round(time.Microsecond)
|
||||||
|
case d < time.Second:
|
||||||
|
return d.Round(time.Millisecond)
|
||||||
|
default:
|
||||||
|
return d.Round(10 * time.Millisecond)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func bytesHuman(n int64) string {
|
||||||
|
const unit = 1024
|
||||||
|
if n < unit {
|
||||||
|
return fmt.Sprintf("%dB", n)
|
||||||
|
}
|
||||||
|
div, exp := int64(unit), 0
|
||||||
|
for v := n / unit; v >= unit; v /= unit {
|
||||||
|
div *= unit
|
||||||
|
exp++
|
||||||
|
}
|
||||||
|
return fmt.Sprintf("%.1f%ciB", float64(n)/float64(div), "KMGT"[exp])
|
||||||
|
}
|
||||||
@@ -0,0 +1,64 @@
|
|||||||
|
package client
|
||||||
|
|
||||||
|
// unackedBuf holds the bytes a stream has sent but the peer has not yet
|
||||||
|
// credited — exactly the region a reattach may have to retransmit
|
||||||
|
// (PROTOCOL.md §7.5).
|
||||||
|
//
|
||||||
|
// It needs no cap of its own: credit is only granted as bytes reach the peer's
|
||||||
|
// terminal socket, so flow control already bounds the outstanding region to one
|
||||||
|
// window. That is what makes byte-exact resumption affordable at all.
|
||||||
|
//
|
||||||
|
// A read offset rather than a copy-down on every trim. Credit arrives once per
|
||||||
|
// half-window, and copying the live remainder each time would add a second
|
||||||
|
// per-byte copy to the whole send path; compacting only once the dead prefix
|
||||||
|
// dominates makes it amortized O(1).
|
||||||
|
type unackedBuf struct {
|
||||||
|
buf []byte
|
||||||
|
head int // bytes at the front already credited, awaiting reclamation
|
||||||
|
baseOff int64 // stream offset of buf[head]
|
||||||
|
}
|
||||||
|
|
||||||
|
// length is how many bytes are still outstanding.
|
||||||
|
func (u *unackedBuf) length() int { return len(u.buf) - u.head }
|
||||||
|
|
||||||
|
// base is the offset of the first byte still held.
|
||||||
|
func (u *unackedBuf) base() int64 { return u.baseOff }
|
||||||
|
|
||||||
|
// end is the offset one past the last byte sent.
|
||||||
|
func (u *unackedBuf) end() int64 { return u.baseOff + int64(u.length()) }
|
||||||
|
|
||||||
|
func (u *unackedBuf) append(p []byte) { u.buf = append(u.buf, p...) }
|
||||||
|
|
||||||
|
// advance drops everything the peer has credited up to off.
|
||||||
|
func (u *unackedBuf) advance(off int64) {
|
||||||
|
drop := int(off - u.baseOff)
|
||||||
|
if drop <= 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if n := u.length(); drop > n {
|
||||||
|
drop = n // only reachable from a peer crediting bytes it was never sent
|
||||||
|
}
|
||||||
|
u.head += drop
|
||||||
|
u.baseOff += int64(drop)
|
||||||
|
switch {
|
||||||
|
case u.head == len(u.buf):
|
||||||
|
u.buf, u.head = u.buf[:0], 0 // fully drained: restart at the front
|
||||||
|
case u.head > len(u.buf)/2:
|
||||||
|
u.buf = append(u.buf[:0], u.buf[u.head:]...)
|
||||||
|
u.head = 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// from returns the outstanding bytes at and after off, or nil when off falls
|
||||||
|
// outside what is still held — which means the peer reported an offset we can no
|
||||||
|
// longer satisfy, and the stream cannot be resumed.
|
||||||
|
func (u *unackedBuf) from(off int64) []byte {
|
||||||
|
skip := off - u.baseOff
|
||||||
|
if skip < 0 || skip > int64(u.length()) {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return u.buf[u.head+int(skip):]
|
||||||
|
}
|
||||||
|
|
||||||
|
// reset releases the buffer once a stream can no longer be resumed.
|
||||||
|
func (u *unackedBuf) reset() { u.buf, u.head = nil, 0 }
|
||||||
@@ -0,0 +1,112 @@
|
|||||||
|
package client
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The retained region is what a reattach replays from, so an off-by-one here is
|
||||||
|
// not a dropped byte but a spliced stream: the peer resumes mid-packet and the
|
||||||
|
// session dies in a way no round-trip test would attribute to this code.
|
||||||
|
|
||||||
|
func TestUnackedTracksOffsets(t *testing.T) {
|
||||||
|
var u unackedBuf
|
||||||
|
u.append([]byte("hello"))
|
||||||
|
u.append([]byte("world"))
|
||||||
|
|
||||||
|
if got := u.length(); got != 10 {
|
||||||
|
t.Fatalf("length = %d, want 10", got)
|
||||||
|
}
|
||||||
|
if got := u.end(); got != 10 {
|
||||||
|
t.Fatalf("end = %d, want 10", got)
|
||||||
|
}
|
||||||
|
if got := u.from(0); !bytes.Equal(got, []byte("helloworld")) {
|
||||||
|
t.Fatalf("from(0) = %q", got)
|
||||||
|
}
|
||||||
|
// A reattach replays from wherever the peer got to, which lands anywhere —
|
||||||
|
// including the middle of a chunk boundary.
|
||||||
|
if got := u.from(3); !bytes.Equal(got, []byte("loworld")) {
|
||||||
|
t.Fatalf("from(3) = %q", got)
|
||||||
|
}
|
||||||
|
if got := u.from(10); len(got) != 0 {
|
||||||
|
t.Fatalf("from(end) = %q, want empty", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUnackedAdvanceDropsCreditedBytes(t *testing.T) {
|
||||||
|
var u unackedBuf
|
||||||
|
u.append([]byte("abcdefghij"))
|
||||||
|
|
||||||
|
u.advance(4)
|
||||||
|
if got := u.length(); got != 6 {
|
||||||
|
t.Fatalf("length after advance = %d, want 6", got)
|
||||||
|
}
|
||||||
|
if got := u.end(); got != 10 {
|
||||||
|
t.Fatalf("end must not move when bytes are dropped: got %d, want 10", got)
|
||||||
|
}
|
||||||
|
if got := u.from(4); !bytes.Equal(got, []byte("efghij")) {
|
||||||
|
t.Fatalf("from(4) = %q", got)
|
||||||
|
}
|
||||||
|
// Below the retained region: the peer named an offset we can no longer
|
||||||
|
// satisfy, which must be reported rather than silently clamped — replaying
|
||||||
|
// the wrong range is worse than refusing to replay.
|
||||||
|
if got := u.from(3); got != nil {
|
||||||
|
t.Fatalf("from(3) below base = %q, want nil", got)
|
||||||
|
}
|
||||||
|
if got := u.from(11); got != nil {
|
||||||
|
t.Fatalf("from(11) past end = %q, want nil", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Interleaving appends and advances is the steady-state pattern: credit arrives
|
||||||
|
// every half window while the sender keeps writing. The buffer must stay exact
|
||||||
|
// across the compaction that eventually triggers.
|
||||||
|
func TestUnackedSurvivesInterleavedAppendAndAdvance(t *testing.T) {
|
||||||
|
var u unackedBuf
|
||||||
|
var sent []byte
|
||||||
|
var acked int64
|
||||||
|
|
||||||
|
for i := 0; i < 200; i++ {
|
||||||
|
chunk := bytes.Repeat([]byte{byte(i)}, 97)
|
||||||
|
sent = append(sent, chunk...)
|
||||||
|
// emit's order: reclaim what has been credited so far, then retain the
|
||||||
|
// new chunk. The base therefore trails the credit that arrived since.
|
||||||
|
base := acked
|
||||||
|
u.advance(base)
|
||||||
|
u.append(chunk)
|
||||||
|
|
||||||
|
if got, want := u.end(), int64(len(sent)); got != want {
|
||||||
|
t.Fatalf("round %d: end = %d, want %d", i, got, want)
|
||||||
|
}
|
||||||
|
if got, want := u.length(), len(sent)-int(base); got != want {
|
||||||
|
t.Fatalf("round %d: length = %d, want %d", i, got, want)
|
||||||
|
}
|
||||||
|
if got, want := u.from(base), sent[base:]; !bytes.Equal(got, want) {
|
||||||
|
t.Fatalf("round %d: retained region diverges from what was sent", i)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The peer can only ever credit bytes it has actually received.
|
||||||
|
if i%3 == 0 {
|
||||||
|
if acked += 61; acked > int64(len(sent)) {
|
||||||
|
acked = int64(len(sent))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Compaction reuses the backing array, so a stream that runs for hours must not
|
||||||
|
// grow one: this is a full window per stream, on both sides.
|
||||||
|
func TestUnackedReclaimsBackingArray(t *testing.T) {
|
||||||
|
var u unackedBuf
|
||||||
|
chunk := bytes.Repeat([]byte{7}, 4096)
|
||||||
|
for i := 0; i < 500; i++ {
|
||||||
|
u.advance(u.end()) // fully credited every round
|
||||||
|
u.append(chunk)
|
||||||
|
}
|
||||||
|
if u.length() != len(chunk) {
|
||||||
|
t.Fatalf("length = %d, want %d", u.length(), len(chunk))
|
||||||
|
}
|
||||||
|
if cap(u.buf) > 8*len(chunk) {
|
||||||
|
t.Fatalf("backing array grew to %d bytes for a %d-byte window", cap(u.buf), len(chunk))
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,372 @@
|
|||||||
|
package client
|
||||||
|
|
||||||
|
import (
|
||||||
|
"crypto/hmac"
|
||||||
|
"crypto/md5"
|
||||||
|
"crypto/sha256"
|
||||||
|
"errors"
|
||||||
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
|
|
||||||
|
"github.com/iceBear67/redapricot/client/wire"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Velocity "modern forwarding" support.
|
||||||
|
//
|
||||||
|
// A Paper backend configured with Velocity modern player-info forwarding runs
|
||||||
|
// in offline mode and instead trusts a signed login payload from its proxy:
|
||||||
|
// during the login phase it sends a Login Plugin Request on channel
|
||||||
|
// "velocity:player_info" and expects a Login Plugin Response whose data is an
|
||||||
|
// HMAC-SHA256 signature followed by the player's real address, UUID, username
|
||||||
|
// and profile properties.
|
||||||
|
//
|
||||||
|
// redapricot is a transparent tunnel, so that request would reach the vanilla
|
||||||
|
// player, who cannot answer it and gets kicked. When a mapping sets
|
||||||
|
// "velocitySecret", the stream answers on the player's behalf: it observes the
|
||||||
|
// player's Handshake and Login Start to learn the protocol version, username
|
||||||
|
// and UUID, swallows the backend's velocity:player_info request instead of
|
||||||
|
// forwarding it, and injects the signed response. Everything else — and
|
||||||
|
// everything after the exchange — is forwarded verbatim. On any traffic that
|
||||||
|
// does not look like a vanilla login (status pings, parse errors, oversized
|
||||||
|
// packets) the stream fails open into pure passthrough.
|
||||||
|
//
|
||||||
|
// The forwarded profile carries no properties (skin/cape textures): the tunnel
|
||||||
|
// never talks to Mojang, exactly like an offline-mode proxy.
|
||||||
|
const (
|
||||||
|
velocityChannel = "velocity:player_info"
|
||||||
|
|
||||||
|
// Forwarding payload versions (Velocity's VelocityConstants). We never use
|
||||||
|
// versions 2/3 (WITH_KEY): they exist only for 1.19–1.19.2 chat signing,
|
||||||
|
// and version 1 remains acceptable to every backend.
|
||||||
|
velocityVersionDefault = 1
|
||||||
|
velocityVersionLazySession = 4
|
||||||
|
|
||||||
|
// Minecraft protocol versions at which the Login Start layout changes.
|
||||||
|
protocol1_19 = 759 // + optional signature key
|
||||||
|
protocol1_19_1 = 760 // + optional profile UUID (after the key)
|
||||||
|
protocol1_19_3 = 761 // key removed, optional UUID stays
|
||||||
|
protocol1_20_2 = 764 // UUID mandatory
|
||||||
|
|
||||||
|
// Handshake intents that enter the login phase.
|
||||||
|
intentLogin = 2
|
||||||
|
intentTransfer = 3
|
||||||
|
|
||||||
|
// Login-phase packet ids (stable across protocol versions).
|
||||||
|
loginC2SPluginResponse = 0x02
|
||||||
|
loginS2CDisconnect = 0x00
|
||||||
|
loginS2CEncryptionRequest = 0x01
|
||||||
|
loginS2CSuccess = 0x02
|
||||||
|
loginS2CSetCompression = 0x03
|
||||||
|
loginS2CPluginRequest = 0x04
|
||||||
|
|
||||||
|
// Sniff-buffer caps. Login-phase packets are small; anything larger means
|
||||||
|
// this is not the exchange we are looking for.
|
||||||
|
maxC2SSniff = 8 << 10
|
||||||
|
maxS2CSniff = 64 << 10
|
||||||
|
)
|
||||||
|
|
||||||
|
var errVelocitySniff = errors.New("velocity: connection does not follow the vanilla login flow")
|
||||||
|
|
||||||
|
// velocityForwarder is the per-stream login interceptor. ObserveC2S is called
|
||||||
|
// from the worker read loop, ProcessS2C from the stream's destination-read
|
||||||
|
// goroutine; the mutex orders them, and passthrough short-circuits both once
|
||||||
|
// interception is over.
|
||||||
|
type velocityForwarder struct {
|
||||||
|
passthrough atomic.Bool // fully transparent, buffers empty: skip the mutex
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
secret []byte
|
||||||
|
srcIP string
|
||||||
|
|
||||||
|
// player -> server observation
|
||||||
|
c2sBuf []byte
|
||||||
|
c2sDone bool
|
||||||
|
handshakeParsed bool
|
||||||
|
protocol int
|
||||||
|
loginStartSeen bool
|
||||||
|
username string
|
||||||
|
uuid [16]byte
|
||||||
|
|
||||||
|
// server -> player interception; done means the s2c side (and with it the
|
||||||
|
// whole interceptor) is finished.
|
||||||
|
s2cBuf []byte
|
||||||
|
done bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func newVelocityForwarder(secret, srcIP string) *velocityForwarder {
|
||||||
|
return &velocityForwarder{secret: []byte(secret), srcIP: srcIP}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Passthrough reports that interception is over and both directions may skip
|
||||||
|
// the forwarder entirely.
|
||||||
|
func (v *velocityForwarder) Passthrough() bool { return v.passthrough.Load() }
|
||||||
|
|
||||||
|
// abortLocked gives up on interception: the stream becomes pure passthrough.
|
||||||
|
// s2cBuf is deliberately kept — ProcessS2C flushes it to the player.
|
||||||
|
func (v *velocityForwarder) abortLocked() {
|
||||||
|
v.done = true
|
||||||
|
v.c2sDone = true
|
||||||
|
v.c2sBuf = nil
|
||||||
|
if len(v.s2cBuf) == 0 {
|
||||||
|
v.passthrough.Store(true)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ObserveC2S watches player->server bytes (already being forwarded verbatim by
|
||||||
|
// the caller) until the Handshake and Login Start have been parsed.
|
||||||
|
func (v *velocityForwarder) ObserveC2S(data []byte) {
|
||||||
|
if v.passthrough.Load() {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
v.mu.Lock()
|
||||||
|
defer v.mu.Unlock()
|
||||||
|
if v.c2sDone {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
v.c2sBuf = append(v.c2sBuf, data...)
|
||||||
|
for !v.c2sDone {
|
||||||
|
_, body, rest, ok, err := nextPacket(v.c2sBuf, maxC2SSniff)
|
||||||
|
if err != nil {
|
||||||
|
v.abortLocked()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !ok {
|
||||||
|
if len(v.c2sBuf) > maxC2SSniff {
|
||||||
|
v.abortLocked()
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
v.c2sBuf = rest
|
||||||
|
if err := v.observeC2SPacket(body); err != nil {
|
||||||
|
v.abortLocked()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
v.c2sBuf = nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// observeC2SPacket handles one player packet: first the Handshake, then Login
|
||||||
|
// Start. Any deviation from the vanilla login flow is an error (→ fail open).
|
||||||
|
func (v *velocityForwarder) observeC2SPacket(body []byte) error {
|
||||||
|
r := wire.NewReader(body)
|
||||||
|
id, err := r.VarInt()
|
||||||
|
if err != nil || id != 0x00 { // Handshake and Login Start are both 0x00
|
||||||
|
return errVelocitySniff
|
||||||
|
}
|
||||||
|
if !v.handshakeParsed {
|
||||||
|
proto, perr := r.VarInt()
|
||||||
|
_, aerr := r.String() // address
|
||||||
|
_, poerr := r.U16() // port
|
||||||
|
intent, ierr := r.VarInt()
|
||||||
|
if perr != nil || aerr != nil || poerr != nil || ierr != nil {
|
||||||
|
return errVelocitySniff
|
||||||
|
}
|
||||||
|
if intent != intentLogin && intent != intentTransfer {
|
||||||
|
return errVelocitySniff // status ping etc.: nothing to intercept
|
||||||
|
}
|
||||||
|
v.protocol = proto
|
||||||
|
v.handshakeParsed = true
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return v.parseLoginStart(r)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (v *velocityForwarder) parseLoginStart(r *wire.Reader) error {
|
||||||
|
name, err := r.String()
|
||||||
|
if err != nil || len(name) == 0 || len(name) > 16 {
|
||||||
|
return errVelocitySniff
|
||||||
|
}
|
||||||
|
if v.protocol >= protocol1_19 && v.protocol < protocol1_19_3 {
|
||||||
|
// Optional chat-signing key: expiry + public key + signature.
|
||||||
|
hasKey, err := r.U8()
|
||||||
|
if err != nil {
|
||||||
|
return errVelocitySniff
|
||||||
|
}
|
||||||
|
if hasKey != 0 {
|
||||||
|
if _, err := r.I64(); err != nil {
|
||||||
|
return errVelocitySniff
|
||||||
|
}
|
||||||
|
for i := 0; i < 2; i++ {
|
||||||
|
n, err := r.VarInt()
|
||||||
|
if err != nil {
|
||||||
|
return errVelocitySniff
|
||||||
|
}
|
||||||
|
if _, err := r.Bytes(n); err != nil {
|
||||||
|
return errVelocitySniff
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
haveUUID := false
|
||||||
|
switch {
|
||||||
|
case v.protocol >= protocol1_20_2:
|
||||||
|
haveUUID = true
|
||||||
|
case v.protocol >= protocol1_19_1:
|
||||||
|
flag, err := r.U8()
|
||||||
|
if err != nil {
|
||||||
|
return errVelocitySniff
|
||||||
|
}
|
||||||
|
haveUUID = flag != 0
|
||||||
|
}
|
||||||
|
if haveUUID {
|
||||||
|
b, err := r.Bytes(16)
|
||||||
|
if err != nil {
|
||||||
|
return errVelocitySniff
|
||||||
|
}
|
||||||
|
copy(v.uuid[:], b)
|
||||||
|
} else {
|
||||||
|
v.uuid = offlineUUID(name)
|
||||||
|
}
|
||||||
|
v.username = name
|
||||||
|
v.loginStartSeen = true
|
||||||
|
v.c2sDone = true
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ProcessS2C consumes one chunk of server->player bytes. It returns the bytes
|
||||||
|
// to forward to the player and, once the velocity query has been answered, the
|
||||||
|
// Login Plugin Response to inject towards the server. Complete packets are
|
||||||
|
// forwarded as they parse; a trailing partial packet stays buffered until the
|
||||||
|
// next chunk.
|
||||||
|
func (v *velocityForwarder) ProcessS2C(data []byte) (forward, inject []byte) {
|
||||||
|
v.mu.Lock()
|
||||||
|
defer v.mu.Unlock()
|
||||||
|
if v.done {
|
||||||
|
// Interception ended from the c2s side while bytes sat buffered here.
|
||||||
|
if len(v.s2cBuf) > 0 {
|
||||||
|
forward = append(v.s2cBuf, data...)
|
||||||
|
v.s2cBuf = nil
|
||||||
|
v.passthrough.Store(true)
|
||||||
|
return forward, nil
|
||||||
|
}
|
||||||
|
v.passthrough.Store(true)
|
||||||
|
return data, nil
|
||||||
|
}
|
||||||
|
v.s2cBuf = append(v.s2cBuf, data...)
|
||||||
|
loop:
|
||||||
|
for {
|
||||||
|
raw, body, rest, ok, err := nextPacket(v.s2cBuf, maxS2CSniff)
|
||||||
|
if err != nil || (!ok && len(v.s2cBuf) > maxS2CSniff) {
|
||||||
|
v.abortLocked() // unconsumed bytes are flushed below
|
||||||
|
break
|
||||||
|
}
|
||||||
|
if !ok {
|
||||||
|
break // partial packet: wait for the next chunk
|
||||||
|
}
|
||||||
|
r := wire.NewReader(body)
|
||||||
|
id, err := r.VarInt()
|
||||||
|
if err != nil {
|
||||||
|
v.abortLocked()
|
||||||
|
break
|
||||||
|
}
|
||||||
|
switch id {
|
||||||
|
case loginS2CPluginRequest:
|
||||||
|
msgID, merr := r.VarInt()
|
||||||
|
channel, cerr := r.String()
|
||||||
|
if merr != nil || cerr != nil {
|
||||||
|
v.abortLocked()
|
||||||
|
break loop
|
||||||
|
}
|
||||||
|
if channel == velocityChannel {
|
||||||
|
if !v.loginStartSeen {
|
||||||
|
// Cannot answer without a parsed Login Start; let the
|
||||||
|
// request through — the backend will kick the player with
|
||||||
|
// its own clear message.
|
||||||
|
v.abortLocked()
|
||||||
|
break loop
|
||||||
|
}
|
||||||
|
inject = v.buildResponseLocked(msgID, r.Remaining())
|
||||||
|
v.s2cBuf = rest // swallow the request: the player never sees it
|
||||||
|
v.done = true
|
||||||
|
break loop
|
||||||
|
}
|
||||||
|
// Another plugin channel (e.g. a mod handshake): the player
|
||||||
|
// answers it itself; forward and keep watching.
|
||||||
|
case loginS2CDisconnect, loginS2CEncryptionRequest, loginS2CSuccess, loginS2CSetCompression:
|
||||||
|
// Login phase is over (or turning encrypted/compressed) and no
|
||||||
|
// velocity query showed up: stop watching.
|
||||||
|
v.done = true
|
||||||
|
default:
|
||||||
|
// Cookie Request (0x05, 1.20.5+) or future packets: forward.
|
||||||
|
}
|
||||||
|
v.s2cBuf = rest
|
||||||
|
forward = append(forward, raw...)
|
||||||
|
if v.done {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if v.done {
|
||||||
|
forward = append(forward, v.s2cBuf...)
|
||||||
|
v.s2cBuf = nil
|
||||||
|
v.c2sBuf = nil
|
||||||
|
v.c2sDone = true
|
||||||
|
v.passthrough.Store(true)
|
||||||
|
}
|
||||||
|
return forward, inject
|
||||||
|
}
|
||||||
|
|
||||||
|
// buildResponseLocked crafts the serverbound Login Plugin Response carrying the
|
||||||
|
// signed forwarding payload (mirrors Velocity's createForwardingData).
|
||||||
|
func (v *velocityForwarder) buildResponseLocked(msgID int, reqData []byte) []byte {
|
||||||
|
// The request data is the backend's maximum supported forwarding version
|
||||||
|
// (absent on very old backends → 1).
|
||||||
|
requested := velocityVersionDefault
|
||||||
|
if len(reqData) > 0 {
|
||||||
|
if n, err := wire.NewReader(reqData).VarInt(); err == nil {
|
||||||
|
requested = n
|
||||||
|
}
|
||||||
|
}
|
||||||
|
version := velocityVersionDefault
|
||||||
|
if requested >= velocityVersionLazySession && v.protocol >= protocol1_19_3 {
|
||||||
|
version = velocityVersionLazySession
|
||||||
|
}
|
||||||
|
payload := wire.NewWriter().
|
||||||
|
VarInt(version).
|
||||||
|
String(v.srcIP).
|
||||||
|
Bytes(v.uuid[:]). // UUID = 16 raw bytes (two big-endian longs)
|
||||||
|
String(v.username).
|
||||||
|
VarInt(0). // profile properties
|
||||||
|
Out()
|
||||||
|
mac := hmac.New(sha256.New, v.secret)
|
||||||
|
mac.Write(payload)
|
||||||
|
body := wire.NewWriter().
|
||||||
|
VarInt(loginC2SPluginResponse).
|
||||||
|
VarInt(msgID).
|
||||||
|
U8(1). // successful
|
||||||
|
Bytes(mac.Sum(nil)).
|
||||||
|
Bytes(payload).
|
||||||
|
Out()
|
||||||
|
return append(wire.AppendVarInt(nil, len(body)), body...)
|
||||||
|
}
|
||||||
|
|
||||||
|
// nextPacket splits one length-prefixed Minecraft packet off buf. raw includes
|
||||||
|
// the length header, body is the packet payload, rest what follows. ok is
|
||||||
|
// false while the packet is still incomplete; err reports a malformed or
|
||||||
|
// oversized length header.
|
||||||
|
func nextPacket(buf []byte, max int) (raw, body, rest []byte, ok bool, err error) {
|
||||||
|
r := wire.NewReader(buf)
|
||||||
|
n, verr := r.VarInt()
|
||||||
|
if verr != nil {
|
||||||
|
if len(buf) >= wire.VarIntMaxBytes {
|
||||||
|
return nil, nil, buf, false, verr
|
||||||
|
}
|
||||||
|
return nil, nil, buf, false, nil // header not complete yet
|
||||||
|
}
|
||||||
|
if n <= 0 || n > max {
|
||||||
|
return nil, nil, buf, false, errVelocitySniff
|
||||||
|
}
|
||||||
|
hdr := len(buf) - len(r.Remaining())
|
||||||
|
if len(buf) < hdr+n {
|
||||||
|
return nil, nil, buf, false, nil
|
||||||
|
}
|
||||||
|
return buf[:hdr+n], buf[hdr : hdr+n], buf[hdr+n:], true, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// offlineUUID derives the offline-mode UUID for a username, identical to
|
||||||
|
// Java's UUID.nameUUIDFromBytes("OfflinePlayer:" + name): a v3 (MD5) UUID.
|
||||||
|
func offlineUUID(name string) [16]byte {
|
||||||
|
sum := md5.Sum([]byte("OfflinePlayer:" + name))
|
||||||
|
sum[6] = sum[6]&0x0f | 0x30 // version 3
|
||||||
|
sum[8] = sum[8]&0x3f | 0x80 // IETF variant
|
||||||
|
return sum
|
||||||
|
}
|
||||||
@@ -0,0 +1,348 @@
|
|||||||
|
package client
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"crypto/hmac"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/iceBear67/redapricot/client/wire"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ---- packet builders (player/backend side) ----
|
||||||
|
|
||||||
|
func mkPacket(body []byte) []byte {
|
||||||
|
return append(wire.AppendVarInt(nil, len(body)), body...)
|
||||||
|
}
|
||||||
|
|
||||||
|
func mkLoginStart(proto int, name string, uuid []byte, keyData bool) []byte {
|
||||||
|
w := wire.NewWriter().VarInt(0x00).String(name)
|
||||||
|
if proto >= protocol1_19 && proto < protocol1_19_3 {
|
||||||
|
if keyData {
|
||||||
|
w.U8(1).I64(1234567890)
|
||||||
|
pub := bytes.Repeat([]byte{0xAA}, 33)
|
||||||
|
sig := bytes.Repeat([]byte{0xBB}, 17)
|
||||||
|
w.VarInt(len(pub)).Bytes(pub).VarInt(len(sig)).Bytes(sig)
|
||||||
|
} else {
|
||||||
|
w.U8(0)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
switch {
|
||||||
|
case proto >= protocol1_20_2:
|
||||||
|
w.Bytes(uuid)
|
||||||
|
case proto >= protocol1_19_1:
|
||||||
|
if uuid != nil {
|
||||||
|
w.U8(1).Bytes(uuid)
|
||||||
|
} else {
|
||||||
|
w.U8(0)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return mkPacket(w.Out())
|
||||||
|
}
|
||||||
|
|
||||||
|
func mkPluginRequest(msgID int, channel string, data []byte) []byte {
|
||||||
|
body := wire.NewWriter().VarInt(loginS2CPluginRequest).VarInt(msgID).String(channel).Bytes(data).Out()
|
||||||
|
return mkPacket(body)
|
||||||
|
}
|
||||||
|
|
||||||
|
// feedLogin drives a full player login prologue through ObserveC2S in n-byte
|
||||||
|
// chunks.
|
||||||
|
func feedLogin(v *velocityForwarder, proto int, intent int, loginStart []byte, chunk int) {
|
||||||
|
stream := wire.BuildHandshake(proto, "mc.example.com", 25565, intent)
|
||||||
|
if loginStart != nil {
|
||||||
|
stream = append(stream, loginStart...)
|
||||||
|
}
|
||||||
|
for len(stream) > 0 {
|
||||||
|
n := chunk
|
||||||
|
if n > len(stream) {
|
||||||
|
n = len(stream)
|
||||||
|
}
|
||||||
|
v.ObserveC2S(stream[:n])
|
||||||
|
stream = stream[n:]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseResponse validates the injected Login Plugin Response and returns the
|
||||||
|
// echoed message id and the signed forwarding payload.
|
||||||
|
func parseResponse(t *testing.T, secret string, inject []byte) (msgID int, payload *wire.Reader) {
|
||||||
|
t.Helper()
|
||||||
|
r := wire.NewReader(inject)
|
||||||
|
plen, err := r.VarInt()
|
||||||
|
if err != nil || plen != len(r.Remaining()) {
|
||||||
|
t.Fatalf("bad response length prefix: %v (declared %d, have %d)", err, plen, len(r.Remaining()))
|
||||||
|
}
|
||||||
|
id, _ := r.VarInt()
|
||||||
|
if id != loginC2SPluginResponse {
|
||||||
|
t.Fatalf("response packet id = %#x, want 0x02", id)
|
||||||
|
}
|
||||||
|
msgID, _ = r.VarInt()
|
||||||
|
ok, _ := r.U8()
|
||||||
|
if ok != 1 {
|
||||||
|
t.Fatalf("response not marked successful")
|
||||||
|
}
|
||||||
|
sig, err := r.Bytes(32)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("response missing signature: %v", err)
|
||||||
|
}
|
||||||
|
data := r.Remaining()
|
||||||
|
mac := hmac.New(sha256.New, []byte(secret))
|
||||||
|
mac.Write(data)
|
||||||
|
if !hmac.Equal(sig, mac.Sum(nil)) {
|
||||||
|
t.Fatalf("forwarding payload signature does not verify")
|
||||||
|
}
|
||||||
|
return msgID, wire.NewReader(data)
|
||||||
|
}
|
||||||
|
|
||||||
|
func assertPayload(t *testing.T, r *wire.Reader, version int, ip, name, uuidHex string) {
|
||||||
|
t.Helper()
|
||||||
|
gotVer, _ := r.VarInt()
|
||||||
|
if gotVer != version {
|
||||||
|
t.Fatalf("forwarding version = %d, want %d", gotVer, version)
|
||||||
|
}
|
||||||
|
gotIP, _ := r.String()
|
||||||
|
if gotIP != ip {
|
||||||
|
t.Fatalf("forwarded address = %q, want %q", gotIP, ip)
|
||||||
|
}
|
||||||
|
gotUUID, err := r.Bytes(16)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("payload missing uuid: %v", err)
|
||||||
|
}
|
||||||
|
if hex.EncodeToString(gotUUID) != uuidHex {
|
||||||
|
t.Fatalf("forwarded uuid = %x, want %s", gotUUID, uuidHex)
|
||||||
|
}
|
||||||
|
gotName, _ := r.String()
|
||||||
|
if gotName != name {
|
||||||
|
t.Fatalf("forwarded username = %q, want %q", gotName, name)
|
||||||
|
}
|
||||||
|
props, err := r.VarInt()
|
||||||
|
if err != nil || props != 0 {
|
||||||
|
t.Fatalf("forwarded properties = %d (%v), want 0", props, err)
|
||||||
|
}
|
||||||
|
if len(r.Remaining()) != 0 {
|
||||||
|
t.Fatalf("trailing bytes in forwarding payload: %x", r.Remaining())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- tests ----
|
||||||
|
|
||||||
|
const testSecret = "unit-secret"
|
||||||
|
|
||||||
|
func TestVelocityInterceptModern(t *testing.T) {
|
||||||
|
uuid, _ := hex.DecodeString("00112233445566778899aabbccddeeff")
|
||||||
|
v := newVelocityForwarder(testSecret, "203.0.113.7")
|
||||||
|
feedLogin(v, 767, intentLogin, mkLoginStart(767, "icybear", uuid, false), 1)
|
||||||
|
|
||||||
|
// Backend query, requesting up to forwarding version 4, fed byte by byte:
|
||||||
|
// nothing may reach the player, and the response appears with the last byte.
|
||||||
|
req := mkPluginRequest(99, velocityChannel, []byte{0x04})
|
||||||
|
var inject []byte
|
||||||
|
for i, b := range req {
|
||||||
|
fwd, inj := v.ProcessS2C([]byte{b})
|
||||||
|
if len(fwd) != 0 {
|
||||||
|
t.Fatalf("byte %d: request leaked to the player: %x", i, fwd)
|
||||||
|
}
|
||||||
|
if inj != nil {
|
||||||
|
inject = inj
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if inject == nil {
|
||||||
|
t.Fatalf("no response was injected")
|
||||||
|
}
|
||||||
|
msgID, payload := parseResponse(t, testSecret, inject)
|
||||||
|
if msgID != 99 {
|
||||||
|
t.Fatalf("echoed message id = %d, want 99", msgID)
|
||||||
|
}
|
||||||
|
assertPayload(t, payload, velocityVersionLazySession, "203.0.113.7", "icybear",
|
||||||
|
"00112233445566778899aabbccddeeff")
|
||||||
|
if !v.Passthrough() {
|
||||||
|
t.Fatalf("interceptor should be passthrough after answering")
|
||||||
|
}
|
||||||
|
garbage := []byte{0xde, 0xad, 0xbe, 0xef}
|
||||||
|
if fwd, inj := v.ProcessS2C(garbage); !bytes.Equal(fwd, garbage) || inj != nil {
|
||||||
|
t.Fatalf("post-login bytes not passed through verbatim")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVelocityNegativeMessageID(t *testing.T) {
|
||||||
|
// Paper picks the message id with ThreadLocalRandom.nextInt(): it is
|
||||||
|
// negative half the time and must be echoed bit-exactly.
|
||||||
|
uuid, _ := hex.DecodeString("00112233445566778899aabbccddeeff")
|
||||||
|
v := newVelocityForwarder(testSecret, "198.51.100.1")
|
||||||
|
feedLogin(v, 767, intentLogin, mkLoginStart(767, "neg", uuid, false), 64)
|
||||||
|
_, inject := v.ProcessS2C(mkPluginRequest(-123456, velocityChannel, []byte{0x04}))
|
||||||
|
if inject == nil {
|
||||||
|
t.Fatalf("no response was injected")
|
||||||
|
}
|
||||||
|
msgID, _ := parseResponse(t, testSecret, inject)
|
||||||
|
if !bytes.Equal(wire.AppendVarInt(nil, msgID), wire.AppendVarInt(nil, -123456)) {
|
||||||
|
t.Fatalf("negative message id not echoed bit-exactly (got %d)", msgID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVelocityOfflineUUIDAndV1(t *testing.T) {
|
||||||
|
// 1.18.2 player: no UUID in Login Start -> Java's offline UUID; an old
|
||||||
|
// backend requesting version 1 gets version 1.
|
||||||
|
v := newVelocityForwarder(testSecret, "192.0.2.9")
|
||||||
|
feedLogin(v, 758, intentLogin, mkLoginStart(758, "Notch", nil, false), 3)
|
||||||
|
_, inject := v.ProcessS2C(mkPluginRequest(7, velocityChannel, []byte{0x01}))
|
||||||
|
if inject == nil {
|
||||||
|
t.Fatalf("no response was injected")
|
||||||
|
}
|
||||||
|
_, payload := parseResponse(t, testSecret, inject)
|
||||||
|
// UUID.nameUUIDFromBytes("OfflinePlayer:Notch".getBytes(UTF_8)).
|
||||||
|
assertPayload(t, payload, velocityVersionDefault, "192.0.2.9", "Notch",
|
||||||
|
"b50ad385829d3141a2167e7d7539ba7f")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVelocityVersionGating(t *testing.T) {
|
||||||
|
// A modern backend (requests 4) behind a pre-1.19.3 player must get v1.
|
||||||
|
v := newVelocityForwarder(testSecret, "192.0.2.9")
|
||||||
|
feedLogin(v, 758, intentLogin, mkLoginStart(758, "Old", nil, false), 5)
|
||||||
|
_, inject := v.ProcessS2C(mkPluginRequest(1, velocityChannel, []byte{0x04}))
|
||||||
|
_, payload := parseResponse(t, testSecret, inject)
|
||||||
|
ver, _ := payload.VarInt()
|
||||||
|
if ver != velocityVersionDefault {
|
||||||
|
t.Fatalf("version = %d, want 1 for a pre-1.19.3 player", ver)
|
||||||
|
}
|
||||||
|
// An empty request (very old backend) also means v1.
|
||||||
|
v2 := newVelocityForwarder(testSecret, "192.0.2.9")
|
||||||
|
feedLogin(v2, 767, intentLogin, mkLoginStart(767, "New", make([]byte, 16), false), 5)
|
||||||
|
_, inject2 := v2.ProcessS2C(mkPluginRequest(1, velocityChannel, nil))
|
||||||
|
_, payload2 := parseResponse(t, testSecret, inject2)
|
||||||
|
ver2, _ := payload2.VarInt()
|
||||||
|
if ver2 != velocityVersionDefault {
|
||||||
|
t.Fatalf("version = %d, want 1 for an empty version request", ver2)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVelocityLoginStartVariants(t *testing.T) {
|
||||||
|
uuid, _ := hex.DecodeString("ffeeddccbbaa99887766554433221100")
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
proto int
|
||||||
|
start []byte
|
||||||
|
uuidHex string
|
||||||
|
}{
|
||||||
|
{"1.19 with key, no uuid", 759, mkLoginStart(759, "Notch", nil, true),
|
||||||
|
"b50ad385829d3141a2167e7d7539ba7f"},
|
||||||
|
{"1.19.1 with key and uuid", 760, mkLoginStart(760, "Notch", uuid, true),
|
||||||
|
"ffeeddccbbaa99887766554433221100"},
|
||||||
|
{"1.19.3 optional uuid present", 761, mkLoginStart(761, "Notch", uuid, false),
|
||||||
|
"ffeeddccbbaa99887766554433221100"},
|
||||||
|
{"1.19.3 optional uuid absent", 761, mkLoginStart(761, "Notch", nil, false),
|
||||||
|
"b50ad385829d3141a2167e7d7539ba7f"},
|
||||||
|
}
|
||||||
|
for _, tc := range cases {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
v := newVelocityForwarder(testSecret, "192.0.2.1")
|
||||||
|
feedLogin(v, tc.proto, intentLogin, tc.start, 2)
|
||||||
|
_, inject := v.ProcessS2C(mkPluginRequest(3, velocityChannel, []byte{0x01}))
|
||||||
|
if inject == nil {
|
||||||
|
t.Fatalf("no response was injected")
|
||||||
|
}
|
||||||
|
_, payload := parseResponse(t, testSecret, inject)
|
||||||
|
assertPayload(t, payload, velocityVersionDefault, "192.0.2.1", "Notch", tc.uuidHex)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVelocityStatusPassthrough(t *testing.T) {
|
||||||
|
v := newVelocityForwarder(testSecret, "192.0.2.1")
|
||||||
|
feedLogin(v, 767, 1 /* status */, nil, 100)
|
||||||
|
if !v.Passthrough() {
|
||||||
|
t.Fatalf("status intent should turn the stream transparent")
|
||||||
|
}
|
||||||
|
data := []byte("not a minecraft packet at all")
|
||||||
|
if fwd, inj := v.ProcessS2C(data); !bytes.Equal(fwd, data) || inj != nil {
|
||||||
|
t.Fatalf("status traffic must pass through untouched")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVelocityOtherChannelForwarded(t *testing.T) {
|
||||||
|
uuid := make([]byte, 16)
|
||||||
|
v := newVelocityForwarder(testSecret, "192.0.2.1")
|
||||||
|
feedLogin(v, 767, intentLogin, mkLoginStart(767, "modded", uuid, false), 50)
|
||||||
|
|
||||||
|
// A modded-handshake query and the velocity query coalesced in one chunk:
|
||||||
|
// the first must reach the player, the second must not.
|
||||||
|
other := mkPluginRequest(1, "fml:loginwrapper", []byte{0x00, 0x01})
|
||||||
|
velo := mkPluginRequest(2, velocityChannel, []byte{0x04})
|
||||||
|
fwd, inject := v.ProcessS2C(append(append([]byte{}, other...), velo...))
|
||||||
|
if !bytes.Equal(fwd, other) {
|
||||||
|
t.Fatalf("non-velocity query not forwarded verbatim:\n got %x\nwant %x", fwd, other)
|
||||||
|
}
|
||||||
|
if inject == nil {
|
||||||
|
t.Fatalf("velocity query in the same chunk was not answered")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVelocityNoQueryLoginSuccess(t *testing.T) {
|
||||||
|
// Backend without velocity forwarding: the first login packet ends
|
||||||
|
// interception and everything flows verbatim.
|
||||||
|
uuid := make([]byte, 16)
|
||||||
|
v := newVelocityForwarder(testSecret, "192.0.2.1")
|
||||||
|
feedLogin(v, 767, intentLogin, mkLoginStart(767, "plain", uuid, false), 50)
|
||||||
|
|
||||||
|
success := mkPacket(wire.NewWriter().VarInt(loginS2CSuccess).Bytes(uuid).String("plain").VarInt(0).Out())
|
||||||
|
tail := []byte("compressed gibberish after login")
|
||||||
|
fwd, inject := v.ProcessS2C(append(append([]byte{}, success...), tail...))
|
||||||
|
if inject != nil {
|
||||||
|
t.Fatalf("nothing should be injected without a velocity query")
|
||||||
|
}
|
||||||
|
want := append(append([]byte{}, success...), tail...)
|
||||||
|
if !bytes.Equal(fwd, want) {
|
||||||
|
t.Fatalf("login success not flushed verbatim")
|
||||||
|
}
|
||||||
|
if !v.Passthrough() {
|
||||||
|
t.Fatalf("interceptor should be passthrough after Login Success")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVelocityQueryBeforeLoginStartFailsOpen(t *testing.T) {
|
||||||
|
// A query arriving before the Login Start was observed cannot be answered:
|
||||||
|
// it must reach the player unmodified (who will then be kicked by the
|
||||||
|
// backend with its own message).
|
||||||
|
v := newVelocityForwarder(testSecret, "192.0.2.1")
|
||||||
|
feedLogin(v, 767, intentLogin, nil, 50) // handshake only
|
||||||
|
req := mkPluginRequest(5, velocityChannel, []byte{0x04})
|
||||||
|
fwd, inject := v.ProcessS2C(req)
|
||||||
|
if inject != nil {
|
||||||
|
t.Fatalf("must not answer without a Login Start")
|
||||||
|
}
|
||||||
|
if !bytes.Equal(fwd, req) {
|
||||||
|
t.Fatalf("unanswerable query not passed through")
|
||||||
|
}
|
||||||
|
if !v.Passthrough() {
|
||||||
|
t.Fatalf("interceptor should fail open")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVelocityOversizedC2SFailsOpen(t *testing.T) {
|
||||||
|
v := newVelocityForwarder(testSecret, "192.0.2.1")
|
||||||
|
// A declared c2s packet length beyond the sniff cap aborts interception.
|
||||||
|
v.ObserveC2S(wire.AppendVarInt(nil, maxC2SSniff+1))
|
||||||
|
if !v.Passthrough() {
|
||||||
|
t.Fatalf("oversized login packet should turn the stream transparent")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestVelocityC2SAbortFlushesBufferedS2C(t *testing.T) {
|
||||||
|
v := newVelocityForwarder(testSecret, "192.0.2.1")
|
||||||
|
feedLogin(v, 767, intentLogin, nil, 50) // handshake only; login pending
|
||||||
|
req := mkPluginRequest(5, velocityChannel, []byte{0x04})
|
||||||
|
half := len(req) / 2
|
||||||
|
if fwd, _ := v.ProcessS2C(req[:half]); len(fwd) != 0 {
|
||||||
|
t.Fatalf("partial packet must stay buffered")
|
||||||
|
}
|
||||||
|
// The c2s side now aborts (e.g. unparseable player bytes) while s2c bytes
|
||||||
|
// sit buffered: they must not be lost.
|
||||||
|
v.ObserveC2S(wire.AppendVarInt(nil, maxC2SSniff+1))
|
||||||
|
fwd, inject := v.ProcessS2C(req[half:])
|
||||||
|
if inject != nil {
|
||||||
|
t.Fatalf("aborted interceptor must not inject")
|
||||||
|
}
|
||||||
|
if !bytes.Equal(fwd, req) {
|
||||||
|
t.Fatalf("buffered s2c bytes lost on abort:\n got %x\nwant %x", fwd, req)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -6,6 +6,7 @@ import (
|
|||||||
"io"
|
"io"
|
||||||
"net"
|
"net"
|
||||||
"sync"
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
"golang.org/x/crypto/chacha20"
|
"golang.org/x/crypto/chacha20"
|
||||||
)
|
)
|
||||||
@@ -13,7 +14,19 @@ import (
|
|||||||
// MaxFrame is the maximum decrypted frame payload size (1 MiB).
|
// MaxFrame is the maximum decrypted frame payload size (1 MiB).
|
||||||
const MaxFrame = 1 << 20
|
const MaxFrame = 1 << 20
|
||||||
|
|
||||||
var errFrameTooBig = errors.New("wire: frame exceeds max size")
|
// WriteTimeout bounds a single frame write. A peer that stops reading must not
|
||||||
|
// be able to park this connection inside WriteFrame forever: the write mutex
|
||||||
|
// is held for the whole socket write, so one stalled write would otherwise
|
||||||
|
// wedge liveness (PING/PONG) and WND/FIN on this tunnel.
|
||||||
|
const WriteTimeout = 30 * time.Second
|
||||||
|
|
||||||
|
var (
|
||||||
|
errFrameTooBig = errors.New("wire: frame exceeds max size")
|
||||||
|
// ErrBroken is returned once a write has failed. The ChaCha20 keystream has
|
||||||
|
// already advanced (and the socket may hold a partial frame), so the
|
||||||
|
// connection can never be resynchronized and is closed for good.
|
||||||
|
ErrBroken = errors.New("wire: connection is broken")
|
||||||
|
)
|
||||||
|
|
||||||
// FramedConn is the encrypted, length-prefixed frame transport (PROTOCOL.md §3.1).
|
// FramedConn is the encrypted, length-prefixed frame transport (PROTOCOL.md §3.1).
|
||||||
// The VarInt length prefix is plaintext; the payload is ChaCha20-encrypted with a
|
// The VarInt length prefix is plaintext; the payload is ChaCha20-encrypted with a
|
||||||
@@ -24,7 +37,9 @@ type FramedConn struct {
|
|||||||
r *bufio.Reader
|
r *bufio.Reader
|
||||||
in *chacha20.Cipher
|
in *chacha20.Cipher
|
||||||
out *chacha20.Cipher
|
out *chacha20.Cipher
|
||||||
wmu sync.Mutex
|
|
||||||
|
wmu sync.Mutex
|
||||||
|
broken bool
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewFramedConn(conn net.Conn, in, out *chacha20.Cipher) *FramedConn {
|
func NewFramedConn(conn net.Conn, in, out *chacha20.Cipher) *FramedConn {
|
||||||
@@ -62,15 +77,29 @@ func (f *FramedConn) ReadFrame() ([]byte, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// WriteFrame encrypts and sends one frame payload. Safe for concurrent callers.
|
// WriteFrame encrypts and sends one frame payload. Safe for concurrent callers.
|
||||||
|
//
|
||||||
|
// The write is bounded by WriteTimeout. On any write error the connection is
|
||||||
|
// marked broken and closed, which unblocks the reader so the owner can tear the
|
||||||
|
// session down instead of leaving every stream parked on the write mutex.
|
||||||
func (f *FramedConn) WriteFrame(payload []byte) error {
|
func (f *FramedConn) WriteFrame(payload []byte) error {
|
||||||
f.wmu.Lock()
|
f.wmu.Lock()
|
||||||
defer f.wmu.Unlock()
|
defer f.wmu.Unlock()
|
||||||
|
if f.broken {
|
||||||
|
return ErrBroken
|
||||||
|
}
|
||||||
ct := make([]byte, len(payload))
|
ct := make([]byte, len(payload))
|
||||||
f.out.XORKeyStream(ct, payload)
|
f.out.XORKeyStream(ct, payload)
|
||||||
out := AppendVarInt(make([]byte, 0, VarIntMaxBytes+len(ct)), len(ct))
|
out := AppendVarInt(make([]byte, 0, VarIntMaxBytes+len(ct)), len(ct))
|
||||||
out = append(out, ct...)
|
out = append(out, ct...)
|
||||||
|
_ = f.conn.SetWriteDeadline(time.Now().Add(WriteTimeout))
|
||||||
_, err := f.conn.Write(out)
|
_, err := f.conn.Write(out)
|
||||||
return err
|
if err != nil {
|
||||||
|
f.broken = true
|
||||||
|
_ = f.conn.Close()
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
_ = f.conn.SetWriteDeadline(time.Time{})
|
||||||
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (f *FramedConn) Close() error { return f.conn.Close() }
|
func (f *FramedConn) Close() error { return f.conn.Close() }
|
||||||
|
|||||||
+533
-140
@@ -1,75 +1,129 @@
|
|||||||
package client
|
package client
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
"log"
|
"log"
|
||||||
"net"
|
"net"
|
||||||
"sync"
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/iceBear67/redapricot/client/wire"
|
"github.com/iceBear67/redapricot/client/wire"
|
||||||
)
|
)
|
||||||
|
|
||||||
// WorkerPool manages up to maxConn worker connections and allocates streams
|
// errPoolClosed is returned by Dial after Close: the set is shutting down
|
||||||
// using the least-loaded strategy (PROTOCOL.md §7.1).
|
// and must not start new dials, so a caller (handleControlRequest,
|
||||||
|
// allocateForResume) gives up rather than waiting on a hub that will never
|
||||||
|
// be used.
|
||||||
|
var errPoolClosed = errors.New("worker set closed")
|
||||||
|
|
||||||
|
// errTooManyTunnels is returned when live+in-flight worker conns already
|
||||||
|
// equal maxTunnels. The ControlRequest is dropped; the hub closes the player
|
||||||
|
// when pendingTimeoutMs fires.
|
||||||
|
var errTooManyTunnels = errors.New("maxTunnels reached")
|
||||||
|
|
||||||
|
// dialAttempts bounds how many times a caller retries Dial when the conn it
|
||||||
|
// was handed dies before the stream could be attached to it. The race is
|
||||||
|
// narrow; an unbounded loop would spin against a hub that is refusing every
|
||||||
|
// connection.
|
||||||
|
const dialAttempts = 3
|
||||||
|
|
||||||
|
// WorkerPool tracks live 1:1 worker connections up to maxTunnels
|
||||||
|
// (PROTOCOL.md §7.1). There is no least-loaded placement and no sharing:
|
||||||
|
// every player gets its own TCP connection.
|
||||||
type WorkerPool struct {
|
type WorkerPool struct {
|
||||||
client *Client
|
client *Client
|
||||||
maxConn int
|
maxTunnels int
|
||||||
|
|
||||||
mu sync.Mutex
|
connSeq atomic.Int64 // conn ids, for log correlation
|
||||||
conns []*WorkerConn
|
|
||||||
|
mu sync.Mutex
|
||||||
|
conns map[*WorkerConn]struct{}
|
||||||
|
dialing int // dials currently in flight
|
||||||
|
closed bool // closeAll ran; no new conns may join
|
||||||
}
|
}
|
||||||
|
|
||||||
func newWorkerPool(c *Client, maxConn int) *WorkerPool {
|
func newWorkerPool(c *Client, maxTunnels int) *WorkerPool {
|
||||||
return &WorkerPool{client: c, maxConn: maxConn}
|
return &WorkerPool{
|
||||||
|
client: c,
|
||||||
|
maxTunnels: maxTunnels,
|
||||||
|
conns: make(map[*WorkerConn]struct{}),
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Allocate returns a worker conn and a fresh stream id to place a new stream on.
|
// Dial opens a dedicated worker conn for one player.
|
||||||
func (p *WorkerPool) Allocate() (*WorkerConn, int, error) {
|
//
|
||||||
|
// A dial is never performed while holding p.mu: session establishment involves
|
||||||
|
// network I/O, and holding the lock across it would park every other player
|
||||||
|
// behind one unresponsive hub. Each caller dials independently; unlike the
|
||||||
|
// old mux pool there is no shared conn to wait for.
|
||||||
|
func (p *WorkerPool) Dial() (*WorkerConn, error) {
|
||||||
p.mu.Lock()
|
p.mu.Lock()
|
||||||
defer p.mu.Unlock()
|
if p.closed {
|
||||||
|
p.mu.Unlock()
|
||||||
var best *WorkerConn
|
return nil, errPoolClosed
|
||||||
bestCount := 0
|
|
||||||
for _, wc := range p.conns {
|
|
||||||
n := wc.streamCount()
|
|
||||||
if best == nil || n < bestCount {
|
|
||||||
best = wc
|
|
||||||
bestCount = n
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
if len(p.conns)+p.dialing >= p.maxTunnels {
|
||||||
needNew := best == nil || (bestCount > SaturationThreshold && len(p.conns) < p.maxConn)
|
p.mu.Unlock()
|
||||||
if needNew {
|
return nil, errTooManyTunnels
|
||||||
wc, err := p.dialWorker()
|
|
||||||
if err != nil {
|
|
||||||
if best == nil {
|
|
||||||
return nil, 0, err
|
|
||||||
}
|
|
||||||
log.Printf("worker dial failed, reusing existing conn: %v", err)
|
|
||||||
} else {
|
|
||||||
p.conns = append(p.conns, wc)
|
|
||||||
best = wc
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
return best, best.newSid(), nil
|
p.dialing++
|
||||||
|
p.mu.Unlock()
|
||||||
|
|
||||||
|
wc, err := p.dialWorker()
|
||||||
|
|
||||||
|
p.mu.Lock()
|
||||||
|
p.dialing--
|
||||||
|
if err != nil {
|
||||||
|
p.mu.Unlock()
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if p.closed {
|
||||||
|
// Close raced this dial: the conn must not enter the set.
|
||||||
|
p.mu.Unlock()
|
||||||
|
_ = wc.fc.Close()
|
||||||
|
return nil, errPoolClosed
|
||||||
|
}
|
||||||
|
if len(p.conns) >= p.maxTunnels {
|
||||||
|
p.mu.Unlock()
|
||||||
|
_ = wc.fc.Close()
|
||||||
|
return nil, errTooManyTunnels
|
||||||
|
}
|
||||||
|
p.conns[wc] = struct{}{}
|
||||||
|
p.mu.Unlock()
|
||||||
|
return wc, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// dialWorker establishes one worker conn. It must be called without p.mu held.
|
||||||
|
// The dial runs on the client's context so Close aborts it mid-handshake.
|
||||||
func (p *WorkerPool) dialWorker() (*WorkerConn, error) {
|
func (p *WorkerPool) dialWorker() (*WorkerConn, error) {
|
||||||
fc, peerWnd, err := p.client.dialSession(MagicWorker)
|
sess, err := p.client.dialSession(p.client.ctx, MagicWorker)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
wc := &WorkerConn{
|
wc := &WorkerConn{
|
||||||
pool: p,
|
pool: p,
|
||||||
fc: fc,
|
fc: sess.fc,
|
||||||
sendWndInit: peerWnd,
|
sendWndInit: sess.peerWnd,
|
||||||
recvWndInit: p.client.streamWnd,
|
resume: sess.resume,
|
||||||
streams: make(map[int]*Stream),
|
grace: p.client.resumeGrace(sess.hubGrace),
|
||||||
nextSid: 1,
|
id: int(p.connSeq.Add(1)),
|
||||||
|
done: make(chan struct{}),
|
||||||
}
|
}
|
||||||
|
if p.client.statsOn() {
|
||||||
|
wc.stats = &connStats{opened: time.Now()}
|
||||||
|
}
|
||||||
|
wc.lastPong.Store(time.Now().UnixMilli())
|
||||||
go wc.readLoop()
|
go wc.readLoop()
|
||||||
log.Printf("opened worker conn (#%d in pool, send window %d, recv window %d)",
|
if sess.heartbeat {
|
||||||
len(p.conns)+1, wc.sendWndInit, wc.recvWndInit)
|
go wc.heartbeatLoop(p.client.cfg.pingInterval(), p.client.cfg.heartbeatTimeout())
|
||||||
|
} else {
|
||||||
|
log.Printf("worker conn: hub does not support the worker heartbeat; " +
|
||||||
|
"a silently dropped path will only be caught by TCP keepalive")
|
||||||
|
}
|
||||||
|
log.Printf("opened worker conn (send window %d, recv window %d, heartbeat %v, resume %v)",
|
||||||
|
wc.sendWndInit, p.client.streamWnd, sess.heartbeat, wc.resume)
|
||||||
return wc, nil
|
return wc, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -82,144 +136,228 @@ func (p *WorkerPool) count() int {
|
|||||||
func (p *WorkerPool) remove(wc *WorkerConn) {
|
func (p *WorkerPool) remove(wc *WorkerConn) {
|
||||||
p.mu.Lock()
|
p.mu.Lock()
|
||||||
defer p.mu.Unlock()
|
defer p.mu.Unlock()
|
||||||
for i, c := range p.conns {
|
delete(p.conns, wc)
|
||||||
if c == wc {
|
|
||||||
p.conns = append(p.conns[:i], p.conns[i+1:]...)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p *WorkerPool) closeAll() {
|
func (p *WorkerPool) closeAll() {
|
||||||
p.mu.Lock()
|
p.mu.Lock()
|
||||||
conns := append([]*WorkerConn(nil), p.conns...)
|
p.closed = true
|
||||||
|
conns := make([]*WorkerConn, 0, len(p.conns))
|
||||||
|
for wc := range p.conns {
|
||||||
|
conns = append(conns, wc)
|
||||||
|
}
|
||||||
p.mu.Unlock()
|
p.mu.Unlock()
|
||||||
for _, wc := range conns {
|
for _, wc := range conns {
|
||||||
_ = wc.fc.Close()
|
_ = wc.fc.Close()
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// WorkerConn is one multiplexed worker connection to the hub.
|
// WorkerConn is one 1:1 worker connection to the hub: it carries exactly one
|
||||||
|
// player. Only genuinely per-connection state lives here. Client-wide values
|
||||||
|
// (the shaper, the advertised receive window, the DATA chunk cap) belong to
|
||||||
|
// Client: reading them through a connection pointer would make every such
|
||||||
|
// read a re-parenting hazard once a stream can migrate between conns.
|
||||||
type WorkerConn struct {
|
type WorkerConn struct {
|
||||||
pool *WorkerPool
|
pool *WorkerPool
|
||||||
fc *wire.FramedConn
|
fc *wire.FramedConn
|
||||||
|
|
||||||
sendWndInit int // hub's advertised per-stream receive window (our send budget)
|
sendWndInit int // hub's advertised receive window (our send budget)
|
||||||
recvWndInit int // our advertised per-stream receive window (bounds each recv queue)
|
|
||||||
|
|
||||||
mu sync.Mutex
|
// resume is whether this conn negotiated stream resumption, and grace how
|
||||||
streams map[int]*Stream
|
// long a stream parked from it may keep trying to reattach. Both are
|
||||||
nextSid int
|
// per-conn: a reattach may land on a different (or restarted) hub, so the
|
||||||
|
// flag must be re-checked on the conn that will carry the RESUME.
|
||||||
|
resume bool
|
||||||
|
grace time.Duration
|
||||||
|
|
||||||
|
done chan struct{} // closed when readLoop exits
|
||||||
|
lastPong atomic.Int64 // unix ms of the most recent PONG
|
||||||
|
|
||||||
|
id int // for log correlation only
|
||||||
|
stats *connStats // nil unless diagnostics are enabled
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
stream *Stream
|
||||||
|
closed bool // readLoop has exited; attach must refuse
|
||||||
}
|
}
|
||||||
|
|
||||||
func (wc *WorkerConn) streamCount() int {
|
// heartbeatLoop proves the worker conn is still carrying frames end to end. TCP
|
||||||
|
// alone cannot tell us: a middlebox that drops an established flow (conntrack
|
||||||
|
// expiry, firewall state loss) sends no FIN or RST, so the read loop would park
|
||||||
|
// forever and the player on this conn would silently fail until the process
|
||||||
|
// restarted.
|
||||||
|
func (wc *WorkerConn) heartbeatLoop(interval, timeout time.Duration) {
|
||||||
|
ticker := time.NewTicker(interval)
|
||||||
|
defer ticker.Stop()
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-wc.done:
|
||||||
|
return
|
||||||
|
case <-ticker.C:
|
||||||
|
if silent := time.Since(time.UnixMilli(wc.lastPong.Load())); silent > timeout {
|
||||||
|
log.Printf("worker conn silent for %s; dropping it", silent.Round(time.Second))
|
||||||
|
_ = wc.fc.Close() // readLoop unblocks and tears everything down
|
||||||
|
return
|
||||||
|
}
|
||||||
|
msg := wire.NewWriter().U8(MuxPing).I64(time.Now().UnixMilli()).Out()
|
||||||
|
if err := wc.fc.WriteFrame(msg); err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// attach publishes a stream on this conn, or reports false if the conn has
|
||||||
|
// already died (or is already bound — which would be a caller bug).
|
||||||
|
//
|
||||||
|
// The check is not advisory. Dial hands out a conn, and the conn's readLoop
|
||||||
|
// can exit before the caller gets here. A blind store would land on a conn
|
||||||
|
// nothing iterates and the stream would never be torn down.
|
||||||
|
func (wc *WorkerConn) attach(st *Stream) bool {
|
||||||
wc.mu.Lock()
|
wc.mu.Lock()
|
||||||
defer wc.mu.Unlock()
|
defer wc.mu.Unlock()
|
||||||
return len(wc.streams)
|
if wc.closed || wc.stream != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
wc.stream = st
|
||||||
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
func (wc *WorkerConn) newSid() int {
|
func (wc *WorkerConn) getStream() *Stream {
|
||||||
wc.mu.Lock()
|
wc.mu.Lock()
|
||||||
defer wc.mu.Unlock()
|
defer wc.mu.Unlock()
|
||||||
sid := wc.nextSid
|
return wc.stream
|
||||||
wc.nextSid++
|
|
||||||
return sid
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func (wc *WorkerConn) registerStream(sid int, st *Stream) {
|
func (wc *WorkerConn) detach() *Stream {
|
||||||
wc.mu.Lock()
|
|
||||||
wc.streams[sid] = st
|
|
||||||
wc.mu.Unlock()
|
|
||||||
}
|
|
||||||
|
|
||||||
func (wc *WorkerConn) getStream(sid int) *Stream {
|
|
||||||
wc.mu.Lock()
|
wc.mu.Lock()
|
||||||
defer wc.mu.Unlock()
|
defer wc.mu.Unlock()
|
||||||
return wc.streams[sid]
|
st := wc.stream
|
||||||
}
|
wc.stream = nil
|
||||||
|
|
||||||
func (wc *WorkerConn) removeStream(sid int) *Stream {
|
|
||||||
wc.mu.Lock()
|
|
||||||
defer wc.mu.Unlock()
|
|
||||||
st := wc.streams[sid]
|
|
||||||
delete(wc.streams, sid)
|
|
||||||
return st
|
return st
|
||||||
}
|
}
|
||||||
|
|
||||||
// readLoop dispatches inbound mux frames. It must never block on a stream's
|
// readLoop dispatches inbound tunnel frames. DATA is only enqueued (the
|
||||||
// destination: DATA is only enqueued (the per-stream writeLoop does the actual
|
// stream's writeLoop does the actual destination writes), so a stalled
|
||||||
// destination writes), so one slow destination cannot stall other streams.
|
// destination cannot stall liveness / WND / FIN dispatch on this conn.
|
||||||
func (wc *WorkerConn) readLoop() {
|
func (wc *WorkerConn) readLoop() {
|
||||||
for {
|
for {
|
||||||
payload, err := wc.fc.ReadFrame()
|
payload, err := wc.fc.ReadFrame()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
|
if wc.stats != nil {
|
||||||
|
wc.stats.framesIn.Add(1)
|
||||||
|
}
|
||||||
r := wire.NewReader(payload)
|
r := wire.NewReader(payload)
|
||||||
ftype, err := r.U8()
|
ftype, err := r.U8()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
sid, err := r.VarInt()
|
|
||||||
if err != nil {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
switch ftype {
|
switch ftype {
|
||||||
case MuxData:
|
case MuxData:
|
||||||
if st := wc.getStream(sid); st != nil {
|
if st := wc.getStream(); st != nil {
|
||||||
st.deliverFromHub(r.Remaining())
|
st.deliverFromHub(r.Remaining())
|
||||||
}
|
}
|
||||||
case MuxWnd:
|
case MuxWnd:
|
||||||
if delta, err := r.VarInt(); err == nil && delta > 0 {
|
if delta, err := r.VarInt(); err == nil && delta > 0 {
|
||||||
if st := wc.getStream(sid); st != nil {
|
if st := wc.getStream(); st != nil {
|
||||||
st.grantSendWnd(delta)
|
st.grantSendWnd(delta)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
case MuxFin:
|
case MuxFin:
|
||||||
// Graceful: drain what is already queued to the destination first.
|
// Graceful: drain what is already queued to the destination first.
|
||||||
if st := wc.removeStream(sid); st != nil {
|
if st := wc.detach(); st != nil {
|
||||||
st.gracefulFin()
|
st.gracefulFin()
|
||||||
}
|
}
|
||||||
case MuxRst:
|
case MuxRst:
|
||||||
if st := wc.removeStream(sid); st != nil {
|
// The reason is an optional trailing byte; older peers send none.
|
||||||
st.teardown(false)
|
reason := RstUnspecified
|
||||||
|
if b, err := r.U8(); err == nil {
|
||||||
|
reason = int(b)
|
||||||
|
}
|
||||||
|
if st := wc.detach(); st != nil {
|
||||||
|
st.onRst(reason)
|
||||||
|
}
|
||||||
|
case MuxResumeAck:
|
||||||
|
accepted, aerr := r.I64()
|
||||||
|
delivered, derr := r.I64()
|
||||||
|
cid, cerr := r.Bytes(CIDLen)
|
||||||
|
if aerr != nil || derr != nil || cerr != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if st := wc.getStream(); st != nil {
|
||||||
|
st.deliverResume(resumeResult{accepted: accepted, delivered: delivered, cid: cid})
|
||||||
|
}
|
||||||
|
case MuxPing:
|
||||||
|
nonce, _ := r.I64()
|
||||||
|
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxPong).I64(nonce).Out())
|
||||||
|
case MuxPong:
|
||||||
|
now := time.Now()
|
||||||
|
wc.lastPong.Store(now.UnixMilli())
|
||||||
|
// The probe's nonce is the timestamp we sent, echoed back, so the
|
||||||
|
// round trip is free to measure and nobody was reading it.
|
||||||
|
if wc.stats != nil {
|
||||||
|
if sent, err := r.I64(); err == nil {
|
||||||
|
wc.stats.observeRTT(now.Sub(time.UnixMilli(sent)))
|
||||||
|
}
|
||||||
}
|
}
|
||||||
default:
|
default:
|
||||||
log.Printf("worker: unknown mux type %d", ftype)
|
log.Printf("worker: unknown frame type %d", ftype)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
// Connection lost: tear down all streams and drop from pool.
|
// Connection lost: tear down the bound stream and drop from the set.
|
||||||
|
close(wc.done)
|
||||||
wc.pool.remove(wc)
|
wc.pool.remove(wc)
|
||||||
wc.mu.Lock()
|
wc.mu.Lock()
|
||||||
streams := make([]*Stream, 0, len(wc.streams))
|
// Marked before the pointer is cleared, under the same lock, so a concurrent
|
||||||
for _, st := range wc.streams {
|
// attach either lands in the field we are about to drain or is refused.
|
||||||
streams = append(streams, st)
|
wc.closed = true
|
||||||
}
|
st := wc.stream
|
||||||
wc.streams = make(map[int]*Stream)
|
wc.stream = nil
|
||||||
wc.mu.Unlock()
|
wc.mu.Unlock()
|
||||||
for _, st := range streams {
|
if st == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// Only the tunnel leg died. Where the session negotiated resumption the
|
||||||
|
// destination socket is kept open and the stream reattaches over a fresh
|
||||||
|
// conn (§7.5); otherwise this is the old, unconditional teardown.
|
||||||
|
// During Close there is no reattach to come: a stream that parked now would
|
||||||
|
// hold its destination socket open past shutdown, so teardown instead.
|
||||||
|
if wc.pool.client.closing.Load() {
|
||||||
|
st.teardown(false)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !st.park(wc.grace) {
|
||||||
st.teardown(false)
|
st.teardown(false)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (wc *WorkerConn) sendSyn(sid int, cid []byte) {
|
func (wc *WorkerConn) sendSyn(cid []byte) error {
|
||||||
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxSyn).VarInt(sid).Bytes(cid).Out())
|
return wc.fc.WriteFrame(wire.NewWriter().U8(MuxSyn).Bytes(cid).Out())
|
||||||
}
|
}
|
||||||
|
|
||||||
func (wc *WorkerConn) sendData(sid int, data []byte) error {
|
func (wc *WorkerConn) sendData(data []byte) error {
|
||||||
return wc.fc.WriteFrame(wire.NewWriter().U8(MuxData).VarInt(sid).Bytes(data).Out())
|
err := wc.fc.WriteFrame(wire.NewWriter().U8(MuxData).Bytes(data).Out())
|
||||||
|
if wc.stats != nil {
|
||||||
|
wc.stats.framesOut.Add(1)
|
||||||
|
if err != nil {
|
||||||
|
wc.stats.writeErrs.Add(1)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
func (wc *WorkerConn) sendFin(sid int) {
|
func (wc *WorkerConn) sendFin() {
|
||||||
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxFin).VarInt(sid).Out())
|
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxFin).Out())
|
||||||
}
|
}
|
||||||
|
|
||||||
func (wc *WorkerConn) sendRst(sid int) {
|
func (wc *WorkerConn) sendRst() {
|
||||||
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxRst).VarInt(sid).Out())
|
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxRst).Out())
|
||||||
}
|
}
|
||||||
|
|
||||||
func (wc *WorkerConn) sendWndUpdate(sid, delta int) {
|
func (wc *WorkerConn) sendWndUpdate(delta int) {
|
||||||
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxWnd).VarInt(sid).VarInt(delta).Out())
|
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxWnd).VarInt(delta).Out())
|
||||||
}
|
}
|
||||||
|
|
||||||
// Stream bridges one player (via the hub) to one destination connection.
|
// Stream bridges one player (via the hub) to one destination connection.
|
||||||
@@ -227,41 +365,123 @@ func (wc *WorkerConn) sendWndUpdate(sid, delta int) {
|
|||||||
// Data from the hub is queued and written to the destination by a dedicated
|
// Data from the hub is queued and written to the destination by a dedicated
|
||||||
// writeLoop goroutine. The queue is bounded by the advertised receive window —
|
// writeLoop goroutine. The queue is bounded by the advertised receive window —
|
||||||
// the hub never sends more un-credited bytes, so overflow is a protocol
|
// the hub never sends more un-credited bytes, so overflow is a protocol
|
||||||
// violation and resets the stream.
|
// violation and resets the tunnel.
|
||||||
type Stream struct {
|
type Stream struct {
|
||||||
wc *WorkerConn
|
client *Client
|
||||||
sid int
|
wc atomic.Pointer[WorkerConn]
|
||||||
cid []byte
|
|
||||||
mapping Mapping
|
mapping Mapping
|
||||||
srcIP string
|
srcIP string
|
||||||
srcPort int
|
srcPort int
|
||||||
|
vel *velocityForwarder // non-nil when the mapping sets velocitySecret
|
||||||
|
|
||||||
|
share shaperShare // this stream's position in the egress fair queue
|
||||||
|
done chan struct{} // closed on teardown; unparks a shaper wait
|
||||||
|
|
||||||
|
// resumable is fixed at creation from the conn's negotiated flag. With it
|
||||||
|
// false none of the bookkeeping below runs and no buffer is ever allocated,
|
||||||
|
// so a client with resumption disabled pays exactly what it used to.
|
||||||
|
resumable bool
|
||||||
|
|
||||||
|
stats *streamStats // nil unless diagnostics are enabled
|
||||||
|
|
||||||
|
// ackedOffset is the running sum of WND deltas received. Credit is granted
|
||||||
|
// only as bytes reach the peer's terminal socket, so it is a sound lower
|
||||||
|
// bound on what has been delivered. Atomic because the worker readLoop
|
||||||
|
// advances it and must never block behind the send path.
|
||||||
|
ackedOffset atomic.Int64
|
||||||
|
|
||||||
|
// sendMu serializes the send path — buffer the chunk, advance the offset,
|
||||||
|
// write the frame — against a reattach's retransmit, so replayed bytes can
|
||||||
|
// never interleave with live ones.
|
||||||
|
//
|
||||||
|
// Deliberately not s.mu: deliverFromHub takes s.mu from the worker readLoop,
|
||||||
|
// and holding s.mu across a WriteFrame would stall frame dispatch (WND/FIN
|
||||||
|
// /heartbeat replies) on this conn.
|
||||||
|
sendMu sync.Mutex
|
||||||
|
un unackedBuf // guarded by sendMu
|
||||||
|
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
cond *sync.Cond
|
cond *sync.Cond
|
||||||
|
cid []byte // takeover capability; re-minted by the hub on each resume
|
||||||
dest net.Conn
|
dest net.Conn
|
||||||
connected bool
|
connected bool
|
||||||
closed bool
|
closed bool
|
||||||
finPending bool // hub sent FIN; close the destination once the queue drains
|
parked bool // worker conn died; awaiting reattach on a fresh one
|
||||||
q [][]byte // hub -> destination, waiting for writeLoop
|
parkedAt time.Time // when the current hang began; diagnostics only
|
||||||
qBytes int
|
finPending bool // hub sent FIN; close the destination once the queue drains
|
||||||
sendWnd int // flow control: budget for destination -> hub DATA
|
finToHub bool // destination closed while parked; FIN owed once reattached
|
||||||
consumed int // flow control: drained bytes not yet credited back to the hub
|
q []qentry // hub/local -> destination, waiting for writeLoop
|
||||||
|
qBytes int // hub bytes only: bounds the peer against its window
|
||||||
|
// acceptedOffset counts hub bytes enqueued toward the destination. This, not
|
||||||
|
// "bytes written", is what a reattach reports: acceptance is synchronous and
|
||||||
|
// stable at park time, whereas delivery is signalled asynchronously and goes
|
||||||
|
// silent exactly when the connection dies — which would under-report and make
|
||||||
|
// the hub retransmit bytes the player already has.
|
||||||
|
acceptedOffset int64
|
||||||
|
// deliveredOffset counts hub bytes actually written to the destination.
|
||||||
|
// Distinct from acceptedOffset and needed for a different job: a reattach
|
||||||
|
// replays from what the peer *accepted*, but sizes the window from what it
|
||||||
|
// *delivered*, because the window is a promise about undelivered bytes.
|
||||||
|
deliveredOffset int64
|
||||||
|
sendWnd int // flow control: budget for destination -> hub DATA
|
||||||
|
consumed int // flow control: drained bytes not yet credited back to the hub
|
||||||
|
resumeWait chan resumeResult
|
||||||
}
|
}
|
||||||
|
|
||||||
func newStream(wc *WorkerConn, sid int, cid []byte, m Mapping, ip string, port int) *Stream {
|
// resumeResult is the hub's answer to a RESUME: how far it got in both senses,
|
||||||
s := &Stream{wc: wc, sid: sid, cid: cid, mapping: m, srcIP: ip, srcPort: port, sendWnd: wc.sendWndInit}
|
// plus a fresh CID — or the reason the reattach was refused.
|
||||||
|
type resumeResult struct {
|
||||||
|
accepted int64
|
||||||
|
delivered int64
|
||||||
|
cid []byte
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
// qentry is one queued write towards the destination. Only hub-originated
|
||||||
|
// entries take part in flow control; locally injected bytes (the velocity
|
||||||
|
// login response) are neither counted against the hub's window nor credited
|
||||||
|
// back when drained.
|
||||||
|
type qentry struct {
|
||||||
|
data []byte
|
||||||
|
fromHub bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func newStream(c *Client, wc *WorkerConn, cid []byte, m Mapping, ip string, port int) *Stream {
|
||||||
|
s := &Stream{client: c, cid: cid, mapping: m, srcIP: ip, srcPort: port,
|
||||||
|
resumable: wc.resume, sendWnd: wc.sendWndInit, done: make(chan struct{})}
|
||||||
|
s.wc.Store(wc)
|
||||||
|
if c.statsOn() {
|
||||||
|
s.stats = &streamStats{opened: time.Now()}
|
||||||
|
}
|
||||||
|
if m.VelocitySecret != "" {
|
||||||
|
s.vel = newVelocityForwarder(m.VelocitySecret, ip)
|
||||||
|
}
|
||||||
s.cond = sync.NewCond(&s.mu)
|
s.cond = sync.NewCond(&s.mu)
|
||||||
return s
|
return s
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// conn returns the stream's current worker conn. May be the original or a
|
||||||
|
// reattach; callers that send must take one snapshot and use it for the
|
||||||
|
// whole operation so a concurrent rebind cannot split a write across conns.
|
||||||
|
func (s *Stream) conn() *WorkerConn { return s.wc.Load() }
|
||||||
|
|
||||||
|
func (s *Stream) name() string {
|
||||||
|
if wc := s.conn(); wc != nil {
|
||||||
|
return fmt.Sprintf("conn%d", wc.id)
|
||||||
|
}
|
||||||
|
return "conn?"
|
||||||
|
}
|
||||||
|
|
||||||
// run dials the destination, optionally writes the PROXY v2 header, then pumps
|
// run dials the destination, optionally writes the PROXY v2 header, then pumps
|
||||||
// destination -> hub (respecting the stream send window when negotiated).
|
// destination -> hub (respecting the send window when negotiated).
|
||||||
func (s *Stream) run() {
|
func (s *Stream) run() {
|
||||||
dest, err := net.DialTimeout("tcp", s.mapping.Destination, 10*time.Second)
|
dest, err := net.DialTimeout("tcp", s.mapping.Destination, 10*time.Second)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Printf("stream %d: dial %s failed: %v", s.sid, s.mapping.Destination, err)
|
log.Printf("stream %s: dial %s failed: %v", s.name(), s.mapping.Destination, err)
|
||||||
s.wc.removeStream(s.sid)
|
if wc := s.conn(); wc != nil {
|
||||||
s.wc.sendRst(s.sid)
|
wc.detach()
|
||||||
|
wc.sendRst()
|
||||||
|
}
|
||||||
s.teardown(false)
|
s.teardown(false)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -272,7 +492,7 @@ func (s *Stream) run() {
|
|||||||
if s.mapping.ProxyProtocol {
|
if s.mapping.ProxyProtocol {
|
||||||
if hdr := s.buildProxyHeader(dest); hdr != nil {
|
if hdr := s.buildProxyHeader(dest); hdr != nil {
|
||||||
if _, err := dest.Write(hdr); err != nil {
|
if _, err := dest.Write(hdr); err != nil {
|
||||||
log.Printf("stream %d: proxy header write: %v", s.sid, err)
|
log.Printf("stream %s: proxy header write: %v", s.name(), err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -285,6 +505,12 @@ func (s *Stream) run() {
|
|||||||
}
|
}
|
||||||
s.dest = dest
|
s.dest = dest
|
||||||
s.connected = true
|
s.connected = true
|
||||||
|
if s.finPending {
|
||||||
|
// The hub FIN'd while we were still dialing, so gracefulFin could not
|
||||||
|
// arm the drain deadline (there was no destination yet). Arm it now,
|
||||||
|
// otherwise writeLoop can block on an unresponsive destination forever.
|
||||||
|
_ = dest.SetWriteDeadline(time.Now().Add(finDrainTimeout))
|
||||||
|
}
|
||||||
s.cond.Broadcast() // wake writeLoop: queued hub bytes can flow now
|
s.cond.Broadcast() // wake writeLoop: queued hub bytes can flow now
|
||||||
s.mu.Unlock()
|
s.mu.Unlock()
|
||||||
|
|
||||||
@@ -293,10 +519,15 @@ func (s *Stream) run() {
|
|||||||
for {
|
for {
|
||||||
n, err := dest.Read(buf)
|
n, err := dest.Read(buf)
|
||||||
if n > 0 {
|
if n > 0 {
|
||||||
if !s.acquireSendWnd(n) {
|
chunk := buf[:n]
|
||||||
break
|
if s.vel != nil && !s.vel.Passthrough() {
|
||||||
|
fwd, inject := s.vel.ProcessS2C(chunk)
|
||||||
|
if len(inject) > 0 {
|
||||||
|
s.injectToDest(inject)
|
||||||
|
}
|
||||||
|
chunk = fwd
|
||||||
}
|
}
|
||||||
if werr := s.wc.sendData(s.sid, buf[:n]); werr != nil {
|
if !s.sendToHub(chunk) {
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -307,6 +538,94 @@ func (s *Stream) run() {
|
|||||||
s.teardown(true)
|
s.teardown(true)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// sendToHub forwards destination bytes to the hub in bounded DATA frames,
|
||||||
|
// honoring both the send window and the client-wide bandwidth cap.
|
||||||
|
// Returns false once the stream closed or the worker conn failed.
|
||||||
|
func (s *Stream) sendToHub(data []byte) bool {
|
||||||
|
for len(data) > 0 {
|
||||||
|
n := len(data)
|
||||||
|
if n > s.client.chunk {
|
||||||
|
n = s.client.chunk
|
||||||
|
}
|
||||||
|
// Credit first, bandwidth second. The reverse order would charge the
|
||||||
|
// budget for bytes still parked on an exhausted window, so the client
|
||||||
|
// would throttle itself below the configured rate. Holding credit while
|
||||||
|
// waiting for tokens is free — credit is per-tunnel, and the hub returns
|
||||||
|
// it as it drains data to the player, independent of our pacing.
|
||||||
|
if !s.acquireSendWnd(n) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
var shaperStall stallClock
|
||||||
|
shaperStall.begin(s.stats != nil)
|
||||||
|
if !s.client.shaper.Acquire(&s.share, n, s.done) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
// Sampled here, before the socket write: emit's write to the hub can
|
||||||
|
// block when the hub is not reading, and charging that time to the
|
||||||
|
// bandwidth cap would blame maxBandwidth for a hub that is not draining.
|
||||||
|
waited := shaperStall.elapsed()
|
||||||
|
if !s.emit(data[:n]) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
if s.stats != nil {
|
||||||
|
s.mu.Lock()
|
||||||
|
// Separated from the window stall on purpose: this one says the
|
||||||
|
// configured cap is the binding constraint, and raising maxBandwidth
|
||||||
|
// is the fix. The window stall says the opposite.
|
||||||
|
s.stats.shaperStall += waited
|
||||||
|
s.stats.bytesUp += int64(n)
|
||||||
|
s.mu.Unlock()
|
||||||
|
}
|
||||||
|
data = data[n:]
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// emit records a chunk for possible retransmission and writes it to the current
|
||||||
|
// worker conn. Returns false once the stream is finished with.
|
||||||
|
//
|
||||||
|
// The record is taken first and unconditionally. A write that fails on a dying
|
||||||
|
// connection has already spent window and may have put part of the frame on the
|
||||||
|
// wire, so the only trustworthy account of what the peer still owes us is the
|
||||||
|
// one taken before the attempt. That also makes a failure survivable: while the
|
||||||
|
// stream can still be resumed the bytes are already safe, and the reattach
|
||||||
|
// replays them from wherever the hub says it got to.
|
||||||
|
func (s *Stream) emit(chunk []byte) bool {
|
||||||
|
s.sendMu.Lock()
|
||||||
|
if s.resumable {
|
||||||
|
// Reclaim what the hub has credited before growing the buffer, so the
|
||||||
|
// outstanding region stays bounded by one window.
|
||||||
|
s.un.advance(s.ackedOffset.Load())
|
||||||
|
s.un.append(chunk)
|
||||||
|
}
|
||||||
|
wc := s.conn()
|
||||||
|
var err error
|
||||||
|
if wc != nil {
|
||||||
|
err = wc.sendData(chunk)
|
||||||
|
} else {
|
||||||
|
err = errPoolClosed
|
||||||
|
}
|
||||||
|
s.sendMu.Unlock()
|
||||||
|
|
||||||
|
if err == nil {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if !s.resumable {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
// The conn is gone but the stream is not: readLoop parks it and a reattach
|
||||||
|
// replays the buffer. Keep pumping the destination — acquireSendWnd stops us
|
||||||
|
// once a full window is outstanding, so nothing is lost and nothing grows
|
||||||
|
// without bound.
|
||||||
|
return !s.isClosed()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Stream) isClosed() bool {
|
||||||
|
s.mu.Lock()
|
||||||
|
defer s.mu.Unlock()
|
||||||
|
return s.closed
|
||||||
|
}
|
||||||
|
|
||||||
// writeLoop is the only writer to the destination. It drains the receive queue,
|
// writeLoop is the only writer to the destination. It drains the receive queue,
|
||||||
// credits the hub as bytes land on the destination socket, and performs the
|
// credits the hub as bytes land on the destination socket, and performs the
|
||||||
// deferred graceful close when a FIN arrived with data still queued.
|
// deferred graceful close when a FIN arrived with data still queued.
|
||||||
@@ -325,39 +644,76 @@ func (s *Stream) writeLoop() {
|
|||||||
s.teardown(false)
|
s.teardown(false)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
data := s.q[0]
|
e := s.q[0]
|
||||||
s.q = s.q[1:]
|
s.q = s.q[1:]
|
||||||
s.qBytes -= len(data)
|
if e.fromHub {
|
||||||
|
s.qBytes -= len(e.data)
|
||||||
|
}
|
||||||
dest := s.dest
|
dest := s.dest
|
||||||
s.mu.Unlock()
|
s.mu.Unlock()
|
||||||
|
|
||||||
if _, err := dest.Write(data); err != nil {
|
if _, err := dest.Write(e.data); err != nil {
|
||||||
s.teardown(true)
|
s.teardown(true)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
s.credit(len(data))
|
if e.fromHub {
|
||||||
|
s.mu.Lock()
|
||||||
|
s.deliveredOffset += int64(len(e.data))
|
||||||
|
if s.stats != nil {
|
||||||
|
s.stats.bytesDown += int64(len(e.data))
|
||||||
|
}
|
||||||
|
s.mu.Unlock()
|
||||||
|
s.credit(len(e.data))
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// deliverFromHub enqueues hub bytes for the destination. Called from the worker
|
// deliverFromHub enqueues hub bytes for the destination. Called from the worker
|
||||||
// readLoop; it never blocks — a peer that exceeds the advertised window is a
|
// readLoop; it never blocks — a peer that exceeds the advertised window is a
|
||||||
// protocol violator and gets the stream reset.
|
// protocol violator and gets the tunnel reset.
|
||||||
func (s *Stream) deliverFromHub(data []byte) {
|
func (s *Stream) deliverFromHub(data []byte) {
|
||||||
|
if s.vel != nil {
|
||||||
|
s.vel.ObserveC2S(data) // observation only; bytes still forwarded verbatim
|
||||||
|
}
|
||||||
s.mu.Lock()
|
s.mu.Lock()
|
||||||
if s.closed || s.finPending {
|
if s.closed || s.finPending {
|
||||||
s.mu.Unlock()
|
s.mu.Unlock()
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if s.qBytes+len(data) > s.wc.recvWndInit {
|
if s.qBytes+len(data) > s.client.streamWnd {
|
||||||
s.mu.Unlock()
|
s.mu.Unlock()
|
||||||
log.Printf("stream %d: peer exceeded flow-control window; resetting", s.sid)
|
log.Printf("stream %s: peer exceeded flow-control window; resetting", s.name())
|
||||||
s.wc.removeStream(s.sid)
|
if wc := s.conn(); wc != nil {
|
||||||
s.wc.sendRst(s.sid)
|
wc.detach()
|
||||||
|
wc.sendRst()
|
||||||
|
}
|
||||||
s.teardown(false)
|
s.teardown(false)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
s.q = append(s.q, data)
|
s.q = append(s.q, qentry{data: data, fromHub: true})
|
||||||
s.qBytes += len(data)
|
s.qBytes += len(data)
|
||||||
|
// Accepted, not delivered: from here the bytes are ours to write, and the
|
||||||
|
// only way we fail to is by destroying the stream — which also ends any
|
||||||
|
// prospect of resuming it. That makes this a sound reattach coordinate.
|
||||||
|
s.acceptedOffset += int64(len(data))
|
||||||
|
if s.stats != nil && s.qBytes > s.stats.qPeak {
|
||||||
|
// How close the receive queue came to the advertised window: near it
|
||||||
|
// means the destination is the slow party.
|
||||||
|
s.stats.qPeak = s.qBytes
|
||||||
|
}
|
||||||
|
s.cond.Broadcast()
|
||||||
|
s.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
// injectToDest queues locally generated bytes (the velocity login response)
|
||||||
|
// for the destination, outside flow-control accounting.
|
||||||
|
func (s *Stream) injectToDest(data []byte) {
|
||||||
|
s.mu.Lock()
|
||||||
|
if s.closed || s.finPending {
|
||||||
|
s.mu.Unlock()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.q = append(s.q, qentry{data: data})
|
||||||
s.cond.Broadcast()
|
s.cond.Broadcast()
|
||||||
s.mu.Unlock()
|
s.mu.Unlock()
|
||||||
}
|
}
|
||||||
@@ -367,9 +723,18 @@ func (s *Stream) deliverFromHub(data []byte) {
|
|||||||
func (s *Stream) acquireSendWnd(n int) bool {
|
func (s *Stream) acquireSendWnd(n int) bool {
|
||||||
s.mu.Lock()
|
s.mu.Lock()
|
||||||
defer s.mu.Unlock()
|
defer s.mu.Unlock()
|
||||||
|
// Timed only when it actually blocks, so a stream that never runs out of
|
||||||
|
// credit never reads the clock. A large windowStall is the signal that the
|
||||||
|
// peer is not draining to its terminal socket — the bottleneck is past the
|
||||||
|
// tunnel, not in it.
|
||||||
|
var stall stallClock
|
||||||
for !s.closed && s.sendWnd < n {
|
for !s.closed && s.sendWnd < n {
|
||||||
|
stall.begin(s.stats != nil)
|
||||||
s.cond.Wait()
|
s.cond.Wait()
|
||||||
}
|
}
|
||||||
|
if s.stats != nil {
|
||||||
|
s.stats.windowStall += stall.elapsed()
|
||||||
|
}
|
||||||
if s.closed {
|
if s.closed {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
@@ -378,6 +743,11 @@ func (s *Stream) acquireSendWnd(n int) bool {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (s *Stream) grantSendWnd(delta int) {
|
func (s *Stream) grantSendWnd(delta int) {
|
||||||
|
// The running sum doubles as the acked offset: the hub grants credit exactly
|
||||||
|
// as bytes reach the player socket, so a byte that has been credited can
|
||||||
|
// never need retransmitting. Advanced without a lock so the worker readLoop
|
||||||
|
// never blocks behind a send in progress.
|
||||||
|
s.ackedOffset.Add(int64(delta))
|
||||||
s.mu.Lock()
|
s.mu.Lock()
|
||||||
s.sendWnd += delta
|
s.sendWnd += delta
|
||||||
s.cond.Broadcast()
|
s.cond.Broadcast()
|
||||||
@@ -386,17 +756,23 @@ func (s *Stream) grantSendWnd(delta int) {
|
|||||||
|
|
||||||
// credit accounts bytes drained to the destination and grants the hub more
|
// credit accounts bytes drained to the destination and grants the hub more
|
||||||
// window once half of our receive window has been consumed.
|
// window once half of our receive window has been consumed.
|
||||||
|
// While parked the grant is only withheld, never dropped: consumed keeps
|
||||||
|
// accumulating and a reattach flushes it on the new conn. Resetting it would
|
||||||
|
// destroy up to half a window of credit per outage, and after a few flaps the
|
||||||
|
// stream would throttle to a crawl.
|
||||||
func (s *Stream) credit(n int) {
|
func (s *Stream) credit(n int) {
|
||||||
s.mu.Lock()
|
s.mu.Lock()
|
||||||
s.consumed += n
|
s.consumed += n
|
||||||
if s.closed || s.consumed*2 < s.wc.recvWndInit {
|
if s.closed || s.parked || s.consumed*2 < s.client.streamWnd {
|
||||||
s.mu.Unlock()
|
s.mu.Unlock()
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
delta := s.consumed
|
delta := s.consumed
|
||||||
s.consumed = 0
|
s.consumed = 0
|
||||||
s.mu.Unlock()
|
s.mu.Unlock()
|
||||||
s.wc.sendWndUpdate(s.sid, delta)
|
if wc := s.conn(); wc != nil {
|
||||||
|
wc.sendWndUpdate(delta)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *Stream) buildProxyHeader(dest net.Conn) []byte {
|
func (s *Stream) buildProxyHeader(dest net.Conn) []byte {
|
||||||
@@ -434,6 +810,13 @@ func (s *Stream) gracefulFin() {
|
|||||||
// teardown closes the stream immediately; notifyHub sends a FIN when true.
|
// teardown closes the stream immediately; notifyHub sends a FIN when true.
|
||||||
// Idempotent; wakes every goroutine parked on the stream.
|
// Idempotent; wakes every goroutine parked on the stream.
|
||||||
func (s *Stream) teardown(notifyHub bool) {
|
func (s *Stream) teardown(notifyHub bool) {
|
||||||
|
// A parked stream owes the hub a FIN it cannot send: the only conn it has is
|
||||||
|
// the one that just failed. Keep it alive so the reattach can deliver it and
|
||||||
|
// the player gets a clean disconnect, rather than hanging until the hub's
|
||||||
|
// grace expires.
|
||||||
|
if notifyHub && s.noteFinWhileParked() {
|
||||||
|
return
|
||||||
|
}
|
||||||
s.mu.Lock()
|
s.mu.Lock()
|
||||||
if s.closed {
|
if s.closed {
|
||||||
s.mu.Unlock()
|
s.mu.Unlock()
|
||||||
@@ -441,14 +824,24 @@ func (s *Stream) teardown(notifyHub bool) {
|
|||||||
}
|
}
|
||||||
s.closed = true
|
s.closed = true
|
||||||
dest := s.dest
|
dest := s.dest
|
||||||
|
close(s.done) // guarded by the idempotence check above, so exactly once
|
||||||
s.cond.Broadcast()
|
s.cond.Broadcast()
|
||||||
s.mu.Unlock()
|
s.mu.Unlock()
|
||||||
|
|
||||||
|
// The stream can no longer be resumed, so the retransmit buffer is dead
|
||||||
|
// weight — up to a full window of it per stream.
|
||||||
|
s.sendMu.Lock()
|
||||||
|
s.un.reset()
|
||||||
|
s.sendMu.Unlock()
|
||||||
|
|
||||||
if dest != nil {
|
if dest != nil {
|
||||||
_ = dest.Close()
|
_ = dest.Close()
|
||||||
}
|
}
|
||||||
s.wc.removeStream(s.sid)
|
if wc := s.conn(); wc != nil {
|
||||||
if notifyHub {
|
wc.detach()
|
||||||
s.wc.sendFin(s.sid)
|
if notifyHub {
|
||||||
|
wc.sendFin()
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
s.logSummary()
|
||||||
}
|
}
|
||||||
|
|||||||
+121
-65
@@ -15,9 +15,9 @@ on the wire, read [PROTOCOL.md](../PROTOCOL.md).
|
|||||||
┌──────────┐ Intent 17, magic 0x01 │ │ │
|
┌──────────┐ Intent 17, magic 0x01 │ │ │
|
||||||
│ Client │ ◀──────── control session ──────│ • pattern reg │ │
|
│ Client │ ◀──────── control session ──────│ • pattern reg │ │
|
||||||
│ (Go) │ ────────────────────────────────│ • CID table │ │
|
│ (Go) │ ────────────────────────────────│ • CID table │ │
|
||||||
│ │ Intent 17, magic 0x02 │ • mux demux │ │
|
│ │ Intent 17, magic 0x02 │ • 1:1 workers │ │
|
||||||
│ │ ═════════ worker conns ═════════│ │ │
|
│ │ ═════════ worker conns ═════════│ │ │
|
||||||
└──────────┘ multiplexed player streams └───────────────┘ │
|
└──────────┘ one TCP conn per player └───────────────┘ │
|
||||||
│ │
|
│ │
|
||||||
▼ MC bytes (+ optional HAProxy v2) │
|
▼ MC bytes (+ optional HAProxy v2) │
|
||||||
┌───────────────┐ │
|
┌───────────────┐ │
|
||||||
@@ -68,15 +68,15 @@ Player Hub Client Destinatio
|
|||||||
│ │ pause player socket,
|
│ │ pause player socket,
|
||||||
│ │ buffer bytes, mint CID
|
│ │ buffer bytes, mint CID
|
||||||
│ │─ ControlRequest(CID, pattern, ip:port) ─▶
|
│ │─ ControlRequest(CID, pattern, ip:port) ─▶
|
||||||
│ │ allocate worker+stream
|
│ │ dial dedicated worker
|
||||||
│ │◀──────── SYN(streamId, CID) ────────────│
|
│ │◀──────────── SYN(CID) ──────────────────│
|
||||||
│ │ takePending(CID) → bind dial destination,
|
│ │ takePending(CID) → bind dial destination,
|
||||||
│ │ forward buffered bytes write HAProxy v2 hdr
|
│ │ forward buffered bytes write HAProxy v2 hdr
|
||||||
│ │─ DATA(streamId, handshake…) ───▶ ── handshake ──▶│
|
│ │─ DATA(handshake…) ─────────────▶ ── handshake ──▶│
|
||||||
│ resume ─────────────────────────────│ bridge stream ⇄ dest
|
│ resume ─────────────────────────────│ bridge conn ⇄ dest
|
||||||
│══════════════ player bytes ══ DATA ══▶│════ DATA ═══▶ dest.write │
|
│══════════════ player bytes ══ DATA ══▶│════ DATA ═══▶ dest.write │
|
||||||
│◀═══════════ dest bytes ═══ DATA ══════│◀═══ DATA ════ dest.read │
|
│◀═══════════ dest bytes ═══ DATA ══════│◀═══ DATA ════ dest.read │
|
||||||
│ player closes ──────────────────────│─ FIN(streamId) ────────▶ close dest │
|
│ player closes ──────────────────────│─ FIN ──────────────────▶ close dest │
|
||||||
```
|
```
|
||||||
|
|
||||||
Key points:
|
Key points:
|
||||||
@@ -87,6 +87,11 @@ Key points:
|
|||||||
player's hostname — in `ControlRequest`, so the client can look it straight up
|
player's hostname — in `ControlRequest`, so the client can look it straight up
|
||||||
in its own route table. Invalid patterns are rejected at registration with a
|
in its own route table. Invalid patterns are rejected at registration with a
|
||||||
non-zero `RegisterAck` status.
|
non-zero `RegisterAck` status.
|
||||||
|
* A **per-IP limiter** runs first: a token bucket (`playerRatePerSec` /
|
||||||
|
`playerBurst`) and a concurrent-socket cap (`maxPlayersPerIp`). It applies
|
||||||
|
only to player intents — never Intent 17 — and unmatched hostnames still
|
||||||
|
consume a token, so a hostname scan is not a free flood. A refusal closes
|
||||||
|
the socket before CID minting or pause.
|
||||||
* The hub **pauses** the player socket the instant it matches, so no player
|
* The hub **pauses** the player socket the instant it matches, so no player
|
||||||
bytes are lost while the takeover is arranged; the buffered handshake is
|
bytes are lost while the takeover is arranged; the buffered handshake is
|
||||||
forwarded **verbatim**, so the real server sees exactly what the player sent
|
forwarded **verbatim**, so the real server sees exactly what the player sent
|
||||||
@@ -94,41 +99,38 @@ Key points:
|
|||||||
* **CID** is 16 random bytes minted by the hub and delivered only over the
|
* **CID** is 16 random bytes minted by the hub and delivered only over the
|
||||||
encrypted control session, so only the intended client learns it. Any worker
|
encrypted control session, so only the intended client learns it. Any worker
|
||||||
connection presenting the correct CID is allowed to take over — that secrecy
|
connection presenting the correct CID is allowed to take over — that secrecy
|
||||||
is what binds a worker stream to the right pending player without any explicit
|
is what binds a worker conn to the right pending player without any explicit
|
||||||
client identity.
|
client identity.
|
||||||
* Disconnects are symmetric: player-close → hub sends `FIN` → client closes the
|
* Disconnects are symmetric: player-close → hub sends `FIN` → client closes the
|
||||||
destination; destination-close → client sends `FIN` → hub closes the player.
|
destination; destination-close → client sends `FIN` → hub closes the player.
|
||||||
|
|
||||||
## 3. Multiplexing (worker connections)
|
## 3. Worker connections (1:1)
|
||||||
|
|
||||||
A worker connection is one encrypted TCP link carrying many **streams**. The
|
A worker connection is one encrypted TCP link carrying **exactly one player**.
|
||||||
frame is intentionally tiny (PROTOCOL.md §7):
|
The frame is intentionally tiny (PROTOCOL.md §7):
|
||||||
|
|
||||||
```
|
```
|
||||||
[plaintext VarInt length][ FrameType u8 | StreamID VarInt | Data… ] (payload encrypted)
|
[plaintext VarInt length][ FrameType u8 | Data… ] (payload encrypted)
|
||||||
```
|
```
|
||||||
|
|
||||||
Only the client opens streams (`SYN`), so stream-id allocation is a simple
|
There is no stream id. The client dials a fresh worker for each
|
||||||
per-connection counter with no coordination.
|
`ControlRequest` and binds it with `SYN(CID)` after `SessionReady`. A second
|
||||||
|
bind on the same conn is a protocol violation.
|
||||||
|
|
||||||
### 3.1 Pool & allocation
|
### 3.1 Allocation
|
||||||
|
|
||||||
The client keeps 1…`maxConn` worker connections and places each new stream on
|
The client holds at most `maxTunnels` live worker connections (default 256). A
|
||||||
the **least-loaded** one. It opens an additional connection only when the
|
`ControlRequest` either gets its own TCP connection or is dropped — the hub
|
||||||
least-loaded connection is *saturated* (more than 8 active streams) and the pool
|
then closes the player when `pendingTimeoutMs` fires.
|
||||||
is below `maxConn`:
|
|
||||||
|
|
||||||
```
|
A dial is **never performed while holding the live-set lock** — session
|
||||||
pick least-loaded conn
|
establishment is network I/O, and one unresponsive hub must not park every
|
||||||
if leastLoaded.streams > 8 and pool.size < maxConn:
|
other player behind it. Each caller dials independently; callers are not
|
||||||
dial a new worker conn and use it
|
serialized behind a single in-flight handshake.
|
||||||
else:
|
|
||||||
use leastLoaded
|
|
||||||
```
|
|
||||||
|
|
||||||
The e2e test `TestConcurrentStreamsUseMultipleConns` drives 20 simultaneous
|
The e2e test `TestEachPlayerGetsOwnWorker` drives concurrent players and
|
||||||
streams with `maxConn=4` and observes them deterministically spread over 3
|
confirms one worker conn each, without exceeding `maxTunnels`;
|
||||||
connections (9 + 9 + 2), confirming the algorithm.
|
`TestDialDoesNotWedgeOnStalledHub` covers the stalled-dial path.
|
||||||
|
|
||||||
## 4. Encryption
|
## 4. Encryption
|
||||||
|
|
||||||
@@ -153,68 +155,122 @@ connections (9 + 9 + 2), confirming the algorithm.
|
|||||||
non-blocking; crypto is CPU-cheap. This trades multi-core scaling for
|
non-blocking; crypto is CPU-cheap. This trades multi-core scaling for
|
||||||
simplicity and correctness.
|
simplicity and correctness.
|
||||||
* **Client:** goroutine-per-concern. One goroutine reads each connection
|
* **Client:** goroutine-per-concern. One goroutine reads each connection
|
||||||
(control or worker); `WriteFrame` is mutex-serialized so many stream goroutines
|
(control or worker); `WriteFrame` is mutex-serialized. Each tunnel has two
|
||||||
can share a worker connection safely. Each stream has two goroutines: `run`
|
goroutines: `run` pumps destination → hub, and `writeLoop` is the only writer
|
||||||
pumps destination → hub, and `writeLoop` is the only writer to the
|
to the destination, draining a queue fed by the worker readLoop. The
|
||||||
destination, draining a per-stream queue fed by the worker readLoop. The
|
readLoop itself never writes to a destination, so a stalled destination cannot
|
||||||
readLoop itself never writes to a destination, so a stalled destination can
|
stall WND / FIN / heartbeat dispatch on that conn.
|
||||||
never block frame dispatch for other streams.
|
|
||||||
|
|
||||||
## 6. Back-pressure & flow control
|
## 6. Back-pressure & flow control
|
||||||
|
|
||||||
Two mechanisms operate at different granularities:
|
Three mechanisms operate at different granularities:
|
||||||
|
|
||||||
* **Per-stream credit windows** (PROTOCOL.md §7.3; the windows are exchanged
|
* **Per-connection credit windows** (PROTOCOL.md §7.3; the windows are exchanged
|
||||||
at session establishment): each stream direction has an independent byte
|
at session establishment): each tunnel direction has an independent byte
|
||||||
budget equal to the receiver's advertised window (default 256 KiB). A sender
|
budget equal to the receiver's advertised window (default 256 KiB). A sender
|
||||||
that exhausts a
|
that exhausts a window pauses *only that player's source* — the hub pauses the
|
||||||
stream's window pauses *only that stream's source* — the hub pauses the one
|
one player socket, the client parks the one destination-reader goroutine.
|
||||||
player socket, the client parks the one destination-reader goroutine. Credit
|
Credit is granted back (`WND` frames, batched at half-window) as bytes are
|
||||||
is granted back (`WND` frames, batched at half-window) as bytes are actually
|
actually written to the terminal socket. The result: a slow player or slow
|
||||||
written to the terminal socket. The result: a slow player or slow destination
|
destination jams its own tunnel at a bounded buffer size and nothing else.
|
||||||
jams its own stream at a bounded buffer size and nothing else. This is what
|
* **TCP back-pressure** on each worker connection: when that socket is
|
||||||
eliminates head-of-line blocking between streams.
|
congested, the hub parks that one player until it drains, and the client's
|
||||||
* **Aggregate TCP back-pressure** on each worker connection: when the shared
|
`WriteFrame` blocks. Because the conn is 1:1, TCP HOL cannot stall another
|
||||||
socket itself is congested (total bandwidth, not one stream), the hub parks
|
player.
|
||||||
all sending players until it drains, and the client's `WriteFrame` blocks.
|
* **Client egress shaping** (optional, `maxBandwidth`; `client/shaper.go`): a
|
||||||
This is fair — when the pipe is genuinely full, everyone should slow down.
|
rate cap on everything the client sends to the hub, across all worker conns.
|
||||||
|
|
||||||
The window also bounds memory: a stream can hold at most one window of
|
The first two mechanisms have no time dimension. A credit window bounds how many
|
||||||
|
bytes are *in flight*, and TCP back-pressure only reacts once the pipe is already
|
||||||
|
full — which on a residential uplink is too late. One player loading chunks fills
|
||||||
|
the line, the standing queue grows to seconds, and every other player's keepalive
|
||||||
|
times out. Nothing in §7.3 prevents that: each tunnel is individually
|
||||||
|
well-behaved, and collectively they still overrun the link.
|
||||||
|
|
||||||
|
The shaper closes that gap with a token bucket for the rate and start-time fair
|
||||||
|
queueing for the split. A global virtual clock advances with each grant; every
|
||||||
|
tunnel remembers where its last request finished, and a new request is stamped
|
||||||
|
`max(tunnel.vfinish, vclock)`. Lowest stamp wins. A tunnel that keeps sending
|
||||||
|
pushes its own stamp further out and yields; a tunnel returning from idle is
|
||||||
|
clamped back to the clock, so it cannot bank credit for time it did not use, but
|
||||||
|
is not penalised for the idleness either. A tunnel sending a few hundred bytes
|
||||||
|
gets a nearer stamp than one sending a full chunk, so keepalives and chat overtake
|
||||||
|
bulk terrain data for free. One tunnel alone still gets the entire rate.
|
||||||
|
|
||||||
|
Two details keep bursts cheap. The bucket banks 200 ms of transmission, so a
|
||||||
|
player joining spends it at once instead of paying for the cap in visible
|
||||||
|
chunk-loading latency. And the DATA chunk shrinks to ~20 ms of transmission when
|
||||||
|
the rate is low (floor 4 KiB), because a fixed 32 KiB chunk is a 256 ms slot at
|
||||||
|
1 Mbps — long enough dead air to drag the other players towards the very timeout
|
||||||
|
the cap exists to prevent. Above ~13 Mbps the chunk stays at the usual 32 KiB.
|
||||||
|
|
||||||
|
This is entirely client-local: nothing about it appears on the wire, and the hub
|
||||||
|
is unaware. Only DATA is shaped — delaying a `FIN`, `WND` or `PONG` would cause
|
||||||
|
the false-death detection §7.4 exists to avoid.
|
||||||
|
|
||||||
|
The window also bounds memory: a tunnel can hold at most one window of
|
||||||
undelivered data per direction (the client's pre-connect handshake buffer is
|
undelivered data per direction (the client's pre-connect handshake buffer is
|
||||||
covered by the same bound).
|
covered by the same bound). With stream resumption enabled (§7) the *sender*
|
||||||
|
holds a second window — the bytes it has sent but the peer has not yet credited,
|
||||||
|
kept so they can be retransmitted after an outage. That is not a new bound so
|
||||||
|
much as the existing one made symmetric: the region is exactly what flow control
|
||||||
|
already declared outstanding, which is why resumption needs no cap of its own.
|
||||||
|
|
||||||
Per-stream flow control is mandatory: the hub rejects a session whose Rekey
|
Per-connection flow control is mandatory: the hub rejects a session whose Rekey
|
||||||
lacks the STREAM_FC flag, and the client rejects a hub that does not echo it —
|
lacks the STREAM_FC flag, and the client rejects a hub that does not echo it —
|
||||||
peers that predate the mechanism cannot connect at all.
|
peers that predate the mechanism cannot connect at all.
|
||||||
|
|
||||||
What remains (by design) is TCP-level head-of-line blocking: a lost packet on
|
TCP-level head-of-line blocking is now confined to one player: a lost packet
|
||||||
a worker connection stalls all its streams for one retransmit. That is inherent
|
stalls only that player's tunnel for one retransmit. The cost is one handshake
|
||||||
to mux-over-TCP; the connection pool is the mitigation, and a datagram
|
and one NAT mapping per player instead of per pool slot.
|
||||||
transport (QUIC) would be the escape hatch if it ever matters.
|
|
||||||
|
|
||||||
## 7. Failure & recovery
|
## 7. Failure & recovery
|
||||||
|
|
||||||
* **Control session drop:** the client reconnects with capped exponential
|
* **Control session drop:** the client retries immediately, then backs off to a
|
||||||
backoff and re-registers all patterns. Existing worker connections and their
|
10s cap, and re-registers all patterns. Existing worker connections and their
|
||||||
live streams are unaffected.
|
players are unaffected — they ride worker conns, which a control-session
|
||||||
* **Worker connection drop:** every stream on it is torn down (destinations
|
close never touches. The hub meanwhile keeps that session's routes as
|
||||||
closed); the hub closes the corresponding player sockets; the client removes
|
*orphaned* for `registrationGraceMs` (PROTOCOL.md §5.2) and **holds** players
|
||||||
the connection from the pool and will dial a fresh one on the next allocation.
|
arriving on them instead of refusing them, replaying the control request once
|
||||||
|
a client re-registers the pattern. Without that, the reconnect window is one
|
||||||
|
in which every new player is told there is no such server.
|
||||||
|
* **Worker connection drop:** the connection leaves the live set either way. What
|
||||||
|
happens to its player depends on whether STREAM_RESUME was negotiated:
|
||||||
|
* *without it* — the tunnel is torn down (destination closed) and the hub
|
||||||
|
closes the player socket, as it always did;
|
||||||
|
* *with it* — the player is **hung** instead (PROTOCOL.md §7.5). The
|
||||||
|
destination socket stays open, the hub pauses the player socket and holds
|
||||||
|
it for its grace period, and the client reattaches over a freshly dialed
|
||||||
|
connection, replaying byte-exactly from the offset the peer reports. The
|
||||||
|
player sees a stall rather than a disconnect.
|
||||||
* **Pending timeout:** if no worker takes over a matched player within
|
* **Pending timeout:** if no worker takes over a matched player within
|
||||||
`pendingTimeoutMs`, the hub drops the pending entry and closes the player.
|
`pendingTimeoutMs`, the hub drops the pending entry and closes the player.
|
||||||
* **Bad PSK / bad timestamp / bad magic:** the hub closes the TCP connection;
|
* **Bad PSK / bad timestamp / bad magic:** the hub closes the TCP connection;
|
||||||
the client's session establishment fails fast.
|
the client's session establishment fails fast.
|
||||||
|
|
||||||
|
Resumption is worth the machinery because a worker connection is only the
|
||||||
|
*middle* leg of the player it carries. When it dies both terminal sockets are
|
||||||
|
usually still healthy, so the old behaviour discarded working connections
|
||||||
|
because a replaceable transport failed. It also has to be byte-exact rather than
|
||||||
|
best-effort: bytes handed to a dying socket are lost with no notification and the
|
||||||
|
frame cipher cannot be resynchronized, so an approximate reattach would splice
|
||||||
|
the tunneled protocol mid-packet, which is worse than a clean close.
|
||||||
|
|
||||||
|
Notably, resumption does not depend on the control session. A blip usually kills
|
||||||
|
both, and the reattach path needs only a worker connection, so recovery does not
|
||||||
|
wait on the control reconnect backoff.
|
||||||
|
|
||||||
## 8. Known limitations
|
## 8. Known limitations
|
||||||
|
|
||||||
1. No AEAD — payload integrity/authenticity is not cryptographically guaranteed.
|
1. No AEAD — payload integrity/authenticity is not cryptographically guaranteed.
|
||||||
2. TCP-level head-of-line blocking within a worker connection (lost packets;
|
2. One handshake and one NAT mapping per player (the cost of dropping mux).
|
||||||
see §6) — per-stream flow control removes the application-level variant only.
|
|
||||||
3. Single-event-loop hub (see §5) bounds throughput to one core.
|
3. Single-event-loop hub (see §5) bounds throughput to one core.
|
||||||
4. `Intent 18` is reserved but only stubbed (the hub logs and closes).
|
4. `Intent 18` is reserved for a plaintext status probe.
|
||||||
5. Pattern ownership is last-writer-wins; two clients registering the identical
|
5. Pattern ownership is last-writer-wins; two clients registering the identical
|
||||||
pattern string will silently reassign it. Overlapping-but-distinct regexes are
|
pattern string will silently reassign it. Overlapping-but-distinct regexes are
|
||||||
both kept, and when several match one hostname the winner is unspecified.
|
both kept, and when several match one hostname the winner is unspecified.
|
||||||
|
6. The player IP limiter matches exact addresses. A `/64` of IPv6 clients looks
|
||||||
|
like many independent IPs; aggregation is a later, isolated change.
|
||||||
|
|
||||||
These are deliberate scope choices for a connectivity-focused P2P tool, not
|
These are deliberate scope choices for a connectivity-focused P2P tool, not
|
||||||
oversights; each is a small, well-isolated change away from being hardened.
|
oversights; each is a small, well-isolated change away from being hardened.
|
||||||
|
|||||||
@@ -0,0 +1,147 @@
|
|||||||
|
package e2e
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/iceBear67/redapricot/client"
|
||||||
|
)
|
||||||
|
|
||||||
|
// startClientWithBandwidth is startClient with an egress cap, for the shaping
|
||||||
|
// tests. Only the client→hub direction is shaped, which in this harness is the
|
||||||
|
// leg carrying the mock destination's echo back to the player.
|
||||||
|
func startClientWithBandwidth(t *testing.T, hubAddr, psk string, maxTunnels int, bandwidth string, mappings []client.Mapping) *client.Client {
|
||||||
|
t.Helper()
|
||||||
|
cfg := &client.Config{
|
||||||
|
Server: hubAddr,
|
||||||
|
PSK: psk,
|
||||||
|
MaxTunnels: maxTunnels,
|
||||||
|
PingIntervalMs: 20000,
|
||||||
|
MaxBandwidth: bandwidth,
|
||||||
|
Mappings: mappings,
|
||||||
|
}
|
||||||
|
c := client.New(cfg)
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
t.Cleanup(func() {
|
||||||
|
cancel()
|
||||||
|
c.Close()
|
||||||
|
})
|
||||||
|
if err := c.Start(ctx); err != nil {
|
||||||
|
t.Fatalf("client start: %v", err)
|
||||||
|
}
|
||||||
|
time.Sleep(200 * time.Millisecond)
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
|
||||||
|
// drain reads a connection until it fails, so a greedy player keeps pulling
|
||||||
|
// bytes instead of stalling on its own receive window.
|
||||||
|
func drain(conn net.Conn) {
|
||||||
|
go func() { _, _ = io.Copy(io.Discard, conn) }()
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestBandwidthCapIsEnforced pushes a payload that cannot fit in the burst and
|
||||||
|
// asserts the transfer takes at least as long as the configured rate implies.
|
||||||
|
// This is the first timing assertion in the suite, so the bounds are wide: the
|
||||||
|
// unshaped path completes this in well under a second, making the lower bound
|
||||||
|
// an unambiguous signal rather than a tight measurement.
|
||||||
|
func TestBandwidthCapIsEnforced(t *testing.T) {
|
||||||
|
const psk = "e2e-bwcap"
|
||||||
|
port := freePort(t)
|
||||||
|
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
|
||||||
|
startHub(t, port, psk)
|
||||||
|
dest := newMockDest(t, modeEcho)
|
||||||
|
startClientWithBandwidth(t, hubAddr, psk, 1, "1MB/s", []client.Mapping{
|
||||||
|
{Pattern: "mc.local", Destination: dest.addr},
|
||||||
|
})
|
||||||
|
|
||||||
|
pc := dialPlayer(t, hubAddr, "mc.local")
|
||||||
|
defer pc.Close()
|
||||||
|
|
||||||
|
payload := make([]byte, 2*1024*1024)
|
||||||
|
for i := range payload {
|
||||||
|
payload[i] = byte(i*31 + 7)
|
||||||
|
}
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
writeErr := make(chan error, 1)
|
||||||
|
go func() {
|
||||||
|
_, err := pc.Write(payload)
|
||||||
|
writeErr <- err
|
||||||
|
}()
|
||||||
|
got := make([]byte, len(payload))
|
||||||
|
_ = pc.SetReadDeadline(time.Now().Add(60 * time.Second))
|
||||||
|
if _, err := io.ReadFull(pc, got); err != nil {
|
||||||
|
t.Fatalf("read echo: %v", err)
|
||||||
|
}
|
||||||
|
elapsed := time.Since(start)
|
||||||
|
if err := <-writeErr; err != nil {
|
||||||
|
t.Fatalf("write payload: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// 2 MiB at 1 MiB/s, minus the 200 KiB the bucket has banked, is ~1.8 s.
|
||||||
|
const floor = 1200 * time.Millisecond
|
||||||
|
if elapsed < floor {
|
||||||
|
t.Errorf("2 MiB echoed back in %v under a 1MB/s cap; expected at least %v, so the cap is not taking effect", elapsed, floor)
|
||||||
|
}
|
||||||
|
if elapsed > 30*time.Second {
|
||||||
|
t.Errorf("transfer took %v, far beyond the ~1.8s the rate implies", elapsed)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCappedBandwidthDoesNotStarveLightStreams is the point of the fair queue:
|
||||||
|
// with the link deliberately capped and three players saturating it, a fourth
|
||||||
|
// player exchanging small messages must keep round-tripping promptly. A plain
|
||||||
|
// FIFO token bucket would leave it queued behind the heavy players' backlog,
|
||||||
|
// which for a real Minecraft client means a keepalive timeout — the exact
|
||||||
|
// failure this feature exists to prevent.
|
||||||
|
func TestCappedBandwidthDoesNotStarveLightStreams(t *testing.T) {
|
||||||
|
const psk = "e2e-bwfair"
|
||||||
|
port := freePort(t)
|
||||||
|
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
|
||||||
|
startHub(t, port, psk)
|
||||||
|
dest := newMockDest(t, modeEcho)
|
||||||
|
// Each player has its own worker conn; the shaper is still the only thing
|
||||||
|
// splitting the shared uplink budget.
|
||||||
|
startClientWithBandwidth(t, hubAddr, psk, 4, "1MB/s", []client.Mapping{
|
||||||
|
{Pattern: "mc.local", Destination: dest.addr},
|
||||||
|
})
|
||||||
|
|
||||||
|
// Three greedy players: each writes megabytes and keeps draining the echo,
|
||||||
|
// so they compete for the cap for the whole test rather than parking on
|
||||||
|
// their own flow-control windows.
|
||||||
|
for i := 0; i < 3; i++ {
|
||||||
|
heavy := dialPlayer(t, hubAddr, "mc.local")
|
||||||
|
defer heavy.Close()
|
||||||
|
drain(heavy)
|
||||||
|
go func() { _, _ = heavy.Write(make([]byte, 8*1024*1024)) }()
|
||||||
|
}
|
||||||
|
time.Sleep(500 * time.Millisecond) // let them saturate the shaper
|
||||||
|
|
||||||
|
light := dialPlayer(t, hubAddr, "mc.local")
|
||||||
|
defer light.Close()
|
||||||
|
payload := make([]byte, 4*1024)
|
||||||
|
for i := range payload {
|
||||||
|
payload[i] = byte(i*13 + 5)
|
||||||
|
}
|
||||||
|
|
||||||
|
var worst time.Duration
|
||||||
|
for i := 0; i < 10; i++ {
|
||||||
|
start := time.Now()
|
||||||
|
playerEcho(t, light, payload)
|
||||||
|
if d := time.Since(start); d > worst {
|
||||||
|
worst = d
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fair sharing puts a 4 KiB round at roughly 4 KiB / (1 MiB/s ÷ 4) ≈ 16 ms of
|
||||||
|
// link time. The bound is two orders of magnitude looser so only genuine
|
||||||
|
// starvation trips it.
|
||||||
|
const limit = 3 * time.Second
|
||||||
|
if worst > limit {
|
||||||
|
t.Errorf("slowest small round-trip took %v (limit %v); heavy streams are crowding out the light one", worst, limit)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,144 @@
|
|||||||
|
package e2e
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/iceBear67/redapricot/client"
|
||||||
|
"github.com/iceBear67/redapricot/client/wire"
|
||||||
|
)
|
||||||
|
|
||||||
|
// A control session dying takes the client's routes with it. Players already
|
||||||
|
// tunneled are unaffected — they ride worker conns — but anyone *arriving*
|
||||||
|
// during the reconnect used to be told there is no such server, even though the
|
||||||
|
// tunnel was a second from being back.
|
||||||
|
//
|
||||||
|
// The hub now keeps those routes as orphaned for its registration grace and
|
||||||
|
// hangs arriving players on them, replaying the control request it never sent
|
||||||
|
// once a client re-registers the pattern.
|
||||||
|
|
||||||
|
// outageRelay starts a hub, an echoing destination, and a client reaching the
|
||||||
|
// hub only through a relay, so the tunnel can be cut without touching the
|
||||||
|
// players — who connect to the hub directly, as they would from the internet.
|
||||||
|
func outageRelay(t *testing.T, psk string, hubCfg map[string]any) (hubAddr string, relay *blackholeRelay) {
|
||||||
|
t.Helper()
|
||||||
|
hubPort := freePort(t)
|
||||||
|
hubAddr = fmt.Sprintf("127.0.0.1:%d", hubPort)
|
||||||
|
startHubCfg(t, hubPort, psk, hubCfg)
|
||||||
|
dest := newMockDest(t, modeEcho)
|
||||||
|
|
||||||
|
relay = newBlackholeRelay(t, hubAddr)
|
||||||
|
startClientWithPing(t, relay.addr, psk, 1, 400, []client.Mapping{
|
||||||
|
{Pattern: "mc.local", Destination: dest.addr},
|
||||||
|
})
|
||||||
|
return hubAddr, relay
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestControlOutageHangsArrivingPlayer is the point of the feature: a player
|
||||||
|
// that shows up while the client is reconnecting gets held and then served,
|
||||||
|
// rather than refused.
|
||||||
|
func TestControlOutageHangsArrivingPlayer(t *testing.T) {
|
||||||
|
const psk = "e2e-ctl-hang"
|
||||||
|
hubAddr, relay := outageRelay(t, psk, nil)
|
||||||
|
|
||||||
|
// Cut the tunnel and keep it cut, so the client cannot re-register.
|
||||||
|
relay.stop()
|
||||||
|
relay.dropAll()
|
||||||
|
// Let the hub see the close and orphan the route before the player arrives.
|
||||||
|
time.Sleep(500 * time.Millisecond)
|
||||||
|
|
||||||
|
pc := resumePlayer(t, hubAddr, "mc.local")
|
||||||
|
if _, err := pc.Write([]byte("held")); err != nil {
|
||||||
|
t.Fatalf("player write during outage: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Nothing can come back yet — but the socket must still be open. Before this
|
||||||
|
// change the hub had already closed it.
|
||||||
|
_ = pc.SetReadDeadline(time.Now().Add(700 * time.Millisecond))
|
||||||
|
if _, err := pc.Read(make([]byte, 1)); err == nil {
|
||||||
|
t.Fatal("player was served while the route was orphaned")
|
||||||
|
} else if !isTimeout(err) {
|
||||||
|
t.Fatalf("player was dropped during the control outage instead of being held: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
relay.restore()
|
||||||
|
|
||||||
|
// The client reconnects, re-registers, and the hub replays the request it
|
||||||
|
// held — so the bytes written during the outage arrive at the destination and
|
||||||
|
// echo back on the same connection.
|
||||||
|
echo := make([]byte, 4)
|
||||||
|
_ = pc.SetReadDeadline(time.Now().Add(30 * time.Second))
|
||||||
|
if _, err := io.ReadFull(pc, echo); err != nil {
|
||||||
|
t.Fatalf("held player never served after the route came back: %v", err)
|
||||||
|
}
|
||||||
|
if string(echo) != "held" {
|
||||||
|
t.Fatalf("echo = %q, want %q", echo, "held")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestControlOutageGraceDisabledClosesPlayer pins the off switch: with the grace
|
||||||
|
// at zero the hub must drop the route the instant its session closes, exactly as
|
||||||
|
// it did before, rather than hanging players for a client that may never return.
|
||||||
|
func TestControlOutageGraceDisabledClosesPlayer(t *testing.T) {
|
||||||
|
const psk = "e2e-ctl-nohang"
|
||||||
|
hubAddr, relay := outageRelay(t, psk, map[string]any{"registrationGraceMs": 0})
|
||||||
|
|
||||||
|
relay.stop()
|
||||||
|
relay.dropAll()
|
||||||
|
time.Sleep(500 * time.Millisecond)
|
||||||
|
|
||||||
|
// No route at all now, so the hub closes the connection during the handshake.
|
||||||
|
pc, err := net.DialTimeout("tcp", hubAddr, 5*time.Second)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("player dial: %v", err)
|
||||||
|
}
|
||||||
|
defer pc.Close()
|
||||||
|
if _, err := pc.Write(playerHandshake("mc.local")); err != nil {
|
||||||
|
t.Fatalf("player handshake: %v", err)
|
||||||
|
}
|
||||||
|
_ = pc.SetReadDeadline(time.Now().Add(10 * time.Second))
|
||||||
|
if _, err := pc.Read(make([]byte, 1)); err == nil || isTimeout(err) {
|
||||||
|
t.Fatalf("player was held with the registration grace disabled (err=%v)", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestControlOutageHangExpiresClosesPlayer covers the other end: a route that is
|
||||||
|
// never reclaimed must not hold its players forever.
|
||||||
|
func TestControlOutageHangExpiresClosesPlayer(t *testing.T) {
|
||||||
|
const psk = "e2e-ctl-expire"
|
||||||
|
hubAddr, relay := outageRelay(t, psk, map[string]any{"registrationGraceMs": 2000})
|
||||||
|
|
||||||
|
relay.stop()
|
||||||
|
relay.dropAll()
|
||||||
|
time.Sleep(500 * time.Millisecond)
|
||||||
|
|
||||||
|
pc := resumePlayer(t, hubAddr, "mc.local")
|
||||||
|
start := time.Now()
|
||||||
|
_ = pc.SetReadDeadline(time.Now().Add(20 * time.Second))
|
||||||
|
if _, err := pc.Read(make([]byte, 1)); err == nil || isTimeout(err) {
|
||||||
|
t.Fatalf("held player was never released after the grace expired (err=%v)", err)
|
||||||
|
}
|
||||||
|
if elapsed := time.Since(start); elapsed < 500*time.Millisecond {
|
||||||
|
t.Fatalf("player closed after %s, so it was refused rather than held", elapsed)
|
||||||
|
} else {
|
||||||
|
t.Logf("held player released after %s", elapsed.Round(100*time.Millisecond))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// isTimeout distinguishes "still held, nothing to read yet" from "the hub closed
|
||||||
|
// us" — which is the whole distinction these tests turn on.
|
||||||
|
func isTimeout(err error) bool {
|
||||||
|
var ne net.Error
|
||||||
|
if errors.As(err, &ne) {
|
||||||
|
return ne.Timeout()
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func playerHandshake(host string) []byte {
|
||||||
|
return wire.BuildHandshake(767, host, 25565, 2)
|
||||||
|
}
|
||||||
+31
-20
@@ -13,15 +13,28 @@ import (
|
|||||||
|
|
||||||
// startClient builds and starts an in-process client against the hub, with the
|
// startClient builds and starts an in-process client against the hub, with the
|
||||||
// given mappings, returning the running client.
|
// given mappings, returning the running client.
|
||||||
func startClient(t *testing.T, hubAddr, psk string, maxConn int, mappings []client.Mapping) *client.Client {
|
func startClient(t *testing.T, hubAddr, psk string, maxTunnels int, mappings []client.Mapping) *client.Client {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
cfg := &client.Config{
|
return startClientWithPing(t, hubAddr, psk, maxTunnels, 20000, mappings)
|
||||||
|
}
|
||||||
|
|
||||||
|
// startClientWithPing is startClient with an explicit heartbeat interval, for
|
||||||
|
// tests that need liveness detection to trigger quickly.
|
||||||
|
func startClientWithPing(t *testing.T, hubAddr, psk string, maxTunnels, pingMs int, mappings []client.Mapping) *client.Client {
|
||||||
|
t.Helper()
|
||||||
|
return startClientCfg(t, &client.Config{
|
||||||
Server: hubAddr,
|
Server: hubAddr,
|
||||||
PSK: psk,
|
PSK: psk,
|
||||||
MaxConn: maxConn,
|
MaxTunnels: maxTunnels,
|
||||||
PingIntervalMs: 20000,
|
PingIntervalMs: pingMs,
|
||||||
Mappings: mappings,
|
Mappings: mappings,
|
||||||
}
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// startClientCfg runs an in-process client from a fully-specified config, for
|
||||||
|
// tests that need a knob the shorthand helpers do not expose.
|
||||||
|
func startClientCfg(t *testing.T, cfg *client.Config) *client.Client {
|
||||||
|
t.Helper()
|
||||||
c := client.New(cfg)
|
c := client.New(cfg)
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
t.Cleanup(func() {
|
t.Cleanup(func() {
|
||||||
@@ -94,7 +107,7 @@ func TestRegexPatternMatch(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestLargeTransfer pushes a multi-megabyte payload both ways to exercise mux
|
// TestLargeTransfer pushes a multi-megabyte payload both ways to exercise
|
||||||
// framing and back-pressure.
|
// framing and back-pressure.
|
||||||
func TestLargeTransfer(t *testing.T) {
|
func TestLargeTransfer(t *testing.T) {
|
||||||
const psk = "e2e-large"
|
const psk = "e2e-large"
|
||||||
@@ -134,17 +147,17 @@ func TestLargeTransfer(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestConcurrentStreamsUseMultipleConns confirms the least-loaded allocator
|
// TestEachPlayerGetsOwnWorker confirms the 1:1 rule: N concurrent players
|
||||||
// opens additional worker connections once streams saturate (>8).
|
// produce N worker connections, and the maxTunnels cap is honoured.
|
||||||
func TestConcurrentStreamsUseMultipleConns(t *testing.T) {
|
func TestEachPlayerGetsOwnWorker(t *testing.T) {
|
||||||
const psk = "e2e-concurrent"
|
const psk = "e2e-concurrent"
|
||||||
const n = 20
|
const n = 8
|
||||||
const maxConn = 4
|
const maxTunnels = 16
|
||||||
port := freePort(t)
|
port := freePort(t)
|
||||||
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
|
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
|
||||||
startHub(t, port, psk)
|
startHub(t, port, psk)
|
||||||
dest := newMockDest(t, modeEcho)
|
dest := newMockDest(t, modeEcho)
|
||||||
c := startClient(t, hubAddr, psk, maxConn, []client.Mapping{
|
c := startClient(t, hubAddr, psk, maxTunnels, []client.Mapping{
|
||||||
{Pattern: "mc.local", Destination: dest.addr},
|
{Pattern: "mc.local", Destination: dest.addr},
|
||||||
})
|
})
|
||||||
|
|
||||||
@@ -154,8 +167,6 @@ func TestConcurrentStreamsUseMultipleConns(t *testing.T) {
|
|||||||
_ = pc.Close()
|
_ = pc.Close()
|
||||||
}
|
}
|
||||||
}()
|
}()
|
||||||
// Establish streams sequentially so allocation is deterministic; keep them
|
|
||||||
// all open to hold streams active.
|
|
||||||
for i := 0; i < n; i++ {
|
for i := 0; i < n; i++ {
|
||||||
pc := dialPlayer(t, hubAddr, "mc.local")
|
pc := dialPlayer(t, hubAddr, "mc.local")
|
||||||
playerEcho(t, pc, []byte(fmt.Sprintf("hello-%d", i)))
|
playerEcho(t, pc, []byte(fmt.Sprintf("hello-%d", i)))
|
||||||
@@ -163,13 +174,13 @@ func TestConcurrentStreamsUseMultipleConns(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
got := c.WorkerConnCount()
|
got := c.WorkerConnCount()
|
||||||
if got < 2 {
|
if got != n {
|
||||||
t.Fatalf("expected >=2 worker conns for %d concurrent streams, got %d", n, got)
|
t.Fatalf("expected %d worker conns for %d players, got %d", n, n, got)
|
||||||
}
|
}
|
||||||
if got > maxConn {
|
if got > maxTunnels {
|
||||||
t.Fatalf("worker conns %d exceed maxConn %d", got, maxConn)
|
t.Fatalf("worker conns %d exceed maxTunnels %d", got, maxTunnels)
|
||||||
}
|
}
|
||||||
t.Logf("%d concurrent streams spread over %d worker conn(s)", n, got)
|
t.Logf("%d players on %d worker conn(s)", n, got)
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestProxyProtocol checks that the client prepends a correct HAProxy v2 header
|
// TestProxyProtocol checks that the client prepends a correct HAProxy v2 header
|
||||||
@@ -264,7 +275,7 @@ func TestBadPSK(t *testing.T) {
|
|||||||
cfg := &client.Config{
|
cfg := &client.Config{
|
||||||
Server: hubAddr,
|
Server: hubAddr,
|
||||||
PSK: "totally-wrong",
|
PSK: "totally-wrong",
|
||||||
MaxConn: 2,
|
MaxTunnels: 2,
|
||||||
PingIntervalMs: 20000,
|
PingIntervalMs: 20000,
|
||||||
Mappings: []client.Mapping{{Pattern: "mc.local", Destination: dest.addr}},
|
Mappings: []client.Mapping{{Pattern: "mc.local", Destination: dest.addr}},
|
||||||
}
|
}
|
||||||
|
|||||||
+22
-2
@@ -4,6 +4,7 @@ import (
|
|||||||
"bufio"
|
"bufio"
|
||||||
"bytes"
|
"bytes"
|
||||||
"encoding/binary"
|
"encoding/binary"
|
||||||
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"net"
|
"net"
|
||||||
@@ -61,14 +62,33 @@ func freePort(t *testing.T) int {
|
|||||||
// startHub launches the Java hub on the given port and blocks until it accepts
|
// startHub launches the Java hub on the given port and blocks until it accepts
|
||||||
// connections. The process is killed on test cleanup.
|
// connections. The process is killed on test cleanup.
|
||||||
func startHub(t *testing.T, port int, psk string) {
|
func startHub(t *testing.T, port int, psk string) {
|
||||||
|
t.Helper()
|
||||||
|
startHubCfg(t, port, psk, nil)
|
||||||
|
}
|
||||||
|
|
||||||
|
// startHubCfg is startHub with extra config keys merged over the defaults, for
|
||||||
|
// tests that need to tune a hub-side knob.
|
||||||
|
func startHubCfg(t *testing.T, port int, psk string, extra map[string]any) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
install := filepath.Join(repoRoot(), "server", "build", "install", "redapricot-server")
|
install := filepath.Join(repoRoot(), "server", "build", "install", "redapricot-server")
|
||||||
if _, err := os.Stat(install); err != nil {
|
if _, err := os.Stat(install); err != nil {
|
||||||
t.Fatalf("hub not built at %s (run scripts/build.sh first): %v", install, err)
|
t.Fatalf("hub not built at %s (run scripts/build.sh first): %v", install, err)
|
||||||
}
|
}
|
||||||
cfg := fmt.Sprintf(`{"listen":"127.0.0.1:%d","psk":%q,"timestampWindowMs":30000,"pendingTimeoutMs":5000}`, port, psk)
|
settings := map[string]any{
|
||||||
|
"listen": fmt.Sprintf("127.0.0.1:%d", port),
|
||||||
|
"psk": psk,
|
||||||
|
"timestampWindowMs": 30000,
|
||||||
|
"pendingTimeoutMs": 5000,
|
||||||
|
}
|
||||||
|
for k, v := range extra {
|
||||||
|
settings[k] = v
|
||||||
|
}
|
||||||
|
cfg, err := json.Marshal(settings)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
cfgPath := filepath.Join(t.TempDir(), "hub.json")
|
cfgPath := filepath.Join(t.TempDir(), "hub.json")
|
||||||
if err := os.WriteFile(cfgPath, []byte(cfg), 0o644); err != nil {
|
if err := os.WriteFile(cfgPath, cfg, 0o644); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,174 @@
|
|||||||
|
package e2e
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"net"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/iceBear67/redapricot/client"
|
||||||
|
)
|
||||||
|
|
||||||
|
// expectClosedSoon fails unless conn is closed (or reset) within d. A player
|
||||||
|
// that entered pending stays open until pendingTimeoutMs, so a fast EOF is
|
||||||
|
// how we tell the limiter dropped the socket before match-side bookkeeping.
|
||||||
|
func expectClosedSoon(t *testing.T, conn net.Conn, d time.Duration) {
|
||||||
|
t.Helper()
|
||||||
|
_ = conn.SetReadDeadline(time.Now().Add(d))
|
||||||
|
n, err := conn.Read(make([]byte, 16))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("expected the hub to close the player, read %d bytes", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestPlayerBurstDropsExtraHandshakes: more arrivals than playerBurst from the
|
||||||
|
// same IP are closed after the handshake and never become pending.
|
||||||
|
func TestPlayerBurstDropsExtraHandshakes(t *testing.T) {
|
||||||
|
const psk = "e2e-rate-burst"
|
||||||
|
port := freePort(t)
|
||||||
|
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
|
||||||
|
startHubCfg(t, port, psk, map[string]any{
|
||||||
|
"playerRatePerSec": 1,
|
||||||
|
"playerBurst": 2,
|
||||||
|
"maxPlayersPerIp": 64,
|
||||||
|
})
|
||||||
|
dest := newMockDest(t, modeEcho)
|
||||||
|
startClient(t, hubAddr, psk, 8, []client.Mapping{
|
||||||
|
{Pattern: "mc.local", Destination: dest.addr},
|
||||||
|
})
|
||||||
|
|
||||||
|
kept := make([]net.Conn, 0, 2)
|
||||||
|
defer func() {
|
||||||
|
for _, c := range kept {
|
||||||
|
_ = c.Close()
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
for i := 0; i < 2; i++ {
|
||||||
|
pc := dialPlayer(t, hubAddr, "mc.local")
|
||||||
|
playerEcho(t, pc, []byte(fmt.Sprintf("ok-%d", i)))
|
||||||
|
kept = append(kept, pc)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Tokens spent, first two still held: extras must die well inside pendingTimeout.
|
||||||
|
for i := 0; i < 2; i++ {
|
||||||
|
extra := dialPlayer(t, hubAddr, "mc.local")
|
||||||
|
expectClosedSoon(t, extra, 1500*time.Millisecond)
|
||||||
|
_ = extra.Close()
|
||||||
|
}
|
||||||
|
|
||||||
|
// The admitted players are unaffected.
|
||||||
|
playerEcho(t, kept[0], []byte("still-here"))
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestMaxPlayersPerIpCapsConcurrentSockets: a second player from the same IP
|
||||||
|
// is refused while the first is still open, and admitted again after it closes.
|
||||||
|
func TestMaxPlayersPerIpCapsConcurrentSockets(t *testing.T) {
|
||||||
|
const psk = "e2e-rate-conc"
|
||||||
|
port := freePort(t)
|
||||||
|
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
|
||||||
|
startHubCfg(t, port, psk, map[string]any{
|
||||||
|
"playerRatePerSec": 0,
|
||||||
|
"playerBurst": 16,
|
||||||
|
"maxPlayersPerIp": 1,
|
||||||
|
})
|
||||||
|
dest := newMockDest(t, modeEcho)
|
||||||
|
startClient(t, hubAddr, psk, 4, []client.Mapping{
|
||||||
|
{Pattern: "mc.local", Destination: dest.addr},
|
||||||
|
})
|
||||||
|
|
||||||
|
first := dialPlayer(t, hubAddr, "mc.local")
|
||||||
|
defer first.Close()
|
||||||
|
playerEcho(t, first, []byte("first"))
|
||||||
|
|
||||||
|
second := dialPlayer(t, hubAddr, "mc.local")
|
||||||
|
expectClosedSoon(t, second, 1500*time.Millisecond)
|
||||||
|
_ = second.Close()
|
||||||
|
|
||||||
|
playerEcho(t, first, []byte("still-first"))
|
||||||
|
_ = first.Close()
|
||||||
|
// The closeHandler runs on the hub event loop; give it a beat to release.
|
||||||
|
time.Sleep(200 * time.Millisecond)
|
||||||
|
|
||||||
|
third := dialPlayer(t, hubAddr, "mc.local")
|
||||||
|
defer third.Close()
|
||||||
|
playerEcho(t, third, []byte("after-release"))
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestUnmatchedHostConsumesRateBudget: a hostname scan is not a free flood.
|
||||||
|
// Two unmatched handshakes spend the burst, so a later matching player is
|
||||||
|
// also dropped.
|
||||||
|
func TestUnmatchedHostConsumesRateBudget(t *testing.T) {
|
||||||
|
const psk = "e2e-rate-scan"
|
||||||
|
port := freePort(t)
|
||||||
|
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
|
||||||
|
startHubCfg(t, port, psk, map[string]any{
|
||||||
|
"playerRatePerSec": 1,
|
||||||
|
"playerBurst": 2,
|
||||||
|
"maxPlayersPerIp": 64,
|
||||||
|
})
|
||||||
|
dest := newMockDest(t, modeEcho)
|
||||||
|
startClient(t, hubAddr, psk, 4, []client.Mapping{
|
||||||
|
{Pattern: "mc.local", Destination: dest.addr},
|
||||||
|
})
|
||||||
|
|
||||||
|
for i := 0; i < 2; i++ {
|
||||||
|
miss := dialPlayer(t, hubAddr, "no.such.host")
|
||||||
|
expectClosedSoon(t, miss, 1500*time.Millisecond)
|
||||||
|
_ = miss.Close()
|
||||||
|
}
|
||||||
|
|
||||||
|
matched := dialPlayer(t, hubAddr, "mc.local")
|
||||||
|
expectClosedSoon(t, matched, 1500*time.Millisecond)
|
||||||
|
_ = matched.Close()
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestIntent17IgnoresPlayerLimiter: the control session and worker conns are
|
||||||
|
// Intent 17, so a limiter tight enough to refuse a second player must not
|
||||||
|
// prevent the client from connecting or taking over the first player.
|
||||||
|
func TestIntent17IgnoresPlayerLimiter(t *testing.T) {
|
||||||
|
const psk = "e2e-rate-intent17"
|
||||||
|
port := freePort(t)
|
||||||
|
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
|
||||||
|
startHubCfg(t, port, psk, map[string]any{
|
||||||
|
"playerRatePerSec": 1,
|
||||||
|
"playerBurst": 1,
|
||||||
|
"maxPlayersPerIp": 1,
|
||||||
|
})
|
||||||
|
dest := newMockDest(t, modeEcho)
|
||||||
|
startClient(t, hubAddr, psk, 4, []client.Mapping{
|
||||||
|
{Pattern: "mc.local", Destination: dest.addr},
|
||||||
|
})
|
||||||
|
|
||||||
|
pc := dialPlayer(t, hubAddr, "mc.local")
|
||||||
|
defer pc.Close()
|
||||||
|
playerEcho(t, pc, []byte("intent17-ok"))
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestPlayerLimiterOffSwitch: both knobs at 0 restore phase-1 behaviour —
|
||||||
|
// many players from 127.0.0.1 all get through.
|
||||||
|
func TestPlayerLimiterOffSwitch(t *testing.T) {
|
||||||
|
const psk = "e2e-rate-off"
|
||||||
|
const n = 6
|
||||||
|
port := freePort(t)
|
||||||
|
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
|
||||||
|
startHubCfg(t, port, psk, map[string]any{
|
||||||
|
"playerRatePerSec": 0,
|
||||||
|
"maxPlayersPerIp": 0,
|
||||||
|
})
|
||||||
|
dest := newMockDest(t, modeEcho)
|
||||||
|
startClient(t, hubAddr, psk, n, []client.Mapping{
|
||||||
|
{Pattern: "mc.local", Destination: dest.addr},
|
||||||
|
})
|
||||||
|
|
||||||
|
conns := make([]net.Conn, 0, n)
|
||||||
|
defer func() {
|
||||||
|
for _, c := range conns {
|
||||||
|
_ = c.Close()
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
for i := 0; i < n; i++ {
|
||||||
|
pc := dialPlayer(t, hubAddr, "mc.local")
|
||||||
|
playerEcho(t, pc, []byte(fmt.Sprintf("off-%d", i)))
|
||||||
|
conns = append(conns, pc)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,187 @@
|
|||||||
|
package e2e
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net"
|
||||||
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/iceBear67/redapricot/client"
|
||||||
|
"github.com/iceBear67/redapricot/client/wire"
|
||||||
|
)
|
||||||
|
|
||||||
|
// blackholeRelay forwards TCP to the hub until it is switched off, after which
|
||||||
|
// already-established pairs silently stop carrying bytes while their sockets
|
||||||
|
// stay open. That is what a stateful middlebox looks like when it forgets a
|
||||||
|
// flow: conntrack expiry, a firewall reload, or a cloud LB idle timeout. No FIN
|
||||||
|
// and no RST ever reach either end, so nothing below the application layer can
|
||||||
|
// notice.
|
||||||
|
// Only flows that already existed when the switch is thrown go dark; new
|
||||||
|
// connections are carried normally, exactly as when a middlebox forgets
|
||||||
|
// established state but keeps forwarding fresh traffic.
|
||||||
|
type blackholeRelay struct {
|
||||||
|
addr string
|
||||||
|
backend string
|
||||||
|
gen atomic.Uint64
|
||||||
|
down atomic.Bool
|
||||||
|
mu sync.Mutex
|
||||||
|
held []net.Conn
|
||||||
|
}
|
||||||
|
|
||||||
|
func newBlackholeRelay(t *testing.T, backend string) *blackholeRelay {
|
||||||
|
t.Helper()
|
||||||
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("relay listen: %v", err)
|
||||||
|
}
|
||||||
|
r := &blackholeRelay{addr: ln.Addr().String(), backend: backend}
|
||||||
|
t.Cleanup(func() {
|
||||||
|
_ = ln.Close()
|
||||||
|
r.mu.Lock()
|
||||||
|
for _, c := range r.held {
|
||||||
|
_ = c.Close()
|
||||||
|
}
|
||||||
|
r.mu.Unlock()
|
||||||
|
})
|
||||||
|
go func() {
|
||||||
|
for {
|
||||||
|
cli, err := ln.Accept()
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
go r.handle(cli)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
return r
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *blackholeRelay) handle(cli net.Conn) {
|
||||||
|
born := r.gen.Load()
|
||||||
|
if r.down.Load() {
|
||||||
|
_ = cli.Close()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
up, err := net.Dial("tcp", r.backend)
|
||||||
|
if err != nil {
|
||||||
|
_ = cli.Close()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
r.mu.Lock()
|
||||||
|
r.held = append(r.held, cli, up)
|
||||||
|
r.mu.Unlock()
|
||||||
|
pipe := func(dst, src net.Conn) {
|
||||||
|
buf := make([]byte, 32*1024)
|
||||||
|
for {
|
||||||
|
n, err := src.Read(buf)
|
||||||
|
if n > 0 && r.gen.Load() == born {
|
||||||
|
if _, werr := dst.Write(buf[:n]); werr != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
go pipe(up, cli)
|
||||||
|
go pipe(cli, up)
|
||||||
|
}
|
||||||
|
|
||||||
|
// blackhole strands every currently-established pair. Later connections are
|
||||||
|
// unaffected.
|
||||||
|
func (r *blackholeRelay) blackhole() { r.gen.Add(1) }
|
||||||
|
|
||||||
|
// dropAll hard-resets every currently-established pair: a real FIN/RST reaches
|
||||||
|
// both ends immediately, as when a middlebox is restarted or a route flaps,
|
||||||
|
// rather than the silent stranding blackhole models. Later connections are
|
||||||
|
// carried normally.
|
||||||
|
//
|
||||||
|
// stop takes the relay out of service: new connections are refused rather than
|
||||||
|
// carried, so the tunnel stays down until restore is called. Models an outage
|
||||||
|
// the client cannot immediately reconnect through.
|
||||||
|
func (r *blackholeRelay) stop() { r.down.Store(true) }
|
||||||
|
|
||||||
|
// restore puts the relay back in service. Flows stranded before it are still
|
||||||
|
// dead — only fresh connections are carried, which is what a client gets after
|
||||||
|
// a middlebox or upstream link comes back.
|
||||||
|
func (r *blackholeRelay) restore() { r.down.Store(false) }
|
||||||
|
|
||||||
|
// This is the fast path into stream resumption: the client learns the conn is
|
||||||
|
// gone at once instead of waiting out a heartbeat timeout.
|
||||||
|
func (r *blackholeRelay) dropAll() {
|
||||||
|
r.gen.Add(1)
|
||||||
|
r.mu.Lock()
|
||||||
|
held := r.held
|
||||||
|
r.held = nil
|
||||||
|
r.mu.Unlock()
|
||||||
|
for _, c := range held {
|
||||||
|
_ = c.Close()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestBlackholedPathRecovers is the end-to-end regression guard for the
|
||||||
|
// stability bug this hardening was written for: with the tunnel's path silently
|
||||||
|
// dropped, the client used to notice nothing at all. Its read loops parked
|
||||||
|
// forever, the dead worker conn stayed in the pool, the hub kept routing players
|
||||||
|
// to a control session nobody was reading, and no player could connect again
|
||||||
|
// until the client process was restarted.
|
||||||
|
//
|
||||||
|
// The heartbeats must now detect the silence, drop both sessions, and let the
|
||||||
|
// existing reconnect path restore service on its own.
|
||||||
|
func TestBlackholedPathRecovers(t *testing.T) {
|
||||||
|
const psk = "e2e-blackhole"
|
||||||
|
hubPort := freePort(t)
|
||||||
|
hubAddr := fmt.Sprintf("127.0.0.1:%d", hubPort)
|
||||||
|
startHub(t, hubPort, psk)
|
||||||
|
dest := newMockDest(t, modeEcho)
|
||||||
|
|
||||||
|
// The client reaches the hub only through the relay; players connect to the
|
||||||
|
// hub directly, as they would from the internet.
|
||||||
|
relay := newBlackholeRelay(t, hubAddr)
|
||||||
|
const pingMs = 400 // heartbeat timeout is 3x this
|
||||||
|
c := startClientWithPing(t, relay.addr, psk, 4, pingMs, []client.Mapping{
|
||||||
|
{Pattern: "mc.local", Destination: dest.addr},
|
||||||
|
})
|
||||||
|
|
||||||
|
play := func(what string) error {
|
||||||
|
pc, err := net.DialTimeout("tcp", hubAddr, 5*time.Second)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer pc.Close()
|
||||||
|
if _, err := pc.Write(wire.BuildHandshake(767, "mc.local", 25565, 2)); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
msg := []byte(what)
|
||||||
|
if _, err := pc.Write(msg); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
got := make([]byte, len(msg))
|
||||||
|
_ = pc.SetReadDeadline(time.Now().Add(15 * time.Second))
|
||||||
|
_, err = io.ReadFull(pc, got)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := play("before"); err != nil {
|
||||||
|
t.Fatalf("baseline round-trip failed: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
relay.blackhole()
|
||||||
|
t.Log("path blackholed: no FIN, no RST, sockets held open")
|
||||||
|
|
||||||
|
// Wait for the heartbeats to fire, the sessions to be dropped, and the
|
||||||
|
// control session to reconnect through a fresh relay pair.
|
||||||
|
deadline := time.Now().Add(45 * time.Second)
|
||||||
|
var lastErr error
|
||||||
|
for time.Now().Before(deadline) {
|
||||||
|
if lastErr = play("after"); lastErr == nil {
|
||||||
|
t.Logf("recovered on its own; worker conns now %d", c.WorkerConnCount())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
time.Sleep(500 * time.Millisecond)
|
||||||
|
}
|
||||||
|
t.Fatalf("client never recovered from the blackholed path (last error: %v)", lastErr)
|
||||||
|
}
|
||||||
@@ -0,0 +1,269 @@
|
|||||||
|
package e2e
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"math/rand"
|
||||||
|
"net"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/iceBear67/redapricot/client"
|
||||||
|
"github.com/iceBear67/redapricot/client/wire"
|
||||||
|
)
|
||||||
|
|
||||||
|
// resumePlayer opens a player connection and returns it, having sent only the
|
||||||
|
// handshake. The caller keeps it open across the outage — which is the whole
|
||||||
|
// point: before stream resumption this socket was closed by the hub the instant
|
||||||
|
// its worker conn died.
|
||||||
|
func resumePlayer(t *testing.T, hubAddr, host string) net.Conn {
|
||||||
|
t.Helper()
|
||||||
|
pc, err := net.DialTimeout("tcp", hubAddr, 5*time.Second)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("player dial: %v", err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() { _ = pc.Close() })
|
||||||
|
if _, err := pc.Write(wire.BuildHandshake(767, host, 25565, 2)); err != nil {
|
||||||
|
t.Fatalf("player handshake: %v", err)
|
||||||
|
}
|
||||||
|
return pc
|
||||||
|
}
|
||||||
|
|
||||||
|
// echoExchange streams payload through an echoing destination and verifies that
|
||||||
|
// what comes back is byte-for-byte identical, calling disrupt once `at` bytes
|
||||||
|
// have made the round trip.
|
||||||
|
//
|
||||||
|
// Comparing the whole stream rather than sampling is deliberate: a resumption
|
||||||
|
// bug does not corrupt bytes, it duplicates or skips a range, and only an exact
|
||||||
|
// comparison of the full sequence catches an off-by-one in the offsets.
|
||||||
|
func echoExchange(t *testing.T, pc net.Conn, payload []byte, at int, disrupt func()) {
|
||||||
|
t.Helper()
|
||||||
|
const chunk = 16 << 10
|
||||||
|
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
wg.Add(1)
|
||||||
|
writeErr := make(chan error, 1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
for off := 0; off < len(payload); off += chunk {
|
||||||
|
end := min(off+chunk, len(payload))
|
||||||
|
_ = pc.SetWriteDeadline(time.Now().Add(60 * time.Second))
|
||||||
|
if _, err := pc.Write(payload[off:end]); err != nil {
|
||||||
|
writeErr <- fmt.Errorf("write at %d: %w", off, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
writeErr <- nil
|
||||||
|
}()
|
||||||
|
|
||||||
|
got := make([]byte, len(payload))
|
||||||
|
read, fired := 0, false
|
||||||
|
for read < len(got) {
|
||||||
|
_ = pc.SetReadDeadline(time.Now().Add(60 * time.Second))
|
||||||
|
n, err := pc.Read(got[read:])
|
||||||
|
read += n
|
||||||
|
if !fired && read >= at {
|
||||||
|
fired = true
|
||||||
|
disrupt()
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("player read failed after %d/%d bytes: %v", read, len(got), err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
wg.Wait()
|
||||||
|
if err := <-writeErr; err != nil {
|
||||||
|
t.Fatalf("player write: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !bytes.Equal(got, payload) {
|
||||||
|
// Report the first divergence: its offset says whether the stream gained
|
||||||
|
// or lost bytes, which is the difference between a retransmit that
|
||||||
|
// replayed too much and one that replayed too little.
|
||||||
|
for i := range got {
|
||||||
|
if got[i] != payload[i] {
|
||||||
|
t.Fatalf("echo diverges at byte %d of %d (sent %#x, got %#x)",
|
||||||
|
i, len(payload), payload[i], got[i])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func randomPayload(seed int64, n int) []byte {
|
||||||
|
p := make([]byte, n)
|
||||||
|
rand.New(rand.NewSource(seed)).Read(p)
|
||||||
|
return p
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestResumePreservesByteStream is the correctness bar for stream resumption:
|
||||||
|
// a worker conn is hard-reset mid-transfer and the *same* player connection must
|
||||||
|
// keep working, with a byte stream that neither gains nor loses a single byte.
|
||||||
|
//
|
||||||
|
// Byte-exactness is the whole difficulty. Frames handed to a dying socket are
|
||||||
|
// lost with no notification and the cipher cannot be resynchronized, so each
|
||||||
|
// side has to replay from the offset the other reports it accepted. Getting that
|
||||||
|
// offset wrong by any amount splices the stream mid-Minecraft-packet, which a
|
||||||
|
// round-trip test that only checked "traffic flows again" would happily pass.
|
||||||
|
func TestResumePreservesByteStream(t *testing.T) {
|
||||||
|
const psk = "e2e-resume"
|
||||||
|
hubPort := freePort(t)
|
||||||
|
hubAddr := fmt.Sprintf("127.0.0.1:%d", hubPort)
|
||||||
|
startHub(t, hubPort, psk)
|
||||||
|
dest := newMockDest(t, modeEcho)
|
||||||
|
|
||||||
|
// Only the tunnel runs through the relay; the player talks to the hub
|
||||||
|
// directly, as it would from the internet. So the drop hits the middle leg
|
||||||
|
// while both terminal sockets stay healthy — exactly the case resumption is
|
||||||
|
// for.
|
||||||
|
relay := newBlackholeRelay(t, hubAddr)
|
||||||
|
c := startClientWithPing(t, relay.addr, psk, 1, 400, []client.Mapping{
|
||||||
|
{Pattern: "mc.local", Destination: dest.addr},
|
||||||
|
})
|
||||||
|
|
||||||
|
pc := resumePlayer(t, hubAddr, "mc.local")
|
||||||
|
payload := randomPayload(1, 3<<20)
|
||||||
|
echoExchange(t, pc, payload, 512<<10, func() {
|
||||||
|
t.Log("hard-resetting the tunnel mid-transfer")
|
||||||
|
relay.dropAll()
|
||||||
|
})
|
||||||
|
t.Logf("stream survived the reset intact; worker conns now %d", c.WorkerConnCount())
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestResumeWithConcurrentStreams covers the failure the single-stream test
|
||||||
|
// cannot reach: two independent tunnels resume at once. A torn rebind that
|
||||||
|
// writes a frame onto the wrong conn would credit or reset the other player's
|
||||||
|
// tunnel, and that only shows up when a second player is there to be corrupted.
|
||||||
|
func TestResumeWithConcurrentStreams(t *testing.T) {
|
||||||
|
const psk = "e2e-resume-multi"
|
||||||
|
hubPort := freePort(t)
|
||||||
|
hubAddr := fmt.Sprintf("127.0.0.1:%d", hubPort)
|
||||||
|
startHub(t, hubPort, psk)
|
||||||
|
dest := newMockDest(t, modeEcho)
|
||||||
|
|
||||||
|
relay := newBlackholeRelay(t, hubAddr)
|
||||||
|
startClientWithPing(t, relay.addr, psk, 4, 400, []client.Mapping{
|
||||||
|
{Pattern: "mc.local", Destination: dest.addr},
|
||||||
|
})
|
||||||
|
|
||||||
|
const players = 3
|
||||||
|
conns := make([]net.Conn, players)
|
||||||
|
for i := range conns {
|
||||||
|
conns[i] = resumePlayer(t, hubAddr, "mc.local")
|
||||||
|
// Distinct payloads: if a reattach crosses two streams the bytes land on
|
||||||
|
// the wrong player, which an identical payload would hide.
|
||||||
|
if _, err := conns[i].Write([]byte(fmt.Sprintf("hello-%d", i))); err != nil {
|
||||||
|
t.Fatalf("player %d warmup write: %v", i, err)
|
||||||
|
}
|
||||||
|
echo := make([]byte, len("hello-0"))
|
||||||
|
_ = conns[i].SetReadDeadline(time.Now().Add(15 * time.Second))
|
||||||
|
if _, err := io.ReadFull(conns[i], echo); err != nil {
|
||||||
|
t.Fatalf("player %d warmup echo: %v", i, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for i := range conns {
|
||||||
|
wg.Add(1)
|
||||||
|
go func(i int) {
|
||||||
|
defer wg.Done()
|
||||||
|
payload := randomPayload(int64(100+i), 768<<10)
|
||||||
|
// Only the first player triggers the reset; the others are mid-flight
|
||||||
|
// when it lands.
|
||||||
|
disrupt := func() {}
|
||||||
|
if i == 0 {
|
||||||
|
disrupt = relay.dropAll
|
||||||
|
}
|
||||||
|
echoExchange(t, conns[i], payload, 128<<10, disrupt)
|
||||||
|
}(i)
|
||||||
|
}
|
||||||
|
wg.Wait()
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestResumeDisabledClosesImmediately pins the off switch. With resumption
|
||||||
|
// declined the hub must not hang the player waiting for a reattach that is never
|
||||||
|
// coming: the socket has to close as it did before the feature existed, so
|
||||||
|
// disabling it is a true revert rather than a slower failure.
|
||||||
|
func TestResumeDisabledClosesImmediately(t *testing.T) {
|
||||||
|
const psk = "e2e-resume-off"
|
||||||
|
hubPort := freePort(t)
|
||||||
|
hubAddr := fmt.Sprintf("127.0.0.1:%d", hubPort)
|
||||||
|
startHub(t, hubPort, psk)
|
||||||
|
dest := newMockDest(t, modeEcho)
|
||||||
|
|
||||||
|
relay := newBlackholeRelay(t, hubAddr)
|
||||||
|
off := false
|
||||||
|
startClientCfg(t, &client.Config{
|
||||||
|
Server: relay.addr,
|
||||||
|
PSK: psk,
|
||||||
|
MaxTunnels: 1,
|
||||||
|
PingIntervalMs: 400,
|
||||||
|
StreamResume: &off,
|
||||||
|
Mappings: []client.Mapping{{Pattern: "mc.local", Destination: dest.addr}},
|
||||||
|
})
|
||||||
|
|
||||||
|
pc := resumePlayer(t, hubAddr, "mc.local")
|
||||||
|
if _, err := pc.Write([]byte("ping")); err != nil {
|
||||||
|
t.Fatalf("warmup write: %v", err)
|
||||||
|
}
|
||||||
|
echo := make([]byte, 4)
|
||||||
|
_ = pc.SetReadDeadline(time.Now().Add(15 * time.Second))
|
||||||
|
if _, err := io.ReadFull(pc, echo); err != nil {
|
||||||
|
t.Fatalf("warmup echo: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
relay.dropAll()
|
||||||
|
|
||||||
|
// Well inside the hub's 20s grace: if the player is still open here, the hub
|
||||||
|
// parked a stream for a client that never opted in.
|
||||||
|
_ = pc.SetReadDeadline(time.Now().Add(10 * time.Second))
|
||||||
|
if _, err := pc.Read(make([]byte, 1)); err == nil {
|
||||||
|
t.Fatal("player socket stayed open after the tunnel dropped with resume disabled")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestResumeGraceExpiryClosesPlayer covers the other end of the lifetime: when
|
||||||
|
// the tunnel never comes back, a hung player must not hang forever. The hub
|
||||||
|
// drops it once its grace expires, without leaking the stream or its buffers.
|
||||||
|
func TestResumeGraceExpiryClosesPlayer(t *testing.T) {
|
||||||
|
const psk = "e2e-resume-grace"
|
||||||
|
hubPort := freePort(t)
|
||||||
|
hubAddr := fmt.Sprintf("127.0.0.1:%d", hubPort)
|
||||||
|
startHubCfg(t, hubPort, psk, map[string]any{"resumeGraceMs": 3000})
|
||||||
|
dest := newMockDest(t, modeEcho)
|
||||||
|
|
||||||
|
relay := newBlackholeRelay(t, hubAddr)
|
||||||
|
startClientCfg(t, &client.Config{
|
||||||
|
Server: relay.addr,
|
||||||
|
PSK: psk,
|
||||||
|
MaxTunnels: 1,
|
||||||
|
PingIntervalMs: 400,
|
||||||
|
ResumeGraceMs: 2000,
|
||||||
|
Mappings: []client.Mapping{{Pattern: "mc.local", Destination: dest.addr}},
|
||||||
|
})
|
||||||
|
|
||||||
|
pc := resumePlayer(t, hubAddr, "mc.local")
|
||||||
|
if _, err := pc.Write([]byte("ping")); err != nil {
|
||||||
|
t.Fatalf("warmup write: %v", err)
|
||||||
|
}
|
||||||
|
echo := make([]byte, 4)
|
||||||
|
_ = pc.SetReadDeadline(time.Now().Add(15 * time.Second))
|
||||||
|
if _, err := io.ReadFull(pc, echo); err != nil {
|
||||||
|
t.Fatalf("warmup echo: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stop carrying the tunnel entirely, so every reattach attempt fails.
|
||||||
|
relay.stop()
|
||||||
|
relay.dropAll()
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
_ = pc.SetReadDeadline(time.Now().Add(30 * time.Second))
|
||||||
|
if _, err := pc.Read(make([]byte, 1)); err == nil {
|
||||||
|
t.Fatal("player socket never closed after the resume grace expired")
|
||||||
|
}
|
||||||
|
if elapsed := time.Since(start); elapsed < time.Second {
|
||||||
|
t.Fatalf("player closed after %s, before any reattach could be attempted", elapsed)
|
||||||
|
} else {
|
||||||
|
t.Logf("hung player released after %s", elapsed.Round(100*time.Millisecond))
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -27,17 +27,15 @@ func echoRounds(t *testing.T, hubAddr string, rounds, size int) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// TestSlowPlayerDoesNotStallOthers: a player that stops reading while megabytes
|
// TestSlowPlayerDoesNotStallOthers: a player that stops reading while megabytes
|
||||||
// are echoed back to it must not stall another stream on the same worker
|
// are echoed back to it must not stall another player. Flow control and the
|
||||||
// connection (maxConn=1 forces sharing). Before per-stream flow control, the
|
// 1:1 worker model keep a jammed tunnel from taking anyone else with it.
|
||||||
// hub paused the whole worker socket once that player's write queue filled,
|
|
||||||
// freezing every other stream's downstream data.
|
|
||||||
func TestSlowPlayerDoesNotStallOthers(t *testing.T) {
|
func TestSlowPlayerDoesNotStallOthers(t *testing.T) {
|
||||||
const psk = "e2e-slowplayer"
|
const psk = "e2e-slowplayer"
|
||||||
port := freePort(t)
|
port := freePort(t)
|
||||||
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
|
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
|
||||||
startHub(t, port, psk)
|
startHub(t, port, psk)
|
||||||
dest := newMockDest(t, modeEcho)
|
dest := newMockDest(t, modeEcho)
|
||||||
startClient(t, hubAddr, psk, 1, []client.Mapping{
|
startClient(t, hubAddr, psk, 2, []client.Mapping{
|
||||||
{Pattern: "mc.local", Destination: dest.addr},
|
{Pattern: "mc.local", Destination: dest.addr},
|
||||||
})
|
})
|
||||||
|
|
||||||
@@ -52,7 +50,7 @@ func TestSlowPlayerDoesNotStallOthers(t *testing.T) {
|
|||||||
// Let the slow stream jam: its flow-control window fills and stays full.
|
// Let the slow stream jam: its flow-control window fills and stays full.
|
||||||
time.Sleep(1 * time.Second)
|
time.Sleep(1 * time.Second)
|
||||||
|
|
||||||
// The fast player shares the single worker conn and must still round-trip.
|
// The fast player has its own worker conn and must still round-trip.
|
||||||
echoRounds(t, hubAddr, 10, 8*1024)
|
echoRounds(t, hubAddr, 10, 8*1024)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -67,7 +65,7 @@ func TestSlowDestinationDoesNotStallOthers(t *testing.T) {
|
|||||||
startHub(t, port, psk)
|
startHub(t, port, psk)
|
||||||
dest := newMockDest(t, modeEcho)
|
dest := newMockDest(t, modeEcho)
|
||||||
hole := newMockDest(t, modeBlackhole)
|
hole := newMockDest(t, modeBlackhole)
|
||||||
startClient(t, hubAddr, psk, 1, []client.Mapping{
|
startClient(t, hubAddr, psk, 2, []client.Mapping{
|
||||||
{Pattern: "mc.local", Destination: dest.addr},
|
{Pattern: "mc.local", Destination: dest.addr},
|
||||||
{Pattern: "hole.local", Destination: hole.addr},
|
{Pattern: "hole.local", Destination: hole.addr},
|
||||||
})
|
})
|
||||||
|
|||||||
@@ -0,0 +1,219 @@
|
|||||||
|
package e2e
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"bytes"
|
||||||
|
"crypto/hmac"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/iceBear67/redapricot/client"
|
||||||
|
"github.com/iceBear67/redapricot/client/wire"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ---- mock velocity-forwarding destination ----
|
||||||
|
|
||||||
|
// veloEvent is what the mock backend saw in the (verified) forwarding payload.
|
||||||
|
type veloEvent struct {
|
||||||
|
version int
|
||||||
|
ip string
|
||||||
|
uuid []byte
|
||||||
|
name string
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type veloDest struct {
|
||||||
|
addr string
|
||||||
|
secret string
|
||||||
|
success []byte // the Login Success packet the backend sends after the exchange
|
||||||
|
events chan veloEvent
|
||||||
|
}
|
||||||
|
|
||||||
|
// newVeloDest starts a mock backend that requires Velocity modern forwarding:
|
||||||
|
// it reads the handshake and Login Start, sends the velocity:player_info
|
||||||
|
// query (with a negative message id, as Paper's random ids often are), verifies
|
||||||
|
// the HMAC-signed response, and finally sends a recognizable Login Success.
|
||||||
|
func newVeloDest(t *testing.T, secret string) *veloDest {
|
||||||
|
t.Helper()
|
||||||
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("velo dest listen: %v", err)
|
||||||
|
}
|
||||||
|
d := &veloDest{
|
||||||
|
addr: ln.Addr().String(),
|
||||||
|
secret: secret,
|
||||||
|
success: mcPacket(wire.NewWriter().
|
||||||
|
VarInt(0x02). // Login Success
|
||||||
|
Bytes(bytes.Repeat([]byte{0x42}, 16)).
|
||||||
|
String("e2ePlayer").
|
||||||
|
VarInt(0).
|
||||||
|
Out()),
|
||||||
|
events: make(chan veloEvent, 16),
|
||||||
|
}
|
||||||
|
t.Cleanup(func() { _ = ln.Close() })
|
||||||
|
go func() {
|
||||||
|
for {
|
||||||
|
conn, err := ln.Accept()
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
go d.handle(conn)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
return d
|
||||||
|
}
|
||||||
|
|
||||||
|
const veloMsgID = -777
|
||||||
|
|
||||||
|
func (d *veloDest) handle(conn net.Conn) {
|
||||||
|
defer conn.Close()
|
||||||
|
ev := d.exchange(conn)
|
||||||
|
d.events <- ev
|
||||||
|
if ev.err == nil {
|
||||||
|
_, _ = conn.Write(d.success)
|
||||||
|
}
|
||||||
|
_, _ = io.Copy(io.Discard, conn) // hold the connection until the peer closes
|
||||||
|
}
|
||||||
|
|
||||||
|
func (d *veloDest) exchange(conn net.Conn) veloEvent {
|
||||||
|
br := bufio.NewReader(conn)
|
||||||
|
if _, err := readMCPacket(br); err != nil { // handshake
|
||||||
|
return veloEvent{err: fmt.Errorf("read handshake: %w", err)}
|
||||||
|
}
|
||||||
|
if _, err := readMCPacket(br); err != nil { // login start
|
||||||
|
return veloEvent{err: fmt.Errorf("read login start: %w", err)}
|
||||||
|
}
|
||||||
|
|
||||||
|
query := mcPacket(wire.NewWriter().
|
||||||
|
VarInt(0x04). // Login Plugin Request
|
||||||
|
VarInt(veloMsgID).
|
||||||
|
String("velocity:player_info").
|
||||||
|
U8(0x04). // max supported forwarding version
|
||||||
|
Out())
|
||||||
|
if _, err := conn.Write(query); err != nil {
|
||||||
|
return veloEvent{err: err}
|
||||||
|
}
|
||||||
|
|
||||||
|
resp, err := readMCPacket(br)
|
||||||
|
if err != nil {
|
||||||
|
return veloEvent{err: fmt.Errorf("read plugin response: %w", err)}
|
||||||
|
}
|
||||||
|
r := wire.NewReader(resp)
|
||||||
|
id, _ := r.VarInt()
|
||||||
|
if id != 0x02 {
|
||||||
|
return veloEvent{err: fmt.Errorf("expected Login Plugin Response, got packet %#x", id)}
|
||||||
|
}
|
||||||
|
msgID, _ := r.VarInt()
|
||||||
|
if !bytes.Equal(wire.AppendVarInt(nil, msgID), wire.AppendVarInt(nil, veloMsgID)) {
|
||||||
|
return veloEvent{err: fmt.Errorf("message id not echoed: got %d", msgID)}
|
||||||
|
}
|
||||||
|
okFlag, _ := r.U8()
|
||||||
|
if okFlag != 1 {
|
||||||
|
return veloEvent{err: fmt.Errorf("response marked unsuccessful")}
|
||||||
|
}
|
||||||
|
sig, err := r.Bytes(32)
|
||||||
|
if err != nil {
|
||||||
|
return veloEvent{err: fmt.Errorf("missing signature: %w", err)}
|
||||||
|
}
|
||||||
|
payload := r.Remaining()
|
||||||
|
mac := hmac.New(sha256.New, []byte(d.secret))
|
||||||
|
mac.Write(payload)
|
||||||
|
if !hmac.Equal(sig, mac.Sum(nil)) {
|
||||||
|
return veloEvent{err: fmt.Errorf("forwarding signature does not verify")}
|
||||||
|
}
|
||||||
|
|
||||||
|
pr := wire.NewReader(payload)
|
||||||
|
var ev veloEvent
|
||||||
|
ev.version, _ = pr.VarInt()
|
||||||
|
ev.ip, _ = pr.String()
|
||||||
|
ev.uuid, _ = pr.Bytes(16)
|
||||||
|
ev.name, err = pr.String()
|
||||||
|
if err != nil {
|
||||||
|
return veloEvent{err: fmt.Errorf("truncated payload: %w", err)}
|
||||||
|
}
|
||||||
|
if props, err := pr.VarInt(); err != nil || props != 0 || len(pr.Remaining()) != 0 {
|
||||||
|
return veloEvent{err: fmt.Errorf("unexpected properties/trailer in payload")}
|
||||||
|
}
|
||||||
|
return ev
|
||||||
|
}
|
||||||
|
|
||||||
|
func mcPacket(body []byte) []byte {
|
||||||
|
return append(wire.AppendVarInt(nil, len(body)), body...)
|
||||||
|
}
|
||||||
|
|
||||||
|
func readMCPacket(br *bufio.Reader) ([]byte, error) {
|
||||||
|
n, err := wire.ReadVarInt(br)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if n <= 0 || n > 1<<20 {
|
||||||
|
return nil, fmt.Errorf("bad packet length %d", n)
|
||||||
|
}
|
||||||
|
pkt := make([]byte, n)
|
||||||
|
if _, err := io.ReadFull(br, pkt); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return pkt, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- test ----
|
||||||
|
|
||||||
|
// TestVelocityForwarding drives a full player login through the hub and a
|
||||||
|
// velocity-enabled mapping: the backend's velocity:player_info query must be
|
||||||
|
// answered by the client (never reaching the player), carrying the player's
|
||||||
|
// real IP, username and UUID, and the player's first bytes must be the
|
||||||
|
// backend's Login Success.
|
||||||
|
func TestVelocityForwarding(t *testing.T) {
|
||||||
|
const psk = "e2e-velocity"
|
||||||
|
const secret = "velo-forwarding-secret"
|
||||||
|
port := freePort(t)
|
||||||
|
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
|
||||||
|
startHub(t, port, psk)
|
||||||
|
dest := newVeloDest(t, secret)
|
||||||
|
startClient(t, hubAddr, psk, 2, []client.Mapping{
|
||||||
|
{Pattern: `velo\.local`, Destination: dest.addr, VelocitySecret: secret},
|
||||||
|
})
|
||||||
|
|
||||||
|
pc := dialPlayer(t, hubAddr, "velo.local") // protocol 767, login intent
|
||||||
|
defer pc.Close()
|
||||||
|
uuid, _ := hex.DecodeString("00112233445566778899aabbccddeeff")
|
||||||
|
loginStart := mcPacket(wire.NewWriter().VarInt(0x00).String("e2ePlayer").Bytes(uuid).Out())
|
||||||
|
if _, err := pc.Write(loginStart); err != nil {
|
||||||
|
t.Fatalf("player login start: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var ev veloEvent
|
||||||
|
select {
|
||||||
|
case ev = <-dest.events:
|
||||||
|
case <-time.After(10 * time.Second):
|
||||||
|
t.Fatalf("backend never completed the forwarding exchange")
|
||||||
|
}
|
||||||
|
if ev.err != nil {
|
||||||
|
t.Fatalf("backend rejected the forwarding exchange: %v", ev.err)
|
||||||
|
}
|
||||||
|
if ev.version != 4 {
|
||||||
|
t.Fatalf("forwarding version = %d, want 4 (lazy session)", ev.version)
|
||||||
|
}
|
||||||
|
if ev.ip != "127.0.0.1" {
|
||||||
|
t.Fatalf("forwarded IP = %q, want the player's real 127.0.0.1", ev.ip)
|
||||||
|
}
|
||||||
|
if ev.name != "e2ePlayer" || !bytes.Equal(ev.uuid, uuid) {
|
||||||
|
t.Fatalf("forwarded profile = %s/%x, want e2ePlayer/%x", ev.name, ev.uuid, uuid)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The player must see the Login Success as its very first bytes — the
|
||||||
|
// velocity query must have been swallowed by the client.
|
||||||
|
got := make([]byte, len(dest.success))
|
||||||
|
_ = pc.SetReadDeadline(time.Now().Add(10 * time.Second))
|
||||||
|
if _, err := io.ReadFull(pc, got); err != nil {
|
||||||
|
t.Fatalf("player read login success: %v", err)
|
||||||
|
}
|
||||||
|
if !bytes.Equal(got, dest.success) {
|
||||||
|
t.Fatalf("player's first bytes are not the Login Success:\n got %x\nwant %x", got, dest.success)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -3,5 +3,14 @@
|
|||||||
"psk": "change-me-to-a-long-random-passphrase",
|
"psk": "change-me-to-a-long-random-passphrase",
|
||||||
"timestampWindowMs": 30000,
|
"timestampWindowMs": 30000,
|
||||||
"pendingTimeoutMs": 10000,
|
"pendingTimeoutMs": 10000,
|
||||||
"streamWindowBytes": 262144
|
"streamWindowBytes": 262144,
|
||||||
|
"sessionIdleTimeoutMs": 90000,
|
||||||
|
"streamResume": true,
|
||||||
|
"resumeGraceMs": 20000,
|
||||||
|
"maxParkedStreams": 256,
|
||||||
|
"statsIntervalMs": 0,
|
||||||
|
"registrationGraceMs": 15000,
|
||||||
|
"playerRatePerSec": 8,
|
||||||
|
"playerBurst": 16,
|
||||||
|
"maxPlayersPerIp": 64
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -12,7 +12,17 @@ public record Config(
|
|||||||
String psk,
|
String psk,
|
||||||
long timestampWindowMs,
|
long timestampWindowMs,
|
||||||
long pendingTimeoutMs,
|
long pendingTimeoutMs,
|
||||||
int streamWindowBytes
|
int streamWindowBytes,
|
||||||
|
long sessionIdleTimeoutMs,
|
||||||
|
boolean streamResume,
|
||||||
|
long resumeGraceMs,
|
||||||
|
int maxParkedStreams,
|
||||||
|
long maxParkedBytes,
|
||||||
|
long statsIntervalMs,
|
||||||
|
long registrationGraceMs,
|
||||||
|
double playerRatePerSec,
|
||||||
|
int playerBurst,
|
||||||
|
int maxPlayersPerIp
|
||||||
) {
|
) {
|
||||||
public static Config load(Path file) throws Exception {
|
public static Config load(Path file) throws Exception {
|
||||||
JsonObject json = new JsonObject(Files.readString(file));
|
JsonObject json = new JsonObject(Files.readString(file));
|
||||||
@@ -29,12 +39,46 @@ public record Config(
|
|||||||
int window = json.getInteger("streamWindowBytes", Protocol.DEFAULT_STREAM_WINDOW);
|
int window = json.getInteger("streamWindowBytes", Protocol.DEFAULT_STREAM_WINDOW);
|
||||||
window = Math.max(Protocol.MIN_STREAM_WINDOW, Math.min(Protocol.MAX_STREAM_WINDOW, window));
|
window = Math.max(Protocol.MIN_STREAM_WINDOW, Math.min(Protocol.MAX_STREAM_WINDOW, window));
|
||||||
|
|
||||||
|
boolean resume = json.getBoolean("streamResume", Boolean.TRUE);
|
||||||
|
// A parked stream can hold up to one window of unacked bytes plus one of
|
||||||
|
// parked player bytes, for the whole grace period, and nothing else
|
||||||
|
// bounds how many streams park at once — so anyone able to kill worker
|
||||||
|
// conns is otherwise a cheap memory amplifier. The default admits ~256
|
||||||
|
// hanging players at the default window.
|
||||||
|
int maxParked = json.getInteger("maxParkedStreams", 256);
|
||||||
|
long maxParkedBytes = json.getLong("maxParkedBytes", (long) maxParked * 2 * window);
|
||||||
|
|
||||||
return new Config(
|
return new Config(
|
||||||
host,
|
host,
|
||||||
port,
|
port,
|
||||||
psk,
|
psk,
|
||||||
json.getLong("timestampWindowMs", 30_000L),
|
json.getLong("timestampWindowMs", 30_000L),
|
||||||
json.getLong("pendingTimeoutMs", 10_000L),
|
json.getLong("pendingTimeoutMs", 10_000L),
|
||||||
window);
|
window,
|
||||||
|
// Comfortably above the client's default 20s ping interval;
|
||||||
|
// 0 disables the watchdog.
|
||||||
|
json.getLong("sessionIdleTimeoutMs", 90_000L),
|
||||||
|
resume,
|
||||||
|
// Must exceed the client's own grace by at least one dial, or the
|
||||||
|
// hub drops a player while its client is still mid-reattach. The
|
||||||
|
// value is advertised in SessionReady precisely so the client can
|
||||||
|
// clamp itself under it rather than rely on matching config.
|
||||||
|
json.getLong("resumeGraceMs", 20_000L),
|
||||||
|
maxParked,
|
||||||
|
maxParkedBytes,
|
||||||
|
json.getLong("statsIntervalMs", 0L),
|
||||||
|
// Long enough to cover a client's control-session reconnect
|
||||||
|
// (its backoff caps at 10s) without holding a player so long
|
||||||
|
// that they give up anyway; 0 disables and restores the old
|
||||||
|
// behaviour of dropping routes the moment a session closes.
|
||||||
|
json.getLong("registrationGraceMs", 15_000L),
|
||||||
|
// Player-only (Intent ∉ {17, 18}). 0 turns that mechanism off.
|
||||||
|
// Unmatched hostnames still consume a token — otherwise a
|
||||||
|
// hostname scan is a free flood. Intent 17 is never admitted
|
||||||
|
// through the limiter: every worker comes from the client's
|
||||||
|
// one IP.
|
||||||
|
Math.max(0, json.getDouble("playerRatePerSec", 8.0)),
|
||||||
|
Math.max(1, json.getInteger("playerBurst", 16)),
|
||||||
|
Math.max(0, json.getInteger("maxPlayersPerIp", 64)));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -7,10 +7,16 @@ import org.apache.logging.log4j.LogManager;
|
|||||||
import org.apache.logging.log4j.Logger;
|
import org.apache.logging.log4j.Logger;
|
||||||
|
|
||||||
import java.nio.charset.StandardCharsets;
|
import java.nio.charset.StandardCharsets;
|
||||||
|
import java.security.SecureRandom;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.HashSet;
|
||||||
|
import java.util.Iterator;
|
||||||
|
import java.util.LinkedHashMap;
|
||||||
|
import java.util.List;
|
||||||
import java.util.Locale;
|
import java.util.Locale;
|
||||||
import java.util.Map;
|
import java.util.Map;
|
||||||
|
import java.util.Set;
|
||||||
import java.util.concurrent.ConcurrentHashMap;
|
import java.util.concurrent.ConcurrentHashMap;
|
||||||
import java.util.concurrent.ThreadLocalRandom;
|
|
||||||
import java.util.regex.Pattern;
|
import java.util.regex.Pattern;
|
||||||
import java.util.regex.PatternSyntaxException;
|
import java.util.regex.PatternSyntaxException;
|
||||||
|
|
||||||
@@ -21,26 +27,76 @@ import java.util.regex.PatternSyntaxException;
|
|||||||
*/
|
*/
|
||||||
public final class Hub {
|
public final class Hub {
|
||||||
private static final Logger LOG = LogManager.getLogger("redapricot.hub");
|
private static final Logger LOG = LogManager.getLogger("redapricot.hub");
|
||||||
|
private static final SecureRandom RNG = new SecureRandom();
|
||||||
|
|
||||||
public final Vertx vertx;
|
public final Vertx vertx;
|
||||||
public final Config config;
|
public final Config config;
|
||||||
public final byte[] pskBytes;
|
public final byte[] pskBytes;
|
||||||
public final String pskAddress;
|
public final String pskAddress;
|
||||||
|
public final IpRateLimiter limiter;
|
||||||
|
|
||||||
private final Map<String, Registration> patterns = new ConcurrentHashMap<>();
|
private final Map<String, Registration> patterns = new ConcurrentHashMap<>();
|
||||||
private final Map<String, PendingPlayer> pending = new ConcurrentHashMap<>();
|
private final Map<String, PendingPlayer> pending = new ConcurrentHashMap<>();
|
||||||
|
|
||||||
/** A compiled routing pattern and the control session that registered it. */
|
/**
|
||||||
private record Registration(Pattern regex, ControlSession session) {}
|
* Every tunneled player, keyed by its current CID, whether live or parked.
|
||||||
|
* Keeping parked streams here rather than on the worker conn is the whole
|
||||||
|
* point: a stream's identity is the player, and state that dies with the
|
||||||
|
* connection cannot survive that connection dying. Insertion-ordered so the
|
||||||
|
* parked cap can evict the oldest first.
|
||||||
|
*/
|
||||||
|
private final Map<String, PlayerStream> streams = new LinkedHashMap<>();
|
||||||
|
private int parkedCount;
|
||||||
|
private long parkedBytes;
|
||||||
|
|
||||||
/** A successful match: the registered pattern that matched and its owning session. */
|
/**
|
||||||
public record Match(String pattern, ControlSession session) {}
|
* A compiled routing pattern and the control session that registered it.
|
||||||
|
*
|
||||||
|
* <p>{@code session} is null while the registration is <b>orphaned</b> — its
|
||||||
|
* client's control session has closed but the route is held open until
|
||||||
|
* {@code orphanDeadline} in case the client reconnects. Players matching an
|
||||||
|
* orphaned route are hung rather than refused.
|
||||||
|
*/
|
||||||
|
private record Registration(Pattern regex, ControlSession session, long orphanDeadline) {
|
||||||
|
Registration(Pattern regex, ControlSession session) {
|
||||||
|
this(regex, session, 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
boolean orphaned() {
|
||||||
|
return session == null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A successful match: the registered pattern that matched and its owning
|
||||||
|
* session, which is null when the route is orphaned (§ control-outage hang).
|
||||||
|
*/
|
||||||
|
public record Match(String pattern, ControlSession session, long orphanDeadline) {}
|
||||||
|
|
||||||
public Hub(Vertx vertx, Config config) {
|
public Hub(Vertx vertx, Config config) {
|
||||||
this.vertx = vertx;
|
this.vertx = vertx;
|
||||||
this.config = config;
|
this.config = config;
|
||||||
this.pskBytes = config.psk().getBytes(StandardCharsets.UTF_8);
|
this.pskBytes = config.psk().getBytes(StandardCharsets.UTF_8);
|
||||||
this.pskAddress = Crypto.pskAddress(config.psk());
|
this.pskAddress = Crypto.pskAddress(config.psk());
|
||||||
|
this.limiter = new IpRateLimiter(
|
||||||
|
config.playerRatePerSec(), config.playerBurst(), config.maxPlayersPerIp());
|
||||||
|
// Unit tests construct a Hub with a null Vertx (no event loop).
|
||||||
|
if (limiter.enabled() && vertx != null) {
|
||||||
|
vertx.setPeriodic(IpRateLimiter.SWEEP_MS, id -> limiter.sweep(System.currentTimeMillis()));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Admit one player socket from {@code ip}. {@code null} means allowed;
|
||||||
|
* the caller must {@link #releasePlayer} on every close path (pending
|
||||||
|
* timeout, unmatched host, player FIN, park eviction).
|
||||||
|
*/
|
||||||
|
public IpRateLimiter.Deny admitPlayer(String ip) {
|
||||||
|
return limiter.admit(ip, System.currentTimeMillis());
|
||||||
|
}
|
||||||
|
|
||||||
|
public void releasePlayer(String ip) {
|
||||||
|
limiter.release(ip, System.currentTimeMillis());
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- pattern registry ----
|
// ---- pattern registry ----
|
||||||
@@ -64,9 +120,29 @@ public final class Hub {
|
|||||||
}
|
}
|
||||||
patterns.put(pattern, new Registration(regex, session));
|
patterns.put(pattern, new Registration(regex, session));
|
||||||
LOG.info("registered pattern '{}' -> {}", pattern, session.id());
|
LOG.info("registered pattern '{}' -> {}", pattern, session.id());
|
||||||
|
replayAwaiting(pattern, session);
|
||||||
return Protocol.REGISTER_OK;
|
return Protocol.REGISTER_OK;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Deliver the control requests held while this pattern had no live session.
|
||||||
|
*
|
||||||
|
* <p>These players connected during the client's reconnect and were hung
|
||||||
|
* instead of refused; the request was never sent, so it is sent now. Each
|
||||||
|
* moves from "waiting for a route" to the ordinary "waiting for a worker",
|
||||||
|
* which means swapping its deadline over to {@code pendingTimeoutMs}.
|
||||||
|
*/
|
||||||
|
private void replayAwaiting(String pattern, ControlSession session) {
|
||||||
|
for (PendingPlayer p : pending.values()) {
|
||||||
|
if (!p.isAwaitingSession() || !p.getPattern().equals(pattern)) continue;
|
||||||
|
p.setAwaitingSession(false);
|
||||||
|
p.setOwner(session);
|
||||||
|
rearm(p, config.pendingTimeoutMs());
|
||||||
|
session.sendControlRequest(p.getCid(), p.getPattern(), p.getPlayerIp(), p.getPlayerPort());
|
||||||
|
LOG.info("replayed control request for hung player {} on session {}", p.getCidHex(), session.id());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
public void unregister(String pattern, ControlSession session) {
|
public void unregister(String pattern, ControlSession session) {
|
||||||
// Remove only if this session still owns the pattern (a newer session may have taken it).
|
// Remove only if this session still owns the pattern (a newer session may have taken it).
|
||||||
patterns.computeIfPresent(pattern, (k, reg) -> reg.session() == session ? null : reg);
|
patterns.computeIfPresent(pattern, (k, reg) -> reg.session() == session ? null : reg);
|
||||||
@@ -80,36 +156,140 @@ public final class Hub {
|
|||||||
public Match match(String address) {
|
public Match match(String address) {
|
||||||
String host = normalizeAddress(address);
|
String host = normalizeAddress(address);
|
||||||
for (Map.Entry<String, Registration> e : patterns.entrySet()) {
|
for (Map.Entry<String, Registration> e : patterns.entrySet()) {
|
||||||
if (e.getValue().regex().matcher(host).matches()) {
|
Registration reg = e.getValue();
|
||||||
return new Match(e.getKey(), e.getValue().session());
|
if (reg.regex().matcher(host).matches()) {
|
||||||
|
return new Match(e.getKey(), reg.session(), reg.orphanDeadline());
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Drop every pattern owned by a (closing) session, plus any players still pending for it. */
|
/**
|
||||||
|
* A control session closed. Its routes are kept as <b>orphaned</b> for
|
||||||
|
* {@code registrationGraceMs}, and the players waiting on them are hung
|
||||||
|
* rather than dropped.
|
||||||
|
*
|
||||||
|
* <p>Without this, a client's reconnect — half a second at best, ten at worst
|
||||||
|
* once its backoff has grown — is a window in which every arriving player is
|
||||||
|
* told there is no such server, even though the tunnel is seconds from being
|
||||||
|
* back. The players already tunneled are unaffected either way; they ride
|
||||||
|
* worker conns, which a control-session close never touches.
|
||||||
|
*
|
||||||
|
* <p>A grace of 0 restores the old behaviour exactly.
|
||||||
|
*/
|
||||||
public void removeSession(ControlSession session) {
|
public void removeSession(ControlSession session) {
|
||||||
patterns.entrySet().removeIf(e -> e.getValue().session() == session);
|
long grace = config.registrationGraceMs();
|
||||||
|
if (grace <= 0) {
|
||||||
|
patterns.entrySet().removeIf(e -> e.getValue().session() == session);
|
||||||
|
pending.values().removeIf(p -> {
|
||||||
|
if (p.getOwner() != session) return false;
|
||||||
|
cancelTimer(p);
|
||||||
|
p.getSocket().close();
|
||||||
|
LOG.info("dropping pending player {} (control session {} closed)", p.getCidHex(), session.id());
|
||||||
|
return true;
|
||||||
|
});
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
long deadline = System.currentTimeMillis() + grace;
|
||||||
|
int orphaned = 0;
|
||||||
|
for (Map.Entry<String, Registration> e : patterns.entrySet()) {
|
||||||
|
Registration reg = e.getValue();
|
||||||
|
if (reg.session() != session) continue;
|
||||||
|
e.setValue(new Registration(reg.regex(), null, deadline));
|
||||||
|
orphaned++;
|
||||||
|
}
|
||||||
|
|
||||||
|
// A player that was already matched is in the same position: its request
|
||||||
|
// went to a session that will never answer, so it waits for the route to
|
||||||
|
// come back and is then replayed like any other.
|
||||||
|
int hung = 0;
|
||||||
|
for (PendingPlayer p : pending.values()) {
|
||||||
|
if (p.getOwner() != session) continue;
|
||||||
|
p.setOwner(null);
|
||||||
|
p.setAwaitingSession(true);
|
||||||
|
rearm(p, grace);
|
||||||
|
hung++;
|
||||||
|
}
|
||||||
|
if (orphaned > 0 || hung > 0) {
|
||||||
|
LOG.info("control session {} closed; holding {} route(s) and {} player(s) for {}ms",
|
||||||
|
session.id(), orphaned, hung, grace);
|
||||||
|
vertx.setTimer(grace, id -> expireOrphans());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Drop routes whose grace ran out, and the players still hung on them. */
|
||||||
|
private void expireOrphans() {
|
||||||
|
long now = System.currentTimeMillis();
|
||||||
|
Set<String> gone = new HashSet<>();
|
||||||
|
patterns.entrySet().removeIf(e -> {
|
||||||
|
Registration reg = e.getValue();
|
||||||
|
if (!reg.orphaned() || reg.orphanDeadline() > now) return false;
|
||||||
|
gone.add(e.getKey());
|
||||||
|
return true;
|
||||||
|
});
|
||||||
|
if (gone.isEmpty()) return;
|
||||||
|
LOG.info("dropping {} orphaned route(s) not reclaimed within the grace period", gone.size());
|
||||||
pending.values().removeIf(p -> {
|
pending.values().removeIf(p -> {
|
||||||
if (p.getOwner() != session) return false;
|
if (!p.isAwaitingSession() || !gone.contains(p.getPattern())) return false;
|
||||||
if (p.getTimerId() >= 0) vertx.cancelTimer(p.getTimerId());
|
cancelTimer(p);
|
||||||
p.getSocket().close();
|
p.getSocket().close();
|
||||||
LOG.info("dropping pending player {} (control session {} closed)", p.getCidHex(), session.id());
|
LOG.info("dropping hung player {} (route '{}' never came back)", p.getCidHex(), p.getPattern());
|
||||||
return true;
|
return true;
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- pending players ----
|
// ---- pending players ----
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Mint a takeover capability. The CID is the only thing authorizing a client
|
||||||
|
* to claim a player (and, with resumption, to reclaim one), so it comes from
|
||||||
|
* a cryptographic source rather than ThreadLocalRandom — a predictable value
|
||||||
|
* would be a session-hijacking primitive.
|
||||||
|
*/
|
||||||
public byte[] newCid() {
|
public byte[] newCid() {
|
||||||
byte[] cid = new byte[Protocol.CID_LEN];
|
byte[] cid = new byte[Protocol.CID_LEN];
|
||||||
ThreadLocalRandom.current().nextBytes(cid);
|
RNG.nextBytes(cid);
|
||||||
return cid;
|
return cid;
|
||||||
}
|
}
|
||||||
|
|
||||||
public void addPending(PendingPlayer p) {
|
public void addPending(PendingPlayer p) {
|
||||||
pending.put(p.getCidHex(), p);
|
pending.put(p.getCidHex(), p);
|
||||||
p.setTimerId(vertx.setTimer(config.pendingTimeoutMs(), id -> {
|
rearm(p, config.pendingTimeoutMs());
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Hang a player whose route is orphaned: hold it until a client re-registers
|
||||||
|
* the pattern, at which point {@link #replayAwaiting} delivers the control
|
||||||
|
* request that was never sent.
|
||||||
|
*
|
||||||
|
* @param deadline wall-clock millis at which the route's grace runs out
|
||||||
|
*/
|
||||||
|
public void addAwaiting(PendingPlayer p, long deadline) {
|
||||||
|
p.setOwner(null);
|
||||||
|
p.setAwaitingSession(true);
|
||||||
|
pending.put(p.getCidHex(), p);
|
||||||
|
rearm(p, Math.max(1, deadline - System.currentTimeMillis()));
|
||||||
|
LOG.info("holding player {} for '{}': route is orphaned, waiting for its client",
|
||||||
|
p.getCidHex(), p.getPattern());
|
||||||
|
}
|
||||||
|
|
||||||
|
public PendingPlayer takePending(byte[] cid) {
|
||||||
|
String hex = Hex.encode(cid);
|
||||||
|
PendingPlayer p = pending.remove(hex);
|
||||||
|
if (p != null) cancelTimer(p);
|
||||||
|
return p;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void removePending(String cidHex) {
|
||||||
|
PendingPlayer p = pending.remove(cidHex);
|
||||||
|
if (p != null) cancelTimer(p);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Replace a pending player's deadline, cancelling whatever it had. */
|
||||||
|
private void rearm(PendingPlayer p, long delayMs) {
|
||||||
|
cancelTimer(p);
|
||||||
|
p.setTimerId(vertx.setTimer(delayMs, id -> {
|
||||||
PendingPlayer removed = pending.remove(p.getCidHex());
|
PendingPlayer removed = pending.remove(p.getCidHex());
|
||||||
if (removed != null) {
|
if (removed != null) {
|
||||||
LOG.warn("pending player {} timed out", p.getCidHex());
|
LOG.warn("pending player {} timed out", p.getCidHex());
|
||||||
@@ -118,16 +298,171 @@ public final class Hub {
|
|||||||
}));
|
}));
|
||||||
}
|
}
|
||||||
|
|
||||||
public PendingPlayer takePending(byte[] cid) {
|
private void cancelTimer(PendingPlayer p) {
|
||||||
String hex = Hex.encode(cid);
|
if (p.getTimerId() >= 0) {
|
||||||
PendingPlayer p = pending.remove(hex);
|
vertx.cancelTimer(p.getTimerId());
|
||||||
if (p != null && p.getTimerId() >= 0) vertx.cancelTimer(p.getTimerId());
|
p.setTimerId(-1);
|
||||||
return p;
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
public void removePending(String cidHex) {
|
// ---- tunneled streams & resumption (§7.5) ----
|
||||||
PendingPlayer p = pending.remove(cidHex);
|
|
||||||
if (p != null && p.getTimerId() >= 0) vertx.cancelTimer(p.getTimerId());
|
/** Register a stream that has just been bound to a worker conn. */
|
||||||
|
public void addStream(PlayerStream st) {
|
||||||
|
streams.put(st.cidHex, st);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Forget a stream for good; its player socket is gone or going. */
|
||||||
|
public void removeStream(PlayerStream st) {
|
||||||
|
if (streams.remove(st.cidHex) == null) return;
|
||||||
|
if (st.parked) unpark(st);
|
||||||
|
st.unacked.clear();
|
||||||
|
// The player's closeHandler was replaced at SYN/RESUME bind, so the
|
||||||
|
// HubConnection cleanup never runs. This is the live/parked release
|
||||||
|
// path; pending/unmatched still go through HubConnection.closeCleanup.
|
||||||
|
releasePlayer(st.playerIp);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Look up a stream by the CID a client presented, live or parked. */
|
||||||
|
public PlayerStream streamByCid(byte[] cid) {
|
||||||
|
return streams.get(Hex.encode(cid));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Player bytes arrived: hand them to whichever conn currently carries the
|
||||||
|
* stream. Routing here rather than from the conn that installed the socket
|
||||||
|
* handler is what lets a stream change conns without rebinding handlers.
|
||||||
|
*/
|
||||||
|
public void onPlayerData(PlayerStream st, io.vertx.core.buffer.Buffer buf) {
|
||||||
|
WorkerConn w = st.worker;
|
||||||
|
if (w != null) {
|
||||||
|
w.playerData(st, buf);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// Parked, so there is nowhere to send: hold the bytes in order and make
|
||||||
|
// sure the socket really is stopped. The player is paused on park, but a
|
||||||
|
// batch already in flight can still land here.
|
||||||
|
if (st.pendingUp == null) st.pendingUp = io.vertx.core.buffer.Buffer.buffer();
|
||||||
|
st.pendingUp.appendBuffer(buf);
|
||||||
|
st.player.pause();
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The player hung up. Drop the stream everywhere and tell the client if it is still bound. */
|
||||||
|
public void onPlayerGone(PlayerStream st) {
|
||||||
|
WorkerConn w = st.worker;
|
||||||
|
removeStream(st);
|
||||||
|
if (w != null) w.playerGone(st);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Hang a player whose worker conn died, instead of closing it.
|
||||||
|
*
|
||||||
|
* <p>Only the tunnel leg failed — the player socket is still perfectly good —
|
||||||
|
* so it is paused and held until the client reattaches the stream over a
|
||||||
|
* fresh conn. The deadline is absolute and fixed at the first park: re-arming
|
||||||
|
* it on each park would let a flapping client hold a player forever.
|
||||||
|
*
|
||||||
|
* @return false if the stream cannot be parked and must be closed instead
|
||||||
|
*/
|
||||||
|
public boolean park(PlayerStream st) {
|
||||||
|
if (!st.resumable || st.parked) return false;
|
||||||
|
long now = System.currentTimeMillis();
|
||||||
|
if (st.graceDeadline == 0) st.graceDeadline = now + config.resumeGraceMs();
|
||||||
|
long remaining = st.graceDeadline - now;
|
||||||
|
if (remaining <= 0) return false;
|
||||||
|
|
||||||
|
st.parked = true;
|
||||||
|
st.worker = null;
|
||||||
|
// Explicitly, not as a side effect of the window filling: an idle stream
|
||||||
|
// has a wide-open window, so nothing else would stop the next keepalive
|
||||||
|
// walking into a send on a dead connection.
|
||||||
|
st.player.pause();
|
||||||
|
st.pausedForAggregate = false; // that conn's drain handler will never fire again
|
||||||
|
|
||||||
|
parkedCount++;
|
||||||
|
parkedBytes += st.parkedBytes();
|
||||||
|
st.timerId = vertx.setTimer(remaining, id -> {
|
||||||
|
LOG.info("parked player {} not reclaimed within grace; closing", st.cidHex);
|
||||||
|
removeStream(st);
|
||||||
|
st.player.close();
|
||||||
|
});
|
||||||
|
enforceParkedCaps(st);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Reclaim a parked stream. Returns null when no stream is parked under this
|
||||||
|
* CID; the caller distinguishes "never heard of it" from "still bound
|
||||||
|
* elsewhere" via {@link #streamByCid}.
|
||||||
|
*/
|
||||||
|
public PlayerStream takeParked(byte[] cid) {
|
||||||
|
PlayerStream st = streams.get(Hex.encode(cid));
|
||||||
|
if (st == null || !st.parked) return null;
|
||||||
|
unpark(st);
|
||||||
|
return st;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Re-key a stream to the freshly minted CID handed out in RESUME_ACK. */
|
||||||
|
public void rekeyStream(PlayerStream st, byte[] cid) {
|
||||||
|
streams.remove(st.cidHex);
|
||||||
|
st.cid = cid;
|
||||||
|
st.cidHex = Hex.encode(cid);
|
||||||
|
streams.put(st.cidHex, st);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void unpark(PlayerStream st) {
|
||||||
|
if (!st.parked) return;
|
||||||
|
st.parked = false;
|
||||||
|
parkedCount--;
|
||||||
|
parkedBytes -= st.parkedBytes();
|
||||||
|
if (parkedBytes < 0) parkedBytes = 0;
|
||||||
|
if (st.timerId >= 0) {
|
||||||
|
vertx.cancelTimer(st.timerId);
|
||||||
|
st.timerId = -1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Bound what hanging players may cost. Each holds up to a window of unsent
|
||||||
|
* bytes plus a window of parked ones for the whole grace period, and nothing
|
||||||
|
* else limits how many park at once — so anyone able to kill worker conns is
|
||||||
|
* otherwise a cheap memory amplifier. Oldest first, since they have the least
|
||||||
|
* grace left to be reclaimed in.
|
||||||
|
*/
|
||||||
|
private void enforceParkedCaps(PlayerStream keep) {
|
||||||
|
if (parkedCount <= config.maxParkedStreams() && parkedBytes <= config.maxParkedBytes()) return;
|
||||||
|
List<PlayerStream> evict = new ArrayList<>();
|
||||||
|
Iterator<PlayerStream> it = streams.values().iterator();
|
||||||
|
while (it.hasNext() && (parkedCount - evict.size() > config.maxParkedStreams()
|
||||||
|
|| parkedBytes > config.maxParkedBytes())) {
|
||||||
|
PlayerStream st = it.next();
|
||||||
|
if (!st.parked || st == keep) continue;
|
||||||
|
evict.add(st);
|
||||||
|
parkedBytes -= st.parkedBytes();
|
||||||
|
}
|
||||||
|
for (PlayerStream st : evict) {
|
||||||
|
LOG.warn("parked-stream cap reached; dropping hanging player {}", st.cidHex);
|
||||||
|
parkedBytes += st.parkedBytes(); // removeStream subtracts it again
|
||||||
|
removeStream(st);
|
||||||
|
st.player.close();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Live and parked stream counts, for the periodic stats line. */
|
||||||
|
public int streamCount() {
|
||||||
|
return streams.size();
|
||||||
|
}
|
||||||
|
|
||||||
|
public int parkedCount() {
|
||||||
|
return parkedCount;
|
||||||
|
}
|
||||||
|
|
||||||
|
public long parkedBytes() {
|
||||||
|
return parkedBytes;
|
||||||
|
}
|
||||||
|
|
||||||
|
public int patternCount() {
|
||||||
|
return patterns.size();
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- helpers ----
|
// ---- helpers ----
|
||||||
|
|||||||
@@ -103,17 +103,45 @@ public final class HubConnection {
|
|||||||
if (intent == Protocol.INTENT_REDAPRICOT) {
|
if (intent == Protocol.INTENT_REDAPRICOT) {
|
||||||
beginRedapricot(address, afterHandshake);
|
beginRedapricot(address, afterHandshake);
|
||||||
} else if (intent == Protocol.INTENT_RESERVED) {
|
} else if (intent == Protocol.INTENT_RESERVED) {
|
||||||
LOG.info("{} reserved intent 18; closing", id);
|
LOG.info("{} reserved intent 18; replying with a status line and closing", id);
|
||||||
socket.close();
|
sendStatusLine();
|
||||||
} else {
|
} else {
|
||||||
handlePlayer(address);
|
handlePlayer(address);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ---- Intent 18 (reserved: management/status) ----
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Reply to an Intent-18 probe with a Minecraft status-response packet
|
||||||
|
* ({@code [Len: VarInt][0x00][JSON: String]}), the same shape a player gets
|
||||||
|
* for a status query (Intent 1), then close. This lets an operator probe
|
||||||
|
* the public port with ordinary tooling without joining the protocol, and
|
||||||
|
* it is the one reply the hub sends in plaintext — Intent 18 never
|
||||||
|
* negotiates encryption.
|
||||||
|
*/
|
||||||
|
private void sendStatusLine() {
|
||||||
|
String json = "{\"description\":{\"text\":\"redapricot hub\"},"
|
||||||
|
+ "\"version\":{\"name\":\"redapricot\",\"protocol\":767},"
|
||||||
|
+ "\"players\":{\"max\":0,\"online\":0}}";
|
||||||
|
ProtoWriter body = new ProtoWriter().u8(0x00).string(json);
|
||||||
|
byte[] bodyBytes = body.toBytes();
|
||||||
|
ProtoWriter pkt = new ProtoWriter().varInt(bodyBytes.length).bytes(bodyBytes);
|
||||||
|
// end() writes the reply and closes after it lands, so the packet is
|
||||||
|
// never cut short by the close racing the flush.
|
||||||
|
socket.end(Buffer.buffer(pkt.toBytes()));
|
||||||
|
}
|
||||||
|
|
||||||
// ---- redapricot session (Intent 17) ----
|
// ---- redapricot session (Intent 17) ----
|
||||||
|
|
||||||
private void beginRedapricot(String address, Buffer afterHandshake) {
|
private void beginRedapricot(String address, Buffer afterHandshake) {
|
||||||
if (!address.equalsIgnoreCase(hub.pskAddress)) {
|
// Strict equality, per PROTOCOL.md §2: the contract is exactly
|
||||||
|
// lowercase_hex(SHA3-224(PSK)), and the client always sends that. An
|
||||||
|
// uppercase or otherwise case-folded variant is not a valid session —
|
||||||
|
// accepting it would widen the acceptance surface beyond what the spec
|
||||||
|
// promises (and what the client ever produces), which is exactly the
|
||||||
|
// kind of "close enough" check that hides a wrong-PSK probe.
|
||||||
|
if (!address.equals(hub.pskAddress)) {
|
||||||
LOG.warn("{} bad PSK address; closing", id);
|
LOG.warn("{} bad PSK address; closing", id);
|
||||||
socket.close();
|
socket.close();
|
||||||
return;
|
return;
|
||||||
@@ -165,6 +193,13 @@ public final class HubConnection {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
peerWindow = Math.min(peerWindow, Protocol.MAX_STREAM_WINDOW);
|
peerWindow = Math.min(peerWindow, Protocol.MAX_STREAM_WINDOW);
|
||||||
|
// Heartbeat is optional: accept it only when the client offered it, so
|
||||||
|
// older clients keep working (they just lose silent-path detection).
|
||||||
|
boolean heartbeat = (flags & Protocol.FLAG_WORKER_HEARTBEAT) != 0;
|
||||||
|
// Resumption likewise. Parking a stream for a client that will never
|
||||||
|
// reattach is strictly worse than closing it — the player hangs for the
|
||||||
|
// whole grace instead of failing fast — so this bit gates the park path.
|
||||||
|
boolean resume = hub.config.streamResume() && (flags & Protocol.FLAG_STREAM_RESUME) != 0;
|
||||||
|
|
||||||
// REKEY = Rand || Timestamp(I64 big-endian). Magic is excluded.
|
// REKEY = Rand || Timestamp(I64 big-endian). Magic is excluded.
|
||||||
byte[] rekey = new byte[randLen + 8];
|
byte[] rekey = new byte[randLen + 8];
|
||||||
@@ -178,32 +213,86 @@ public final class HubConnection {
|
|||||||
frames.switchCiphers(
|
frames.switchCiphers(
|
||||||
Crypto.decryptCipher(rekey, Crypto.DIR_C2S),
|
Crypto.decryptCipher(rekey, Crypto.DIR_C2S),
|
||||||
Crypto.encryptCipher(rekey, Crypto.DIR_S2C));
|
Crypto.encryptCipher(rekey, Crypto.DIR_S2C));
|
||||||
// Echo the accepted flags plus our own receive window.
|
// Echo the accepted flags plus our own receive window. When resumption is
|
||||||
frames.send(new ProtoWriter()
|
// accepted, our grace period follows: the client clamps its own retry
|
||||||
|
// budget under it, which turns a cross-config invariant ("the hub must
|
||||||
|
// wait longer than the client retries") into a negotiated one that
|
||||||
|
// operator skew cannot break.
|
||||||
|
int accepted = Protocol.FLAG_STREAM_FC
|
||||||
|
| (heartbeat ? Protocol.FLAG_WORKER_HEARTBEAT : 0)
|
||||||
|
| (resume ? Protocol.FLAG_STREAM_RESUME : 0);
|
||||||
|
ProtoWriter ready = new ProtoWriter()
|
||||||
.u8(Protocol.CTL_SESSION_READY)
|
.u8(Protocol.CTL_SESSION_READY)
|
||||||
.varInt(Protocol.FLAG_STREAM_FC)
|
.varInt(accepted)
|
||||||
.varInt(hub.config.streamWindowBytes())
|
.varInt(hub.config.streamWindowBytes());
|
||||||
.toBytes());
|
if (resume) ready.varInt((int) hub.config.resumeGraceMs());
|
||||||
|
frames.send(ready.toBytes());
|
||||||
|
|
||||||
if (magic == Protocol.MAGIC_CONTROL) {
|
if (magic == Protocol.MAGIC_CONTROL) {
|
||||||
ControlSession session = new ControlSession(hub, frames, id);
|
ControlSession session = new ControlSession(hub, frames, id);
|
||||||
frames.setHandler(session::onFrame);
|
frames.setHandler(session::onFrame);
|
||||||
closeCleanup = session::onClose;
|
closeCleanup = session::onClose;
|
||||||
LOG.info("{} control session established", id);
|
LOG.info("{} control session established (heartbeat {})", id, heartbeat);
|
||||||
} else if (magic == Protocol.MAGIC_WORKER) {
|
} else if (magic == Protocol.MAGIC_WORKER) {
|
||||||
WorkerConn worker = new WorkerConn(hub, frames, id, peerWindow, hub.config.streamWindowBytes());
|
WorkerConn worker = new WorkerConn(hub, frames, id, peerWindow, hub.config.streamWindowBytes(), resume);
|
||||||
frames.setHandler(worker::onFrame);
|
frames.setHandler(worker::onFrame);
|
||||||
closeCleanup = worker::onClose;
|
closeCleanup = worker::onClose;
|
||||||
LOG.info("{} worker conn established (peer window {})", id, peerWindow);
|
LOG.info("{} worker conn established (peer window {}, heartbeat {}, resume {})",
|
||||||
|
id, peerWindow, heartbeat, resume);
|
||||||
} else {
|
} else {
|
||||||
LOG.warn("{} bad magic {}; closing", id, magic);
|
LOG.warn("{} bad magic {}; closing", id, magic);
|
||||||
frames.close();
|
frames.close();
|
||||||
|
return;
|
||||||
}
|
}
|
||||||
|
armIdleWatchdog();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Drop an established redapricot session that has gone silent. Clients ping
|
||||||
|
* both their control session and every worker conn, so silence means the
|
||||||
|
* path is dead — without this the hub would keep a zombie control session
|
||||||
|
* registered and keep routing players into it, and zombie worker conns would
|
||||||
|
* hold player sockets open forever. Player connections are never subject to
|
||||||
|
* this; only authenticated sessions are.
|
||||||
|
*/
|
||||||
|
private void armIdleWatchdog() {
|
||||||
|
long idleMs = hub.config.sessionIdleTimeoutMs();
|
||||||
|
if (idleMs <= 0) return;
|
||||||
|
long timer = hub.vertx.setPeriodic(idleMs / 2, tid -> {
|
||||||
|
if (frames.isClosed()) {
|
||||||
|
hub.vertx.cancelTimer(tid);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
long silent = System.currentTimeMillis() - frames.lastFrameAt();
|
||||||
|
if (silent > idleMs) {
|
||||||
|
LOG.warn("{} session silent for {}ms; closing", id, silent);
|
||||||
|
hub.vertx.cancelTimer(tid);
|
||||||
|
frames.close();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
Runnable inner = closeCleanup;
|
||||||
|
closeCleanup = () -> {
|
||||||
|
hub.vertx.cancelTimer(timer);
|
||||||
|
inner.run();
|
||||||
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- player connection ----
|
// ---- player connection ----
|
||||||
|
|
||||||
private void handlePlayer(String address) {
|
private void handlePlayer(String address) {
|
||||||
|
// Before match / CID / pause: unmatched hostnames still consume a token,
|
||||||
|
// otherwise a hostname scan is a free flood. Intent 17 never reaches
|
||||||
|
// this method (PROTOCOL.md §9.1).
|
||||||
|
String ip = socket.remoteAddress() != null ? socket.remoteAddress().host() : "0.0.0.0";
|
||||||
|
if (hub.admitPlayer(ip) != null) {
|
||||||
|
socket.close();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// Release on every close of this socket: pending timeout, unmatched
|
||||||
|
// host, player FIN, park eviction. Later cleanups wrap this, they
|
||||||
|
// must not replace it.
|
||||||
|
closeCleanup = () -> hub.releasePlayer(ip);
|
||||||
|
|
||||||
String host = Hub.normalizeAddress(address);
|
String host = Hub.normalizeAddress(address);
|
||||||
Hub.Match matched = hub.match(address);
|
Hub.Match matched = hub.match(address);
|
||||||
if (matched == null) {
|
if (matched == null) {
|
||||||
@@ -217,18 +306,29 @@ public final class HubConnection {
|
|||||||
String pattern = matched.pattern();
|
String pattern = matched.pattern();
|
||||||
byte[] cid = hub.newCid();
|
byte[] cid = hub.newCid();
|
||||||
String cidHex = Hex.encode(cid);
|
String cidHex = Hex.encode(cid);
|
||||||
String ip = socket.remoteAddress() != null ? socket.remoteAddress().host() : "0.0.0.0";
|
|
||||||
int port = socket.remoteAddress() != null ? socket.remoteAddress().port() : 0;
|
int port = socket.remoteAddress() != null ? socket.remoteAddress().port() : 0;
|
||||||
|
|
||||||
socket.pause();
|
socket.pause();
|
||||||
Buffer buffered = hs.copy(); // handshake + any pipelined bytes, forwarded verbatim
|
Buffer buffered = hs.copy(); // handshake + any pipelined bytes, forwarded verbatim
|
||||||
|
|
||||||
PendingPlayer p = new PendingPlayer(cid, cidHex, socket, buffered, pattern, ip, port, session);
|
PendingPlayer p = new PendingPlayer(cid, cidHex, socket, buffered, pattern, ip, port);
|
||||||
hub.addPending(p);
|
closeCleanup = () -> {
|
||||||
closeCleanup = () -> hub.removePending(cidHex);
|
hub.removePending(cidHex);
|
||||||
|
hub.releasePlayer(ip);
|
||||||
|
};
|
||||||
|
|
||||||
session.sendControlRequest(cid, pattern, ip, port);
|
if (session == null) {
|
||||||
LOG.info("{} player {}:{} host '{}' matched pattern '{}' cid={}",
|
// The route is orphaned: its client's control session has closed and
|
||||||
id, ip, port, host, pattern, cidHex);
|
// has not come back yet. Hold the player rather than telling it there
|
||||||
|
// is no such server — the request is replayed the moment a client
|
||||||
|
// re-registers the pattern.
|
||||||
|
hub.addAwaiting(p, matched.orphanDeadline());
|
||||||
|
} else {
|
||||||
|
p.setOwner(session);
|
||||||
|
hub.addPending(p);
|
||||||
|
session.sendControlRequest(cid, pattern, ip, port);
|
||||||
|
}
|
||||||
|
LOG.info("{} player {}:{} host '{}' matched pattern '{}' cid={}{}",
|
||||||
|
id, ip, port, host, pattern, cidHex, session == null ? " (held: route orphaned)" : "");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -24,6 +24,9 @@ public final class HubServer extends AbstractVerticle {
|
|||||||
.setHost(config.host())
|
.setHost(config.host())
|
||||||
.setPort(config.port())
|
.setPort(config.port())
|
||||||
.setTcpNoDelay(true)
|
.setTcpNoDelay(true)
|
||||||
|
// Probe idle sockets so a peer that becomes unreachable is
|
||||||
|
// eventually detected even when no frames are in flight.
|
||||||
|
.setTcpKeepAlive(true)
|
||||||
.setReuseAddress(true);
|
.setReuseAddress(true);
|
||||||
|
|
||||||
NetServer server = vertx.createNetServer(opts);
|
NetServer server = vertx.createNetServer(opts);
|
||||||
@@ -32,10 +35,27 @@ public final class HubServer extends AbstractVerticle {
|
|||||||
if (ar.succeeded()) {
|
if (ar.succeeded()) {
|
||||||
LOG.info("redapricot hub listening on {}:{}", config.host(), ar.result().actualPort());
|
LOG.info("redapricot hub listening on {}:{}", config.host(), ar.result().actualPort());
|
||||||
LOG.info("PSK handshake address: {}", hub.pskAddress);
|
LOG.info("PSK handshake address: {}", hub.pskAddress);
|
||||||
|
armStats();
|
||||||
startPromise.complete();
|
startPromise.complete();
|
||||||
} else {
|
} else {
|
||||||
startPromise.fail(ar.cause());
|
startPromise.fail(ar.cause());
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Periodic one-line snapshot of what the hub is holding. Off unless
|
||||||
|
* statsIntervalMs is set, so it costs nothing by default.
|
||||||
|
*
|
||||||
|
* <p>Parked streams and the bytes they retain are the numbers worth watching:
|
||||||
|
* they are the memory stream resumption trades for keeping players connected,
|
||||||
|
* and the first place a resumption problem shows up as a trend.
|
||||||
|
*/
|
||||||
|
private void armStats() {
|
||||||
|
long interval = config.statsIntervalMs();
|
||||||
|
if (interval <= 0) return;
|
||||||
|
vertx.setPeriodic(interval, id -> LOG.info(
|
||||||
|
"stats streams={} parked={} parkedBytes={} patterns={}",
|
||||||
|
hub.streamCount(), hub.parkedCount(), hub.parkedBytes(), hub.patternCount()));
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,134 @@
|
|||||||
|
package io.icybear.redapricot;
|
||||||
|
|
||||||
|
import org.apache.logging.log4j.LogManager;
|
||||||
|
import org.apache.logging.log4j.Logger;
|
||||||
|
|
||||||
|
import java.util.Iterator;
|
||||||
|
import java.util.LinkedHashMap;
|
||||||
|
import java.util.Map;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Per-IP admission for <em>player</em> connections only (PROTOCOL.md §9.1).
|
||||||
|
*
|
||||||
|
* <p>Intent 17 (control session + every worker conn) is never admitted through
|
||||||
|
* here. Those sockets all come from the client's one address; limiting them
|
||||||
|
* would be the hub throttling its own client. e2e shares {@code 127.0.0.1}
|
||||||
|
* between players and the client for the same reason.
|
||||||
|
*
|
||||||
|
* <p>Event-loop confined: no locking. A {@code 0} rate or concurrent cap turns
|
||||||
|
* that mechanism off; both {@code 0} makes {@link #admit} a no-op.
|
||||||
|
*/
|
||||||
|
public final class IpRateLimiter {
|
||||||
|
private static final Logger LOG = LogManager.getLogger("redapricot.limit");
|
||||||
|
|
||||||
|
/** Idle buckets older than this are dropped so a one-shot flood cannot leak. */
|
||||||
|
static final long SWEEP_MS = 60_000;
|
||||||
|
private static final long DENY_LOG_INTERVAL_MS = 2_000;
|
||||||
|
|
||||||
|
public enum Deny { RATE, CONCURRENT }
|
||||||
|
|
||||||
|
private final double ratePerSec; // 0 = token bucket off
|
||||||
|
private final double burst;
|
||||||
|
private final int maxConcurrent; // 0 = concurrent cap off
|
||||||
|
private final Map<String, Bucket> buckets = new LinkedHashMap<>();
|
||||||
|
|
||||||
|
static final class Bucket {
|
||||||
|
double tokens;
|
||||||
|
long lastRefillMs;
|
||||||
|
int concurrent;
|
||||||
|
long lastActivityMs;
|
||||||
|
long lastDenyLogMs;
|
||||||
|
int deniesSinceLog;
|
||||||
|
}
|
||||||
|
|
||||||
|
public IpRateLimiter(double ratePerSec, double burst, int maxConcurrent) {
|
||||||
|
this.ratePerSec = Math.max(0, ratePerSec);
|
||||||
|
this.burst = Math.max(1, burst);
|
||||||
|
this.maxConcurrent = Math.max(0, maxConcurrent);
|
||||||
|
}
|
||||||
|
|
||||||
|
public boolean enabled() {
|
||||||
|
return ratePerSec > 0 || maxConcurrent > 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Consume one player admission for {@code ip}. {@code null} means allowed
|
||||||
|
* and the caller <em>must</em> {@link #release} when the socket closes.
|
||||||
|
*/
|
||||||
|
public Deny admit(String ip, long nowMs) {
|
||||||
|
if (!enabled()) return null;
|
||||||
|
Bucket b = bucket(ip, nowMs);
|
||||||
|
b.lastActivityMs = nowMs;
|
||||||
|
refill(b, nowMs);
|
||||||
|
if (ratePerSec > 0 && b.tokens < 1.0) {
|
||||||
|
noteDeny(ip, b, nowMs, Deny.RATE);
|
||||||
|
return Deny.RATE;
|
||||||
|
}
|
||||||
|
if (maxConcurrent > 0 && b.concurrent >= maxConcurrent) {
|
||||||
|
noteDeny(ip, b, nowMs, Deny.CONCURRENT);
|
||||||
|
return Deny.CONCURRENT;
|
||||||
|
}
|
||||||
|
if (ratePerSec > 0) b.tokens -= 1.0;
|
||||||
|
b.concurrent++;
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
public void release(String ip, long nowMs) {
|
||||||
|
Bucket b = buckets.get(ip);
|
||||||
|
if (b == null) return;
|
||||||
|
if (b.concurrent > 0) b.concurrent--;
|
||||||
|
b.lastActivityMs = nowMs;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Drop idle empty buckets. Safe to call on a timer. */
|
||||||
|
public void sweep(long nowMs) {
|
||||||
|
Iterator<Map.Entry<String, Bucket>> it = buckets.entrySet().iterator();
|
||||||
|
while (it.hasNext()) {
|
||||||
|
Bucket b = it.next().getValue();
|
||||||
|
if (b.concurrent == 0 && nowMs - b.lastActivityMs >= SWEEP_MS) {
|
||||||
|
it.remove();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Visible for tests. */
|
||||||
|
int bucketCount() {
|
||||||
|
return buckets.size();
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Visible for tests. */
|
||||||
|
int concurrent(String ip) {
|
||||||
|
Bucket b = buckets.get(ip);
|
||||||
|
return b == null ? 0 : b.concurrent;
|
||||||
|
}
|
||||||
|
|
||||||
|
private Bucket bucket(String ip, long nowMs) {
|
||||||
|
Bucket b = buckets.get(ip);
|
||||||
|
if (b != null) return b;
|
||||||
|
b = new Bucket();
|
||||||
|
b.tokens = burst;
|
||||||
|
b.lastRefillMs = nowMs;
|
||||||
|
b.lastActivityMs = nowMs;
|
||||||
|
buckets.put(ip, b);
|
||||||
|
return b;
|
||||||
|
}
|
||||||
|
|
||||||
|
private void refill(Bucket b, long nowMs) {
|
||||||
|
if (ratePerSec <= 0) return;
|
||||||
|
double elapsed = (nowMs - b.lastRefillMs) / 1000.0;
|
||||||
|
if (elapsed <= 0) return;
|
||||||
|
b.tokens = Math.min(burst, b.tokens + elapsed * ratePerSec);
|
||||||
|
b.lastRefillMs = nowMs;
|
||||||
|
}
|
||||||
|
|
||||||
|
private void noteDeny(String ip, Bucket b, long nowMs, Deny why) {
|
||||||
|
b.deniesSinceLog++;
|
||||||
|
if (b.lastDenyLogMs != 0 && nowMs - b.lastDenyLogMs < DENY_LOG_INTERVAL_MS) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
LOG.warn("dropping player from {}: {} ({} similar since last log)",
|
||||||
|
ip, why == Deny.RATE ? "rate" : "maxPlayersPerIp", b.deniesSinceLog);
|
||||||
|
b.lastDenyLogMs = nowMs;
|
||||||
|
b.deniesSinceLog = 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -17,7 +17,22 @@ public final class PendingPlayer {
|
|||||||
private final String pattern;
|
private final String pattern;
|
||||||
private final String playerIp;
|
private final String playerIp;
|
||||||
private final int playerPort;
|
private final int playerPort;
|
||||||
private final ControlSession owner; // control session this player was routed to
|
|
||||||
|
/**
|
||||||
|
* Control session this player was routed to, or null while the route is
|
||||||
|
* orphaned. Mutable because a hung player is rebound to whichever session
|
||||||
|
* re-registers its pattern.
|
||||||
|
*/
|
||||||
|
@Setter
|
||||||
|
private ControlSession owner;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Hung waiting for a control session to come back, rather than waiting for a
|
||||||
|
* worker to claim it. No ControlRequest has been delivered yet, so this
|
||||||
|
* player is the hub's to replay once a route reappears.
|
||||||
|
*/
|
||||||
|
@Setter
|
||||||
|
private boolean awaitingSession;
|
||||||
|
|
||||||
@Setter
|
@Setter
|
||||||
private long timerId = -1;
|
private long timerId = -1;
|
||||||
|
|||||||
@@ -0,0 +1,86 @@
|
|||||||
|
package io.icybear.redapricot;
|
||||||
|
|
||||||
|
import io.vertx.core.buffer.Buffer;
|
||||||
|
import io.vertx.core.net.NetSocket;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* One tunneled player: the player socket plus the flow-control state of the
|
||||||
|
* worker conn carrying it (PROTOCOL.md §7.3).
|
||||||
|
*
|
||||||
|
* <p>This is deliberately <em>not</em> owned by {@link WorkerConn}. A tunnel's
|
||||||
|
* identity is the player, not the connection it happens to ride: the worker conn
|
||||||
|
* is a replaceable transport, and state that dies with it cannot be recovered
|
||||||
|
* when it drops.
|
||||||
|
*
|
||||||
|
* <p>Confined to the hub's single event loop, so the mutable fields need no
|
||||||
|
* synchronization.
|
||||||
|
*/
|
||||||
|
public final class PlayerStream {
|
||||||
|
/** Capability that authorized the takeover; also the resume key. Re-minted on each reattach. */
|
||||||
|
byte[] cid;
|
||||||
|
String cidHex;
|
||||||
|
final NetSocket player;
|
||||||
|
/** Registered pattern that matched, echoed to the client. */
|
||||||
|
final String pattern;
|
||||||
|
final String playerIp;
|
||||||
|
final int playerPort;
|
||||||
|
|
||||||
|
/** The conn currently carrying this player. */
|
||||||
|
WorkerConn worker;
|
||||||
|
|
||||||
|
/** Budget for player -> client DATA. */
|
||||||
|
int sendWnd;
|
||||||
|
/** Player bytes awaiting send window; the player is paused while non-null. */
|
||||||
|
Buffer pendingUp;
|
||||||
|
/** client -> player bytes flushed but not yet granted back. */
|
||||||
|
int credited;
|
||||||
|
|
||||||
|
// Pause reasons. Vert.x pause() is a flag rather than a counter, so a socket
|
||||||
|
// can be paused for several reasons at once and must only be resumed once
|
||||||
|
// none of them hold — see WorkerConn#maybeResumePlayer.
|
||||||
|
boolean pausedForWindow; // this tunnel's send window is exhausted
|
||||||
|
boolean pausedForAggregate; // the worker socket's write queue is full
|
||||||
|
boolean parked; // the worker conn died; hanging until a reattach
|
||||||
|
|
||||||
|
/** Whether the conn carrying this player negotiated resumption (§7.5). */
|
||||||
|
boolean resumable;
|
||||||
|
|
||||||
|
// Resumption bookkeeping (§7.5). Three distinct offsets, and conflating them
|
||||||
|
// is the classic mistake: what to retransmit is measured from what the peer
|
||||||
|
// *accepted*, while the flow-control window is measured from what it
|
||||||
|
// *credited*. The gap between the two is credit still owed.
|
||||||
|
long sentOffset; // bytes handed to the wire
|
||||||
|
long ackedOffset; // running sum of WND deltas received
|
||||||
|
long acceptedOffset; // client -> player bytes taken off the wire
|
||||||
|
long deliveredOffset; // client -> player bytes actually written to the socket
|
||||||
|
final UnackedBytes unacked = new UnackedBytes();
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Absolute wall-clock deadline for reattaching, fixed at the first park. Not
|
||||||
|
* re-armed on a later park: a flapping hub would otherwise keep extending it
|
||||||
|
* and hang the player indefinitely.
|
||||||
|
*/
|
||||||
|
long graceDeadline;
|
||||||
|
long timerId = -1;
|
||||||
|
|
||||||
|
PlayerStream(PendingPlayer p, WorkerConn worker, int sendWnd) {
|
||||||
|
this.cid = p.getCid();
|
||||||
|
this.cidHex = p.getCidHex();
|
||||||
|
this.player = p.getSocket();
|
||||||
|
this.pattern = p.getPattern();
|
||||||
|
this.playerIp = p.getPlayerIp();
|
||||||
|
this.playerPort = p.getPlayerPort();
|
||||||
|
this.worker = worker;
|
||||||
|
this.sendWnd = sendWnd;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Whether the player socket should be flowing right now. */
|
||||||
|
boolean shouldFlow() {
|
||||||
|
return !pausedForWindow && !pausedForAggregate && !parked;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Roughly how much this tunnel holds while parked, for the hub-wide cap. */
|
||||||
|
int parkedBytes() {
|
||||||
|
return unacked.length() + (pendingUp != null ? pendingUp.length() : 0);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -26,18 +26,44 @@ public final class Protocol {
|
|||||||
public static final int REGISTER_OK = 0x00;
|
public static final int REGISTER_OK = 0x00;
|
||||||
public static final int REGISTER_ERR_PATTERN = 0x01; // pattern is not a valid regular expression
|
public static final int REGISTER_ERR_PATTERN = 0x01; // pattern is not a valid regular expression
|
||||||
|
|
||||||
// Worker-conn mux frame types
|
// Worker-conn frame types (one player per conn; no stream id).
|
||||||
public static final int MUX_SYN = 0x00;
|
public static final int MUX_SYN = 0x00;
|
||||||
public static final int MUX_DATA = 0x01;
|
public static final int MUX_DATA = 0x01;
|
||||||
public static final int MUX_FIN = 0x02;
|
public static final int MUX_FIN = 0x02;
|
||||||
public static final int MUX_RST = 0x03;
|
public static final int MUX_RST = 0x03;
|
||||||
public static final int MUX_WND = 0x04; // per-stream flow-control credit grant
|
public static final int MUX_WND = 0x04; // per-connection flow-control credit grant
|
||||||
|
public static final int MUX_PING = 0x05; // liveness probe
|
||||||
|
public static final int MUX_PONG = 0x06; // liveness reply, echoes the nonce
|
||||||
|
/** Reattach a parked player to this conn: CID + the client's accepted offset (§7.5). */
|
||||||
|
public static final int MUX_RESUME = 0x07;
|
||||||
|
/** Hub's answer to RESUME: its accepted offset plus a freshly minted CID. */
|
||||||
|
public static final int MUX_RESUME_ACK = 0x08;
|
||||||
|
|
||||||
|
// RST reason codes (optional trailing byte; absence means "unspecified").
|
||||||
|
// Distinguishing them matters for resume: "unknown stream" is terminal,
|
||||||
|
// "already bound" means a racing attempt won and this one must not tear down.
|
||||||
|
public static final int RST_UNSPECIFIED = 0x00;
|
||||||
|
public static final int RST_UNKNOWN_STREAM = 0x01; // CID unknown, expired, or hub restarted
|
||||||
|
public static final int RST_ALREADY_BOUND = 0x02; // another RESUME won the race
|
||||||
|
public static final int RST_RESUME_ABANDONED = 0x03;
|
||||||
|
public static final int RST_FLOW_CONTROL = 0x04;
|
||||||
|
public static final int RST_DIAL_FAILED = 0x05;
|
||||||
|
|
||||||
// Session-establishment feature flags (trailing VarInt on the Rekey message,
|
// Session-establishment feature flags (trailing VarInt on the Rekey message,
|
||||||
// echoed after the SessionReady type byte when accepted).
|
// echoed after the SessionReady type byte when accepted).
|
||||||
public static final int FLAG_STREAM_FC = 0x01;
|
public static final int FLAG_STREAM_FC = 0x01;
|
||||||
|
/** Connection-level PING/PONG on worker conns, so a silently dropped path is detected. */
|
||||||
|
public static final int FLAG_WORKER_HEARTBEAT = 0x02;
|
||||||
|
/**
|
||||||
|
* Stream resumption (§7.5): on a worker-conn drop the hub hangs the player
|
||||||
|
* socket instead of closing it, and the client reattaches that player
|
||||||
|
* byte-exactly over a fresh conn. When accepted, the hub appends its resume
|
||||||
|
* grace period to SessionReady so the client can bound its own retry budget
|
||||||
|
* against it.
|
||||||
|
*/
|
||||||
|
public static final int FLAG_STREAM_RESUME = 0x04;
|
||||||
|
|
||||||
// Per-stream flow-control window bounds (bytes).
|
// Per-connection flow-control window bounds (bytes).
|
||||||
public static final int DEFAULT_STREAM_WINDOW = 256 * 1024;
|
public static final int DEFAULT_STREAM_WINDOW = 256 * 1024;
|
||||||
public static final int MIN_STREAM_WINDOW = 32 * 1024;
|
public static final int MIN_STREAM_WINDOW = 32 * 1024;
|
||||||
public static final int MAX_STREAM_WINDOW = 8 << 20;
|
public static final int MAX_STREAM_WINDOW = 8 << 20;
|
||||||
|
|||||||
@@ -0,0 +1,95 @@
|
|||||||
|
package io.icybear.redapricot;
|
||||||
|
|
||||||
|
import io.vertx.core.buffer.Buffer;
|
||||||
|
|
||||||
|
import java.util.ArrayDeque;
|
||||||
|
import java.util.Deque;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The bytes a stream has sent but the client has not yet credited — exactly the
|
||||||
|
* region a reattach may have to retransmit (PROTOCOL.md §7.5).
|
||||||
|
*
|
||||||
|
* <p>It needs no cap of its own: credit is only granted as bytes reach the
|
||||||
|
* client's destination socket, so flow control already bounds the outstanding
|
||||||
|
* region to one window. That is what makes byte-exact resumption affordable.
|
||||||
|
*
|
||||||
|
* <p>A deque of the chunks already materialized by the send path, rather than one
|
||||||
|
* growing {@link Buffer}: appending to a Buffer reallocates and recopies as it
|
||||||
|
* grows, which would add a second per-byte copy to the whole upstream path. Here
|
||||||
|
* retention is free — the chunk was allocated to be sent anyway.
|
||||||
|
*/
|
||||||
|
final class UnackedBytes {
|
||||||
|
private final Deque<byte[]> chunks = new ArrayDeque<>();
|
||||||
|
private int head; // bytes of the first chunk already credited
|
||||||
|
private long base; // stream offset of the first live byte
|
||||||
|
private int length; // live bytes across all chunks
|
||||||
|
|
||||||
|
int length() {
|
||||||
|
return length;
|
||||||
|
}
|
||||||
|
|
||||||
|
long base() {
|
||||||
|
return base;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Offset one past the last byte handed to the wire. */
|
||||||
|
long end() {
|
||||||
|
return base + length;
|
||||||
|
}
|
||||||
|
|
||||||
|
void append(byte[] chunk) {
|
||||||
|
if (chunk.length == 0) return;
|
||||||
|
chunks.addLast(chunk);
|
||||||
|
length += chunk.length;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Drop everything the client has credited up to {@code off}. */
|
||||||
|
void advance(long off) {
|
||||||
|
long drop = off - base;
|
||||||
|
if (drop <= 0) return;
|
||||||
|
if (drop > length) drop = length; // only from a peer crediting bytes never sent
|
||||||
|
while (drop > 0) {
|
||||||
|
byte[] first = chunks.peekFirst();
|
||||||
|
int avail = first.length - head;
|
||||||
|
int take = (int) Math.min(drop, avail);
|
||||||
|
head += take;
|
||||||
|
base += take;
|
||||||
|
length -= take;
|
||||||
|
drop -= take;
|
||||||
|
if (head == first.length) {
|
||||||
|
chunks.removeFirst();
|
||||||
|
head = 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The outstanding bytes at and after {@code off}, or {@code null} when
|
||||||
|
* {@code off} falls outside what is still held — which means the peer named
|
||||||
|
* an offset we can no longer satisfy and the stream cannot be resumed.
|
||||||
|
*/
|
||||||
|
Buffer from(long off) {
|
||||||
|
long skip = off - base;
|
||||||
|
if (skip < 0 || skip > length) return null;
|
||||||
|
Buffer out = Buffer.buffer((int) (length - skip));
|
||||||
|
int start = head;
|
||||||
|
for (byte[] chunk : chunks) {
|
||||||
|
int avail = chunk.length - start;
|
||||||
|
if (skip >= avail) {
|
||||||
|
skip -= avail;
|
||||||
|
start = 0;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
out.appendBytes(chunk, start + (int) skip, avail - (int) skip);
|
||||||
|
skip = 0;
|
||||||
|
start = 0;
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
void clear() {
|
||||||
|
chunks.clear();
|
||||||
|
head = 0;
|
||||||
|
length = 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -4,109 +4,116 @@ import io.icybear.redapricot.net.EncryptedFrames;
|
|||||||
import io.icybear.redapricot.util.ProtoReader;
|
import io.icybear.redapricot.util.ProtoReader;
|
||||||
import io.icybear.redapricot.util.ProtoWriter;
|
import io.icybear.redapricot.util.ProtoWriter;
|
||||||
import io.vertx.core.buffer.Buffer;
|
import io.vertx.core.buffer.Buffer;
|
||||||
import io.vertx.core.net.NetSocket;
|
|
||||||
import lombok.RequiredArgsConstructor;
|
import lombok.RequiredArgsConstructor;
|
||||||
import org.apache.logging.log4j.LogManager;
|
import org.apache.logging.log4j.LogManager;
|
||||||
import org.apache.logging.log4j.Logger;
|
import org.apache.logging.log4j.Logger;
|
||||||
|
|
||||||
import java.util.HashMap;
|
|
||||||
import java.util.HashSet;
|
|
||||||
import java.util.Map;
|
|
||||||
import java.util.Set;
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* An authenticated worker connection (Magic 0x02). Multiplexes many player
|
* An authenticated worker connection (Magic 0x02). Carries exactly one player:
|
||||||
* streams; the client opens streams via SYN(CID) to take over pending players.
|
* the TCP connection is the tunnel (PROTOCOL.md §7). The client binds it with
|
||||||
|
* SYN(CID) (or RESUME) after SessionReady.
|
||||||
*
|
*
|
||||||
* <p>Every stream has a credit window in both directions (PROTOCOL.md §7.3),
|
* <p>The connection has a credit window in both directions (PROTOCOL.md §7.3),
|
||||||
* so one slow player only ever stalls its own stream — the shared worker
|
* so a slow player only ever stalls itself.
|
||||||
* socket is never paused because of a single stream.
|
|
||||||
*/
|
*/
|
||||||
@RequiredArgsConstructor
|
@RequiredArgsConstructor
|
||||||
public final class WorkerConn {
|
public final class WorkerConn {
|
||||||
private static final Logger LOG = LogManager.getLogger("redapricot.worker");
|
private static final Logger LOG = LogManager.getLogger("redapricot.worker");
|
||||||
|
|
||||||
/** Cap on a single DATA frame so no stream monopolizes the shared link for long. */
|
/** Cap on a single DATA frame so one write cannot occupy the link for long. */
|
||||||
private static final int CHUNK = 32 * 1024;
|
private static final int CHUNK = 32 * 1024;
|
||||||
|
|
||||||
private final Hub hub;
|
private final Hub hub;
|
||||||
private final EncryptedFrames frames;
|
private final EncryptedFrames frames;
|
||||||
private final String id;
|
private final String id;
|
||||||
private final int sendWndInit; // client's advertised per-stream receive window (our send budget)
|
private final int sendWndInit; // client's advertised receive window (our send budget)
|
||||||
private final int recvWndInit; // our advertised per-stream receive window (basis for credit grants)
|
private final int recvWndInit; // our advertised receive window (basis for credit grants)
|
||||||
|
/**
|
||||||
|
* Whether this conn negotiated stream resumption. Gates the park path: a
|
||||||
|
* client that will never reattach is better served by an immediate close than
|
||||||
|
* by a player left hanging for the whole grace period.
|
||||||
|
*/
|
||||||
|
private final boolean resume;
|
||||||
|
|
||||||
private final Map<Integer, StreamState> streams = new HashMap<>();
|
/** The one player bound to this conn, or null until SYN/RESUME. */
|
||||||
|
private PlayerStream stream;
|
||||||
|
|
||||||
// Aggregate backpressure for the single shared worker socket: players parked
|
private boolean workerDrainArmed = false; // whether the worker socket's drainHandler is set
|
||||||
// until its write queue drains. Per-stream fairness is the credit windows'
|
|
||||||
// job; this only reacts to the whole pipe being congested.
|
|
||||||
private final Set<StreamState> upstreamPaused = new HashSet<>();
|
|
||||||
private boolean workerDrainArmed = false; // whether the worker socket's single drainHandler is set
|
|
||||||
|
|
||||||
/** Per-stream flow-control bookkeeping. */
|
|
||||||
private final class StreamState {
|
|
||||||
final NetSocket player;
|
|
||||||
int sendWnd = sendWndInit; // budget for player -> client DATA
|
|
||||||
Buffer pendingUp; // player bytes awaiting send window (player is paused meanwhile)
|
|
||||||
boolean pausedForWindow;
|
|
||||||
int credited; // client -> player bytes flushed but not yet granted back
|
|
||||||
|
|
||||||
StreamState(NetSocket player) {
|
|
||||||
this.player = player;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
public void onFrame(byte[] payload) {
|
public void onFrame(byte[] payload) {
|
||||||
ProtoReader r = new ProtoReader(payload);
|
ProtoReader r = new ProtoReader(payload);
|
||||||
int type = r.readUByte();
|
int type = r.readUByte();
|
||||||
int sid = r.readVarInt();
|
|
||||||
switch (type) {
|
switch (type) {
|
||||||
case Protocol.MUX_SYN -> handleSyn(sid, r.readBytes(Protocol.CID_LEN));
|
case Protocol.MUX_SYN -> handleSyn(r.readBytes(Protocol.CID_LEN));
|
||||||
case Protocol.MUX_DATA -> handleData(sid, r.readBuffer(r.remaining()));
|
case Protocol.MUX_DATA -> handleData(r.readBuffer(r.remaining()));
|
||||||
case Protocol.MUX_WND -> handleWnd(sid, r.readVarInt());
|
case Protocol.MUX_WND -> handleWnd(r.readVarInt());
|
||||||
case Protocol.MUX_FIN, Protocol.MUX_RST -> closeStream(sid);
|
case Protocol.MUX_RESUME ->
|
||||||
|
handleResume(r.readBytes(Protocol.CID_LEN), r.readI64(), r.readI64());
|
||||||
|
case Protocol.MUX_FIN, Protocol.MUX_RST -> closeBound();
|
||||||
|
case Protocol.MUX_PING -> sendPong(r.readI64());
|
||||||
|
case Protocol.MUX_PONG -> { /* liveness only; arrival is what matters */ }
|
||||||
case Protocol.FRAME_ERROR -> LOG.warn("worker {} error frame", id);
|
case Protocol.FRAME_ERROR -> LOG.warn("worker {} error frame", id);
|
||||||
default -> LOG.warn("worker {} unknown mux type {}", id, type);
|
default -> LOG.warn("worker {} unknown frame type {}", id, type);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private void handleSyn(int sid, byte[] cid) {
|
private void handleSyn(byte[] cid) {
|
||||||
|
if (stream != null) {
|
||||||
|
LOG.warn("worker {} SYN on an already-bound conn; closing", id);
|
||||||
|
sendRst(Protocol.RST_ALREADY_BOUND);
|
||||||
|
frames.close();
|
||||||
|
return;
|
||||||
|
}
|
||||||
PendingPlayer p = hub.takePending(cid);
|
PendingPlayer p = hub.takePending(cid);
|
||||||
if (p == null) {
|
if (p == null) {
|
||||||
LOG.warn("worker {} SYN for unknown CID", id);
|
LOG.warn("worker {} SYN for unknown CID", id);
|
||||||
sendRst(sid);
|
sendRst(Protocol.RST_UNKNOWN_STREAM);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
NetSocket player = p.getSocket();
|
PlayerStream st = new PlayerStream(p, this, sendWndInit);
|
||||||
StreamState st = new StreamState(player);
|
st.resumable = resume;
|
||||||
streams.put(sid, st);
|
stream = st;
|
||||||
|
hub.addStream(st);
|
||||||
|
|
||||||
// From now on the player socket belongs to this stream.
|
// From now on the player socket belongs to this tunnel. The handlers are
|
||||||
player.handler(buf -> {
|
// installed once and route through the hub, which dispatches to whichever
|
||||||
sendUpstream(sid, st, buf);
|
// conn currently carries the player.
|
||||||
checkAggregate(st);
|
//
|
||||||
});
|
// They must not call this conn's methods directly: a lambda defined here
|
||||||
player.closeHandler(v -> onPlayerGone(sid, st));
|
// captures `this`, so after the player moves to another conn it would keep
|
||||||
player.exceptionHandler(t -> onPlayerGone(sid, st));
|
// writing into the dead one's transport, where sends are silently dropped
|
||||||
|
// and the player goes mute with nothing logged. Re-installing handlers on
|
||||||
|
// every reattach would be the other option, but a Vert.x socket resumed
|
||||||
|
// while a stale handler is still attached loses bytes, so routing beats
|
||||||
|
// rebinding.
|
||||||
|
st.player.handler(buf -> hub.onPlayerData(st, buf));
|
||||||
|
st.player.closeHandler(v -> hub.onPlayerGone(st));
|
||||||
|
st.player.exceptionHandler(t -> hub.onPlayerGone(st));
|
||||||
|
|
||||||
// Forward the buffered handshake (and any pipelined bytes), then resume.
|
// Forward the buffered handshake (and any pipelined bytes), then resume.
|
||||||
sendUpstream(sid, st, p.getBuffered());
|
sendUpstream(st, p.getBuffered());
|
||||||
if (!st.pausedForWindow) player.resume();
|
maybeResumePlayer(st);
|
||||||
|
checkAggregate(st);
|
||||||
|
LOG.info("worker {} bound to {}", id, st.pattern);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Player bytes arrived on the player this conn currently carries. */
|
||||||
|
void playerData(PlayerStream st, Buffer buf) {
|
||||||
|
sendUpstream(st, buf);
|
||||||
checkAggregate(st);
|
checkAggregate(st);
|
||||||
LOG.info("worker {} stream {} bound to {}", id, sid, p.getPattern());
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Send player bytes to the client, chunked and clipped to the stream window;
|
* Send player bytes to the client, chunked and clipped to the send window;
|
||||||
* the overflow is parked in {@code pendingUp} and the player socket paused
|
* the overflow is parked in {@code pendingUp} and the player socket paused
|
||||||
* until the client grants more credit.
|
* until the client grants more credit.
|
||||||
*/
|
*/
|
||||||
private void sendUpstream(int sid, StreamState st, Buffer buf) {
|
private void sendUpstream(PlayerStream st, Buffer buf) {
|
||||||
if (st.pendingUp != null) { // still waiting for window; keep ordering
|
if (st.pendingUp != null) { // still waiting for window; keep ordering
|
||||||
st.pendingUp.appendBuffer(buf);
|
st.pendingUp.appendBuffer(buf);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
int off = drainUpstream(sid, st, buf, 0);
|
int off = drainUpstream(st, buf, 0);
|
||||||
if (off < buf.length()) {
|
if (off < buf.length()) {
|
||||||
st.pendingUp = buf.getBuffer(off, buf.length());
|
st.pendingUp = buf.getBuffer(off, buf.length());
|
||||||
if (!st.pausedForWindow) {
|
if (!st.pausedForWindow) {
|
||||||
@@ -116,111 +123,263 @@ public final class WorkerConn {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Send from {@code buf[off..]} within the stream window, chunked; returns the new offset. */
|
/** Send from {@code buf[off..]} within the send window, chunked; returns the new offset. */
|
||||||
private int drainUpstream(int sid, StreamState st, Buffer buf, int off) {
|
private int drainUpstream(PlayerStream st, Buffer buf, int off) {
|
||||||
while (off < buf.length() && st.sendWnd > 0) {
|
while (off < buf.length() && st.sendWnd > 0) {
|
||||||
int n = Math.min(Math.min(CHUNK, st.sendWnd), buf.length() - off);
|
int n = Math.min(Math.min(CHUNK, st.sendWnd), buf.length() - off);
|
||||||
sendData(sid, buf.getBytes(off, off + n));
|
byte[] chunk = buf.getBytes(off, off + n);
|
||||||
|
if (st.resumable) {
|
||||||
|
// Retain before sending. A frame written to a dying socket is
|
||||||
|
// lost with no notification, so the only trustworthy record of
|
||||||
|
// what the client still owes us is taken before the attempt.
|
||||||
|
// Retention is free here: the chunk was materialized to be sent.
|
||||||
|
st.unacked.advance(st.ackedOffset);
|
||||||
|
st.unacked.append(chunk);
|
||||||
|
}
|
||||||
|
st.sentOffset += n;
|
||||||
|
sendData(chunk);
|
||||||
st.sendWnd -= n;
|
st.sendWnd -= n;
|
||||||
off += n;
|
off += n;
|
||||||
}
|
}
|
||||||
return off;
|
return off;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** The client granted {@code delta} more bytes of credit on a stream. */
|
/** The client granted {@code delta} more bytes of credit. */
|
||||||
private void handleWnd(int sid, int delta) {
|
private void handleWnd(int delta) {
|
||||||
StreamState st = streams.get(sid);
|
PlayerStream st = stream;
|
||||||
if (st == null || delta <= 0) return;
|
if (st == null || delta <= 0) return;
|
||||||
|
// The running sum doubles as the acked offset: the client grants credit
|
||||||
|
// exactly as bytes reach the destination socket, so a credited byte can
|
||||||
|
// never need retransmitting.
|
||||||
|
st.ackedOffset += delta;
|
||||||
st.sendWnd += delta;
|
st.sendWnd += delta;
|
||||||
if (st.pendingUp != null) {
|
if (st.pendingUp != null) {
|
||||||
Buffer pending = st.pendingUp;
|
Buffer pending = st.pendingUp;
|
||||||
int off = drainUpstream(sid, st, pending, 0);
|
int off = drainUpstream(st, pending, 0);
|
||||||
st.pendingUp = off >= pending.length() ? null : pending.getBuffer(off, pending.length());
|
st.pendingUp = off >= pending.length() ? null : pending.getBuffer(off, pending.length());
|
||||||
}
|
}
|
||||||
if (st.pendingUp == null && st.pausedForWindow) {
|
if (st.pendingUp == null) st.pausedForWindow = false;
|
||||||
st.pausedForWindow = false;
|
maybeResumePlayer(st);
|
||||||
if (!upstreamPaused.contains(st)) st.player.resume();
|
|
||||||
}
|
|
||||||
checkAggregate(st);
|
checkAggregate(st);
|
||||||
}
|
}
|
||||||
|
|
||||||
private void handleData(int sid, Buffer data) {
|
private void handleData(Buffer data) {
|
||||||
StreamState st = streams.get(sid);
|
PlayerStream st = stream;
|
||||||
if (st == null) return;
|
if (st == null) return;
|
||||||
// Never pause the shared socket: the client bounds what it sends per
|
// Never pause the worker socket: the client bounds what it sends to our
|
||||||
// stream to our advertised window, so a slow player only piles up a
|
// advertised window, so a slow player only piles up a bounded amount in
|
||||||
// bounded amount in its own write queue; credit is granted back as the
|
// its own write queue; credit is granted back as the write completes
|
||||||
// write completes (i.e. the bytes reached the player socket).
|
// (i.e. the bytes reached the player socket).
|
||||||
int len = data.length();
|
int len = data.length();
|
||||||
|
// Accepted the moment the bytes are taken off the wire, not when the write
|
||||||
|
// completes. Completion is asynchronous and suppressed once the connection
|
||||||
|
// closes — precisely when a reattach needs this number to be right — so
|
||||||
|
// reporting delivery would under-count and make the client replay bytes
|
||||||
|
// the player already has.
|
||||||
|
st.acceptedOffset += len;
|
||||||
st.player.write(data).onComplete(ar -> {
|
st.player.write(data).onComplete(ar -> {
|
||||||
if (ar.failed() || frames.isClosed() || streams.get(sid) != st) return;
|
if (ar.failed()) return;
|
||||||
|
// Both counters advance even if this conn has since died or the player
|
||||||
|
// has moved on. Discarding them would destroy up to half a window of
|
||||||
|
// credit per outage, and — worse — leave the delivered offset that a
|
||||||
|
// reattach restates the window from permanently short.
|
||||||
|
st.deliveredOffset += len;
|
||||||
st.credited += len;
|
st.credited += len;
|
||||||
|
if (st.worker != this || frames.isClosed()) return;
|
||||||
if (st.credited * 2 >= recvWndInit) {
|
if (st.credited * 2 >= recvWndInit) {
|
||||||
int delta = st.credited;
|
int delta = st.credited;
|
||||||
st.credited = 0;
|
st.credited = 0;
|
||||||
sendWnd(sid, delta);
|
sendWnd(delta);
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
private void closeStream(int sid) {
|
/**
|
||||||
StreamState st = streams.remove(sid);
|
* Reattach a parked player to this connection (§7.5).
|
||||||
|
*
|
||||||
|
* <p>Runs to completion in one event-loop turn — rebind, acknowledge, replay —
|
||||||
|
* so the hub's single-threaded model makes the ordering race-free by
|
||||||
|
* construction, with no interleaving of live and replayed bytes to reason
|
||||||
|
* about.
|
||||||
|
*/
|
||||||
|
private void handleResume(byte[] cid, long clientAccepted, long clientDelivered) {
|
||||||
|
if (stream != null) {
|
||||||
|
LOG.warn("worker {} RESUME on an already-bound conn; closing", id);
|
||||||
|
sendRst(Protocol.RST_ALREADY_BOUND);
|
||||||
|
frames.close();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
PlayerStream st = hub.takeParked(cid);
|
||||||
|
if (st == null) {
|
||||||
|
// Tell a player we have never heard of apart from one that is still
|
||||||
|
// bound: the first is terminal for the client, the second means a
|
||||||
|
// racing attempt won and this one must leave the player alone.
|
||||||
|
boolean bound = hub.streamByCid(cid) != null;
|
||||||
|
LOG.warn("worker {} RESUME for {} CID", id, bound ? "still-bound" : "unknown");
|
||||||
|
sendRst(bound ? Protocol.RST_ALREADY_BOUND : Protocol.RST_UNKNOWN_STREAM);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Delivery is a strictly stronger fact than credit — the client only
|
||||||
|
// credits what it has delivered — so the reported offset can be adopted
|
||||||
|
// wholesale. Doing so also repairs the ledger: the grants destroyed by the
|
||||||
|
// outage are exactly the gap between the two, and without this the
|
||||||
|
// retained region would carry that dead prefix for the stream's whole life.
|
||||||
|
st.ackedOffset = Math.max(st.ackedOffset, clientDelivered);
|
||||||
|
st.unacked.advance(st.ackedOffset);
|
||||||
|
Buffer replay = st.unacked.from(clientAccepted);
|
||||||
|
if (replay == null) {
|
||||||
|
LOG.warn("worker {} RESUME at offset {} outside the retained region [{}, {}]; closing player",
|
||||||
|
id, clientAccepted, st.unacked.base(), st.unacked.end());
|
||||||
|
sendRst(Protocol.RST_UNKNOWN_STREAM);
|
||||||
|
hub.removeStream(st);
|
||||||
|
st.player.close();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
st.worker = this;
|
||||||
|
st.resumable = resume;
|
||||||
|
stream = st;
|
||||||
|
|
||||||
|
// Restate the window rather than patching it. Three offsets, three jobs:
|
||||||
|
// the replay above is measured from what the client *accepted*, the window
|
||||||
|
// from what it *delivered* (the window being a promise about undelivered
|
||||||
|
// bytes), and never from what it *credited* — credit travels as deltas, and
|
||||||
|
// the grants in flight when the connection died are gone for good, so a
|
||||||
|
// window derived from them stays permanently short. When a full window was
|
||||||
|
// outstanding at the drop that means permanently zero, which deadlocks:
|
||||||
|
// nothing can be sent, so no credit can ever come back.
|
||||||
|
int outstanding = st.unacked.length();
|
||||||
|
st.sendWnd = Math.max(0, sendWndInit - outstanding);
|
||||||
|
// Symmetrically, drop our own pending credit instead of flushing it: the
|
||||||
|
// delivered offset in the ack already carries everything those deltas
|
||||||
|
// would have, and sending both would grant the same bytes twice.
|
||||||
|
st.credited = 0;
|
||||||
|
|
||||||
|
// A fresh capability per reattach keeps a CID single-use, so a leaked one
|
||||||
|
// never grants more than the outage it was observed in.
|
||||||
|
byte[] newCid = hub.newCid();
|
||||||
|
hub.rekeyStream(st, newCid);
|
||||||
|
frames.send(new ProtoWriter()
|
||||||
|
.u8(Protocol.MUX_RESUME_ACK)
|
||||||
|
.i64(st.acceptedOffset)
|
||||||
|
.i64(st.deliveredOffset)
|
||||||
|
.bytes(newCid)
|
||||||
|
.toBytes());
|
||||||
|
|
||||||
|
// Replayed straight to the wire: it must not be re-charged against the
|
||||||
|
// window or re-appended to the retained region, both of which
|
||||||
|
// drainUpstream would do.
|
||||||
|
for (int off = 0; off < replay.length(); off += CHUNK) {
|
||||||
|
int end = Math.min(off + CHUNK, replay.length());
|
||||||
|
sendData(replay.getBytes(off, end));
|
||||||
|
}
|
||||||
|
|
||||||
|
if (st.pendingUp != null) {
|
||||||
|
Buffer pending = st.pendingUp;
|
||||||
|
int off = drainUpstream(st, pending, 0);
|
||||||
|
st.pendingUp = off >= pending.length() ? null : pending.getBuffer(off, pending.length());
|
||||||
|
}
|
||||||
|
if (st.pendingUp == null) st.pausedForWindow = false;
|
||||||
|
maybeResumePlayer(st);
|
||||||
|
checkAggregate(st);
|
||||||
|
LOG.info("worker {} resumed ({} bytes replayed, {} outstanding)",
|
||||||
|
id, replay.length(), outstanding);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void closeBound() {
|
||||||
|
PlayerStream st = stream;
|
||||||
|
stream = null;
|
||||||
if (st != null) {
|
if (st != null) {
|
||||||
upstreamPaused.remove(st);
|
st.worker = null;
|
||||||
|
hub.removeStream(st);
|
||||||
st.player.close();
|
st.player.close();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Park the player if the shared worker socket's write queue is congested. */
|
/** Park the player if the worker socket's write queue is congested. */
|
||||||
private void checkAggregate(StreamState st) {
|
private void checkAggregate(PlayerStream st) {
|
||||||
if (frames.writeQueueFull() && upstreamPaused.add(st)) {
|
if (!st.pausedForAggregate && frames.writeQueueFull()) {
|
||||||
|
st.pausedForAggregate = true;
|
||||||
st.player.pause();
|
st.player.pause();
|
||||||
armWorkerDrain();
|
armWorkerDrain();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Register (once) the shared worker socket's single drain handler; on drain, wake parked players. */
|
/** Register (once) the worker socket's drain handler; on drain, wake the player. */
|
||||||
private void armWorkerDrain() {
|
private void armWorkerDrain() {
|
||||||
if (workerDrainArmed) return;
|
if (workerDrainArmed) return;
|
||||||
workerDrainArmed = true;
|
workerDrainArmed = true;
|
||||||
frames.socket().drainHandler(v -> {
|
frames.socket().drainHandler(v -> {
|
||||||
workerDrainArmed = false;
|
workerDrainArmed = false;
|
||||||
if (upstreamPaused.isEmpty()) return;
|
PlayerStream st = stream;
|
||||||
StreamState[] parked = upstreamPaused.toArray(new StreamState[0]);
|
if (st == null || !st.pausedForAggregate) return;
|
||||||
upstreamPaused.clear();
|
st.pausedForAggregate = false;
|
||||||
for (StreamState st : parked) {
|
maybeResumePlayer(st);
|
||||||
if (!st.pausedForWindow) st.player.resume();
|
|
||||||
}
|
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
/** The player side of a stream vanished: drop it from every table, release any backpressure it held, and FIN the peer if still live. */
|
/**
|
||||||
private void onPlayerGone(int sid, StreamState st) {
|
* Resume the player socket if no reason to hold it applies any more.
|
||||||
boolean wasLive = streams.remove(sid) == st;
|
*
|
||||||
upstreamPaused.remove(st);
|
* <p>The single arbitration point for every pause reason. Vert.x
|
||||||
if (wasLive) sendFin(sid);
|
* {@code pause()} is a flag rather than a counter, so resuming while another
|
||||||
|
* reason still holds would let bytes through that we have nowhere to put.
|
||||||
|
*/
|
||||||
|
private void maybeResumePlayer(PlayerStream st) {
|
||||||
|
if (st.shouldFlow()) st.player.resume();
|
||||||
}
|
}
|
||||||
|
|
||||||
private void sendData(int sid, byte[] data) {
|
/** The player side of the tunnel this conn carries vanished: unbind it and FIN the client. */
|
||||||
frames.send(new ProtoWriter().u8(Protocol.MUX_DATA).varInt(sid).bytes(data).toBytes());
|
void playerGone(PlayerStream st) {
|
||||||
|
boolean wasLive = stream == st;
|
||||||
|
if (wasLive) stream = null;
|
||||||
|
st.worker = null;
|
||||||
|
if (wasLive) sendFin();
|
||||||
}
|
}
|
||||||
|
|
||||||
private void sendFin(int sid) {
|
private void sendData(byte[] data) {
|
||||||
frames.send(new ProtoWriter().u8(Protocol.MUX_FIN).varInt(sid).toBytes());
|
frames.send(new ProtoWriter().u8(Protocol.MUX_DATA).bytes(data).toBytes());
|
||||||
}
|
}
|
||||||
|
|
||||||
private void sendRst(int sid) {
|
private void sendFin() {
|
||||||
frames.send(new ProtoWriter().u8(Protocol.MUX_RST).varInt(sid).toBytes());
|
frames.send(new ProtoWriter().u8(Protocol.MUX_FIN).toBytes());
|
||||||
}
|
}
|
||||||
|
|
||||||
private void sendWnd(int sid, int delta) {
|
/** The reason is a trailing byte, optional on the wire; peers that predate it send none. */
|
||||||
frames.send(new ProtoWriter().u8(Protocol.MUX_WND).varInt(sid).varInt(delta).toBytes());
|
private void sendRst(int reason) {
|
||||||
|
frames.send(new ProtoWriter().u8(Protocol.MUX_RST).u8(reason).toBytes());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private void sendWnd(int delta) {
|
||||||
|
frames.send(new ProtoWriter().u8(Protocol.MUX_WND).varInt(delta).toBytes());
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Answer the client's liveness probe, echoing its nonce. */
|
||||||
|
private void sendPong(long nonce) {
|
||||||
|
frames.send(new ProtoWriter().u8(Protocol.MUX_PONG).i64(nonce).toBytes());
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Only the tunnel leg died. Where the session negotiated resumption the
|
||||||
|
* player socket is hung rather than closed, and waits for the client to
|
||||||
|
* reattach over a fresh conn (§7.5); otherwise this is the old,
|
||||||
|
* unconditional close.
|
||||||
|
*/
|
||||||
public void onClose() {
|
public void onClose() {
|
||||||
for (StreamState st : streams.values()) st.player.close();
|
PlayerStream st = stream;
|
||||||
streams.clear();
|
stream = null;
|
||||||
upstreamPaused.clear();
|
if (st == null) {
|
||||||
LOG.info("worker {} closed", id);
|
LOG.info("worker {} closed (unbound)", id);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
st.worker = null;
|
||||||
|
if (hub.park(st)) {
|
||||||
|
LOG.info("worker {} closed (player hung for reattach)", id);
|
||||||
|
} else {
|
||||||
|
hub.removeStream(st);
|
||||||
|
st.player.close();
|
||||||
|
LOG.info("worker {} closed (player dropped)", id);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -23,6 +23,7 @@ public final class EncryptedFrames {
|
|||||||
private FrameHandler handler;
|
private FrameHandler handler;
|
||||||
private Buffer buf = Buffer.buffer();
|
private Buffer buf = Buffer.buffer();
|
||||||
private boolean closed = false;
|
private boolean closed = false;
|
||||||
|
private long lastFrameAt = System.currentTimeMillis();
|
||||||
|
|
||||||
public EncryptedFrames(NetSocket socket, Cipher in, Cipher out, FrameHandler handler) {
|
public EncryptedFrames(NetSocket socket, Cipher in, Cipher out, FrameHandler handler) {
|
||||||
this.socket = socket;
|
this.socket = socket;
|
||||||
@@ -71,6 +72,7 @@ public final class EncryptedFrames {
|
|||||||
byte[] pt = in.update(ct);
|
byte[] pt = in.update(ct);
|
||||||
if (pt == null) pt = new byte[0];
|
if (pt == null) pt = new byte[0];
|
||||||
buf = buf.getBuffer(hdr + payloadLen, buf.length());
|
buf = buf.getBuffer(hdr + payloadLen, buf.length());
|
||||||
|
lastFrameAt = System.currentTimeMillis();
|
||||||
FrameHandler h = handler;
|
FrameHandler h = handler;
|
||||||
if (h != null) {
|
if (h != null) {
|
||||||
try {
|
try {
|
||||||
@@ -96,6 +98,9 @@ public final class EncryptedFrames {
|
|||||||
|
|
||||||
public boolean writeQueueFull() { return socket.writeQueueFull(); }
|
public boolean writeQueueFull() { return socket.writeQueueFull(); }
|
||||||
|
|
||||||
|
/** Wall-clock millis when the last complete frame was decoded; basis for idle detection. */
|
||||||
|
public long lastFrameAt() { return lastFrameAt; }
|
||||||
|
|
||||||
public void close() {
|
public void close() {
|
||||||
if (closed) return;
|
if (closed) return;
|
||||||
closed = true;
|
closed = true;
|
||||||
|
|||||||
@@ -77,7 +77,10 @@ class CryptoCodecTest {
|
|||||||
|
|
||||||
/** A Hub whose event loop is never touched (register/match/normalize use no Vert.x state). */
|
/** A Hub whose event loop is never touched (register/match/normalize use no Vert.x state). */
|
||||||
private static Hub testHub() {
|
private static Hub testHub() {
|
||||||
return new Hub(null, new Config("0.0.0.0", 25565, "test-psk", 30_000L, 10_000L, Protocol.DEFAULT_STREAM_WINDOW));
|
return new Hub(null, new Config("0.0.0.0", 25565, "test-psk", 30_000L, 10_000L,
|
||||||
|
Protocol.DEFAULT_STREAM_WINDOW, 90_000L,
|
||||||
|
true, 20_000L, 256, 256L * 2 * Protocol.DEFAULT_STREAM_WINDOW, 0L, 15_000L,
|
||||||
|
0, 1, 0));
|
||||||
}
|
}
|
||||||
|
|
||||||
private static ControlSession testSession(Hub hub, String id) {
|
private static ControlSession testSession(Hub hub, String id) {
|
||||||
|
|||||||
@@ -0,0 +1,86 @@
|
|||||||
|
package io.icybear.redapricot;
|
||||||
|
|
||||||
|
import org.junit.jupiter.api.Test;
|
||||||
|
|
||||||
|
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||||
|
import static org.junit.jupiter.api.Assertions.assertNotNull;
|
||||||
|
import static org.junit.jupiter.api.Assertions.assertNull;
|
||||||
|
|
||||||
|
class IpRateLimiterTest {
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void burstThenRefill() {
|
||||||
|
IpRateLimiter lim = new IpRateLimiter(2, 2, 0);
|
||||||
|
long t = 1_000;
|
||||||
|
assertNull(lim.admit("1.1.1.1", t));
|
||||||
|
assertNull(lim.admit("1.1.1.1", t));
|
||||||
|
assertEquals(IpRateLimiter.Deny.RATE, lim.admit("1.1.1.1", t));
|
||||||
|
// 500 ms at 2/s = 1 token.
|
||||||
|
assertNull(lim.admit("1.1.1.1", t + 500));
|
||||||
|
assertEquals(IpRateLimiter.Deny.RATE, lim.admit("1.1.1.1", t + 500));
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void concurrentCapIndependentOfRate() {
|
||||||
|
IpRateLimiter lim = new IpRateLimiter(0, 16, 1);
|
||||||
|
long t = 1_000;
|
||||||
|
assertNull(lim.admit("10.0.0.1", t));
|
||||||
|
assertEquals(1, lim.concurrent("10.0.0.1"));
|
||||||
|
assertEquals(IpRateLimiter.Deny.CONCURRENT, lim.admit("10.0.0.1", t));
|
||||||
|
lim.release("10.0.0.1", t);
|
||||||
|
assertEquals(0, lim.concurrent("10.0.0.1"));
|
||||||
|
assertNull(lim.admit("10.0.0.1", t));
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void ipsAreIndependent() {
|
||||||
|
IpRateLimiter lim = new IpRateLimiter(1, 1, 1);
|
||||||
|
long t = 1_000;
|
||||||
|
assertNull(lim.admit("a", t));
|
||||||
|
assertNull(lim.admit("b", t));
|
||||||
|
assertEquals(IpRateLimiter.Deny.RATE, lim.admit("a", t));
|
||||||
|
assertEquals(IpRateLimiter.Deny.RATE, lim.admit("b", t));
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void bothOffIsNoOp() {
|
||||||
|
IpRateLimiter lim = new IpRateLimiter(0, 16, 0);
|
||||||
|
long t = 1_000;
|
||||||
|
for (int i = 0; i < 100; i++) {
|
||||||
|
assertNull(lim.admit("1.2.3.4", t));
|
||||||
|
}
|
||||||
|
assertEquals(0, lim.bucketCount());
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void sweepDropsIdleEmptyBuckets() {
|
||||||
|
IpRateLimiter lim = new IpRateLimiter(8, 8, 64);
|
||||||
|
long t = 1_000;
|
||||||
|
assertNull(lim.admit("9.9.9.9", t));
|
||||||
|
lim.release("9.9.9.9", t);
|
||||||
|
assertEquals(1, lim.bucketCount());
|
||||||
|
lim.sweep(t + IpRateLimiter.SWEEP_MS - 1);
|
||||||
|
assertEquals(1, lim.bucketCount());
|
||||||
|
lim.sweep(t + IpRateLimiter.SWEEP_MS);
|
||||||
|
assertEquals(0, lim.bucketCount());
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void sweepKeepsLiveBuckets() {
|
||||||
|
IpRateLimiter lim = new IpRateLimiter(8, 8, 64);
|
||||||
|
long t = 1_000;
|
||||||
|
assertNull(lim.admit("9.9.9.9", t));
|
||||||
|
lim.sweep(t + IpRateLimiter.SWEEP_MS * 2);
|
||||||
|
assertEquals(1, lim.bucketCount());
|
||||||
|
assertEquals(1, lim.concurrent("9.9.9.9"));
|
||||||
|
}
|
||||||
|
|
||||||
|
@Test
|
||||||
|
void denyDoesNotConsumeASlot() {
|
||||||
|
IpRateLimiter lim = new IpRateLimiter(1, 1, 8);
|
||||||
|
long t = 1_000;
|
||||||
|
assertNull(lim.admit("x", t));
|
||||||
|
assertNotNull(lim.admit("x", t));
|
||||||
|
assertEquals(1, lim.concurrent("x"));
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user