add credit based mux window

This commit is contained in:
iceBear67
2026-07-15 14:59:32 +00:00
parent ada07e0e36
commit bbe4efbe16
15 changed files with 651 additions and 139 deletions
+54 -7
View File
@@ -140,8 +140,16 @@ Magic : u8 # 0x01 = control session, 0x02 = worker conn
RandLen : VarInt # 8 ≤ RandLen ≤ 64 RandLen : VarInt # 8 ≤ RandLen ≤ 64
Rand : Bytes[RandLen] # cryptographically random Rand : Bytes[RandLen] # cryptographically random
Timestamp : I64 # client's epoch milliseconds Timestamp : I64 # client's epoch milliseconds
Flags : VarInt # feature flags; bit 0x01 (STREAM_FC) MUST be set
RecvWindow: VarInt # client's per-stream receive window, bytes (§7.3)
``` ```
`Flags` is a bitfield of features. Bit `0x01` (STREAM_FC) declares
**per-stream flow control** (§7.3) and is mandatory: `RecvWindow` advertises
the client's per-stream receive window in bytes and must be positive. The hub
closes the connection if the flag is missing, `RecvWindow` is absent or
non-positive, or the fields are malformed.
The hub: The hub:
1. Decrypts frame 1 with Phase A. 1. Decrypts frame 1 with Phase A.
@@ -158,9 +166,13 @@ frame (both directions) is Phase B, counters reset to 0.
The hub then sends one Phase-B frame to confirm success: The hub then sends one Phase-B frame to confirm success:
``` ```
SessionReady : payload = [ 0x00 ] SessionReady : payload = [ 0x00, Flags: VarInt, RecvWindow: VarInt ]
``` ```
The hub echoes the accepted flags (STREAM_FC set) followed by its own
per-stream receive window. A client must reject a SessionReady without the
STREAM_FC flag or without a positive window (an unsupported hub).
A hub that rejects the session simply closes the TCP connection (optionally A hub that rejects the session simply closes the TCP connection (optionally
after a Phase-B `Error` frame, §6). After `SessionReady`: after a Phase-B `Error` frame, §6). After `SessionReady`:
@@ -258,6 +270,7 @@ Data : Bytes[...] # remainder of the frame payload
| `0x01` | DATA | both | raw tunneled bytes for the stream. | | `0x01` | DATA | both | raw tunneled bytes for the stream. |
| `0x02` | FIN | both | *(empty)* — graceful close of the stream (both directions). This is the "disconnect" the hub sends when the player leaves. | | `0x02` | FIN | both | *(empty)* — graceful close of the stream (both directions). This is the "disconnect" the hub sends when the player leaves. |
| `0x03` | RST | both | *(optional 1 byte reason)* — abnormal close (e.g. CID unknown/expired, destination dial failed). | | `0x03` | RST | both | *(optional 1 byte reason)* — abnormal close (e.g. CID unknown/expired, destination dial failed). |
| `0x04` | WND | both | `Delta: VarInt` — flow-control credit grant (§7.3). |
There is no explicit SYN-ACK: success is implied by the hub forwarding the There is no explicit SYN-ACK: success is implied by the hub forwarding the
buffered Handshake as the stream's first `DATA`; failure is an `RST`. buffered Handshake as the stream's first `DATA`; failure is an `RST`.
@@ -294,11 +307,33 @@ configurable, `1..8`). To place a new stream:
closes the destination. When the destination closes, the client sends `FIN`; closes the destination. When the destination closes, the client sends `FIN`;
the hub closes the player socket. `RST` is treated the same way (hard close). the hub closes the player socket. `RST` is treated the same way (hard close).
Data on a worker conn is subject to that TCP connection's back-pressure. Each Data on a worker conn is subject to that TCP connection's back-pressure for
stream has a bounded outbound queue on the receiving side; overflow resets the its **aggregate** bandwidth; *per-stream* fairness is governed by the credit
stream (`RST`). (This is a deliberate simplification — no per-stream credit windows of §7.3.
windows — acceptable for the interactive, low-throughput Minecraft handshake +
gameplay traffic pattern.) ### 7.3 Per-stream flow control
Every stream carries an independent credit window per direction:
* Each side advertised its **receive window** W (bytes) at session setup. A
sender may have at most W un-credited DATA bytes outstanding per stream; the
initial budget is W, spent as DATA is sent (`Data` length only — SYN/FIN/RST
frames are free) starting with the very first DATA on the stream (including
the hub's forwarded handshake).
* The receiver returns credit with `WND(Delta)` once bytes are **delivered to
the terminal socket** (written to the player / destination connection), not
when they are merely buffered. Receivers should batch grants (the reference
implementations send one `WND` per W/2 bytes consumed).
* A sender whose window is exhausted pauses reading **that stream's source
socket only**; the shared worker conn is never paused because of a single
stream. A receiver that observes more than W un-credited bytes on a stream
may reset it (`RST`) as a protocol violation.
* Senders should also cap individual DATA payloads (the reference
implementations use 32 KiB) so one stream cannot monopolize the link for a
full 1-MiB frame.
Both windows may differ (each side enforces the one its peer advertised).
`Delta` must be positive; a `WND` for an unknown stream id is ignored.
## 8. HAProxy protocol v2 (optional) ## 8. HAProxy protocol v2 (optional)
@@ -327,10 +362,14 @@ big-endian.
"listen": "0.0.0.0:25565", "listen": "0.0.0.0:25565",
"psk": "change-me", "psk": "change-me",
"timestampWindowMs": 30000, "timestampWindowMs": 30000,
"pendingTimeoutMs": 10000 "pendingTimeoutMs": 10000,
"streamWindowBytes": 262144
} }
``` ```
`streamWindowBytes` (optional, default 262144, clamped to [32768, 8388608]) is
the hub's advertised per-stream receive window (§7.3).
### 9.2 Client — JSON ### 9.2 Client — JSON
```json ```json
@@ -339,12 +378,16 @@ big-endian.
"psk": "change-me", "psk": "change-me",
"maxConn": 4, "maxConn": 4,
"pingIntervalMs": 20000, "pingIntervalMs": 20000,
"streamWindowBytes": 262144,
"mappings": [ "mappings": [
{ "pattern": "mc\\.example\\.com", "destination": "127.0.0.1:25566", "proxyProtocol": true } { "pattern": "mc\\.example\\.com", "destination": "127.0.0.1:25566", "proxyProtocol": true }
] ]
} }
``` ```
`streamWindowBytes` (optional, default 262144, clamped to [32768, 8388608]) is
the client's advertised per-stream receive window (§7.3).
Each `pattern` is a regular expression (§5.1) matched against the whole Each `pattern` is a regular expression (§5.1) matched against the whole
normalized player hostname, case-insensitively. Escape literal dots (`mc\.example\.com`, normalized player hostname, case-insensitively. Escape literal dots (`mc\.example\.com`,
which is `mc\\.example\\.com` in JSON); an unescaped `.` matches any character. which is `mc\\.example\\.com` in JSON); an unescaped `.` matches any character.
@@ -369,4 +412,8 @@ subdomain or `(alpha|beta)\.mc\.net` for a fixed set.
| max frame payload | 1 MiB | | max frame payload | 1 MiB |
| saturation threshold | active streams `> 8` | | saturation threshold | active streams `> 8` |
| max worker conns | `max_conn ∈ [1,8]` | | max worker conns | `max_conn ∈ [1,8]` |
| feature flag: per-stream flow control | `0x01` |
| stream window default / bounds | 256 KiB, clamped to [32 KiB, 8 MiB] |
| WND grant batching (reference) | one grant per window/2 consumed |
| DATA chunk cap (reference) | 32 KiB |
``` ```
+9 -5
View File
@@ -202,7 +202,9 @@ forwarding and case-insensitive matching, regex wildcard pattern routing,
multi-megabyte transfers, concurrent multi-megabyte transfers, concurrent
streams spreading across multiple worker connections, HAProxy v2 source-address streams spreading across multiple worker connections, HAProxy v2 source-address
propagation, player- and destination-initiated disconnect propagation, wrong-PSK propagation, player- and destination-initiated disconnect propagation, wrong-PSK
rejection, and dropping of unmatched hostnames. The Go and Java crypto layers are rejection, dropping of unmatched hostnames, stream isolation under a slow
player and under a slow destination (no head-of-line blocking), and rejection
of pre-flow-control peers. The Go and Java crypto layers are
independently pinned to the same SHA3-224 test vector so they cannot silently independently pinned to the same SHA3-224 test vector so they cannot silently
drift apart. drift apart.
@@ -212,10 +214,12 @@ drift apart.
ChaCha20-encrypted (no AEAD tag) to minimize overhead. This protects against ChaCha20-encrypted (no AEAD tag) to minimize overhead. This protects against
casual sniffing, not a determined active attacker (see the note at the top of casual sniffing, not a determined active attacker (see the note at the top of
[PROTOCOL.md](PROTOCOL.md) and [docs/architecture.md](docs/architecture.md) §8). [PROTOCOL.md](PROTOCOL.md) and [docs/architecture.md](docs/architecture.md) §8).
* **No per-stream flow control.** Multiplexing relies on TCP back-pressure per * **Per-stream flow control.** Each stream has credit-based windows in both
worker connection, so one very slow stream can head-of-line-block others on directions (windows exchanged at session setup, default 256 KiB), so a slow
the same connection. Raising `maxConn` spreads load. Fine for interactive player or slow destination jams only its own stream at a bounded buffer — no
Minecraft traffic; not a general-purpose high-throughput mux. application-level head-of-line blocking between streams. What remains is
TCP-level HOL (packet loss stalls a whole worker connection briefly);
raising `maxConn` spreads that.
* **Single hub event loop.** The hub deploys one Vert.x verticle, so all state * **Single hub event loop.** The hub deploys one Vert.x verticle, so all state
is confined to one event loop (no locking). Throughput is bounded by one core; is confined to one event loop (no locking). Throughput is bounded by one core;
ample for hundreds of players, not designed for tens of thousands. ample for hundreds of players, not designed for tens of thousands.
+45 -14
View File
@@ -24,6 +24,8 @@ type Client struct {
mappings map[string]Mapping // normalized pattern -> mapping mappings map[string]Mapping // normalized pattern -> mapping
pool *WorkerPool pool *WorkerPool
streamWnd int // our advertised per-stream receive window (bytes)
mu sync.Mutex mu sync.Mutex
ctrl *wire.FramedConn ctrl *wire.FramedConn
} }
@@ -44,16 +46,32 @@ func New(cfg *Config) *Client {
for _, m := range cfg.Mappings { for _, m := range cfg.Mappings {
c.mappings[NormalizeAddress(m.Pattern)] = m c.mappings[NormalizeAddress(m.Pattern)] = m
} }
c.streamWnd = clampWindow(cfg.StreamWindowBytes)
c.pool = newWorkerPool(c, cfg.MaxConn) c.pool = newWorkerPool(c, cfg.MaxConn)
return c return c
} }
func clampWindow(w int) int {
if w <= 0 {
return DefaultStreamWindow
}
if w < MinStreamWindow {
return MinStreamWindow
}
if w > MaxStreamWindow {
return MaxStreamWindow
}
return w
}
// dialSession opens a TCP connection, performs the Intent-17 handshake, the // dialSession opens a TCP connection, performs the Intent-17 handshake, the
// Phase-A rekey, and reads SessionReady, returning an established frame conn. // Phase-A rekey, and reads SessionReady, returning an established frame conn
func (c *Client) dialSession(magic byte) (*wire.FramedConn, error) { // and the hub's advertised per-stream receive window. Per-stream flow control
// is mandatory: a hub that does not echo the STREAM_FC flag is rejected.
func (c *Client) dialSession(magic byte) (fc *wire.FramedConn, peerWnd int, err error) {
conn, err := net.DialTimeout("tcp", c.cfg.Server, 10*time.Second) conn, err := net.DialTimeout("tcp", c.cfg.Server, 10*time.Second)
if err != nil { if err != nil {
return nil, err return nil, 0, err
} }
if tcp, ok := conn.(*net.TCPConn); ok { if tcp, ok := conn.(*net.TCPConn); ok {
_ = tcp.SetNoDelay(true) _ = tcp.SetNoDelay(true)
@@ -68,24 +86,26 @@ func (c *Client) dialSession(magic byte) (*wire.FramedConn, error) {
// 1. plaintext Minecraft Handshake, Intent 17, address = hex(SHA3-224(PSK)). // 1. plaintext Minecraft Handshake, Intent 17, address = hex(SHA3-224(PSK)).
hs := wire.BuildHandshake(ProtocolVersion, c.pskAddr, c.serverPort, IntentRedapricot) hs := wire.BuildHandshake(ProtocolVersion, c.pskAddr, c.serverPort, IntentRedapricot)
if _, err := conn.Write(hs); err != nil { if _, err := conn.Write(hs); err != nil {
return nil, err return nil, 0, err
} }
// 2. Phase-A ciphers derived from the PSK. // 2. Phase-A ciphers derived from the PSK.
fc := wire.NewFramedConn(conn, fc = wire.NewFramedConn(conn,
wire.CipherFor(c.pskBytes, wire.DirS2C), // in: server -> client wire.CipherFor(c.pskBytes, wire.DirS2C), // in: server -> client
wire.CipherFor(c.pskBytes, wire.DirC2S), // out: client -> server wire.CipherFor(c.pskBytes, wire.DirC2S), // out: client -> server
) )
// 3. Rekey frame (Phase A). // 3. Rekey frame (Phase A), including the mandatory feature flags and our
// per-stream receive window.
rnd := make([]byte, 16) rnd := make([]byte, 16)
if _, err := crand.Read(rnd); err != nil { if _, err := crand.Read(rnd); err != nil {
return nil, err return nil, 0, err
} }
ts := time.Now().UnixMilli() ts := time.Now().UnixMilli()
rekeyMsg := wire.NewWriter().U8(magic).VarInt(len(rnd)).Bytes(rnd).I64(ts).Out() rekeyMsg := wire.NewWriter().U8(magic).VarInt(len(rnd)).Bytes(rnd).I64(ts).
VarInt(FlagStreamFC).VarInt(c.streamWnd).Out()
if err := fc.WriteFrame(rekeyMsg); err != nil { if err := fc.WriteFrame(rekeyMsg); err != nil {
return nil, err return nil, 0, err
} }
// 4. Switch to Phase-B ciphers: REKEY = Rand || Timestamp(I64 BE). // 4. Switch to Phase-B ciphers: REKEY = Rand || Timestamp(I64 BE).
@@ -99,16 +119,26 @@ func (c *Client) dialSession(magic byte) (*wire.FramedConn, error) {
wire.CipherFor(rekey, wire.DirC2S), wire.CipherFor(rekey, wire.DirC2S),
) )
// 5. SessionReady. // 5. SessionReady: the type byte followed by the hub's accepted flags and
// its per-stream receive window. Both are required.
payload, err := fc.ReadFrame() payload, err := fc.ReadFrame()
if err != nil { if err != nil {
return nil, err return nil, 0, err
} }
if len(payload) < 1 || payload[0] != CtlSessionReady { if len(payload) < 1 || payload[0] != CtlSessionReady {
return nil, fmt.Errorf("expected SessionReady, got %v", payload) return nil, 0, fmt.Errorf("expected SessionReady, got %v", payload)
}
r := wire.NewReader(payload[1:])
flags, ferr := r.VarInt()
hubWnd, werr := r.VarInt()
if ferr != nil || werr != nil || flags&FlagStreamFC == 0 || hubWnd <= 0 {
return nil, 0, fmt.Errorf("hub did not accept per-stream flow control (unsupported hub version?)")
}
if hubWnd > MaxStreamWindow {
hubWnd = MaxStreamWindow
} }
ok = true ok = true
return fc, nil return fc, hubWnd, nil
} }
// Start establishes the control session and registers all patterns. It returns // Start establishes the control session and registers all patterns. It returns
@@ -119,7 +149,7 @@ func (c *Client) Start(ctx context.Context) error {
} }
func (c *Client) connectControl(ctx context.Context) error { func (c *Client) connectControl(ctx context.Context) error {
fc, err := c.dialSession(MagicControl) fc, _, err := c.dialSession(MagicControl)
if err != nil { if err != nil {
return fmt.Errorf("control connect: %w", err) return fmt.Errorf("control connect: %w", err)
} }
@@ -233,6 +263,7 @@ func (c *Client) handleControlRequest(cid []byte, pattern, ip string, port int,
st := newStream(wc, sid, cid, mapping, ip, port) st := newStream(wc, sid, cid, mapping, ip, port)
wc.registerStream(sid, st) wc.registerStream(sid, st)
wc.sendSyn(sid, cid) wc.sendSyn(sid, cid)
go st.writeLoop()
go st.run() go st.run()
} }
+1
View File
@@ -3,6 +3,7 @@
"psk": "change-me-to-a-long-random-passphrase", "psk": "change-me-to-a-long-random-passphrase",
"maxConn": 4, "maxConn": 4,
"pingIntervalMs": 20000, "pingIntervalMs": 20000,
"streamWindowBytes": 262144,
"mappings": [ "mappings": [
{ {
"pattern": "mc\\.example\\.com", "pattern": "mc\\.example\\.com",
+20 -5
View File
@@ -29,10 +29,24 @@ const (
MuxData = 0x01 MuxData = 0x01
MuxFin = 0x02 MuxFin = 0x02
MuxRst = 0x03 MuxRst = 0x03
MuxWnd = 0x04
FrameError = 0x7F FrameError = 0x7F
SaturationThreshold = 8 SaturationThreshold = 8
// Session-establishment feature flags (trailing VarInt on the Rekey message).
FlagStreamFC = 0x01
// Per-stream flow-control window bounds (bytes). The advertised window is the
// receiver's promise of how much un-credited DATA it will buffer per stream.
DefaultStreamWindow = 256 * 1024
MinStreamWindow = 32 * 1024
MaxStreamWindow = 8 << 20
// DataChunkSize caps a single DATA frame's payload so no stream monopolizes
// the shared worker connection for long.
DataChunkSize = 32 * 1024
) )
// Mapping routes a registered pattern to a real destination. // Mapping routes a registered pattern to a real destination.
@@ -44,11 +58,12 @@ type Mapping struct {
// Config is the client configuration (PROTOCOL.md §9.2). // Config is the client configuration (PROTOCOL.md §9.2).
type Config struct { type Config struct {
Server string `json:"server"` Server string `json:"server"`
PSK string `json:"psk"` PSK string `json:"psk"`
MaxConn int `json:"maxConn"` MaxConn int `json:"maxConn"`
PingIntervalMs int `json:"pingIntervalMs"` PingIntervalMs int `json:"pingIntervalMs"`
Mappings []Mapping `json:"mappings"` StreamWindowBytes int `json:"streamWindowBytes"` // per-stream receive window; 0 = default
Mappings []Mapping `json:"mappings"`
} }
// LoadConfig reads and validates a JSON config file. // LoadConfig reads and validates a JSON config file.
+168 -38
View File
@@ -55,18 +55,21 @@ func (p *WorkerPool) Allocate() (*WorkerConn, int, error) {
} }
func (p *WorkerPool) dialWorker() (*WorkerConn, error) { func (p *WorkerPool) dialWorker() (*WorkerConn, error) {
fc, err := p.client.dialSession(MagicWorker) fc, peerWnd, err := p.client.dialSession(MagicWorker)
if err != nil { if err != nil {
return nil, err return nil, err
} }
wc := &WorkerConn{ wc := &WorkerConn{
pool: p, pool: p,
fc: fc, fc: fc,
streams: make(map[int]*Stream), sendWndInit: peerWnd,
nextSid: 1, recvWndInit: p.client.streamWnd,
streams: make(map[int]*Stream),
nextSid: 1,
} }
go wc.readLoop() go wc.readLoop()
log.Printf("opened worker conn (#%d in pool)", len(p.conns)+1) log.Printf("opened worker conn (#%d in pool, send window %d, recv window %d)",
len(p.conns)+1, wc.sendWndInit, wc.recvWndInit)
return wc, nil return wc, nil
} }
@@ -101,6 +104,9 @@ type WorkerConn struct {
pool *WorkerPool pool *WorkerPool
fc *wire.FramedConn fc *wire.FramedConn
sendWndInit int // hub's advertised per-stream receive window (our send budget)
recvWndInit int // our advertised per-stream receive window (bounds each recv queue)
mu sync.Mutex mu sync.Mutex
streams map[int]*Stream streams map[int]*Stream
nextSid int nextSid int
@@ -140,6 +146,9 @@ func (wc *WorkerConn) removeStream(sid int) *Stream {
return st return st
} }
// readLoop dispatches inbound mux frames. It must never block on a stream's
// destination: DATA is only enqueued (the per-stream writeLoop does the actual
// destination writes), so one slow destination cannot stall other streams.
func (wc *WorkerConn) readLoop() { func (wc *WorkerConn) readLoop() {
for { for {
payload, err := wc.fc.ReadFrame() payload, err := wc.fc.ReadFrame()
@@ -160,9 +169,20 @@ func (wc *WorkerConn) readLoop() {
if st := wc.getStream(sid); st != nil { if st := wc.getStream(sid); st != nil {
st.deliverFromHub(r.Remaining()) st.deliverFromHub(r.Remaining())
} }
case MuxFin, MuxRst: case MuxWnd:
if delta, err := r.VarInt(); err == nil && delta > 0 {
if st := wc.getStream(sid); st != nil {
st.grantSendWnd(delta)
}
}
case MuxFin:
// Graceful: drain what is already queued to the destination first.
if st := wc.removeStream(sid); st != nil { if st := wc.removeStream(sid); st != nil {
st.shutdown(false) st.gracefulFin()
}
case MuxRst:
if st := wc.removeStream(sid); st != nil {
st.teardown(false)
} }
default: default:
log.Printf("worker: unknown mux type %d", ftype) log.Printf("worker: unknown mux type %d", ftype)
@@ -178,7 +198,7 @@ func (wc *WorkerConn) readLoop() {
wc.streams = make(map[int]*Stream) wc.streams = make(map[int]*Stream)
wc.mu.Unlock() wc.mu.Unlock()
for _, st := range streams { for _, st := range streams {
st.shutdown(false) st.teardown(false)
} }
} }
@@ -198,7 +218,16 @@ func (wc *WorkerConn) sendRst(sid int) {
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxRst).VarInt(sid).Out()) _ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxRst).VarInt(sid).Out())
} }
func (wc *WorkerConn) sendWndUpdate(sid, delta int) {
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxWnd).VarInt(sid).VarInt(delta).Out())
}
// Stream bridges one player (via the hub) to one destination connection. // Stream bridges one player (via the hub) to one destination connection.
//
// Data from the hub is queued and written to the destination by a dedicated
// writeLoop goroutine. The queue is bounded by the advertised receive window —
// the hub never sends more un-credited bytes, so overflow is a protocol
// violation and resets the stream.
type Stream struct { type Stream struct {
wc *WorkerConn wc *WorkerConn
sid int sid int
@@ -207,25 +236,33 @@ type Stream struct {
srcIP string srcIP string
srcPort int srcPort int
mu sync.Mutex mu sync.Mutex
dest net.Conn cond *sync.Cond
connected bool dest net.Conn
preBuf []byte connected bool
closed bool closed bool
finPending bool // hub sent FIN; close the destination once the queue drains
q [][]byte // hub -> destination, waiting for writeLoop
qBytes int
sendWnd int // flow control: budget for destination -> hub DATA
consumed int // flow control: drained bytes not yet credited back to the hub
} }
func newStream(wc *WorkerConn, sid int, cid []byte, m Mapping, ip string, port int) *Stream { func newStream(wc *WorkerConn, sid int, cid []byte, m Mapping, ip string, port int) *Stream {
return &Stream{wc: wc, sid: sid, cid: cid, mapping: m, srcIP: ip, srcPort: port} s := &Stream{wc: wc, sid: sid, cid: cid, mapping: m, srcIP: ip, srcPort: port, sendWnd: wc.sendWndInit}
s.cond = sync.NewCond(&s.mu)
return s
} }
// run dials the destination, optionally writes the PROXY v2 header, flushes any // run dials the destination, optionally writes the PROXY v2 header, then pumps
// buffered hub bytes, then pumps destination -> hub. // destination -> hub (respecting the stream send window when negotiated).
func (s *Stream) run() { func (s *Stream) run() {
dest, err := net.DialTimeout("tcp", s.mapping.Destination, 10*time.Second) dest, err := net.DialTimeout("tcp", s.mapping.Destination, 10*time.Second)
if err != nil { if err != nil {
log.Printf("stream %d: dial %s failed: %v", s.sid, s.mapping.Destination, err) log.Printf("stream %d: dial %s failed: %v", s.sid, s.mapping.Destination, err)
s.wc.removeStream(s.sid) s.wc.removeStream(s.sid)
s.wc.sendRst(s.sid) s.wc.sendRst(s.sid)
s.teardown(false)
return return
} }
if tcp, ok := dest.(*net.TCPConn); ok { if tcp, ok := dest.(*net.TCPConn); ok {
@@ -240,7 +277,6 @@ func (s *Stream) run() {
} }
} }
// Atomically flush pre-connect buffer and enable direct writes.
s.mu.Lock() s.mu.Lock()
if s.closed { if s.closed {
s.mu.Unlock() s.mu.Unlock()
@@ -248,18 +284,18 @@ func (s *Stream) run() {
return return
} }
s.dest = dest s.dest = dest
if len(s.preBuf) > 0 {
_, _ = dest.Write(s.preBuf)
s.preBuf = nil
}
s.connected = true s.connected = true
s.cond.Broadcast() // wake writeLoop: queued hub bytes can flow now
s.mu.Unlock() s.mu.Unlock()
// destination -> hub // destination -> hub
buf := make([]byte, 32*1024) buf := make([]byte, DataChunkSize)
for { for {
n, err := dest.Read(buf) n, err := dest.Read(buf)
if n > 0 { if n > 0 {
if !s.acquireSendWnd(n) {
break
}
if werr := s.wc.sendData(s.sid, buf[:n]); werr != nil { if werr := s.wc.sendData(s.sid, buf[:n]); werr != nil {
break break
} }
@@ -268,7 +304,99 @@ func (s *Stream) run() {
break break
} }
} }
s.shutdown(true) s.teardown(true)
}
// writeLoop is the only writer to the destination. It drains the receive queue,
// credits the hub as bytes land on the destination socket, and performs the
// deferred graceful close when a FIN arrived with data still queued.
func (s *Stream) writeLoop() {
for {
s.mu.Lock()
for !s.closed && (!s.connected || (len(s.q) == 0 && !s.finPending)) {
s.cond.Wait()
}
if s.closed {
s.mu.Unlock()
return
}
if len(s.q) == 0 { // finPending and fully drained
s.mu.Unlock()
s.teardown(false)
return
}
data := s.q[0]
s.q = s.q[1:]
s.qBytes -= len(data)
dest := s.dest
s.mu.Unlock()
if _, err := dest.Write(data); err != nil {
s.teardown(true)
return
}
s.credit(len(data))
}
}
// deliverFromHub enqueues hub bytes for the destination. Called from the worker
// readLoop; it never blocks — a peer that exceeds the advertised window is a
// protocol violator and gets the stream reset.
func (s *Stream) deliverFromHub(data []byte) {
s.mu.Lock()
if s.closed || s.finPending {
s.mu.Unlock()
return
}
if s.qBytes+len(data) > s.wc.recvWndInit {
s.mu.Unlock()
log.Printf("stream %d: peer exceeded flow-control window; resetting", s.sid)
s.wc.removeStream(s.sid)
s.wc.sendRst(s.sid)
s.teardown(false)
return
}
s.q = append(s.q, data)
s.qBytes += len(data)
s.cond.Broadcast()
s.mu.Unlock()
}
// acquireSendWnd blocks until the stream may send n more bytes to the hub.
// Returns false if the stream closed while waiting.
func (s *Stream) acquireSendWnd(n int) bool {
s.mu.Lock()
defer s.mu.Unlock()
for !s.closed && s.sendWnd < n {
s.cond.Wait()
}
if s.closed {
return false
}
s.sendWnd -= n
return true
}
func (s *Stream) grantSendWnd(delta int) {
s.mu.Lock()
s.sendWnd += delta
s.cond.Broadcast()
s.mu.Unlock()
}
// credit accounts bytes drained to the destination and grants the hub more
// window once half of our receive window has been consumed.
func (s *Stream) credit(n int) {
s.mu.Lock()
s.consumed += n
if s.closed || s.consumed*2 < s.wc.recvWndInit {
s.mu.Unlock()
return
}
delta := s.consumed
s.consumed = 0
s.mu.Unlock()
s.wc.sendWndUpdate(s.sid, delta)
} }
func (s *Stream) buildProxyHeader(dest net.Conn) []byte { func (s *Stream) buildProxyHeader(dest net.Conn) []byte {
@@ -283,28 +411,29 @@ func (s *Stream) buildProxyHeader(dest net.Conn) []byte {
return BuildProxyV2(srcIP, s.srcPort, dstTCP.IP, dstTCP.Port) return BuildProxyV2(srcIP, s.srcPort, dstTCP.IP, dstTCP.Port)
} }
// deliverFromHub writes bytes coming from the hub to the destination, buffering // finDrainTimeout bounds how long a FIN'd stream may keep draining its queue
// until the destination connection is established. // into the destination, so a dead destination cannot hold the stream forever.
func (s *Stream) deliverFromHub(data []byte) { const finDrainTimeout = 10 * time.Second
// gracefulFin marks the hub-side close; writeLoop finishes the queue and then
// tears the stream down.
func (s *Stream) gracefulFin() {
s.mu.Lock() s.mu.Lock()
if s.closed { if s.closed || s.finPending {
s.mu.Unlock() s.mu.Unlock()
return return
} }
if !s.connected { s.finPending = true
s.preBuf = append(s.preBuf, data...) if s.dest != nil {
s.mu.Unlock() _ = s.dest.SetWriteDeadline(time.Now().Add(finDrainTimeout))
return
} }
dest := s.dest s.cond.Broadcast()
s.mu.Unlock() s.mu.Unlock()
if _, err := dest.Write(data); err != nil {
s.shutdown(true)
}
} }
// shutdown closes the stream; notifyHub sends a FIN to the hub when true. // teardown closes the stream immediately; notifyHub sends a FIN when true.
func (s *Stream) shutdown(notifyHub bool) { // Idempotent; wakes every goroutine parked on the stream.
func (s *Stream) teardown(notifyHub bool) {
s.mu.Lock() s.mu.Lock()
if s.closed { if s.closed {
s.mu.Unlock() s.mu.Unlock()
@@ -312,6 +441,7 @@ func (s *Stream) shutdown(notifyHub bool) {
} }
s.closed = true s.closed = true
dest := s.dest dest := s.dest
s.cond.Broadcast()
s.mu.Unlock() s.mu.Unlock()
if dest != nil { if dest != nil {
+35 -16
View File
@@ -154,25 +154,43 @@ connections (9 + 9 + 2), confirming the algorithm.
simplicity and correctness. simplicity and correctness.
* **Client:** goroutine-per-concern. One goroutine reads each connection * **Client:** goroutine-per-concern. One goroutine reads each connection
(control or worker); `WriteFrame` is mutex-serialized so many stream goroutines (control or worker); `WriteFrame` is mutex-serialized so many stream goroutines
can share a worker connection safely. A per-stream mutex guards the small can share a worker connection safely. Each stream has two goroutines: `run`
"buffer until the destination is connected, then write directly" handoff so pumps destination → hub, and `writeLoop` is the only writer to the
the forwarded handshake never races ahead of later bytes. destination, draining a per-stream queue fed by the worker readLoop. The
readLoop itself never writes to a destination, so a stalled destination can
never block frame dispatch for other streams.
## 6. Back-pressure ## 6. Back-pressure & flow control
There is no per-stream credit window. Flow is governed by TCP back-pressure on Two mechanisms operate at different granularities:
each worker connection:
* Hub → player: if a player socket's write queue fills, the hub pauses the * **Per-stream credit windows** (PROTOCOL.md §7.3; the windows are exchanged
worker connection socket and resumes on drain. at session establishment): each stream direction has an independent byte
* Destination → hub: the client's `WriteFrame` blocks when the worker socket is budget equal to the receiver's advertised window (default 256 KiB). A sender
congested, which naturally stops the client reading the destination. that exhausts a
stream's window pauses *only that stream's source* — the hub pauses the one
player socket, the client parks the one destination-reader goroutine. Credit
is granted back (`WND` frames, batched at half-window) as bytes are actually
written to the terminal socket. The result: a slow player or slow destination
jams its own stream at a bounded buffer size and nothing else. This is what
eliminates head-of-line blocking between streams.
* **Aggregate TCP back-pressure** on each worker connection: when the shared
socket itself is congested (total bandwidth, not one stream), the hub parks
all sending players until it drains, and the client's `WriteFrame` blocks.
This is fair — when the pipe is genuinely full, everyone should slow down.
The consequence is head-of-line blocking *within* a worker connection: one very The window also bounds memory: a stream can hold at most one window of
slow player can stall other streams sharing that connection. `maxConn` spreads undelivered data per direction (the client's pre-connect handshake buffer is
streams across connections to mitigate this. For interactive Minecraft traffic covered by the same bound).
(small client→server packets, bursty server→client chunk data) this is a good
trade for a near-zero-overhead mux. Per-stream flow control is mandatory: the hub rejects a session whose Rekey
lacks the STREAM_FC flag, and the client rejects a hub that does not echo it —
peers that predate the mechanism cannot connect at all.
What remains (by design) is TCP-level head-of-line blocking: a lost packet on
a worker connection stalls all its streams for one retransmit. That is inherent
to mux-over-TCP; the connection pool is the mitigation, and a datagram
transport (QUIC) would be the escape hatch if it ever matters.
## 7. Failure & recovery ## 7. Failure & recovery
@@ -190,7 +208,8 @@ trade for a near-zero-overhead mux.
## 8. Known limitations ## 8. Known limitations
1. No AEAD — payload integrity/authenticity is not cryptographically guaranteed. 1. No AEAD — payload integrity/authenticity is not cryptographically guaranteed.
2. No per-stream flow control (see §6). 2. TCP-level head-of-line blocking within a worker connection (lost packets;
see §6) — per-stream flow control removes the application-level variant only.
3. Single-event-loop hub (see §5) bounds throughput to one core. 3. Single-event-loop hub (see §5) bounds throughput to one core.
4. `Intent 18` is reserved but only stubbed (the hub logs and closes). 4. `Intent 18` is reserved but only stubbed (the hub logs and closes).
5. Pattern ownership is last-writer-wins; two clients registering the identical 5. Pattern ownership is last-writer-wins; two clients registering the identical
+11 -1
View File
@@ -129,6 +129,7 @@ type destMode int
const ( const (
modeEcho destMode = iota // echo every post-handshake byte modeEcho destMode = iota // echo every post-handshake byte
modeEchoOnceClose // echo one read, then close the connection modeEchoOnceClose // echo one read, then close the connection
modeBlackhole // accept but never read: immediate write back-pressure
) )
type proxyInfo struct { type proxyInfo struct {
@@ -154,6 +155,7 @@ type mockDest struct {
mode destMode mode destMode
events chan destEvent events chan destEvent
connClosed chan struct{} connClosed chan struct{}
done chan struct{}
} }
func newMockDest(t *testing.T, mode destMode) *mockDest { func newMockDest(t *testing.T, mode destMode) *mockDest {
@@ -168,8 +170,12 @@ func newMockDest(t *testing.T, mode destMode) *mockDest {
mode: mode, mode: mode,
events: make(chan destEvent, 128), events: make(chan destEvent, 128),
connClosed: make(chan struct{}, 128), connClosed: make(chan struct{}, 128),
done: make(chan struct{}),
} }
t.Cleanup(func() { _ = ln.Close() }) t.Cleanup(func() {
_ = ln.Close()
close(d.done)
})
go d.serve() go d.serve()
return d return d
} }
@@ -189,6 +195,10 @@ func (d *mockDest) handle(conn net.Conn) {
_ = conn.Close() _ = conn.Close()
d.connClosed <- struct{}{} d.connClosed <- struct{}{}
}() }()
if d.mode == modeBlackhole {
<-d.done // hold the connection open without ever reading
return
}
br := bufio.NewReader(conn) br := bufio.NewReader(conn)
var ev destEvent var ev destEvent
+131
View File
@@ -0,0 +1,131 @@
package e2e
import (
crand "crypto/rand"
"fmt"
"net"
"testing"
"time"
"github.com/iceBear67/redapricot/client"
"github.com/iceBear67/redapricot/client/wire"
)
// echoRounds pushes payload through the tunnel `rounds` times on one player
// connection, failing the test if any round stalls.
func echoRounds(t *testing.T, hubAddr string, rounds, size int) {
t.Helper()
fast := dialPlayer(t, hubAddr, "mc.local")
defer fast.Close()
payload := make([]byte, size)
for i := range payload {
payload[i] = byte(i*13 + 5)
}
for i := 0; i < rounds; i++ {
playerEcho(t, fast, payload)
}
}
// TestSlowPlayerDoesNotStallOthers: a player that stops reading while megabytes
// are echoed back to it must not stall another stream on the same worker
// connection (maxConn=1 forces sharing). Before per-stream flow control, the
// hub paused the whole worker socket once that player's write queue filled,
// freezing every other stream's downstream data.
func TestSlowPlayerDoesNotStallOthers(t *testing.T) {
const psk = "e2e-slowplayer"
port := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
startHub(t, port, psk)
dest := newMockDest(t, modeEcho)
startClient(t, hubAddr, psk, 1, []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr},
})
// Slow player: 4 MiB goes out, gets echoed back, and is never read. The
// write blocks once buffers fill; the error on test-end close is expected.
slow := dialPlayer(t, hubAddr, "mc.local")
defer slow.Close()
go func() {
_, _ = slow.Write(make([]byte, 4*1024*1024))
}()
// Let the slow stream jam: its flow-control window fills and stays full.
time.Sleep(1 * time.Second)
// The fast player shares the single worker conn and must still round-trip.
echoRounds(t, hubAddr, 10, 8*1024)
}
// TestSlowDestinationDoesNotStallOthers: a destination that never reads must
// only stall its own stream. Before the async delivery queue, the client wrote
// to destinations inline in the worker readLoop, so one blocked destination
// froze every stream on the connection.
func TestSlowDestinationDoesNotStallOthers(t *testing.T) {
const psk = "e2e-slowdest"
port := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
startHub(t, port, psk)
dest := newMockDest(t, modeEcho)
hole := newMockDest(t, modeBlackhole)
startClient(t, hubAddr, psk, 1, []client.Mapping{
{Pattern: "mc.local", Destination: dest.addr},
{Pattern: "hole.local", Destination: hole.addr},
})
// This player's destination never reads: the client-side write jams after
// kernel buffers fill, with the stream's queue bounded by its window.
stuck := dialPlayer(t, hubAddr, "hole.local")
defer stuck.Close()
go func() {
_, _ = stuck.Write(make([]byte, 2*1024*1024))
}()
time.Sleep(1 * time.Second)
echoRounds(t, hubAddr, 10, 8*1024)
}
// TestLegacyClientRejected: per-stream flow control is mandatory. A client that
// performs the old session establishment — a Rekey message without the trailing
// feature flags — must be closed by the hub before SessionReady.
func TestLegacyClientRejected(t *testing.T) {
const psk = "e2e-legacyreject"
port := freePort(t)
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
startHub(t, port, psk)
conn, err := net.DialTimeout("tcp", hubAddr, 5*time.Second)
if err != nil {
t.Fatalf("dial hub: %v", err)
}
defer conn.Close()
pskBytes := []byte(psk)
hs := wire.BuildHandshake(767, wire.PSKAddress(pskBytes), 25565, 17)
if _, err := conn.Write(hs); err != nil {
t.Fatalf("handshake: %v", err)
}
fc := wire.NewFramedConn(conn,
wire.CipherFor(pskBytes, wire.DirS2C),
wire.CipherFor(pskBytes, wire.DirC2S),
)
rnd := make([]byte, 16)
if _, err := crand.Read(rnd); err != nil {
t.Fatal(err)
}
// Pre-flow-control Rekey: Magic|RandLen|Rand|Timestamp with no flags.
legacyRekey := wire.NewWriter().
U8(0x01). // control-session magic
VarInt(len(rnd)).
Bytes(rnd).
I64(time.Now().UnixMilli()).
Out()
if err := fc.WriteFrame(legacyRekey); err != nil {
t.Fatalf("rekey: %v", err)
}
_ = conn.SetReadDeadline(time.Now().Add(10 * time.Second))
if payload, err := fc.ReadFrame(); err == nil {
t.Fatalf("expected the hub to close a legacy session, got frame %v", payload)
}
}
+2 -1
View File
@@ -2,5 +2,6 @@
"listen": "0.0.0.0:25565", "listen": "0.0.0.0:25565",
"psk": "change-me-to-a-long-random-passphrase", "psk": "change-me-to-a-long-random-passphrase",
"timestampWindowMs": 30000, "timestampWindowMs": 30000,
"pendingTimeoutMs": 10000 "pendingTimeoutMs": 10000,
"streamWindowBytes": 262144
} }
@@ -11,7 +11,8 @@ public record Config(
int port, int port,
String psk, String psk,
long timestampWindowMs, long timestampWindowMs,
long pendingTimeoutMs long pendingTimeoutMs,
int streamWindowBytes
) { ) {
public static Config load(Path file) throws Exception { public static Config load(Path file) throws Exception {
JsonObject json = new JsonObject(Files.readString(file)); JsonObject json = new JsonObject(Files.readString(file));
@@ -25,11 +26,15 @@ public record Config(
String psk = json.getString("psk"); String psk = json.getString("psk");
if (psk == null || psk.isEmpty()) throw new IllegalArgumentException("psk is required"); if (psk == null || psk.isEmpty()) throw new IllegalArgumentException("psk is required");
int window = json.getInteger("streamWindowBytes", Protocol.DEFAULT_STREAM_WINDOW);
window = Math.max(Protocol.MIN_STREAM_WINDOW, Math.min(Protocol.MAX_STREAM_WINDOW, window));
return new Config( return new Config(
host, host,
port, port,
psk, psk,
json.getLong("timestampWindowMs", 30_000L), json.getLong("timestampWindowMs", 30_000L),
json.getLong("pendingTimeoutMs", 10_000L)); json.getLong("pendingTimeoutMs", 10_000L),
window);
} }
} }
@@ -4,6 +4,7 @@ import io.icybear.redapricot.crypto.Crypto;
import io.icybear.redapricot.net.EncryptedFrames; import io.icybear.redapricot.net.EncryptedFrames;
import io.icybear.redapricot.util.Hex; import io.icybear.redapricot.util.Hex;
import io.icybear.redapricot.util.ProtoReader; import io.icybear.redapricot.util.ProtoReader;
import io.icybear.redapricot.util.ProtoWriter;
import io.icybear.redapricot.util.VarInt; import io.icybear.redapricot.util.VarInt;
import io.vertx.core.buffer.Buffer; import io.vertx.core.buffer.Buffer;
import io.vertx.core.net.NetSocket; import io.vertx.core.net.NetSocket;
@@ -145,6 +146,26 @@ public final class HubConnection {
return; return;
} }
// Mandatory trailing feature flags: the client must offer per-stream flow
// control and advertise its receive window. Anything else is an
// unsupported (pre-flow-control) peer and is rejected.
int flags;
int peerWindow;
try {
flags = r.readVarInt();
peerWindow = r.readVarInt();
} catch (RuntimeException e) {
LOG.warn("{} rekey without feature flags (unsupported client version); closing", id);
frames.close();
return;
}
if ((flags & Protocol.FLAG_STREAM_FC) == 0 || peerWindow <= 0) {
LOG.warn("{} client did not offer per-stream flow control; closing", id);
frames.close();
return;
}
peerWindow = Math.min(peerWindow, Protocol.MAX_STREAM_WINDOW);
// REKEY = Rand || Timestamp(I64 big-endian). Magic is excluded. // REKEY = Rand || Timestamp(I64 big-endian). Magic is excluded.
byte[] rekey = new byte[randLen + 8]; byte[] rekey = new byte[randLen + 8];
System.arraycopy(rand, 0, rekey, 0, randLen); System.arraycopy(rand, 0, rekey, 0, randLen);
@@ -157,7 +178,12 @@ public final class HubConnection {
frames.switchCiphers( frames.switchCiphers(
Crypto.decryptCipher(rekey, Crypto.DIR_C2S), Crypto.decryptCipher(rekey, Crypto.DIR_C2S),
Crypto.encryptCipher(rekey, Crypto.DIR_S2C)); Crypto.encryptCipher(rekey, Crypto.DIR_S2C));
frames.send(new byte[]{(byte) Protocol.CTL_SESSION_READY}); // Echo the accepted flags plus our own receive window.
frames.send(new ProtoWriter()
.u8(Protocol.CTL_SESSION_READY)
.varInt(Protocol.FLAG_STREAM_FC)
.varInt(hub.config.streamWindowBytes())
.toBytes());
if (magic == Protocol.MAGIC_CONTROL) { if (magic == Protocol.MAGIC_CONTROL) {
ControlSession session = new ControlSession(hub, frames, id); ControlSession session = new ControlSession(hub, frames, id);
@@ -165,10 +191,10 @@ public final class HubConnection {
closeCleanup = session::onClose; closeCleanup = session::onClose;
LOG.info("{} control session established", id); LOG.info("{} control session established", id);
} else if (magic == Protocol.MAGIC_WORKER) { } else if (magic == Protocol.MAGIC_WORKER) {
WorkerConn worker = new WorkerConn(hub, frames, id); WorkerConn worker = new WorkerConn(hub, frames, id, peerWindow, hub.config.streamWindowBytes());
frames.setHandler(worker::onFrame); frames.setHandler(worker::onFrame);
closeCleanup = worker::onClose; closeCleanup = worker::onClose;
LOG.info("{} worker conn established", id); LOG.info("{} worker conn established (peer window {})", id, peerWindow);
} else { } else {
LOG.warn("{} bad magic {}; closing", id, magic); LOG.warn("{} bad magic {}; closing", id, magic);
frames.close(); frames.close();
@@ -31,6 +31,16 @@ public final class Protocol {
public static final int MUX_DATA = 0x01; public static final int MUX_DATA = 0x01;
public static final int MUX_FIN = 0x02; public static final int MUX_FIN = 0x02;
public static final int MUX_RST = 0x03; public static final int MUX_RST = 0x03;
public static final int MUX_WND = 0x04; // per-stream flow-control credit grant
// Session-establishment feature flags (trailing VarInt on the Rekey message,
// echoed after the SessionReady type byte when accepted).
public static final int FLAG_STREAM_FC = 0x01;
// Per-stream flow-control window bounds (bytes).
public static final int DEFAULT_STREAM_WINDOW = 256 * 1024;
public static final int MIN_STREAM_WINDOW = 32 * 1024;
public static final int MAX_STREAM_WINDOW = 8 << 20;
// Any redapricot connection // Any redapricot connection
public static final int FRAME_ERROR = 0x7F; public static final int FRAME_ERROR = 0x7F;
@@ -17,20 +17,44 @@ import java.util.Set;
/** /**
* An authenticated worker connection (Magic 0x02). Multiplexes many player * An authenticated worker connection (Magic 0x02). Multiplexes many player
* streams; the client opens streams via SYN(CID) to take over pending players. * streams; the client opens streams via SYN(CID) to take over pending players.
*
* <p>Every stream has a credit window in both directions (PROTOCOL.md §7.3),
* so one slow player only ever stalls its own stream — the shared worker
* socket is never paused because of a single stream.
*/ */
@RequiredArgsConstructor @RequiredArgsConstructor
public final class WorkerConn { public final class WorkerConn {
private static final Logger LOG = LogManager.getLogger("redapricot.worker"); private static final Logger LOG = LogManager.getLogger("redapricot.worker");
/** Cap on a single DATA frame so no stream monopolizes the shared link for long. */
private static final int CHUNK = 32 * 1024;
private final Hub hub; private final Hub hub;
private final EncryptedFrames frames; private final EncryptedFrames frames;
private final String id; private final String id;
private final Map<Integer, NetSocket> streams = new HashMap<>(); private final int sendWndInit; // client's advertised per-stream receive window (our send budget)
private final int recvWndInit; // our advertised per-stream receive window (basis for credit grants)
// Backpressure for the single shared worker socket, arbitrated across all streams. private final Map<Integer, StreamState> streams = new HashMap<>();
private final Set<NetSocket> upstreamPaused = new HashSet<>(); // players parked until the worker write queue drains
private final Set<Integer> downstreamBlocked = new HashSet<>(); // sids whose player write queue is full; non-empty => worker read paused // Aggregate backpressure for the single shared worker socket: players parked
private boolean workerDrainArmed = false; // whether the worker socket's single drainHandler is set // until its write queue drains. Per-stream fairness is the credit windows'
// job; this only reacts to the whole pipe being congested.
private final Set<StreamState> upstreamPaused = new HashSet<>();
private boolean workerDrainArmed = false; // whether the worker socket's single drainHandler is set
/** Per-stream flow-control bookkeeping. */
private final class StreamState {
final NetSocket player;
int sendWnd = sendWndInit; // budget for player -> client DATA
Buffer pendingUp; // player bytes awaiting send window (player is paused meanwhile)
boolean pausedForWindow;
int credited; // client -> player bytes flushed but not yet granted back
StreamState(NetSocket player) {
this.player = player;
}
}
public void onFrame(byte[] payload) { public void onFrame(byte[] payload) {
ProtoReader r = new ProtoReader(payload); ProtoReader r = new ProtoReader(payload);
@@ -39,6 +63,7 @@ public final class WorkerConn {
switch (type) { switch (type) {
case Protocol.MUX_SYN -> handleSyn(sid, r.readBytes(Protocol.CID_LEN)); case Protocol.MUX_SYN -> handleSyn(sid, r.readBytes(Protocol.CID_LEN));
case Protocol.MUX_DATA -> handleData(sid, r.readBuffer(r.remaining())); case Protocol.MUX_DATA -> handleData(sid, r.readBuffer(r.remaining()));
case Protocol.MUX_WND -> handleWnd(sid, r.readVarInt());
case Protocol.MUX_FIN, Protocol.MUX_RST -> closeStream(sid); case Protocol.MUX_FIN, Protocol.MUX_RST -> closeStream(sid);
case Protocol.FRAME_ERROR -> LOG.warn("worker {} error frame", id); case Protocol.FRAME_ERROR -> LOG.warn("worker {} error frame", id);
default -> LOG.warn("worker {} unknown mux type {}", id, type); default -> LOG.warn("worker {} unknown mux type {}", id, type);
@@ -53,72 +78,126 @@ public final class WorkerConn {
return; return;
} }
NetSocket player = p.getSocket(); NetSocket player = p.getSocket();
streams.put(sid, player); StreamState st = new StreamState(player);
streams.put(sid, st);
// From now on the player socket belongs to this stream. // From now on the player socket belongs to this stream.
player.handler(buf -> { player.handler(buf -> {
sendData(sid, buf.getBytes()); sendUpstream(sid, st, buf);
if (frames.writeQueueFull()) { checkAggregate(st);
player.pause();
upstreamPaused.add(player);
armWorkerDrain();
}
}); });
player.closeHandler(v -> onPlayerGone(sid, player)); player.closeHandler(v -> onPlayerGone(sid, st));
player.exceptionHandler(t -> onPlayerGone(sid, player)); player.exceptionHandler(t -> onPlayerGone(sid, st));
// Forward the buffered handshake (and any pipelined bytes), then resume. // Forward the buffered handshake (and any pipelined bytes), then resume.
sendData(sid, p.getBuffered().getBytes()); sendUpstream(sid, st, p.getBuffered());
player.resume(); if (!st.pausedForWindow) player.resume();
checkAggregate(st);
LOG.info("worker {} stream {} bound to {}", id, sid, p.getPattern()); LOG.info("worker {} stream {} bound to {}", id, sid, p.getPattern());
} }
private void handleData(int sid, Buffer data) { /**
NetSocket player = streams.get(sid); * Send player bytes to the client, chunked and clipped to the stream window;
if (player == null) return; * the overflow is parked in {@code pendingUp} and the player socket paused
player.write(data); * until the client grants more credit.
// A slow player pauses the shared worker read side; the block set lets us resume only */
// once every blocked player has drained (and release it if a player disconnects meanwhile). private void sendUpstream(int sid, StreamState st, Buffer buf) {
if (player.writeQueueFull() && downstreamBlocked.add(sid)) { if (st.pendingUp != null) { // still waiting for window; keep ordering
if (downstreamBlocked.size() == 1) frames.socket().pause(); st.pendingUp.appendBuffer(buf);
player.drainHandler(v -> unblockDownstream(sid)); return;
} }
int off = drainUpstream(sid, st, buf, 0);
if (off < buf.length()) {
st.pendingUp = buf.getBuffer(off, buf.length());
if (!st.pausedForWindow) {
st.pausedForWindow = true;
st.player.pause();
}
}
}
/** Send from {@code buf[off..]} within the stream window, chunked; returns the new offset. */
private int drainUpstream(int sid, StreamState st, Buffer buf, int off) {
while (off < buf.length() && st.sendWnd > 0) {
int n = Math.min(Math.min(CHUNK, st.sendWnd), buf.length() - off);
sendData(sid, buf.getBytes(off, off + n));
st.sendWnd -= n;
off += n;
}
return off;
}
/** The client granted {@code delta} more bytes of credit on a stream. */
private void handleWnd(int sid, int delta) {
StreamState st = streams.get(sid);
if (st == null || delta <= 0) return;
st.sendWnd += delta;
if (st.pendingUp != null) {
Buffer pending = st.pendingUp;
int off = drainUpstream(sid, st, pending, 0);
st.pendingUp = off >= pending.length() ? null : pending.getBuffer(off, pending.length());
}
if (st.pendingUp == null && st.pausedForWindow) {
st.pausedForWindow = false;
if (!upstreamPaused.contains(st)) st.player.resume();
}
checkAggregate(st);
}
private void handleData(int sid, Buffer data) {
StreamState st = streams.get(sid);
if (st == null) return;
// Never pause the shared socket: the client bounds what it sends per
// stream to our advertised window, so a slow player only piles up a
// bounded amount in its own write queue; credit is granted back as the
// write completes (i.e. the bytes reached the player socket).
int len = data.length();
st.player.write(data).onComplete(ar -> {
if (ar.failed() || frames.isClosed() || streams.get(sid) != st) return;
st.credited += len;
if (st.credited * 2 >= recvWndInit) {
int delta = st.credited;
st.credited = 0;
sendWnd(sid, delta);
}
});
} }
private void closeStream(int sid) { private void closeStream(int sid) {
NetSocket player = streams.remove(sid); StreamState st = streams.remove(sid);
unblockDownstream(sid); if (st != null) {
if (player != null) { upstreamPaused.remove(st);
upstreamPaused.remove(player); st.player.close();
player.close();
} }
} }
/** Register (once) the shared worker socket's single drain handler; on drain, wake every parked player. */ /** Park the player if the shared worker socket's write queue is congested. */
private void checkAggregate(StreamState st) {
if (frames.writeQueueFull() && upstreamPaused.add(st)) {
st.player.pause();
armWorkerDrain();
}
}
/** Register (once) the shared worker socket's single drain handler; on drain, wake parked players. */
private void armWorkerDrain() { private void armWorkerDrain() {
if (workerDrainArmed) return; if (workerDrainArmed) return;
workerDrainArmed = true; workerDrainArmed = true;
frames.socket().drainHandler(v -> { frames.socket().drainHandler(v -> {
workerDrainArmed = false; workerDrainArmed = false;
if (upstreamPaused.isEmpty()) return; if (upstreamPaused.isEmpty()) return;
NetSocket[] parked = upstreamPaused.toArray(new NetSocket[0]); StreamState[] parked = upstreamPaused.toArray(new StreamState[0]);
upstreamPaused.clear(); upstreamPaused.clear();
for (NetSocket pl : parked) pl.resume(); for (StreamState st : parked) {
if (!st.pausedForWindow) st.player.resume();
}
}); });
} }
/** A blocked player drained or vanished: drop its downstream block, resuming the shared worker read side when none remain. */
private void unblockDownstream(int sid) {
if (downstreamBlocked.remove(sid) && downstreamBlocked.isEmpty()) {
frames.socket().resume();
}
}
/** The player side of a stream vanished: drop it from every table, release any backpressure it held, and FIN the peer if still live. */ /** The player side of a stream vanished: drop it from every table, release any backpressure it held, and FIN the peer if still live. */
private void onPlayerGone(int sid, NetSocket player) { private void onPlayerGone(int sid, StreamState st) {
boolean wasLive = streams.remove(sid) != null; boolean wasLive = streams.remove(sid) == st;
upstreamPaused.remove(player); upstreamPaused.remove(st);
unblockDownstream(sid);
if (wasLive) sendFin(sid); if (wasLive) sendFin(sid);
} }
@@ -134,10 +213,13 @@ public final class WorkerConn {
frames.send(new ProtoWriter().u8(Protocol.MUX_RST).varInt(sid).toBytes()); frames.send(new ProtoWriter().u8(Protocol.MUX_RST).varInt(sid).toBytes());
} }
private void sendWnd(int sid, int delta) {
frames.send(new ProtoWriter().u8(Protocol.MUX_WND).varInt(sid).varInt(delta).toBytes());
}
public void onClose() { public void onClose() {
for (NetSocket player : streams.values()) player.close(); for (StreamState st : streams.values()) st.player.close();
streams.clear(); streams.clear();
downstreamBlocked.clear();
upstreamPaused.clear(); upstreamPaused.clear();
LOG.info("worker {} closed", id); LOG.info("worker {} closed", id);
} }
@@ -77,7 +77,7 @@ class CryptoCodecTest {
/** A Hub whose event loop is never touched (register/match/normalize use no Vert.x state). */ /** A Hub whose event loop is never touched (register/match/normalize use no Vert.x state). */
private static Hub testHub() { private static Hub testHub() {
return new Hub(null, new Config("0.0.0.0", 25565, "test-psk", 30_000L, 10_000L)); return new Hub(null, new Config("0.0.0.0", 25565, "test-psk", 30_000L, 10_000L, Protocol.DEFAULT_STREAM_WINDOW));
} }
private static ControlSession testSession(Hub hub, String id) { private static ControlSession testSession(Hub hub, String id) {