add credit based mux window
This commit is contained in:
+54
-7
@@ -140,8 +140,16 @@ Magic : u8 # 0x01 = control session, 0x02 = worker conn
|
|||||||
RandLen : VarInt # 8 ≤ RandLen ≤ 64
|
RandLen : VarInt # 8 ≤ RandLen ≤ 64
|
||||||
Rand : Bytes[RandLen] # cryptographically random
|
Rand : Bytes[RandLen] # cryptographically random
|
||||||
Timestamp : I64 # client's epoch milliseconds
|
Timestamp : I64 # client's epoch milliseconds
|
||||||
|
Flags : VarInt # feature flags; bit 0x01 (STREAM_FC) MUST be set
|
||||||
|
RecvWindow: VarInt # client's per-stream receive window, bytes (§7.3)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
`Flags` is a bitfield of features. Bit `0x01` (STREAM_FC) declares
|
||||||
|
**per-stream flow control** (§7.3) and is mandatory: `RecvWindow` advertises
|
||||||
|
the client's per-stream receive window in bytes and must be positive. The hub
|
||||||
|
closes the connection if the flag is missing, `RecvWindow` is absent or
|
||||||
|
non-positive, or the fields are malformed.
|
||||||
|
|
||||||
The hub:
|
The hub:
|
||||||
|
|
||||||
1. Decrypts frame 1 with Phase A.
|
1. Decrypts frame 1 with Phase A.
|
||||||
@@ -158,9 +166,13 @@ frame (both directions) is Phase B, counters reset to 0.
|
|||||||
The hub then sends one Phase-B frame to confirm success:
|
The hub then sends one Phase-B frame to confirm success:
|
||||||
|
|
||||||
```
|
```
|
||||||
SessionReady : payload = [ 0x00 ]
|
SessionReady : payload = [ 0x00, Flags: VarInt, RecvWindow: VarInt ]
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The hub echoes the accepted flags (STREAM_FC set) followed by its own
|
||||||
|
per-stream receive window. A client must reject a SessionReady without the
|
||||||
|
STREAM_FC flag or without a positive window (an unsupported hub).
|
||||||
|
|
||||||
A hub that rejects the session simply closes the TCP connection (optionally
|
A hub that rejects the session simply closes the TCP connection (optionally
|
||||||
after a Phase-B `Error` frame, §6). After `SessionReady`:
|
after a Phase-B `Error` frame, §6). After `SessionReady`:
|
||||||
|
|
||||||
@@ -258,6 +270,7 @@ Data : Bytes[...] # remainder of the frame payload
|
|||||||
| `0x01` | DATA | both | raw tunneled bytes for the stream. |
|
| `0x01` | DATA | both | raw tunneled bytes for the stream. |
|
||||||
| `0x02` | FIN | both | *(empty)* — graceful close of the stream (both directions). This is the "disconnect" the hub sends when the player leaves. |
|
| `0x02` | FIN | both | *(empty)* — graceful close of the stream (both directions). This is the "disconnect" the hub sends when the player leaves. |
|
||||||
| `0x03` | RST | both | *(optional 1 byte reason)* — abnormal close (e.g. CID unknown/expired, destination dial failed). |
|
| `0x03` | RST | both | *(optional 1 byte reason)* — abnormal close (e.g. CID unknown/expired, destination dial failed). |
|
||||||
|
| `0x04` | WND | both | `Delta: VarInt` — flow-control credit grant (§7.3). |
|
||||||
|
|
||||||
There is no explicit SYN-ACK: success is implied by the hub forwarding the
|
There is no explicit SYN-ACK: success is implied by the hub forwarding the
|
||||||
buffered Handshake as the stream's first `DATA`; failure is an `RST`.
|
buffered Handshake as the stream's first `DATA`; failure is an `RST`.
|
||||||
@@ -294,11 +307,33 @@ configurable, `1..8`). To place a new stream:
|
|||||||
closes the destination. When the destination closes, the client sends `FIN`;
|
closes the destination. When the destination closes, the client sends `FIN`;
|
||||||
the hub closes the player socket. `RST` is treated the same way (hard close).
|
the hub closes the player socket. `RST` is treated the same way (hard close).
|
||||||
|
|
||||||
Data on a worker conn is subject to that TCP connection's back-pressure. Each
|
Data on a worker conn is subject to that TCP connection's back-pressure for
|
||||||
stream has a bounded outbound queue on the receiving side; overflow resets the
|
its **aggregate** bandwidth; *per-stream* fairness is governed by the credit
|
||||||
stream (`RST`). (This is a deliberate simplification — no per-stream credit
|
windows of §7.3.
|
||||||
windows — acceptable for the interactive, low-throughput Minecraft handshake +
|
|
||||||
gameplay traffic pattern.)
|
### 7.3 Per-stream flow control
|
||||||
|
|
||||||
|
Every stream carries an independent credit window per direction:
|
||||||
|
|
||||||
|
* Each side advertised its **receive window** W (bytes) at session setup. A
|
||||||
|
sender may have at most W un-credited DATA bytes outstanding per stream; the
|
||||||
|
initial budget is W, spent as DATA is sent (`Data` length only — SYN/FIN/RST
|
||||||
|
frames are free) starting with the very first DATA on the stream (including
|
||||||
|
the hub's forwarded handshake).
|
||||||
|
* The receiver returns credit with `WND(Delta)` once bytes are **delivered to
|
||||||
|
the terminal socket** (written to the player / destination connection), not
|
||||||
|
when they are merely buffered. Receivers should batch grants (the reference
|
||||||
|
implementations send one `WND` per W/2 bytes consumed).
|
||||||
|
* A sender whose window is exhausted pauses reading **that stream's source
|
||||||
|
socket only**; the shared worker conn is never paused because of a single
|
||||||
|
stream. A receiver that observes more than W un-credited bytes on a stream
|
||||||
|
may reset it (`RST`) as a protocol violation.
|
||||||
|
* Senders should also cap individual DATA payloads (the reference
|
||||||
|
implementations use 32 KiB) so one stream cannot monopolize the link for a
|
||||||
|
full 1-MiB frame.
|
||||||
|
|
||||||
|
Both windows may differ (each side enforces the one its peer advertised).
|
||||||
|
`Delta` must be positive; a `WND` for an unknown stream id is ignored.
|
||||||
|
|
||||||
## 8. HAProxy protocol v2 (optional)
|
## 8. HAProxy protocol v2 (optional)
|
||||||
|
|
||||||
@@ -327,10 +362,14 @@ big-endian.
|
|||||||
"listen": "0.0.0.0:25565",
|
"listen": "0.0.0.0:25565",
|
||||||
"psk": "change-me",
|
"psk": "change-me",
|
||||||
"timestampWindowMs": 30000,
|
"timestampWindowMs": 30000,
|
||||||
"pendingTimeoutMs": 10000
|
"pendingTimeoutMs": 10000,
|
||||||
|
"streamWindowBytes": 262144
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
`streamWindowBytes` (optional, default 262144, clamped to [32768, 8388608]) is
|
||||||
|
the hub's advertised per-stream receive window (§7.3).
|
||||||
|
|
||||||
### 9.2 Client — JSON
|
### 9.2 Client — JSON
|
||||||
|
|
||||||
```json
|
```json
|
||||||
@@ -339,12 +378,16 @@ big-endian.
|
|||||||
"psk": "change-me",
|
"psk": "change-me",
|
||||||
"maxConn": 4,
|
"maxConn": 4,
|
||||||
"pingIntervalMs": 20000,
|
"pingIntervalMs": 20000,
|
||||||
|
"streamWindowBytes": 262144,
|
||||||
"mappings": [
|
"mappings": [
|
||||||
{ "pattern": "mc\\.example\\.com", "destination": "127.0.0.1:25566", "proxyProtocol": true }
|
{ "pattern": "mc\\.example\\.com", "destination": "127.0.0.1:25566", "proxyProtocol": true }
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
`streamWindowBytes` (optional, default 262144, clamped to [32768, 8388608]) is
|
||||||
|
the client's advertised per-stream receive window (§7.3).
|
||||||
|
|
||||||
Each `pattern` is a regular expression (§5.1) matched against the whole
|
Each `pattern` is a regular expression (§5.1) matched against the whole
|
||||||
normalized player hostname, case-insensitively. Escape literal dots (`mc\.example\.com`,
|
normalized player hostname, case-insensitively. Escape literal dots (`mc\.example\.com`,
|
||||||
which is `mc\\.example\\.com` in JSON); an unescaped `.` matches any character.
|
which is `mc\\.example\\.com` in JSON); an unescaped `.` matches any character.
|
||||||
@@ -369,4 +412,8 @@ subdomain or `(alpha|beta)\.mc\.net` for a fixed set.
|
|||||||
| max frame payload | 1 MiB |
|
| max frame payload | 1 MiB |
|
||||||
| saturation threshold | active streams `> 8` |
|
| saturation threshold | active streams `> 8` |
|
||||||
| max worker conns | `max_conn ∈ [1,8]` |
|
| max worker conns | `max_conn ∈ [1,8]` |
|
||||||
|
| feature flag: per-stream flow control | `0x01` |
|
||||||
|
| stream window default / bounds | 256 KiB, clamped to [32 KiB, 8 MiB] |
|
||||||
|
| WND grant batching (reference) | one grant per window/2 consumed |
|
||||||
|
| DATA chunk cap (reference) | 32 KiB |
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -202,7 +202,9 @@ forwarding and case-insensitive matching, regex wildcard pattern routing,
|
|||||||
multi-megabyte transfers, concurrent
|
multi-megabyte transfers, concurrent
|
||||||
streams spreading across multiple worker connections, HAProxy v2 source-address
|
streams spreading across multiple worker connections, HAProxy v2 source-address
|
||||||
propagation, player- and destination-initiated disconnect propagation, wrong-PSK
|
propagation, player- and destination-initiated disconnect propagation, wrong-PSK
|
||||||
rejection, and dropping of unmatched hostnames. The Go and Java crypto layers are
|
rejection, dropping of unmatched hostnames, stream isolation under a slow
|
||||||
|
player and under a slow destination (no head-of-line blocking), and rejection
|
||||||
|
of pre-flow-control peers. The Go and Java crypto layers are
|
||||||
independently pinned to the same SHA3-224 test vector so they cannot silently
|
independently pinned to the same SHA3-224 test vector so they cannot silently
|
||||||
drift apart.
|
drift apart.
|
||||||
|
|
||||||
@@ -212,10 +214,12 @@ drift apart.
|
|||||||
ChaCha20-encrypted (no AEAD tag) to minimize overhead. This protects against
|
ChaCha20-encrypted (no AEAD tag) to minimize overhead. This protects against
|
||||||
casual sniffing, not a determined active attacker (see the note at the top of
|
casual sniffing, not a determined active attacker (see the note at the top of
|
||||||
[PROTOCOL.md](PROTOCOL.md) and [docs/architecture.md](docs/architecture.md) §8).
|
[PROTOCOL.md](PROTOCOL.md) and [docs/architecture.md](docs/architecture.md) §8).
|
||||||
* **No per-stream flow control.** Multiplexing relies on TCP back-pressure per
|
* **Per-stream flow control.** Each stream has credit-based windows in both
|
||||||
worker connection, so one very slow stream can head-of-line-block others on
|
directions (windows exchanged at session setup, default 256 KiB), so a slow
|
||||||
the same connection. Raising `maxConn` spreads load. Fine for interactive
|
player or slow destination jams only its own stream at a bounded buffer — no
|
||||||
Minecraft traffic; not a general-purpose high-throughput mux.
|
application-level head-of-line blocking between streams. What remains is
|
||||||
|
TCP-level HOL (packet loss stalls a whole worker connection briefly);
|
||||||
|
raising `maxConn` spreads that.
|
||||||
* **Single hub event loop.** The hub deploys one Vert.x verticle, so all state
|
* **Single hub event loop.** The hub deploys one Vert.x verticle, so all state
|
||||||
is confined to one event loop (no locking). Throughput is bounded by one core;
|
is confined to one event loop (no locking). Throughput is bounded by one core;
|
||||||
ample for hundreds of players, not designed for tens of thousands.
|
ample for hundreds of players, not designed for tens of thousands.
|
||||||
|
|||||||
+45
-14
@@ -24,6 +24,8 @@ type Client struct {
|
|||||||
mappings map[string]Mapping // normalized pattern -> mapping
|
mappings map[string]Mapping // normalized pattern -> mapping
|
||||||
pool *WorkerPool
|
pool *WorkerPool
|
||||||
|
|
||||||
|
streamWnd int // our advertised per-stream receive window (bytes)
|
||||||
|
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
ctrl *wire.FramedConn
|
ctrl *wire.FramedConn
|
||||||
}
|
}
|
||||||
@@ -44,16 +46,32 @@ func New(cfg *Config) *Client {
|
|||||||
for _, m := range cfg.Mappings {
|
for _, m := range cfg.Mappings {
|
||||||
c.mappings[NormalizeAddress(m.Pattern)] = m
|
c.mappings[NormalizeAddress(m.Pattern)] = m
|
||||||
}
|
}
|
||||||
|
c.streamWnd = clampWindow(cfg.StreamWindowBytes)
|
||||||
c.pool = newWorkerPool(c, cfg.MaxConn)
|
c.pool = newWorkerPool(c, cfg.MaxConn)
|
||||||
return c
|
return c
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func clampWindow(w int) int {
|
||||||
|
if w <= 0 {
|
||||||
|
return DefaultStreamWindow
|
||||||
|
}
|
||||||
|
if w < MinStreamWindow {
|
||||||
|
return MinStreamWindow
|
||||||
|
}
|
||||||
|
if w > MaxStreamWindow {
|
||||||
|
return MaxStreamWindow
|
||||||
|
}
|
||||||
|
return w
|
||||||
|
}
|
||||||
|
|
||||||
// dialSession opens a TCP connection, performs the Intent-17 handshake, the
|
// dialSession opens a TCP connection, performs the Intent-17 handshake, the
|
||||||
// Phase-A rekey, and reads SessionReady, returning an established frame conn.
|
// Phase-A rekey, and reads SessionReady, returning an established frame conn
|
||||||
func (c *Client) dialSession(magic byte) (*wire.FramedConn, error) {
|
// and the hub's advertised per-stream receive window. Per-stream flow control
|
||||||
|
// is mandatory: a hub that does not echo the STREAM_FC flag is rejected.
|
||||||
|
func (c *Client) dialSession(magic byte) (fc *wire.FramedConn, peerWnd int, err error) {
|
||||||
conn, err := net.DialTimeout("tcp", c.cfg.Server, 10*time.Second)
|
conn, err := net.DialTimeout("tcp", c.cfg.Server, 10*time.Second)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, 0, err
|
||||||
}
|
}
|
||||||
if tcp, ok := conn.(*net.TCPConn); ok {
|
if tcp, ok := conn.(*net.TCPConn); ok {
|
||||||
_ = tcp.SetNoDelay(true)
|
_ = tcp.SetNoDelay(true)
|
||||||
@@ -68,24 +86,26 @@ func (c *Client) dialSession(magic byte) (*wire.FramedConn, error) {
|
|||||||
// 1. plaintext Minecraft Handshake, Intent 17, address = hex(SHA3-224(PSK)).
|
// 1. plaintext Minecraft Handshake, Intent 17, address = hex(SHA3-224(PSK)).
|
||||||
hs := wire.BuildHandshake(ProtocolVersion, c.pskAddr, c.serverPort, IntentRedapricot)
|
hs := wire.BuildHandshake(ProtocolVersion, c.pskAddr, c.serverPort, IntentRedapricot)
|
||||||
if _, err := conn.Write(hs); err != nil {
|
if _, err := conn.Write(hs); err != nil {
|
||||||
return nil, err
|
return nil, 0, err
|
||||||
}
|
}
|
||||||
|
|
||||||
// 2. Phase-A ciphers derived from the PSK.
|
// 2. Phase-A ciphers derived from the PSK.
|
||||||
fc := wire.NewFramedConn(conn,
|
fc = wire.NewFramedConn(conn,
|
||||||
wire.CipherFor(c.pskBytes, wire.DirS2C), // in: server -> client
|
wire.CipherFor(c.pskBytes, wire.DirS2C), // in: server -> client
|
||||||
wire.CipherFor(c.pskBytes, wire.DirC2S), // out: client -> server
|
wire.CipherFor(c.pskBytes, wire.DirC2S), // out: client -> server
|
||||||
)
|
)
|
||||||
|
|
||||||
// 3. Rekey frame (Phase A).
|
// 3. Rekey frame (Phase A), including the mandatory feature flags and our
|
||||||
|
// per-stream receive window.
|
||||||
rnd := make([]byte, 16)
|
rnd := make([]byte, 16)
|
||||||
if _, err := crand.Read(rnd); err != nil {
|
if _, err := crand.Read(rnd); err != nil {
|
||||||
return nil, err
|
return nil, 0, err
|
||||||
}
|
}
|
||||||
ts := time.Now().UnixMilli()
|
ts := time.Now().UnixMilli()
|
||||||
rekeyMsg := wire.NewWriter().U8(magic).VarInt(len(rnd)).Bytes(rnd).I64(ts).Out()
|
rekeyMsg := wire.NewWriter().U8(magic).VarInt(len(rnd)).Bytes(rnd).I64(ts).
|
||||||
|
VarInt(FlagStreamFC).VarInt(c.streamWnd).Out()
|
||||||
if err := fc.WriteFrame(rekeyMsg); err != nil {
|
if err := fc.WriteFrame(rekeyMsg); err != nil {
|
||||||
return nil, err
|
return nil, 0, err
|
||||||
}
|
}
|
||||||
|
|
||||||
// 4. Switch to Phase-B ciphers: REKEY = Rand || Timestamp(I64 BE).
|
// 4. Switch to Phase-B ciphers: REKEY = Rand || Timestamp(I64 BE).
|
||||||
@@ -99,16 +119,26 @@ func (c *Client) dialSession(magic byte) (*wire.FramedConn, error) {
|
|||||||
wire.CipherFor(rekey, wire.DirC2S),
|
wire.CipherFor(rekey, wire.DirC2S),
|
||||||
)
|
)
|
||||||
|
|
||||||
// 5. SessionReady.
|
// 5. SessionReady: the type byte followed by the hub's accepted flags and
|
||||||
|
// its per-stream receive window. Both are required.
|
||||||
payload, err := fc.ReadFrame()
|
payload, err := fc.ReadFrame()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, 0, err
|
||||||
}
|
}
|
||||||
if len(payload) < 1 || payload[0] != CtlSessionReady {
|
if len(payload) < 1 || payload[0] != CtlSessionReady {
|
||||||
return nil, fmt.Errorf("expected SessionReady, got %v", payload)
|
return nil, 0, fmt.Errorf("expected SessionReady, got %v", payload)
|
||||||
|
}
|
||||||
|
r := wire.NewReader(payload[1:])
|
||||||
|
flags, ferr := r.VarInt()
|
||||||
|
hubWnd, werr := r.VarInt()
|
||||||
|
if ferr != nil || werr != nil || flags&FlagStreamFC == 0 || hubWnd <= 0 {
|
||||||
|
return nil, 0, fmt.Errorf("hub did not accept per-stream flow control (unsupported hub version?)")
|
||||||
|
}
|
||||||
|
if hubWnd > MaxStreamWindow {
|
||||||
|
hubWnd = MaxStreamWindow
|
||||||
}
|
}
|
||||||
ok = true
|
ok = true
|
||||||
return fc, nil
|
return fc, hubWnd, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// Start establishes the control session and registers all patterns. It returns
|
// Start establishes the control session and registers all patterns. It returns
|
||||||
@@ -119,7 +149,7 @@ func (c *Client) Start(ctx context.Context) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (c *Client) connectControl(ctx context.Context) error {
|
func (c *Client) connectControl(ctx context.Context) error {
|
||||||
fc, err := c.dialSession(MagicControl)
|
fc, _, err := c.dialSession(MagicControl)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("control connect: %w", err)
|
return fmt.Errorf("control connect: %w", err)
|
||||||
}
|
}
|
||||||
@@ -233,6 +263,7 @@ func (c *Client) handleControlRequest(cid []byte, pattern, ip string, port int,
|
|||||||
st := newStream(wc, sid, cid, mapping, ip, port)
|
st := newStream(wc, sid, cid, mapping, ip, port)
|
||||||
wc.registerStream(sid, st)
|
wc.registerStream(sid, st)
|
||||||
wc.sendSyn(sid, cid)
|
wc.sendSyn(sid, cid)
|
||||||
|
go st.writeLoop()
|
||||||
go st.run()
|
go st.run()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -3,6 +3,7 @@
|
|||||||
"psk": "change-me-to-a-long-random-passphrase",
|
"psk": "change-me-to-a-long-random-passphrase",
|
||||||
"maxConn": 4,
|
"maxConn": 4,
|
||||||
"pingIntervalMs": 20000,
|
"pingIntervalMs": 20000,
|
||||||
|
"streamWindowBytes": 262144,
|
||||||
"mappings": [
|
"mappings": [
|
||||||
{
|
{
|
||||||
"pattern": "mc\\.example\\.com",
|
"pattern": "mc\\.example\\.com",
|
||||||
|
|||||||
@@ -29,10 +29,24 @@ const (
|
|||||||
MuxData = 0x01
|
MuxData = 0x01
|
||||||
MuxFin = 0x02
|
MuxFin = 0x02
|
||||||
MuxRst = 0x03
|
MuxRst = 0x03
|
||||||
|
MuxWnd = 0x04
|
||||||
|
|
||||||
FrameError = 0x7F
|
FrameError = 0x7F
|
||||||
|
|
||||||
SaturationThreshold = 8
|
SaturationThreshold = 8
|
||||||
|
|
||||||
|
// Session-establishment feature flags (trailing VarInt on the Rekey message).
|
||||||
|
FlagStreamFC = 0x01
|
||||||
|
|
||||||
|
// Per-stream flow-control window bounds (bytes). The advertised window is the
|
||||||
|
// receiver's promise of how much un-credited DATA it will buffer per stream.
|
||||||
|
DefaultStreamWindow = 256 * 1024
|
||||||
|
MinStreamWindow = 32 * 1024
|
||||||
|
MaxStreamWindow = 8 << 20
|
||||||
|
|
||||||
|
// DataChunkSize caps a single DATA frame's payload so no stream monopolizes
|
||||||
|
// the shared worker connection for long.
|
||||||
|
DataChunkSize = 32 * 1024
|
||||||
)
|
)
|
||||||
|
|
||||||
// Mapping routes a registered pattern to a real destination.
|
// Mapping routes a registered pattern to a real destination.
|
||||||
@@ -48,6 +62,7 @@ type Config struct {
|
|||||||
PSK string `json:"psk"`
|
PSK string `json:"psk"`
|
||||||
MaxConn int `json:"maxConn"`
|
MaxConn int `json:"maxConn"`
|
||||||
PingIntervalMs int `json:"pingIntervalMs"`
|
PingIntervalMs int `json:"pingIntervalMs"`
|
||||||
|
StreamWindowBytes int `json:"streamWindowBytes"` // per-stream receive window; 0 = default
|
||||||
Mappings []Mapping `json:"mappings"`
|
Mappings []Mapping `json:"mappings"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+160
-30
@@ -55,18 +55,21 @@ func (p *WorkerPool) Allocate() (*WorkerConn, int, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (p *WorkerPool) dialWorker() (*WorkerConn, error) {
|
func (p *WorkerPool) dialWorker() (*WorkerConn, error) {
|
||||||
fc, err := p.client.dialSession(MagicWorker)
|
fc, peerWnd, err := p.client.dialSession(MagicWorker)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
wc := &WorkerConn{
|
wc := &WorkerConn{
|
||||||
pool: p,
|
pool: p,
|
||||||
fc: fc,
|
fc: fc,
|
||||||
|
sendWndInit: peerWnd,
|
||||||
|
recvWndInit: p.client.streamWnd,
|
||||||
streams: make(map[int]*Stream),
|
streams: make(map[int]*Stream),
|
||||||
nextSid: 1,
|
nextSid: 1,
|
||||||
}
|
}
|
||||||
go wc.readLoop()
|
go wc.readLoop()
|
||||||
log.Printf("opened worker conn (#%d in pool)", len(p.conns)+1)
|
log.Printf("opened worker conn (#%d in pool, send window %d, recv window %d)",
|
||||||
|
len(p.conns)+1, wc.sendWndInit, wc.recvWndInit)
|
||||||
return wc, nil
|
return wc, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -101,6 +104,9 @@ type WorkerConn struct {
|
|||||||
pool *WorkerPool
|
pool *WorkerPool
|
||||||
fc *wire.FramedConn
|
fc *wire.FramedConn
|
||||||
|
|
||||||
|
sendWndInit int // hub's advertised per-stream receive window (our send budget)
|
||||||
|
recvWndInit int // our advertised per-stream receive window (bounds each recv queue)
|
||||||
|
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
streams map[int]*Stream
|
streams map[int]*Stream
|
||||||
nextSid int
|
nextSid int
|
||||||
@@ -140,6 +146,9 @@ func (wc *WorkerConn) removeStream(sid int) *Stream {
|
|||||||
return st
|
return st
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// readLoop dispatches inbound mux frames. It must never block on a stream's
|
||||||
|
// destination: DATA is only enqueued (the per-stream writeLoop does the actual
|
||||||
|
// destination writes), so one slow destination cannot stall other streams.
|
||||||
func (wc *WorkerConn) readLoop() {
|
func (wc *WorkerConn) readLoop() {
|
||||||
for {
|
for {
|
||||||
payload, err := wc.fc.ReadFrame()
|
payload, err := wc.fc.ReadFrame()
|
||||||
@@ -160,9 +169,20 @@ func (wc *WorkerConn) readLoop() {
|
|||||||
if st := wc.getStream(sid); st != nil {
|
if st := wc.getStream(sid); st != nil {
|
||||||
st.deliverFromHub(r.Remaining())
|
st.deliverFromHub(r.Remaining())
|
||||||
}
|
}
|
||||||
case MuxFin, MuxRst:
|
case MuxWnd:
|
||||||
|
if delta, err := r.VarInt(); err == nil && delta > 0 {
|
||||||
|
if st := wc.getStream(sid); st != nil {
|
||||||
|
st.grantSendWnd(delta)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
case MuxFin:
|
||||||
|
// Graceful: drain what is already queued to the destination first.
|
||||||
if st := wc.removeStream(sid); st != nil {
|
if st := wc.removeStream(sid); st != nil {
|
||||||
st.shutdown(false)
|
st.gracefulFin()
|
||||||
|
}
|
||||||
|
case MuxRst:
|
||||||
|
if st := wc.removeStream(sid); st != nil {
|
||||||
|
st.teardown(false)
|
||||||
}
|
}
|
||||||
default:
|
default:
|
||||||
log.Printf("worker: unknown mux type %d", ftype)
|
log.Printf("worker: unknown mux type %d", ftype)
|
||||||
@@ -178,7 +198,7 @@ func (wc *WorkerConn) readLoop() {
|
|||||||
wc.streams = make(map[int]*Stream)
|
wc.streams = make(map[int]*Stream)
|
||||||
wc.mu.Unlock()
|
wc.mu.Unlock()
|
||||||
for _, st := range streams {
|
for _, st := range streams {
|
||||||
st.shutdown(false)
|
st.teardown(false)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -198,7 +218,16 @@ func (wc *WorkerConn) sendRst(sid int) {
|
|||||||
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxRst).VarInt(sid).Out())
|
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxRst).VarInt(sid).Out())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (wc *WorkerConn) sendWndUpdate(sid, delta int) {
|
||||||
|
_ = wc.fc.WriteFrame(wire.NewWriter().U8(MuxWnd).VarInt(sid).VarInt(delta).Out())
|
||||||
|
}
|
||||||
|
|
||||||
// Stream bridges one player (via the hub) to one destination connection.
|
// Stream bridges one player (via the hub) to one destination connection.
|
||||||
|
//
|
||||||
|
// Data from the hub is queued and written to the destination by a dedicated
|
||||||
|
// writeLoop goroutine. The queue is bounded by the advertised receive window —
|
||||||
|
// the hub never sends more un-credited bytes, so overflow is a protocol
|
||||||
|
// violation and resets the stream.
|
||||||
type Stream struct {
|
type Stream struct {
|
||||||
wc *WorkerConn
|
wc *WorkerConn
|
||||||
sid int
|
sid int
|
||||||
@@ -208,24 +237,32 @@ type Stream struct {
|
|||||||
srcPort int
|
srcPort int
|
||||||
|
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
|
cond *sync.Cond
|
||||||
dest net.Conn
|
dest net.Conn
|
||||||
connected bool
|
connected bool
|
||||||
preBuf []byte
|
|
||||||
closed bool
|
closed bool
|
||||||
|
finPending bool // hub sent FIN; close the destination once the queue drains
|
||||||
|
q [][]byte // hub -> destination, waiting for writeLoop
|
||||||
|
qBytes int
|
||||||
|
sendWnd int // flow control: budget for destination -> hub DATA
|
||||||
|
consumed int // flow control: drained bytes not yet credited back to the hub
|
||||||
}
|
}
|
||||||
|
|
||||||
func newStream(wc *WorkerConn, sid int, cid []byte, m Mapping, ip string, port int) *Stream {
|
func newStream(wc *WorkerConn, sid int, cid []byte, m Mapping, ip string, port int) *Stream {
|
||||||
return &Stream{wc: wc, sid: sid, cid: cid, mapping: m, srcIP: ip, srcPort: port}
|
s := &Stream{wc: wc, sid: sid, cid: cid, mapping: m, srcIP: ip, srcPort: port, sendWnd: wc.sendWndInit}
|
||||||
|
s.cond = sync.NewCond(&s.mu)
|
||||||
|
return s
|
||||||
}
|
}
|
||||||
|
|
||||||
// run dials the destination, optionally writes the PROXY v2 header, flushes any
|
// run dials the destination, optionally writes the PROXY v2 header, then pumps
|
||||||
// buffered hub bytes, then pumps destination -> hub.
|
// destination -> hub (respecting the stream send window when negotiated).
|
||||||
func (s *Stream) run() {
|
func (s *Stream) run() {
|
||||||
dest, err := net.DialTimeout("tcp", s.mapping.Destination, 10*time.Second)
|
dest, err := net.DialTimeout("tcp", s.mapping.Destination, 10*time.Second)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
log.Printf("stream %d: dial %s failed: %v", s.sid, s.mapping.Destination, err)
|
log.Printf("stream %d: dial %s failed: %v", s.sid, s.mapping.Destination, err)
|
||||||
s.wc.removeStream(s.sid)
|
s.wc.removeStream(s.sid)
|
||||||
s.wc.sendRst(s.sid)
|
s.wc.sendRst(s.sid)
|
||||||
|
s.teardown(false)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if tcp, ok := dest.(*net.TCPConn); ok {
|
if tcp, ok := dest.(*net.TCPConn); ok {
|
||||||
@@ -240,7 +277,6 @@ func (s *Stream) run() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Atomically flush pre-connect buffer and enable direct writes.
|
|
||||||
s.mu.Lock()
|
s.mu.Lock()
|
||||||
if s.closed {
|
if s.closed {
|
||||||
s.mu.Unlock()
|
s.mu.Unlock()
|
||||||
@@ -248,18 +284,18 @@ func (s *Stream) run() {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
s.dest = dest
|
s.dest = dest
|
||||||
if len(s.preBuf) > 0 {
|
|
||||||
_, _ = dest.Write(s.preBuf)
|
|
||||||
s.preBuf = nil
|
|
||||||
}
|
|
||||||
s.connected = true
|
s.connected = true
|
||||||
|
s.cond.Broadcast() // wake writeLoop: queued hub bytes can flow now
|
||||||
s.mu.Unlock()
|
s.mu.Unlock()
|
||||||
|
|
||||||
// destination -> hub
|
// destination -> hub
|
||||||
buf := make([]byte, 32*1024)
|
buf := make([]byte, DataChunkSize)
|
||||||
for {
|
for {
|
||||||
n, err := dest.Read(buf)
|
n, err := dest.Read(buf)
|
||||||
if n > 0 {
|
if n > 0 {
|
||||||
|
if !s.acquireSendWnd(n) {
|
||||||
|
break
|
||||||
|
}
|
||||||
if werr := s.wc.sendData(s.sid, buf[:n]); werr != nil {
|
if werr := s.wc.sendData(s.sid, buf[:n]); werr != nil {
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
@@ -268,7 +304,99 @@ func (s *Stream) run() {
|
|||||||
break
|
break
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
s.shutdown(true)
|
s.teardown(true)
|
||||||
|
}
|
||||||
|
|
||||||
|
// writeLoop is the only writer to the destination. It drains the receive queue,
|
||||||
|
// credits the hub as bytes land on the destination socket, and performs the
|
||||||
|
// deferred graceful close when a FIN arrived with data still queued.
|
||||||
|
func (s *Stream) writeLoop() {
|
||||||
|
for {
|
||||||
|
s.mu.Lock()
|
||||||
|
for !s.closed && (!s.connected || (len(s.q) == 0 && !s.finPending)) {
|
||||||
|
s.cond.Wait()
|
||||||
|
}
|
||||||
|
if s.closed {
|
||||||
|
s.mu.Unlock()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if len(s.q) == 0 { // finPending and fully drained
|
||||||
|
s.mu.Unlock()
|
||||||
|
s.teardown(false)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
data := s.q[0]
|
||||||
|
s.q = s.q[1:]
|
||||||
|
s.qBytes -= len(data)
|
||||||
|
dest := s.dest
|
||||||
|
s.mu.Unlock()
|
||||||
|
|
||||||
|
if _, err := dest.Write(data); err != nil {
|
||||||
|
s.teardown(true)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.credit(len(data))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// deliverFromHub enqueues hub bytes for the destination. Called from the worker
|
||||||
|
// readLoop; it never blocks — a peer that exceeds the advertised window is a
|
||||||
|
// protocol violator and gets the stream reset.
|
||||||
|
func (s *Stream) deliverFromHub(data []byte) {
|
||||||
|
s.mu.Lock()
|
||||||
|
if s.closed || s.finPending {
|
||||||
|
s.mu.Unlock()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if s.qBytes+len(data) > s.wc.recvWndInit {
|
||||||
|
s.mu.Unlock()
|
||||||
|
log.Printf("stream %d: peer exceeded flow-control window; resetting", s.sid)
|
||||||
|
s.wc.removeStream(s.sid)
|
||||||
|
s.wc.sendRst(s.sid)
|
||||||
|
s.teardown(false)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.q = append(s.q, data)
|
||||||
|
s.qBytes += len(data)
|
||||||
|
s.cond.Broadcast()
|
||||||
|
s.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
// acquireSendWnd blocks until the stream may send n more bytes to the hub.
|
||||||
|
// Returns false if the stream closed while waiting.
|
||||||
|
func (s *Stream) acquireSendWnd(n int) bool {
|
||||||
|
s.mu.Lock()
|
||||||
|
defer s.mu.Unlock()
|
||||||
|
for !s.closed && s.sendWnd < n {
|
||||||
|
s.cond.Wait()
|
||||||
|
}
|
||||||
|
if s.closed {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
s.sendWnd -= n
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Stream) grantSendWnd(delta int) {
|
||||||
|
s.mu.Lock()
|
||||||
|
s.sendWnd += delta
|
||||||
|
s.cond.Broadcast()
|
||||||
|
s.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
// credit accounts bytes drained to the destination and grants the hub more
|
||||||
|
// window once half of our receive window has been consumed.
|
||||||
|
func (s *Stream) credit(n int) {
|
||||||
|
s.mu.Lock()
|
||||||
|
s.consumed += n
|
||||||
|
if s.closed || s.consumed*2 < s.wc.recvWndInit {
|
||||||
|
s.mu.Unlock()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
delta := s.consumed
|
||||||
|
s.consumed = 0
|
||||||
|
s.mu.Unlock()
|
||||||
|
s.wc.sendWndUpdate(s.sid, delta)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *Stream) buildProxyHeader(dest net.Conn) []byte {
|
func (s *Stream) buildProxyHeader(dest net.Conn) []byte {
|
||||||
@@ -283,28 +411,29 @@ func (s *Stream) buildProxyHeader(dest net.Conn) []byte {
|
|||||||
return BuildProxyV2(srcIP, s.srcPort, dstTCP.IP, dstTCP.Port)
|
return BuildProxyV2(srcIP, s.srcPort, dstTCP.IP, dstTCP.Port)
|
||||||
}
|
}
|
||||||
|
|
||||||
// deliverFromHub writes bytes coming from the hub to the destination, buffering
|
// finDrainTimeout bounds how long a FIN'd stream may keep draining its queue
|
||||||
// until the destination connection is established.
|
// into the destination, so a dead destination cannot hold the stream forever.
|
||||||
func (s *Stream) deliverFromHub(data []byte) {
|
const finDrainTimeout = 10 * time.Second
|
||||||
|
|
||||||
|
// gracefulFin marks the hub-side close; writeLoop finishes the queue and then
|
||||||
|
// tears the stream down.
|
||||||
|
func (s *Stream) gracefulFin() {
|
||||||
s.mu.Lock()
|
s.mu.Lock()
|
||||||
if s.closed {
|
if s.closed || s.finPending {
|
||||||
s.mu.Unlock()
|
s.mu.Unlock()
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if !s.connected {
|
s.finPending = true
|
||||||
s.preBuf = append(s.preBuf, data...)
|
if s.dest != nil {
|
||||||
s.mu.Unlock()
|
_ = s.dest.SetWriteDeadline(time.Now().Add(finDrainTimeout))
|
||||||
return
|
|
||||||
}
|
}
|
||||||
dest := s.dest
|
s.cond.Broadcast()
|
||||||
s.mu.Unlock()
|
s.mu.Unlock()
|
||||||
if _, err := dest.Write(data); err != nil {
|
|
||||||
s.shutdown(true)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// shutdown closes the stream; notifyHub sends a FIN to the hub when true.
|
// teardown closes the stream immediately; notifyHub sends a FIN when true.
|
||||||
func (s *Stream) shutdown(notifyHub bool) {
|
// Idempotent; wakes every goroutine parked on the stream.
|
||||||
|
func (s *Stream) teardown(notifyHub bool) {
|
||||||
s.mu.Lock()
|
s.mu.Lock()
|
||||||
if s.closed {
|
if s.closed {
|
||||||
s.mu.Unlock()
|
s.mu.Unlock()
|
||||||
@@ -312,6 +441,7 @@ func (s *Stream) shutdown(notifyHub bool) {
|
|||||||
}
|
}
|
||||||
s.closed = true
|
s.closed = true
|
||||||
dest := s.dest
|
dest := s.dest
|
||||||
|
s.cond.Broadcast()
|
||||||
s.mu.Unlock()
|
s.mu.Unlock()
|
||||||
|
|
||||||
if dest != nil {
|
if dest != nil {
|
||||||
|
|||||||
+35
-16
@@ -154,25 +154,43 @@ connections (9 + 9 + 2), confirming the algorithm.
|
|||||||
simplicity and correctness.
|
simplicity and correctness.
|
||||||
* **Client:** goroutine-per-concern. One goroutine reads each connection
|
* **Client:** goroutine-per-concern. One goroutine reads each connection
|
||||||
(control or worker); `WriteFrame` is mutex-serialized so many stream goroutines
|
(control or worker); `WriteFrame` is mutex-serialized so many stream goroutines
|
||||||
can share a worker connection safely. A per-stream mutex guards the small
|
can share a worker connection safely. Each stream has two goroutines: `run`
|
||||||
"buffer until the destination is connected, then write directly" handoff so
|
pumps destination → hub, and `writeLoop` is the only writer to the
|
||||||
the forwarded handshake never races ahead of later bytes.
|
destination, draining a per-stream queue fed by the worker readLoop. The
|
||||||
|
readLoop itself never writes to a destination, so a stalled destination can
|
||||||
|
never block frame dispatch for other streams.
|
||||||
|
|
||||||
## 6. Back-pressure
|
## 6. Back-pressure & flow control
|
||||||
|
|
||||||
There is no per-stream credit window. Flow is governed by TCP back-pressure on
|
Two mechanisms operate at different granularities:
|
||||||
each worker connection:
|
|
||||||
|
|
||||||
* Hub → player: if a player socket's write queue fills, the hub pauses the
|
* **Per-stream credit windows** (PROTOCOL.md §7.3; the windows are exchanged
|
||||||
worker connection socket and resumes on drain.
|
at session establishment): each stream direction has an independent byte
|
||||||
* Destination → hub: the client's `WriteFrame` blocks when the worker socket is
|
budget equal to the receiver's advertised window (default 256 KiB). A sender
|
||||||
congested, which naturally stops the client reading the destination.
|
that exhausts a
|
||||||
|
stream's window pauses *only that stream's source* — the hub pauses the one
|
||||||
|
player socket, the client parks the one destination-reader goroutine. Credit
|
||||||
|
is granted back (`WND` frames, batched at half-window) as bytes are actually
|
||||||
|
written to the terminal socket. The result: a slow player or slow destination
|
||||||
|
jams its own stream at a bounded buffer size and nothing else. This is what
|
||||||
|
eliminates head-of-line blocking between streams.
|
||||||
|
* **Aggregate TCP back-pressure** on each worker connection: when the shared
|
||||||
|
socket itself is congested (total bandwidth, not one stream), the hub parks
|
||||||
|
all sending players until it drains, and the client's `WriteFrame` blocks.
|
||||||
|
This is fair — when the pipe is genuinely full, everyone should slow down.
|
||||||
|
|
||||||
The consequence is head-of-line blocking *within* a worker connection: one very
|
The window also bounds memory: a stream can hold at most one window of
|
||||||
slow player can stall other streams sharing that connection. `maxConn` spreads
|
undelivered data per direction (the client's pre-connect handshake buffer is
|
||||||
streams across connections to mitigate this. For interactive Minecraft traffic
|
covered by the same bound).
|
||||||
(small client→server packets, bursty server→client chunk data) this is a good
|
|
||||||
trade for a near-zero-overhead mux.
|
Per-stream flow control is mandatory: the hub rejects a session whose Rekey
|
||||||
|
lacks the STREAM_FC flag, and the client rejects a hub that does not echo it —
|
||||||
|
peers that predate the mechanism cannot connect at all.
|
||||||
|
|
||||||
|
What remains (by design) is TCP-level head-of-line blocking: a lost packet on
|
||||||
|
a worker connection stalls all its streams for one retransmit. That is inherent
|
||||||
|
to mux-over-TCP; the connection pool is the mitigation, and a datagram
|
||||||
|
transport (QUIC) would be the escape hatch if it ever matters.
|
||||||
|
|
||||||
## 7. Failure & recovery
|
## 7. Failure & recovery
|
||||||
|
|
||||||
@@ -190,7 +208,8 @@ trade for a near-zero-overhead mux.
|
|||||||
## 8. Known limitations
|
## 8. Known limitations
|
||||||
|
|
||||||
1. No AEAD — payload integrity/authenticity is not cryptographically guaranteed.
|
1. No AEAD — payload integrity/authenticity is not cryptographically guaranteed.
|
||||||
2. No per-stream flow control (see §6).
|
2. TCP-level head-of-line blocking within a worker connection (lost packets;
|
||||||
|
see §6) — per-stream flow control removes the application-level variant only.
|
||||||
3. Single-event-loop hub (see §5) bounds throughput to one core.
|
3. Single-event-loop hub (see §5) bounds throughput to one core.
|
||||||
4. `Intent 18` is reserved but only stubbed (the hub logs and closes).
|
4. `Intent 18` is reserved but only stubbed (the hub logs and closes).
|
||||||
5. Pattern ownership is last-writer-wins; two clients registering the identical
|
5. Pattern ownership is last-writer-wins; two clients registering the identical
|
||||||
|
|||||||
+11
-1
@@ -129,6 +129,7 @@ type destMode int
|
|||||||
const (
|
const (
|
||||||
modeEcho destMode = iota // echo every post-handshake byte
|
modeEcho destMode = iota // echo every post-handshake byte
|
||||||
modeEchoOnceClose // echo one read, then close the connection
|
modeEchoOnceClose // echo one read, then close the connection
|
||||||
|
modeBlackhole // accept but never read: immediate write back-pressure
|
||||||
)
|
)
|
||||||
|
|
||||||
type proxyInfo struct {
|
type proxyInfo struct {
|
||||||
@@ -154,6 +155,7 @@ type mockDest struct {
|
|||||||
mode destMode
|
mode destMode
|
||||||
events chan destEvent
|
events chan destEvent
|
||||||
connClosed chan struct{}
|
connClosed chan struct{}
|
||||||
|
done chan struct{}
|
||||||
}
|
}
|
||||||
|
|
||||||
func newMockDest(t *testing.T, mode destMode) *mockDest {
|
func newMockDest(t *testing.T, mode destMode) *mockDest {
|
||||||
@@ -168,8 +170,12 @@ func newMockDest(t *testing.T, mode destMode) *mockDest {
|
|||||||
mode: mode,
|
mode: mode,
|
||||||
events: make(chan destEvent, 128),
|
events: make(chan destEvent, 128),
|
||||||
connClosed: make(chan struct{}, 128),
|
connClosed: make(chan struct{}, 128),
|
||||||
|
done: make(chan struct{}),
|
||||||
}
|
}
|
||||||
t.Cleanup(func() { _ = ln.Close() })
|
t.Cleanup(func() {
|
||||||
|
_ = ln.Close()
|
||||||
|
close(d.done)
|
||||||
|
})
|
||||||
go d.serve()
|
go d.serve()
|
||||||
return d
|
return d
|
||||||
}
|
}
|
||||||
@@ -189,6 +195,10 @@ func (d *mockDest) handle(conn net.Conn) {
|
|||||||
_ = conn.Close()
|
_ = conn.Close()
|
||||||
d.connClosed <- struct{}{}
|
d.connClosed <- struct{}{}
|
||||||
}()
|
}()
|
||||||
|
if d.mode == modeBlackhole {
|
||||||
|
<-d.done // hold the connection open without ever reading
|
||||||
|
return
|
||||||
|
}
|
||||||
br := bufio.NewReader(conn)
|
br := bufio.NewReader(conn)
|
||||||
|
|
||||||
var ev destEvent
|
var ev destEvent
|
||||||
|
|||||||
@@ -0,0 +1,131 @@
|
|||||||
|
package e2e
|
||||||
|
|
||||||
|
import (
|
||||||
|
crand "crypto/rand"
|
||||||
|
"fmt"
|
||||||
|
"net"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/iceBear67/redapricot/client"
|
||||||
|
"github.com/iceBear67/redapricot/client/wire"
|
||||||
|
)
|
||||||
|
|
||||||
|
// echoRounds pushes payload through the tunnel `rounds` times on one player
|
||||||
|
// connection, failing the test if any round stalls.
|
||||||
|
func echoRounds(t *testing.T, hubAddr string, rounds, size int) {
|
||||||
|
t.Helper()
|
||||||
|
fast := dialPlayer(t, hubAddr, "mc.local")
|
||||||
|
defer fast.Close()
|
||||||
|
payload := make([]byte, size)
|
||||||
|
for i := range payload {
|
||||||
|
payload[i] = byte(i*13 + 5)
|
||||||
|
}
|
||||||
|
for i := 0; i < rounds; i++ {
|
||||||
|
playerEcho(t, fast, payload)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestSlowPlayerDoesNotStallOthers: a player that stops reading while megabytes
|
||||||
|
// are echoed back to it must not stall another stream on the same worker
|
||||||
|
// connection (maxConn=1 forces sharing). Before per-stream flow control, the
|
||||||
|
// hub paused the whole worker socket once that player's write queue filled,
|
||||||
|
// freezing every other stream's downstream data.
|
||||||
|
func TestSlowPlayerDoesNotStallOthers(t *testing.T) {
|
||||||
|
const psk = "e2e-slowplayer"
|
||||||
|
port := freePort(t)
|
||||||
|
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
|
||||||
|
startHub(t, port, psk)
|
||||||
|
dest := newMockDest(t, modeEcho)
|
||||||
|
startClient(t, hubAddr, psk, 1, []client.Mapping{
|
||||||
|
{Pattern: "mc.local", Destination: dest.addr},
|
||||||
|
})
|
||||||
|
|
||||||
|
// Slow player: 4 MiB goes out, gets echoed back, and is never read. The
|
||||||
|
// write blocks once buffers fill; the error on test-end close is expected.
|
||||||
|
slow := dialPlayer(t, hubAddr, "mc.local")
|
||||||
|
defer slow.Close()
|
||||||
|
go func() {
|
||||||
|
_, _ = slow.Write(make([]byte, 4*1024*1024))
|
||||||
|
}()
|
||||||
|
|
||||||
|
// Let the slow stream jam: its flow-control window fills and stays full.
|
||||||
|
time.Sleep(1 * time.Second)
|
||||||
|
|
||||||
|
// The fast player shares the single worker conn and must still round-trip.
|
||||||
|
echoRounds(t, hubAddr, 10, 8*1024)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestSlowDestinationDoesNotStallOthers: a destination that never reads must
|
||||||
|
// only stall its own stream. Before the async delivery queue, the client wrote
|
||||||
|
// to destinations inline in the worker readLoop, so one blocked destination
|
||||||
|
// froze every stream on the connection.
|
||||||
|
func TestSlowDestinationDoesNotStallOthers(t *testing.T) {
|
||||||
|
const psk = "e2e-slowdest"
|
||||||
|
port := freePort(t)
|
||||||
|
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
|
||||||
|
startHub(t, port, psk)
|
||||||
|
dest := newMockDest(t, modeEcho)
|
||||||
|
hole := newMockDest(t, modeBlackhole)
|
||||||
|
startClient(t, hubAddr, psk, 1, []client.Mapping{
|
||||||
|
{Pattern: "mc.local", Destination: dest.addr},
|
||||||
|
{Pattern: "hole.local", Destination: hole.addr},
|
||||||
|
})
|
||||||
|
|
||||||
|
// This player's destination never reads: the client-side write jams after
|
||||||
|
// kernel buffers fill, with the stream's queue bounded by its window.
|
||||||
|
stuck := dialPlayer(t, hubAddr, "hole.local")
|
||||||
|
defer stuck.Close()
|
||||||
|
go func() {
|
||||||
|
_, _ = stuck.Write(make([]byte, 2*1024*1024))
|
||||||
|
}()
|
||||||
|
|
||||||
|
time.Sleep(1 * time.Second)
|
||||||
|
|
||||||
|
echoRounds(t, hubAddr, 10, 8*1024)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestLegacyClientRejected: per-stream flow control is mandatory. A client that
|
||||||
|
// performs the old session establishment — a Rekey message without the trailing
|
||||||
|
// feature flags — must be closed by the hub before SessionReady.
|
||||||
|
func TestLegacyClientRejected(t *testing.T) {
|
||||||
|
const psk = "e2e-legacyreject"
|
||||||
|
port := freePort(t)
|
||||||
|
hubAddr := fmt.Sprintf("127.0.0.1:%d", port)
|
||||||
|
startHub(t, port, psk)
|
||||||
|
|
||||||
|
conn, err := net.DialTimeout("tcp", hubAddr, 5*time.Second)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("dial hub: %v", err)
|
||||||
|
}
|
||||||
|
defer conn.Close()
|
||||||
|
|
||||||
|
pskBytes := []byte(psk)
|
||||||
|
hs := wire.BuildHandshake(767, wire.PSKAddress(pskBytes), 25565, 17)
|
||||||
|
if _, err := conn.Write(hs); err != nil {
|
||||||
|
t.Fatalf("handshake: %v", err)
|
||||||
|
}
|
||||||
|
fc := wire.NewFramedConn(conn,
|
||||||
|
wire.CipherFor(pskBytes, wire.DirS2C),
|
||||||
|
wire.CipherFor(pskBytes, wire.DirC2S),
|
||||||
|
)
|
||||||
|
rnd := make([]byte, 16)
|
||||||
|
if _, err := crand.Read(rnd); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
// Pre-flow-control Rekey: Magic|RandLen|Rand|Timestamp with no flags.
|
||||||
|
legacyRekey := wire.NewWriter().
|
||||||
|
U8(0x01). // control-session magic
|
||||||
|
VarInt(len(rnd)).
|
||||||
|
Bytes(rnd).
|
||||||
|
I64(time.Now().UnixMilli()).
|
||||||
|
Out()
|
||||||
|
if err := fc.WriteFrame(legacyRekey); err != nil {
|
||||||
|
t.Fatalf("rekey: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_ = conn.SetReadDeadline(time.Now().Add(10 * time.Second))
|
||||||
|
if payload, err := fc.ReadFrame(); err == nil {
|
||||||
|
t.Fatalf("expected the hub to close a legacy session, got frame %v", payload)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -2,5 +2,6 @@
|
|||||||
"listen": "0.0.0.0:25565",
|
"listen": "0.0.0.0:25565",
|
||||||
"psk": "change-me-to-a-long-random-passphrase",
|
"psk": "change-me-to-a-long-random-passphrase",
|
||||||
"timestampWindowMs": 30000,
|
"timestampWindowMs": 30000,
|
||||||
"pendingTimeoutMs": 10000
|
"pendingTimeoutMs": 10000,
|
||||||
|
"streamWindowBytes": 262144
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -11,7 +11,8 @@ public record Config(
|
|||||||
int port,
|
int port,
|
||||||
String psk,
|
String psk,
|
||||||
long timestampWindowMs,
|
long timestampWindowMs,
|
||||||
long pendingTimeoutMs
|
long pendingTimeoutMs,
|
||||||
|
int streamWindowBytes
|
||||||
) {
|
) {
|
||||||
public static Config load(Path file) throws Exception {
|
public static Config load(Path file) throws Exception {
|
||||||
JsonObject json = new JsonObject(Files.readString(file));
|
JsonObject json = new JsonObject(Files.readString(file));
|
||||||
@@ -25,11 +26,15 @@ public record Config(
|
|||||||
String psk = json.getString("psk");
|
String psk = json.getString("psk");
|
||||||
if (psk == null || psk.isEmpty()) throw new IllegalArgumentException("psk is required");
|
if (psk == null || psk.isEmpty()) throw new IllegalArgumentException("psk is required");
|
||||||
|
|
||||||
|
int window = json.getInteger("streamWindowBytes", Protocol.DEFAULT_STREAM_WINDOW);
|
||||||
|
window = Math.max(Protocol.MIN_STREAM_WINDOW, Math.min(Protocol.MAX_STREAM_WINDOW, window));
|
||||||
|
|
||||||
return new Config(
|
return new Config(
|
||||||
host,
|
host,
|
||||||
port,
|
port,
|
||||||
psk,
|
psk,
|
||||||
json.getLong("timestampWindowMs", 30_000L),
|
json.getLong("timestampWindowMs", 30_000L),
|
||||||
json.getLong("pendingTimeoutMs", 10_000L));
|
json.getLong("pendingTimeoutMs", 10_000L),
|
||||||
|
window);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ import io.icybear.redapricot.crypto.Crypto;
|
|||||||
import io.icybear.redapricot.net.EncryptedFrames;
|
import io.icybear.redapricot.net.EncryptedFrames;
|
||||||
import io.icybear.redapricot.util.Hex;
|
import io.icybear.redapricot.util.Hex;
|
||||||
import io.icybear.redapricot.util.ProtoReader;
|
import io.icybear.redapricot.util.ProtoReader;
|
||||||
|
import io.icybear.redapricot.util.ProtoWriter;
|
||||||
import io.icybear.redapricot.util.VarInt;
|
import io.icybear.redapricot.util.VarInt;
|
||||||
import io.vertx.core.buffer.Buffer;
|
import io.vertx.core.buffer.Buffer;
|
||||||
import io.vertx.core.net.NetSocket;
|
import io.vertx.core.net.NetSocket;
|
||||||
@@ -145,6 +146,26 @@ public final class HubConnection {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Mandatory trailing feature flags: the client must offer per-stream flow
|
||||||
|
// control and advertise its receive window. Anything else is an
|
||||||
|
// unsupported (pre-flow-control) peer and is rejected.
|
||||||
|
int flags;
|
||||||
|
int peerWindow;
|
||||||
|
try {
|
||||||
|
flags = r.readVarInt();
|
||||||
|
peerWindow = r.readVarInt();
|
||||||
|
} catch (RuntimeException e) {
|
||||||
|
LOG.warn("{} rekey without feature flags (unsupported client version); closing", id);
|
||||||
|
frames.close();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if ((flags & Protocol.FLAG_STREAM_FC) == 0 || peerWindow <= 0) {
|
||||||
|
LOG.warn("{} client did not offer per-stream flow control; closing", id);
|
||||||
|
frames.close();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
peerWindow = Math.min(peerWindow, Protocol.MAX_STREAM_WINDOW);
|
||||||
|
|
||||||
// REKEY = Rand || Timestamp(I64 big-endian). Magic is excluded.
|
// REKEY = Rand || Timestamp(I64 big-endian). Magic is excluded.
|
||||||
byte[] rekey = new byte[randLen + 8];
|
byte[] rekey = new byte[randLen + 8];
|
||||||
System.arraycopy(rand, 0, rekey, 0, randLen);
|
System.arraycopy(rand, 0, rekey, 0, randLen);
|
||||||
@@ -157,7 +178,12 @@ public final class HubConnection {
|
|||||||
frames.switchCiphers(
|
frames.switchCiphers(
|
||||||
Crypto.decryptCipher(rekey, Crypto.DIR_C2S),
|
Crypto.decryptCipher(rekey, Crypto.DIR_C2S),
|
||||||
Crypto.encryptCipher(rekey, Crypto.DIR_S2C));
|
Crypto.encryptCipher(rekey, Crypto.DIR_S2C));
|
||||||
frames.send(new byte[]{(byte) Protocol.CTL_SESSION_READY});
|
// Echo the accepted flags plus our own receive window.
|
||||||
|
frames.send(new ProtoWriter()
|
||||||
|
.u8(Protocol.CTL_SESSION_READY)
|
||||||
|
.varInt(Protocol.FLAG_STREAM_FC)
|
||||||
|
.varInt(hub.config.streamWindowBytes())
|
||||||
|
.toBytes());
|
||||||
|
|
||||||
if (magic == Protocol.MAGIC_CONTROL) {
|
if (magic == Protocol.MAGIC_CONTROL) {
|
||||||
ControlSession session = new ControlSession(hub, frames, id);
|
ControlSession session = new ControlSession(hub, frames, id);
|
||||||
@@ -165,10 +191,10 @@ public final class HubConnection {
|
|||||||
closeCleanup = session::onClose;
|
closeCleanup = session::onClose;
|
||||||
LOG.info("{} control session established", id);
|
LOG.info("{} control session established", id);
|
||||||
} else if (magic == Protocol.MAGIC_WORKER) {
|
} else if (magic == Protocol.MAGIC_WORKER) {
|
||||||
WorkerConn worker = new WorkerConn(hub, frames, id);
|
WorkerConn worker = new WorkerConn(hub, frames, id, peerWindow, hub.config.streamWindowBytes());
|
||||||
frames.setHandler(worker::onFrame);
|
frames.setHandler(worker::onFrame);
|
||||||
closeCleanup = worker::onClose;
|
closeCleanup = worker::onClose;
|
||||||
LOG.info("{} worker conn established", id);
|
LOG.info("{} worker conn established (peer window {})", id, peerWindow);
|
||||||
} else {
|
} else {
|
||||||
LOG.warn("{} bad magic {}; closing", id, magic);
|
LOG.warn("{} bad magic {}; closing", id, magic);
|
||||||
frames.close();
|
frames.close();
|
||||||
|
|||||||
@@ -31,6 +31,16 @@ public final class Protocol {
|
|||||||
public static final int MUX_DATA = 0x01;
|
public static final int MUX_DATA = 0x01;
|
||||||
public static final int MUX_FIN = 0x02;
|
public static final int MUX_FIN = 0x02;
|
||||||
public static final int MUX_RST = 0x03;
|
public static final int MUX_RST = 0x03;
|
||||||
|
public static final int MUX_WND = 0x04; // per-stream flow-control credit grant
|
||||||
|
|
||||||
|
// Session-establishment feature flags (trailing VarInt on the Rekey message,
|
||||||
|
// echoed after the SessionReady type byte when accepted).
|
||||||
|
public static final int FLAG_STREAM_FC = 0x01;
|
||||||
|
|
||||||
|
// Per-stream flow-control window bounds (bytes).
|
||||||
|
public static final int DEFAULT_STREAM_WINDOW = 256 * 1024;
|
||||||
|
public static final int MIN_STREAM_WINDOW = 32 * 1024;
|
||||||
|
public static final int MAX_STREAM_WINDOW = 8 << 20;
|
||||||
|
|
||||||
// Any redapricot connection
|
// Any redapricot connection
|
||||||
public static final int FRAME_ERROR = 0x7F;
|
public static final int FRAME_ERROR = 0x7F;
|
||||||
|
|||||||
@@ -17,21 +17,45 @@ import java.util.Set;
|
|||||||
/**
|
/**
|
||||||
* An authenticated worker connection (Magic 0x02). Multiplexes many player
|
* An authenticated worker connection (Magic 0x02). Multiplexes many player
|
||||||
* streams; the client opens streams via SYN(CID) to take over pending players.
|
* streams; the client opens streams via SYN(CID) to take over pending players.
|
||||||
|
*
|
||||||
|
* <p>Every stream has a credit window in both directions (PROTOCOL.md §7.3),
|
||||||
|
* so one slow player only ever stalls its own stream — the shared worker
|
||||||
|
* socket is never paused because of a single stream.
|
||||||
*/
|
*/
|
||||||
@RequiredArgsConstructor
|
@RequiredArgsConstructor
|
||||||
public final class WorkerConn {
|
public final class WorkerConn {
|
||||||
private static final Logger LOG = LogManager.getLogger("redapricot.worker");
|
private static final Logger LOG = LogManager.getLogger("redapricot.worker");
|
||||||
|
|
||||||
|
/** Cap on a single DATA frame so no stream monopolizes the shared link for long. */
|
||||||
|
private static final int CHUNK = 32 * 1024;
|
||||||
|
|
||||||
private final Hub hub;
|
private final Hub hub;
|
||||||
private final EncryptedFrames frames;
|
private final EncryptedFrames frames;
|
||||||
private final String id;
|
private final String id;
|
||||||
private final Map<Integer, NetSocket> streams = new HashMap<>();
|
private final int sendWndInit; // client's advertised per-stream receive window (our send budget)
|
||||||
|
private final int recvWndInit; // our advertised per-stream receive window (basis for credit grants)
|
||||||
|
|
||||||
// Backpressure for the single shared worker socket, arbitrated across all streams.
|
private final Map<Integer, StreamState> streams = new HashMap<>();
|
||||||
private final Set<NetSocket> upstreamPaused = new HashSet<>(); // players parked until the worker write queue drains
|
|
||||||
private final Set<Integer> downstreamBlocked = new HashSet<>(); // sids whose player write queue is full; non-empty => worker read paused
|
// Aggregate backpressure for the single shared worker socket: players parked
|
||||||
|
// until its write queue drains. Per-stream fairness is the credit windows'
|
||||||
|
// job; this only reacts to the whole pipe being congested.
|
||||||
|
private final Set<StreamState> upstreamPaused = new HashSet<>();
|
||||||
private boolean workerDrainArmed = false; // whether the worker socket's single drainHandler is set
|
private boolean workerDrainArmed = false; // whether the worker socket's single drainHandler is set
|
||||||
|
|
||||||
|
/** Per-stream flow-control bookkeeping. */
|
||||||
|
private final class StreamState {
|
||||||
|
final NetSocket player;
|
||||||
|
int sendWnd = sendWndInit; // budget for player -> client DATA
|
||||||
|
Buffer pendingUp; // player bytes awaiting send window (player is paused meanwhile)
|
||||||
|
boolean pausedForWindow;
|
||||||
|
int credited; // client -> player bytes flushed but not yet granted back
|
||||||
|
|
||||||
|
StreamState(NetSocket player) {
|
||||||
|
this.player = player;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
public void onFrame(byte[] payload) {
|
public void onFrame(byte[] payload) {
|
||||||
ProtoReader r = new ProtoReader(payload);
|
ProtoReader r = new ProtoReader(payload);
|
||||||
int type = r.readUByte();
|
int type = r.readUByte();
|
||||||
@@ -39,6 +63,7 @@ public final class WorkerConn {
|
|||||||
switch (type) {
|
switch (type) {
|
||||||
case Protocol.MUX_SYN -> handleSyn(sid, r.readBytes(Protocol.CID_LEN));
|
case Protocol.MUX_SYN -> handleSyn(sid, r.readBytes(Protocol.CID_LEN));
|
||||||
case Protocol.MUX_DATA -> handleData(sid, r.readBuffer(r.remaining()));
|
case Protocol.MUX_DATA -> handleData(sid, r.readBuffer(r.remaining()));
|
||||||
|
case Protocol.MUX_WND -> handleWnd(sid, r.readVarInt());
|
||||||
case Protocol.MUX_FIN, Protocol.MUX_RST -> closeStream(sid);
|
case Protocol.MUX_FIN, Protocol.MUX_RST -> closeStream(sid);
|
||||||
case Protocol.FRAME_ERROR -> LOG.warn("worker {} error frame", id);
|
case Protocol.FRAME_ERROR -> LOG.warn("worker {} error frame", id);
|
||||||
default -> LOG.warn("worker {} unknown mux type {}", id, type);
|
default -> LOG.warn("worker {} unknown mux type {}", id, type);
|
||||||
@@ -53,72 +78,126 @@ public final class WorkerConn {
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
NetSocket player = p.getSocket();
|
NetSocket player = p.getSocket();
|
||||||
streams.put(sid, player);
|
StreamState st = new StreamState(player);
|
||||||
|
streams.put(sid, st);
|
||||||
|
|
||||||
// From now on the player socket belongs to this stream.
|
// From now on the player socket belongs to this stream.
|
||||||
player.handler(buf -> {
|
player.handler(buf -> {
|
||||||
sendData(sid, buf.getBytes());
|
sendUpstream(sid, st, buf);
|
||||||
if (frames.writeQueueFull()) {
|
checkAggregate(st);
|
||||||
player.pause();
|
|
||||||
upstreamPaused.add(player);
|
|
||||||
armWorkerDrain();
|
|
||||||
}
|
|
||||||
});
|
});
|
||||||
player.closeHandler(v -> onPlayerGone(sid, player));
|
player.closeHandler(v -> onPlayerGone(sid, st));
|
||||||
player.exceptionHandler(t -> onPlayerGone(sid, player));
|
player.exceptionHandler(t -> onPlayerGone(sid, st));
|
||||||
|
|
||||||
// Forward the buffered handshake (and any pipelined bytes), then resume.
|
// Forward the buffered handshake (and any pipelined bytes), then resume.
|
||||||
sendData(sid, p.getBuffered().getBytes());
|
sendUpstream(sid, st, p.getBuffered());
|
||||||
player.resume();
|
if (!st.pausedForWindow) player.resume();
|
||||||
|
checkAggregate(st);
|
||||||
LOG.info("worker {} stream {} bound to {}", id, sid, p.getPattern());
|
LOG.info("worker {} stream {} bound to {}", id, sid, p.getPattern());
|
||||||
}
|
}
|
||||||
|
|
||||||
private void handleData(int sid, Buffer data) {
|
/**
|
||||||
NetSocket player = streams.get(sid);
|
* Send player bytes to the client, chunked and clipped to the stream window;
|
||||||
if (player == null) return;
|
* the overflow is parked in {@code pendingUp} and the player socket paused
|
||||||
player.write(data);
|
* until the client grants more credit.
|
||||||
// A slow player pauses the shared worker read side; the block set lets us resume only
|
*/
|
||||||
// once every blocked player has drained (and release it if a player disconnects meanwhile).
|
private void sendUpstream(int sid, StreamState st, Buffer buf) {
|
||||||
if (player.writeQueueFull() && downstreamBlocked.add(sid)) {
|
if (st.pendingUp != null) { // still waiting for window; keep ordering
|
||||||
if (downstreamBlocked.size() == 1) frames.socket().pause();
|
st.pendingUp.appendBuffer(buf);
|
||||||
player.drainHandler(v -> unblockDownstream(sid));
|
return;
|
||||||
}
|
}
|
||||||
|
int off = drainUpstream(sid, st, buf, 0);
|
||||||
|
if (off < buf.length()) {
|
||||||
|
st.pendingUp = buf.getBuffer(off, buf.length());
|
||||||
|
if (!st.pausedForWindow) {
|
||||||
|
st.pausedForWindow = true;
|
||||||
|
st.player.pause();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Send from {@code buf[off..]} within the stream window, chunked; returns the new offset. */
|
||||||
|
private int drainUpstream(int sid, StreamState st, Buffer buf, int off) {
|
||||||
|
while (off < buf.length() && st.sendWnd > 0) {
|
||||||
|
int n = Math.min(Math.min(CHUNK, st.sendWnd), buf.length() - off);
|
||||||
|
sendData(sid, buf.getBytes(off, off + n));
|
||||||
|
st.sendWnd -= n;
|
||||||
|
off += n;
|
||||||
|
}
|
||||||
|
return off;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The client granted {@code delta} more bytes of credit on a stream. */
|
||||||
|
private void handleWnd(int sid, int delta) {
|
||||||
|
StreamState st = streams.get(sid);
|
||||||
|
if (st == null || delta <= 0) return;
|
||||||
|
st.sendWnd += delta;
|
||||||
|
if (st.pendingUp != null) {
|
||||||
|
Buffer pending = st.pendingUp;
|
||||||
|
int off = drainUpstream(sid, st, pending, 0);
|
||||||
|
st.pendingUp = off >= pending.length() ? null : pending.getBuffer(off, pending.length());
|
||||||
|
}
|
||||||
|
if (st.pendingUp == null && st.pausedForWindow) {
|
||||||
|
st.pausedForWindow = false;
|
||||||
|
if (!upstreamPaused.contains(st)) st.player.resume();
|
||||||
|
}
|
||||||
|
checkAggregate(st);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void handleData(int sid, Buffer data) {
|
||||||
|
StreamState st = streams.get(sid);
|
||||||
|
if (st == null) return;
|
||||||
|
// Never pause the shared socket: the client bounds what it sends per
|
||||||
|
// stream to our advertised window, so a slow player only piles up a
|
||||||
|
// bounded amount in its own write queue; credit is granted back as the
|
||||||
|
// write completes (i.e. the bytes reached the player socket).
|
||||||
|
int len = data.length();
|
||||||
|
st.player.write(data).onComplete(ar -> {
|
||||||
|
if (ar.failed() || frames.isClosed() || streams.get(sid) != st) return;
|
||||||
|
st.credited += len;
|
||||||
|
if (st.credited * 2 >= recvWndInit) {
|
||||||
|
int delta = st.credited;
|
||||||
|
st.credited = 0;
|
||||||
|
sendWnd(sid, delta);
|
||||||
|
}
|
||||||
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
private void closeStream(int sid) {
|
private void closeStream(int sid) {
|
||||||
NetSocket player = streams.remove(sid);
|
StreamState st = streams.remove(sid);
|
||||||
unblockDownstream(sid);
|
if (st != null) {
|
||||||
if (player != null) {
|
upstreamPaused.remove(st);
|
||||||
upstreamPaused.remove(player);
|
st.player.close();
|
||||||
player.close();
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Register (once) the shared worker socket's single drain handler; on drain, wake every parked player. */
|
/** Park the player if the shared worker socket's write queue is congested. */
|
||||||
|
private void checkAggregate(StreamState st) {
|
||||||
|
if (frames.writeQueueFull() && upstreamPaused.add(st)) {
|
||||||
|
st.player.pause();
|
||||||
|
armWorkerDrain();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Register (once) the shared worker socket's single drain handler; on drain, wake parked players. */
|
||||||
private void armWorkerDrain() {
|
private void armWorkerDrain() {
|
||||||
if (workerDrainArmed) return;
|
if (workerDrainArmed) return;
|
||||||
workerDrainArmed = true;
|
workerDrainArmed = true;
|
||||||
frames.socket().drainHandler(v -> {
|
frames.socket().drainHandler(v -> {
|
||||||
workerDrainArmed = false;
|
workerDrainArmed = false;
|
||||||
if (upstreamPaused.isEmpty()) return;
|
if (upstreamPaused.isEmpty()) return;
|
||||||
NetSocket[] parked = upstreamPaused.toArray(new NetSocket[0]);
|
StreamState[] parked = upstreamPaused.toArray(new StreamState[0]);
|
||||||
upstreamPaused.clear();
|
upstreamPaused.clear();
|
||||||
for (NetSocket pl : parked) pl.resume();
|
for (StreamState st : parked) {
|
||||||
|
if (!st.pausedForWindow) st.player.resume();
|
||||||
|
}
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
/** A blocked player drained or vanished: drop its downstream block, resuming the shared worker read side when none remain. */
|
|
||||||
private void unblockDownstream(int sid) {
|
|
||||||
if (downstreamBlocked.remove(sid) && downstreamBlocked.isEmpty()) {
|
|
||||||
frames.socket().resume();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/** The player side of a stream vanished: drop it from every table, release any backpressure it held, and FIN the peer if still live. */
|
/** The player side of a stream vanished: drop it from every table, release any backpressure it held, and FIN the peer if still live. */
|
||||||
private void onPlayerGone(int sid, NetSocket player) {
|
private void onPlayerGone(int sid, StreamState st) {
|
||||||
boolean wasLive = streams.remove(sid) != null;
|
boolean wasLive = streams.remove(sid) == st;
|
||||||
upstreamPaused.remove(player);
|
upstreamPaused.remove(st);
|
||||||
unblockDownstream(sid);
|
|
||||||
if (wasLive) sendFin(sid);
|
if (wasLive) sendFin(sid);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -134,10 +213,13 @@ public final class WorkerConn {
|
|||||||
frames.send(new ProtoWriter().u8(Protocol.MUX_RST).varInt(sid).toBytes());
|
frames.send(new ProtoWriter().u8(Protocol.MUX_RST).varInt(sid).toBytes());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private void sendWnd(int sid, int delta) {
|
||||||
|
frames.send(new ProtoWriter().u8(Protocol.MUX_WND).varInt(sid).varInt(delta).toBytes());
|
||||||
|
}
|
||||||
|
|
||||||
public void onClose() {
|
public void onClose() {
|
||||||
for (NetSocket player : streams.values()) player.close();
|
for (StreamState st : streams.values()) st.player.close();
|
||||||
streams.clear();
|
streams.clear();
|
||||||
downstreamBlocked.clear();
|
|
||||||
upstreamPaused.clear();
|
upstreamPaused.clear();
|
||||||
LOG.info("worker {} closed", id);
|
LOG.info("worker {} closed", id);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -77,7 +77,7 @@ class CryptoCodecTest {
|
|||||||
|
|
||||||
/** A Hub whose event loop is never touched (register/match/normalize use no Vert.x state). */
|
/** A Hub whose event loop is never touched (register/match/normalize use no Vert.x state). */
|
||||||
private static Hub testHub() {
|
private static Hub testHub() {
|
||||||
return new Hub(null, new Config("0.0.0.0", 25565, "test-psk", 30_000L, 10_000L));
|
return new Hub(null, new Config("0.0.0.0", 25565, "test-psk", 30_000L, 10_000L, Protocol.DEFAULT_STREAM_WINDOW));
|
||||||
}
|
}
|
||||||
|
|
||||||
private static ControlSession testSession(Hub hub, String id) {
|
private static ControlSession testSession(Hub hub, String id) {
|
||||||
|
|||||||
Reference in New Issue
Block a user