init
This commit is contained in:
iceBear67
@@ -0,0 +1,166 @@
|
||||
# Working on simplepages
|
||||
|
||||
Read [`README.md`](README.md) first for what this is, and
|
||||
[`docs/operations.md`](docs/operations.md) for how it is run. This file is for
|
||||
whoever — human or agent — is changing the code.
|
||||
|
||||
Everything here exists to protect one property: **a request sees the whole old
|
||||
deployment or the whole new one, never a mixture.** If a change makes that
|
||||
property harder to reason about, it is the wrong change however much code it
|
||||
saves.
|
||||
|
||||
## Before you call a change done
|
||||
|
||||
```sh
|
||||
make fmt vet test
|
||||
make race
|
||||
make atomicity # -race -count=20, takes ~80s
|
||||
go test ./cmd/pages -run TestCLIImportGraph
|
||||
```
|
||||
|
||||
Run the last two whenever you touch `internal/site`, `internal/deploy`, or
|
||||
anything the CLI imports. `make race` is not optional for changes in those
|
||||
packages — the interesting failures here are all interleavings.
|
||||
|
||||
## Layout
|
||||
|
||||
| Path | |
|
||||
|---|---|
|
||||
| `api/` | Wire types, error codes, path builders. **Stdlib only** — it is shared with the CLI, and that is the whole reason the CLI stays dependency-free. |
|
||||
| `cmd/pages-server/` | Assembly: config, storage, recovery, listeners, background workers, signals. Also holds the end-to-end tests. |
|
||||
| `cmd/pages/` | CLI entry point plus `deps_test.go`, which enforces the import graph. |
|
||||
| `internal/store/` | SQLite: dual pool, migrations, queries, fsck. |
|
||||
| `internal/cas/` | Content-addressed blob store. |
|
||||
| `internal/site/` | ★ The read path: registry, immutable deployment snapshots, URL resolution, the HTTP handler. |
|
||||
| `internal/deploy/` | Deployment lifecycle, assembly, startup recovery, reconciler, GC. |
|
||||
| `internal/webroot/` | `$WEBROOT/~name` symlink maintenance. |
|
||||
| `internal/auth/` | Token mint/parse/verify, middleware, failed-auth rate limiting. |
|
||||
| `internal/adminapi/` | Management API routes and handlers. |
|
||||
| `internal/config/`, `internal/httpx/`, `internal/cache/`, `internal/pathutil/`, `internal/version/` | Support. |
|
||||
| `internal/client/`, `internal/clicmd/`, `internal/cliutil/` | CLI: HTTP client and deploy flow, commands, command tree and rendering. |
|
||||
|
||||
## Invariants
|
||||
|
||||
Each of these is load-bearing. Changing one is a design decision, not a
|
||||
refactor.
|
||||
|
||||
**1. Load `p.Active()` exactly once per request.** `internal/site/serve.go`
|
||||
takes the snapshot at the top and every later access uses that value. A second
|
||||
`Active()` call in the same request is the bug this whole design prevents: the
|
||||
two loads could straddle an activation and serve a mixed page. `Deployment` is
|
||||
immutable after construction, so holding the pointer is free and correct.
|
||||
|
||||
**2. Activation order is fixed.** Project lock → require `ready` → build the
|
||||
snapshot → **commit to SQLite** → `atomic.Pointer.Store` → best-effort
|
||||
`webroot.Point`. Database before memory, so a crash between the two restarts
|
||||
into the state the database already recorded. The reverse order leaves a
|
||||
process serving A while the database says B. Building the snapshot before the
|
||||
commit means a failure there changes nothing at all.
|
||||
|
||||
**3. Never trust a client-supplied hash.** `cas.Store.Put` rehashes the received
|
||||
stream and compares against the claimed digest before linking the blob into
|
||||
place; a mismatch is a 400 and the temp file is discarded. Without this a client
|
||||
could claim another project's digest, upload arbitrary bytes, and poison that
|
||||
blob for every project referencing it. This check is what makes cross-project
|
||||
dedup safe — do not move it, skip it for "already known" digests, or trust a
|
||||
digest because the transport was TLS.
|
||||
|
||||
**4. Tokens never appear in logs, query strings, or error messages.** There is a
|
||||
test that runs a request through the middleware into a buffer and greps for the
|
||||
token; keep it passing. The CLI prefers `PAGES_TOKEN` and `--token-file` over
|
||||
`--token` because argv is world-readable via `/proc/<pid>/cmdline` on a shared
|
||||
runner, and the help text says so. The bootstrap token file and the CLI config
|
||||
file are 0600.
|
||||
|
||||
**5. All writes go through `db.Tx`.** `store.DB` exposes `Reader()` for queries
|
||||
and `Tx(ctx, fn)` for everything else. There is deliberately no `Writer()` — the
|
||||
single write connection plus `BEGIN IMMEDIATE` plus busy retries is the only
|
||||
thing keeping `SQLITE_BUSY` off the hot path. Tests write through `db.Tx` too.
|
||||
|
||||
**6. Upload paths pass both `fs.ValidPath` and `filepath.Localize`,** and are
|
||||
additionally rejected for NUL or control bytes, a segment over 255 bytes, a
|
||||
total over 4096, duplicates, and case-insensitive collisions (the assembled tree
|
||||
may land on a case-insensitive filesystem, where a collision silently
|
||||
overwrites). One check is not enough: `fs.ValidPath` allows backslashes and
|
||||
Windows reserved names, `Localize` allows things `ValidPath` rejects.
|
||||
|
||||
**7. The read path serves from the CAS by digest.** The only filesystem path
|
||||
constructed while serving is `cas/ab/cd/<64 hex>`, derived from a `[32]byte`
|
||||
that came out of a map lookup. No user-controlled string reaches the filesystem,
|
||||
so read-path traversal is not defended against — it is structurally impossible.
|
||||
Serving from the assembled directory instead would give that back.
|
||||
|
||||
**8. No archive extraction, ever.** Files arrive one at a time; the server never
|
||||
unpacks or decompresses anything. That is what makes zip bombs and tar-slip
|
||||
inapplicable rather than mitigated. "Just accept a tarball, it's fewer round
|
||||
trips" reopens both — it needs a fresh security review, not a patch.
|
||||
|
||||
**9. `RequireProject` compares resolved project IDs,** never the name string
|
||||
from the URL. Names can be reused after a delete; IDs cannot. Same rule for
|
||||
deployments: confirm `{id}` really belongs to `{name}` before changing anything.
|
||||
|
||||
**10. `webroot.Reconcile` only removes entries that are symlinks pointing inside
|
||||
`$DATA_DIR/deployments`.** `$WEBROOT` belongs to the operator and will contain
|
||||
files that are none of our business.
|
||||
|
||||
**11. Assembled files and blobs are 0444, directories 0755, and client-supplied
|
||||
modes are ignored.** There is no `mode` column in the schema on purpose, so a
|
||||
setuid bit has nowhere to be stored even if someone tries to send one. On a
|
||||
hardlinking filesystem the assembled file *is* the blob's inode — writing to a
|
||||
file under `$WEBROOT` corrupts it for every project that references it.
|
||||
|
||||
**12. The CLI's dependency graph is enforced, not documented.**
|
||||
`cmd/pages/deps_test.go` fails if `go list -deps ./cmd/pages` reaches
|
||||
`modernc.org/sqlite`, `modernc.org/libc`, `database/sql`, `github.com/BurntSushi/toml`,
|
||||
`net/http/httptest`, or `internal/{store,site,deploy,cas,auth,adminapi}`.
|
||||
`database/sql` is a probe: it can only appear if server code leaked in. If you
|
||||
need a type in both the CLI and the server, it belongs in `api/` (stdlib only)
|
||||
or `internal/pathutil` (stdlib only).
|
||||
|
||||
## Conventions
|
||||
|
||||
**Comments explain why, not what.** The code already says what it does. A
|
||||
comment earns its place by recording the reasoning that is not recoverable from
|
||||
the code — why this order, why not the obvious alternative, what breaks if
|
||||
someone changes it. Match the density of the surrounding file; several of the
|
||||
sharper decisions are commented in place precisely so a later reader does not
|
||||
"optimize" them away (`internal/cache`'s note about not caching file handles is
|
||||
the canonical example).
|
||||
|
||||
**Do not over-design.** No metrics nobody reads, no abstraction with one
|
||||
implementation, no configuration knob without a caller who needs it. If a
|
||||
simpler thing works, ship the simpler thing.
|
||||
|
||||
**Dependencies.** Three direct ones, on purpose: `modernc.org/sqlite` (pure Go,
|
||||
so `CGO_ENABLED=0` cross-compiles), `github.com/BurntSushi/toml` (server config
|
||||
only), `golang.org/x/sync/errgroup` (bounded concurrency, both sides). No web
|
||||
framework, router, ORM, migration tool, CLI framework, logging library
|
||||
(`log/slog`), or UUID library (`crypto/rand` + hex). Adding a fourth needs a
|
||||
reason that survives being written down.
|
||||
|
||||
**Tests** are table-driven with names that read as sentences
|
||||
(`TestAnInFlightRequestSurvivesTheContentBeingCollected`). Two gotchas worth
|
||||
knowing before you debug an unexpected pass: retention tests must patch
|
||||
`retention_grace_s = 0`, because a deployment that was never activated is aged
|
||||
from `created_at` and is otherwise protected for the full grace period; and blob
|
||||
collection tests need a negative `Service.BlobGrace`, since zero means "use the
|
||||
one-hour default", not "collect now".
|
||||
|
||||
**CLI rendering:** `cliutil.Bool` prints `yes`/`no`, and `cliutil.Truncate(s, n)`
|
||||
returns `n-1` characters plus an ellipsis. Both have caught tests out.
|
||||
|
||||
## Scope
|
||||
|
||||
v1 (M0–M5) is complete: deploy, atomic switch, rollback, crash recovery,
|
||||
retention and GC, and the atomicity proof tests.
|
||||
|
||||
Deferred, deliberately — do not build these speculatively: precompressed
|
||||
`br`/`gzip` variants (the `deployment_files.encoding` column is already there
|
||||
for it), per-project header rules, host routing (one domain per project, which
|
||||
is also the fix for the same-origin limit), autoindex, a metrics endpoint, a
|
||||
resident-manifest cap, and packaging.
|
||||
|
||||
**Known limit:** one server process per database. The in-memory registry is
|
||||
updated by the process that wrote the change, so a second process on the same
|
||||
SQLite file serves stale content until it restarts. Horizontal scaling needs a
|
||||
change-notification mechanism first.
|
||||
Reference in New Issue
Block a user