pages/AGENTS.md

Working on simplepages

Read README.md first for what this is, and docs/operations.md for how it is run. This file is for whoever — human or agent — is changing the code.

Everything here exists to protect one property: a request sees the whole old deployment or the whole new one, never a mixture. If a change makes that property harder to reason about, it is the wrong change however much code it saves.

Before you call a change done

make fmt vet test
make race
make atomicity                            # -race -count=20, takes ~80s
go test ./cmd/pages -run TestCLIImportGraph

Run the last two whenever you touch internal/site, internal/deploy, or anything the CLI imports. make race is not optional for changes in those packages — the interesting failures here are all interleavings.

Layout

Path
api/	Wire types, error codes, path builders. Stdlib only — it is shared with the CLI, and that is the whole reason the CLI stays dependency-free.
cmd/pages-server/	Assembly: config, storage, recovery, listeners, background workers, signals. Also holds the end-to-end tests.
cmd/pages/	CLI entry point plus deps_test.go, which enforces the import graph.
internal/store/	SQLite: dual pool, migrations, queries, fsck.
internal/cas/	Content-addressed blob store.
internal/site/	★ The read path: registry, immutable deployment snapshots, URL resolution, the HTTP handler.
internal/deploy/	Deployment lifecycle, assembly, startup recovery, reconciler, GC.
internal/webroot/	$WEBROOT/~name symlink maintenance.
internal/auth/	Token mint/parse/verify, middleware, failed-auth rate limiting.
internal/adminapi/	Management API routes and handlers.
internal/config/, internal/httpx/, internal/cache/, internal/pathutil/, internal/version/	Support.
internal/client/, internal/clicmd/, internal/cliutil/	CLI: HTTP client and deploy flow, commands, command tree and rendering.

Invariants

Each of these is load-bearing. Changing one is a design decision, not a refactor.

1. Load p.Active() exactly once per request. internal/site/serve.go takes the snapshot at the top and every later access uses that value. A second Active() call in the same request is the bug this whole design prevents: the two loads could straddle an activation and serve a mixed page. Deployment is immutable after construction, so holding the pointer is free and correct.

2. Activation order is fixed. Project lock → require ready → build the snapshot → commit to SQLite → atomic.Pointer.Store → best-effort webroot.Point. Database before memory, so a crash between the two restarts into the state the database already recorded. The reverse order leaves a process serving A while the database says B. Building the snapshot before the commit means a failure there changes nothing at all.

3. Never trust a client-supplied hash. cas.Store.Put rehashes the received stream and compares against the claimed digest before linking the blob into place; a mismatch is a 400 and the temp file is discarded. Without this a client could claim another project's digest, upload arbitrary bytes, and poison that blob for every project referencing it. This check is what makes cross-project dedup safe — do not move it, skip it for "already known" digests, or trust a digest because the transport was TLS.

4. Tokens never appear in logs, query strings, or error messages. There is a test that runs a request through the middleware into a buffer and greps for the token; keep it passing. The CLI prefers PAGES_TOKEN and --token-file over --token because argv is world-readable via /proc/<pid>/cmdline on a shared runner, and the help text says so. The bootstrap token file and the CLI config file are 0600.

5. All writes go through db.Tx. store.DB exposes Reader() for queries and Tx(ctx, fn) for everything else. There is deliberately no Writer() — the single write connection plus BEGIN IMMEDIATE plus busy retries is the only thing keeping SQLITE_BUSY off the hot path. Tests write through db.Tx too.

6. Upload paths pass both fs.ValidPath and filepath.Localize, and are additionally rejected for NUL or control bytes, a segment over 255 bytes, a total over 4096, duplicates, and case-insensitive collisions (the assembled tree may land on a case-insensitive filesystem, where a collision silently overwrites). One check is not enough: fs.ValidPath allows backslashes and Windows reserved names, Localize allows things ValidPath rejects.

7. The read path serves from the CAS by digest. The only filesystem path constructed while serving is cas/ab/cd/<64 hex>, derived from a [32]byte that came out of a map lookup. No user-controlled string reaches the filesystem, so read-path traversal is not defended against — it is structurally impossible. Serving from the assembled directory instead would give that back.

8. No archive extraction, ever. Files arrive one at a time; the server never unpacks or decompresses anything. That is what makes zip bombs and tar-slip inapplicable rather than mitigated. "Just accept a tarball, it's fewer round trips" reopens both — it needs a fresh security review, not a patch.

9. RequireProject compares resolved project IDs, never the name string from the URL. Names can be reused after a delete; IDs cannot. Same rule for deployments: confirm {id} really belongs to {name} before changing anything.

10. webroot.Reconcile only removes entries that are symlinks pointing inside $DATA_DIR/deployments. $WEBROOT belongs to the operator and will contain files that are none of our business.

11. Assembled files and blobs are 0444, directories 0755, and client-supplied modes are ignored. There is no mode column in the schema on purpose, so a setuid bit has nowhere to be stored even if someone tries to send one. On a hardlinking filesystem the assembled file is the blob's inode — writing to a file under $WEBROOT corrupts it for every project that references it.

12. The CLI's dependency graph is enforced, not documented. cmd/pages/deps_test.go fails if go list -deps ./cmd/pages reaches modernc.org/sqlite, modernc.org/libc, database/sql, github.com/BurntSushi/toml, net/http/httptest, or internal/{store,site,deploy,cas,auth,adminapi}. database/sql is a probe: it can only appear if server code leaked in. If you need a type in both the CLI and the server, it belongs in api/ (stdlib only) or internal/pathutil (stdlib only).

Conventions

Comments explain why, not what. The code already says what it does. A comment earns its place by recording the reasoning that is not recoverable from the code — why this order, why not the obvious alternative, what breaks if someone changes it. Match the density of the surrounding file; several of the sharper decisions are commented in place precisely so a later reader does not "optimize" them away (internal/cache's note about not caching file handles is the canonical example).

Do not over-design. No metrics nobody reads, no abstraction with one implementation, no configuration knob without a caller who needs it. If a simpler thing works, ship the simpler thing.

Dependencies. Three direct ones, on purpose: modernc.org/sqlite (pure Go, so CGO_ENABLED=0 cross-compiles), github.com/BurntSushi/toml (server config only), golang.org/x/sync/errgroup (bounded concurrency, both sides). No web framework, router, ORM, migration tool, CLI framework, logging library (log/slog), or UUID library (crypto/rand + hex). Adding a fourth needs a reason that survives being written down.

Tests are table-driven with names that read as sentences (TestAnInFlightRequestSurvivesTheContentBeingCollected). Two gotchas worth knowing before you debug an unexpected pass: retention tests must patch retention_grace_s = 0, because a deployment that was never activated is aged from created_at and is otherwise protected for the full grace period; and blob collection tests need a negative Service.BlobGrace, since zero means "use the one-hour default", not "collect now".

CLI rendering: cliutil.Bool prints yes/no, and cliutil.Truncate(s, n) returns n-1 characters plus an ellipsis. Both have caught tests out.

Scope

v1 (M0–M5) is complete: deploy, atomic switch, rollback, crash recovery, retention and GC, and the atomicity proof tests.

Deferred, deliberately — do not build these speculatively: precompressed br/gzip variants (the deployment_files.encoding column is already there for it), per-project header rules, host routing (one domain per project, which is also the fix for the same-origin limit), autoindex, a metrics endpoint, a resident-manifest cap, and packaging.

Known limit: one server process per database. The in-memory registry is updated by the process that wrote the change, so a second process on the same SQLite file serves stale content until it restarts. Horizontal scaling needs a change-notification mechanism first.