Files

T

2026-08-15 07:13:00 +00:00

20 KiB

Raw Blame History

Operating simplepages

pages-server is a static-site host built around one guarantee: a request either sees the whole old deployment or the whole new one, never a mixture. Everything below either explains how to run it or tells you where that guarantee stops.

Read Security model and its limits before you put this on a network. Three of the items there are not bugs to be fixed later — they are properties of the design, and one of them (same-origin) decides whether this server is suitable for your workload at all.

1. What it does

CI job                        pages-server                     visitor
------                        ------------                     -------
pages deploy ./dist  ──►  upload only the changed blobs
                          assemble a directory tree
                          ┌─────────────────────────┐
                          │ ONE atomic pointer swap │  ◄── GET /~demo/
                          └─────────────────────────┘
                          repoint $WEBROOT/~demo

Unlike rsync, no file of the new version is visible until every file of it is present and verified. The switch itself is a single atomic.Pointer store, so a request that started before the swap keeps reading the old snapshot to completion — including a large file already halfway down the wire.

Rollback is the same operation pointed at an older deployment id, and costs the same single store.

2. Installation

Build both binaries. The server needs SQLite (pure Go, so no cgo and no C toolchain); the CLI deliberately shares no server code at all.

CGO_ENABLED=0 go build -o /usr/local/bin/pages-server ./cmd/pages-server
CGO_ENABLED=0 go build -o /usr/local/bin/pages         ./cmd/pages

pages is what goes into CI runners. Its dependency graph is enforced by a test (go test ./cmd/pages -run TestCLIImportGraph), so it can never accidentally start dragging in the database driver.

Directories

Path	Contents	Who writes it
`$DATA_DIR/pages.db`	projects, deployments, keys, blob refcounts	server only
`$DATA_DIR/cas/`	content-addressed blobs, `0444`	server only
`$DATA_DIR/deployments/<project_id>/<deployment_id>/`	assembled trees	server only
`$WEBROOT/~<project>`	symlink to the active deployment	server only

$WEBROOT and $DATA_DIR must not overlap; the server refuses to start if they do.

systemd

[Unit]
Description=simplepages static site server
After=network-online.target

[Service]
Type=exec
ExecStartPre=/usr/local/bin/pages-server --config /etc/pages-server/config.toml --check-config
ExecStart=/usr/local/bin/pages-server --config /etc/pages-server/config.toml
Restart=on-failure
StateDirectory=pages-server
ReadWritePaths=/var/lib/pages-server /srv/www
ProtectSystem=strict
PrivateTmp=yes
NoNewPrivileges=yes
UMask=0022

[Install]
WantedBy=multi-user.target

--check-config validates and exits without binding a port, which is why it is safe in ExecStartPre.

Do not use DynamicUser=yes. The UID it allocates rotates, and the assembled trees are hardlinks whose ownership must stay stable across restarts.

3. Configuration

Precedence is flag > PAGES_* environment variable > config file > default.

# /etc/pages-server/config.toml
data_dir   = "/var/lib/pages-server"
webroot    = "/srv/www"
listen     = ":8080"                    # static content
api_listen = "127.0.0.1:8081"           # management API
site_url   = "https://pages.example.com"

trusted_proxy_cidrs = ["127.0.0.1/32", "::1/128"]
assemble_mode       = "auto"            # auto | hardlink | copy | none

log_level  = "info"
log_format = "json"

gc_interval        = "15m"
reconcile_interval = "5m"
shutdown_grace     = "30s"

read_header_timeout = "10s"
read_timeout        = "5m"
idle_timeout        = "120s"

[limits]
max_file_bytes         = 268435456      # 256 MiB
max_manifest_files     = 50000
max_concurrent_uploads = 32
max_manifest_bytes     = 67108864       # 64 MiB of manifest JSON
max_json_bytes         = 1048576        # 1 MiB for ordinary API bodies

Every key above has a matching PAGES_* variable (PAGES_DATA_DIR, PAGES_API_LISTEN, PAGES_ASSEMBLE_MODE, PAGES_RECONCILE_INTERVAL, …). The ones worth overriding on the command line also have flags: --data-dir, --webroot, --listen, --api-listen, --site-url, --assemble-mode, --log-level, --log-format, --gc-interval, --shutdown-grace, plus --config, --check-config and --version. Run pages-server -h for the authoritative list.

listen and api_listen must differ, and webroot may be neither equal to nor nested inside data_dir (in either direction) — the reconciler prunes symlinks from one and the GC deletes trees from the other, so overlapping them would let each destroy the other's state.

The two listeners are separate on purpose

listen serves nothing but site content. api_listen serves nothing but the management API, and defaults to loopback. Neither answers for the other — there is a test that asserts exactly this. Expose the API through your reverse proxy on its own hostname, or not at all; nothing forces a management surface onto the origin your sites are served from.

TLS is the reverse proxy's job in both cases.

`site_url`

Purely cosmetic: it is the origin API responses quote back to a CI job so the job can print where its build landed. Behind a proxy the server cannot learn this name, so you have to tell it. Leaving it empty just omits the url field.

`assemble_mode`

auto (default) — probe for hardlink support at startup, fall back to copy.
hardlink — require hardlinks; fail loudly if the filesystem refuses.
copy — always copy. Correct everywhere, uses disk proportional to content.
none — skip on-disk assembly entirely. Content is served straight from the CAS and $WEBROOT symlinks are not maintained. Choose this only if nothing outside pages-server reads the files.

On overlayfs (Docker's default upper layer, among others) cross-layer link() either triggers a copy-up or fails outright, so auto may silently land on copy. Check the startup log line reporting the detected mode if disk usage surprises you.

`trusted_proxy_cidrs`

X-Forwarded-For is honoured only when the direct peer address falls inside one of these prefixes. The client IP is used for the failed-authentication rate limiter, so getting this wrong either lets one proxy IP absorb everyone's budget (too narrow) or lets a client forge its own identity (too wide). Set it to your proxy's address and nothing else.

4. First run

On first start, if no unrevoked admin key exists, the server mints one and writes the token to $DATA_DIR/bootstrap-token with mode 0600, logging a warning. This is the only place the server ever writes a token to disk.

set -x PAGES_SERVER http://127.0.0.1:8081
set -x PAGES_TOKEN (cat /var/lib/pages-server/bootstrap-token)

pages project create demo
pages key create --project demo --name github-actions -o json   # token shown ONCE

Then delete the bootstrap file. It is not recreated as long as one unrevoked admin key exists.

Give CI a project-scoped key, never the admin one. A project key can create, upload, finalize and activate deployments in its own project and read that project's settings — and nothing else, in any other project.

5. Deploying

pages deploy ./dist --project demo

What happens, and what each step buys you:

Scan. Walk the directory, hash every file with SHA-256 in parallel. Symlinks are refused by default (--follow-symlinks opts in, and still refuses targets outside the tree). Devices, sockets and FIFOs are always refused.
Create. A pending deployment row. No filesystem is touched.
Manifest. Send every path + digest + size; the server answers with the digests it does not already have. This is where the incremental win shows up — a rebuild that changes one file uploads one blob, regardless of how big the site is. The CLI prints the saving.
Upload. Only the missing blobs, bounded concurrency (--concurrency), exponential backoff honouring Retry-After (--retries).
Finalize. The server verifies nothing is missing, then assembles the tree into …/<id>.staging, fsyncs it, and renames it into place.
Activate (unless --activate=false) — the atomic swap.

Interrupted deploys need no special handling. Re-run pages deploy. The blobs that made it are still in the CAS, so the new deployment's manifest negotiation returns a much smaller missing set. There is no resume protocol because none is needed.

Rollback

pages deployment list --project demo    # newest first; ACTIVE marks what is live
pages deployment activate dpl_…         # one pointer store

The superseded deployment stays ready on disk until retention expires it, so rollback costs nothing but the swap. pages deployment show dpl_… --files gives the manifest of any of them if you need to see what a version contained before switching to it, and pages deployment delete dpl_… removes one — except the active one, which answers 409 deployment_active until something else is activated.

Every pages deployment command accepts a project key, so a CI job can roll its own project back without an admin credential. pages system gc and pages system fsck are server-wide and need an admin key.

Base paths — the one thing that will bite you

Sites are served under /~PROJECT/. A build that assumes it lives at / will request /assets/app.js, get a 404, and look broken. Set the base path at build time:

Tool	Setting
Vite	`vite build --base=/~demo/`
Create React App	`PUBLIC_URL=/~demo`
Next.js (static export)	`basePath: '/~demo'`
Hugo	`--baseURL=/~demo/`
plain HTML	`<base href="/~demo/">`

This is inherent to prefix hosting; the server cannot fix it. If you need sites at the root of their own domain, you need host routing (§8).

Project settings

pages project update demo --spa --not-found-file 404.html --retention 20

--index-file (default index.html) — what a directory URL serves.
--not-found-file — served with a 404 status. Without it, 404s are bare.
--spa — serve the index document for unknown paths, but only when the request's Accept header includes text/html. That condition matters: it is why a missing /assets/app.js still returns a real 404 instead of an HTML page with status 200 and an afternoon of Unexpected token '<'.
--cache-control (default public, max-age=0, must-revalidate) — correct by construction here, because every response carries a strong content-hash ETag. Repeat visits are cheap 304s and deploys are visible immediately. Only raise max-age for content served under hashed filenames.
--retention / --retention-grace — see §6.

6. Garbage collection and retention

The sweep runs every gc_interval, and on demand via pages system gc (admin), which is POST /api/v1/gc. pages system gc --dry-run reports what would go without deleting anything — it cannot report the bytes, because nothing was deleted and the content is all still referenced.

Per project, a deployment is deleted when all of these hold:

it is not active;
it was superseded more than retention_grace ago (default 1 h) — a deployment that was never activated is measured from when it was created, so an uploaded-but-unused one is kept for the grace period too;
it is not among the newest retention_count finished deployments (default 10).

Failed deployments are dropped after 24 h. Blobs are removed once their refcount reaches zero and they have been unreferenced for an hour.

Why the grace period exists. A request that has already taken its snapshot and is about to open a blob gets an hour of slack. Combined with POSIX's guarantee that an open file descriptor keeps an inode alive, an in-flight download survives even a forced GC. If the server ever logs a missing blob on the read path, that is a genuine bug — it is logged at ERROR precisely so it cannot be mistaken for normal behaviour.

pages system fsck (POST /api/v1/fsck, admin) recomputes every blob refcount from the manifests and reports drift; --repair ({"repair": true}) corrects it. On a healthy server it always reports none — the counts are maintained by database triggers — so it is for the cases outside normal operation: a restored backup, or a database edited by hand. A count that reads low is the dangerous one, because it is content the collector will delete while a deployment still needs it.

7. Security model and its limits

Three of these are accepted properties of the v1 design, not deferred work. They are stated here so the decision is yours and not a surprise.

7.1 ⚠️ Path routing is not a security boundary between projects

/~a/ and /~b/ are the same origin. That means project A's JavaScript can:

fetch('/~b/secret.json') and read the response;
read and write cookies and localStorage for the whole host;
register a service worker scoped to /, intercepting every request to every project on that host, including future ones.

There is no browser mechanism that prevents this. X-Content-Type-Options: nosniff is always sent, and it does not help here.

Only host projects that trust each other. If you need to host sites from mutually distrusting tenants, give each project its own domain so each gets its own origin. That is host routing, which v1 does not implement.

When hardlink assembly is in use (the default when supported), a file under $WEBROOT/~demo/ and its blob in $DATA_DIR/cas/ are the same inode. Editing that file in place corrupts the blob — and therefore corrupts every other project and every other deployment referencing the same content.

Mitigations in place: blobs and assembled files are 0444, directories 0755, and client-supplied file modes are ignored entirely (there is no mode column in the schema, so a setuid bit can never reach disk).

Treat $WEBROOT as read-only to everything except pages-server. Do not point rsync --delete at it, do not let a deploy script chmod -R it, do not edit a file "just to check something". If you must modify content, deploy it.

Setting assemble_mode = "copy" removes the shared inode at the cost of disk.

7.3 ⚠️ The manifest endpoint is a blob existence oracle

A project-scoped key can put any digest in a manifest and learn from the missing response whether the server already holds that content — including content belonging to another project.

The attacker must already know the exact SHA-256, so this only ever confirms "somebody here hosts this file I already have". It cannot be used to read content or to enumerate anything.

This is the price of cross-project deduplication, which is the feature that makes repeated CI deploys fast. If it matters to you, the fixes are: partition the CAS per project (losing all dedup), or always report digests the calling project has never referenced as missing (losing cross-project dedup only).

7.4 What is structurally prevented

Read-path traversal. The only filesystem path built while serving is cas/ab/cd/<64 hex>, derived from a 32-byte digest that came out of an in-memory map lookup. No user-controlled byte reaches the filesystem on the read path, so traversal is not blocked — it is impossible.
Write-path traversal. Every manifest path must satisfy both fs.ValidPath and filepath.Localize, and NUL bytes, control characters, segments over 255 bytes, total lengths over 4096, duplicates, and case-insensitively colliding paths are all rejected.
Poisoned blobs. The server never trusts a client's digest: it recomputes SHA-256 over the received stream and rejects a mismatch. Without this, a client could claim another project's digest and overwrite shared content.
Zip bombs and tar-slip. There is no archive format and no server-side decompression. Files arrive individually. Do not "improve" this into tarball upload without re-reviewing this section.
Symlink escape from uploaded content. The server creates only directories and hardlinks inside a deployment tree; it never creates a symlink there.

7.5 Tokens

Format pgs_<keyid>_<secret>: an 80-bit key id and a 256-bit random secret. Stored as SHA-256 (not bcrypt — the secret is uniformly random, so there is no dictionary to defend against, and a KDF per request would hand every unauthenticated client a CPU-exhaustion lever), compared in constant time.

Tokens are returned once, at creation.
They are never logged, never accepted in a query string, never included in an error message. A test asserts this by running a request through the middleware and grepping the log output.
Prefer PAGES_TOKEN or --token-file over --token. On a shared CI runner argv is world-readable through /proc/<pid>/cmdline.
Revocation is immediate: the auth cache is invalidated by a generation bump.
The CLI config file is written 0600.

Failed authentications are rate limited per client IP (see trusted_proxy_cidrs). Successful ones are not.

8. Scaling limits

Run exactly one pages-server process per database. The in-memory registry is updated synchronously by the process that performed the write; a second process on the same SQLite file would keep serving stale content indefinitely, because nothing tells it the active deployment changed.

Vertically there is plenty of room — serving is a map lookup plus a file read, and reads never touch SQLite. The bound worth knowing is memory: resident manifests cost roughly 100 bytes per file, and only active deployments are resident. A thousand projects of fifty thousand files each would be about 6 GB.

Horizontal scaling would need a change-notification mechanism (polling deployments.updated_at, or external pub/sub). It is not in v1.

9. Monitoring and troubleshooting

/healthz and /readyz are served on both listeners, so either can be the one your load balancer targets. /healthz never touches the database; /readyz returns 503 until the database is reachable and the registry is loaded.

GET /api/v1/system/info (admin) reports version, uptime, project and deployment counts, CAS size and the detected link mode. pages system info formats it.

Access logs are one line per request with method, path, status, bytes, dur_ms, project, deployment, key_id and req_id.

Symptom	Cause	Fix
Site returns 503	project exists, nothing activated yet	deploy, or activate an existing deployment
Site returns 404 at `/~name/`	no such project (404 and 503 are deliberately different answers)	check the name
Assets 404, page loads	build has the wrong base path	rebuild with `--base=/~name/` (§5)
`$WEBROOT/~name` missing	external tool removed it, or `assemble_mode = "none"`	the reconciler restores it within `reconcile_interval`
Disk grows steadily	retention too generous, or copy-mode assembly	lower `--retention`, run `pages system gc`, check the link mode in the startup log
ERROR "blob missing from the content store"	genuine bug, or somebody deleted CAS files	run `pages system fsck`; redeploy affected projects
Deploy hangs at upload	`max_concurrent_uploads` saturated by another job	wait, or raise the limit

Recovering a database with an intact CAS is normal: the server marks blobs whose files are gone as absent at startup and the next deploy re-uploads them. Losing both means redeploying, which for a CI-driven site is one pipeline run.

Graceful shutdown

SIGINT/SIGTERM stops accepting connections, gives in-flight requests shutdown_grace to finish, cancels background workers, checkpoints the WAL and closes cleanly. A second signal forces an immediate close.

10. Backups

Stop the server, or use sqlite3 pages.db ".backup" — a plain cp of a live WAL database is not safe. Then copy $DATA_DIR/cas/.

$DATA_DIR/deployments/ does not need backing up: it is reconstructible from the CAS and the database. $WEBROOT does not need backing up either; it holds only symlinks, regenerated on the next activation and by the reconciler within reconcile_interval.

20 KiB Raw Blame History