pages/docs/operations.md

# Operating simplepages

`pages-server` is a static-site host built around one guarantee: **a request
either sees the whole old deployment or the whole new one, never a mixture.**
Everything below either explains how to run it or tells you where that
guarantee stops.

Read [Security model and its limits](#security-model-and-its-limits) before you
put this on a network. Three of the items there are not bugs to be fixed later —
they are properties of the design, and one of them (same-origin) decides whether
this server is suitable for your workload at all.

---

## 1. What it does

```
CI job                        pages-server                     visitor
------                        ------------                     -------
pages deploy ./dist  ──►  upload only the changed blobs
                          assemble a directory tree
                          ┌─────────────────────────┐
                          │ ONE atomic pointer swap │  ◄── GET /~demo/
                          └─────────────────────────┘
                          repoint $WEBROOT/~demo
```

Unlike `rsync`, no file of the new version is visible until every file of it is
present and verified. The switch itself is a single `atomic.Pointer` store, so a
request that started before the swap keeps reading the old snapshot to
completion — including a large file already halfway down the wire.

Rollback is the same operation pointed at an older deployment id, and costs the
same single store.

---

## 2. Installation

Build both binaries. The server needs SQLite (pure Go, so no cgo and no C
toolchain); the CLI deliberately shares no server code at all.

```sh
CGO_ENABLED=0 go build -o /usr/local/bin/pages-server ./cmd/pages-server
CGO_ENABLED=0 go build -o /usr/local/bin/pages         ./cmd/pages
```

`pages` is what goes into CI runners. Its dependency graph is enforced by a
test (`go test ./cmd/pages -run TestCLIImportGraph`), so it can never
accidentally start dragging in the database driver.

### Directories

| Path | Contents | Who writes it |
|---|---|---|
| `$DATA_DIR/pages.db` | projects, deployments, keys, blob refcounts | server only |
| `$DATA_DIR/cas/` | content-addressed blobs, `0444` | server only |
| `$DATA_DIR/deployments/<project_id>/<deployment_id>/` | assembled trees | server only |
| `$WEBROOT/~<project>` | symlink to the active deployment | server only |

`$WEBROOT` and `$DATA_DIR` must not overlap; the server refuses to start if they
do.

### systemd

```ini
[Unit]
Description=simplepages static site server
After=network-online.target

[Service]
Type=exec
ExecStartPre=/usr/local/bin/pages-server --config /etc/pages-server/config.toml --check-config
ExecStart=/usr/local/bin/pages-server --config /etc/pages-server/config.toml
Restart=on-failure
StateDirectory=pages-server
ReadWritePaths=/var/lib/pages-server /srv/www
ProtectSystem=strict
PrivateTmp=yes
NoNewPrivileges=yes
UMask=0022

[Install]
WantedBy=multi-user.target
```

`--check-config` validates and exits without binding a port, which is why it is
safe in `ExecStartPre`.

**Do not use `DynamicUser=yes`.** The UID it allocates rotates, and the
assembled trees are hardlinks whose ownership must stay stable across restarts.

---

## 3. Configuration

Precedence is **flag > `PAGES_*` environment variable > config file > default**.

```toml
# /etc/pages-server/config.toml
data_dir   = "/var/lib/pages-server"
webroot    = "/srv/www"
listen     = ":8080"                    # static content
api_listen = "127.0.0.1:8081"           # management API
site_url   = "https://pages.example.com"

trusted_proxy_cidrs = ["127.0.0.1/32", "::1/128"]
assemble_mode       = "auto"            # auto | hardlink | copy | none

log_level  = "info"
log_format = "json"

gc_interval        = "15m"
reconcile_interval = "5m"
shutdown_grace     = "30s"

read_header_timeout = "10s"
read_timeout        = "5m"
idle_timeout        = "120s"

[limits]
max_file_bytes         = 268435456      # 256 MiB
max_manifest_files     = 50000
max_concurrent_uploads = 32
max_manifest_bytes     = 67108864       # 64 MiB of manifest JSON
max_json_bytes         = 1048576        # 1 MiB for ordinary API bodies
```

Every key above has a matching `PAGES_*` variable (`PAGES_DATA_DIR`,
`PAGES_API_LISTEN`, `PAGES_ASSEMBLE_MODE`, `PAGES_RECONCILE_INTERVAL`, …). The
ones worth overriding on the command line also have flags: `--data-dir`,
`--webroot`, `--listen`, `--api-listen`, `--site-url`, `--assemble-mode`,
`--log-level`, `--log-format`, `--gc-interval`, `--shutdown-grace`, plus
`--config`, `--check-config` and `--version`. Run `pages-server -h` for the
authoritative list.

`listen` and `api_listen` must differ, and `webroot` may be neither equal to nor
nested inside `data_dir` (in either direction) — the reconciler prunes symlinks
from one and the GC deletes trees from the other, so overlapping them would let
each destroy the other's state.

### The two listeners are separate on purpose

`listen` serves nothing but site content. `api_listen` serves nothing but the
management API, and defaults to loopback. Neither answers for the other — there
is a test that asserts exactly this. Expose the API through your reverse proxy on
its own hostname, or not at all; nothing forces a management surface onto the
origin your sites are served from.

TLS is the reverse proxy's job in both cases.

### `site_url`

Purely cosmetic: it is the origin API responses quote back to a CI job so the
job can print where its build landed. Behind a proxy the server cannot learn
this name, so you have to tell it. Leaving it empty just omits the `url` field.

### `assemble_mode`

- `auto` (default) — probe for hardlink support at startup, fall back to copy.
- `hardlink` — require hardlinks; fail loudly if the filesystem refuses.
- `copy` — always copy. Correct everywhere, uses disk proportional to content.
- `none` — skip on-disk assembly entirely. Content is served straight from the
  CAS and **`$WEBROOT` symlinks are not maintained**. Choose this only if
  nothing outside `pages-server` reads the files.

On overlayfs (Docker's default upper layer, among others) cross-layer `link()`
either triggers a copy-up or fails outright, so `auto` may silently land on
copy. Check the startup log line reporting the detected mode if disk usage
surprises you.

### `trusted_proxy_cidrs`

`X-Forwarded-For` is honoured **only** when the direct peer address falls inside
one of these prefixes. The client IP is used for the failed-authentication rate
limiter, so getting this wrong either lets one proxy IP absorb everyone's
budget (too narrow) or lets a client forge its own identity (too wide). Set it
to your proxy's address and nothing else.

---

## 4. First run

On first start, if no unrevoked admin key exists, the server mints one and
writes the token to `$DATA_DIR/bootstrap-token` with mode `0600`, logging a
warning. **This is the only place the server ever writes a token to disk.**

```fish
set -x PAGES_SERVER http://127.0.0.1:8081
set -x PAGES_TOKEN (cat /var/lib/pages-server/bootstrap-token)

pages project create demo
pages key create --project demo --name github-actions -o json   # token shown ONCE
```

Then delete the bootstrap file. It is not recreated as long as one unrevoked
admin key exists.

Give CI a **project-scoped** key, never the admin one. A project key can create,
upload, finalize and activate deployments in its own project and read that
project's settings — and nothing else, in any other project.

---

## 5. Deploying

```sh
pages deploy ./dist --project demo
```

What happens, and what each step buys you:

1. **Scan.** Walk the directory, hash every file with SHA-256 in parallel.
   Symlinks are refused by default (`--follow-symlinks` opts in, and still
   refuses targets outside the tree). Devices, sockets and FIFOs are always
   refused.
2. **Create.** A `pending` deployment row. No filesystem is touched.
3. **Manifest.** Send every path + digest + size; the server answers with the
   digests it does *not* already have. This is where the incremental win shows
   up — a rebuild that changes one file uploads one blob, regardless of how big
   the site is. The CLI prints the saving.
4. **Upload.** Only the missing blobs, bounded concurrency (`--concurrency`),
   exponential backoff honouring `Retry-After` (`--retries`).
5. **Finalize.** The server verifies nothing is missing, then assembles the tree
   into `…/<id>.staging`, fsyncs it, and renames it into place.
6. **Activate** (unless `--activate=false`) — the atomic swap.

**Interrupted deploys need no special handling.** Re-run `pages deploy`. The
blobs that made it are still in the CAS, so the new deployment's manifest
negotiation returns a much smaller missing set. There is no resume protocol
because none is needed.

### Rollback

```sh
pages deployment list --project demo    # newest first; ACTIVE marks what is live
pages deployment activate dpl_…         # one pointer store
```

The superseded deployment stays `ready` on disk until retention expires it, so
rollback costs nothing but the swap. `pages deployment show dpl_… --files` gives
the manifest of any of them if you need to see what a version contained before
switching to it, and `pages deployment delete dpl_…` removes one — except the
active one, which answers `409 deployment_active` until something else is
activated.

Every `pages deployment` command accepts a project key, so a CI job can roll its
own project back without an admin credential. `pages system gc` and `pages
system fsck` are server-wide and need an admin key.

### Base paths — the one thing that will bite you

Sites are served under `/~PROJECT/`. A build that assumes it lives at `/` will
request `/assets/app.js`, get a 404, and look broken. **Set the base path at
build time:**

| Tool | Setting |
|---|---|
| Vite | `vite build --base=/~demo/` |
| Create React App | `PUBLIC_URL=/~demo` |
| Next.js (static export) | `basePath: '/~demo'` |
| Hugo | `--baseURL=/~demo/` |
| plain HTML | `<base href="/~demo/">` |

This is inherent to prefix hosting; the server cannot fix it. If you need sites
at the root of their own domain, you need host routing (§8).

### Project settings

```sh
pages project update demo --spa --not-found-file 404.html --retention 20
```

- `--index-file` (default `index.html`) — what a directory URL serves.
- `--not-found-file` — served *with* a 404 status. Without it, 404s are bare.
- `--spa` — serve the index document for unknown paths, **but only when the
  request's `Accept` header includes `text/html`.** That condition matters: it
  is why a missing `/assets/app.js` still returns a real 404 instead of an HTML
  page with status 200 and an afternoon of `Unexpected token '<'`.
- `--cache-control` (default `public, max-age=0, must-revalidate`) — correct by
  construction here, because every response carries a strong content-hash ETag.
  Repeat visits are cheap 304s and deploys are visible immediately. Only raise
  `max-age` for content served under hashed filenames.
- `--retention` / `--retention-grace` — see §6.

---

## 6. Garbage collection and retention

The sweep runs every `gc_interval`, and on demand via `pages system gc` (admin),
which is `POST /api/v1/gc`. `pages system gc --dry-run` reports what would go
without deleting anything — it cannot report the bytes, because nothing was
deleted and the content is all still referenced.

Per project, a deployment is deleted when **all** of these hold:

- it is not active;
- it was superseded more than `retention_grace` ago (default 1 h) — a
  deployment that was never activated is measured from when it was created, so
  an uploaded-but-unused one is kept for the grace period too;
- it is not among the newest `retention_count` finished deployments (default 10).

Failed deployments are dropped after 24 h. Blobs are removed once their
refcount reaches zero and they have been unreferenced for an hour.

**Why the grace period exists.** A request that has already taken its snapshot
and is about to open a blob gets an hour of slack. Combined with POSIX's
guarantee that an open file descriptor keeps an inode alive, an in-flight
download survives even a forced GC. If the server ever logs a missing blob on
the read path, that is a genuine bug — it is logged at ERROR precisely so it
cannot be mistaken for normal behaviour.

`pages system fsck` (`POST /api/v1/fsck`, admin) recomputes every blob refcount
from the manifests and reports drift; `--repair` (`{"repair": true}`) corrects
it. On a healthy server it always reports none — the counts are maintained by
database triggers — so it is for the cases outside normal operation: a restored
backup, or a database edited by hand. A count that reads *low* is the dangerous
one, because it is content the collector will delete while a deployment still
needs it.

---

## 7. Security model and its limits

Three of these are accepted properties of the v1 design, not deferred work.
They are stated here so the decision is yours and not a surprise.

### 7.1 ⚠️ Path routing is not a security boundary between projects

`/~a/` and `/~b/` are the **same origin**. That means project A's JavaScript
can:

- `fetch('/~b/secret.json')` and read the response;
- read and write cookies and `localStorage` for the whole host;
- register a service worker scoped to `/`, intercepting **every request to
  every project on that host**, including future ones.

There is no browser mechanism that prevents this. `X-Content-Type-Options:
nosniff` is always sent, and it does not help here.

**Only host projects that trust each other.** If you need to host sites from
mutually distrusting tenants, give each project its own domain so each gets its
own origin. That is host routing, which v1 does not implement.

### 7.2 ⚠️ Assembled files share inodes with CAS blobs

When hardlink assembly is in use (the default when supported), a file under
`$WEBROOT/~demo/` and its blob in `$DATA_DIR/cas/` are **the same inode**.
Editing that file in place corrupts the blob — and therefore corrupts every
other project and every other deployment referencing the same content.

Mitigations in place: blobs and assembled files are `0444`, directories `0755`,
and client-supplied file modes are ignored entirely (there is no `mode` column
in the schema, so a setuid bit can never reach disk).

**Treat `$WEBROOT` as read-only to everything except `pages-server`.** Do not
point `rsync --delete` at it, do not let a deploy script `chmod -R` it, do not
edit a file "just to check something". If you must modify content, deploy it.

Setting `assemble_mode = "copy"` removes the shared inode at the cost of disk.

### 7.3 ⚠️ The manifest endpoint is a blob existence oracle

A project-scoped key can put any digest in a manifest and learn from the
`missing` response whether the server already holds that content — including
content belonging to another project.

The attacker must already know the exact SHA-256, so this only ever confirms
"somebody here hosts this file I already have". It cannot be used to read
content or to enumerate anything.

This is the price of cross-project deduplication, which is the feature that
makes repeated CI deploys fast. If it matters to you, the fixes are: partition
the CAS per project (losing all dedup), or always report digests the calling
project has never referenced as missing (losing cross-project dedup only).

### 7.4 What is structurally prevented

- **Read-path traversal.** The only filesystem path built while serving is
  `cas/ab/cd/<64 hex>`, derived from a 32-byte digest that came out of an
  in-memory map lookup. No user-controlled byte reaches the filesystem on the
  read path, so traversal is not blocked — it is impossible.
- **Write-path traversal.** Every manifest path must satisfy both
  `fs.ValidPath` and `filepath.Localize`, and NUL bytes, control characters,
  segments over 255 bytes, total lengths over 4096, duplicates, and
  case-insensitively colliding paths are all rejected.
- **Poisoned blobs.** The server never trusts a client's digest: it recomputes
  SHA-256 over the received stream and rejects a mismatch. Without this, a
  client could claim another project's digest and overwrite shared content.
- **Zip bombs and tar-slip.** There is no archive format and no server-side
  decompression. Files arrive individually. *Do not "improve" this into tarball
  upload without re-reviewing this section.*
- **Symlink escape from uploaded content.** The server creates only directories
  and hardlinks inside a deployment tree; it never creates a symlink there.

### 7.5 Tokens

Format `pgs_<keyid>_<secret>`: an 80-bit key id and a 256-bit random secret.
Stored as SHA-256 (not bcrypt — the secret is uniformly random, so there is no
dictionary to defend against, and a KDF per request would hand every
unauthenticated client a CPU-exhaustion lever), compared in constant time.

- Tokens are returned **once**, at creation.
- They are **never logged, never accepted in a query string, never included in
  an error message.** A test asserts this by running a request through the
  middleware and grepping the log output.
- Prefer `PAGES_TOKEN` or `--token-file` over `--token`. On a shared CI runner
  `argv` is world-readable through `/proc/<pid>/cmdline`.
- Revocation is immediate: the auth cache is invalidated by a generation bump.
- The CLI config file is written `0600`.

Failed authentications are rate limited per client IP (see
`trusted_proxy_cidrs`). Successful ones are not.

---

## 8. Scaling limits

**Run exactly one `pages-server` process per database.** The in-memory registry
is updated synchronously by the process that performed the write; a second
process on the same SQLite file would keep serving stale content indefinitely,
because nothing tells it the active deployment changed.

Vertically there is plenty of room — serving is a map lookup plus a file read,
and reads never touch SQLite. The bound worth knowing is memory: resident
manifests cost roughly 100 bytes per file, and only active deployments are
resident. A thousand projects of fifty thousand files each would be about 6 GB.

Horizontal scaling would need a change-notification mechanism (polling
`deployments.updated_at`, or external pub/sub). It is not in v1.

---

## 9. Monitoring and troubleshooting

`/healthz` and `/readyz` are served on **both** listeners, so either can be the
one your load balancer targets. `/healthz` never touches the database;
`/readyz` returns 503 until the database is reachable and the registry is
loaded.

`GET /api/v1/system/info` (admin) reports version, uptime, project and
deployment counts, CAS size and the detected link mode. `pages system info`
formats it.

Access logs are one line per request with `method`, `path`, `status`, `bytes`,
`dur_ms`, `project`, `deployment`, `key_id` and `req_id`.

| Symptom | Cause | Fix |
|---|---|---|
| Site returns 503 | project exists, nothing activated yet | deploy, or activate an existing deployment |
| Site returns 404 at `/~name/` | no such project (404 and 503 are deliberately different answers) | check the name |
| Assets 404, page loads | build has the wrong base path | rebuild with `--base=/~name/` (§5) |
| `$WEBROOT/~name` missing | external tool removed it, or `assemble_mode = "none"` | the reconciler restores it within `reconcile_interval` |
| Disk grows steadily | retention too generous, or copy-mode assembly | lower `--retention`, run `pages system gc`, check the link mode in the startup log |
| ERROR "blob missing from the content store" | genuine bug, or somebody deleted CAS files | run `pages system fsck`; redeploy affected projects |
| Deploy hangs at upload | `max_concurrent_uploads` saturated by another job | wait, or raise the limit |

**Recovering a database with an intact CAS** is normal: the server marks blobs
whose files are gone as absent at startup and the next deploy re-uploads them.
Losing both means redeploying, which for a CI-driven site is one pipeline run.

### Graceful shutdown

`SIGINT`/`SIGTERM` stops accepting connections, gives in-flight requests
`shutdown_grace` to finish, cancels background workers, checkpoints the WAL and
closes cleanly. A second signal forces an immediate close.

---

## 10. Backups

Stop the server, or use `sqlite3 pages.db ".backup"` — a plain `cp` of a live
WAL database is not safe. Then copy `$DATA_DIR/cas/`.

`$DATA_DIR/deployments/` does **not** need backing up: it is reconstructible
from the CAS and the database. `$WEBROOT` does not need backing up either; it
holds only symlinks, regenerated on the next activation and by the reconciler
within `reconcile_interval`.