syncbot/AGENTS.md

AGENTS.md

Notes for anyone (human or agent) changing this repository. User-facing docs live in README.md; this file is about the code.

What this is

A daemon that mirrors git repositories from src to dst on a timer, driven by a hot-reloadable TOML file. Go 1.24, one third-party dependency (github.com/BurntSushi/toml), ~1500 lines of non-test code.

Design goals, in priority order: do not corrupt or delete anything on dst, stay small in memory, stay simple. When a change trades any of these for convenience, it is probably the wrong change.

Commands

go build ./...
go test ./...                       # includes end-to-end tests that shell out to real git
go test -race -count=1 ./...        # what CI runs
go vet ./...
gofmt -l .                          # must print nothing; CI fails otherwise
go run . -config config.example.toml -check   # needs SRC_TOKEN set to anything

The tests need a git binary on PATH. They create real repositories under t.TempDir() and never touch the network.

Layout

main.go                    flags, logger, signal handling, config file watcher
signal_unix.go/_other.go   build-tagged SIGHUP wiring (no-op on non-unix)
internal/config/           TOML parsing, validation, defaulting → []Job
internal/gitx/             hermetic wrapper around the git CLI
  proc_unix.go/_other.go   build-tagged process-group creation and kill
internal/syncer/           one sync cycle for one repository
internal/manager/          per-repo goroutines, reload diffing, backoff
  http.go                  /healthz /readyz /status /metrics

Dependency direction is strictly main → manager → syncer → gitx → config. Nothing under internal/ imports manager.

Invariants

These are load-bearing. Breaking one is a bug even if the tests still pass.

1. config.Job is fully resolved. All defaulting, env expansion and validation happen in config.resolve. Downstream code never asks "was this field set?" — it reads the value. This is also what makes reload diffing work: Apply decides whether a repo changed with reflect.DeepEqual on Job, so a Job must contain no pointers, maps, funcs, or timestamps, and must be deterministic for identical input.

2. Repository URLs never reach disk. They are passed as command-line arguments to git fetch / git push / git ls-remote, never written into .git/config via git remote add. URLs may embed tokens; mirror directories may outlive the process.

3. Secrets are scrubbed from every string that escapes. gitx.Run runs both its log lines and git's output through scrub using Options.Secrets, and URLs through RedactURL. Any new code path that logs or wraps git output must do the same. There is a test asserting a token never appears in an error.

4. Every git invocation goes through gitx.Run. It supplies the isolated HOME, disables system/global gitconfig, drops inherited GIT_SSH_COMMAND and friends, sets GIT_TERMINAL_PROMPT=0, and puts the child in its own process group so a timeout kills ssh too. Calling exec.Command("git", ...) directly anywhere else reintroduces all of those problems.

5. The syncer holds no state between cycles. Every cycle re-derives truth from ls-remote on both ends. Do not add a state file, a "last synced SHA" cache, or an in-memory map[repo]refs shortcut — the self-healing behaviour (recovering from external force-pushes and partial pushes) comes entirely from not trusting anything remembered.

6. An empty source never empties a destination unless allow_empty = true. The guard is in syncer.Sync; keep it before the push, not inside it.

7. One goroutine per mirror directory, ever. Apply cancels the goroutine being replaced and passes its done channel to the successor as waitFor, so the successor blocks until the predecessor has exited. Two goroutines running git in the same bare repo will corrupt it.

8. Apply must not block. It is called from the config watcher; a repo may be 25 minutes into a 30-minute sync. Hence the waitFor handoff above rather than waiting inline.

Gotchas discovered the hard way

• limitedWriter.Write must return len(p), not the number of bytes it chose to keep. os/exec treats a short write as an error and would abort an otherwise healthy git run once output exceeded the cap.

• signal.Notify with an empty slice subscribes to every signal. The reloadSignals() call site is guarded with a length check for the non-unix build where the slice is empty.

• Git refuses to delete the branch that HEAD points at (receive.denyDeleteCurrent). This shows up when pruning a renamed default branch; GitHub behaves the same way. It is server-side behaviour, not something to work around in the client. The syncer test fixture sets receive.denyDeleteCurrent ignore on its bare dst for exactly this reason. Documented in the README's troubleshooting section.

• url.UserPassword percent-encodes. RedactURL uses the literal string redacted as the placeholder; *** came back as %2A%2A%2A.

• Deploy keys mounted read-only are usually 0644 and ssh rejects them. gitx.usableKey stages a 0600 copy in the per-sync temp dir. Do not "fix" this by chmod'ing the original — the operator often cannot make it writable.

• IdentitiesOnly=yes is required. Without it ssh offers every key the agent knows about and GitHub closes the connection with "too many authentication failures" before reaching the right one.

• expandEnv deliberately only understands ${VAR}. Bare $VAR is left alone because $ is common in passwords, and an undefined variable is a hard error rather than an empty expansion, which would otherwise produce a URL that fails in a confusing way much later.

Testing conventions

internal/syncer/syncer_test.go has a harness that builds real source and bare destination repositories and drives Sync against them. New sync behaviour belongs there rather than in a mock — the interesting bugs in this program are all in how git actually behaves. Existing cases cover first sync, no-op cycles, new commits, prune, force-push after a rewrite, repairing drift on dst, the empty-source refusal, ref filtering, unreachable sources, and timeouts.

Manager tests assert reload semantics: unchanged repos keep running, changed repos restart, removed repos stop. They run under -race; the wg in Manager exists because a goroutine replaced by a reload could otherwise outlive Stop and race t.TempDir() cleanup.

When adding a config field

1. Add it to Repo (and Global if it should be inheritable) in config.go.
2. Resolve it in resolveJob — with an explicit default, never a zero value that downstream code has to interpret.
3. Add it to Job, keeping the type comparable (see invariant 1).
4. Thread it through syncer.Sync / gitx.
5. Document it in the README table and, if it is interesting, config.example.toml.
6. Add a case to config_test.go. unknownKeys rejects unrecognised keys, so a field that is parsed but not registered will make previously valid configs fail.