Production deployment
End-to-end runbook for taking auto-swe from a fresh clone to a running production deployment. For local development, use the quickstart in README.md. For OAuth provider configuration, see oauth-setup.md.
Before and after you upgrade, read
upgrade-notes.md. It lists fixed defects that exposed credentials in earlier builds (Kubernetes audit logs, Docker host process arguments, Pod objects) and the rotation and clean-up each one needs. Check which entries match your backend and the commit your worker image was built from.
0. What you’re deploying
Section titled “0. What you’re deploying”Six long-running processes plus one Docker daemon, and two more that are optional:
| Service | Image | Purpose |
|---|---|---|
postgres |
pgvector/pgvector:pg18 |
App DB — relational state + pgvector for semantic memory. |
postgres-temporal |
postgres:18-alpine |
Separate DB for Temporal history. Do not combine with the app DB. |
temporal (server+admin+ui) |
temporalio/server:1.31.2 + admin-tools 1.31 + ui 2.53.3 |
Workflow orchestration runtime + setup container + web UI on :8233. |
gateway |
built from packages/gateway/Dockerfile |
Fastify HTTP API on :8080. Stateless, scale horizontally. |
worker |
built from packages/worker/Dockerfile |
Temporal worker. Spawns ephemeral Docker workspaces through docker-socket-proxy. |
docker-socket-proxy |
haproxy:3.4-alpine (DOCKER_SOCKET_PROXY_IMAGE) with infra/docker-socket-proxy/haproxy.cfg |
Filters the Docker API for the worker; the only app container that mounts the host socket. Its idle timeouts (65 min) outlast the longest container step. |
web |
built from packages/web/Dockerfile |
Next.js dashboard on :3000. Stateless, scale horizontally. |
otel-lgtm (optional) |
grafana/otel-lgtm:0.34.0 |
Grafana + Loki + Tempo + Mimir bundle for traces, logs, metrics. |
| object store (optional) | dxflrs/garage (bundled) / AWS S3 / Cloudflare R2 / Backblaze B2 |
S3-compatible artifact store for large step outputs (diffs, logs, scan reports). Without it the worker falls back to Postgres-inline storage which inflates the app DB. |
The worker reaches the Docker daemon through docker-socket-proxy (DOCKER_HOST=tcp://docker-socket-proxy:2375, on an internal network only the two share) and spawns ephemeral node:24-alpine-style containers per work request. The base image comes from the connection’s executorImage, falling back to the WORKSPACE_IMAGE environment variable (default node:24-alpine); an explicit image on a node still wins. Leave a connection’s executor image blank to inherit that default — the column has no database default, so blank means “inherit” rather than a pinned value, and a connection that names an image keeps it. Clearing the field in the connection’s settings is how you opt one back in (see architecture.md §8 for the container’s hardening posture). The proxy is haproxy with a path allowlist in infra/docker-socket-proxy/haproxy.cfg: it passes only the API groups the worker uses (_ping, version, containers — which covers docker cp and pause —, exec, images, volumes) and answers everything else with a 403 (build, networks, swarm, services, secrets, plugins, system, info, events). The file names what each allowed path is for; a new Docker operation in the worker needs its path there. It limits the API surface, not container options: the worker may still create a container with --privileged or a host bind mount, so anyone with code execution inside the worker container can still reach root on its host. Keep the worker host isolated.
The workspace clones and pushes with the run’s GitHub credential (the platform’s, or the launcher’s
own) inside the executor image, so choosing the image is choosing code that sees that credential.
Only a platform admin may set a connection’s executor image; a team lead may clear it, or save the
connection with the image it already has. The team shell-image allowlist does not apply: its entries
are approved for sandboxed shell steps, which never hold a credential. build-executor.yml writes
each freshly tagged image to its repository, so its GATEWAY_TOKEN must belong to a platform
admin. It builds defaults/Dockerfile.node unless a run sets use_repo_dockerfile, which builds the
repository’s own .auto-swe/Dockerfile instead. defaults/ also carries Dockerfile.python and
Dockerfile.jvm and Dockerfile.go; the stack input (node|python|jvm|go, default node) selects which curated
Dockerfile is built.
Prebuilt executor images. docker.yml builds the four curated Dockerfiles on every pull request
and, after the same CI gate as the app images, publishes them beside them as
ghcr.io/<repository>/executor-node, executor-python, executor-jvm and executor-go (and to
EXTERNAL_REGISTRY_URL when that is set), tagged main, latest and <timestamp>-<commit>. Point
WORKSPACE_IMAGE (the Node image) and WORKSPACE_IMAGE_PYTHON, _JVM and _GO at them to give a
repository the image for its detected stack (repositories.md).
Reference them by digest (…@sha256:<digest>), not by a tag: the image runs next to the clone
credential, and a tag that moves changes what runs there without a deploy. A GHCR package is private
until its visibility is changed, so a worker, and on Kubernetes the pull secret named in
K8S_WORKSPACE_IMAGE_PULL_SECRETS, needs access to it.
Each build also records its settings on the connection (executorImageBuild). A weekly schedule
(cron: '0 6 * * 1') rebuilds every active repository that has such a record, so the workflow’s
GATEWAY_TOKEN must be able to list repositories as well as update them.
The workspace image tracks its own major, not the services’. It stays on node:24-alpine while the gateway, worker and web images run Node 26, because that image puts yarn, pnpm and corepack on PATH and the Node 26 image puts none of them there — Corepack is no longer bundled with the Node distribution.
This is a harder constraint than “some target repos use Yarn”. The built-in quality gates are yarn commands — DEFAULT_COMMANDS in activities/qualityGates.ts defines runBuild, runLint, runTests, runTypecheck and runVulnScan that way — and they run with docker exec inside this container. They are the fallback for a repository whose detected root is Yarn, Node with no recorded package manager, or undetected; a detected npm or pnpm root, or a Python, JVM or Go root, runs its own commands instead (repositories.md). On an image without yarn, every gate that falls back exits 127, and the failure reads as the agent producing broken code rather than as a missing tool. A per-connection gateCommands override sidesteps it, but that is opt-in.
So raise this default only alongside an image that provides yarn — not by pointing it at a newer stock upstream tag. defaults/Dockerfile.node, the base for executor images built by build-executor.yml, is on Node 26 and installs Corepack explicitly for exactly that reason.
Two things to know about the service images themselves. Their Node major is a Current release, not an LTS one, until Node 26 reaches LTS in October 2026. And the worker’s runtime stage is Debian-based, so moving it to Node 26 also moved it from bookworm to trixie (glibc 2.36 → 2.41); that stage installs no distro packages, so there is nothing pinned to break, but an OS-level surprise there will not look Node-related.
Local-dev shortcut.
yarn docker:infra:upbrings up Garage (thegaragecontainer indocker-compose.infra.yml) and provisions the access key and theauto-swe-artifactsbucket on first boot — there is no separate bucket-create step. Uncomment theARTIFACT_S3_*block in.env.example(defaults match the container) to flip the worker onto S3 mode locally. Garage ships no web console; useaws s3 ls --endpoint-url http://localhost:9000or read./data/garagedirectly.
1. Pre-flight checklist
Section titled “1. Pre-flight checklist”Before touching infrastructure, gather these:
- Domain + TLS. Reverse-proxy in front of the gateway (
https://api.example.com) and the web app (https://app.example.com). Both must serve HTTPS; better-auth refuses to issue secure cookies otherwise. The proxy must pass the gateway’s/api/auth/*,/api/v1/*and/.well-known/*paths through unchanged (the OAuth authorization server for MCP clients is discovered at/.well-known/oauth-authorization-server/api/auth). - GitHub PAT with
reposcope, or a GitHub App (short-lived installation tokens; seegithub-app-setup.md). - GitHub webhook secret — any strong random string; you’ll add it to GitHub repo webhooks pointing at
https://api.example.com/api/v1/webhooks/git. - LLM provider key(s) — configured via the admin UI (
/models) after first boot. There is no env-var fallback for LLM credentials: model + credential config is fully DB-driven (seemodel-configuration.md). - Email transport — pick one of SMTP (
SMTP_HOST/PORT/USER/PASS+AUTH_FROM_EMAIL) or Resend (RESEND_API_KEY+AUTH_FROM_EMAIL). Required for magic-link sign-in and password reset outside development — without one those requests fail, and the gateway logs an error that never contains the link. Only withNODE_ENV=developmentortestdoes it print the link to stdout instead. - OAuth credentials (optional but recommended) — register a GitHub OAuth app, a Google OAuth client, and/or an Okta OIDC app, callback
{BETTER_AUTH_URL}/api/auth/callback/{github,google,okta}. Credentials are environment variables (GITHUB_CLIENT_ID/SECRET,GOOGLE_CLIENT_ID/SECRET,OKTA_ISSUER+OKTA_CLIENT_ID/SECRET). Seeoauth-setup.md. - Slack credentials (optional) — configured via
/admin/integrations(Slack tab) after first boot. - S3-compatible artifact store (optional but recommended in prod) — configured with the
ARTIFACT_S3_*environment variables. Without a bucket, large step outputs are stored inline in Postgres.
Generate strong secrets:
openssl rand -base64 32 # BETTER_AUTH_SECRET (must be ≥32 chars)openssl rand -base64 48 # JWT_SECRETopenssl rand -base64 24 # SEED_ADMIN_PASSWORD (or use the IdP)2. Required environment variables
Section titled “2. Required environment variables”The full annotated template is .env.example. yarn env:setup builds a .env from it:
yarn env:setup init --profile prod # prompts for the API and web domains on a TTYyarn env:setup init --profile prod --yes --set DOMAIN_API=api.example.com --set DOMAIN_APP=app.example.comyarn env:setup check --profile prod # pre-flight before `just prod-*`yarn env:setup sync # after pulling a change to .env.exampleThe prod profile generates a strong URL-safe POSTGRES_PASSWORD (and rebuilds DATABASE_URL from it),
ARTIFACT_S3_ACCESS_KEY / ARTIFACT_S3_SECRET_KEY and GARAGE_RPC_SECRET, sets NODE_ENV=production,
and derives PUBLIC_URL, BETTER_AUTH_URL and NEXT_PUBLIC_API_URL from DOMAIN_API and CORS_ORIGIN
and WEB_URL from DOMAIN_APP. check --profile prod exits non-zero while any of those URLs is unset or
points at localhost / 127.0.0.1, while a dev default (POSTGRES_PASSWORD=password, the
autoswelocal… S3 credentials) remains, or while a required secret is empty or malformed. sync re-renders
the file from the current template keeping every value you set, moves keys the template does not know into a
trailing “Local additions” section, and writes a .env.bak-<timestamp> first. --dry-run prints the change
with secret values masked. Secrets that already have a value are never regenerated.
The minimum-viable production set:
# DatastoreDATABASE_URL=postgresql://USER:PASSWORD@db-host:5432/engineering_system # pgvector-enabledPOSTGRES_USER=engineeringPOSTGRES_PASSWORD=<from secrets manager>POSTGRES_DB=engineering_system
# TemporalTEMPORAL_ADDRESS=temporal:7233 # or your managed Temporal Cloud endpointTEMPORAL_NAMESPACE=default
# LLM — no env vars. Provider keys and model selection are configured in the# admin UI (/models) after first boot and stored encrypted in the# DB; nothing in the codebase reads ANTHROPIC_API_KEY / OPENAI_API_KEY.
# Source control — set here for bootstrap only; managed via /admin/integrations thereafter# If these are set they act as fallback when the DB row hasn't been configured yet.# Remove them once you've saved the values in the admin UI.GITHUB_TOKEN=ghp_...GITHUB_WEBHOOK_SECRET=<random>
# Browser auth (better-auth) — MUST set both in prod or the gateway refuses to startBETTER_AUTH_URL=https://api.example.comBETTER_AUTH_SECRET=<openssl rand -base64 32>
# JWT signing — used by the session-token bridge (browser bearer) and OAuth state tokensJWT_SECRET=<openssl rand -base64 48># Or, for RS256:# JWT_PRIVATE_KEY_PATH=/etc/auto-swe/jwt-private.pem# JWT_PUBLIC_KEY_PATH=/etc/auto-swe/jwt-public.pem
# Web app — read at request time by the root Server Component and injected# into window.__APP_CONFIG__, so a change is a container restart, not a rebuildNEXT_PUBLIC_API_URL=https://api.example.comAPI_INTERNAL_URL=http://gateway:8080 # server-only: gateway as reached from the web and worker containers (the in-app assistant's tools); web defaults to NEXT_PUBLIC_API_URLNEXT_PUBLIC_TEMPORAL_UI_URL=https://temporal.example.com # optional; omit to hide the linkNEXT_PUBLIC_SUPPORT_URL=https://help.example.com # optional; https only; omit to hide itNEXT_PUBLIC_SUPPORT_EMAIL=help@example.com # optional; shown as a mailto linkNEXT_PUBLIC_SUPPORT_SLACK=#platform-help # optional; up to 80 characters; an https URL becomes a link
# Browser-facing gatewayCORS_ORIGIN=https://app.example.com,https://admin.example.com # first entry = better-auth client originPUBLIC_URL=https://api.example.com # used for Slack OAuth callbackTRUST_PROXY=10.0.0.0/8 # behind a reverse proxy: the proxy IPs/CIDRs whose X-Forwarded-For # is trusted (or `true`). Unset = do not trust itRATE_LIMIT_MAX=200 # optional: requests per client per window (default 200)RATE_LIMIT_WINDOW_SECONDS=60 # optional: window length in seconds (default 60)WEBHOOK_RATE_LIMIT_MAX=6000 # optional: requests per client address per window on the GitHub webhook routes (default 6000)
# Config encryption — REQUIRED for gateway and worker to start# Encrypts all DB-stored secrets (GitHub token, Slack tokens, S3 credentials, OAuth secrets)CONFIG_ENCRYPTION_KEY=<openssl rand -base64 32> # base64-encoded 32 bytes
# Seed (only needed for first boot — the admin user)SEED_ADMIN_PASSWORD=<openssl rand -base64 24>SEED_ADMIN_EMAIL=admin@example.com # optional, defaults to admin@auto-swe.local
# Magic-link / password-reset email — pick ONE# Option A — SMTPSMTP_HOST=smtp.example.comSMTP_PORT=587SMTP_USER=postmaster@example.comSMTP_PASS=<from secrets manager>AUTH_FROM_EMAIL=auth@example.com# Option B — Resend# RESEND_API_KEY=re_...# AUTH_FROM_EMAIL=auth@example.com
# OAuth sign-in and artifact storage are environment-only — they are read at# process start, so they have no admin-UI form. Slack is configured via the# admin UI (/admin/integrations) after first boot; its env vars below are only# bootstrap fallbacks.## OAuth sign-in (gateway restart required to change)# GITHUB_CLIENT_ID=...# GITHUB_CLIENT_SECRET=...# GOOGLE_CLIENT_ID=...# GOOGLE_CLIENT_SECRET=...# OKTA_ISSUER=https://dev-12345.okta.com/oauth2/default# OKTA_CLIENT_ID=...# OKTA_CLIENT_SECRET=...## Slack (env fallback — prefer /admin/integrations → Slack tab)# SLACK_CLIENT_ID=...# SLACK_CLIENT_SECRET=...# SLACK_SIGNING_SECRET=...# SLACK_BOT_TOKEN=xoxb-...## Artifact store# ARTIFACT_S3_ACCESS_KEY=... # compose maps this to AWS_ACCESS_KEY_ID# ARTIFACT_S3_SECRET_KEY=... # compose maps this to AWS_SECRET_ACCESS_KEY# ARTIFACT_S3_BUCKET=auto-swe-artifacts# ARTIFACT_S3_REGION=us-east-1# ARTIFACT_S3_PREFIX=workflow-artifacts# ARTIFACT_S3_ENDPOINT=https://... # only if not AWS S3 — Garage, R2, B2, …# ARTIFACT_S3_FORCE_PATH_STYLE=true # true for Garage and B2; false for AWS S3 and R2
# Observability — OTLP over gRPC (the exporters are the *-otlp-grpc packages),# so point this at the collector's gRPC port, not the HTTP oneOTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
# Worker (optional tuning)WEB_URL=https://app.example.com # base URL for the "Open approvals" link (to /approvals) in Slack HITL # notifications (defaults to http://localhost:3000 — # set it or Slack links point at localhost)# WORKER_MAX_CONCURRENT_ACTIVITIES=10 # cap on concurrent activity executions per worker # (each typically holds a Docker workspace)# Agent workspace sizing and sidecar images (all optional; defaults shown; the# worker refuses to start on a value it cannot use)# WORKSPACE_MEMORY=4g# WORKSPACE_CPUS=2# WORKSPACE_PIDS_LIMIT=512# WORKSPACE_IMAGE=node:24-alpine# WORKSPACE_IMAGE_PYTHON=registry.example/executor-python:1 # per-stack executor images (optional)# WORKSPACE_IMAGE_JVM=registry.example/executor-jvm:1# WORKSPACE_IMAGE_GO=registry.example/executor-go:1# WORKSPACE_METADATA_BLOCK_IMAGE=alpine:3.20# WORKSPACE_BLOCK_METADATA=true # only the literal `false` disables the metadata blackhole# SCANNER_REGEX_BUDGET_MS=250 # per-pattern scanner wall-clock budget# The Claude Code harness's model proxy (the compose files turn it on): the worker holds the model# key and meters every call; a workspace gets a per-turn token. Unset the port to turn it off on the Docker backend;# the Kubernetes backend refuses to start without it.# npm registry proxy — workspaces install packages through the worker, which# alone holds each registry's credential. Empty REGISTRY_PROXY_PORT disables it.# REGISTRY_PROXY_PORT=8791# REGISTRY_PROXY_URL=http://{hostname}:8791 # how a workspace reaches this worker; {hostname}# # becomes the replica's hostname# REGISTRY_PROXY_BIND=0.0.0.0# REGISTRY_PROXY_CACHE_DIR= # tarball cache dir; empty disables the cache# REGISTRY_PROXY_CACHE_MAX_BYTES=2147483648 # cache cap (default 2 GiB)
# HARNESS_MODEL_PROXY_PORT=8790# HARNESS_MODEL_PROXY_URL=http://{hostname}:8790 # how a workspace reaches this worker; {hostname}# # is the worker's own (default http://host.docker.internal:<port>)# HARNESS_MODEL_PROXY_BIND=0.0.0.0# WORKSPACE_NETWORK=auto-swe-workspaces # a Docker network workspaces join instead of the default bridge# WORKSPACE_DNS=1.1.1.1,8.8.8.8 # resolvers every workspace uses; set an internal one when the repository# # host (a split-horizon GHE name) resolves to an unreachable address on public DNS.# # Agent-run code can then resolve every name that resolver knows# WORKSPACE_BACKEND=docker # `kubernetes` runs agent workspaces as pods (see Kubernetes above); anything else is Docker# K8S_WORKSPACE_NAMESPACE=auto-swe-workspaces # the one namespace workspace pods live in# K8S_WORKSPACE_RUN_AS_UID=1000 # uid/gid of the workspace process; must be non-root and match the image# K8S_WORKSPACE_RUNTIME_CLASS= # optional RuntimeClass (gVisor/Kata); empty = the cluster default# K8S_WORKSPACE_STORAGE_LIMIT=10Gi # ephemeral-storage request/limit and the size of the /workspace volume# K8S_WORKSPACE_MAX_LIFETIME_SEC=86400 # hard pod lifetime the kubelet enforces (activeDeadlineSeconds)# K8S_WORKSPACE_IMAGE_PULL_SECRETS= # comma-separated Secret names in the workspace namespace# K8S_REQUIRE_NETWORK_POLICY=true # only the literal `false` skips the start-up NetworkPolicy check# K8S_KUBECTL=kubectl # the kubectl binary the worker runsMagic-link transport gotcha.
nodemailer.sendMail()resolves on SMTP2xx(relay accepted) — not delivery. Always test end-to-end against a real inbox after configuring SMTP/Resend, and check the audit log foracceptedvsrejectedarrays.
2b. Admin UI configuration (post-boot)
Section titled “2b. Admin UI configuration (post-boot)”Integration credentials that an operator rotates during normal use are stored encrypted in the DB and managed via the admin UI; their env vars remain accepted as fallbacks. What a process needs at start-up — sign-in provider credentials, artifact storage, and workspace sizing — is environment-only (see configuration.md).
| Admin page | What it configures | Restart required? |
|---|---|---|
/models |
LLM provider credentials and per-role model selection | No — resolved fresh per activity call |
/admin/integrations → GitHub |
GitHub PAT, webhook secret, GitHub Enterprise URLs, GitHub App credentials | No |
/admin/integrations → Slack |
Slack client ID/secret, signing secret, bot token | No — read on every request |
/admin/workflow-defaults |
Branch prefix, PR title/body templates, default team slug | No — resolved fresh per workflow activity |
Bootstrap order (first deployment):
- Start gateway + web only —
docker compose -f docker-compose.infra.yml -f docker-compose.prod.yml up -d gateway webin the compose stack (locally,yarn dev:gatewayandyarn dev:web). - Sign in as admin.
/models→ Credentials → add a provider credential (the seed already created the agents + embedding config)./admin/integrations→ GitHub tab → enter your PAT and webhook secret → Save./admin/integrations→ any other tabs you need (Slack, Issue Tracker, Knowledge Base, Figma).- Start the worker (
… up -d worker, oryarn dev:workerlocally). The worker reads all config from the DB. - Optionally clear the
GITHUB_TOKENandGITHUB_WEBHOOK_SECRETenv vars — the DB config is now the source of truth.
Restarts. Everything set through environment variables — sign-in credentials, storage, workspace sizing, the scheduled-sweep settings — is read at process start, so changing one means restarting the service that reads it. Every admin-UI change (GitHub token and webhook secret, Slack credentials, workflow defaults) takes effect on the next request or activity call.
3. Database setup
Section titled “3. Database setup”The shipped schema lives in packages/shared/src/prisma/migrations/ — a consolidated baseline, the
hand-written DDL migration beside it, and one migration per schema change made since the schema was
first deployed, each named for the change it makes:
| Migration | What it adds |
|---|---|
20260618000000_init |
The consolidated baseline, generated from schema.prisma via prisma migrate diff --from-empty --to-schema --script — the vector extension, every enum, table, column, FK and Prisma-expressible index (including the repo_dependencies edge table and connections.package_names) |
20260618000001_custom_constraints_and_indexes |
Everything the DSL can’t express: the HNSW vector indexes on memory_items.embedding and knowledge_items.embedding, the partial and expression indexes the scope cascade and the automation ledger rely on, the singleton / scope-discriminator / per-source-filter CHECK constraints, api_tokens’ exactly-one-owner CHECK, array-column NOT NULLs, the governed-knowledge scope-guard triggers, and the embedding-config seed |
The split between the two is by kind of DDL, not by when it was written. The schema is deployed,
so the pair is applied in the wild and is never edited or regenerated: a schema change appends its
own migration after them, and prisma migrate deploy applies whatever is pending. See the
prisma-pgvector-hnsw skill.
Directory names follow Prisma’s <timestamp>_<name> convention: 20260618000000_init for the
consolidated baseline and 20260618000001_custom_constraints_and_indexes for the hand-written file
beside it. The stamp is a sentinel — the day before the repository’s earliest content migration —
because that ordering is the contract: migrate deploy applies migrations lexicographically by
directory name, so the baseline has to sort before every migration whose DDL it contains. It is
deliberately not the date the consolidation happened.
The oauth_* and jwks tables must exist before MCP clients can use the gateway, so deploy the migration
first. The gateway does boot without them: it seeds the MCP resource into oauth_resources while
initialising authentication, and tolerates a missing table by deferring that seed to the first use of the
resource. Until the migration is applied the OAuth endpoints fail, and sign-in is unaffected. The OAuth provider distinguishes a
NULL array column from an empty one, so only the array columns it requires are NOT NULL
(oauth_clients.redirect_uris and scopes on the access-token, refresh-token and consent tables); every
other array there stays nullable on purpose.
# 1. Create the database with the pgvector extensionpsql -h db-host -U postgres -c 'CREATE DATABASE engineering_system;'psql -h db-host -U postgres -d engineering_system -c 'CREATE EXTENSION IF NOT EXISTS vector;'
# 2. Apply migrations (run from a host that has the repo + DATABASE_URL set)yarn workspace @auto-swe/shared exec prisma migrate deploy
# 3. Seed the admin user, default team, default workflow templateyarn db:seed# ↳ builds @auto-swe/shared first (the gateway half resolves it through its compiled# dist/), then runs provisionAuthAdmin to set up the better-auth credential account# so the seeded admin can sign in via the password tab on /login.# Needs SEED_ADMIN_PASSWORD (both halves hash it; neither generates one) and# BETTER_AUTH_SECRET (provisionAuthAdmin runs without NODE_ENV, which the# gateway treats as production) in the environment or the root .env.Inside Docker. The gateway image’s entrypoint runs node node_modules/prisma/build/index.js migrate deploy before starting Fastify, so the runtime image needs the full prisma CLI and its transitive dependencies — prisma is declared in the package’s dependencies (not devDependencies) so yarn workspaces focus --production keeps them. The CLI is invoked by path, so no .bin symlink is involved. The simplest pattern for a separate migration step is a one-shot init container. Details in the prisma-docker-migrations skill.
For routine application thereafter, prisma migrate deploy is idempotent. Never use prisma migrate dev in production — it will silently try to drop the HNSW index every time unrelated schema changes are made. See the prisma-pgvector-hnsw skill.
4. Temporal setup
Section titled “4. Temporal setup”You have two reasonable paths:
a. Self-hosted Temporal (Docker Compose / Kubernetes)
Section titled “a. Self-hosted Temporal (Docker Compose / Kubernetes)”The shipped docker-compose.infra.yml provisions:
postgres-temporal(a separate Postgres from the app DB — needed because Temporal owns the schema)temporal-setup(one-shot container that installs the Temporal schema)temporal(the server)temporal-setup-namespace(one-shot container that creates the namespace) and, for local development only,temporal-ui(port8233, behind thetemporal-uiCompose profile; the localyarn docker:*/justentry points enable it, the production recipes do not)
For real production, run these on dedicated infrastructure (not on the gateway/worker hosts). At minimum:
- Multiple Temporal server replicas behind an internal load balancer.
- Daily backups of
postgres-temporal. - Mutual TLS for the worker → Temporal gRPC connection.
Starting without Temporal
Section titled “Starting without Temporal”The gateway does not itself need Temporal to boot. It connects in the background, retrying with backoff (1 s, doubling to 30 s), so an unreachable Temporal at startup delays nothing: sign-in, reads and settings that do not touch workflows work straight away. The system schedules (consolidation, the sweeps, eval) are synced once the connection is up. Per-channel and per-work-request schedules are not re-synced at startup: they are reconciled when their own rows are written, which is refused while Temporal is not connected (below). The shipped Compose files still gate the gateway on Temporal’s health, so this path applies to deployments that start the gateway independently, such as managed Temporal or an orchestrator without that dependency.
GET /healthstays200whatever Temporal is doing, so container liveness checks (the gateway image’sHEALTHCHECK) never restart a gateway for a Temporal outage. The body reports{"status":"ok","temporal":"connected"}or"temporal":"connecting".connectedmeans only that the first connection has been made: the gateway does not probe, the SDK reconnects on its own, and a later outage still readsconnected.connectingmeans no workflow operation can run yet.- Until it connects, these answer
503with{"error":{"code":"TEMPORAL_UNAVAILABLE", ...}}andRetry-After: 5: launching, retrying and cancelling runs, signals, human-step responses, eval runs, schedule changes, and the other routes that call Temporal. Saving the consolidation, eval and re-validation schedule settings, and any Slack channel change to whether or when a digest or poll runs (activating, deactivating, enabling, changing a cron, registering with one enabled, deleting), are refused before anything is saved, so the database and the schedules never disagree; other channel edits go through, including a dashboard save that leaves the schedule fields as they were. The synchronousPOST /generate, the refine route and the explain route keep their own503codes (GENERATION_UNAVAILABLE,REFINE_UNAVAILABLE,EXPLAIN_UNAVAILABLE); the asynchronousPOST /generate/jobsandGET /generate/jobs/:idanswerTEMPORAL_UNAVAILABLE. Editing a memory item still succeeds; its re-embed is best-effort and is skipped when Temporal is not connected. - A launch leaves no rows behind, and a human step answered in that window is put back to pending.
A webhook-triggered launch in the window gets
503; GitHub does not redeliver on its own, so redeliver it from the webhook’s delivery log once/healthreportsconnected. - Lookups that decide whether a run is finished treat “not connected” as unconfirmed, never as
finished: agent-run admission keeps its concurrency slots and a re-run is refused rather than started
on top of a possibly live one.
GET /scheduled-work-requestsreturnsexists: null, unavailable: truefor each schedule rather than reporting it missing. - After connecting, the SDK’s gRPC channel reconnects by itself if the server goes away; calls during
such an outage fail with the SDK’s own error rather than
TEMPORAL_UNAVAILABLE.
b. Temporal Cloud
Section titled “b. Temporal Cloud”Point TEMPORAL_ADDRESS at your Temporal Cloud endpoint and supply the namespace + mTLS certs via the SDK. This removes the Postgres/server burden entirely. The codebase does not wire up the mTLS path — that is a small change in packages/gateway/src/plugins/temporal.ts and packages/worker/src/index.ts when you need it.
5. Building the application images
Section titled “5. Building the application images”Each service has a multi-stage Dockerfile (packages/{gateway,worker,web}/Dockerfile) using the Yarn 4 workspaces focus --production pattern. From the repo root:
yarn docker:app:build # builds gateway + worker + web# or:docker compose -f docker-compose.infra.yml -f docker-compose.app.yml build gateway worker webFor a real registry push, the typical CI flow is:
SHA=$(git rev-parse --short HEAD)docker build -f packages/gateway/Dockerfile -t registry.example.com/auto-swe/gateway:$SHA .docker build -f packages/worker/Dockerfile -t registry.example.com/auto-swe/worker:$SHA .docker build -f packages/web/Dockerfile -t registry.example.com/auto-swe/web:$SHA .docker push registry.example.com/auto-swe/gateway:$SHAdocker push registry.example.com/auto-swe/worker:$SHAdocker push registry.example.com/auto-swe/web:$SHAThe web image takes no URL build args.
NEXT_PUBLIC_API_URL,NEXT_PUBLIC_TEMPORAL_UI_URLand the optional support contact variables (NEXT_PUBLIC_SUPPORT_*, see configuration.md §8) are read at request time by the root Server Component (app/layout.tsx) and injected intowindow.__APP_CONFIG__, so one image serves every environment — set them as runtime env on the container. The exception isNEXT_PUBLIC_APP_VERSION, the login-footer label: it is inlined bynext build, and the shipped Dockerfile does not thread it through as a build arg, so the containerised dashboard shows0.0.0unless you build the web bundle with that variable set.
5b. Deploying with the prod compose stack
Section titled “5b. Deploying with the prod compose stack”If you’d rather run prebuilt images than build your own, the repo ships three compose overlays that deploy the GHCR images published by .github/workflows/docker.yml:
| File | What it adds |
|---|---|
docker-compose.prod.yml |
gateway / worker / web from registry images (pull_policy: always, restart: unless-stopped; gateway and worker take env_file: .env, web does not). Overlays the infra file — it reuses postgres, temporal, and (when the objectstore profile is active) garage from docker-compose.infra.yml. |
docker-compose.traefik.yml |
Fronts gateway + web with Traefik TLS. Strips the published host ports (ports: !reset []), attaches both to an external traefik network in addition to the stack’s default one (which they still need to reach Postgres, Temporal and each other), and adds websecure routers with certResolver=webcert for DOMAIN_API (gateway) and DOMAIN_APP (web). Requires a Traefik instance you run separately, with the traefik Docker network already created. |
docker-compose.watchtower.yml |
Adds a scoped Watchtower container (WATCHTOWER_SCOPE, label-gated, cleanup on) that re-pulls the three app images every WATCHTOWER_POLL_INTERVAL seconds (compose default 300; .env.example suggests 86400). |
# Base prod stack (infra + prebuilt app images)docker compose -f docker-compose.infra.yml -f docker-compose.prod.yml up -d
# With Traefik TLS fronting and Watchtower auto-updatedocker compose \ -f docker-compose.infra.yml \ -f docker-compose.prod.yml \ -f docker-compose.traefik.yml \ -f docker-compose.watchtower.yml \ up -dThings to know:
- Pin image tags via
APP_IMAGE_GATEWAY/APP_IMAGE_WORKER/APP_IMAGE_WEB. The compose defaults are the mutable:maintags. CI also publishes an immutable<timestamp>-<commit>tag per build (plus:latestand git-tag refs) — prefer pinning those in production so a rollback is a one-line.envchange rather than registry archaeology. - Image publishing is CI-gated. The publish job in
docker.ymlruns only after itschecksjob, which callsci.ymlitself — docs drift, source invariants, typecheck, lint, tests, build, and the real-database migrations job — so:mainonly moves on builds that pass everything a pull request must. If you run Watchtower against:main, that gate is your only protection — a passing-but-bad commit still auto-deploys. Pinned immutable tags + manual bumps are the conservative choice. - Required secrets. The gateway and worker images run with
NODE_ENV=productionand refuse the in-source dev fallbacks, so.envmust contain real values forJWT_SECRET,BETTER_AUTH_SECRET(≥32 chars), andCONFIG_ENCRYPTION_KEY.docker-compose.prod.ymlenforces all three with:?interpolation errors atuptime, and both processes additionally validateCONFIG_ENCRYPTION_KEYat boot — a key that is present but not base64-encoded 32 bytes exits with a descriptive error before anything is served. - The web container does not receive
.env. It is handed only what the dashboard reads —NEXT_PUBLIC_API_URL,API_INTERNAL_URL,NEXT_PUBLIC_TEMPORAL_UI_URL,NEXT_PUBLIC_GRAFANA_URL, the optionalNEXT_PUBLIC_SUPPORT_URL,NEXT_PUBLIC_SUPPORT_EMAILandNEXT_PUBLIC_SUPPORT_SLACK,NODE_ENV,PORTandHOSTNAME— so no secret reaches the Next.js server. Compose still interpolates those values from.env.NEXT_PUBLIC_API_URLis required (:?) and must be the gateway’s public URL, since the browser uses it;API_INTERNAL_URLdefaults tohttp://gateway:8080for the server’s own fetches. A new variable the dashboard reads has to be added to thewebservice explicitly. - Managed DB. Set
DATABASE_URL_OVERRIDEto point gateway + worker at a managed Postgres instead of the composepostgresservice. - Worker DinD. Same as everywhere else: the worker uses
DOCKER_HOSTpointing atdocker-socket-proxy, which mounts the host socket. NoDOCKER_GIDis needed.DOCKER_SOCKET_PROXY_IMAGEoverrides the haproxy image (defaulthaproxy:3.4-alpine); the allowlist and timeouts are ininfra/docker-socket-proxy/haproxy.cfg, mounted read-only, so the deployment needs that file next to the compose files. Keep itstimeout client/server/tunnelabove the longest step: a foregrounddocker runwaits on the API without sending a byte until the container exits, and a quieter exec stream is cut at the proxy’s idle timeout. Watchtower (docker-compose.watchtower.yml) keeps its own direct socket mount: it is a separate service, not on the proxy’s network, and its access is not filtered by the proxy. - Worker health. The worker serves no HTTP. Its image
HEALTHCHECKpasses while/tmp/worker-alivehas been touched in the last two minutes; the worker touches it every 10 seconds only while its Temporal worker is running, so a stopped or wedged worker reports unhealthy. The check does not see a worker that polls but fails every activity. - Object store selection.
COMPOSE_PROFILESin.envdecides whether the bundled Garage service runs.objectstore(the.env.exampledefault) starts it; an empty value starts nothing, and the worker’sdepends_onisrequired: falseso it comes up regardless. See §5c.
5c. Choosing an object store
Section titled “5c. Choosing an object store”The artifact store is S3-compatible, and the worker only ever issues PutObject and GetObject against it. Anything speaking those two verbs works — the choice is operational, not one of compatibility.
Bundled Garage (default). A single-node Garage container, configured by infra/garage/garage.toml and started with --single-node --default-bucket. It provisions the cluster layout, the access key and the bucket on first boot from ARTIFACT_S3_ACCESS_KEY / ARTIFACT_S3_SECRET_KEY / ARTIFACT_S3_BUCKET, and re-running is a no-op, so there is nothing to bootstrap separately. It idles around 5 MB of RAM. Its state is ./data/garage, which needs backing up alongside the two Postgres volumes (§9).
Two constraints worth knowing before you set credentials:
- Garage rejects secret keys shorter than 16 characters.
ARTIFACT_S3_SECRET_KEYmust clear that bar or the container exits on first boot withInvalid default access key. GARAGE_RPC_SECRETmust be set. Single-node deployments never put it on the wire, but Garage requires it.infra/garage/garage.tomldoes not carry one;docker-compose.infra.ymlfalls back to a fixed, public value so the local quickstart needs no configuration, and the production overlay cannot require it with:?the way it does the other secrets — Compose interpolates every service, including one whose profile is off, so a requirement there would also stop deployments that use hosted S3.yarn env:setup check --profile prodis the check instead: it fails whileCOMPOSE_PROFILESincludesobjectstoreand the secret is empty or not 64 hex characters. Generate withopenssl rand -hex 32.
Hosted S3 instead. Set COMPOSE_PROFILES= (empty) in .env and point the artifact variables at the provider. No compose file changes:
| Provider | ARTIFACT_S3_ENDPOINT |
ARTIFACT_S3_REGION |
ARTIFACT_S3_FORCE_PATH_STYLE |
|---|---|---|---|
| AWS S3 | (omit) | your region | false |
| Cloudflare R2 | https://<account-id>.r2.cloudflarestorage.com |
auto |
false |
| Backblaze B2 | https://s3.<region>.backblazeb2.com |
your region | true |
ARTIFACT_S3_ACCESS_KEY / ARTIFACT_S3_SECRET_KEY become the provider’s credentials — compose passes them into the container as the standard AWS_* names the SDK reads, but keeps them under a distinct name outside so an ambient AWS_ACCESS_KEY_ID in the deploying shell cannot silently win over .env. The bucket must already exist — only the bundled store self-provisions one. These same settings can be managed at /admin/integrations → Storage instead, which takes precedence over the environment.
Migrating an existing MinIO deployment. Garage’s on-disk format is unrelated to MinIO’s, so pointing it at ./data/minio will not work and existing objects will not appear. ArtifactRef rows in the app DB hold keys, not blobs, so any object left behind becomes a broken artifact link rather than a visible failure. Copy the objects across before cutting over, with both stores running:
# Old MinIO on :9000, new Garage on :9100 (GARAGE_API_PORT=9100 temporarily).rclone copy minio:auto-swe-artifacts garage:auto-swe-artifacts --progressThen swap GARAGE_API_PORT back to 9000 and drop the MinIO service. If you would rather not migrate, the artifacts are recoverable output rather than source of truth — leaving them behind costs you the history on already-completed runs, nothing that blocks new ones.
6. Running the services
Section titled “6. Running the services”The simplest production layout maps the local Docker Compose 1:1 onto the production hosts:
| Host | Containers | Notes |
|---|---|---|
db-1 |
postgres (pgvector) |
Managed Postgres (RDS / Cloud SQL) is fine. Enable the vector extension, version 0.8 or later (scoped memory search uses its iterative index scan). |
temporal-1..N |
postgres-temporal, temporal, temporal-ui |
Or Temporal Cloud — skip the host entirely. |
gateway-1..N |
gateway |
Behind your HTTPS reverse proxy. Stateless; scale horizontally. |
worker-1..N |
worker + Docker daemon |
Isolated. Reaches Docker via the socket proxy. |
web-1..N |
web |
Behind HTTPS reverse proxy. Stateless; scale horizontally. |
obs-1 (optional) |
otel-lgtm or your collector + Grafana |
OTLP/gRPC on :4317. |
Reverse proxy routing:
https://app.example.com → web-*:3000https://api.example.com → gateway-*:8080https://temporal.example.com (optional, internal only) → temporal-1:8233Kubernetes
Section titled “Kubernetes”The same images run on Kubernetes; there is no shipped Helm chart. There are two ways to run the worker:
- Docker backend (default). The worker is the only stateful pod (Docker socket): run it as a single-replica StatefulSet on a dedicated node pool with the socket mounted via
hostPath. Everything else is a stock Deployment. - Kubernetes backend (
WORKSPACE_BACKEND=kubernetes). Agent workspaces are pods in their own namespace, so the worker needs no Docker socket and nodocker-socket-proxy. Shell steps, container steps and sidecars are not available on this backend (they fail withWORKSPACE_BACKEND_UNSUPPORTED), and the worker is a single replica (its proxies keep per-workspace state in memory). Seearchitecture.mdand its Limitations.
To run the Kubernetes backend:
- Apply
infra/kubernetes/workspace-namespace.yamlafter adjusting the marked values: the namespace enforces PodSecurityrestricted, a NetworkPolicy denies all ingress and allows egress only to DNS, the worker’s proxy ports and the public internet (never the cloud metadata endpoint or a private range), and a Role/RoleBinding gives the worker’s ServiceAccountcreate/get/list/watch/deleteon pods,createonpods/exec, andcreate/deleteonsecretsin that namespace only (the per-pod Secret that carries the registry-proxy token; the worker never reads one, so grant noget,listorpatchon secrets, and keep other Secrets out of the namespace). The worker checks these verbs at start and refuses to start without them. - Run the cluster on a CNI that enforces NetworkPolicy (GKE Dataplane V2; EKS VPC CNI with network policy enabled; AKS with Azure NPM, Cilium or Calico). The worker checks this at start: it launches a probe pod and refuses to start if the pod can reach
169.254.169.254or the cluster API address. - Set
HARNESS_MODEL_PROXY_PORTandHARNESS_MODEL_PROXY_URL(the model proxy is required on this backend: the worker refuses to start without the port, or with a URL that is unset, uses the Docker host alias or contains{hostname}, because the provider API key would otherwise be placed inside the workspace pod), andREGISTRY_PROXY_URLif you run the registry proxy, to a Service name pods can resolve (the{hostname}placeholder is the worker pod’s name, which other pods cannot resolve; the default Docker host alias does not exist in a cluster), and the matching*_BINDto0.0.0.0. - Set a node-pool
podPidsLimit: Kubernetes has no per-pod PID cap, soWORKSPACE_PIDS_LIMITis not applied. - Make sure the executor image runs as a non-root user. The curated images use uid 1000; set
K8S_WORKSPACE_RUN_AS_UIDif yours differs.
The environment variables are listed below. Unset ones take the defaults shown.
7. Post-boot smoke test
Section titled “7. Post-boot smoke test”# Gateway healthcheck (any auth provider is fine if configured)curl -sf https://api.example.com/api/v1/auth/providers
# Sign in to the web app as the seeded adminopen https://app.example.com# email: SEED_ADMIN_EMAIL, password: SEED_ADMIN_PASSWORD
# In the dashboard:# 1. Settings → API tokens → "+ New token" → save the PAT to your secret manager# 2. Connections → "+ Add connection" → connect a real git repo (use a test one first)# 3. Dashboard "+ New request" → select the ticket-to-merge template → smallest possible task; verify the run starts# 4. Watch the request in /requests (open it from the list) and the run in the Temporal UI
# Then verify webhooks both directions# - GitHub merge of the test PR fires POST /api/v1/webhooks/git → workflow completes# - If the repo has CI, POST /api/v1/webhooks/ci → ciFixLoop kicks in on failures8. Day-2 operations
Section titled “8. Day-2 operations”| Concern | Where to look |
|---|---|
| Workflow visibility | Temporal UI (:8233), or /requests, /runs, /requests/:id in the dashboard. |
| Cost tracking | /admin/usage (every LLM and embedding call, runs or not), WorkflowRun.costUsdAccrued, /admin/analytics, the llm_cost_usd_total metric. Unknown models log llm.cost_pricing_known=false. |
| Metrics & traces | Grafana (:3001 with the bundled otel-lgtm) → auto-swe — LLM & workflow overview, provisioned from infra/grafana/. With your own collector, import infra/grafana/dashboards/auto-swe-overview.json, and put infra/grafana/provisioning/auto-swe-alerts.yaml in your Grafana’s provisioning/alerting directory for the transcript alert rules; both expect Prometheus and Tempo datasources with UIDs prometheus and tempo. A run’s activities are in the trace of the request that started it; search Tempo by span.temporal.workflow_id. Gateway and worker both export metrics. |
| Logs | stdout/stderr on both services. Both services also export their logs over OTLP to Loki — the gateway’s request logs, the worker’s Temporal and activity lines (audit lines included) — each carrying its trace id. A custom CMD must keep --import ./packages/<service>/dist/instrument.js, or HTTP spans go missing (see architecture.md §8). |
| Per-team A/B experiments | /library/[id] → set experimentVersion + experimentSplit. |
| Rotating LLM models | Change model spec at /models (takes effect on next activity call). A model with no price is recorded at $0 — price it on its model_catalog_entries row (see model configuration). |
| Rotating GitHub PAT | /admin/integrations → GitHub → enter new token → Save. No restart required. |
| Rotating Slack bot token | /admin/integrations → Slack → enter new bot token → Save. No restart required. |
| Rotating S3 credentials | Update ARTIFACT_S3_ACCESS_KEY / ARTIFACT_S3_SECRET_KEY (or AWS_*) in .env → restart the worker. |
| Rotating OAuth app creds | Update GITHUB_CLIENT_SECRET / GOOGLE_CLIENT_SECRET / OKTA_CLIENT_SECRET in .env → restart the gateway. |
| Rotating secrets | BETTER_AUTH_SECRET / JWT_SECRET invalidate all existing sessions/tokens. Communicate before rotating. |
| Sessions admin | /admin/sessions (revoke any session); /admin/api-tokens (revoke PATs across all users). |
| Shell-step audit | workflow_shell_audit rows are kept as long as their template version; deleting a template removes its rows. |
| Lesson retention | memory_items grows over time; no automatic pruning. Manual DELETE is fine — drops the row from the HNSW index. |
| pgvector index rebuild | REINDEX INDEX idx_memory_items_embedding; — only needed after a bulk import or if recall degrades. |
9. Backup + DR
Section titled “9. Backup + DR”Two stateful datastores; both need backups:
postgres(app DB) —pg_dump. Recovery is straightforward. Restoring loses no functionality; the worker just resumes whatever Temporal still has in flight.postgres-temporal(workflow history) —pg_dump. Loss of this DB means in-flight workflows are gone forever (signals, history, timers). For long-running humanMergeSignal waits (up to 7 days), this is real data loss. Treat this DB at least as carefully as the app DB.
Container workspaces are ephemeral — never back them up. The Docker daemon on the worker host can be restored from scratch.
10. Hardening checklist
Section titled “10. Hardening checklist”-
BETTER_AUTH_URLandCORS_ORIGINpoint at your real HTTPS domain (gateway refuses to issue secure cookies otherwise).BETTER_AUTH_URLis also the issuer and audience of the OAuth tokens MCP clients receive (seeoauth-setup.md), so it must be the public URL clients reach and must not change once clients are connected. -
BETTER_AUTH_SECRETis ≥32 chars and matches across all gateway replicas. -
JWT_SECRET(or RSA key pair) is set, and is not the.env.exampleplaceholder. -
SEED_ADMIN_PASSWORDis rotated or the seed admin is deleted after first sign-in. - OAuth consent screens are published (Google) and homepage URLs filled (GitHub) for prod-grade UX.
- Magic-link transport is verified end-to-end against a real inbox (not just SMTP 2xx).
- Worker host is isolated — separate VPC subnet, no shared Docker socket (the worker holds none; only
docker-socket-proxydoes), no inbound traffic. - Infra ports are not world-reachable.
docker-compose.infra.ymlpublishes Postgres (5432), Temporal (7233, plus 8233 for the Web UI only when thetemporal-uiprofile is enabled, which production must not do) and the object store (9000) — anddocker-compose.app.ymlGrafana (3001) and OTLP (4317/4318) — on loopback by default so the runbook can reach them from the host. An override such asPOSTGRES_PORT=55432without the127.0.0.1:prefix publishes on every interface; keep the prefix, or firewall the port. The localdocker-compose.app.ymlalso binds the gateway (8080) and dashboard (3000) toAPP_BIND_ADDRESS, loopback by default;docker-compose.prod.ymlpublishes those two on every interface (or not at all under the Traefik overlay), so put them behind the reverse proxy. -
RATE_LIMIT_MAX/RATE_LIMIT_WINDOW_SECONDSare left at the defaults (200 per minute per client) unless a suite drives one signed-in user hard: an end-to-end (Playwright) run loads many pages as a single user and exhausts 200 per minute. Raise them only for that environment. Both are read once at startup and a value that is not a positive integer stops the gateway from booting. -
WEBHOOK_RATE_LIMIT_MAX(default 6000 per window, the same window asRATE_LIMIT_WINDOW_SECONDS) is sized above the GitHub App’s peak delivery rate. The GitHub-signed webhook routes (/api/v1/webhooks/git,/access,/ci) have this ceiling instead ofRATE_LIMIT_MAX, per client address and per route: every delivery from a GitHub host arrives from one address, an App installed on many repositories can send hundreds of deliveries a minute, and GitHub does not retry a429, so a delivery over the ceiling is lost. The tuned value is read once at startup and an invalid one stops the gateway from booting. - Reverse proxy enforces HTTPS and forwards
X-Forwarded-For/X-Forwarded-Proto, andTRUST_PROXYnames that proxy’s IPs or CIDRs. The gateway rate-limits a verified user by user id and everything else by client IP; withoutTRUST_PROXYevery anonymous request behind the proxy shares the proxy’s IP and one limit. Do not set it when clients can reach the gateway directly — they could then choose their own IP. - With the bundled object store (
COMPOSE_PROFILES=objectstore),GARAGE_RPC_SECRETis set in.env. Compose does not enforce it in production (§5c); left unset, Garage runs on the infra file’s public placeholder. - Postgres connection uses TLS (
?sslmode=require). - S3 artifact store has lifecycle policy for old workflow artifacts (the DB stores references; the worker never deletes the objects itself).
- Temporal namespace retention is set deliberately (default in self-hosted = 30d; tune for your humanMergeSignal wait).
-
OTEL_EXPORTER_OTLP_ENDPOINTis set and the collector is reachable — otherwise traces, metrics and logs silently drop. - GitHub PAT and webhook secret are set via
/admin/integrations(or env var fallback). Secrets are random per-environment. -
CONFIG_ENCRYPTION_KEY(base64 32-byte random) is set and backed up — it encrypts all DB-stored secrets (GitHub token, Slack tokens, S3 key, OAuth secrets). Loss = all stored credentials are unreadable.
Limitations
Section titled “Limitations”- The prebuilt executor images are amd64 only and rebuild only when
mainmoves.docker.ymlbuilds for the platforms inBUILD_PLATFORMS, and has no schedule, so a base-image fix reachesexecutor-*at the next push tomainrather than weekly; pin by digest and rebuild on your own cadence if you need one. - Only
docker-compose.prod.ymlruns the production Temporal dynamic config. It points Temporal atinfra/dynamicconfig/production-sql.yaml, which omits the development-onlysystem.forceSearchAttributesCacheRefreshOnRead(a search-attribute cache refresh on every read). A deployment built ondocker-compose.app.ymlinstead keepsdevelopment-sql.yaml. The two files are kept identical otherwise by hand; nothing checks it. - The Docker socket proxy narrows the API, not container options. See §0: a worker that can create containers can still request privileged mode or host bind mounts.
- The Docker socket proxy runs as root. It must open the host socket, which is root-owned; the container is read-only, drops every capability and runs nothing but haproxy. CI exercises the worker’s Docker calls through the same image and config.
- The proxy’s allowlist is by path prefix, not by operation. It refuses the bulk and exfiltrating operations (prune, push, load, image export and import from a URL), but every other operation in an allowed group passes, including ones the worker does not use, such as
kill,renameand image delete. - The infra images are tag-pinned, not digest-pinned. The app Dockerfiles, the agent executor image, the socket proxy default and
otel-lgtmpintag@sha256:<digest>(the multi-arch index digest, so amd64 and arm64 both resolve), and Dependabot bumps tag and digest together. The images indocker-compose.infra.yml(pgvector, postgres, the Temporal images, Garage) and Watchtower stay tag-only because each tag is a${VAR:-default}override point: a digest appended after the variable would win over the tag, so a deployment’s override would be silently ignored. A registry can still move what those tags resolve to.DOCKER_SOCKET_PROXY_IMAGEreplaces the whole reference, digest included, so an override there is as pinned as the value it supplies. - The GitHub webhook ceiling is per process, per address and per route. Gateway replicas do not share it. A delivery over
WEBHOOK_RATE_LIMIT_MAXfrom one address in a window is answered429and is not retried by GitHub. Anyone who can make the GitHub host send junk deliveries to the endpoint (anyone able to add a webhook on a repository the host serves) counts against the same address bucket as the valid ones, so the ceiling bounds that, it does not prevent it. Unsigned requests are cheap to refuse but not free: each costs one secret lookup and one HMAC check. WhenTRUST_PROXYtrusts a hop that forwards a client-chosenX-Forwarded-For, a caller can pick the address, and so the bucket, it is counted in. The other webhook routes (template tokens, Jira, inbound events) keep their own limits.
11. Where this guide ends
Section titled “11. Where this guide ends”This doc covers infrastructure setup and the first happy-path workflow. For everything else:
| To understand… | Read |
|---|---|
| Conventions and implementation gotchas | AGENTS.md |
| Workflow engine, data model, auth, infrastructure | architecture.md |
| Agent roles, skills, tools, scanners, tracing | agents.md |
| Model + credential configuration | model-configuration.md |
| OAuth providers and magic-link email | oauth-setup.md |
| The full doc map | docs/README.md |
© 2026 Jorge Barnaby
Source-available under FSL-1.1-MIT