Operations

Health endpoints

/health

Liveness. Use for simple process checks.

/ready

Readiness. Use for orchestration because it waits for DB and migrations.

/metrics

Prometheus metrics for queues, jobs, HTTP requests, uptime, and AI usage.

Useful commands

docker compose ps
docker compose logs -f gittensory
curl http://localhost:8787/ready
curl http://localhost:8787/metrics

Important log events

selfhost_listening
selfhost_migrations_applied
selfhost_ai_provider
selfhost_ai_review_plan
selfhost_embed_provider
selfhost_vectorize
selfhost_job_dead
selfhost_cron_error
review_context_fetch_failed
selfhost_webhook_enqueue_failed
selfhost_webhook_enqueue_binding_missing

Observability profile

The observability profile starts Prometheus, Alertmanager, Loki, Promtail, and Grafana with dashboards for infra, review activity, and AI usage.

Postgres installs also expose database internals through the bundled Postgres exporter: connection pressure, lock waits, long transactions, deadlocks, database/table growth, dead tuples, autovacuum activity, and backup freshness. Backup freshness appears when the backup profile is active.

When OpenTelemetry and Sentry are enabled, job audit logs and Sentry events include trace_id/span_id fields so an operator can jump from a failed job or issue to the matching trace in Grafana or Tempo.

docker compose --profile postgres --profile observability up -d
docker compose --profile postgres --profile observability --profile backup up -d

Alerting — required for a 24/7 deployment

Alertmanager ships with a valid but silent default: every alert routes to a name-only receiver that discards it, so docker compose --profile observability up -d always starts clean even before you've configured anywhere to send notifications. This is intentional — the shipped config can't bake in a Slack/Discord/email destination that works for everyone — but it means nothing pages anyone until you edit alertmanager/alertmanager.yml yourself. Treat this as a required step, not an optional one, for any deployment you expect to run unattended.

The fastest verified path: create a Discord channel webhook (channel settings → Integrations → Webhooks → New Webhook), then uncomment the discord receiver block in alertmanager/alertmanager.yml and point the root route at it. Slack, email, and a generic webhook receiver (for PagerDuty or a custom handler) are also ready to uncomment in the same file.

Until you do, alerts are still visible without any extra setup: open Grafana and check the Alerts row on the main dashboard, which lists every currently-firing alert directly from Prometheus, independent of Alertmanager routing. Use this as your fallback check if you haven't wired up push notifications yet — it's exactly what the Dead jobs stay at zero routine check below is watching for.

Dead-lettered jobs also get one automatic revival attempt every 30 minutes (QUEUE_DEAD_LETTER_REVIVE_INTERVAL_MS), as long as the job hasn't already been revived more than a small, bounded number of extra times (QUEUE_DEAD_LETTER_AUTO_RETRY_MAX_EXTRA_ATTEMPTS, default 3) — so a job that died from a bug that's since been fixed and redeployed recovers on its own within the next cycle, without needing direct database access. A job that keeps failing the same way eventually exhausts this budget and stays dead, which is exactly what the alert above is watching for.

Resource profiles

Measured rows below come from a real production instance running the full profile set (qdrant + redis + observability + backup + postgres + ollama) at steady state —docker stats and docker system df snapshots, not a lab benchmark. Estimated rows are reasoned from that same baseline plus each service's declared deploy.resources.limits and image size in docker-compose.yml — they have not been measured directly and could be off, especially for CPU under real load. Treat estimates as a starting point for capacity planning, not a guarantee.

Disk

Measured on the same production instance: 48GB of 151GB used on the host root volume (32%) at steady state. docker system df breakdown:

Images

22.59GB total, 19.24GB (85%) reclaimable via prune.

Volumes

20.57GB total, 5.4GB (26%) reclaimable — this is real application state (databases, vector index, backups), so most of it is never pruned.

Build cache

6.39GB total, 3.55GB (56%) reclaimable.

The reclaimable image and build-cache space here is expected steady state, not a leak — this instance runs scripts/deploy-selfhost-prebuilt.sh, which rebuilds the image from the current git checkout on every deploy and intentionally keeps prior layers around in the build cache for faster rebuilds. The gittensory-docker-safe-prune systemd timer (below) already runs daily against this exact instance and reclaims it on a schedule, so this is not a number to chase down manually.

When a compose default might need to change

Every deploy.resources.limits.memory in docker-compose.yml is operator-overridable via .env (see the *_MEM_LIMIT variables in .env.example). Against the measured full-profile data above, none of the current defaults look miscalibrated enough to change: nothing sits consistently near its limit in a way that risks an OOM kill under normal load (Grafana's ~60% is the closest and still has real headroom), and nothing is so oversized relative to plausible usage that it should be lowered — including Ollama's comparatively large 8GiB ceiling, which is sized for holding one quantized model in memory, not idle overhead. The one real gap is --profile runners, which ships with no limit at all; that is a known, documented tradeoff (see the table above and docker-compose.override.yml.example) rather than an oversight, since the right ceiling depends entirely on the host's core count and how many runner replicas you run.

Docker resource hygiene

Every service in docker-compose.yml caps its own container logs (10MB × 3 rotated files) out of the box, so log growth alone won't fill your disk. Unused Docker images and build cache are a separate, larger disk-growth vector on a host that rebuilds or pulls images repeatedly over months — Docker does not reclaim either automatically.

Install the provided host-level timer to reclaim both on a schedule (anything unused for less than 7 days is left alone, so a recent deploy is never at risk):

sudo cp systemd/gittensory-docker-prune.service.example /etc/systemd/system/gittensory-docker-prune.service
sudo cp systemd/gittensory-docker-prune.timer.example /etc/systemd/system/gittensory-docker-prune.timer
sudo $EDITOR /etc/systemd/system/gittensory-docker-prune.service   # set WorkingDirectory / ExecStart to your path
sudo systemctl daemon-reload
sudo systemctl enable --now gittensory-docker-prune.timer

Run it manually at any time with docker system df before and after to see what it reclaimed: sh scripts/selfhost-docker-prune.sh.

This should always prune containers, images, and build cache — never volumes. Pruning a volume deletes real application state (the database, backups, vector index, or a runner's registration and job data), not disposable build output, so it is never part of routine cleanup unless you intentionally want to delete that state.

Self-hosted runner temp storage

If you run --profile runners, keep every runner job's scratch/temp writes on the mounted runner-work volume, never the container's plain /tmp. A container's own /tmp lives in Docker's overlay/containerd snapshot storage — a CI job that writes high-volume temp data there (language toolchain caches, build artifacts, ad hoc mktemp calls) grows the host's Docker root storage directly, not the volume, so it is invisible to volume-scoped cleanup and can fill the disk out from under the whole stack. The shipped runner service points TMPDIR, TMP, and TEMP at /tmp/runner/tmp (a subdirectory of the mounted runner-work volume) and keeps RUNNER_WORKDIR at /tmp/runner on the same volume. A one-shot runner-tmp-init service creates that directory on the volume (and makes it world-writable, matching real /tmp permissions) before the runner container starts, so this works out of the box on a fresh volume with no manual steps.

Adding a second or third runner service in docker-compose.override.yml for higher CI throughput? Each one needs its own runner-work-style volume, its own init step, and the same temp env — YAML anchors don't cross separate compose files, so repeat the extension block in your override file:

x-runner-tmp-env: &runner-tmp-env
  TMPDIR: /tmp/runner/tmp
  TMP: /tmp/runner/tmp
  TEMP: /tmp/runner/tmp

services:
  runner-2-tmp-init:
    image: alpine:3.20
    profiles: ["runners"]
    volumes:
      - runner-work-2:/tmp/runner
    command: ["sh", "-c", "mkdir -p /tmp/runner/tmp && chmod 1777 /tmp/runner/tmp"]

  runner-2:
    image: myoung34/github-runner:ubuntu-jammy
    profiles: ["runners"]
    depends_on:
      runner-2-tmp-init:
        condition: service_completed_successfully
    environment:
      <<: *runner-tmp-env
      RUNNER_NAME: gittensory-runner-2
      RUNNER_SCOPE: ${RUNNER_SCOPE:-repo}
      REPO_URL: ${RUNNER_REPO_URL:-}
      RUNNER_TOKEN: ${RUNNER_TOKEN:-}
      RUNNER_WORKDIR: /tmp/runner
    volumes:
      - runner-work-2:/tmp/runner

volumes:
  runner-work-2:

Sentry tracing

Leave SENTRY_TRACES_SAMPLE_RATE unset or blank to disable trace export, or set a positive sample rate such as 0.05 to send sampled review spans to Sentry. The custom OpenTelemetry provider installs Sentry hooks for review-stage spans carrying repo, PR, operation, outcome, and hashed installation tags.

Sentry cron monitors

When SENTRY_DSN is set, the self-host runtime emits Sentry monitor check-ins for the recurring loops where silent stoppage matters most. Leaving SENTRY_DSN unset keeps monitor reporting off.

scheduled loop

The two-minute maintenance tick that fans out sweeps, backfills, and refresh jobs.

Orb export

The hourly outcome export loop used by brokered self-host deployments.

Orb relay drain

The pull-mode relay loop for installations that receive events outbound from Orb.

Orb relay register

The recurring retry loop that (re-)registers this instance with the relay broker.

Queue dead-letter revive

The 30-minute (by default) sweep that retries dead-lettered jobs still under the auto-retry ceiling.

A missed monitor means the process may still be alive but the recurring work is not checking in on schedule. Pair the monitor with queue depth, dead-job counts, and the structured error log for the same subsystem.

Routine checks

• Queue pending count is not growing without processing.
• Dead jobs stay at zero or are investigated promptly.
• Webhook deliveries are recent and have 2xx responses, with no enqueue failures.
• AI usage matches expected review volume and model/effort choices.
• REES and RAG failures are visible and bounded.
• Postgres connections, lock waits, slow transactions, dead tuples, and table growth are stable.
• Backups are recent and restore-tested.

Updating and rolling back

Both update paths below only ever restart the gittensory app service (--no-deps) — they never touch other compose-profile services or their state (Postgres, Redis, Qdrant, and Grafana's own grafana-data volume), and they never touch .env keys other than the one they persist for next time. That means .env, the gittensory-config/ mount, every data volume — including the app's own /data volume where Codex/Claude Code auth material lives — and any docker-compose.override.yml are preserved automatically across an update. You don't need to back those up or re-supply them just to run either script, and you only need to recreate a profile service yourself if you're deliberately upgrading that service (its own image tag in docker-compose.yml, or a Postgres/Redis/Qdrant major-version bump) rather than the app.

Path 1: pull a published image

scripts/deploy-selfhost-image.sh pulls a tag or digest, restarts only the gittensory service, waits for it to report healthy via docker inspect's health status (configurable timeout, default 180s), and then persists the resolved image reference back to GITTENSORY_IMAGE in .env so the next plain invocation reuses it.

# Re-pull whatever GITTENSORY_IMAGE already resolves to (safe no-op restart if the tag is unchanged
# and nothing new was pushed under it)
./scripts/deploy-selfhost-image.sh

# Pin an exact release tag or content digest
./scripts/deploy-selfhost-image.sh ghcr.io/jsonbored/gittensory-selfhost:orb-v0.1.0
GITTENSORY_IMAGE=ghcr.io/jsonbored/gittensory-selfhost@sha256:... ./scripts/deploy-selfhost-image.sh

The pull always runs with --policy always, so re-running the script against an unchanged tag is safe: if the registry has nothing new, it just restarts the same image and the health-check wait passes immediately.

Path 2: build from the current git checkout

scripts/deploy-selfhost-prebuilt.sh is for a source-based deploy (this is how GITTENSORY_VERSION ends up as a short git SHA instead of an image tag). It builds the bundle inside a Dockerized Node container — the host itself never needs Node or npm installed — then restarts only the gittensory service the same way as the image path.

git pull
./scripts/deploy-selfhost-prebuilt.sh

SENTRY_RELEASE defaults to gittensory-selfhost@<short git SHA of the current HEAD> unless you override it, so each deploy from a new commit gets a distinct release id automatically. When SENTRY_AUTH_TOKEN, SENTRY_ORG, and SENTRY_PROJECT are all configured, the script also injects and uploads Sentry source maps for that release before restarting the service (set SELFHOST_SKIP_SENTRY_UPLOAD=1 to skip this even when those three are present).

Rollback: no dedicated command today

There is no rollback script. Rolling back means re-running one of the two scripts above pointed at an older target:

• Image-based: re-run deploy-selfhost-image.sh with the prior tag or digest (docker inspect on the running container, or your own deploy log, has the digest you were on before the update).
• Source-based: git checkout the prior commit, then re-run deploy-selfhost-prebuilt.sh.

Before and after any update

Before updating:

• Source-based deploys: git status is clean (no uncommitted local changes the build would silently pick up or drop).
• A current, verified backup exists if the update includes schema changes — see Backup and scaling.

After updating, work through the same checks as any other health pass — see Health endpoints and Useful commands above: confirm /ready returns 200, docker compose ps shows the service healthy, and tail recent logs for startup errors or an unexpected absence of selfhost_listening / selfhost_migrations_applied.

Neither /health nor /ready reports a version, so confirm the deployed release directly — GITTENSORY_IMAGE or SENTRY_RELEASE in .env records what the deploy script just resolved, and docker inspect confirms what the running container actually has:

grep -E '^(GITTENSORY_IMAGE|GITTENSORY_VERSION|SENTRY_RELEASE)=' .env
docker inspect --format '{{.Config.Image}}' "$(docker compose ps -q gittensory)"

If an operating check fails, go to Self-host troubleshooting.