Host state & backups¶

Everything runs on the single Spark host. This page is the operator reference for the authoritative on-disk state (the volumes and host binds that hold data you cannot rebuild from code) and the manual re-provision / backup steps a fresh git clone && deploy.sh does not do for you — the ones that have bitten us.

Current state¶

Every Dataland service runs on spark (ege@100.124.170.43, tailnet IP), with the production checkout under /home/cobanov/DATALAND/. All inter-service traffic stays on the dataland-network Docker bridge; the only things that leave the host are:

Cloudflare tunnels (host systemd cloudflared) — public ingress for dataland.chat, knowledge.dataland.chat, and the museum dashboard. TLS terminates at Cloudflare; the tunnel ingresses to 127.0.0.1 on the host. (The docs.dataland.chat site is separate — built and served by Cloudflare Pages from docs/src/, not through the tunnel.)
Tailscale tailnet — direct operator + peer access to the data-plane and ops ports. Every stateful/ops service is published twice in compose.yml: once on 127.0.0.1 (local tooling + SSH-tunnel) and once on a *_PUBLIC_BIND host IP (the tailnet interface) for direct peer access. Never 0.0.0.0 — several of these have weak (Postgres) or no (Qdrant) auth, so the tailnet is the trust boundary.
Public internet — outbound only: the Gemini API, GCS buckets, and OneSignal / Discord / Slack. GCP is purely an API/storage consumer.

The single load-bearing cross-network flow is museum-api → the external RDC redis (Refik Anadol data center: the wearable/sensor source of truth). Its endpoint and availability are museum-owned; we build to the access pattern they sanction.

Out-of-band access¶

When the host OS hangs or Tailscale on spark is down, SSH and every tailnet bind above are unreachable — so a GL.iNet GLKVM (GL-RM10) is wired to the Spark host for out-of-band recovery. It gives a full KVM console (video + keyboard) and remote power control independent of the host OS, and it runs its own Tailscale, so it stays reachable even when spark itself is not.

Reach it: https://agentkvm.minotaur-banded.ts.net (tailnet 100.78.171.4).
Use it for: BIOS/boot, a hung kernel, a stuck deploy that took SSH down, or a hard power-cycle — anything you would otherwise need physical access for.
It is a separate tailnet node (agentkvm), not part of dataland-network and not behind Cloudflare; the tailnet ACL is its trust boundary.

Service inventory¶

Service	Container	Host port	Stateful?
Agent	`dataland-agent`	4141	Stateless (Postgres-backed)
Auth	`dataland-auth`	9000	Signing keys in the `auth-data` volume; accounts in the on-host `dataland` Postgres
RAG	`dataland-rag`	4143	Stateless (recompute from GCS)
Knowledge	`dataland-knowledge`	4152	SQLite + uploads on a host bind
Postgres	`dataland-postgres`	5432	Stateful — dump/restore
Qdrant	`dataland-qdrant`	4146/4147	Stateful — re-ingest from GCS possible but slow
Museum API	`dataland-museum`	4144	In-memory cache only
Notification worker	`dataland-notification-worker`	—	Consumer-group state in redis
Notification API	`dataland-notification-api`	8080	Consumer-group state in redis
Redis	`dataland-redis`	4145	Stateful — AOF persistence

Runtime UIDs differ per image

This matters for the GCS-key ACL step below. The agent (and auth, which shares the agent image) runs as uid 10001. The rag and knowledge images run as uid 1000. All three mount the same secrets/gcp-key.json read-only — so the key file must be readable by both uids (see GCS key ACL).

Volumes & host binds that MUST persist¶

A rebuild is "done" only when this state lands intact on the host. The named volumes are declared at the bottom of compose.yml; the binds come from .env.

State	Kind	Source → container path	Owner	Authoritative?	Recovery if lost
`dataland_postgres-data`	named volume	`… → /var/lib/postgresql/data`	postgres	Yes — auth users, agent conversations	`pg_dump`/`pg_restore` only
`dataland_qdrant-data`	named volume	`… → /qdrant/storage`	qdrant	No (downstream of GCS)	Snapshot, or re-`/ingest/sync` (hours)
`dataland_redis-data`	named volume	`… → /data`	redis	Partial (stream + AOF)	RDB/AOF restore, or replay RDC bridge
`dataland_auth-data`	named volume	`… → /app/data`	auth	Yes — RS256 signing key (`auth_rsa_private.pem`), `auth_rsa_kid.txt`, `extra_jwks.json`	No rebuild — restore or rotate (key-rotation event)
knowledge data	host bind `../dataland-knowledge/data` → `/app/data`	knowledge	Yes — `catalog.sqlite3`, uploaded images, thumbnails	`cp`/`rsync` + `backup-webui.sh`
museum images	GCS, not on host	`dataland-public/artworks`, `cobanov-public/chapters`	RAG / knowledge / catalog	n/a (lives in GCS)	Nothing to move; re-ingest pointers
app logs	host bind `${DATALAND_LOG_DIR}` → `/app/logs`	all services	No (operational)	New host can start empty

Two binds, not volumes

The two pieces of authoritative state that are host bind mounts (not Docker named volumes) are the easiest to forget in a docker volume-centric migration:

Knowledge catalog + uploads at the compose bind ../dataland-knowledge/data (/home/cobanov/DATALAND/dataland-knowledge/data). Holds catalog.sqlite3, projects/<slug>/images/, museum/, and thumbnails/<slug>/. Use backup-webui.sh (WAL-safe SQLite .backup + tarball of museum projects thumbnails) — it verifies the dump and writes a manifest. A bare docker volume migration will silently leave this behind.
Logs at ${DATALAND_LOG_DIR} (prod default /home/cobanov/DATALAND/logs). Not authoritative, but the path must exist on the new host or every container fails to mount /app/logs. Both defaults only exist on the prod VDS — set them explicitly on any other host.

For the full backup/restore command set and DR scenarios, see the backup & restore runbook in the repo (reports/backup-restore.md). reset-stack.sh drops every named volume in one command and is not reversible — read that runbook first.

Re-provision after a host rebuild¶

These are the manual steps a fresh checkout + deploy.sh cannot do for you. Run them in order.

1. Secrets (`.env` + GCS key)¶

# from the parent checkout dir on the new host, e.g. /home/cobanov/DATALAND
cp dataland-infrastructure/.env.example .env
# each service also gets its own dataland-<repo>/.env (loaded via compose env_file:);
# the root .env is only the compose --env-file used for ${...} interpolation.
mkdir -p secrets && chmod 700 secrets   # (1)!
# fill the real secrets in .env (and each dataland-<repo>/.env), then:
chmod 600 .env   # (2)!

0700 on secrets/ keeps the directory (and the gcp-key.json you drop in next) reachable only by the deploy user. Anything looser exposes the service-account key on a shared host.
Lock the .env files to the deploy user before you populate them, not after.

compose.yml reads the root .env for interpolation and each service loads its own dataland-<repo>/.env. These keys are required (compose.yml uses the :? form and refuses to render without them):

Var	Gate
`REDIS_PASSWORD`	`--requirepass` on redis; every consumer authenticates
`RDC_REDIS_URL`	museum-api streams from the live RDC redis only
`MUSEUM_PASSWORD`, `MUSEUM_SESSION_SECRET`	museum dashboard gate
`KNOWLEDGE_PASSWORD`, `KNOWLEDGE_SESSION_SECRET`	knowledge (Catalog Studio) gate

deploy.sh fails fast on placeholder secrets

Before rebuilding, deploy.sh runs the real agent boot guard (app.runtime.assert_boot_required_env) from the current dataland/agent:latest image against the agent's env. If any production secret is still a placeholder/default, the deploy aborts before anything is rebuilt — this prevents the crash-loop outage where the boot guard would otherwise crash-loop the fresh container and take chat offline. The guard is a no-op outside APP_ENV=production, and is skipped on the very first deploy when no dataland/agent:latest image exists yet. So on a brand-new host the first deploy will not catch placeholders — validate the env by hand (scripts/check-env-drift.sh) before the first deploy.sh.

2. GCS key ACL for the runtime user¶

Put the GCP service-account key at secrets/gcp-key.json, then lock it down:

chmod 600 secrets/gcp-key.json   # (1)!
chown "$(id -u)":"$(id -g)" secrets/gcp-key.json   # (2)!
stat -c '%a %n' secrets/gcp-key.json   # (3)!

The service-account key is a long-lived GCS credential, so it stays owner-read only by default. Note the gotcha below — 0600 is the starting posture, because the runtime container uids must still be able to read it.
Set ownership to the current deploy user explicitly. On a rebuild the host uid mapping is the thing most likely to have shifted, and a wrong owner here is exactly what breaks the in-container read.
Sanity check: expected output is 600 secrets/gcp-key.json.

secrets/gcp-key.json is bind-mounted :ro into agent, auth, rag, and knowledge at /app/gcp-key.json. The catch is that those containers run as non-root users, and the container UID must be able to read the host file through the bind:

agent + auth (same image) → uid 10001
rag, knowledge → uid 1000

uid 10001 must be able to read the key (GCS key ACL gotcha)

A chmod 600 key owned by the host deploy user is only readable by that user's uid inside the container. If the deploy user's host uid is not 10001/1000, the agent (uid 10001) will fail to read /app/gcp-key.json and GCS-backed calls break, while the host operator sees a perfectly fine cat secrets/gcp-key.json. After a rebuild the host uid mapping is the thing most likely to have changed. Two clean fixes:

# Option A — keep 600 but make the file owned-by / group-readable to the runtime uids.
sudo chown 10001:10001 secrets/gcp-key.json   # (1)!
sudo chmod 640 secrets/gcp-key.json   # (2)!

# Option B — group both runtime uids and grant the group read.

10001 is the agent/auth runtime uid. For rag/knowledge you would chown to 1000 instead, or use the group approach in Option B to cover both.
0640 (owner read/write + group read) keeps the key off world-read while letting the new owning uid read it. Prefer this over 0644.

Verify from inside each container after deploy.sh:

scripts/dl agent cat /app/gcp-key.json >/dev/null && echo agent-ok   # (1)!
scripts/dl rag   cat /app/gcp-key.json >/dev/null && echo rag-ok   # (2)!

Reads the key as uid 10001 from inside the container and discards the contents. This is the check that catches the uid-mismatch gotcha that a host-side cat hides.
Same proof for uid 1000 (rag, and by extension knowledge).

3. JWKS mirror (`AUTH_EXTRA_JWKS_JSON`)¶

auth runs from the agent image (uv run python auth_server.py) as the on-host dataland-auth service. Its RS256 signing key (auth_rsa_private.pem + auth_rsa_kid.txt, kid=dataland-rs256-1) lives in the dataland_auth-data volume (or is supplied via AUTH_RSA_PRIVATE_PEM/AUTH_RSA_KID). On top of that it serves a mirror of the external CMS signing key's public JWK alongside its own:

flowchart LR
  CMS["CMS / mobile backend<br/>signs tokens (kid dataland-rs256-1)"]
  subgraph auth["auth (dataland-auth :9000)"]
    local["RS256 signing key<br/>(auth-data volume)"]
    extra["AUTH_EXTRA_JWKS_JSON / extra_jwks.json<br/>(public CMS key, mirror)"]
    served["/.well-known/jwks.json<br/>(local key first, then mirror)"]
  end
  Agent["dataland-agent<br/>verifies JWT against each JWKS URL"]
  CMSremote["CMS-staging JWKS<br/>(fallback / sole validator if mirror absent)"]

  CMS -->|public key| extra
  local --> served
  extra --> served
  served --> Agent
  CMSremote -.->|fallback| Agent

Why it matters: the agent verifies a token by trying each configured JWKS URL in turn. With the CMS key absent from the served JWKS, the agent falls through to the external CMS-staging endpoint as the sole validator for all chat auth — a single point of failure. A JWKS only ever exposes public material (never the signing key), so mirroring it demotes the external endpoint to a pure backup. The agent WARNs when a fallback JWKS provider is the only validator.

Provision the mirror via the AUTH_EXTRA_JWKS_JSON env (inline public JWK) or the on-disk data/extra_jwks.json file. After updating it, recreate the auth container so it re-reads the merged key set at startup, then confirm the served JWKS carries both keys:

docker compose -f dataland-infrastructure/compose.yml --env-file .env up -d --build --no-deps auth
curl -fsS http://localhost:9000/.well-known/jwks.json | jq '.keys | length'   # (1)!

Expect >1: one entry for the local signing key plus the mirrored dataland-rs256-1. A 1 means the mirror did not load and the external CMS endpoint is still the sole validator.

A malformed or missing source is logged (auth.extra_jwks.bad_source) and never raised — a bad mirror must not take auth, and thus all chat, down. The local key is always authoritative: an extra key reusing dataland-rs256-1 is skipped so a stale mirror can never shadow the key auth signs with.

Never restore stale auth signing material

auth_rsa_private.pem is signing material. If it is lost and you cannot prove the old key was never compromised, rotate instead of restoring: mint a fresh key, recreate auth, re-provision the mirror, and notify any client that pinned the prior JWKS. Treat a restore as a key-rotation event.

4. Museum knowledge re-ingest (only if Qdrant is rebuilt)¶

Qdrant is recomputable — vectors derive from GCS documents via RAG. If you carry the qdrant-data volume (or a per-collection snapshot) you skip this. If you start with an empty Qdrant, re-ingest:

curl -fsS -X DELETE http://localhost:4146/collections/knowledge   # (1)!
curl -fsS -X POST  http://localhost:4143/ingest/sync -H "X-API-Key: $API_KEY"   # (2)!

Drops the knowledge collection on the Qdrant REST port (4146). Skip it if you carried the volume.
Triggers a full drop-and-re-embed from GCS via RAG (port 4143); the X-API-Key header is required. Budget hours, not minutes.

The museum content — 20 sections + scenes + the overview — is (re-)ingested into the Qdrant knowledge collection through the knowledge service's RAG live-sync (text → /ingest/file, images → /ingest/image; rag slugs museum-section-<slug> / museum-scene-<slug>, replace-by-slug with UUIDv5 ids). After a knowledge-data restore, re-run the live-sync (or scripts/sync_from_qdrant.py) to reconcile the catalog against Qdrant — drift between the two is the most common after-restore footgun.

5. Tailnet binds + Cloudflare ingress¶

compose.yml publishes the data-plane/ops services on each host's tailnet IP via *_PUBLIC_BIND (default 100.124.170.43). On a new host these must point at that host's tailnet interface IP, never 0.0.0.0 and never 127.0.0.1 (which would collide with the loopback line already in compose). Set, at minimum: QDRANT_PUBLIC_BIND, REDIS_PUBLIC_BIND, RAG_PUBLIC_BIND, NOTIFICATION_PUBLIC_BIND, POSTGRES_PUBLIC_BIND, AUTH_PUBLIC_BIND.

Finally, repoint Cloudflare tunnel ingress for dataland.chat, knowledge.dataland.chat, and the museum dashboard at the new origin (host systemd cloudflared, ingressing to 127.0.0.1). Swap DNS / origin last so rollback stays a one-line origin change.

Cross-network dependencies¶

Each service's external touchpoints — useful for sizing firewall asks.

Service	Reaches out to	Direction
Agent	RAG, museum-api, notification-api, auth JWKS, CMS JWKS, Gemini API, Postgres, internal redis	Egress
Auth	on-host Postgres (`dataland` db, `auth_server_users`)	Egress
Museum API	External RDC redis (museum LAN), internal redis, RAG	Egress
RAG	GCS (`dataland-public`, `dataland-private`, `cobanov-public`), Gemini API, Qdrant	Egress
Knowledge	RAG, GCS (no DB — SQLite is local)	Egress
Notification worker / api	Internal redis, Agent (service token), OneSignal, Discord/Slack	Egress
Internal Redis	—	None outbound

The one critical cross-network flow is museum-api → RDC redis on the museum's network. Every other external touchpoint is public internet (Gemini, GCS, OneSignal) or our own internal infra.

Post-rebuild smoke (acceptance)¶

Before declaring a host rebuild done, confirm — in this order:

deploy.sh completes (its boot guard passed against the agent's .env).
docker compose -f dataland-infrastructure/compose.yml --env-file .env ps shows every container healthy.
GCS key is readable inside agent, rag, knowledge (the docker exec cat checks above).
auth serves >1 JWK (curl -fsS http://localhost:9000/.well-known/jwks.json | jq '.keys | length').
Qdrant knowledge reports ~its expected point count (or your last re-ingest count).
dataland-agent answers a museum-mode chat round-trip end-to-end, and a welcome push fires on the first empty /museum message.
Telemetry flows: museum:telemetry is being XADD-ed and notification rules fire.