Architecture¶

Dataland is the live AI-art museum by Refik Anadol Studio. The visitor-facing experience — a personalized AI guide that knows where you are standing, what your body is doing, and what artwork is in front of you — is backed by a small set of Python microservices that run as a Docker Compose stack on a single host (the Spark DGX VDS, ege@100.124.170.43, repos under /home/cobanov/DATALAND/).

The stack is organised around two concentric rings:

Data plane — the stateful stores (Postgres, Redis, Qdrant) plus the external RDC Redis owned by Refik Anadol Studio.
App services — the business logic (auth, agent, rag, museum-api, notification-worker, notification-api, knowledge). The auth service runs on this host (container dataland-auth, port 9000) from the same agent image (see Auth).

App-level observability is out-of-process: each service ships traces/spans to Logfire. The self-hosted Prometheus/Grafana/Alertmanager metrics stack was turned off (2026-07) and removed from compose.yml — it is retained in git history if it is ever reinstated (see Observability). GCP is only an API/storage consumer — the stack calls the Gemini API and reads/writes GCS buckets; no application services run in the cloud.

One Docker bridge network, dataland-network, connects everything. Cross-service traffic uses container DNS names (http://dataland-rag:4143), never host-published ports. Public ingress is a single Cloudflare Tunnel; operator access is over a Tailscale tailnet.

Service map¶

graph TB
  subgraph ext["External — Refik Anadol Studio"]
    RDC[("RDC Redis<br/>wearable + sensor truth<br/>msgpack + plain UTF-8")]
  end

  subgraph dnet["dataland-network (Docker bridge)"]
    subgraph data["Data plane"]
      PG[("postgres<br/>agent DB + auth users (dataland)")]
      RD[("dataland-redis<br/>museum:telemetry · ticket state · dedup")]
      QD[("qdrant<br/>knowledge · images · scenes")]
    end
    subgraph app["App services"]
      AU["auth :9000<br/>JWKS issuer + mirror"]
      AG["agent :4141<br/>chat SSE + tools"]
      RG["rag :4143<br/>hybrid retrieval"]
      MU["museum-api :5001→4144<br/>RDC bridge + dashboard"]
      NW["notification-worker<br/>rules → push/ops"]
      NA["notification-api :8080<br/>DLQ · state · ops"]
      WU["knowledge :4152<br/>Catalog Studio CMS"]
    end
  end

  GCS[("GCS buckets<br/>dataland-public · dataland-private · cobanov-public")]
  GEM["Gemini API"]

  RDC -->|PSUBSCRIBE wearables/visitors| MU
  MU -->|XADD| RD
  RD -->|XREADGROUP| NW
  NW -->|POST /v1/service/chat/museum| AG
  NW -->|OneSignal push| OS["OneSignal"]
  NW -->|ops alert| OPS["Discord / Slack"]

  AG --> RG
  AG --> MU
  AG -.->|JWKS RS256| AU
  AG --> PG
  AG --> RD
  AG -->|/v1/ops/complaint · /v1/ops/welcome| NA
  AG --> GEM
  AU --> PG

  WU --> RG
  WU --> GCS
  RG --> QD
  RG --> GCS
  RG --> GEM

  CF["Cloudflare Tunnel (host systemd)"] -.->|public ingress| AG
  CF -.-> MU
  CF -.-> WU
  TS["Tailscale tailnet"] -.->|*_PUBLIC_BIND| data
  TS -.-> NA

Repo	Container(s)	Role
`dataland-agent`	`agent` (`:4141`), `auth` (`:9000`)	FastAPI + pydantic-ai chat agent (museum + general, SSE) and a static test chat client. The same image also runs `auth_server.py` as the `auth` service (see Auth).
`dataland-rag`	`rag` (`:4143`)	Retrieval: ingest + hybrid search over Qdrant. Gemini captioning + embeddings. GCS-backed assets.
`dataland-museum`	`museum-api` (`:5001`→`:4144`)	Bridge to the external RDC Redis; publishes `museum:telemetry`, mirrors active tickets, serves the chapter catalog + ops dashboard.
`dataland-notification`	`notification-worker`, `notification-api` (`:8080`)	Telemetry rules engine → OneSignal pushes + Discord/Slack ops alerts. DLQ + replay.
`dataland-knowledge`	`knowledge` (`:4152`)	The "Catalog Studio" CMS for curators. SQLite catalog + GCS uploads + RAG live-sync.
`dataland-infrastructure`	— (compose + scripts)	The `compose.yml` single source of truth, `deploy.sh`, smoke scripts, notification rules, and the source for this docs site (built by Cloudflare Pages).

Three things to remember about the network:

Container DNS only inside the stack. Service-to-service calls use names like dataland-rag, dataland-museum, dataland-postgres — these resolve over dataland-network, not via the host. Host-published ports are for operators and Cloudflare ingress, not for internal callers.
Stateful services and ops surfaces bind to 127.0.0.1 + a tailnet IP only. Postgres, Redis, Qdrant, and notification-api publish to 127.0.0.1:<port> (local tooling / SSH tunnel) and to a *_PUBLIC_BIND (default the tailnet interface 100.124.170.43) so tailnet peers reach them directly — never 0.0.0.0.
Cloudflare is the only public ingress. cloudflared runs as a host systemd service in token mode and routes the public hostnames (dataland.chat, knowledge.dataland.chat, and the museum dashboard) to the right local port. There is no nginx/Traefik in front of the stack. The docs.dataland.chat site is separate — it is built and served by Cloudflare Pages from docs/src/, not through the tunnel.

Core data flows¶

1. Visitor chat over SSE (incl. museum init/welcome)¶

The agent serves two chat modes over server-sent events: museum mode (POST /v1/chat/museum[/multimodal], ticket-bound, location/vitals-aware) and general mode (POST /v1/chat/general[/multimodal]). All chat endpoints sit behind the /v1 prefix and require an RS256 JWT (Authorization: Bearer).

The first museum message is special. There is no separate registration call any more — sending an empty first /museum message is the registration and returns an instant, personalized welcome with no LLM round-trip.

sequenceDiagram
  autonumber
  participant App as Mobile app
  participant Agent as agent
  participant Auth as auth (JWKS)
  participant PG as postgres
  participant NA as notification-api
  participant Museum as museum-api
  participant RAG as rag

  App->>Agent: POST /v1/chat/museum {ticket_id, message:""} (Bearer JWT)
  Agent->>Auth: verify RS256 via JWKS (cached, multi-provider)
  Agent->>PG: register_ticket(user, ticket_id)
  Note over Agent,PG: conversation_id == ticket_id
  alt empty first message (init)
    Agent-->>App: SSE static welcome — no LLM, no RAG
    Agent->>NA: POST /v1/ops/welcome (off-path welcome push)
    Note right of NA: ticket-deduped vs RDC visit_started
  else real message
    Agent->>Agent: schedule_complaint_check() off-path
    Agent->>Museum: GET /api/tickets/{id}/vitals (get_visitor_vitals)
    Agent->>Museum: GET /api/chapters (get_room_info / get_scene_flow)
    Agent->>RAG: POST /search (search_knowledge / search_artwork_images)
    Agent-->>App: SSE token stream
    Agent-->>App: SSE follow-up suggestions (restored on reload)
  end

Key facts about this flow:

Conversation identity. For museum chat, conversation_id == ticket_id; the same ticket_id always resumes the same chat. register_ticket is idempotent and returns whether the ticket was newly created. There is no separate /register or /current endpoint.
Instant welcome. An empty first message returns welcome_message(full_name) — a fixed greeting kept in sync with the notification service's visit_started copy. New ticket → the welcome is persisted; re-init on an existing ticket → it is streamed without duplicating. The welcome push is fired off-path to notification-api /v1/ops/welcome, which ticket-dedups it against the RDC-driven visit_started welcome.
Anonymous-first. Identity is ticket_id ↔ external_id ↔ OneSignal; no registered/email account is required. full_name may be empty and the welcome degrades gracefully ("Welcome to Dataland!").
Agent tools. The museum agent exposes get_visitor_vitals, get_room_info, get_scene_flow, search_knowledge, and search_artwork_images. Tools speak real room names, never bare codes: Data Pavilion (GA), Latent Gallery (GB), Infinity Room (GC), The Sanctuary (GD), Discovery Portal (ON), Lobby (LO).
Silent complaint detection. Every real visitor message triggers an off-path server-side LLM judge (schedule_complaint_check). It never changes the reply or emits an SSE event; a detected complaint is POSTed to notification-api /v1/ops/complaint with the visitor's identity, per-session deduped, and audit-logged.
Timeouts. The chat run is capped at agent_run_timeout_seconds = 60s; the suggestion call at 15s. The RAG /search client read timeout is 25s so a single museum-knowledge query completes on the first attempt instead of timing out → retrying → hitting the agent wall-clock cap.

2. RDC telemetry → notification rules → push¶

museum-api reads exclusively from the external RDC Redis (Refik Anadol data center: wearable/sensor source of truth). There is no host/port fallback — the service refuses to start if RDC_REDIS_URL is empty. The RDC feed is mixed encoding: most payloads are msgpack, but some channels (Q-SYS / audio) are plain UTF-8, which the decoders handle.

sequenceDiagram
  autonumber
  participant Band as Empatica wearable
  participant RDC as RDC Redis (external)
  participant Museum as museum-api
  participant Redis as dataland-redis
  participant Worker as notification-worker
  participant Agent as agent
  participant OS as OneSignal
  participant OPS as Discord / Slack

  Band->>RDC: PUBLISH BioSensors / position / status
  RDC-->>Museum: PSUBSCRIBE pmessage (msgpack / UTF-8)
  Museum->>Museum: resolve serial→ticket→room→chapter, normalise
  Museum->>Redis: XADD museum:telemetry (telemetry bridge)
  Museum->>Redis: overwrite museum:active_ticket_ids @1Hz
  Redis-->>Worker: XREADGROUP
  Worker->>Worker: evaluate rules (gating + cooldown)
  alt push channel
    Worker->>OS: push (external_id, fallback user_id)
  end
  alt chat channel
    Worker->>Agent: POST /v1/service/chat/museum
  end
  alt ops alert
    Worker->>OPS: OpsNotifier fan-out
  end

How the bridge resolves context, per BioSensors message:

serial → ticket by walking Visitors:ActiveTicketIDs + per-ticket Visitors:*:EmpaticaDeviceID (cached, 60s TTL, cleared on an AssignDevice event).
serial → room from Wearables:WatchDevices:<serial>:RoomCode.
room → gallery → chapter via ROOM_TO_GALLERY + GALLERY_ART_IDS, reading Galleries:<gallery>:<art_id>:SceneControl:RTChapter/DDSChapter, then joining the chapters.json index for chapter_name / art_name.

The normalised event (ticket_id, heart_rate, skin_conductance, body_temperature, room_code, chapter_*, simulator_source: "rdc-bridge", …) is XADD-ed onto dataland-redis::museum:telemetry. The museum-simulator container (compose profile simulator) can publish synthetic events to the same stream; both flow through the same rule engine, distinguished by simulator_source.

The active-ticket mirror is a separate 1 Hz loop that fully overwrites the dataland-redis SET museum:active_ticket_ids with RDC's Visitors:ActiveTicketIDs. This key is a fixed cross-service contract — the agent (session_state.py) reads the exact same key to decide whether a ticket is live.

The notification engine rules (config in config/notification-rules.toml): visit_started (welcome), heart_rate, heart_rate_drop, skin_conductance, spo2, temperature, artwork_engagement, room_transition, session_flow, experience_tip, visit_ended. Gating logic:

Content gating. Content/condition notifications are gated until the visitor reaches an exhibit gallery (GALLERY_ROOM_CODES = {GA, GB, GC, GD}). Only rules marked pre_gallery_exempt (the welcome + checkout) fire before then.
Cooldown. Telemetry rules share a per-ticket cooldown (telemetry_cooldown_seconds = 240). Rules marked ignore_cooldown (room-transition) bypass it so transitions always fire.
Session-flow + checkout. Gallery-B session-flow pushes and visit_ended / checkout pushes.
Resolver fallback. The OneSignal resolver falls back to user_id when external_id is absent.

Ops alerts go through a swappable OpsNotifier: OPS_NOTIFIER_PROVIDER takes a comma-separated list for multi-provider fan-out (Discord + Slack). The service also keeps a DLQ + replay path; notification-api exposes the DLQ / state / rule surfaces plus /v1/ops/complaint and /v1/ops/welcome.

Telemetry bridge is the only path into dataland-redis

museum-api reads RDC, normalises, and writes museum:telemetry. If MUSEUM_TELEMETRY_BRIDGE_ENABLED=false, the notification worker has nothing to consume off live data. Never replay museum-simulation-playback Avro into the live dataland-redis — use a dedicated container.

3. Curator content → RAG ingestion → Qdrant → agent retrieval¶

Curators use the Catalog Studio (knowledge, knowledge.dataland.chat). It is a non-dev CMS with two workspaces: Projects (the Refik artwork catalog) and Museum (sections / scenes / overview). SQLite is the source of truth; GCS holds the uploaded assets; RAG is kept in step by live-sync.

sequenceDiagram
  autonumber
  participant Curator
  participant WebUI as knowledge
  participant GCS
  participant RAG as rag
  participant QD as qdrant
  participant Agent as agent

  Curator->>WebUI: create/update/delete project or museum entity + upload images
  WebUI->>WebUI: persist SQLite (source of truth)
  WebUI->>GCS: PUT images (dataland-public/artworks, cobanov-public/chapters)
  WebUI->>RAG: DELETE /ingest/by-project-slug/<slug>  (replace-by-slug)
  WebUI->>RAG: POST /ingest/file  (rendered markdown → knowledge)
  WebUI->>RAG: POST /ingest/image (raw bytes → images, Gemini caption)
  RAG->>RAG: chunk + embed (gemini-embedding, 3072-dim) + BM25 sparse
  RAG->>QD: upsert points (deterministic UUIDv5 ids)
  Note over WebUI,QD: fire-and-forget; a RAG/GCS hiccup never blocks the save
  Agent->>RAG: POST /search (hybrid dense + BM25 + rerank)
  RAG->>QD: query knowledge / images / scenes
  RAG-->>Agent: reranked passages + image hits

RAG slug + payload conventions (the contract between webui and rag):

Project flow → markdown to the knowledge collection (webui-<slug>.md), images to the images collection. Replace semantics: DELETE /ingest/by-project-slug/<slug> first, then re-ingest. Deterministic UUIDv5 point ids give clean upserts with no duplicate accumulation.
Museum flow uses namespaced slugs: museum (overview), museum-section-<slug>, museum-scene-<slug>; payload entity_type is museum / section / scene. Same replace-by-slug + UUIDv5 model.

RAG's retrieval side (/search) is a hybrid pipeline: dense vectors (gemini-embedding, embedding_dim = 3072) + a BM25 sparse channel (Qdrant/bm25), blended via RRF, then reranked with the jinaai/jina-reranker-v2-base-multilingual cross-encoder (FastEmbed/ONNX). The optional text-scroll channel is off by default (text_search_enabled = false). Captioning for /ingest/image uses gemini-3.5-flash.

Collections currently hold roughly: knowledge ~4969 points, images ~1485 points, plus scenes.

Data stores¶

Store	Container / location	Holds	Notes
dataland-redis	`redis:7-alpine`, `dataland-redis`	`museum:telemetry` stream, `museum:active_ticket_ids` SET, ticket/session state, dedup keys (`welcome_sent:<ticket>`, complaint dedup), HR history, rate-limit buckets	`--requirepass`, `appendonly yes`, `maxmemory 512mb` / `noeviction` (drain, don't drop). 1 GB / 0.5 core.
RDC Redis	external, Refik Anadol data center	Wearable BioSensors / BlueIoT position, `Visitors:` control plane, `Galleries::SceneControl:*`, `Visitors:ActiveTicketIDs`	Read-only from Dataland. Mixed msgpack + plain-UTF-8 encoding. `museum-api` PSUBSCRIBEs; required at boot (no fallback).
Postgres	`postgres:16-alpine`, `dataland-postgres`	Agent DB (users, tickets, conversations, messages, runs) plus the auth server's accounts (`auth_server_users`), all in the `dataland` database	On-host instance shared by `agent` and `auth`. 1 GB / 1 core, tuned `shared_buffers=256MB`. See Auth.
Qdrant	`qdrant/qdrant`, `dataland-qdrant`	Collections `knowledge`, `images`, `scenes` (3072-dim vectors + BM25 sparse)	No API key — the tailnet/`127.0.0.1` binding is the trust boundary. 4 GB / 2 cores.

Authentication¶

Chat endpoints require an RS256 JWT validated against JWKS. The on-host auth service (auth_server.py, container dataland-auth, port 9000) issues and serves keys at /.well-known/jwks.json; the agent reaches it over the docker network at http://dataland-auth:9000. The agent's app/auth.py caches JWKS via PyJWKClient (jwks_cache_ttl = 3600s), verifies RS256, requires exp, and skips aud (mobile tokens carry a client-scoped audience the agent does not constrain). The user identity comes from user_id (or sub); the local User row is auto-created/updated on first sight.

flowchart LR
  Token["Bearer RS256 JWT"] --> AgentAuth["agent app/auth.py"]
  AgentAuth -->|"1. primary"| Local["auth JWKS (dataland-auth:9000)<br/>kid dataland-rs256-1"]
  AgentAuth -->|"2. fallback (WARN)"| CMS["external CMS JWKS"]
  subgraph mirror["JWKS mirror"]
    CMSkey["CMS signing key (public JWK)"] --> Extra["AUTH_EXTRA_JWKS_JSON"]
    Extra --> Served["auth serves merged JWKS"]
  end

JWKS mirror. The agent validates a token by trying each configured JWKS endpoint in order (JWKS_URL then JWKS_URLS). A token signed by the CMS would otherwise only be validatable by the external CMS JWKS endpoint — a single point of failure for chat auth. So auth mirrors the CMS signing key (kid = dataland-rs256-1) by loading its public JWK (from AUTH_EXTRA_JWKS_JSON or data/extra_jwks.json) and serving a merged JWKS (local signing key first, then extra keys, deduped by kid; the local key never gets shadowed). Re-run the mirror provisioning after a CMS key rotation. The agent emits a WARN whenever a fallback JWKS provider is the sole validator of a token, making the SPoF condition alertable.

Other auth surfaces use a single shared password (no per-user accounts): the museum dashboard (MUSEUM_PASSWORD) and the Catalog Studio (KNOWLEDGE_PASSWORD). Service-to-service calls (/v1/service/*, /v1/ops/*, RAG, notification ops) use bearer tokens: AGENT_SERVICE_TOKEN, NOTIFICATION_OPS_TOKEN, RAG_API_KEY.

Deployment topology¶

The stack is one Docker Compose project (dataland-infrastructure/compose.yml) on the Spark DGX VDS. Public traffic enters through a single Cloudflare Tunnel (host systemd); operators reach internal surfaces over Tailscale.

graph LR
  subgraph internet["Public internet"]
    User["Visitor / curator browser"]
  end
  subgraph cf["Cloudflare"]
    Tunnel["cloudflared tunnel (TLS)"]
  end
  subgraph host["Spark DGX VDS (100.124.170.43)"]
    cfd["cloudflared (host systemd)"]
    subgraph compose["docker compose: dataland-network"]
      AG2["agent :4141 → dataland.chat"]
      WU2["knowledge :4152 → knowledge.dataland.chat"]
      AU2["auth :9000 (internal JWKS)"]
      MU2["museum-api :4144"]
      rest["rag · notification · stores"]
    end
  end
  subgraph tnet["Tailscale tailnet"]
    Operator["operator peer"]
  end

  User --> Tunnel --> cfd
  cfd --> AG2
  cfd --> WU2
  Operator -.->|*_PUBLIC_BIND| rest
  Operator -.-> MU2

Concern	Owner
Service config	`/home/cobanov/DATALAND/.env` (compose `--env-file` for interpolation) + per-repo `dataland-<repo>/.env` (loaded via compose `env_file:`)
Secrets (GCP key)	`/home/cobanov/DATALAND/secrets/gcp-key.json` (mode 600)
Compose stack	`dataland-infrastructure/compose.yml` (single source of truth)
Simulator overlay	`compose.sim.yml` — `museum-simulator` (writes main redis) + isolated `redis-sim`/`telemetry-sim`; started by explicit service names
Deploy	Push to `main`; GitHub Actions on Spark redeploys through Coolify
Smoke runner	`dataland-infrastructure/scripts/smoke.sh`
Notification rules	`dataland-infrastructure/config/notification-rules.toml`
Public ingress	Cloudflare Tunnel (host systemd)
Operator ingress	Tailscale tailnet via `*_PUBLIC_BIND`
Out-of-band console	`agentkvm` GLKVM (GL-RM10) on the tailnet — host console + power, independent of the host OS (see Host state & backups)

Deploy guardrail. deploy.sh (set -euo pipefail) runs a production secret check before rebuilding: if the prod env still holds placeholder/default secrets it aborts with a non-zero exit and prints the flagged keys, rather than triggering the agent's boot guard into a crash-loop deploy.

Models¶

The stack uses gemini-3.5-flash for all generative work — the chat agent (agent_model = google-gla:gemini-3.5-flash) and RAG's Gemini image captioning. RAG vectors use gemini-embedding (3072-dim). The Gemini API is called directly; it is the only GCP surface the stack depends on besides the GCS buckets.

Resource budget¶

The host is a single 20-core machine; compose.yml pins memory + cores per service. The two heaviest tenants:

rag — 12 GB / 12 cores. The jina cross-encoder reranker is multi-threaded ONNX inference; at 2 cores a single 20-candidate rerank took ~25s (CPU pegged), so it is allowed 12 cores while leaving 8 for the rest of the stack.
qdrant — 4 GB / 2 cores. Three collections (knowledge, images, scenes) with 3072-dim vectors.

Everything else lives within 64 MB – 1 GB. See each service page for its exact budget: Agent, RAG, Museum, Notification, Knowledge.