Skip to content

Running the stack

The whole stack is one compose source of truth, dataland-infrastructure/compose.yml. The service repos ship no compose files of their own. Every command below is a plain docker compose invocation run by Coolify. See Deploy.

The three commands that matter

Start the whole stack

docker compose -f dataland-infrastructure/compose.yml --env-file .env up -d

Brings every service up with the existing images — no rebuild. This is the command after a host reboot or after the stack was stopped. Containers that are already running are left alone; only missing ones are created.

Full deploy (pull + rebuild everything)

# Deploy is a push to main. See docs/src/ops/deploy.md

Runs the full ritual: pulls all six repos, fails fast if the agent's .env still holds placeholder secrets (the pre-deploy boot guard), mints a fresh IMAGE_TAG, rebuilds the whole stack with --build, and retags the built images to :latest. The output ends with the active tag and the rollback command.

Rebuild ONE service (the everyday command)

docker compose -f dataland-infrastructure/compose.yml --env-file .env \
  up -d --build --no-deps rag

Builds and recreates just that service. --no-deps keeps recreating rag from rippling into notification-* and friends. Service names are the compose keys: agent, auth, rag, museum-api, knowledge, notification-worker, notification-api, postgres, redis, qdrant.

Day-to-day helpers

docker compose -f dataland-infrastructure/compose.yml --env-file .env ps          # container status
docker compose -f dataland-infrastructure/compose.yml --env-file .env logs -f agent  # follow one service's logs
docker compose -f dataland-infrastructure/compose.yml --env-file .env config      # validate the rendered compose config

Repo-scoped scripts cover the rest:

bash dataland-infrastructure/scripts/check-env-drift.sh   # cross-repo .env.example drift check
bash dataland-infrastructure/scripts/smoke.sh             # read-only smoke suite against the running stack
bash dataland-infrastructure/scripts/backup-webui.sh      # knowledge catalog + image backup
bash dataland-infrastructure/scripts/install-boot.sh      # install the host boot units (auto-start on reboot)

Simulator paths (dev/test only — the simulator is not part of the production compose):

bash dataland-infrastructure/scripts/start-simulator.sh   # isolated sidecar stack (needs SIM_TICKET_ID)
bash dataland-infrastructure/scripts/stop-simulator.sh

The compose.sim.yml overlay adds museum-simulator (writes to the main redis) plus an isolated redis-sim/telemetry-sim pair on a dedicated sidecar redis. See Deploy → Simulator.

First-time setup (new host)

  1. Clone the repos side by side under one root (e.g. /home/cobanov/DATALAND).
  2. Populate the env files. Each service loads its own dataland-<repo>/.env via the compose env_file: directive, and the root .env is the compose --env-file used only for ${...} interpolation (image tags, ports, *_PUBLIC_BIND). Copy each .env.example to its .env and fill in the secrets.
  3. bash dataland-infrastructure/scripts/check-env-drift.sh until it reports OK.
  4. Push to main. The workflow builds on Spark, redeploys through Coolify, then polls /health.

See Deploy for the full pipeline, and Coolify migration for why the old path is gone.

Full teardown is a deliberate act

reset-stack.sh is gone. Deleting a volume in Coolify drops the data: postgres, qdrant, redis, and the auth signing key. There is no one-command teardown any more, so each volume is a separate, deliberate step:

# Coolify -> resource -> Storages -> delete volume, then redeploy (destructive)

Old habits

Running docker compose up inside a service repo no longer works — the per-repo compose files were removed because they drifted from the canonical stack and silently created a fake dataland-network when started first. The loud "no configuration file" error is the intended behavior; use

docker compose -f dataland-infrastructure/compose.yml --env-file .env \
  up -d --build --no-deps <service>

instead. For code-level dev without a container, run uv run inside the service repo with its repo-local .env.