Skip to content

Deploy

Coolify owns the stack. A push to main in a service repo builds the image on Spark, pins the new tag in Coolify, and redeploys that service. No operator runs a deploy command by hand.

Where this runs

Production lives on the Spark DGX VDS (ege@100.124.170.43). Coolify sits at http://100.124.170.43:8000, on the tailnet and the museum LAN only — never public, never behind the Cloudflare tunnel. The public surface is fronted by Cloudflare tunnels. Service-to-service traffic still uses the dataland-network bridge. See Services for per-service detail.

The old path is gone

deploy.sh, reset-stack.sh and the Makefile no longer exist. The per-repo .env files and /home/cobanov/DATALAND/.env are legacy: no running container reads them. Coolify holds every value. See Coolify migration for why.

Container names carry a UUID

Coolify appends a UUID to every container name, so docker exec dataland-agent fails. The old names survive as network aliases, so container-to-container URLs such as http://dataland-agent:4141 and dataland-redis:6379 still resolve and need no change.

For anything that needs a container name, use the helper:

scripts/dl --list                 # (1)!
scripts/dl --name agent           # (2)!
scripts/dl agent uv run alembic upgrade head
scripts/dl redis redis-cli PING
  1. Every container on dataland-network, with status.
  2. Prints the resolved name, for example agent-5qwdlo0cynw03kzaat5n1vpc.

Without the helper, the same lookup inline:

docker exec "$(docker ps -q --filter network=dataland-network --filter name=^agent-)" \
  uv run alembic upgrade head

The network=dataland-network filter is what keeps the Meta project's postgres out of the match.

Routine deploy

Push to main. That is the whole procedure.

flowchart TD
    A["push to main"] --> B["GitHub Actions<br/>self-hosted runner ON spark"]
    B --> C["docker build<br/>tags: dataland/&lt;svc&gt;:latest<br/>ghcr.io/dataland-ai/&lt;svc&gt;:sha-&lt;short&gt;"]
    C --> D["push to GHCR<br/>registry of record"]
    D --> E["PATCH /api/v1/services/&lt;uuid&gt;/envs<br/>set &lt;SVC&gt;_IMAGE_TAG = sha-&lt;short&gt;"]
    E --> F["POST /api/v1/deploy?uuid=&lt;uuid&gt;"]
    F --> G["poll /health<br/>30 tries, 10 s apart"]

Four properties of this pipeline are worth knowing:

  • The runner is on the production host. The built image lands in the same Docker daemon Coolify deploys from, so the Coolify resources keep pull_policy: never and only recreate onto the new image.
  • GHCR is the registry of record. Every build pushes sha-<commit>, which is what makes a rollback to any commit possible.
  • The image tag is a Coolify variable. The workflow patches <SVC>_IMAGE_TAG before it redeploys. Without that step a push would build a new image and redeploy the old one, and nothing would fail.
  • One repo can deploy several services. The agent repo redeploys both agent and auth, because both run the same image.

The workflow needs two org-level values, already configured: the secret COOLIFY_TOKEN and the variable COOLIFY_URL (http://localhost:8000, from the runner's own host).

Deploy or restart by hand

Through the browser: open Coolify, pick the resource, then Redeploy or Restart.

Through the API, from the tailnet:

UUID=5qwdlo0cynw03kzaat5n1vpc                       # (1)!
curl -X POST -H "Authorization: Bearer $TOKEN" \
  "http://100.124.170.43:8000/api/v1/deploy?uuid=$UUID"

curl -X POST -H "Authorization: Bearer $TOKEN" \
  "http://100.124.170.43:8000/api/v1/services/$UUID/restart"   # (2)!
  1. Service UUIDs come from GET /api/v1/services. Read the token from the secret inventory. Never pass it as a literal argument in a shared shell.
  2. Restart takes POST. A GET answers 405.

Change a value

Coolify holds every environment variable. Editing a .env file on the host changes nothing.

curl -H "Authorization: Bearer $TOKEN" \
  "http://100.124.170.43:8000/api/v1/services/$UUID/envs"          # (1)!

curl -X PATCH -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"key":"SOME_KEY","value":"some-value"}' \
  "http://100.124.170.43:8000/api/v1/services/$UUID/envs"          # (2)!
  1. Read the current values.
  2. Write one value. A restart follows, or the running container keeps the old value.

Rollback

Coolify → Deployments → Redeploy the previous entry. Or set <SVC>_IMAGE_TAG back to an earlier sha-<commit> and redeploy. Every build sits in GHCR, so any past commit is a candidate.

Smoke test

After a deploy:

bash dataland-infrastructure/scripts/smoke.sh

Read-only suite per service (agent, museum, rag-v2, notification, knowledge), green and red rollup, logs to dataland-infrastructure/.smoke-logs/.

Exit code Meaning
0 all green
1 one or more suites failed
2 one or more skipped (tunnel down, suite missing)

For full coverage including write paths, open the internal-port tunnel and turn writes on:

ssh -L 4143:127.0.0.1:4143 -L 8080:127.0.0.1:8080 -L 4145:127.0.0.1:4145 \
    ege@100.124.170.43   # (1)!

SMOKE_ALLOW_WRITES=1 \
SMOKE_NOTIFICATION_REDIS_CLEANUP=tunneled \
bash dataland-infrastructure/scripts/smoke.sh   # (2)!
  1. Forwards the loopback-only internal ports (4143, 8080, 4145) over SSH, so the write-path suites reach services bound to 127.0.0.1.
  2. SMOKE_ALLOW_WRITES=1 moves the suite from read-only to write paths, which is safe only behind the tunnel. SMOKE_NOTIFICATION_REDIS_CLEANUP=tunneled tells the suite to clean up its notification Redis test keys through the forwarded port.

scripts/smoke-visit-flow.sh exercises the ticket, telemetry and notification path end to end, and tails the notification worker to confirm rules fire.

Logs

Coolify → resource → Logs is the fastest route. From a shell:

docker logs -f "$(scripts/dl --name agent)"
docker logs -f "$(scripts/dl --name museum-api)"

Each service also writes JSONL to ${DATALAND_LOG_DIR:-/home/cobanov/DATALAND/logs}/<service>/. The Docker json-file driver is capped at 10 MiB × 5 rotations, 50 MiB per container, so a chatty service cannot fill the host disk. The bind-mounted JSONL needs its own logrotate config — see reports/runbook.md.

Migrations stay manual

Alembic migrations are not applied automatically. The move to Coolify did not change that.

scripts/dl agent uv run alembic upgrade head

See Running the stack.

Simulator

For local end-to-end testing without an Empatica band, layer compose.sim.yml:

SIM_TICKET_ID=<a-real-ticket> \
SIM_REDIS_PASSWORD=$(openssl rand -hex 24) \
SIM_FORCE_HR_SPIKE=1 SIM_FORCE_EDA_SPIKE=1 \
bash dataland-infrastructure/scripts/start-simulator.sh   # (1)!

docker logs -f "$(scripts/dl --name notification-worker)"  # (2)!
  1. SIM_TICKET_ID must be a real ticket, so the synthetic telemetry maps to a routable visitor. SIM_REDIS_PASSWORD is minted fresh for the isolated sidecar redis, never reused from production. The two spike flags inject heart-rate and EDA spikes, so the notification rules fire.
  2. Tail the worker to watch the rules trigger. The overlay points notification-worker at the sim redis, so these are sim events only.

The overlay does three things:

  1. Starts a sidecar dataland-redis-sim (separate volume, loopback port 4149).
  2. Starts dataland-telemetry-sim, a Python publisher that XADDs synthetic events to museum:telemetry on the sim redis.
  3. Recreates notification-worker and notification-api with REDIS_HOST pointed at the sim redis. The main redis keeps serving every other service.

Stop with bash dataland-infrastructure/scripts/stop-simulator.sh.

Never replay simulation telemetry into live redis

The simulator must stay isolated on its own redis container. Replaying museum-simulation telemetry into the production redis would corrupt the live museum:telemetry stream that the notification consumers drain.

Destructive reset

reset-stack.sh is gone. Coolify owns the lifecycle, and there is no one-command teardown any more. To rebuild a service from empty, delete its volumes in Coolify → resource → Storages, then redeploy.

Read the backup notes first

Dropping a volume is not reversible from inside the stack. Read reports/backup-restore.md before you touch postgres-data, qdrant-data, redis-data, auth-data, or the knowledge SQLite bind. auth-data holds the auth server signing key, so dropping it mints a new kid and invalidates every issued JWT — see Auth.

Inspect

scripts/dl --list                                    # (1)!
docker images 'ghcr.io/dataland-ai/*'                # (2)!
curl -H "Authorization: Bearer $TOKEN" \
  "http://100.124.170.43:8000/api/v1/services"       # (3)!
  1. Every container on dataland-network, with status.
  2. Every image tag on the host. Each sha-<commit> is a rollback candidate.
  3. Every Coolify service with its UUID and status.

Adding a new service

  1. Add the service to the Coolify compose resource, attached to dataland-network, with a network alias that keeps its dataland-<name> DNS name.
  2. Pin mem_limit and cpus to a realistic budget. The defaults are unbounded.
  3. Bind the host port to 127.0.0.1:, and optionally a *_PUBLIC_BIND tailnet IP. Never 0.0.0.0.
  4. Add a healthcheck.
  5. Add a deploy.yml workflow in the service repo, modelled on the agent's: build, push to GHCR, patch <SVC>_IMAGE_TAG, redeploy, poll /health.
  6. Set the service environment variables in Coolify, not in a .env file.
  7. If it is public-facing, add the Cloudflare ingress route (Zero Trust → Tunnels → minotaur tunnel → Public Hostname → Add).
  8. Document it: copy a service page in docs/src/services/, add it to docs/mkdocs.yml nav so the --strict build passes, then push.