Deploy¶

Coolify owns the stack. A push to main in a service repo builds the image on Spark, pins the new tag in Coolify, and redeploys that service. No operator runs a deploy command by hand.

Where this runs

Production lives on the Spark DGX VDS (ege@100.124.170.43). Coolify sits at http://100.124.170.43:8000, on the tailnet and the museum LAN only — never public, never behind the Cloudflare tunnel. The public surface is fronted by Cloudflare tunnels. Service-to-service traffic still uses the dataland-network bridge. See Services for per-service detail.

The old path is gone

deploy.sh, reset-stack.sh and the Makefile no longer exist. The per-repo .env files and /home/cobanov/DATALAND/.env are legacy: no running container reads them. Coolify holds every value. See Coolify migration for why.

Container names carry a UUID¶

Coolify appends a UUID to every container name, so docker exec dataland-agent fails. The old names survive as network aliases, so container-to-container URLs such as http://dataland-agent:4141 and dataland-redis:6379 still resolve and need no change.

For anything that needs a container name, use the helper:

scripts/dl --list                 # (1)!
scripts/dl --name agent           # (2)!
scripts/dl agent uv run alembic upgrade head
scripts/dl redis redis-cli PING

Every container on dataland-network, with status.
Prints the resolved name, for example agent-5qwdlo0cynw03kzaat5n1vpc.

Without the helper, the same lookup inline:

docker exec "$(docker ps -q --filter network=dataland-network --filter name=^agent-)" \
  uv run alembic upgrade head

The network=dataland-network filter is what keeps the Meta project's postgres out of the match.

Routine deploy¶

Push to main. That is the whole procedure, except for the agent: it goes to staging first and reaches production only by promote (see below).

flowchart TD
    A["push to main"] --> B["GitHub Actions<br/>self-hosted runner ON spark"]
    B --> C["docker build<br/>tags: dataland/&lt;svc&gt;:latest<br/>ghcr.io/dataland-ai/&lt;svc&gt;:sha-&lt;short&gt;"]
    C --> D["push to GHCR<br/>registry of record"]
    D --> E["PATCH /api/v1/services/&lt;uuid&gt;/envs<br/>set &lt;SVC&gt;_IMAGE_TAG = sha-&lt;short&gt;"]
    E --> F["POST /api/v1/deploy?uuid=&lt;uuid&gt;"]
    F --> G["poll /health<br/>30 tries, 10 s apart"]

Four properties of this pipeline are worth knowing:

The runner is on the production host. The built image lands in the same Docker daemon Coolify deploys from, so the Coolify resources keep pull_policy: never and only recreate onto the new image.
GHCR is the registry of record. Every build pushes sha-<commit>, which is what makes a rollback to any commit possible.
The image tag is a Coolify variable. The workflow patches <SVC>_IMAGE_TAG before it redeploys. Without that step a push would build a new image and redeploy the old one, and nothing would fail.
One repo can deploy several services. The agent repo redeploys both agent and auth, because both run the same image.

The workflow needs two org-level values, already configured: the secret COOLIFY_TOKEN and the variable COOLIFY_URL (http://localhost:8000, from the runner's own host).

The agent: staging first, then promote¶

The agent repo does not deploy production on a push (COB-471, 2026-10-19).

flowchart TD
    A["push to main"] --> B["tests"]
    B --> C["build sha-&lt;short&gt;, push to GHCR"]
    C --> D["deploy.sh TARGET=staging<br/>agent-staging :14141"]
    D --> E["smoke suite against staging<br/>(blocks promote)"]
    F["gh workflow run deploy.yml -f promote=&lt;sha&gt;"] --> G{"did that commit's<br/>staging job pass?"}
    G -- no --> H["refused, nothing deployed"]
    G -- yes --> I["deploy.sh TARGET=production<br/>agent (+ auth if its files changed)"]

scripts/deploy.sh keeps a new image only when /health reports its SHA, /health/full has no agent_model error and a smoke chat reaches done. Otherwise it writes the previous AGENT_IMAGE_TAG back and redeploys. It also refuses to deploy a commit older than the one already live.
agent-staging is a Coolify service in the staging environment of the Dataland project (xursbygrk46ichysnm0ytdcq). Port 14141, network alias dataland-agent-staging only, database agent_staging in the same Postgres. It shares auth, RAG and museum-api with production and reads the live Redis. It has no ops token (no welcome pushes, no complaint tickets), GCS uploads are off, and it has its own service token. Staging never redeploys auth.
Answer quality between two builds is compared with the eval set in the agent repo: uv run python -m evals run --base-url http://100.124.170.43:14141 ..., then python -m evals compare old.json new.json.
Promote is the only way production changes. To go back to an older commit, set the tag by hand (see Rollback below); deploy.sh will not go backwards.

Deploy or restart by hand¶

Through the browser: open Coolify, pick the resource, then Redeploy or Restart.

Through the API, from the tailnet:

UUID=5qwdlo0cynw03kzaat5n1vpc                       # (1)!
curl -X POST -H "Authorization: Bearer $TOKEN" \
  "http://100.124.170.43:8000/api/v1/deploy?uuid=$UUID"

curl -X POST -H "Authorization: Bearer $TOKEN" \
  "http://100.124.170.43:8000/api/v1/services/$UUID/restart"   # (2)!

Service UUIDs come from GET /api/v1/services. Read the token from the secret inventory. Never pass it as a literal argument in a shared shell.
Restart takes POST. A GET answers 405.

Change a value¶

Coolify holds every environment variable. Editing a .env file on the host changes nothing.

curl -H "Authorization: Bearer $TOKEN" \
  "http://100.124.170.43:8000/api/v1/services/$UUID/envs"          # (1)!

curl -X PATCH -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"key":"SOME_KEY","value":"some-value"}' \
  "http://100.124.170.43:8000/api/v1/services/$UUID/envs"          # (2)!

Read the current values.
Write one value. A restart follows, or the running container keeps the old value.

Rollback¶

Coolify → Deployments → Redeploy the previous entry. Or set <SVC>_IMAGE_TAG back to an earlier sha-<commit> and redeploy. Every build sits in GHCR, so any past commit is a candidate.

Smoke test¶

After a deploy:

bash dataland-infrastructure/scripts/smoke.sh

Read-only suite per service (agent, museum, rag-v2, notification, knowledge), green and red rollup, logs to dataland-infrastructure/.smoke-logs/.

Exit code	Meaning
`0`	all green
`1`	one or more suites failed
`2`	one or more skipped (tunnel down, suite missing)

For full coverage including write paths, open the internal-port tunnel and turn writes on:

ssh -L 4143:127.0.0.1:4143 -L 8080:127.0.0.1:8080 -L 4145:127.0.0.1:4145 \
    ege@100.124.170.43   # (1)!

SMOKE_ALLOW_WRITES=1 \
SMOKE_NOTIFICATION_REDIS_CLEANUP=tunneled \
bash dataland-infrastructure/scripts/smoke.sh   # (2)!

Forwards the loopback-only internal ports (4143, 8080, 4145) over SSH, so the write-path suites reach services bound to 127.0.0.1.
SMOKE_ALLOW_WRITES=1 moves the suite from read-only to write paths, which is safe only behind the tunnel. SMOKE_NOTIFICATION_REDIS_CLEANUP=tunneled tells the suite to clean up its notification Redis test keys through the forwarded port.

scripts/smoke-visit-flow.sh exercises the ticket, telemetry and notification path end to end, and tails the notification worker to confirm rules fire.

Logs¶

Coolify → resource → Logs is the fastest route. From a shell:

docker logs -f "$(scripts/dl --name agent)"
docker logs -f "$(scripts/dl --name museum-api)"

Each service also writes JSONL to ${DATALAND_LOG_DIR:-/home/cobanov/DATALAND/logs}/<service>/. The Docker json-file driver is capped at 10 MiB × 5 rotations, 50 MiB per container, so a chatty service cannot fill the host disk. The bind-mounted JSONL needs its own logrotate config — see reports/runbook.md.

Migrations stay manual¶

Alembic migrations are not applied automatically. The move to Coolify did not change that.

scripts/dl agent uv run alembic upgrade head

See Running the stack.

Simulator¶

For local end-to-end testing without an Empatica band, layer compose.sim.yml:

SIM_TICKET_ID=<a-real-ticket> \
SIM_REDIS_PASSWORD=$(openssl rand -hex 24) \
SIM_FORCE_HR_SPIKE=1 SIM_FORCE_EDA_SPIKE=1 \
bash dataland-infrastructure/scripts/start-simulator.sh   # (1)!

docker logs -f "$(scripts/dl --name notification-worker)"  # (2)!

SIM_TICKET_ID must be a real ticket, so the synthetic telemetry maps to a routable visitor. SIM_REDIS_PASSWORD is minted fresh for the isolated sidecar redis, never reused from production. The two spike flags inject heart-rate and EDA spikes, so the notification rules fire.
Tail the worker to watch the rules trigger. The overlay points notification-worker at the sim redis, so these are sim events only.

The overlay does three things:

Starts a sidecar dataland-redis-sim (separate volume, loopback port 4149).
Starts dataland-telemetry-sim, a Python publisher that XADDs synthetic events to museum:telemetry on the sim redis.
Recreates notification-worker and notification-api with REDIS_HOST pointed at the sim redis. The main redis keeps serving every other service.

Stop with bash dataland-infrastructure/scripts/stop-simulator.sh.

Never replay simulation telemetry into live redis

The simulator must stay isolated on its own redis container. Replaying museum-simulation telemetry into the production redis would corrupt the live museum:telemetry stream that the notification consumers drain.

Destructive reset¶

reset-stack.sh is gone. Coolify owns the lifecycle, and there is no one-command teardown any more. To rebuild a service from empty, delete its volumes in Coolify → resource → Storages, then redeploy.

Read the backup notes first

Dropping a volume is not reversible from inside the stack. Read reports/backup-restore.md before you touch postgres-data, qdrant-data, redis-data, auth-data, or the knowledge SQLite bind. auth-data holds the auth server signing key, so dropping it mints a new kid and invalidates every issued JWT — see Auth.

Inspect¶

scripts/dl --list                                    # (1)!
docker images 'ghcr.io/dataland-ai/*'                # (2)!
curl -H "Authorization: Bearer $TOKEN" \
  "http://100.124.170.43:8000/api/v1/services"       # (3)!

Every container on dataland-network, with status.
Every image tag on the host. Each sha-<commit> is a rollback candidate.
Every Coolify service with its UUID and status.

Adding a new service¶

Add the service to the Coolify compose resource, attached to dataland-network, with a network alias that keeps its dataland-<name> DNS name.
Pin mem_limit and cpus to a realistic budget. The defaults are unbounded.
Bind the host port to 127.0.0.1:, and optionally a *_PUBLIC_BIND tailnet IP. Never 0.0.0.0.
Add a healthcheck.
Add a deploy.yml workflow in the service repo, modelled on the agent's: build, push to GHCR, patch <SVC>_IMAGE_TAG, redeploy, poll /health.
Set the service environment variables in Coolify, not in a .env file.
If it is public-facing, add the Cloudflare ingress route (Zero Trust → Tunnels → minotaur tunnel → Public Hostname → Add).
Document it: copy a service page in docs/src/services/, add it to docs/mkdocs.yml nav so the --strict build passes, then push.