Coolify migration & runbook¶

Tracked as RAS-119.

This page describes how the Dataland stack moves from hand-run docker compose to Coolify, and how to operate it afterwards. Read Public ports and Running the stack first — the port and bind policy does not change.

Spark is the live production host

Phases 0 to 2 do not touch the running stack. Only phase 3 stops containers. Run coolify/preflight.sh before every phase, and keep the output with the change record.

Why move¶

Today	With Coolify
One production environment. No test, no dev.	`production`, `test` and `development` as separate environments of one project.
Status through `docker ps` over SSH.	Status, logs, terminal and metrics per service in a browser.
Six hand-edited `.env` files. File permissions have broken deploys.	Values held by Coolify, versioned, with an audit trail.
Rollback = remember the previous `IMAGE_TAG`.	Deployment history, roll back from a list.
Images built on the production host by `deploy.sh`.	Images built by GitHub Actions and pulled by tag.
No CI. Only `test.yml` on the agent and `pip-audit.yml` on rag.	Push to `main` builds, pushes to GHCR, deploys to `test`.

What Coolify forces us to change, and why¶

Four properties of Coolify shape the whole design. Ignore any one of them and the migration breaks at runtime, not at deploy time.

1. No build contexts across repositories¶

A Coolify compose resource clones one repository. The current compose.yml builds five images from five sibling repositories (context: ../dataland-agent). That cannot work.

Answer: every image is pre-built and pulled from GHCR (ghcr.io/dataland-ai/<service>). A GitHub Actions workflow in each service repo builds and pushes it.

2. Container names get a UUID¶

Coolify adds a UUID to container names. The services address each other by the current names — http://dataland-agent:4141, dataland-redis:6379, dataland-postgres:5432 — and those names also sit inside the per-repo .env files. Under Coolify they stop resolving.

Answer: every service declares a network alias that restores its current name:

    networks:
      dataland-network:
        aliases:
          - dataland-agent

An alias belongs to the network, not to the container, so it survives any rename. This is the highest-risk item in the migration. preflight.sh step 7 records the names that must keep resolving.

3. Volume names get a prefix¶

Coolify prefixes volume names with the resource UUID. The four production volumes already hold live data.

Answer: declare them external: true with their exact names. Compose then attaches them as they are. It never creates, copies or removes them. If a name is wrong the deploy fails at once and changes nothing — which is the safe failure.

volumes:
  postgres-data:
    external: true
    name: dataland_postgres-data

4. arm64 host, private repositories¶

Spark is aarch64 and every repo is private, so GitHub-hosted arm64 runners are a paid add-on. The RAG image also needs CUDA on arm64.

Answer: register a self-hosted Actions runner on Spark. It is native arm64, it has the GPU, and it warms the same Docker layer cache the deploy pulls from.

Environment variables: why they move last¶

The stack reads six files: one shared .env with 136 variables, plus a per-repo .env for each service (agent 43, rag 54, knowledge 24, museum 23, notification 40).

117 variable names appear in more than one file, and some genuinely conflict:

Variable	rag	knowledge	agent
`APP_PORT`	4143	4152	—
`DATABASE_URL`	—	`…/knowledge`	`…/agent`
`API_KEY`	rag key	catalog token	agent key

Merging them into one Coolify env store would therefore break the stack silently. So the cutover does not move them. coolify/compose.prod.yml reads the same files by absolute path:

    env_file:
      - /home/cobanov/DATALAND/.env
      - /home/cobanov/DATALAND/dataland-agent/.env

Env semantics are then identical before and after the cutover, and the risky part of the migration — aliases and volume adoption — is tested on its own. Moving the variables into Coolify is phase 5, one service at a time, with a prefix per service (RAG_APP_PORT → APP_PORT: ${RAG_APP_PORT}).

Files in this repo¶

Path	Purpose
`coolify/compose.prod.yml`	The production resource. GHCR images, aliases, external volumes, host-path env files.
`coolify/compose.nonprod.yml`	Serves both `test` and `development`. Own volumes, own network, loopback ports in the 1xxxx range, CPU rag image. Set `ENV_SLUG`.
`coolify/preflight.sh`	Read-only checks. Run before every phase.
`coolify/workflows/build-and-push.yml`	Template to copy into each service repo.
`compose.yml`	The legacy stack. Keep it. It is the rollback path.

Phase 0 — repository work¶

No production impact.

Commit the memory-limit drift into compose.yml (auth 512m, museum 1g, agent 2g, knowledge 2g). The live host already runs these values. If the rollback path keeps the old limits, a rollback quietly downgrades the containers.
Review coolify/compose.prod.yml against compose.yml, service by service.
Copy coolify/workflows/build-and-push.yml into each service repo as .github/workflows/build-and-push.yml and set IMAGE_NAME.

Validate both files anywhere Docker runs:

docker compose -f coolify/compose.prod.yml config --quiet

Phase 1 — install Coolify¶

Coolify runs its own containers on ports that preflight.sh step 2 confirms are free (80, 443, 8000, 6001, 6002). The install is safe only after the step below.

Do not run the plain installer on this host

The documented one-liner restarts the Docker daemon on Spark, and it writes a Docker address pool that overlaps the host LAN. Read this section fully. Both problems are removed by one file written first.

Why the plain installer is unsafe here¶

The installer audit (2026-08-13, install.sh, 1056 lines) found two problems specific to this host.

1. It restarts the Docker daemon. Spark has no /etc/docker/daemon.json. That sets EXISTING_POOL_CONFIGURED=false, so the installer writes a new daemon.json, sets NEED_MERGE=true, and calls restart_docker_service, which runs systemctl restart docker. That bounces all 18 containers on the box — the 10 dataland services and the 8 Atlas ones. It is an outage, and it also risks the documented CDI failure: a daemon event has revoked the RAG container's /dev/nvidia* device-cgroup access before and dropped the reranker silently to CPU (see the comment on the rag service in compose.yml).

2. Its default address pool overlaps the host LAN. The installer defaults to DOCKER_ADDRESS_POOL_BASE=10.0.0.0/8. Spark's LAN is 10.14.0.0/16 and its default gateway is 10.14.0.1 over Wi-Fi. Docker's IPAM does not read the host routing table — it only avoids subnets other Docker networks already hold. A later network allocation inside 10.14.0.0/16 would therefore black-hole the host's own default route. Tailscale rides that LAN, so remote access would go with it, and recovery would need the out-of-band agentkvm console.

Write daemon.json first, then install¶

Writing the file does not restart Docker and does not disturb any running container. Docker reads it at its next start. The installer then finds a configuration that already matches, reports "Configuration is up to date", and skips the restart.

The pool below stays inside 172.16.0.0/12 — the same RFC1918 block the existing networks already use (172.17 to 172.20) — so it cannot collide with the 10.14.0.0/16 LAN or the 100.64.0.0/10 tailnet. Docker's IPAM skips the /16s that are already allocated.

# On Spark, as root. Nothing restarts.
test -f /etc/docker/daemon.json && { echo "ALREADY EXISTS — stop and re-audit"; exit 1; }
cat > /etc/docker/daemon.json <<'EOF'
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "default-address-pools": [
    {"base":"172.16.0.0/12","size":24}
  ]
}
EOF
jq . /etc/docker/daemon.json      # must parse
docker info >/dev/null && echo "daemon untouched, containers still up"

The three log values must match the installer's expectation exactly (json-file, 10m, 3). If they differ, the installer sets NEED_MERGE=true and restarts Docker after all.

Then install, pinning the pool so the installer cannot substitute its default:

ssh ege@100.124.170.43 'bash -s' < coolify/preflight.sh   # must report 0 failed

curl -fsSL https://cdn.coollabs.io/coolify/install.sh -o /tmp/coolify-install.sh
# Read it before running it. The audit above was made against the 2026-08-13
# copy, and the file is not versioned upstream.
DOCKER_ADDRESS_POOL_BASE=172.16.0.0/12 DOCKER_ADDRESS_POOL_SIZE=24 \
  sudo -E bash /tmp/coolify-install.sh

Watch the output for Configuration is up to date. If it prints restarting Docker daemon instead, the stack has just bounced — go straight to the verification below.

Verify after the install, either way¶

docker ps --format '{{.Names}}\t{{.Status}}' | sort
curl -fsS http://127.0.0.1:4141/health && echo   # agent
curl -fsS http://127.0.0.1:4152/health && echo   # knowledge
# The RAG container must still hold the GPU. This is the check that catches
# the silent CDI degradation.
scripts/dl rag nvidia-smi -L

Then, in the browser at http://100.124.170.43:8000:

Create the first admin user. Do this at once — the first visitor to that page claims the instance.
Add the localhost server. Coolify reaches Docker through the local socket.
Create the project Dataland with environments production, test and development.
Install the GitHub App on the dataland-ai organisation (Sources → GitHub). The repositories are private, so the app or a deploy key is required.

Do not expose port 8000

Bind the dashboard to the tailnet only, and do not add it to the Cloudflare tunnel. Coolify holds every deploy credential in the stack.

Register the Actions runner¶

# On Spark, as ege
mkdir -p ~/actions-runner && cd ~/actions-runner
curl -o runner.tar.gz -L \
  https://github.com/actions/runner/releases/latest/download/actions-runner-linux-arm64.tar.gz
tar xzf runner.tar.gz
./config.sh --url https://github.com/dataland-ai \
            --token <ORG_RUNNER_TOKEN> \
            --labels self-hosted,linux,ARM64,spark
sudo ./svc.sh install ege && sudo ./svc.sh start

Get <ORG_RUNNER_TOKEN> from GitHub → dataland-ai → Settings → Actions → Runners → New runner. The token is valid for one hour.

Then run each build-and-push workflow by hand once, and confirm the images appear under the organisation's packages.

Phase 2 — prove it in `test`¶

No production impact. Nothing here may reference /home/cobanov/DATALAND.

# On Spark: build the non-production config tree
mkdir -p /home/cobanov/DATALAND-test/{secrets,dataland-agent,dataland-rag,dataland-knowledge,dataland-museum,dataland-notification}

Fill each .env from the matching production file, then change every value that must not be shared:

separate database and redis passwords;
MUSEUM_TELEMETRY_BRIDGE_ENABLED=false, or point RDC_REDIS_URL at the simulator — never at the live RDC redis;
a test OneSignal app, so no push reaches a real visitor;
test-only session secrets and dashboard passwords.

In Coolify, add a Docker Compose resource to the test environment:

source: dataland-ai/dataland-infrastructure, branch main;
compose path: coolify/compose.nonprod.yml;
environment variables: ENV_SLUG=test plus the values the file interpolates.

Deploy, then check:

curl -fsS http://127.0.0.1:14141/health          # agent
curl -fsS http://127.0.0.1:14152/health          # knowledge
docker exec <test-agent> getent hosts dataland-redis   # aliases resolve

Run the smoke suite against http://127.0.0.1:14141. See the smoke section of Deploy.

Only continue when test is green.

Phase 3 — production cutover¶

This phase stops containers. Plan a window and tell the museum.

Back up first¶

cd /home/cobanov/DATALAND
ts=$(date -u +%Y%m%d-%H%M%S)
mkdir -p _backups/coolify-cutover-$ts
for db in agent auth knowledge; do
  scripts/dl postgres pg_dump -Fc -U dataland "$db" \
    > "_backups/coolify-cutover-$ts/$db.dump"
done
# Copy each volume, do not move it. The originals stay in place.
for v in postgres-data qdrant-data redis-data auth-data; do
  docker run --rm -v "dataland_$v:/src:ro" \
    -v "$PWD/_backups/coolify-cutover-$ts:/out" alpine \
    tar czf "/out/$v.tar.gz" -C /src .
done

Cut over¶

ssh ege@100.124.170.43 'bash -s' < coolify/preflight.sh   # must report 0 failed

cd /home/cobanov/DATALAND
# `stop`, NOT `down`. `stop` leaves the containers, the network and every
# volume in place, so the rollback is one command.
docker compose -f dataland-infrastructure/compose.yml --env-file .env stop

Then add the production Docker Compose resource in Coolify:

compose path: coolify/compose.prod.yml;
set each *_IMAGE_TAG to the exact SHA tag the build produced. Never latest in production — latest makes a redeploy unrepeatable.

Deploy, then verify:

docker ps --format '{{.Names}}\t{{.Status}}'
curl -fsS http://127.0.0.1:4141/health
curl -fsS http://127.0.0.1:4152/health
curl -fsS -H "X-API-Key: $RAG_API_KEY" http://127.0.0.1:4143/health

# The aliases must resolve, or service-to-service calls fail.
agent=$(docker ps --filter name=agent --format '{{.Names}}' | head -1)
for h in dataland-postgres dataland-redis dataland-rag dataland-museum \
         dataland-auth dataland-notification-api; do
  docker exec "$agent" getent hosts "$h" || echo "BROKEN: $h"
done

# The RAG container must still hold the GPU.
docker exec $(docker ps --filter name=rag --format '{{.Names}}' | head -1) nvidia-smi -L

Last, confirm the public hostnames through the Cloudflare tunnel: dataland.chat and knowledge.dataland.chat. The tunnel is not touched in this phase — it still points at the same host ports.

Roll back¶

Any failed check, at any point:

cd /home/cobanov/DATALAND
docker compose -f dataland-infrastructure/compose.yml --env-file .env up -d

The old containers were only stopped, and the volumes were never moved, so this restores the previous state. Stop the Coolify resource first, or the two stacks fight over the same host ports.

Phase 4 — CI/CD¶

test deploys on every push to main, through the webhook step in build-and-push.yml.
Production stays manual. Promote by setting the *_IMAGE_TAG to a SHA tag that test has already proved, then redeploy.
Add a scheduled Postgres backup in Coolify (Databases → Backups), and keep the pg_dump cron until the Coolify backup has been restored once in a drill.

Phase 5 — clean up¶

Only after production has run on Coolify for a full week.

Disable dataland-stack.service. Coolify owns the restart policy. Keep the unit file until the first host reboot proves the new path.
Retire deploy.sh.
Move the environment variables into Coolify, one service at a time, with a per-service prefix. Reduce the .env files to a break-glass copy.
Move the Atlas stack (/opt/atlas) the same way.

Day-to-day operation after the move¶

Task	Before	After
Deploy one service	`docker compose up -d --build --no-deps <svc>`	Push to `main`; promote the tag in Coolify
Read logs	`docker compose logs -f <svc>`	Coolify → resource → Logs
Restart a service	`docker compose restart <svc>`	Coolify → resource → Restart
Roll back	Recall the old `IMAGE_TAG`	Coolify → Deployments → Redeploy
Change a value	Edit `.env`, redeploy	Coolify → Environment Variables (phase 5)
Run a migration	`scripts/dl agent uv run alembic upgrade head`	Unchanged. Still an explicit operator step.

Migrations stay manual

Alembic migrations are not applied automatically, before or after this move. See Running the stack.