Skip to content

Auth — RS256 JWT issuance, JWKS, and the CMS-key mirror

auth is the identity provider that issues and serves the signing material the rest of the stack trusts. It is the auth_server.py entrypoint of the agent repo (dataland-agent) — the same image as the agent, run as its own compose service on Spark.

Two distinct roles converge here:

  1. Issuer — auth signs RS256 JWTs (access + refresh) for accounts it stores in Postgres (argon2id-hashed passwords), and exposes a standard JWKS endpoint so any verifier can validate tokens without ever talking back to the issuer.
  2. JWKS mirror — the same JWKS endpoint also serves the external CMS / mobile-backend public key (kid=dataland-rs256-1), so the agent can validate production-issued chat tokens locally, removing the chat-auth single point of failure.

Visitor identity for the museum chat flow is handled upstream by the CMS / mobile backend, which signs the real production tokens. auth is the agent's primary JWKS source; the external CMS endpoint is demoted to a pure backup. See Agent for how tokens are consumed on the request path.
Compose service auth
Container dataland-auth
Image the agent image dataland/agent:${IMAGE_TAG}
Entrypoint uv run python auth_server.py
Container port 9000
Internal URL http://dataland-auth:9000 (on dataland-network; internal only)
Data store on-host dataland-postgres (the dataland database), table auth_server_users
Signing material dataland_auth-data volume (/app/data) — auth_rsa_private.pem + auth_rsa_kid.txt
Repo dataland-agent (auth_server.py)

What it does

  • Issues RS256 JWTs — POST /api/auth/signup and POST /api/auth/login return an access token (1 h TTL) and a refresh token (7 day TTL), both signed with the local RSA private key under the header kid.
  • Serves JWKS at GET /.well-known/jwks.json — the local public key plus any mirrored external public keys.
  • Stores accounts in Postgres table auth_server_users (in the on-host dataland database, via AUTH_DATABASE_URL, which falls back to DATABASE_URL). Passwords are argon2id.
  • Hosts a dark-themed web UI at / — a login/signup form that mints a token and can deep-link into the agent's test chat client with ?token=….
  • GET /api/auth/me — returns the current user record for a valid Bearer token.

Auth issuance vs. token verification live in two places

auth (auth_server.py) signs and serves keys. The agent (app/auth.py) verifies tokens against one or more JWKS URLs. They share the repo and image but are different processes with different jobs. Do not conflate the auth server's private key (signing) with the agent's verification path (public-key only).

Architecture

graph LR
  subgraph dnet["dataland-network (Spark)"]
    AU["auth (dataland-auth)<br/>auth_server.py :9000"]
    AG["agent<br/>app/auth.py"]
    PG[("postgres<br/>dataland db · auth_server_users")]
    VOL[["dataland_auth-data volume<br/>RSA key + extra_jwks.json"]]
  end
  CMS["CMS / mobile backend<br/>kid=dataland-rs256-1"]

  AU -->|"asyncpg pool"| PG
  VOL --> AU
  AG -.->|"JWKS_URL (primary)"| AU
  AG -.->|"JWKS_URLS (fallback, backup)"| CMS
  CMS -.->|"public key mirrored via<br/>AUTH_EXTRA_JWKS_JSON"| AU

RSA signing keys

On first boot auth_server.py generates a 2048-bit RSA key pair and a kid (form local-<8hex>) and persists them to the dataland_auth-data volume (/app/data). Environment variables can override the on-disk material — useful for pinning a specific key so existing JWTs stay valid across a rebuild:

Source Provides Notes
data/auth_rsa_private.pem + data/auth_rsa_kid.txt on-disk key + kid The default. Generated on first boot and kept in the dataland_auth-data volume.
AUTH_RSA_PRIVATE_PEM PKCS8 RSA private key (signing material — secret) Optional override; use it to pin a preserved key.
AUTH_RSA_KID the kid stamped into every token header The kid the dedup logic protects: an extra mirrored key reusing it is always skipped.
AUTH_EXTRA_JWKS_JSON mirrored external public JWKs (kid=dataland-rs256-1) The CMS-key mirror; absent it, the external CMS endpoint becomes the sole validator (the SPOF this feature removes).

The RSA private key is the one irreplaceable secret

Losing the signing key (and any restorable backup) invalidates every previously issued JWT — every active staff session and any minted token dies, and the agent re-fetches a brand-new kid it has never seen. The key lives in the dataland_auth-data volume; back it up. Rotating it is a key-rotation event — re-provision the JWKS mirror and notify any client that pinned the prior JWKS.

The public key is exported as a JWK with kid, use="sig", alg="RS256".

The JWKS endpoint and the CMS-key mirror

The problem it solves

The agent verifies a token by trying each configured JWKS URL in order (JWKS_URL first, then each entry of JWKS_URLS). Production chat tokens are signed by the CMS backend with kid=dataland-rs256-1. With that key absent from the local JWKS, the only place the agent can resolve it is the external CMS endpoint — making that external endpoint the sole source of the verification key for all chat auth. If the CMS endpoint blips, every chat request 401s.

The fix

auth serves the external public key alongside its own. A JWKS only ever exposes public material — never the private signing key — so mirroring it is safe. With the key present, chat auth resolves against auth (the primary), and the external endpoint becomes a pure backup.

How the served key set is built

auth_server.py merges keys at startup:

  1. The local public JWK is always first and authoritative.
  2. Extra public JWKs are loaded from inline env JSON first (AUTH_EXTRA_JWKS_JSON), then the on-disk file (AUTH_EXTRA_JWKS_PATH, default data/extra_jwks.json).
  3. Entries are accepted only if they carry both kty and kid; everything else is dropped as noise.
  4. Keys are deduplicated by kid — an extra key reusing the local kid is skipped, so a stale mirror can never shadow the key this server actually signs with.
  5. A malformed or missing extra-key source yields zero extra keys, logged via the auth.extra_jwks.bad_source observability event — it never raises. A bad mirror must not take auth (and thus all chat) down.

When extras load successfully, the server emits auth.extra_jwks.loaded with the count and the mirrored kids.

flowchart TD
  A["local public JWK<br/>(kid = AUTH_RSA_KID)"] --> M{merge by kid}
  B["AUTH_EXTRA_JWKS_JSON<br/>(inline env, first)"] --> L[load_extra_jwks]
  C["data/extra_jwks.json<br/>(AUTH_EXTRA_JWKS_PATH)"] --> L
  L --> M
  M -->|local first, dedup by kid,<br/>local kid never overridden| S["/.well-known/jwks.json<br/>{ keys: [...] }"]

Provisioning the mirror

Set the CMS public JWK in AUTH_EXTRA_JWKS_JSON (inline JSON) or drop it in data/extra_jwks.json. The value is the upstream public JWK:

{ "keys": [ { "kty": "RSA", "use": "sig", "alg": "RS256",
  "kid": "dataland-rs256-1", "n": "<modulus>", "e": "AQAB" } ] }

Only the public JWK belongs here (n/e, never d); a JWKS must never carry private material. Entries are kept only if they have both kty and kid. After updating the key, restart auth so auth_server.py re-reads the merged key set at startup, then verify both keys are served:

curl -fsS http://dataland-auth:9000/.well-known/jwks.json | jq '.keys[].kid'  # (1)!
# -> the local signing kid (AUTH_RSA_KID)
# -> "dataland-rs256-1" (the mirrored CMS kid)
  1. Confirms the merged key set serves the local authoritative key first, then the mirrored dataland-rs256-1. Seeing only one kid means the mirror did not load (check the auth.extra_jwks.bad_source event).
Env var Default Purpose
AUTH_EXTRA_JWKS_JSON (unset) Inline JSON ({"keys":[…]} or a bare list) of extra public JWKs. Loaded first.
AUTH_EXTRA_JWKS_PATH data/extra_jwks.json On-disk file of extra public JWKs, loaded after the inline source.

Re-run after a CMS key rotation

If the CMS rotates dataland-rs256-1, update the mirror (AUTH_EXTRA_JWKS_JSON or data/extra_jwks.json) and restart auth — otherwise the agent silently falls back to the external endpoint as sole validator and starts emitting the FALLBACK JWKS warning.

How the agent verifies tokens

Verification lives in the agent at dataland-agent/app/auth.py. The flow (_verify_access_token_sync):

  1. If AUTH_SKIP=true, the signature and expiry are not checked — the token is decoded unverified and a WARNING is logged. Never enable in production (the boot guard refuses it).
  2. Otherwise, for each URL in settings.auth_jwks_urls (i.e. JWKS_URL followed by every JWKS_URLS entry, de-duplicated, order preserved):
  3. Resolve the signing key via a cached PyJWKClient (jwks_cache_ttl, default 3600 s). On a cache miss the keys are re-fetched once before giving up — this transparently handles key rotation.
  4. jwt.decode(..., algorithms=["RS256"], options={"require": ["exp"], "verify_aud": False}). exp is required; aud is intentionally not checked because CMS tokens scope aud to the mobile client UUID and the agent treats any authenticated caller as a valid consumer.
  5. First URL that verifies wins; the loop breaks.

  6. If verification succeeds on a fallback provider (index > 0), the agent logs:

    JWT accepted by FALLBACK JWKS provider <url> -- local JWKS is missing this signing key; chat auth depends on this external endpoint

    This is the alert signal: the local JWKS is missing the key and should be re-mirrored. Treat it as an actionable warning, not noise. 4. If no provider verifies, the agent returns 401 Invalid or expired token.

After verification, _normalize_access_payload enforces token shape:

  • token_type, if present, must be access — refresh tokens are rejected with 403.
  • A user identifier is required: user_id if present, else sub. Missing both → 401 Token missing user identifier claim.
  • get_or_create_user auto-creates a local agent user from the claims (user_id, email, full_name, location, profile_photo_url, joined_date, access_permissions, stripe_customer_id) and stashes it on request.state.current_user for the logging middleware.
sequenceDiagram
  participant C as Client
  participant AG as agent /v1/*
  participant AU as auth JWKS (primary)
  participant CMS as CMS JWKS (fallback)
  C->>AG: Bearer <RS256 JWT>
  AG->>AU: fetch JWKS (cached, TTL 3600s)
  alt kid resolves locally (mirror present)
    AU-->>AG: public key for kid
    AG->>AG: verify sig + exp, normalize claims
    AG-->>C: 200 (SSE / JSON)
  else kid missing locally
    AG->>CMS: fallback fetch
    CMS-->>AG: public key
    AG->>AG: WARN "FALLBACK provider ... chat auth depends on external"
    AG-->>C: 200 (but mirror should be re-provisioned)
  end

Login & signup (argon2id)

Passwords are hashed with argon2id via argon2-cffi defaults (time_cost=3, memory_cost=64 MiB, parallelism=4), landing at ~70–90 ms/hash on the VDS — costly for an attacker, invisible to a user. The previous scheme was sha256(salt + password), a fast GPU-friendly hash.

  • New rows store an $argon2… string; the salt column is set to the literal "argon2" (argon2 encodes its own salt + parameters in the hash). The salt column is retained only for legacy verification.
  • Verification dispatches by format: $argon2… → argon2id verify; 64-char hex → legacy sha256(salt+password) constant-time compare. Any malformed/corrupt row returns False rather than raising.
  • Opportunistic upgrade: a legacy sha256 row that authenticates successfully is re-hashed to argon2id on that login. No new logins ever land sha256, so every user migrates on next sign-in.

Schema (auth_server_users)

Created idempotently at startup (CREATE TABLE IF NOT EXISTS + additive ALTERs). Columns: id, sub (UUID, the JWT subject), email (unique, lower-cased), hash, salt, full_name, location, profile_photo_url, stripe_customer_id, created_at. Rows lacking a sub are backfilled with a fresh UUID on boot.

Issued token claims

_mint_tokens produces two RS256 JWTs signed with the local kid:

Token TTL Notable claims
access 1 hour sub, email, full_name, location, profile_photo_url, stripe_customer_id, joined_date, access_permissions: [], iss, aud, iat, exp, jti
refresh 7 days token_type: "refresh", sub, iss, aud, iat, exp, jti

iss comes from AUTH_ISSUER (defaults to http://dataland-auth:9000); aud defaults to dataland-agent (AUTH_AUDIENCE). Because the agent runs verify_aud=False, the aud value is informational on the agent side.

API reference

Method Path Auth Description
GET / none Login/signup web UI (mints a token, can deep-link to the agent test chat)
GET /.well-known/jwks.json none Public JWKS (local key + mirrored extras)
POST /api/auth/signup none Create account → { access, refresh, user, … }
POST /api/auth/login none Authenticate → { access, refresh, user, … }
GET /api/auth/me Bearer Current user record (verifies with the local public key)
GET /metrics — Prometheus-format text (service="dataland-auth")

signup requires email + password (min length 4); a duplicate email returns 409. login returns 401 on bad credentials. me verifies the Bearer token against the local public key only (verify_aud=False) and looks the user up by sub.

me here vs. the agent's /v1/auth/me

GET /api/auth/me on auth verifies against the local key and is for the auth server's own UI/flows. The visitor-facing GET /v1/auth/me lives on the agent and runs the full multi-JWKS verification path. They are different endpoints on different services.

Key env vars

auth (the issuer/server)

AGENT_URL=https://dataland.chat          # (1)!
AUTH_DATABASE_URL=postgresql://***@dataland-postgres:5432/dataland   # (2)!
AUTH_RSA_KID=                            # (3)!
AUTH_PORT=9000                           # (4)!
AUTH_ISSUER=http://dataland-auth:9000    # (5)!
AUTH_AUDIENCE=dataland-agent             # (6)!
AUTH_EXTRA_JWKS_JSON={"keys":[...]}      # (7)!
AUTH_EXTRA_JWKS_PATH=/app/data/extra_jwks.json   # (8)!
LOG_FILE_DIR=/app/logs
  1. Baked into the web UI's "open agent" deep-link so the minted token can be handed off to the agent's test chat client (?token=…). Cosmetic only — it does not affect token issuance.
  2. The Postgres DSN. Falls back to DATABASE_URL, then a local-dev default; a postgresql+asyncpg:// form is normalized to postgresql:// for asyncpg, and credentials are masked in the boot banner. Points at the same on-host dataland-postgres instance the agent uses.
  3. Optional override for the signing kid. Left unset, auth generates/reads local-<8hex> from the dataland_auth-data volume. Pair with AUTH_RSA_PRIVATE_PEM to pin a preserved key.
  4. Listen / container port, default 9000.
  5. The iss claim stamped into issued tokens. The agent does not enforce it, but it identifies who minted the token.
  6. The aud claim (default dataland-agent). Informational on the agent side because verification runs with verify_aud=False.
  7. The CMS-key mirror ({"keys":[…]} or a bare list). Loaded first.
  8. On-disk file of extra public JWKs, loaded after the inline JSON source.

dataland-agent (the verifier) — auth-relevant settings

JWKS_URL=http://dataland-auth:9000/.well-known/jwks.json   # (1)!
JWKS_URLS=                                                 # (2)!
JWKS_CACHE_TTL=3600                                        # (3)!
AUTH_SKIP=false                                            # (4)!
  1. The primary JWKS source, tried first on every verification — the on-Spark auth service over the docker network. auth mirrors the CMS key so the agent verifies production chat tokens against it. The boot guard refuses to start production if this is empty.
  2. Optional fallback providers, comma-separated or a JSON array, tried in order after JWKS_URL. Keep this empty in production once the mirror is in place; a committed default pointing at staging once let prod silently accept staging-signed tokens. A token that verifies here (index > 0) triggers the FALLBACK warning.
  3. PyJWKClient cache lifetime in seconds (default 3600). On a cache miss the keys are re-fetched once before giving up, which is how transparent key rotation works.
  4. Bypass switch — when true, neither signature nor exp is checked and the token is decoded unverified (with a WARNING). NEVER true in production; assert_boot_required_env() crash-loops the agent if it is set under APP_ENV=production.

Keep JWKS_URLS empty in production once the mirror is in place

A committed default pointing at staging once caused prod to silently accept staging-signed tokens (including self-issued ones). With the CMS key mirrored into auth, production verifies entirely against the primary; leave JWKS_URLS empty unless you deliberately need a second issuer.

Production boot guard

The agent refuses to boot in production with broken auth. production_required_env_issues() (in app/runtime.py) flags:

  • JWKS_URL is empty; tokens cannot be verified in production
  • AUTH_SKIP is true; production deploys must verify JWTs

assert_boot_required_env() raises RuntimeError (crash-loop) on any such issue when APP_ENV=production; non-production stays warn-only. deploy.sh runs this same guard against .env before rebuilding, so a bad auth config aborts the deploy instead of shipping a crash-looping container. See Deploy.

Reaching it

# JWKS (over the docker network, or via an SSH tunnel to the host):
curl -fsS http://dataland-auth:9000/.well-known/jwks.json | jq  # (1)!

# Mint a token, then verify the agent accepts it:
TOKEN=$(curl -fsS -X POST http://dataland-auth:9000/api/auth/login \
  -H 'Content-Type: application/json' \
  -d '{"email":"curator@example.com","password":"secret"}' | jq -r .access)  # (2)!

curl -fsS http://dataland-auth:9000/api/auth/me -H "Authorization: Bearer $TOKEN" | jq  # (3)!

# Confirm the agent verifies the same token against its JWKS chain:
curl -fsS https://dataland.chat/v1/auth/me -H "Authorization: Bearer $TOKEN" | jq  # (4)!
  1. Hits the JWKS on the internal service URL. After the mirror is provisioned you should see the local signing kid plus dataland-rs256-1. This is the exact URL the agent uses for JWKS_URL.
  2. Extracts the access token (1 h TTL) from a login response. jq -r strips the quotes so $TOKEN is the raw JWT; the refresh token in the same response is ignored here.
  3. Verifies against auth's local public key only (verify_aud=False), looking the user up by sub. This is the auth server's own me, not the agent's.
  4. Exercises the agent's /v1/auth/me, which runs the full multi-JWKS chain. If this succeeds while the agent logs a FALLBACK warning, the mirror is missing the kid and should be re-provisioned.

State & secrets

State Where Notes
RSA private key + kid dataland_auth-data volume (auth_rsa_private.pem, auth_rsa_kid.txt) Back it up — it is the one irreplaceable secret. Optional AUTH_RSA_PRIVATE_PEM/AUTH_RSA_KID env override.
CMS-key mirror AUTH_EXTRA_JWKS_JSON env or data/extra_jwks.json kid dataland-rs256-1.
Accounts on-host dataland Postgres, table auth_server_users Reached via AUTH_DATABASE_URL (falls back to DATABASE_URL).

Operational notes & gotchas

Losing the RSA private key invalidates every issued token

The signing key is in the dataland_auth-data volume. Lose it with no restorable backup and every previously issued JWT fails verification and the agent re-fetches a brand-new kid. Treat any restore/replacement as a key-rotation event.

The FALLBACK JWKS provider warning means re-mirror the key

If the agent logs that a fallback provider validated a token, the served JWKS is missing that kid. Re-provision the mirror (AUTH_EXTRA_JWKS_JSON or data/extra_jwks.json) and restart auth so verification returns to the primary and the external CMS endpoint is back to pure-backup status.

Legacy passwords self-heal

Legacy sha256 rows keep working and upgrade to argon2id on the next successful login. No bulk migration is needed — just expect a brief, one-time UPDATE per legacy user on their first sign-in after the rollover.

See also

  • Agent — consumes these tokens; full multi-JWKS verification path and /v1/auth/me.
  • RAG — separate API_KEY bearer auth, unrelated to JWKS.