Observability¶

Self-hosted metrics stack turned off (2026-07)

The Prometheus + Grafana + Alertmanager + exporters stack was removed from compose.yml and is not running. App-level tracing via Logfire is the only live observability. The metrics stack is retained in git history and can be reinstated if it is ever needed again.

Dataland's live observability is Logfire (OpenTelemetry) — trace-level forensics for a single visitor's chat turn, a single push, a single ingest. Every Python service ships spans and structured events to it, and distributed tracing stitches a chat turn that hops agent → rag → museum into one trace.

flowchart LR
  subgraph svc["Python services"]
    A[dataland-agent]
    AU[dataland-auth]
    R[dataland-rag]
    M[dataland-museum]
    NW[notification-worker]
    NA[notification-api]
    W[dataland-knowledge]
  end

  svc -- "OTLP spans + events" --> LF[(Logfire)]

Logfire (traces, AI spans, structured events)¶

Every Python service is instrumented through a small per-repo observability.py wrapper around the logfire SDK. The wrapper exposes a uniform internal API (configure_observability(), instrument_fastapi(), instrument_common_clients(), event(), span(), and on the agent set_attributes()), so the call sites look identical across repos even though each service tunes its own instrumentation.

Instrumented services and their service_name:

`service_name`	Repo / process	Notes
`dataland-agent`	dataland-agent (FastAPI, uvicorn `--workers ${UVICORN_WORKERS:-2}`)	pydantic-ai + google-genai + SQLAlchemy + httpx instrumented
`dataland-auth`	`auth_server.py` (agent image), on-Spark `auth` service	RS256 JWKS issuer
`dataland-rag`	dataland-rag	google-genai instrumented; AI content captured by default in prod, redacted
`dataland-museum`	dataland-museum (museum-api)	RDC bridge spans + `auth.login.*` events
`dataland-notification`	dataland-notification (both the worker and api processes report under this name)	redis tracing off by default
`dataland-knowledge`	dataland-knowledge	the Catalog Studio CMS
`dataland-simulator`	dataland-museum simulator (only when the `compose.sim.yml` overlay is on)	dev/load only

Enabling telemetry¶

Logfire is token-gated and fail-open. Set a write token in the service's .env:

LOGFIRE_TOKEN=your-logfire-write-token       # (1)!
LOGFIRE_ENVIRONMENT=production
LOGFIRE_SEND_TO_LOGFIRE=if-token-present      # (2)!
LOGFIRE_SYSTEM_METRICS=true                   # (3)!
LOGFIRE_CAPTURE_AI_CONTENT=false              # (4)!

The write token is the master switch. With it empty, every service still boots normally and simply never exports spans — Logfire is token-gated and fail-open.
Accepts true / false / if-token-present. The default if-token-present ships spans only when LOGFIRE_TOKEN is set; true errors if there is no token, false never ships.
Per-process CPU / mem / GC gauges. Enabled on agent, rag, museum and notification.
Attaches prompts + completions to spans (also sets OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT). Keep false on agent / museum / knowledge in prod. RAG defaults this on in prod; the auto-instrumented google-genai capture path bypasses RAG's redact() scrubber, so never put secrets in prompts.

LOGFIRE_SEND_TO_LOGFIRE is parsed by every service's _send_to_logfire():

true / 1 / yes / on → always ship (errors if no token).
false / 0 / no / off → never ship.
anything else (the default if-token-present) → ship only when LOGFIRE_TOKEN is set.

Services start with no token

With an empty LOGFIRE_TOKEN, every service still boots normally; it just doesn't export spans. configure_observability() is idempotent and guarded by a module-level _CONFIGURED flag, so re-importing is safe. event() / span() never raise — span() falls back to a nullcontext() on any error, so observability can never break the request path.

AI content capture leaks prompts

LOGFIRE_CAPTURE_AI_CONTENT controls whether prompts and completions are attached to spans (it also sets OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT).

agent / museum / knowledge: default false. Keep it off in production.
dataland-rag: defaults to on in production so a mis-captioned image has its Gemini prompt + response in the trace. RAG runs every explicit attribute through structured_log.redact() first, scrubbing Bearer … / Basic … tokens and secret-shaped keys. The auto-instrumented google-genai capture path bypasses that scrubber, so do not put secrets in prompts. Flip it off explicitly with LOGFIRE_CAPTURE_AI_CONTENT=false in any environment where prompt content is sensitive.

What gets auto-instrumented¶

instrument_common_clients() wires the relevant integrations per service:

Integration	agent	rag	museum	notification
FastAPI (request spans)	✅	✅	✅	✅
httpx (outbound HTTP)	✅	✅	✅	✅
pydantic-ai (LLM run spans)	✅	—	—	—
google-genai (Gemini calls)	✅	✅	—	—
SQLAlchemy (query spans)	✅	—	—	—
system metrics (`LOGFIRE_SYSTEM_METRICS`)	✅	✅	✅	✅
Redis	opt-in	—	—	opt-in

Redis tracing is off by default on both the agent and the notification worker because the Redis traffic (session/ticket-state lookups on the agent; XREADGROUP / XACK / INCRBY loops on the notification consumer) is high-volume plumbing with no business signal. The interesting spans are the FastAPI request, the SQLAlchemy queries, the pydantic-ai run, and the hand-written notification.process_telemetry span. Set LOGFIRE_INSTRUMENT_REDIS=true for short-window debugging only.

Enrich the HTTP span instead of nesting

The agent's set_attributes(...) (used in routers/chat.py, routers/conversations.py, middleware.py) attaches domain fields like ticket.id, conversation.id, chat.mode onto the active auto-instrumented HTTP span. Live-view queries can then filter on these without drilling into child spans. It no-ops silently when no span is recording.

Useful filters¶

service_name = 'dataland-agent'
service_name = 'dataland-notification'
tags contains 'dataland'

Every hand-emitted event() / span() is tagged dataland (plus a per-service tag like notification or rag), so tags contains 'dataland' isolates the curated events from the firehose of auto-instrumented spans.

Key event and span names¶

These are the actual names emitted by the code (obs.event(...) / obs.span(...)), grouped by flow. Distributed tracing is on (distributed_tracing=True), so a chat turn that hops agent → rag → museum stitches into one trace.

Flow	Events / spans (real names)
Agent lifecycle	`agent.startup`, `agent.shutdown`
Agent service resolver	`service.ticket.resolve` (span), `service.ticket.resolved`, `service.ticket.resolve_not_found`
Museum lifecycle	`museum.startup`, `museum.shutdown`, `museum.rdc.subscriber`, `museum.telemetry.publisher`
Museum ticket→user	`museum.ticket_user.resolve` (span), `museum.ticket_user.resolved`
Museum auth	`auth.login.success`, `auth.login.failed`, `auth.login.rate_limited`
RAG search	`rag.search` (span), `rag.search.completed`, `rag.images.search_text` / `rag.images.search_text_completed`, `rag.images.search_image` / `rag.images.search_image_completed`
RAG ingest	`rag.ingest.file` / `rag.ingest.file_completed` / `rag.ingest.file_failed`, `rag.ingest.image` / `rag.ingest.image_completed`, `rag.ingest.image_rate_limited`, `rag.ingest.image_caption_quota_exhausted`, `rag.ingest.sync` / `rag.ingest.sync_completed`
Notification telemetry	`notification.process_telemetry` (span), `notification.telemetry.processed`, `notification.telemetry.skipped_missing_ticket`, `notification.telemetry.skipped_disabled`
Notification rules	`notification.rule.triggered`, `notification.rule.rate_limited`, `notification.rule.skipped_missing_required`, `notification.rules.reloaded` / `notification.rules.reload_failed`
Notification push	`notification.push.send` (span), `notification.push.sent`, `notification.push.failed`, `notification.push.missing_recipient`, `notification.push.skipped_credentials`
Notification welcome	`notification.welcome.sent`, `notification.welcome.unresolved`
Notification ↔ agent	`notification.agent.resolve_ticket` (span), `notification.agent.ticket_resolved`, `notification.agent.ticket_not_found`, `notification.agent.open_chat`, `notification.agent.chat_opened`, `notification.agent.chat_wall_clock_timeout`
Ops alerts	`notification.ops_alert.send` (span), `notification.ops_alert.sent`, `notification.ops_alert.failed`, `notification.ops_alert.multi_subfailure`, `notification.ops_alert.skipped_no_url`, `notification.complaint.notifier_disabled`

Key operational signals¶

A few data-plane signals worth watching even without a metrics scraper:

Notification DLQ depth — the canonical "rules are silently failing" signal. dataland-notification-api:8080/metrics renders the notification_dlq_depth gauge straight off Redis; entries land in museum:telemetry:dlq. Inspect via the notification-api DLQ surface and replay once the root cause is fixed.
Museum bridge freshness — museum-api's /api/bridge/metrics reports seconds-since-last-published. If the RDC bridge goes silent, telemetry stops reaching museum:telemetry even though docker ps stays green — this is the data-plane health check the green dot can't give you.
Fallback-JWKS WARN — a log signal, not a metric. The agent logs JWT accepted by FALLBACK JWKS provider … local JWKS is missing this signing key; chat auth depends on this external endpoint whenever a token is validated only by a non-primary JWKS provider. It means the auth JWKS is missing the CMS signing key and chat auth has reverted to a single point of failure — re-provision the AUTH_EXTRA_JWKS_JSON mirror and recreate auth. Grep the agent's structured log / Logfire for FALLBACK JWKS.

Structured JSONL logs¶

Alongside the human-readable console/text logs, each service writes a redacted, structured JSONL stream (structured_log.py, public API setup_structured_logging(service=...), log_event(...), redact(...)):

One JSON object per line.
Daily rotation at UTC midnight, gzip on rotation, 30-day retention.
<LOG_DIR>/<service>-YYYY-MM-DD.jsonl (≥ INFO) and <service>-error-YYYY-MM-DD.jsonl (≥ WARNING).
LOG_DIR defaults to <repo>/logs, overridden by LOG_FILE_DIR so containers mount it at /app/logs.
Mandatory redaction: keys matching secret|token|password|hash|salt|cookie|authorization|api[-_]?key|bearer|jwt|access[-_]?key|client[-_]?secret are replaced with ***REDACTED*** recursively, and inline Bearer … tokens inside string values are scrubbed before write.

This is the durable, on-disk forensic record when Logfire isn't shipping (no token) or when you're SSH'd into the box debugging an incident.

Health checks vs. depth checks¶

Every service has a docker healthcheck — docker compose ps shows status.

A green dot is a liveness probe, not a depth probe

museum-api's /health returns 200 even when its RDC subscriber is firing connection-timeout errors every few seconds. The notification api's /health only PINGs Redis. For real data-plane health, trust XLEN museum:telemetry, the /api/bridge/metrics freshness gauge, the notification DLQ depth, and log tails — not the green dot in docker ps.