xray

Appearance

xray architecture ​

This doc is the map for anyone contributing to xray. Read it to learn four things:

  • The three processes that make up the system.
  • The two write paths that put data into storage.
  • The read path that feeds the inspector (the web UI).
  • The trust boundary that keeps those paths apart.

End-user integration instructions live in integrate.md.

A few terms show up a lot. Here they are in plain words:

  • Replay: one run of one test conversation against a real voice agent.
  • Turn: one back-and-forth step in a conversation. Either the user speaks or the agent speaks.
  • Span: one timed event from the agent's code, in OpenTelemetry format. For example, "the LLM call took 800ms."
  • OTLP: OpenTelemetry Protocol. The wire format agents use to send spans.
  • VAD: Voice Activity Detection. It scans audio and marks where someone is actually speaking.

TL;DR ​

  • Three independent processes: the driver (test side, Python), the agent worker (dev's code, Python), and xray itself (a single Bun process serving SPA + HTTP API + OTLP receiver + a background job worker).
  • xray has exactly two write surfaces: the SDK control plane (the driver POSTs Conversations / Replays here, the only trusted source for those rows) and the OTLP/HTTP receiver (both sides emit spans here; routed by xray.replay.id, filtered by vocabulary).
  • Server-side analysis. The driver uploads a 48kHz int16 stereo WAV (left = user, right = agent) on completion. The server runs per-channel VAD, derives turn boundaries from the segments, and writes speech_segments + replay_turns rows. The driver waits via SSE on /v1/replays/:id/events.
  • Storage is one SQLite file at /data/xray.db plus the bunqueue job DB at /data/bunqueue.db, plus audio bytes on disk under XRAY_AUDIO_ROOT. No external services. No second container. See single-image-distribution.md for why this is non-negotiable.
  • The inspector SPA is served by the same Bun process that owns the API: one image, one port, one volume.

The three processes ​

Why three processes ​

Each process has one job. Here is what each one does.

The driver runs in CI or on the dev's laptop. It owns the test work:

  • It holds the test spec (the conversation you wrote in code).
  • It plays the user audio into the room.
  • It captures the agent audio coming back.
  • It writes the stereo WAV (one audio file, user on the left channel, agent on the right).
  • It uploads that WAV, then waits via SSE for the server to finish VAD and turn derivation.

The driver is also the only thing that mints LiveKit JWTs. (A JWT is a signed login token. LiveKit is the real-time audio service the agent runs in.) Each JWT carries the xray attribute: replay_id, conversation_hash, and modality. That JWT is how the agent side learns which replay it is inside.

The agent worker is the dev's own LiveKit Agents code. It has one thin xray wrapper: async with xray.attach(ctx, …).

  • It runs the same way it would in production. In production, no xray attribute is on the JWT, so attach does nothing.
  • Its only job, from xray's point of view, is to emit OTEL spans.

xray is the single Bun image. It takes both inputs and renders the inspector. The analyze-replay job runs in-process via bunqueue in embedded mode. No second container. No Redis. No separate worker process.

The driver and the agent worker never talk to each other directly. They share state through two channels:

  1. The LiveKit room (audio plus the JWT attribute).
  2. xray itself (every span lands under the same xray.replay.id).

The two write paths ​

xray has exactly two write surfaces. Every byte that changes state in /data/xray.db arrives through one of them.

They are coupled by trust. The OTLP receiver never creates Conversation or Replay rows. That is exclusively the SDK control plane's job.

Control plane (driver only) ​

sdk/python/src/xray/orchestrator.py:run(...) POSTs to these endpoints in order:

  1. POST /v1/conversations. This is a Valibot-validated upsert, keyed by hash. The body is multipart. It has a spec JSON part with name plus turns (and optional judges / live). It also has one named file part per RecordedAudio turn, keyed by the turn's declared upload_key. Here is what the server does with it:

    • It reads each audio part and sha256s the bytes.
    • It substitutes that hash into the canonical turn.
    • It then hashes the canonical spec JSON ({turns, judges}) to derive conversation_hash.

    So changing a judge forks a new Conversation. Re-POSTing the same hash with a different name updates the row's display label (last-write-wins). The SDK never hashes anything.

  2. POST /v1/replays. This creates the Replay row eagerly at lifecycle_state='pending' and returns replay_id. This must happen before the runtime emits its first span. Otherwise the OTLP receiver would drop those spans as "unknown replay_id." The body is {conversation_hash, run_config?, run_config_name?}. When a run_config is present, the server hashes its canonical JSON, upserts the matching run_configs row (applying run_config_name as a label, last-write-wins) and stamps replays.run_config_hash — all in the same transaction as the row insert. run_config_name is a sibling of run_config, not a key inside it, so a label can never enter the identity hash. A replay sent without a run_config belongs to no group. An emptyrun_config is rejected with a 400: since the label is not hashed, every name-only config would otherwise land in one group whose label flips last-write-wins.

  3. POST /v1/replays/:id/audio. This uploads the stereo WAV (left = user, right = agent, wall-clock-aligned, written under XRAY_AUDIO_ROOT/<replay_id>/replay.<ext>). The server flips lifecycle_state to recording_uploaded.

  4. POST /v1/replays/:id/analyze. This enqueues the bunqueue analyze-replay job. The server transitions to lifecycle_state='analyzing' with analysis_step='vad'. It returns 202 Accepted with the bunqueue job id.

  5. GET /v1/replays/:id/events (SSE). The SDK streams state, progress, evaluation_complete, and failed events. The evaluation_complete payload carries the full ReplayResult (passed/failed verdict + per-assertion + per-judge + per-turn metrics). So the SDK can return immediately without a follow-up GET. A heartbeat : line every 15s keeps proxies from idling out. The SDK closes the stream when lifecycle_state hits a terminal value.

  6. GET /v1/replays/:id/result. This is the same ReplayResult payload outside the SSE stream, for late subscribers and inspector hydration.

  7. PATCH /v1/replays/:id. The SDK uses this only for driver-side failures (failure_reason='driver_aborted' / audio_missing / agent_not_joined). Lifecycle transitions during the analyze chain are server-owned.

OTLP receiver (both sides) ​

src/server/otlp/otlp.service.ts accepts both application/json and application/x-protobuf. It normalises them to a JSON-shape that the Valibot schema validates. Then it dispatches each span through the vocabulary registry (src/server/otlp/vocabularies/registry.ts).

Each registered vocabulary is one file. To add a new one (for example, a provider-specific semconv), drop a file in vocabularies/ plus one line in registry.ts.

The receiver is a filter, not a gate. Two kinds of input get dropped:

  • Unknown vocabulary is silently dropped. So an agent worker emitting noisy framework spans doesn't pollute storage.
  • An unknown xray.replay.id is silently dropped. So an agent running in production, where there is no replay context, doesn't write rows.

xray vocabulary (src/server/otlp/vocabularies/xray.ts). These are the recognized span names: xray.turn, xray.stage.stt, xray.stage.tts. They land in the raw spans table for the inspector's timeline but produce no structured rows. Turn boundaries come from server-side VAD, and assertion plus judge outcomes come from the server's evaluate-replay job walking the declared catalog. xray.assertion and xray.judge are no longer recognized. The spec-0001 server reads its checks from the Assertion / Judge variants declared on the conversation, not from driver-emitted spans.

Tool / model → turn attribution is timestamp-based, not span-tag based. It is also derived, not stored. tool_calls / model_usage rows carry only their wall-clock started_at. Turn membership is computed at eval/read time. The server maps started_at onto the audio timeline (audio_offset_ms = started_at − replays.recording_started_at, the anchor the driver sends via the X-Recording-Started-At upload header). Then it tests that offset against the VAD-derived turn window [turn_start_ms, turn_end_ms). There is no turn_idx column on those tables and no backfill stage. The origin is always replays.recording_started_at (the audio sample-0 wall-clock). replays.started_at (row-creation time, which precedes the recording by the room-connect + agent-join latency) must never be used.

gen_ai semconv (gen-ai-semconv.ts). This dispatches on gen_ai.operation.name: execute_tool → tool_calls; chat / text_completion → model_usage (model TTFT lifted from gen_ai.response.time_to_first_chunk, seconds → ms). Langfuse vocabulary (langfuse.ts) extracts the same shapes from Langfuse observations: generation → model_usage, tool → tool_calls. See wire-contract.md for the full attribute contract.

Replay lifecycle (single replay, time order) ​

Two things to notice in this diagram.

First, the audio plane (LiveKit) and the observability plane (OTLP) are separate. Audio never goes through xray during the run. xray just receives the post-hoc stereo WAV at the end. The agent worker's STT is the dev's STT. xray sees only its emitted OTEL spans.

Second, the replay row is created before any spans land. This is what makes the OTLP receiver's "unknown replay_id → drop" rule safe. By the time the agent worker emits its first span, the Replay row already exists. So the receiver routes the span correctly.

Storage ​

replay_turns is the join point between the spec (conversations.turns_json) and the observed execution. The rows are written by the analyze-replay worker after it runs VAD on each channel of the uploaded stereo WAV.

speech_segments carries the raw VAD output: one row per detected voiced chunk per channel. The inspector renders these alongside the turn boundaries. That helps you debug overlap, silence, and latency.

Turn boundaries are not simply "the speaker changed". A pause inside one speaker's audio splits it into two turns only when the other side took the floor across that pause — otherwise the audio after the pause is the same utterance continuing. Without that rule, an agent that resumed talking at the moment a user cut in had its own tail filed as a separate turn, which put the barge-in on a turn nobody was asserting about (issue #126).

tool_calls, model_usage, and spans are written by the OTLP receiver as it ingests gen_ai.* / Langfuse / xray.* spans.

Read path: what the inspector sees ​

The inspector (src/client/inspector/ + slice folders under src/client/) is a React SPA. Bun's HTML bundler builds it, and the same Bun process that owns the API serves it. There is no client-side build step in CI. Bun builds it at request time and at container start.

Every read endpoint lives in src/server/<slice>/<slice>.router.ts. The service layer (<slice>.service.ts) does the actual SQL via Drizzle on bun:sqlite. The slice convention is documented in code-layout.md.

Distribution ​

The shipped artifact is a Docker image. CI publishes it to GHCR (ghcr.io/xray-eval/xray) on tagged releases. Operators run

docker run -v ./data:/data ghcr.io/xray-eval/xray

(XRAY_AUDIO_ROOT defaults to <XRAY_DATA_DIR>/audio, so it needn't be passed.)

That is the whole install. The image carries the Bun process, the pre-built SPA, the SQLite schema (migrated at startup), and the bunqueue worker (embedded, same process). Nothing else. No SaaS. No hosted version. No second container.

This single-image promise is load-bearing for several other choices in the codebase. SQLite over Postgres. bun:sqlite over a network driver. Embedded reads over a separate query service. Embedded bunqueue worker over a separate queue process. See single-image-distribution.md before proposing any change that would break it.

Two SQLite files in /data/. xray owns xray.db (conversations, replays, and so on). bunqueue owns bunqueue.db (jobs, DLQ). This is an acknowledged tradeoff against the "one file" reading of the rule: single volume, two files, no second process. The operator backs up the whole /data volume. The path is configurable via BUNQUEUE_DATA_PATH.