Skip to content

AI / GenAI API 0.1 — De Facto Specification

Status: Descriptive. Documents the AI/LLM/image-gen surface as it ships today (retroactively numbered 0.1; unversioned). Written from production source on branch version/0.40.0. Quirks flagged ⚠︎. Server surface lives in routes/ai.ts (mounted /api/ai, index.ts:334); all routes require requireAuthWithImpersonation.

1. Provider model

Four LLM providers, hardcoded (ai.ts:9-30): openai (GPT-4o), gemini (2.5 Flash), claude (Sonnet 4), grok (Grok 3 Mini), each with an env key (OPENAI_API_KEY, GEMINI_API_KEY, CLAUDE_API_KEY, GROK_API_KEY) and an OpenAI-compatible chat-completions endpoint.

Per-request selection via resolveLLMConfig(req) (ai.ts:38-72) reading headers:

Header Meaning
x-ai-provider which provider (default openai)
x-ai-key user-supplied key — overrides the server env key
x-ai-model model id override

Priority: user header key > server env key; falls back to the first server-configured provider. Client helpers: aiHeaders.ts (getAIHeaders, pickImageGenProviderHeaders), config store aiConfigStore.ts (Zustand-persisted; user keys never sent to server except as request headers).

2. Endpoints

Method Path Purpose Line
GET /api/ai/providers list env-configured providers :78
GET /api/ai/models/:provider list models (24 h cache) :275
POST /api/ai/models/refresh force model-cache refresh :229
GET /api/ai/models/cache-status cache freshness (debug) :255
GET /api/ai/image-models[/:provider] image-gen models across/one provider :357/:372
POST /api/ai/parse-character-sheet spreadsheet/PDF → token metadata + actions :406
POST /api/ai/parse-character spreadsheet/PDF → full Character :523
POST /api/ai/generate-monster prompt → 5e stat block :926
POST /api/ai/generate-encounter prompt + totalCR → balanced encounter :1107
POST /api/optimize-character character → build suggestions (score + tips) :1233
POST /api/ai/detect-vbl legacy: GPT-4 Vision → VBL shapes :718
POST /api/ai/detect-vbl-pipeline 3-stage VBL pipeline (§5) :1368

3. Image generation

ImageGenProvider = 'openai' | 'grok' | 'gemini' (claude has no image models). Models are hand-curated in the IMAGE_MODELS constant (ai.ts:314-334) — image APIs lack reliable list-models endpoints, so this constant is the single source of truth that modules (e.g. genai) fetch. User-level image keys are injected via resolveImageProviderKey (ai.ts:343-350) and, client-side, the useImageGenAuthFetch hook (useImageGenAuthFetch.ts) patches /api/ai/image-models* requests with the user's key.

⚠︎ Singular-header limitation (aiHeaders.ts:54-59): x-ai-provider/x-ai-key are singular, so one request unlocks only one user-level provider; with multiple user image keys, the first by IMAGE_GEN_PROVIDERS order wins and the rest stay hidden. (Matches the genai-image-models-user-key-gap known issue.)

4. GenAI sibling service

Image synthesis is delegated to the genai module (its own sibling repo), called at POST /api/modules/genai/generate, forwarding x-ai-key/x-ai-provider. Request {prompt, modelId, provider} → response {imageUrl}; the platform then downloads the URL through its own storage pipeline. This rides the Module API router mount, not a bespoke transport.

5. Sub-pipelines with their own type contracts

  • VBL detection pipeline — types vblPipeline.ts (Sam2*, PrimitiveCandidate, VBLFinal), orchestrator pipeline/orchestrator.ts. Three stages: SAM2 segmentation (external GPU service, env SAM2_SERVICE_URL
    • circuit breaker) → contour/geometry extraction (CPU) → GPT cleanup (optional, graceful fallback). Mode auto-detects from SAM2_SERVICE_URL (sam2_contour_only) else pure_gpt; override via VBL_PIPELINE_MODE.

6. Known gaps

  1. Unversioned — no version negotiation on any AI endpoint; providers/models hardcoded.
  2. Singular x-ai-* header caps user-level providers at one per request (§3).
  3. IMAGE_MODELS is hand-maintained — drifts as vendors add/deprecate models (note the DALL·E-deprecation comment at ai.ts:315-318).
  4. SAM2 is an external dependency — dev/self-hosted-gated; absence falls back to pure-GPT VBL.