AI Architecture
In short, we are in-browser first wherever possible. For one, it's faster (for certain tasks), it's private and secure, and it doesn't require paid hardware or API access. However, we also want to allow users to use external models if they want to.
Where each task runs
Corrected in v0.119.1: this table used to say sheet parsing ran on an in-browser Qwen 1.5B and wall detection in the browser. Neither was true.
| Task | Model(s) | Runs |
|---|---|---|
| Tokenization | TensorFlow.js BlazeFace, vendored in apps/web/public/models/blazeface |
Browser |
| Portrait Background Removal | MediaPipe SelfieSegmenter, vendored in apps/web/public/models/mediapipe |
Browser |
| Background Removal (any subject) | BiRefNet_lite (@huggingface/transformers) |
Server |
| Background Removal (OpenAI / Gemini) | Image edit models (/api/images/remove-background-openai, -gemini) |
Provider; user key, else operator's |
| Object removal — the click | SAM 2.1 hiera_tiny, ONNX, from VITE_SEGMENT_MODEL_URL |
Browser |
| Object removal — the sweep | Replicate meta/sam-2, or SAM2_SERVICE_URL |
Provider; user token, else operator's |
| AI fill | OpenAI images/edits / Gemini image (/api/images/inpaint, server-core/src/inpaint) |
Provider; user key, else operator's |
| Character Sheet Parsing | Server LLM (/api/ai/parse-character*), OpenAI by default |
Provider; user key, else operator's |
| VBL Auto-Detect | SAM 2 contours plus OpenAI vision/chat (/api/ai/detect-vbl*) |
Server + provider |
| Library tagging | Ollama llava, else Gemini / Anthropic / OpenAI (workers/imageClassifier.ts) |
Server + provider |
| Library CLIP score | CLIP ONNX from VITE_LIBRARY_CLIP_MODEL_URL (off when unset) |
Browser |
| Monster / Encounter Generation | Qwen 2.5 3B via @mlc-ai/web-llm by default (weights from huggingface.co, runtime from raw.githubusercontent.com); or the server LLM |
Browser, or provider |
| Token name suggestion | Server LLM (/api/ai/suggest-token-name) |
Provider; user key, else operator's |
| Map generator | Server LLM for the layout; genai module (gpt-image-1) for missing assets | Provider; user key, else operator's |
"User key, else operator's" is resolveLLMConfig / resolveImageProviderKey:
the x-ai-* headers win, and the server's env key is the fallback.
API Support
We will support the following API-based providers:
- OpenAI
- Gemini
- Claude
- Grok
Users provide their own API keys for these providers, and set their preferences for which AI task uses which model. The UI also allows model switching at the time of execution.
Server Configuration
For users running their own server, they can enter API Keys to enable the model that key uses.
For users on hosted servers, they will configure their api keys on a per-user basis. This functionality still exists when the api keys are entered on the server, where those keys serve as the default for all users, but each user can override with their own keys if they want to use their own accounts for billing or access reasons.
UI
In the Settings page (route /settings), there will be an "AI Configuration" section, where users can set their preferences for which models to use for which tasks, and enter any necessary API keys. There will also be a link to documentation about how to set up and use AI features.
- If the server has environment variables set for any API keys, the user will be informed that these models are already enabled, and that they can enter their own keys to use their own accounts if they want to. It will not show the exact API keys that are set on the server for security reasons.
In any UI element that uses AI, there will be several UI elements:
- If no AI is enabled for this flow, a notification will explain that, with a link to Settings to change it
- If an in-browser model is enabled but not downloaded, a notification will prompt them to download the model, which should cache it for future use. Without the model, the UI should not attempt to perform in-browser AI tasks.
- If multiple options are enabled for this flow, the UI will show which option is currently active, and allow the user to switch between options on the fly.