Skip to content

v0.87.0 — Image Manipulation, phases 5–6: object removal and separation

Click the boulder, get the boulder. The ask's headline capability, held since v0.72.0 on one decision: whether to fund a GPU segmentation service. SAM2_SERVICE_URL was set by no Terraform, no workflow and no compose file — the server-side path was referenced everywhere and existed nowhere.

D5 is answered, and the answer is not an instance. The deployment study — lost in v0.81.0's migration, restored and revised here with the option it had skipped — found that a hosted inference API beats every scale-to-zero GPU on cost, ops and cold start until ~800 sweeps a day, that Cloud Run's advertised 5-second cold start measures at two minutes, and that neither Bedrock nor Vertex has SAM 2 at all. So: the browser runs the click (SAM 2's decoder over a cached embedding, milliseconds), Replicate runs the sweep (Meta-official meta/sam-2, ~$0.021/run, sweep-only — it has no point prompts), the operator's own GPU is an upgrade rung, and a SegmentationProvider interface makes the tier an environment variable. The token is the user's first and the operator's second, like every other AI key.

Two things deliberately left out. VBL auto-detect stays blocked: SAM 2 has already proven unable to find walls reliably, and a hosted SAM 2 returns the same masks. And the no-WebGPU click fallback is deferred until the study's §8.1 is measured.

D4 and D7, answered the same day. User-level API keys are a platform feature, not genai's, and remove-background-openai already talks to the inpainting endpoint — so the fill is a platform route on the same key ladder, with the browser content-aware fill as the no-key tier and the user's preferred image model choosing (gpt-image-2 joins the list). And the fill is one call per tile, not one per object: the edits endpoint takes one mask with any number of holes. What D7 really asked was where 17 cutouts go, and the answer is one disposition for the batch, with the quota checked once for N before anything lands.

Order. Phase 5a (the provider) first, because it is the only part of Phase 5 verifiable against a real service today. Then 5.5, promoted from an optimisation to the spine; then 5.3; then Phase 6. 5.4 waits on D4, 5.6 on D7.

Shipped 2026-09-14. All 19 tasks, in that order, plus 5.3b — pick the object on the map itself, with a live outline under the cursor and the bare map while you do — added on 2026-09-11 after Phase 6. No module release. Measured along the way: WebGPU encode 733 ms, decode 16 ms; a 16×16 browser sweep finds 58 objects on a café map in ~10 s; the service's point prompt encodes in 742 ms on a CPU and every click after is 22 ms.

Phase 5a — the segmentation provider

  • [x] P.0 — The study comes back, and the decisions are written down. segmentation-deployment.md restored into the developer docs with three corrections and a new section; the roadmap page's task list restored with the study's deltas applied; D5, the VBL block and the no-WebGPU deferral recorded; ROADMAP.md gets the slot.
  • [x] P.1 — One interface, two implementations. SegmentationProvider with segmentAuto and isAvailable; Sam2Client as the service implementation; a factory over SEGMENTATION_PROVIDER; shared types gain 'hosted', provider and costUsd.
  • [x] P.2 — The Replicate implementation. POST /v1/predictions with Prefer: wait, the Files API above 1 MB, output URLs fetched before they expire, masks normalised to the service's shape. Recorded-fixture contract test.
  • [x] P.3 — POST /api/images/segment. Gated, third-party, user-first key, fundedBy, 501 with an actionable message when nothing is configured.
  • [x] P.4 — The user’s own token. replicate in Settings and on x-ai-keys; GET /api/ai/providers reports it.
  • [x] P.5 — Every place an operator learns the variable exists. Secrets Manager, ecs.tf, cloudrun.tf, .env.example, self-hosting, AI policy.

Phase 5 — object removal and separation

  • [x] 5.5 — The in-browser segmentation path. Encoder + decoder as two ONNX sessions, loaded the way libraryClassifier.ts loads; ORT's WASM served from our origin. Measured: WebGPU encode 733 ms warm, decode 16 ms.
  • [x] 5.5a — The embedding cache and its lifetime. Session memory, keyed by image identity, three at a time, LRU, in-flight deduplicated.
  • [x] 5.3 — The extract operation. extract-object with a point picker in the Retouch pane; the hole is the primary, the cutout a side effect.
  • [x] 5.1 — A point-prompt endpoint on the SAM 2 service (demoted).
  • [x] 5.4 — The fill. POST /api/images/inpaint on the existing key ladder, tiled, masked-region composite; browser pull–push fill as the no-key tier; a cutout's background is not a hole (holeMaskFromAlpha).
  • [x] 5.6 — Multi-object extraction. One sweep, N local cutouts, one tiled fill, batch-disposed through Phase 6. Needs Phase 6.

Phase 5b — pick on the map (added 2026-09-11, after Phase 6)

  • [x] 5.3b — Pick the object on the map. "Pick on map" from the Retouch picker: the modal hides, other layers are hidden client-locally, the map enters a pick mode on the pickSquares pattern with a live outline under the cursor (shader-thresholded 256² logits on a plane over the token, one shared embedding with the modal). Esc returns to the modal.

Also in this release

  • [x] Resize on Map is a modal, not a hover submenu. The v0.83.0 submenu opened on hover, over the menu, and made the items beneath it — Retouch, Move/Copy to Map — unreachable on the way to them. The menu entry now opens a modal the bar owns; one resize per opening.
  • [x] A System 8 feat's "About" shows the document, not the brief. The add-feat flow never persisted the wiki url (the add-power flow always had), and System 8's provider lists but does not detail, so the modal had nothing but the one-line brief. CharacterFeat.url is now written on add, and for the feats already stored without one the sheet finds the catalogue record by slug or name and follows its url — opened in the doc viewer, so links slide rather than navigate.

Phase 6 — side-effect disposition

  • [x] 6.1 — The disposition model. Pure (surface, kind, ctx) → destinations; ArtworkVariant.sideEffects[] in the shared type and both zod schemas.
  • [x] 6.2 — On /play: add to the map. Token modal creation, pre-filled, layer picked in the pane (objects by default for an object).
  • [x] 6.3 — While editing a character: add as characters or unlinked tokens.
  • [x] 6.4 — Save to the library, classified. Typed, not classified.
  • [x] 6.5 — Campaign art: library or discard.
  • [x] 6.6 — Discard is explicit. Held on the variant or in a persisted store until sent somewhere or discarded by a button.