Segmentation deployment — where SAM 2 actually runs

1. The question D5 actually asks

D5 is phrased as "do we run a GPU segmentation service in production?", and all four options it lists — stand one up, go browser-only, buy a third-party API, or let the operator bring their own GPU — assume segmentation is one workload. It is not. It is two, with latency budgets two orders of magnitude apart, and almost everything below follows from separating them.

There is a third workload the 2026-08 draft folded into the second, and the revision separates it because a vendor turned out to serve one and not the other:

The current service does only the sweep. It runs SAM2AutomaticMaskGenerator with points_per_side=32, which is 1,024 decoder invocations plus NMS — a deliberately expensive whole-image pass built for the VBL detector, where nobody is waiting. Task 5.1 already says a click needs SAM2ImagePredictor instead. What this document adds is that once you have split the predictor's two halves, the decode half does not need a server at all, and the encode half and the sweep are the only things D5 is really about.

2. What the clouds offer, measured against that

2a. First-party clouds

Neither Bedrock nor Vertex has SAM 2 as a managed API (re-checked 2026-09-10). Bedrock's 2026 catalogue has no segmentation model at all. Vertex Model Garden carries SAM v1 as a community notebook, not a first-class SAM 2 tile, so on GCP you build the container either way. AWS SageMaker JumpStart carries Meta SAM 2.1 first-party, with all three task modes including the point-prompt image predictor 5.1 needs — but JumpStart deploys to a real-time endpoint, an always-on GPU in the ~$1/hr class, which is the cost D5 is trying to avoid, so the catalogue convenience buys us little.

The finding that constrains everything: AWS has no Cloud Run

There is no AWS service that gives scale-to-zero GPU with a seconds-level cold start. Every AWS option is bound to an EC2 launch, so the wake-up is minutes.

GCP does have one: Cloud Run with an L4, GA, genuinely scale-to-zero, per-second billing (~$0.67/hr for the GPU in tier-1 regions, plus a mandatory 4 vCPU / 16 GiB floor that adds roughly another $0.45/hr while an instance is up — call it ~$1.10/hr all-in). Limits worth knowing: one GPU per instance, 8 vCPU / 32 GiB recommended, and only Iowa, N. Virginia, Belgium, Netherlands, Singapore and Mumbai.

Correction (2026-09-10). The 2026-08 draft repeated Google's "instances with pre-installed drivers start in ~5 seconds" figure as if it were the cold start. It is not. Independent measurement of a real model-serving container found a full cold start of ~105–120 s when no instance had been active for ten minutes, 6–7 s when one had been active within five, and ~1.5 s hot. The five seconds is the driver attach; the container pull and the model load sit on top of it. For our traffic shape — a user opens Retouch, clicks once, and the next user arrives an hour later — the number that matters is the two minutes, and that puts Cloud Run in the same job, not click class as AWS.

Vertex AI dedicated endpoints also do min_replica_count=0, but the request that arrives at zero replicas gets a 429 and is dropped while scale-up runs, which needs client retry logic.

The serverless GPU platforms

Sitting between the clouds and the model-API vendors: you bring a container, they scale it to zero. Same shape as Cloud Run, different cold-start engineering.

What that means

If segmentation must be a server call on our own instance, then the first call of a session is a 10–120 s wait everywhere, and on AWS it is minutes. That is defensible for the sweep and for the fill (§6) and indefensible for "click the boulder and see it highlight". Which is why the split in §1 matters — and why the next section exists.

2b. Third-party on-demand inference APIs (new, 2026-09-10)

This is the option the first draft skipped. The vendors below keep popular models warm across all their customers, so the per-request cold start is a few seconds rather than the minute-plus of a single-tenant instance, and there is no standing cost at all.

Two things this table settles that the first draft could not:

• Replicate is not a click API. The 2026-09-10 research summary initially described meta/sam-2 as accepting point and box prompts; its published schema shows it does not. Point-prompted SAM on Replicate exists only as unofficial community models, most of them SAM v1. Replicate answers the VBL detector and 5.6's sweep; it does not answer 5.1. fal.ai covers both halves under one key, which is the argument for it if one vendor is wanted.
• Per click, a hosted API is still the wrong shape for the primary path. Every call is a round trip, and at Replicate's rate the 27-boulder case done click-by-click on a server is $0.57 per map for something a cached embedding does for free in the browser. The split in §3 stands; the hosted API is the fallback rung, not the default.

2c. Standing cost and the break-even (new, 2026-09-10)

Once traffic keeps an instance warm, an L4-class GPU runs $430–600/month on-demand (Cloud Run is the expensive end because of the CPU/memory floor); a committed-use GCE g2-standard-4 or EC2 g6.xlarge is ~$300/month. Against that:

Nothing in this project's traffic is within an order of magnitude of either. The hosted API is cheaper than a warm instance until it is not, and the crossover is far enough away that it can be a numbers-driven decision later rather than a guess now.

3. The proposal: the decoder runs in the browser

SAM 2's image path decomposes cleanly, and the expensive half is not the half a click invokes.

  image ──► [ Hiera image encoder ]──► embedding ──┐
             ~1 forward pass, seconds              │
                                                   ├──► [ mask decoder ] ──► 3 candidate masks
  click ───► [ prompt encoder ]────────────────────┘     milliseconds, per click
             point_coords / point_labels

The embedding is computed once per image and then answers every subsequent click for free. The decoder is small enough to run on WASM, let alone WebGPU. So the interaction the ask describes — click, see the edges, click again to refine, pick between the three candidates multimask_output returns — is local and instant regardless of what we do or do not deploy.

Three tiers, and the honest trade between them

Tier B is less attractive than it first sounds, and this correction matters. The intuition is "encode on the server, decode in the browser, ship the small thing". But the embedding is not small. SAM 2's neck emits multi-scale features — at 1024×1024 input roughly 256²×32, 128²×64 and 64²×256 — which is on the order of 8 MB at fp16 and 17 MB at fp32, i.e. larger than the source image. Meanwhile Tier C ships back a mask, which is a single-channel PNG measured in kilobytes. No hosted API in §2b returns an embedding, so Tier B is only reachable from our own container.

So Tier B only wins where the embedding is amortised over many clicks on one image — which is precisely the ask's "27 boulders" case, and precisely not the "lift one character out of a group shot" case. It is a real option for 5.6 and a poor default.

(These byte counts are derived from the architecture, not measured. §8 lists them as something to measure before anyone builds against them.)

Tier C′ — the operator's own GPU

D5's fourth option. Architecturally this is Tier C with a different payer: the same POST /segment call, to a service the operator runs on their own machine, which is a mode this project already recognises — the Terms of Service permit personal self-hosting and Library overhaul already designs around the self-hosted operator.

It is further along than it looks. run-local.sh auto-detects CUDA through nvidia-smi and installs the matching PyTorch build; _resolve_device handles cuda / mps / cpu; and the service README already documents SAM2_SERVICE_URL=http://localhost:8080 under "Connecting to the Platform". Nothing needs building. What is missing is that nothing the platform ships mentions it.

Two things it buys that Tier A structurally cannot:

• A model the browser cannot download. Local runs hiera_large (~900 MB) where a browser realistically gets hiera_tiny. That is a quality ceiling, not a latency number, and it is the answer to §8.3 if the small-model quality turns out to matter.
• 5.6. The 27-boulder sweep is 1,024 decoder passes plus NMS. Minutes on CPU, seconds on a discrete GPU, awkward on WebGPU.

Two things that keep it from being an answer to D5 on its own:

• Hosted users cannot bring a GPU, and they are nearly all users. Tier C′ is an upgrade rung above Tier A, never a replacement for it.
• It is not uniformly faster. The service defaults to SAM2_DEVICE=cpu, and hiera_small on CPU is slower than WebGPU on the same machine's integrated GPU. An operator without CUDA who follows a "run it locally for better results" instruction makes their segmentation worse. Availability should prefer the browser when a configured service reports a CPU backend, and any documentation for this mode has to say so out loud.

Its marginal cost is near zero provided §9's 5.2 is built as a configured fallback rather than a hardcoded path — at which point local mode is one environment variable and no new code.

4. Why Tier A is cheap to build here

This is not a new capability so much as a fourth instance of a pattern this codebase has already run three times.

The genuinely new code is: an ONNX export of the encoder and decoder as two sessions, a tensor-plumbing layer between them, and the click-to-mask UI. There is prior art for the export and for the browser runtime — samexporter-style encoder/decoder splits, published sam2.1-hiera-tiny ONNX exports, and at least two working end-to-end WebGPU SAM 2 implementations — so the export is a task, not a research project.

5. The model-hosting rule this must not break

backgroundRemoval.ts once resolved its WASM from cdn.jsdelivr.net/…@latest and its weights from storage.googleapis.com/…/latest/ — unpinned, unsigned, third-party, on the critical path of a feature the user already clicked. Phase 0 existed to fix that. A browser segmentation path must not reintroduce it.

The rule, copied from libraryClassifier because it is already right:

• Weights live at a URL a deployment names (VITE_SEGMENT_MODEL_URL, or one URL per session), never a hard-coded CDN and never @latest.
• Nothing is bundled and nothing is fetched on page load — only on the first click that needs it, which is the only moment it is worth anything.
• The fetch goes through the Cache API so an ~80–150 MB download is paid once per browser rather than once per visit, and a cache failure falls through to a plain fetch rather than to no segmentation.
• The session is memoised as a promise, not a value.
• ORT's own WASM artefacts are served from our origin too, not from jsDelivr.

That last point is the one most easily missed: onnxruntime-web will happily fetch its .wasm from a CDN by default, which is the same mistake wearing a different hat.

The hosted-API rung has its own version of this rule: the image leaves our infrastructure. D3 already names "sends the user's image to someone else" as the cost of a third-party path, and the AI policy is where that has to be disclosed if a hosted segmentation provider is ever the default for anyone.

6. The fill still wants a GPU, and it is allowed to be slow

Tier A removes the interactive GPU requirement. It does not remove every GPU requirement, because the epic's scope boundaries already say the platform never ships a generative model, and task 5.4's tier-one fill is genai inpainting.

The saving grace is that the fill has a completely different latency profile. Nobody expects "make this floor look like floor" to return in 40 ms. So the fill is exactly the workload a queued job serves well: a hosted inpainting API, SageMaker Async Inference (min capacity 0, queue in, S3 out) or ECS Managed Instances on a g6f slice, woken per job, with the existing notification machinery telling the user when it lands. A two-minute cold start is acceptable for a job the user has been told is a job.

And 5.4's tier two — the browser-side content-aware fill — remains the reason the capability is not module-gated end to end.

7. If GCP compute is on the table after all

Recorded because it was asked, and because a premise here is commonly got wrong.

The GCS access blocker does not apply to a segmentation service. The migration is held up on serving private stored images out of GCS. A SAM 2 encode service is stateless — bytes in, embedding or mask out, nothing persisted — so it never creates a GCS object and never needs a signed URL for one. Being stuck in AWS for storage does not mean being stuck for compute. Whether adding a second cloud for one feature is worth two IAM models and two bills is a separate and entirely fair objection — and §2b makes the same objection to a second cloud that it makes to a first instance: a hosted API is a vendor, not an environment.

If it were done, there are two shapes:

• Client-as-proxy. Browser → Cloud Run → mask back → composite → the existing S3 presigned upload, with the storage path untouched. Four things it needs: the container verifies the caller's Clerk JWT against Clerk's JWKS (you cannot hand a browser a GCP invoker credential, and this distributes no new secret); hard per-user rate limiting, because the actual risk is an internet-exposed GPU any valid token can burn; OPTIONS handled in the container, since Cloud Run adds no CORS for you; and either end-to-end HTTP/2 or a client-side downscale, because HTTP/1 requests cap at 32 MB. Return a mask, not a composited image — kilobytes instead of megabytes.
• Server-to-server. Keep Cloud Run --no-allow-unauthenticated and have the ECS task role exchange a signed GetCallerIdentity for a short-lived GCP token via Workload Identity Federation, so no GCP service-account key ever lives in AWS. Cleaner posture; costs a double hop (S3 → our server → GCP → back) plus AWS egress.

The hybrid is better than either: the server authorises, the client transfers. Our API mints a short-lived scoped token; the browser moves the bytes.

8. What is still unknown

Nothing below should be treated as settled by this document, and the first three want measuring before Phase 5 commits to a tier.

1. Encode latency on real hardware. ✅ Measured 2026-09-10 (task 5.5, sam2.1_hiera_tiny samexporter export, Apple-silicon laptop): WebGPU encode 733 ms warm (1.75 s on the first run, shader compilation included), decode 16 ms warm (~600 ms the first call). WASM encode 4.9 s with 4 threads in Node, which a browser without cross-origin isolation runs single-threaded — so expect ~15 s there. The click is interactive on WebGPU and a wait on WASM; the no-WebGPU fallback question stays deferred, but it is a question about a fifteen-second wait, not about "no browser segmentation".
2. The embedding's real size. ✅ Measured: image_embed 256·64² + high_res_feats_0 32·256² + high_res_feats_1 64·128² = 16 MB fp32, as derived. Tier B stays dismissed.
3. The quality gap between hiera_tiny and the hiera_small the service already defaults to (SAM2_MODEL_CONFIG=configs/sam2.1/sam2.1_hiera_s.yaml), on our images — battle maps and character art, not SA-V. Single-image point-prompt quality is believed to degrade gently with encoder size, but "believed" is doing real work in that sentence, and D3's honesty obligation ("say when you didn't") applies if we ship the smaller one by default.
4. Where an embedding lives between clicks. Session memory only is the cheap answer and probably the right one for a first release. Storing it beside the artwork would make the 27-boulder case fast on a second visit, and would also be the largest blob this project has proposed storing — which is a D2-adjacent question that D2 did not consider.
5. Whether the VBL pipeline should follow. It has its own reasons to stay server-side (it runs headless, not in a tab). With a hosted sweep provider (§2b) it can stop falling back to GPT for want of a SAM2_SERVICE_URL — that is the cheapest single improvement in this document — but someone should decide whether VBL's masks are worth $0.021 a map before it is switched on by default.
6. Whether SAM 3's concept prompting replaces 5.6's grid sweep. (New.) "Every boulder" as a text prompt at $0.005 is cheaper and likely better than a 1,024-pass grid plus a mask-picker UI, but it is a different model with a different licence and it has not been tried on a battle map.

9. What this would change in Phase 5

Task deltas, if D5 lands on browser-first with a hosted fallback. Written here rather than applied to the task list, because D5 is not settled.

• 5.1 survives but is demoted. The point-prompt path in the service is still worth having — it is what a self-hoster's GPU and any headless caller use — but it stops being the thing the UI is built on. (Built 2026-09-13 as POST /segment/point on the service and POST /api/images/segment/point on the platform; the provider interface carries the click as an optional method, since Replicate's model cannot.)
• 5.2 gains a provider slot, and the slot is what makes every rung free. POST /api/images/segment becomes the fallback for browsers without WebGPU rather than the primary path. SAM2_SERVICE_URL stops being the only way to configure it: a SEGMENTATION_PROVIDER of none | service | replicate | fal chooses the backend behind one SegmentationProvider interface, and Sam2Client becomes one implementation of it. The same route then serves a hosted GPU, a hosted API and a self-hoster's own machine with no branch between them — and Sam2Client's circuit breaker already half-implements the reachability probe isAvailable needs. Written as "browser first; server when a provider is configured and reachable".
• 5.5 stops being an optimisation and becomes the spine. Its own text already anticipates this: "If D5 lands on 'no GPU service in production', this task is Phase 5, not an optimisation of it."
• A new task appears — the embedding cache and its lifetime (§8.4) — which does not exist in the current plan under any number.
• 5.6 is where the hosted sweep earns its keep. Replicate's meta/sam-2 is a drop-in for the existing automatic-mask endpoint, so 5.6's mask-set presentation can be built against a provider that exists today, with the VBL detector getting the same provider for free. It should be written to reuse one sweep across the whole mask set rather than re-running per object.

10. Recommendation

Tier A for the click; a hosted API — not an instance — for everything that is not a click; the operator's own GPU as an upgrade rung. (Revised 2026-09-10.)

• Do not stand up an instance. Nothing in §2 beats a hosted API on cost, ops or cold start until traffic is near ~800 sweeps a day (§2c), and every self-hosted option — Cloud Run included, per the correction in §2a — makes the first click of a session a 10–120 s wait. When the crossover is visible in the bill it is a provider-slot change, not a redesign.
• The per-click loop is browser-side (Tier A). No hosted option gives millisecond hover-and-highlight, and a hosted click is priced per click. This is the same rule D3 already settled on: browser-first, with the server offered on the result.
• Replicate meta/sam-2 is the sweep provider — for the VBL detector today and for 5.6 when it ships — because it is Meta-official, hiera_large, and exactly the endpoint the current service exposes. It is not a click provider. If a hosted click fallback is needed for no-WebGPU browsers, that is fal.ai sam2/image, and if one vendor is preferred over two, fal.ai covers both halves.
• Try SAM 3 before building the grid sweep UI for 5.6. Text-prompted "every boulder" is the cheapest possible implementation of batch extraction, and the half of it that is not platform code is a genai module release the roadmap already expects.
• Tier C′ rides along for free and should be offered rather than chosen: an operator with a 4090 sets one environment variable and gets a better model and a usable 5.6, and an operator without one is correctly left on the browser path.

That leaves D5 answerable as "no always-on GPU; a hosted API for the sweep and the fill; the browser for the click; the operator's own if they have one" — but §8.1 should be measured before the click half of that is written down as settled, because if WASM encode is unusable the hosted click fallback stops being optional.

11. Sources (2026-09-10)

• Replicate meta/sam-2 and its input/output schema
• fal.ai sam2/image, sam2/auto-segment, sam-3/image, fal GPU pricing
• Meta SAM 2.1 on SageMaker JumpStart
• Vertex Model Garden SAM v1 notebook
• Cloud Run GPU cold-start measurements, Cloud Run GPU pricing
• Modal pricing, RunPod pricing