Library image classification

Model in use

The library image classifier (imageClassifier.ts) uses Ollama with llava by default (overridable via OLLAMA_VISION_MODEL). It also has a fallback path that calls OpenAI gpt-4o-mini (detail: "low", image pre-resized to 512px) — see imageClassifier.ts and imageClassifier.ts. There is no Gemini path implemented yet.

Capacity estimates per provider

These are the practical ceilings per user/API key, assuming one image = one classification call.

OpenAI gpt-4o-mini (current fallback)

• Tokens per image: "low" detail on gpt-4o-mini ≈ ~2,833 image tokens (the mini scales image tokens ~33× vs gpt-4o's 85) + ~200 prompt + ~100 output ≈ ~3,200 tokens/image.
• Cost: ≈ $0.0005 per image (~$0.50 per 1,000 images, ~$5 per 10K).
• Rate limits (Tier 1, the default for a new paid key):
  • 500 RPM → ~30K images/hour theoretical
  • 200K TPM → ~60 images/minute sustained → ~3,600/hour
  • 10K RPD → ~10,000 images/day hard cap
• Tier 5 (high spend): 30K RPM, 150M TPM, no RPD cap → effectively hundreds of thousands per day.
• Free trial / no payment: not usable for vision at meaningful scale.

OpenAI gpt-4o (if you wanted higher quality)

• ~85 image tokens (low detail), but ~17× more expensive per token than mini → ~$0.002/image.
• Same RPM/TPM tiers; 10K images/day at Tier 1 = ~$20/day.

Google Gemini (e.g. gemini-2.0-flash / 1.5-flash)

• Tokens per image: ~258 for ≤384px, ~258 per tile above that → very cheap.
• Free tier: 15 RPM, 1,500 RPD, 1M TPD → ~1,500 images/day free.
• Paid tier: 2,000 RPM, no RPD; cost ≈ $0.00002–$0.0001/image → tens of thousands/day trivially, hundreds of thousands within rate limits. Much cheaper than OpenAI.

Anthropic Claude (e.g. claude-3.5-haiku / sonnet)

• Image tokens scale by area; a 512×512 image ≈ ~600 tokens.
• Haiku: ~$0.0005/image; Sonnet: ~$0.005/image.
• Tier 1: 50 RPM, 50K input TPM → ~3K–4K images/hour, no hard daily cap → realistically tens of thousands/day on Haiku.

Rough summary

For most self-hosters importing a personal asset library (a few hundred to a few thousand items), any of these providers handle it comfortably in one session — the binding constraint is usually OpenAI's 10K/day RPD on Tier 1, not token cost. Gemini Flash is the cheapest by ~10× if you want to add a third provider path.

Server load per image

For the flow cloud storage → your API server → vision provider, here's the breakdown per image:

Memory & CPU

• sharp resize/recompress is the dominant cost. For typical library images (2–8 MB JPEGs/PNGs decoded to 20–100 MB raw RGBA in memory):
  • ~50–200 ms CPU on a modest server core
  • Peak RSS spike of ~30–150 MB per concurrent image (sharp uses libvips, mostly streaming, but decoded buffers are held)
• Base64 encode of the resized 512px output (~30–80 KB) → trivial, sub-millisecond.
• JSON parsing of the API response → negligible.

Network

• Inbound from Dropbox/Drive: full original size (1–10 MB typical, up to 50 MB+ for high-res maps). This is the biggest bandwidth cost and you pay egress on the cloud-storage side (Drive/Dropbox are free, S3 isn't).
• Outbound to OpenAI/Gemini/Anthropic: the resized ~30–80 KB image as base64 (~40–110 KB on the wire). Already what imageClassifier.ts does for OpenAI (imageClassifier.ts).
• Wall-clock per image: dominated by the vision API itself — typically 2–8 seconds for gpt-4o-mini, 1–3 s for Gemini Flash, 2–5 s for Claude Haiku. The image fetch + resize is ~200–500 ms on top.

Concurrency math

On a small VPS (2 vCPU, 2 GB RAM) you could comfortably run 4–8 concurrent classifications without thrashing — the bottleneck is RAM during sharp decode of large source images, not CPU (most time is spent awaiting the API). At ~5 s/image and 6 concurrent, that's ~4,300 images/hour per server — well under any provider's rate limits.

Does the whole image have to proxy through?

Yes, in practice — and you want it that way. Three reasons:

1. Pre-processing is mandatory for cost control. A 4000×3000 battle map sent at full size to gpt-4o-mini with detail: "high" is ~25K image tokens (~$0.004 each) vs. ~2.8K tokens (~$0.0005) at detail: "low" after a 512px resize. ~8× cost reduction. The current code already does this. Same logic applies to Gemini (per-tile billing above 384px) and Claude (area-based).
2. Format normalization. Cloud storage may hand you HEIC, WEBP, TIFF, animated GIF, 16-bit PNG, CMYK JPEG, etc. Provider APIs accept a narrow set (JPEG/PNG/WebP/GIF). sharp normalizes this for free.
3. No provider supports "fetch from URL" universally. OpenAI accepts image_url with a public HTTPS URL — but cloud-storage URLs are usually short-lived signed URLs or require OAuth, so OpenAI's fetcher can't reach them. Gemini and Anthropic require inline base64 anyway. So you'd be doing the proxy regardless.

One small optimization worth knowing

You can stream the download from cloud storage directly into sharp without ever holding the full original in a Buffer:

const stream = await dropbox.filesDownloadStream({ path });
const resized = await pipeline(stream, sharp().resize(512, 512, { fit: "inside" }).jpeg({ quality: 85 })).toBuffer();

This caps peak memory at sharp's internal tile buffers (~10–20 MB) regardless of source file size, so you can safely process 50 MB TIFFs on a small server. Worth doing if hosted users will be importing large map files.

Suggested rate-limiting layers for hosted

1. Per-user concurrency cap (e.g. 2 in-flight classifications) — protects RAM.
2. Per-user RPM (e.g. 30/min) — protects your provider quota and bills.
3. Per-user daily image cap tied to subscription tier — protects your wallet.
4. Global concurrency cap at the worker pool — protects the box.
5. Backoff on 429s from the provider — already implicit; make it visible to the user.

The provider's own latency (~3–5 s/image) is, as you noted, a soft natural rate limiter, but adding explicit caps prevents one user from burning a whole tier's RPM.