Vision-Language Models (VLM)

01Where Things Stand (May 2026)

Vision-Language Models (VLMs) — I still hear Multimodal Large Language Models (MLLMs) in papers — jointly read pixels and text. Between 2023 and 2026 they went from lab demos (BLIP-2, MiniGPT-4) to something I reach for in production: document extraction, GUI agents, robotics, everyday chat. The pattern I see winning is native multimodal pretraining: one transformer ingests interleaved image, video, and text tokens, with a vision encoder (or pixel patchifier) feeding the same decoder that writes language.

02What Is a Vision-Language Model

Capabilities I treat as baseline in 2026

• Visual question answering — open-ended Q&A about an image.
• Document understanding — read PDFs, invoices, forms, slides without OCR pre-step.
• Chart, diagram, table reading — numerical extraction with reasoning.
• OCR + handwriting — generally on par with dedicated OCR for printed text; weaker on stylized handwriting.
• Visual grounding — output coordinates / bounding boxes for referred objects (frontier & many open models).
• Multi-image & in-context vision — compare, diff, follow visual instructions.
• Video understanding — up to minutes of video on frontier; seconds-to-minutes on open weights.
• GUI / computer use — observe screenshots and emit click/type/scroll actions.
• Embodied / robotics (VLA) — emit motor or end-effector actions from vision + instructions.

Capabilities Still Fragile

• Counting many small objects (> 20) reliably.
• Precise pixel-accurate localization without specialized heads.
• Long video (> 1 hour) without sampling artifacts.
• Reasoning about novel diagrams that require domain symbol knowledge.
• Subtle factual visual hallucinations (color, count, relative position).

03Anatomy & Architecture

At its core, every modern VLM is a four-stage pipeline:

3.1 The Vision Encoder

The encoder maps an image into a sequence of embeddings, one per spatial patch. Three lineages dominate:

3.2 The Language Model

Almost every open VLM in 2026 is built on top of an existing LLM backbone — typically a Llama, Qwen, Mistral, Gemma, or DeepSeek variant. The choice of backbone constrains: context length, tool-use formatting, system-prompt behavior, license, and quantization support. A common mistake is assuming the VLM inherits the base LLM's full text quality; instruction-tuning on multimodal data often regresses pure-text performance by 3–8 % on hard reasoning benchmarks.

3.3 The Connector (a.k.a. Projector, Adapter, Resampler)

The connector compresses or projects vision-encoder outputs into the LLM's embedding space. This is where most engineering tradeoffs live; see Section 4.

04Connector Patterns

4.1 When to pick which

05Training Lifecycle

5.1 Data sources that matter

• Image-caption webscale — LAION-5B, COYO-700M, DataComp; the workhorses for alignment.
• Interleaved doc corpora — MMC4, OBELICS — teaches multi-image reasoning.
• Synthetic instruction — LLaVA-Instruct, ShareGPT4V, Cambrian-7M, Cauldron; GPT-4V-generated dialogues remain the single highest-leverage SFT data.
• OCR & document — IDL, PDFA, DocVQA, ChartQA, InfographicVQA.
• Grounding — RefCOCO, GRIT, Visual Genome with normalized box format.
• Video — VideoChatGPT, NeXT-QA, ActivityNet, Ego4D, LLaVA-Video-178K.
• GUI — SeeClick, OS-Atlas, Mind2Web, AndroidControl, Computer-Use traces.

5.2 LoRA, QLoRA, and parameter-efficient fine-tuning

For most production use cases, fine-tune only the connector + LoRA adapters on the LLM. Rule of thumb:

062026 Model Landscape

I bucket the market into frontier closed, frontier open, and specialist models. The table below is my shortlist when I am choosing what to pilot — May 2026 snapshot, re-check release notes before you commit.

6.1 Frontier closed (API-only)

6.2 Frontier open-weight (self-hosting viable)

6.3 Specialists worth knowing

07Image Tokenization & Resolution Strategies

The single most expensive variable at inference time is the number of vision tokens per image. A 4K screenshot in a naive 14×14 ViT becomes ~85k tokens — exhausting context windows and budgets. Modern VLMs use one of three strategies:

7.1 Token-budget rules of thumb

• For chat-style use, cap per-image tokens at 1500–2000. Quality plateaus above that for everything except small-text OCR.
• For document Q&A, use per-page tiling and pass pages serially in a structured prompt — never one giant image.
• For multi-image input, prefer uniform sizing across turns; mixed sizes break some caching layers.
• Token-dropping (FastV, VisionZip, PruMerge, SparseVLM) reliably saves 40–70 % of vision tokens with <1 % quality loss on standard benchmarks; test on your data.

08Beyond Single Images — Video, Documents, Charts, UI

8.1 Video

Three practical approaches:

1. Uniform frame sampling (1–2 fps) — simplest; works to ~5 minutes on most open VLMs.
2. Keyframe / scene-cut sampling — pre-process with a shot detector; pass 16–64 frames.
3. Native video tokens (Qwen3-VL, Gemini 2.5, GPT-5) — model expects a video tensor or URI and handles temporal compression internally.

For long video, combine retrieval (Video-RAG) with frame sampling — embed clips, retrieve relevant windows, then pass only those to the VLM. M-RoPE / temporal positional embeddings let modern VLMs maintain absolute-time awareness across long sequences.

8.2 Documents and PDFs

The document pipeline I use when accuracy matters more than demo speed:

1. Render each page to image at 150–200 DPI (higher only for fine-print).
2. Tile aggressively for dense pages (Qwen3-VL or InternVL handle this natively).
3. Pair with layout tokens from a lightweight layout model (e.g., DocLayout-YOLO) for structured extraction.
4. Use a generalist VLM (Claude, GPT-5, Qwen3-VL-72B) for end-to-end Q&A and citation; reserve specialist OCR (GOT-OCR 2.0, MinerU) for archival extraction at scale.

8.3 Charts, tables, math

• Charts: ChartQA, ChartGemma, ChartLlama; emit table + answer to reduce hallucination.
• Tables: render → VLM → JSON schema, validated with Pydantic.
• Math/diagrams: MathVista, MathVerse evaluation; chain-of-thought prompting helps; for diagrammatic geometry, code-execution agents outperform pure VLMs.

8.4 GUI and screenshots

For agents that drive UIs, the screenshot resolution policy is critical: too small → can't read buttons, too large → cost explodes. Use a two-pass recipe — low-res for planning, high-res crops for clicking. Specialist GUI models (UI-TARS, OS-Atlas) ship with click-coordinate output formats; align your harness to that schema.

09Inference & Serving

9.1 Recommended serving stacks (2026)

9.2 Prefix and image caching

9.3 Batching strategy

• VLM batching is dominated by prefill, not decode. Use continuous batching with chunked prefill so large vision-token prompts don't starve other requests.
• For mixed text-only + image traffic, run two pools or use a router — image traffic will otherwise create head-of-line blocking.
• Track P50/P95 vision tokens per request as a first-class metric.

10Quantization & Optimization

11Evaluation & Benchmarks

11.1 Public benchmarks worth tracking

11.2 Build your own eval set

Public benchmarks tell you whether a model is generally capable. They do not tell you whether it will work on your images. Every team shipping VLMs should build:

1. 200–500 golden examples drawn from real production traffic (with consent & redaction).
2. Pairwise human or LLM-judge scoring with explicit rubrics (grounding, hallucination, format).
3. Adversarial slices: low-light, occluded, multi-lingual, low-DPI scans, rotated.
4. Tracked over time in a dashboard with diff-vs-baseline.

12Hallucinations & Safety

12.1 Common failure modes

• Object hallucination — describing objects that aren't there (especially in low-confidence regions or after long context).
• Color/count/spatial errors — left/right confusion, miscount of similar objects, color drift.
• Hyper-confident OCR — fabricating plausible numbers when the image is unreadable.
• Prompt-induced confirmation bias — leading questions ("Is the cat on the table?") nudge the model to confirm.
• Cross-image leakage — info from earlier turns bleeds into the current answer.

12.2 Mitigations

• Use neutral framing in prompts; avoid presupposing answers.
• Ask for citations to image regions (bounding boxes or "according to the top-right panel").
• Run self-consistency at temperature 0 + 1 high-temp sample; flag disagreement for human review.
• Cascade: VLM → verifier. A second model (or a rule) checks the answer's compatibility with the image.
• Use models trained with visual RL/DPO for hallucination (POVID, RLHF-V, VLFeedback).
• For numbers, force a show-your-work output (extract the cell, then compute).

12.3 Safety surface specific to VLMs

• Image-as-jailbreak — text rendered inside an image bypasses some text safety filters. Defense: run prompt-injection detection on extracted image text.
• PII leakage — VLMs willingly transcribe IDs, license plates, faces. Filter or redact before display.
• Bias in perception — counts, ages, attributes can be systematically biased; audit per-cohort.
• CSAM / illegal content — gateway your image input with hash + classifier checks; this is non-negotiable.

13Agents, Computer Use & Vision-Language-Action

13.1 Computer Use

Claude Computer Use, GPT-5 Operator, and open-weights stacks like UI-TARS, OS-Atlas, and ShowUI let a VLM drive a desktop or browser. What I have learned in practice:

• Click accuracy gates success — use a specialist grounding model if your generalist's coordinates jitter.
• Add a visual verifier that confirms the post-action screenshot matches intent before continuing.
• Cap step counts and add a circuit breaker — runaway loops are the #1 incident class.
• Sandbox by default. The model will follow instructions written in webpages and emails.

13.2 Vision-Language-Action (VLA) for robotics

VLA models output low-level robot commands instead of text. RT-2, OpenVLA, π0 / π0.5, Gemini Robotics, and Figure's Helix dominate 2026 demos. The architectural twist: action tokens are interleaved with language tokens in the same decoder. When I wire up a VLA stack, I plan for:

• Discrete action vocabularies (binned end-effector deltas) — easy to log and replay.
• 50–200 Hz action loops; the VLM runs at lower rate with a fast low-level controller.
• Safety bounds enforced outside the model (workspace limits, force ceilings).

14Shipping VLMs in Production

14.1 Pattern library

14.2 Prompting patterns I use in production

• Put the task before the image. Most models do better when the question precedes the image; some (Claude) accept either, but consistency reduces cache misses.
• Be explicit about output format. Ask for JSON or XML and validate. Free-form prose invites hallucination.
• Force grounding with phrases like "Quote the exact text from the image" or "Give the bounding box [x1, y1, x2, y2] for each item."
• Restrict the answer space. "Choose from A/B/C/D/Unknown" reduces wandering and produces measurable accuracy.
• Allow refusal. Include "If you cannot see X, reply 'NOT_VISIBLE'." This dramatically reduces fabrication.

// Production prompt skeleton
SYSTEM: You are a document understanding assistant.
        Read the image carefully. If a value is illegible,
        return "ILLEGIBLE" — never guess.

USER:   [IMAGE]
        Extract the following fields as JSON:
        - invoice_number (string)
        - total_amount (number, no currency symbol)
        - line_items (array of {description, quantity, unit_price})
        For each value, also provide bbox: [x1,y1,x2,y2] in 0–1000 coords.
        Output ONLY valid JSON. No prose, no markdown.

14.3 Observability

• Log: model id, image hash, vision-token count, latency (TTFT, TPS), output schema validity, refusal flag.
• Sample 1–5 % of requests for human review.
• Track drift weekly: same eval set, same model, watch for regressions tied to provider updates.
• Tag-aware dashboards by image type (PDF page, screenshot, photo) — failure modes differ wildly.

15Cost & Throughput Engineering

15.1 Cost levers ranked by impact

1. Prefix cache reusable images (logo, base template, system shot) — typical 5–15× cost reduction.
2. Right-size resolution per task — most chat-style use needs ≤1024×1024.
3. Cascade: small open VLM screens, frontier handles only ambiguous 5–20 %.
4. Vision-token pruning (FastV, VisionZip) on self-hosted — 40–70 % prefill saving.
5. FP8 / INT4 weights — 2–4× throughput on the same GPU.
6. Speculative decoding — 1.5–3× decode speedup, lossless.
7. Batched async — never block on a single request when your traffic is mixed.

15.2 GPU memory budgeting (rough)

Add ~10–25% headroom for KV cache and activations; more if you accept many concurrent long-context image requests.

16What I Am Watching Next

17How I Choose a Model

17.1 Mistakes I have seen (or made)

• Picking a model without an eval set. I have watched teams learn this in production — painfully.
• Sending raw 4K images. I cap resolution early; extra pixels rarely buy accuracy on chat-style tasks.
• Ignoring prefix caching. Most repeating-image workloads are 5–15× over budget without it.
• Free-form output. Always demand JSON / XML with a validator.
• One-shot agent loops. Add a verifier and a circuit breaker; runaway loops are the most common incident.
• Trusting general benchmarks. MMMU does not predict your invoice-reading accuracy.
• Skipping quantization checks on visual tasks. INT4 can silently degrade OCR by 10 %+.

18Glossary & References

18.1 Glossary

18.2 Reference papers and projects

• Radford et al., CLIP (2021) — contrastive image-text pretraining.
• Alayrac et al., Flamingo (2022) — gated cross-attention, in-context multimodal.
• Li et al., BLIP-2 (2023) — Q-Former alignment.
• Liu et al., LLaVA & LLaVA-NeXT (2023–24) — open visual instruction tuning recipe.
• Zhai et al., SigLIP / SigLIP 2 (2023–25) — sigmoid loss image encoders.
• Bai et al., Qwen-VL · Qwen2-VL · Qwen3-VL (2023–26) — dynamic-resolution VLMs.
• Chen et al., InternVL · InternVL 2.5 · InternVL 3 (2024–26).
• Mistral AI, Pixtral 12B · Pixtral Large (2024–25) — encoder-free design.
• Allen AI, Molmo & PixMo (2024) — open data, open weights, pointing.
• Microsoft, Phi-3.5 Vision · Phi-4 Multimodal (2024–25) — small-but-mighty.
• Anthropic, Claude 3.5/4 with vision, Computer Use API.
• OpenAI, GPT-4V / GPT-4o / GPT-5 system cards.
• Google DeepMind, Gemini 1.5/2.0/2.5 technical reports.
• Brohan et al., RT-2 (2023); Kim et al., OpenVLA (2024); Physical Intelligence π0/π0.5 (2024–25) — VLA models.
• Chen et al., FastV (2024); Yang et al., VisionZip (2024) — visual-token pruning.
• VLMEvalKit, lmms-eval — community evaluation harnesses.