Multimodal Applications: Bringing Vision Capabilities into Your Product

Over the past year, leading large models (Claude, GPT, Gemini, and Qwen-VL) have all gained native support for image input. "Multimodal" has gone from a buzzword to a foundational capability that can be used directly in products. But "can see images" does not mean "Vision should be used for everything." This article explains what these models are genuinely good at, how to use them, and where the pitfalls lie.

1. What Vision Models Do Well

Breaking image understanding into its component tasks, current Vision models excel at several things:

• Document and screenshot parsing: Converting charts, invoices, forms, and UI screenshots into structured information. This is already production-ready.
• OCR + semantics: Not only recognizing text, but also understanding its meaning ("Which line item is the total on this invoice, and how much is the tax?").
• Image description and Q&A: Describing content and answering questions about an image.
• Visual reasoning: Finding bugs in UI screenshots, reading trends from charts, and understanding diagrams.

Weaknesses (do not rely on them for these):

• Precise localization and measurement: Pixel-level coordinates and exact dimensions are unreliable.
• Dense small objects: When asked to identify hundreds of small objects individually, the model will miss some.
• Specialized domains: General-purpose Vision is insufficient for medical imaging, satellite imagery, and other fields that require domain-specific training.

The decision rule is: If the task is to understand image content and reason about it, use Vision. If it requires precise pixel operations or specialized recognition, use a dedicated model such as OCR or object detection.

2. Typical Product Use Cases

Vision has already demonstrated commercial value in several use cases:

1. Intelligent document processing. Upload a photo of an invoice, contract, or report → extract structured fields such as amounts, dates, and clauses for storage. This is far more robust than traditional OCR plus rule-based parsing because it can understand layout changes, handwriting, and nested tables.

2. Customer service and support. A user submits a photo or screenshot of an "equipment failure" → the model identifies the problem and suggests a solution. The experience is ten times better than asking users to "describe your problem."

3. Content moderation. Process an image or screenshot → determine whether it violates policy or contains sensitive information. This can serve as a preliminary filter before human review.

4. Accessibility. Describe image content for users with visual impairments by automatically generating alt text.

5. UI and front-end engineering. Converting screenshots to code, reviewing UIs, and reproducing designs. Claude and Gemini are already very capable in this area.

3. Engineering Essentials

1. How to send images. There are two options: inline base64 for small images and real-time use, or URL references for large images and multiple images. base64 is simple but consumes tokens; with URLs, the model service must be able to access the resource over the public internet or the same internal network.

// Claude image input (base64)
const message = await client.messages.create({
  model: "claude-sonnet-5",
  messages: [{
    role: "user",
    content: [
      { type: "image", source: { type: "base64", media_type: "image/png", data: base64 }},
      { type: "text", text: "Extract the total amount and date from this invoice, and output JSON" },
    ],
  }],
});

2. Token costs. This is the easiest way for a Vision application to go wrong: images are converted into tokens based on their resolution, and one high-resolution image can consume thousands of tokens. A single 4K screenshot may cost more than the entire prompt.

Three ways to control costs:

• Preprocess and resize: Before uploading, resize images to a resolution sufficient for the model. For most tasks, 1024–1568px is enough; do not send the original 4K image.
• Crop: Send only the region of interest, such as the amount field on an invoice, rather than the entire image.
• Compress: Use WebP or an appropriate JPEG quality setting to reduce file size.

Rule of thumb: Preprocessing before upload can cut Vision costs by an order of magnitude. A Vision application without preprocessing will produce a painful bill.

3. Prompt design. Prompts for Vision tasks need to be more specific:

• State exactly what to extract ("amount, date, and tax" rather than "extract information").
• Specify the output format with a JSON schema.
• Handle uncertainty ("Return null if it is unreadable; do not guess").

With a vague prompt and an image, the model is likely to "improvise" by inventing an answer it considers plausible.

4. Combine it with OCR. For high-accuracy text recognition, dedicated OCR tools such as PaddleOCR and Tesseract remain more accurate and less expensive than Vision. Combining OCR for text extraction with Vision for semantic understanding and layout reconstruction works better than either one alone.

4. Limitations and Pitfalls

• Hallucinations. The model may "see" things that are not in the image, especially when prompted to do so. Cross-check critical information, such as by asking for the extracted amount again or verifying it with OCR.
• Inconsistency. Asking about the same image twice may produce different details. When consistency is important, apply post-processing or run the task multiple times and use the consensus result.
• Privacy. Images may contain PII or sensitive information. Redact them before sending them to a third-party API, or use a self-hosted Vision model.
• Version drift. After a model upgrade, the same image may be parsed differently. Maintain a Vision evaluation set just as you would for RAG.

5. Deciding Whether to Use Vision

Not every requirement that involves images should use Vision. Use this decision tree:

Need to extract or understand information from an image?
  ├ Yes → Is the information primarily text? (invoice/document/screenshot)
  │       ├ Yes → Try OCR first; use Vision if it is not enough
  │       └ No (chart/scene/semantics) → Vision
  └ Need precise pixels/measurement/dense recognition → Dedicated CV model; do not use Vision

Summary

Vision models have made image understanding a product-grade capability. Document parsing, customer support, moderation, accessibility, and UI engineering are currently their sweet spots. But using them well is less about knowing how to call an API and more about engineering:

• Preprocess with resizing, cropping, and compression to control costs.
• Use precise prompts that specify what you need and how it should be returned.
• Cross-check critical information to prevent hallucinations; do not rely on a single output.
• Combine with OCR: High-accuracy text recognition remains inexpensive and reliable.

Remember: Vision is not an "all-purpose image processor"; it is a "reasoning engine that can see." Use it for understanding and reasoning rather than pixel operations, and keep image token costs under control. It can then become a highly differentiated capability in your product.