ColVec1.1: Increasing Accuracy, Reducing Latency on Visual Retrieval

Enterprise search and multimodal retrieval-augmented generation often fail on the documents teams care about most: financial reports filled with tables, technical manuals with diagrams, presentation decks, scientific papers, and PDFs where layout carries as much meaning as the text itself.

Traditional retrieval pipelines typically address this problem in one of two ways. They either flatten every page into OCR-extracted text or use a vision-language model to generate a textual description of each page before indexing it. Both approaches introduce additional processing stages, discard visual relationships, and create another place for information to be lost.

Earlier this year, we introduced ColVec1 and made the case for retrieving directly from rendered document pages. ColVec1 demonstrated that an OCR-free, retrieval-specific model could compete at the top of ViDoRe V3 without converting visually structured documents into an intermediate text representation. ColVec1.1 advances that approach with stronger retrieval accuracy, bidirectional attention, more compact embeddings, improved multilingual training, and greater control over the quality-efficiency tradeoff.

Search the page, not a reconstruction of it

The OCR-free design provides several practical advantages:

• End-to-end visual retrieval: Search rendered pages without requiring OCR or an intermediate vision model to summarize them.
• Preservation of document structure: Retain signals from tables, charts, figures, spatial relationships, typography, and page layout.
• Multilingual retrieval: Retrieve documents and queries across English, French, Spanish, German, Italian, and Portuguese.
• Tunable page resolution: Adjust the number of visual tokens used per page to trade retrieval detail for memory and latency.
• Unified query and document encoder: Process text queries and visual documents with the same model rather than maintaining separate embedding towers.

The result is a retrieval model designed for financial reports, technical manuals, scientific publications, presentations, forms, textbooks, and multilingual enterprise archives—not only clean, text-heavy documents.

Image from: ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios

Introducing ColVec1.1-4B and ColVec1.1-8B

ColVec1.1 introduces a set of concrete, deployment-relevant improvements over ColVec1 that directly impact retrieval quality, efficiency, and usability in production systems:

• Much smaller embeddings: Reduces projection size from 2,560 to 640 dimensions (75% reduction), significantly lowering index size, memory bandwidth, and compute cost.
• Higher retrieval accuracy: Improves average performance across ViDoRe V3 tasks achieving a new SOTA.
• Bidirectional attention for better context: Removes the left-to-right constraint in full-attention layers so each token can attend to the entire page rather than only preceding tokens, which matters for tables, figures, and layout-dependent semantics.
• Revised data pipeline: Intelligent data curation with on-the-fly augmentation.
• More effective training strategy: Introduces a staged pipeline combining positive retrieval training with improved loss formulation, and checkpoint merging for stronger ranking behavior.
• Improved multilingual coverage: Uses a more balanced and curated dataset with expanded query translations across six languages, reducing domain and language bias.
• More efficient visual representations: Increases supported visual token budget (up to 1,792 tokens), allowing finer-grained page understanding while maintaining a more compact embedding space overall.
• Better hard-negative discrimination: Explicit training on near-miss documents improves robustness in cases where pages are visually or semantically similar but contextually different.
• More flexible deployment profile: Supports both high-performance (8B) and resource-efficient (4B) models with a unified retrieval interface and configurable quality–latency tradeoffs.

webAI-ColVec1.1-8B

Our flagship visual retrieval model and the highest-performing model across ViDoRe V3 tasks.

ColVec1.1-8B is built on the Qwen3.5-9B vision-language backbone. We removed the language-model output head from the released checkpoint, reducing the model to approximately 8.4 billion parameters and reducing memory requirements by ~11%.

The model projects each query and page token into a 640-dimensional embedding space, compared with the 2,560-dimensional output of ColVec1-9B. This reduces database memory requirements and computations by 2.86× (≈65%) at highest fidelity or 4x (75%) at similar fidelity.

webAI-ColVec1.1-4B

A smaller model designed for deployments where accelerator memory, latency, or local execution is more constrained. ColVec1.1-4B also produces 640-dimensional token embeddings and achieves the highest leaderboard score among currently published models in its size class.

They use the same retrieval interface and support document images, rendered PDF pages, and natural-language queries.

A new state of the art on the ViDoRe V3

ViDoRe V3 evaluates visual document retrieval across complex, multilingual, and domain-specific collections. It has become a critical benchmark for evaluating how well retrieval systems perform in real enterprise environments where documents are inherently visual and structurally complex. Unlike traditional text-only benchmarks, it reflects the challenges faced in production systems across finance, legal, manufacturing, and scientific domains, where key information is embedded in tables, charts, multi-column layouts, and multilingual reports.

Strong performance on ViDoRe V3 is therefore a strong indicator that a model can reliably power real-world retrieval-augmented generation systems, enabling employees and applications to surface precise evidence from large-scale document repositories rather than relying on lossy OCR or text summarization pipelines.

The table below reflects the performance of ColVec1.1 on the ViDoRe V3 leaderboard. ColVec1.1-8B establishes a new state of the art on the full benchmark, while ColVec1.1-4B is the best-performing model in its weight class.

Why multi-vector retrieval?

Many embedding systems compress an entire page into a single vector. This makes indexing and nearest-neighbor search efficient, but it forces every concept, table cell, label, figure, and layout relationship into one fixed-size representation.

That compression can be especially destructive for dense visual documents. A page might contain a title, two charts, a footnote, a table, and several unrelated sections. A single-vector embedding must blend all of those signals together.

ColVec1.1 instead uses ColBERT-style late interaction. It creates a sequence of token-level vectors for the query and another sequence for the document page.

For a query with token embeddings (q1,…,qm) and a document with token embeddings (d1,…,dn), the MaxSim score is:

For each query token, MaxSim finds the most similar token on the page. It then adds those best-match scores together.

This means different parts of a query can align with different regions of the same page. A query about “2025 European revenue growth” might match:

• “2025” in a table row,
• “European” in a regional heading,
• “revenue” in the table title,
• and “growth” in a chart annotation.

The page does not have to be compressed into a single representation before those relationships are evaluated. Multi-vector retrieval preserves the fine-grained evidence needed to distinguish pages that are visually or semantically similar but answer different questions. ColVec1.1 uses a single shared model to generate these embeddings for both text queries and document images.

Retrieval quality is only part of the system

Benchmark performance matters, but production retrieval also depends on index footprint, document-encoding throughput, query latency, reranking architecture, and the ability to control what evidence reaches the generation model.

ColVec1.1 is designed to improve that complete retrieval layer.

Direct page retrieval removes OCR and page-summarization dependencies from the indexing path. Multi-vector embeddings preserve local evidence rather than collapsing it into a single representation. The 640-dimensional projection makes late-interaction retrieval more practical at scale. The tunable visual-token budget lets teams choose how much page detail they need to preserve.

This also improves the downstream RAG experience. Instead of presenting a language model with disconnected OCR fragments, the system can retrieve and return the original pages that contain the relevant tables, figures, text, and surrounding context. Users can inspect the evidence in the same form in which it appeared in the source document.

ColVec1.1 provides two model sizes for different production constraints and deployment flexibility.

Looking ahead

ColVec1 established that specialized, OCR-free retrieval models could compete at the highest level of visual document search. ColVec1.1 shows that retrieval quality does not have to come at the expense of deployment efficiency.Built on architectural improvements, enhanced data curation, and a refined training pipeline, ColVec1.1 shows that retrieval quality does not have to come at the expense of deployment efficiency

The result is state of the art ViDoRe V3 performance, a substantially smaller flagship embedding representation, and a more flexible foundation for enterprise search and multimodal RAG.

The models are available now:

• webAI-ColVec1.1-4B
• webAI-ColVec1.1-8B

Use them on the documents that are hardest for conventional retrieval systems: the pages where the answer is not only in the words, but in the way the page is structured. Models are available in webAI products and offerings to enterprise customers.