02Multimodal AI · Search & Recommendation · Read 11 min

From CLIP benchmarks to a production multimodal platform

Selecting representations by task, building an incremental pipeline and serving product neighbours through an observable system.

IN ONE SENTENCE

The platform assigns models by task, reuses unchanged embeddings and makes relevance, latency, coverage and cost trade-offs measurable.

CONTEXT

The program began with visual and semantic search experiments, then expanded to recommendations for products with no interaction history. I contributed to connecting model benchmarks, catalogue preparation, vector indexing, APIs and observability in an architecture that became a production pipeline.

SUMMARY

Benchmarks on public catalogues showed that a dedicated text encoder led semantic search, while a domain-specific vision-language model improved visual similarity. The production architecture computes representations incrementally, combines text and images with configurable weights, builds neighbours and monitors pipeline coverage and performance.

Key diagram

The multimodal pipeline, from catalogue to product neighbours

01 · FIELDS

NAMEw=1.0

DESCRIPTIONw=0.5

IMAGEw=0.8

→ENCODE

02 · EMBEDDINGS

→WEIGHT

03 · FUSIONΣ w_f · e_fL2 normalize

→INDEX

04 · HNSW

05 · TOP-K

01Similar item.92

02Similar item.87

03Similar item.81

Text and images are encoded separately, fused through versioned configuration and searched in a neighbour index. Production adds monitoring, output tests and fallbacks.

A case study focused on transferable methods, decisions and learnings. No client data or confidential information is disclosed.

Contribution & production02

How do CLIP experiments become an operational production pipeline?

I contributed to connecting model selection, incremental computation, configurable fusion, vector indexing, serving and monitoring. The system now runs in production with explicit limits and fallback strategies.

Operational quality

Relevance and coverage

Latency and load

Versions, failures and fallbacks

Key idea

Productionization rests on a verifiable decision chain: models by use case, incremental computation, load-tested indexing, output tests, observability and fallbacks.

01 — Model selection

A task-specific benchmark avoids choosing an encoder by reputation.

Two reproducible protocols separated textual search from image similarity. On public catalogues, FashionCLIP improved visual NDCG@10 by 7.9% on Amazon Berkeley Objects and 4.6% on H&M compared with CLIP. For semantic search, a dedicated text encoder remained stronger.

The decision was to preserve several specialized spaces until a hybrid benchmark could justify merging them. Because categories provide only a proxy for relevance, results were interpreted with explicit limitations and complemented by qualitative inspection.

02 — Pipeline

Incremental computation separates expensive inference from product strategy.

Each product field—name, description and image—has a fingerprint and a versioned representation. Only new or modified content is re-encoded. Removed products follow an explicit deletion rule, while image errors and optional fields use observable fallback strategies.

Embeddings are combined with configurable weights and normalized. Fusion strategy can therefore change without recomputing the entire catalogue. Separation by model and field also lets several versions coexist during migration.

03 — Serving & scale

The architecture makes retrieval trade-offs measurable.

ClickHouse, HNSW indexes and in-memory strategies were compared on relevance, latency and load behavior. FastAPI exposes the required operations, while orchestration, containerization and load testing make the path reproducible.

A GPU embedding service was also evaluated to saturation to separate hardware gains from batching and orchestration limits. Those measurements informed capacity planning instead of becoming a technology preference by default.

04 — Production

Output tests and monitoring extend the benchmark after deployment.

The multimodal pipeline is now in production. Operations monitor catalogue coverage, encoding failures, neighbour availability, latency and model versions. Migrations and metadata consistency receive functional checks because a query may technically succeed while still returning an incorrect ranking.

Experimental components stayed outside the critical path when they did not add enough value. Automatic segmentation, for example, introduced failures and cost without a sufficiently robust improvement for the first version. This decision reduces complexity while retaining an ablation protocol for future catalogues.

Evidence frame

Anonymized professional multimodal-retrieval program

A synthesis of public-data benchmarks and productionization work now operating in production. Environments, catalogues, internal volumes and identifiers remain anonymized.

Methods and references

Radford et al. (2021) — CLIP
FashionCLIP and public Amazon Berkeley Objects / H&M datasets
Hierarchical Navigable Small World (HNSW)
NDCG@K, Recall@K, latency and load testing