Implementing Video Intelligence in the Cloud: AI Video Analysis with TwelveLabs Part 4 – TwelveLabs Marengo 3.0 Embedding and Retrieval Strategies and Implementation Guide | Amazon Web Services

Marengo 3.0 maps all modalities into a unified vector space, enabling any-to-any search.

Three embedding/search strategies: Fused Embeddings; Multi-Vector Retrieval (Score-based, RRF); Intent-based routing.

Fused Embeddings sums and normalizes the three modality vectors per segment into a single index; default fusion weights Visual 0.8, Audio 0.1, Transcription 0.05; changing weights requires reprocessing and fixed weights ignore query intent and hinder debugging.

Marengo returns separate visual, audio, and transcription embeddings per clip; Multi-Vector storage keeps originals in separate indices so weights can be adjusted later, enabling reversibility, easier debugging, and flexibility.

Score-based Fusion multiplies modality cosine similarities by weights and sums; RRF is also mentioned; overall challenges include modality weighting, multi-vector return strategies, routing, ranking, and score calibration.