Meta Releases TRIBE v2: A Brain Encoding Model That Predicts fMRI Responses Across Video, Audio, and Text Stimuli

TRIBE v2 aligns AI representations with brain activity to predict fMRI, tRIBE v2 aligns latent representations of AI architectures with human brain activity to predict fMRI responses.

Text encoder extracts contextualized text embeddings from LLaMA 3.2-3B, text embeddings are extracted from LLaMA 3.2-3B with 1,024 preceding words prepended and mapped to a 2 Hz grid.

Video encoder processes video with V-JEPA2-Giant on 64-frame, 4-second segments, video is processed using V-JEPA2-Giant on 64-frame segments covering the preceding 4 seconds per time-bin.

Audio encoder processes audio with Wav2Vec-BERT 2.0 and resamples representations to 2 Hz, audio is processed with Wav2Vec-BERT 2.0 and resampled to 2 Hz to match stimulus frequency.

Model architecture concatenates three D=384 embeddings into a Dmodel=1152 sequence and inputs it to an 8-layer Transformer, embeddings are projected to a shared dimension D=384 and concatenated to form a model dimension of 1152 fed into an 8-layer Transformer.