OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation

Researchers introduced OmniVAE, a variational autoencoder for audio-video generation that learns shared semantics during training.

It uses segment-level contrastive alignment and modality-specific distillation from pretrained teachers to capture both common events and modality details.

The unified latent space improved text-to-audio-video generation quality and audio-video synchronization with little reconstruction loss.