Generative AI ~50 min

Multimodal Systems

How AI systems perceive the world through multiple senses simultaneously. From CLIP's unified embedding space to GPT-4V's visual reasoning to Gemini's native multimodality—learn how modern systems bridge text, images, audio, and video.

Learning Objectives

Understand what multimodal learning means and why it matters
Explain contrastive learning and CLIP's shared embedding space
Distinguish different multimodal architectures: fusion strategies and their trade-offs
Understand how vision-language models like GPT-4V and LLaVA process images
Explore the frontier: video understanding, audio integration, and native multimodality
Recognize challenges: hallucination, grounding, and cross-modal alignment

The Multimodal Paradigm

Multimodal AI

Systems that can process, understand, and generate information across multiple modalities (text, images, audio, video) within a unified framework, enabling cross-modal reasoning and generation.

Intuition

Humans understand the world through multiple senses simultaneously. When you read "the dog barked," you can imagine how a dog looks and sounds. Multimodal AI aims to develop this same cross-modal understanding: a system that can describe images, answer questions about videos, generate images from text, and reason across modalities.

Technical Framework

The core challenge: different modalities have fundamentally different structures. Text is discrete tokens in sequence. Images are 2D grids of pixels. Audio is 1D waveforms. Video adds temporal dynamics. Multimodal systems must:

Encode each modality into a representation
Align representations so cross-modal relationships emerge
Fuse information across modalities for unified reasoning
Decode back to any target modality

In Practice

Vision-Language Models: GPT-4V, Gemini, Claude 3, LLaVA—understand images and respond in text.

Text-to-Image: DALL-E, Stable Diffusion, Midjourney—generate images from text descriptions.

Any-to-Any: Emerging systems (Gemini 1.5, GPT-4o) can handle multiple input modalities and generate in multiple output modalities.

CLIP: The Foundation of Modern Multimodality

CLIP (Contrastive Language-Image Pre-training)

A model that learns a shared embedding space for images and text through contrastive learning on 400M image-text pairs from the internet. Any image and any text can be embedded and compared in the same vector space.

CLIP Architecture

Image Encoder

ViT or ResNet

Image Embedding
v ∈ ℝᵈ

Text Encoder

Transformer

Text Embedding
t ∈ ℝᵈ

Shared Embedding Space

cosine similarity(v, t)

✓ High: matching pairs ✗ Low: non-matching

Contrastive Learning Objective

Given a batch of N (image, text) pairs, CLIP learns to predict which image goes with which text. The training creates N correct pairs and N²-N incorrect pairs within each batch.

L = -log [exp(sim(vᵢ, tᵢ)/τ) / Σⱼ exp(sim(vᵢ, tⱼ)/τ)]

InfoNCE loss: maximize similarity for true pairs, minimize for false pairs

The temperature τ controls how "peaked" the distribution is. After training, semantically related images and texts cluster together in embedding space.

◈

Why CLIP Changed Everything

Zero-shot classification: Describe any class in text, find matching images
Open vocabulary: Not limited to pre-defined categories
Transfer learning: CLIP embeddings work for downstream tasks
Image generation: CLIP guides diffusion models via text embeddings

Image Search

Search images with natural language queries. Embed query, find nearest image embeddings.

Zero-Shot Classification

"A photo of a [cat/dog/bird]"—compare image to each text embedding, pick highest.

CLIP + Diffusion

Text encoder for Stable Diffusion. Guide image generation toward text description.

Content Moderation

Embed policy descriptions, flag images that embed close to prohibited content.

Vision-Language Models: Images as First-Class Input

While CLIP aligns image and text embeddings, Vision-Language Models (VLMs) go further: they process images directly within a language model, enabling visual reasoning, description, and question-answering.

Approach 1: Adapter-Based (LLaVA, MiniGPT-4)

Image → ViT Encoder → Visual Tokens

↓

Projection/Adapter Layer

↓

Visual Tokens + Text Tokens → Frozen LLM

Project visual features into the LLM's embedding space. The LLM treats visual tokens like text tokens. Often keeps LLM frozen, training only the projection.

✓ Leverages existing LLM capabilities ✓ Efficient training ✗ Visual understanding limited by projection

Approach 2: Native Multimodal (Gemini, GPT-4V)

Image → Visual Tokenizer → Visual Tokens

Text → Text Tokenizer → Text Tokens

↓

Interleaved Tokens → Unified Transformer

Train a single model from scratch on interleaved multimodal data. Images and text are processed by the same transformer with shared attention.

✓ Deep cross-modal integration ✓ Better visual understanding ✗ Requires massive training from scratch

Approach 3: Q-Former (BLIP-2)

Image → Frozen ViT → Image Features

↓

Q-Former (learnable queries) → Compact Visual Rep

↓

Compact Tokens → LLM

Use learnable "queries" that attend to image features and produce a fixed number of tokens. Bridges frozen vision encoder to frozen LLM efficiently.

✓ Both encoders frozen ✓ Very parameter-efficient ✗ Information bottleneck in Q-Former

Visual Tokenization Strategies

How do you convert a continuous image into discrete tokens for a transformer?

Patch Embedding (ViT-style): Split image into patches (e.g., 16×16), linearly embed each patch. A 224×224 image → 196 tokens.

VQ-VAE Tokens: Use a discrete codebook. Each patch maps to a discrete token ID. Enables generation.

Dynamic Resolution: GPT-4V and Gemini handle variable image sizes with adaptive tokenization—more tokens for larger/complex images.

Text-to-Image: Generating Visual Content

The reverse direction: given text, generate images. Modern text-to-image combines text encoders (CLIP or T5), diffusion models, and conditioning mechanisms.

Stable Diffusion Pipeline

Text Encoding

CLIP text encoder maps prompt to embeddings

"A cat sitting on a book" → [e₁, e₂, ..., eₙ]

→

Latent Diffusion

U-Net denoises in latent space, guided by text

Cross-attention injects text at each layer

→

VAE Decode

Decode latent to pixel space

64×64×4 latent → 512×512×3 image

How Text Guides Generation

Text conditioning happens through cross-attention in the U-Net. At each attention layer:

Q = W_Q · image_features K, V = W_K · text_emb, W_V · text_emb

Image features query into text embeddings—learning what parts of the description are relevant for each spatial location

◈

DALL-E 3's Innovation: Caption Improvement

DALL-E 3 doesn't just use user prompts directly—it rewrites them with an LLM to be more detailed and unambiguous. "A cat" becomes "A fluffy orange tabby cat sitting alertly on a weathered wooden surface, soft natural lighting, photorealistic style." This dramatically improves prompt following.

Beyond Image-Text: Video, Audio, and More

Video Understanding

Challenge: Videos add temporal dimension. A 10s video at 30fps = 300 frames. Can't process each frame independently.

Approaches:

Sample key frames + image understanding
3D convolutions (like 2D conv but across time)
Temporal attention: attend across frames
Token compression: represent video as fewer tokens

Systems: GPT-4V (limited), Gemini 1.5 (1M token context), Video-LLaVA, Sora (generation)

Audio Integration

Challenge: Audio is high-dimensional (44kHz = 44,000 samples/sec). Speech, music, environmental sounds all differ.

Approaches:

Mel spectrograms → treat as images
Audio tokenizers (EnCodec, SoundStream)
Whisper embeddings for speech

Systems: GPT-4o (native audio), Gemini (audio input), AudioPaLM, MusicGen

Native Multimodality

Goal: One model that naturally handles any combination of input/output modalities.

Architecture:

Unified tokenization across modalities
Single transformer processes interleaved sequences
Modality-specific heads for generation

Systems: Gemini, GPT-4o, Chameleon (Meta). These represent the frontier—truly unified perception.

Multimodal Fusion Strategies

How and when do you combine information from different modalities? This architectural choice significantly impacts what the model can learn.

Early Fusion

Modality A → Tokens

Modality B → Tokens

Concatenate

→ Joint Transformer → Output

Combine at input level. Full cross-modal attention from the start.

Deep integration Computationally heavy

Late Fusion

Modality A → Encoder A → Rep A

Modality B → Encoder B → Rep B

Combine (concat, add, project)

→ Decision Layer → Output

Process modalities separately, combine at the end.

Modular, reusable encoders Limited cross-modal reasoning

Cross-Attention Fusion

Modality A → Self-Attention Layers

Modality B → Self-Attention Layers

Cross-Attention (A attends to B)

→ More Layers → Output

Dedicated cross-attention layers for modality interaction.

Controlled integration Efficient

Challenges in Multimodal AI

Hallucination

VLMs can "see" things that aren't there. When asked about an image, they may invent objects, attributes, or relationships based on statistical priors rather than visual evidence. "How many dogs are in the image?" might get "3" when there are 2—because 3 is common in training data.

Mitigation: Better grounding, explicit verification, chain-of-thought visual reasoning

Grounding

Does the model truly understand which parts of an image correspond to which words? Or is it pattern matching? Grounding tests require localization: "Where is the red ball?" should point to specific pixels.

Mitigation: Grounding annotations in training, region-text pairing, object detection integration

Modality Imbalance

Text data is vastly more abundant and structured than image/video data. Models may rely too heavily on text priors, ignoring visual information when it conflicts with textual expectations.

Mitigation: Balanced training, visual-only tasks, adversarial probing

Compositionality

"A red cube on a blue sphere" vs "a blue cube on a red sphere"—models struggle with binding attributes to objects and spatial relationships. Text-to-image often fails on precise compositions.

Mitigation: Structured representations, compositional training data, explicit reasoning

Efficiency

Images are expensive: one 224×224 image → 196 tokens. Videos explode: 10 seconds → thousands of tokens. Long-context multimodal is a major computational challenge.

Mitigation: Token compression, sparse attention, hierarchical representations

Evaluation

How do you measure multimodal understanding? Benchmarks are limited. Models can game specific tests without true understanding. Human eval is expensive and subjective.

Mitigation: Diverse benchmarks, adversarial testing, capability-specific probes

Common Misconceptions

✗

"Multimodal models 'see' images like humans do"

VLMs process images as collections of patch embeddings. They don't have biological vision—no eye movements, no depth perception, no real-time continuous processing. Their "understanding" is statistical correlation.

The accurate framing: Multimodal models learn statistical associations between visual patterns and language. They can be remarkably capable, but their perception differs fundamentally from human vision.

✗

"CLIP understands the meaning of images"

CLIP learns to match images and text that co-occur on the internet. It captures visual-linguistic correlations, not semantic understanding. It can be fooled by adversarial examples or distribution shifts.

The accurate framing: CLIP learns a powerful cross-modal embedding space that captures internet-scale correlations. This is useful but not equivalent to understanding. Always probe for failure modes.

✗

"More modalities always means better understanding"

Adding modalities introduces training complexity, potential for modality conflict, and computational overhead. Poorly integrated modalities can degrade performance on individual tasks.

The accurate framing: Multimodality is valuable when modalities provide complementary information and are well-integrated. The architecture and training matter more than simply "more inputs."

⚗ Interactive Lab: Multimodal Embedding Space

Explore how CLIP-style models create a shared embedding space where images and text can be compared. See how semantic similarity works across modalities.