Build Multimodal Models

Course overview

Build multimodal models by leveraging available models and datasets, preprocessing steps per modality, and approaches for classification, VQA and image editing. The module covers vision, audio, text and multimodal generation.

Accessing resources: how to navigate models and datasets, identify popular text‑to‑image models and preprocess multimodal data.

Unimodal models: practical approaches for vision, audio and text tasks such as image classification, object detection and background removal.

Multimodal classification: zero‑shot image classification with CLIP, automatic caption quality evaluation and multimodal sentiment analysis.

Multimodal generation: Visual Question Answering using ViLT and document VQA with LayoutLM, plus image editing using diffusion models.