Multimodal and Generative Models

About this course

Models that see, hear, read documents and generate images, video and speech are now part of everyday products. This course explains how vision-language models, diffusion and flow-matching models, and speech recognition and synthesis models work, then puts them to work in document intelligence, controllable generation and voice agents. It closes on evaluation and on the safety, provenance and rights questions that generated media raises, including content credentials and EU AI Act transparency duties.

For: ml practitioner, software engineer, student.

Learning outcomes

On successful completion you will be able to:

1. 1Explain how vision-language models, diffusion and flow-matching models, and speech recognition and synthesis models are built and trained.
2. 2Build applications that combine text, image, audio and document understanding using current multimodal models and APIs.
3. 3Analyse the quality, latency and cost characteristics of multimodal models and select suitable models for a use case.
4. 4Evaluate multimodal outputs with appropriate automatic metrics and human review, including bias and failure analysis.
5. 5Assess the safety, provenance and rights implications of generated media, including impersonation risk, watermarking, content credentials and EU AI Act transparency duties.

Modules and lessons

1. Week 1Vision and language⌄

   • Vision-language models
   • Document intelligence with multimodal modelslab
2. Week 2Image and video generation⌄

   • Diffusion and flow-matching models
   • Video generation and controllable generationlab
3. Week 3Speech and audio⌄

   • Speech recognition
   • Speech synthesis and voice agentslab
4. Week 4Evaluation, safety and provenance⌄

   • Evaluating multimodal systems
   • Safety, provenance and rights in generated media

Assessment

• Quiz (20%)

  Four weekly quizzes drawn from the lesson quiz banks; all count.

• Lab (35%)

  Three graded labs on document extraction, controllable image generation and a voice loop, each with a measured result.

• Project (30%)

  Build a small multimodal application for a chosen use case, with a model-selection rationale and an evaluation report.

• Case analysis (15%)

  Written analysis of a real generated-media incident and the technical and organisational controls that would have prevented it.

Pass mark 60%. Grades: Distinction, Merit, Pass.