Fine-Tuning, Alignment and Model Customisation

About this course

Fine-tuning is powerful, expensive to maintain and often unnecessary. This course starts with the decision (prompting, retrieval, fine-tuning or a combination) and then teaches each customisation method properly: building and documenting datasets, supervised and parameter-efficient fine-tuning with LoRA and QLoRA, preference optimisation from RLHF to DPO and its successors, reinforcement learning for reasoning, and distillation into smaller models. Learners evaluate every tuned model against strong baselines and take responsibility for its data rights, documentation and lifecycle.

For: ml practitioner, software engineer, student.

Learning outcomes

On successful completion you will be able to:

1. 1Explain the methods for customising language models (supervised and parameter-efficient fine-tuning, preference optimisation, reinforcement learning and distillation) and the problem each one solves.
2. 2Analyse a customisation request and decide whether prompting, retrieval, fine-tuning or a combination is justified, quantifying expected gains, costs and risks.
3. 3Prepare high-quality training and preference datasets, including synthetic data, with documented provenance, licensing and quality controls.
4. 4Fine-tune an open-weight model with LoRA or QLoRA and align it with a preference-optimisation method on limited hardware.
5. 5Evaluate a customised model against its base model and a prompted baseline for task quality, regressions, safety and calibration, and decide whether to ship it.
6. 6Take responsibility for the legal, ethical and lifecycle implications of a customised model, including data rights, documentation and maintenance as base models change.

Modules and lessons

1. Week 1When and why to customise⌄

   • To fine-tune or not to fine-tunecase study
   • The customisation landscape
2. Week 2Data for customisation⌄

   • Building fine-tuning datasetslab
   • Synthetic data and preference data
3. Week 3Supervised and parameter-efficient fine-tuning⌄

   • Supervised fine-tuning in practice
   • LoRA, QLoRA and adapterslab
4. Week 4Alignment and preference optimisation⌄

   • From RLHF to direct preference optimisation
   • Reinforcement learning for reasoning and tool uselab
5. Week 5Distillation and evaluation⌄

   • Distilling large models into small ones
   • Evaluating a customised modellab
6. Week 6Responsible customisation in production⌄

   • Serving and maintaining customised models
   • Legal, ethical and documentation duties

Assessment

• Quiz (15%)

  Six weekly quizzes drawn from the lesson quiz banks; best five count.

• Lab (30%)

  Four graded labs covering dataset construction, QLoRA fine-tuning, DPO alignment and a ship-or-not evaluation.

• Project (40%)

  Customise an open-weight model for a provided task - dataset, training runs, an evaluation report against the base model and a prompted baseline, and a model card.

• Viva (15%)

  Twenty-minute review in which the learner defends the decision to fine-tune (or not) and the model card.

Pass mark 60%. Grades: Distinction, Merit, Pass.