AI Infrastructure, Inference and FinOps

About this course

The infrastructure layer of AI architecture. Learners work from the hardware up: accelerators, memory and interconnects; the anatomy of LLM inference; batching, KV-cache management, quantisation and speculative decoding; serving engines such as vLLM, SGLang and TensorRT-LLM; and multi-tenant inference platforms. They then plan capacity, choose hosting (including EU sovereign options), model unit costs and apply FinOps and sustainability levers to defend an infrastructure decision to finance and leadership.

For: architect, ml practitioner, tech leader, software engineer.

Learning outcomes

On successful completion you will be able to:

1. 1Explain the accelerator, memory and interconnect landscape for AI (GPUs, TPUs and other accelerators, HBM, NVLink and InfiniBand) and how it shapes the cost of training and inference.
2. 2Analyse LLM inference performance in terms of prefill and decode, KV-cache memory, batching and parallelism, and identify the binding constraint for a given workload.
3. 3Evaluate serving engines and optimisation techniques (quantisation, speculative decoding, caching, batching) against latency, throughput, quality and cost targets using reproducible benchmarks.
4. 4Design a scalable, multi-tenant inference platform with routing, autoscaling, isolation and SLOs, including hosting options for EU sovereign and regulated environments.
5. 5Build a capacity plan and unit-cost model for an AI workload and recommend FinOps levers such as commitments, caching, model cascades and batch processing.
6. 6Take responsibility for infrastructure decisions that balance cost, performance, sustainability and vendor risk, and justify them to finance and leadership.

Modules and lessons

1. Week 1Hardware foundations⌄

   • The AI accelerator landscape
   • Training versus inference infrastructure
2. Week 2Inference performance⌄

   • Anatomy of LLM inferencelab
   • Batching, paged attention and KV-cache management
3. Week 3Optimisation techniques⌄

   • Quantisation and distillation in productionlab
   • Speculative decoding and other decoding accelerations
4. Week 4Serving platforms⌄

   • Serving engines: vLLM, SGLang and TensorRT-LLMlab
   • Designing a multi-tenant inference platform
5. Week 5Hosting, sovereignty and capacity⌄

   • Hosting choices and EU sovereignty
   • Capacity planning for AI workloads
6. Week 6FinOps and sustainability⌄

   • FinOps for AIlab
   • Energy, sustainability and the infrastructure business case

Assessment

• Quiz (15%)

  Six weekly quizzes drawn from the lesson quiz banks; best five count.

• Lab (30%)

  Three graded labs - inference benchmarking, a quantisation trade-off study and a serving-engine comparison - each with a reproducible notebook and a one-page findings note.

• Design review (35%)

  Inference platform design for a provided workload, with architecture views, SLOs, a capacity plan and a unit-cost model.

• Viva (20%)

  Fifteen-minute investment-committee defence of the platform design and hosting choice, with mentors acting as CFO and CTO.

Pass mark 60%. Grades: Distinction, Merit, Pass.