Training programme overview
Advanced program to instrument and monitor AI systems: traces, tokens, costs, latency, response quality, incidents and SLOs to improve reliability and observability of LLM applications.
Establishes LLMOps foundations: relevant metrics (tokens, latency, cost), collection architecture and integration with ML/infra pipelines.
Introduces tracing for AI workflows: request correlation, multi‑model chains, cost attribution and execution reconstruction for post‑mortem analysis.
Explains quality monitoring: automated metrics, human sampling, drift detection and dashboards to track degradation and bias.
Covers operations and incident management: alerting, playbooks, post‑mortems and defining/measuring SLOs for LLM services.
Training · Advanced
Courses included in this training programme
LLMOps Foundations
Foundations of LLMOps and AI observability: differentiate MLOps, LLMOps and app observability, map the AI lifecycle (prompt, retrieval, tools, model, output, feedback), and set SLOs for quality, latency, cost and safety using logs, traces and metrics.
4 lessons · 20 activities · 0 exercises · 3 videos
Tracing AI Workflows
Tracing AI workflows: instrument model calls (provider, model, tokens, latency), trace RAG retrieval and reranking steps, capture tool calls and agent loops, and apply redaction to protect privacy in traces.
4 lessons · 24 activities · 12 exercises · 3 videos
Quality Monitoring
Quality monitoring: online and offline evaluation strategies, prompt and version tracking, feedback collection and annotation workflows, and regression gates before deployment to prevent performance degradation.
4 lessons · 20 activities · 0 exercises · 3 videos
Operations and Incidents
Operations and incident handling for LLM systems: this module addresses monitoring token cost spikes and latency, fallback routing for provider outages, safety-incident triage, and postmortems aimed at continuous reliability improvements for AI services in production.
4 lessons · 24 activities · 12 exercises · 3 videos
Access the platform
Open the catalogue to review course details, access conditions, and your personal progress. Sign in.