AI for DevOps & Platform Engineers

ML workflows differ from software workflows in fundamental ways — ignoring those differences causes production failures. This module maps MLOps to DevOps concepts you already know, then covers the new disciplines that have no software equivalent. The MLOps lifecycle: experimentation → training → evaluation → deployment → monitoring → retraining — how this maps to and diverges from the software delivery lifecycle. Experiment tracking with MLflow: logging metrics, parameters, and artefacts, comparing runs, and building reproducible experiments. Model versioning: semantic versioning for models, model lineage, and dependency tracking. Dataset versioning with DVC and Delta Lake. Model registry: staging environments (development → staging → production), approval workflows, and rollback mechanisms. The difference between real-time inference and batch inference — operational implications of each. Terraform patterns for MLOps infrastructure.

The gap between a working notebook model and a production-serving model API is where most ML projects die. This module covers model serving end-to-end. Docker for ML: model containers, large artefact management, layer caching, and multi-stage builds. Model serving frameworks: FastAPI (flexibility), Triton Inference Server (performance), and vLLM (LLM-specific). vLLM deep dive: serving open-source LLMs (Llama, Mistral, Qwen and similar) with continuous batching, PagedAttention, and tensor parallelism. LLM inference optimisation: quantisation (GGUF, AWQ, GPTQ, INT8), KV cache management, speculative decoding, and flash attention. Streaming inference with Server-Sent Events (SSE) and WebSocket patterns. Health checks, graceful shutdown, and Kubernetes readiness/liveness probes. API gateway integration: rate limiting, authentication, and request routing for model endpoints.

CI/CD for ML is more complex than CI/CD for software — you're testing code, model behaviour, data quality, and evaluation results simultaneously. ML pipeline orchestration: Kubeflow Pipelines, Prefect, and Airflow — when to use each. Automated evaluation gates: integrating RAGAS, custom LLM evaluation, and accuracy thresholds into pipelines as quality gates. Testing strategies: unit tests for preprocessing, integration tests for pipeline components, and end-to-end smoke tests. Deployment strategies: canary deployment, blue-green deployment, and shadow mode (run new model in parallel without serving outputs). Automated rollback: detecting quality regressions in production and triggering rollback to the previous model. GitHub Actions and GitLab CI patterns for ML pipelines. Data-triggered and drift-triggered retraining pipeline invocation.

GPU compute is the most expensive line item in any AI infrastructure budget — and most teams manage it poorly. GPU taxonomy: A100 vs H100 vs A10G vs T4 — when each is appropriate for training vs inference vs fine-tuning. Cloud GPU options across AWS (p3, p4, g5), Azure (NDv4, NCv3), and GCP — cost models and availability patterns. Spot and preemptible instances: checkpointing strategies, fault-tolerant training, and interruption handling. Inference cost optimisation: batching strategies, dynamic batching, request queuing, and throughput-latency trade-offs. Quantisation for cost reduction: INT8/INT4 quantisation — accuracy impact analysis and when it is acceptable. Multi-GPU and distributed inference: tensor parallelism and pipeline parallelism for large models. GPU utilisation monitoring: identifying underutilised compute. Cost allocation and chargeback: attributing GPU costs to teams and use cases.

Models degrade silently. Input distributions shift. Upstream data changes. Without systematic monitoring, production AI quietly delivers worse results. Types of drift: data drift (input distribution shifts), concept drift (relationship between inputs and outputs changes), model drift (behaviour changes without input changes). Statistical drift detection: Kolmogorov-Smirnov test, Population Stability Index, Jensen-Shannon divergence — implementation and threshold tuning. Feature drift monitoring for individual feature distributions. LLM-specific drift: prompt length distributions, output quality scores, token distributions, and response time patterns. Model performance monitoring over time. Building drift detection pipelines with Evidently AI and Whylogs. Automated retraining triggers: when to retrain vs rollback vs alert a human.

High-quality AI systems require high-quality features delivered in real time. This module covers the feature engineering infrastructure layer. Feature store architecture: online store (low-latency serving) vs offline store (historical training features) and the consistency problem between them. Platforms compared: Feast, Tecton, Hopsworks, and Vertex AI Feature Store. Feature engineering pipelines: batch and streaming feature computation. Apache Kafka for real-time feature computation: streaming transforms and windowing. Feature freshness guarantees: time-to-live, staleness monitoring, and consistency checks. Data versioning: DVC, Delta Lake, and Apache Iceberg for reproducible training datasets. Point-in-time correct data: avoiding training-serving skew. Data quality validation with Great Expectations.

AI systems need a different observability strategy from traditional software — tracking quality, cost, and behaviour, not just errors. LLM-specific metrics: tokens per request, cost per request, time-to-first-token, throughput, and cache hit rates. Prompt and response logging: what to log, storage cost, retention policies, and privacy considerations. Structured LLM log schemas for downstream analysis. Building observability dashboards with Grafana and Datadog. Distributed tracing for AI pipelines: tracing requests through retrieval → reranking → generation. Cost attribution per request, user, and feature. AI incident taxonomy: hallucination, quality degradation, cost spike, and security incidents. Building AI runbooks. SLOs, SLIs, error budgets, and on-call policies for AI systems.

AI infrastructure introduces new security challenges beyond traditional application security. Secrets management for AI: storing and rotating LLM API keys, model registry credentials, and database tokens using Vault or AWS Secrets Manager. Enforcement via pre-commit hooks and CI/CD scanning. Network security: VPC isolation, private endpoints, and network policies for model serving infrastructure. IAM policies for ML pipelines: least-privilege access to model artefacts, training jobs, and feature stores. Supply chain security: verifying model artefact provenance, checking checksums, and scanning for malicious weights. Container security: vulnerability scanning, distroless base images, and signed images. Audit logging for AI infrastructure: logging all model API calls, training jobs, and data access. Data encryption at rest and in transit for training data, artefacts, and inference logs.

Reference architectures for the most common production AI deployment patterns — with documented trade-offs, failure modes, and operational considerations. RAG system architecture: ingestion pipeline → vector store → retrieval → reranking → generation → caching. Index freshness, cache invalidation, and pipeline monitoring. AI agent backend architecture: stateless vs stateful agent backends, memory persistence, tool execution sandboxing, rate limiting for agent loops. Batch inference architecture: scheduling, parallelism, output storage, and cost optimisation. Real-time scoring: low-latency feature serving + model inference for sub-100ms predictions. Multi-model orchestration: routing, A/B testing infrastructure. Event-driven AI architecture using message queues. Disaster recovery for AI: backup strategies for vector stores, model artefacts, and training data. Cost-latency-quality trade-off framework.

Build and demonstrate a complete, production-grade MLOps platform integrating all modules. The capstone must include: MLflow experiment tracking and model registry; automated CI/CD with evaluation gate and canary deployment; production model serving with autoscaling; feature pipeline with real-time serving; drift detection and automated retraining; full observability with cost dashboards, quality monitoring, and distributed tracing; security hardening with secrets management, network isolation, and audit logging; incident runbook for 3 failure scenarios; and Terraform IaC for the entire platform. Demonstrated live with load testing, a simulated drift event triggering retraining, and a simulated security incident response.

Learning Outcomes

• ✓Build production MLOps infrastructure from scratch: experiment tracking, model registry, automated training pipelines, and model versioning on AWS, Azure, or GCP
• ✓Containerise, serve, and scale LLM and ML model APIs — achieving sub-150ms p95 latency with proper health checks, autoscaling, and zero-downtime deployments
• ✓Design and implement CI/CD pipelines for ML models with automated evaluation gates, canary deployments, shadow mode testing, and quality-threshold-based rollback mechanisms
• ✓Manage GPU infrastructure efficiently: instance selection, spot instance strategies, quantisation (GGUF, AWQ, GPTQ), batching, and cost attribution dashboards
• ✓Build comprehensive AI observability: token cost tracking, latency profiling, prompt logging, quality monitoring, and incident runbooks for AI system failures
• ✓Implement drift detection and automated retraining pipelines covering data drift, concept drift, and model performance degradation with configurable alerting
• ✓Build and operate feature stores and streaming data pipelines serving real-time and batch features to ML models and AI systems at production scale
• ✓Apply platform security for AI infrastructure: secrets management for LLM API keys, network isolation for model endpoints, IAM policies for ML pipelines, and supply chain security for model artefacts
• ✓Design reference architectures for RAG systems, AI agent backends, batch inference, and real-time scoring — with documented trade-offs and operational considerations

FAQs