From Research to Deployment

The Production Gap

The Reality

87% of ML projects never make it to production. The gap between a working prototype and a reliable production system is vast—filled with infrastructure, operations, monitoring, and organizational challenges that notebooks don't prepare you for.

Why It's Hard

In research, you optimize for model accuracy. In production, you optimize for reliability, latency, cost, maintainability, and the ability to update without breaking things. These are different disciplines.

Production Requirements

Reliability: 99.9% uptime, graceful degradation
Latency: p50 < 100ms, p99 < 500ms for many applications
Scalability: Handle 10x traffic spikes
Cost efficiency: Inference at scale is expensive
Reproducibility: Same code + data = same model
Observability: Know when things go wrong

What Changes

Research	Production
Static dataset	Streaming, changing data
Manual experiments	Automated pipelines
Accuracy matters	Latency, cost, reliability matter too
One environment	Dev, staging, prod, edge
Train once	Continuous training
Evaluate offline	Monitor in real-time

The MLOps Lifecycle

End-to-End ML System

📊

Data Pipeline

Data ingestion
Validation
Transformation
Feature engineering
Feature store

→

🔬

Training Pipeline

Experiment tracking
Hyperparameter search
Model training
Evaluation
Model registry

→

🚀

Deployment

Model packaging
Serving infrastructure
A/B testing
Canary releases
Rollback capability

→

📈

Monitoring

Performance metrics
Data drift detection
Model drift detection
Alerting
Feedback collection

← Continuous Feedback Loop →

Data Pipeline and Feature Engineering

In production, data is the most challenging part. It's messy, constantly changing, and the same features need to be computed identically for training and inference.

Data Validation

Before data enters your pipeline, validate it. Catch schema changes, null values, distribution shifts, and anomalies early.

Tools: Great Expectations, TFX Data Validation, Pandera

Schema validation (types, columns)
Statistical validation (ranges, distributions)
Semantic validation (business rules)
Freshness validation (data is recent)

Feature Stores

A centralized repository for features that ensures consistency between training and serving. Features computed once, used everywhere.

Tools: Feast, Tecton, AWS SageMaker Feature Store

Reuse: Share features across models
Consistency: Same computation for training/serving
Time-travel: Point-in-time correct features
Low latency: Precomputed for real-time serving

Training-Serving Skew

When features are computed differently in training vs serving, models perform worse in production than in evaluation. This is a silent killer.

Common causes:

Different code paths for offline/online computation
Data leakage in training (future data available)
Different preprocessing libraries or versions
Aggregation windows computed differently

◈

The Data Flywheel

Production systems should collect data that improves future models. Log predictions, outcomes, and user feedback. Build active learning pipelines that identify valuable examples. The best production systems get better automatically.

Experiment Tracking and Reproducibility

Reproducibility

Given the same code, data, and configuration, you should get the same (or statistically equivalent) model. Without reproducibility, you can't debug, compare, or reliably improve.

Code Versioning

Git for everything. Tag releases. Link commits to experiments.

Data Versioning

DVC, Delta Lake, or LakeFS. Version datasets like code.

Environment Versioning

Docker containers, conda environments. Pin all dependencies.

Experiment Tracking

MLflow, Weights & Biases, Neptune. Log everything automatically.

Model Registry

Central catalog of trained models with metadata and lineage.

What to Track for Every Experiment

✓ Git commit hash

✓ Data version/hash

✓ Hyperparameters

✓ Random seeds

✓ Environment/dependencies

✓ All metrics over time

✓ Model artifacts

✓ Training time/resources

Model Deployment Patterns

Batch Inference

Run predictions on large datasets periodically (hourly, daily).

Use when:

Predictions don't need to be immediate
Large volume of predictions
Complex models that are slow

Example: Nightly recommendation generation, fraud scoring for next-day review

Real-Time Inference (Online)

Serve predictions on-demand via API, typically sub-second latency.

Use when:

User-facing applications
Decisions needed immediately
Input data available at request time

Example: Search ranking, fraud detection at checkout, chatbots

Streaming Inference

Process continuous data streams, making predictions as events arrive.

Use when:

Data arrives continuously
Need to react to events quickly
State needs to be maintained

Example: Real-time anomaly detection, live video analysis

Edge Inference

Run models on device (phone, IoT, browser) rather than server.

Use when:

Privacy-sensitive data
No/unreliable network
Ultra-low latency required

Example: Face ID, voice assistants, autonomous vehicles

🛠️ New to DevOps? Quick Crash Course

What is an API?

An API (Application Programming Interface) is a way for programs to talk to each other. Think of it like a waiter at a restaurant: you tell the waiter what you want (request), and they bring it back (response). Your model becomes an API endpoint that receives data and returns predictions.

What is a Container (Docker)?

A container packages your code + all its dependencies (Python version, libraries, etc.) into a single portable unit. It's like a shipping container: everything inside is bundled together and works the same anywhere. Docker is the tool that creates and runs these containers.

What is Kubernetes (K8s)?

Kubernetes manages many containers across many machines. It handles scaling (run more copies when busy), healing (restart crashed containers), and routing (send requests to healthy instances). Think of it as the "operating system" for running containers in production.

Don't worry! You don't need to master all of this to understand ML systems. Most teams have dedicated infrastructure engineers. Your job is to know these exist and communicate effectively.

Serving Infrastructure Options

Model Servers

Dedicated servers for ML inference: TensorFlow Serving, Triton, TorchServe

Optimized for ML Batching, GPU support More infrastructure

Containers (K8s)

Package model + server in Docker, deploy on Kubernetes

Flexible Cloud-native Complexity

Serverless

AWS Lambda, Google Cloud Functions, Azure Functions

Zero ops Auto-scaling Cold starts Size limits

Managed ML Platforms

SageMaker, Vertex AI, Azure ML

Integrated Managed Vendor lock-in Cost

Hands-On: Deployment Artifacts

Here are production-ready templates for the most common deployment patterns. Copy and adapt these for your own projects.

📄 Dockerfile (FastAPI + Model)

# Dockerfile for ML Model Serving
FROM python:3.10-slim

WORKDIR /app

# Install dependencies first (cache layer)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy model and code
COPY model/ ./model/
COPY app.py .

# Expose port for inference
EXPOSE 8000

# Run with uvicorn
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

📄 app.py (FastAPI Server)

from fastapi import FastAPI
from pydantic import BaseModel
import torch

app = FastAPI()

# Load model at startup (not per-request!)
model = torch.load("model/model.pt")
model.eval()

class PredictionRequest(BaseModel):
    text: str

class PredictionResponse(BaseModel):
    prediction: str
    confidence: float

@app.post("/predict", response_model=PredictionResponse)
async def predict(request: PredictionRequest):
    with torch.no_grad():
        # Your preprocessing here
        output = model(preprocess(request.text))
        pred_class = output.argmax().item()
        confidence = output.softmax(dim=-1).max().item()
    
    return PredictionResponse(
        prediction=CLASS_NAMES[pred_class],
        confidence=confidence
    )

@app.get("/health")
async def health():
    return {"status": "healthy"}

📄 docker-compose.yml (Local Development)

version: '3.8'
services:
  ml-api:
    build: .
    ports:
      - "8000:8000"
    volumes:
      - ./model:/app/model  # Hot-reload models
    environment:
      - MODEL_PATH=/app/model/model.pt
      - LOG_LEVEL=debug
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3

📄 requirements.txt

torch==2.0.1
fastapi==0.100.0
uvicorn[standard]==0.23.0
pydantic==2.0.0
numpy==1.24.0

Quick Commands

Build: docker build -t my-model-api:v1 .

Run: docker run -p 8000:8000 my-model-api:v1

Test: curl -X POST http://localhost:8000/predict -H "Content-Type: application/json" -d '{"text": "Hello world"}'

Safe Deployment Strategies

Don't deploy new models directly to all users. Use progressive rollout strategies to catch problems before they affect everyone.

Shadow Deployment

Traffic →

Prod Model (serves) New Model (logs only)

Run new model in parallel, compare outputs without serving to users. Zero risk to users.

Canary Deployment

1% → 5% → 25% → 50% → 100%

Gradually increase traffic to new model. Monitor metrics at each stage. Rollback if problems appear.

A/B Testing

Group A: Old Model Group B: New Model

Randomly assign users to variants. Measure business metrics (not just ML metrics). Statistical significance before decision.

Blue-Green Deployment

Blue (current) ⇄ Green (new)

Maintain two production environments. Switch traffic instantly. Easy rollback by switching back.

Production Monitoring

The Monitoring Mandate

ML models degrade silently. Unlike traditional software that crashes when broken, ML models continue producing outputs—just worse ones. You won't know unless you monitor.

System Metrics

Traditional DevOps monitoring—infrastructure health.

Latency (p50, p95, p99)
Throughput (requests/sec)
Error rates
CPU/Memory/GPU utilization
Queue depths

Data Quality Metrics

Is the input data what the model expects?

Missing values
Schema violations
Out-of-range values
Data freshness
Volume anomalies

Data Drift

Has the input distribution changed since training?

Feature distributions (PSI, KL divergence)
Embedding drift
Covariate shift detection

Why it matters: Models trained on old distributions may perform poorly on new ones.

Model Performance

Is the model still accurate? (Requires ground truth)

Accuracy, precision, recall over time
Performance by segment
Prediction distribution changes
Concept drift detection

Challenge: Ground truth often delayed (did the user click? did the loan default?)

Business Metrics

Is the model achieving its business goal?

Conversion rates
Revenue impact
User engagement
Customer satisfaction

Key insight: ML metrics (accuracy) don't always correlate with business metrics.

Fairness Metrics

Are there disparities in how the model treats groups?

Performance by demographic
Prediction rate parity
Error rate parity

◈

Monitoring Without Ground Truth

When labels are delayed or unavailable, monitor proxy signals: prediction confidence distributions, prediction rate changes, user behavior (do users accept recommendations?), and input data drift. These can signal problems before you have ground truth.

Continuous Training and Feedback Loops

Static models decay. Production systems need mechanisms to retrain models on fresh data—automatically or with human oversight.

Scheduled Retraining

Retrain on a fixed schedule (daily, weekly, monthly). Simple but may retrain unnecessarily or not quickly enough.

Best for: Stable domains, predictable drift

Triggered Retraining

Retrain when drift or performance degradation is detected. More efficient but requires good monitoring.

Best for: Variable drift, cost-sensitive environments

Online Learning

Update model continuously as new data arrives. No discrete retraining—model always current.

Best for: High-frequency data, real-time adaptation

Caution: Vulnerable to data quality issues and adversarial inputs

Human-in-the-Loop Retraining

Fully automated retraining can propagate errors. Consider requiring human approval for:

Model promotions to production
Significant training data changes
Models in high-stakes domains
First deployment of new architectures

Common Misconceptions

✗

"If the model works in testing, it'll work in production"

Test environments differ from production in data distribution, scale, latency requirements, and edge cases. Models that ace offline evaluation often struggle with real-world messiness.

The accurate framing: Offline evaluation is necessary but not sufficient. Use shadow deployments, canary releases, and continuous monitoring to validate in production.

✗

"Deploy once and you're done"

The world changes. Data drifts. User behavior evolves. Competitors adapt. A model that's excellent today may be mediocre in three months.

The accurate framing: ML systems need continuous monitoring, regular evaluation, and mechanisms for updating. Deployment is the beginning, not the end.

✗

"The ML code is the hard part"

In production systems, ML code is typically 5-10% of the codebase. The rest is data pipelines, serving infrastructure, monitoring, testing, and configuration management.

The accurate framing: ML engineering is mostly engineering. Data quality, infrastructure reliability, and operational excellence determine production success.

⚗ Interactive Lab: Deployment Decision Tree

Answer questions about your use case to get deployment architecture recommendations.

What is the latency requirement?

Key Considerations

Latency: Determines real-time vs batch architecture
Data location: Privacy and connectivity constraints
Scale: Influences infrastructure complexity and cost
Model size: Large models need more resources

✓ Check Your Understanding

🎓

Congratulations!

You've completed the Foundations of Artificial Intelligence curriculum. You now have a rigorous understanding of:

What AI actually is and how intelligence emerges from optimization
Core ML methods: decision boundaries, neural networks, backpropagation
Modern architectures: embeddings, attention, transformers, LLMs
Generative AI: diffusion models, multimodal systems
Evaluation, responsible AI, and production deployment

This foundation prepares you to dive deeper into specialized areas: computer vision, NLP, reinforcement learning, AI safety research, or applied ML engineering. Keep learning, stay curious, and build responsibly.

Learning Objectives

The Production Gap

The MLOps Lifecycle

Data Pipeline

Training Pipeline

Deployment

Monitoring

Data Pipeline and Feature Engineering

Data Validation

Feature Stores

Training-Serving Skew

The Data Flywheel

Experiment Tracking and Reproducibility

Code Versioning

Data Versioning

Environment Versioning

Experiment Tracking

Model Registry

What to Track for Every Experiment

Model Deployment Patterns

Batch Inference

Real-Time Inference (Online)

Streaming Inference

Edge Inference

🛠️ New to DevOps? Quick Crash Course

Serving Infrastructure Options

Model Servers

Containers (K8s)

Serverless

Managed ML Platforms

Hands-On: Deployment Artifacts

Quick Commands

Safe Deployment Strategies

Shadow Deployment

Canary Deployment

A/B Testing

Blue-Green Deployment

Production Monitoring

System Metrics

Data Quality Metrics

Data Drift

Model Performance

Business Metrics

Fairness Metrics

Monitoring Without Ground Truth

Continuous Training and Feedback Loops

Scheduled Retraining

Triggered Retraining

Online Learning

Human-in-the-Loop Retraining

Common Misconceptions

"If the model works in testing, it'll work in production"

"Deploy once and you're done"

"The ML code is the hard part"

⚗ Interactive Lab: Deployment Decision Tree

Recommended Architecture

Key Considerations

✓ Check Your Understanding

Congratulations!