Last reviewed: September 4, 2026. This guide focuses on predictive machine-learning systems. Generative-AI applications add prompt, retrieval, safety, evaluation, and token-cost concerns.

What MLOps is for

MLOps applies software, data, and operational engineering to the machine-learning lifecycle. The aim is not “seamless deployment”; it is to make model changes reproducible, reviewable, testable, observable, reversible, and owned. Google Cloud describes MLOps as automation and monitoring across integration, testing, release, deployment, infrastructure, and continuous training.

1. Define ownership, risk, and acceptance criteria

  • Name owners for data, model behavior, platform, security, compliance, and business outcomes.
  • Document intended use, excluded use, affected users, error costs, dependencies, human oversight, and fallback behavior.
  • Set release criteria before training: data checks, offline metrics, subgroup tests, latency, cost, robustness, privacy, and safety thresholds.
  • Use stronger review and approval for higher-impact systems.

2. Capture end-to-end lineage

Conceptual illustration of versioned code, data, configuration, and model artifacts
Reproducibility requires traceable relationships among code, data, features, configuration, environment, evaluation, and the released artifact.

Assign immutable identifiers to source code, training and evaluation data snapshots or queries, feature definitions, configuration, dependencies, container images, model artifacts, evaluation reports, approvals, and deployments. Record who produced each artifact, when, with which inputs, and why it was promoted.

Git is appropriate for source and small text artifacts; large or sensitive data and model binaries usually need an artifact store, registry, access controls, checksums, retention policy, and provenance metadata. Versioning improves traceability but does not by itself guarantee exact reproduction.

3. Separate CI, training, and release

Conceptual continuous-integration, training, validation, and release pipeline
A production pipeline should create and evaluate a candidate before a controlled release.
  1. Continuous integration: test code, schemas, transforms, contracts, packaging, infrastructure definitions, and security controls.
  2. Training pipeline: create an immutable candidate from approved inputs in a reproducible environment.
  3. Evaluation gate: compare the candidate with the current champion and predefined baselines on frozen datasets and risk checks.
  4. Continuous delivery: make the approved artifact deployable with documented rollback; do not silently retrain during deployment.
  5. Controlled release: use shadow, canary, blue/green, or phased rollout when appropriate, with stop conditions and an accountable approver.

4. Test data and features as production dependencies

  • Validate schema, types, units, ranges, category changes, missingness, freshness, joins, duplicates, and target leakage.
  • Enforce point-in-time correctness: training features must reflect what would have been available when each prediction was made.
  • Test training/serving transformation parity and establish contracts with upstream producers.
  • Choose a feature store only when reuse, online serving, governance, or point-in-time retrieval justifies its cost. A versioned batch pipeline may be simpler.

5. Evaluate the whole system

Model quality is only one release gate. Test preprocessing, postprocessing, API contracts, failure behavior, authorization, dependency vulnerabilities, resource limits, latency, throughput, cost, observability, and rollback. Evaluate task metrics on representative data, include uncertainty, compare relevant subgroups, and preserve an untouched final test where feasible.

A/B or interleaving tests can measure user and business outcomes, but they require an appropriate experimental design. A canary deployment primarily limits blast radius; it does not automatically establish causal business impact.

6. Monitor what can change

Conceptual monitoring view for data, predictions, outcomes, service health, and risk
Monitor the data-model-service-decision system, not only an accuracy number.
  • Service: availability, latency, throughput, errors, saturation, and cost.
  • Data: schema violations, freshness, missingness, ranges, categorical changes, and pipeline failures.
  • Predictions: score distribution, abstentions, policy overrides, and segment coverage.
  • Outcomes: performance and calibration when labels arrive, business KPIs, harmful impacts, complaints, and appeals.
  • Security and governance: access, data exfiltration indicators, unexpected dependencies, approval bypass, and audit completeness.

Choose thresholds from a risk model and baseline behavior. Drift is evidence of change, not a diagnosis: investigate upstream changes, seasonality, population shifts, and label quality before acting.

7. Automate candidate retraining, not blind promotion

A schedule, new labeled data, performance degradation, policy change, or drift alert may open an investigation or launch a candidate-training pipeline. The candidate must pass the same data, quality, risk, security, and operational gates as any other release. Keep the current model available until the candidate is approved, and support rollback or fallback if the new release fails.

8. Manage infrastructure and supply-chain risk

  • Review infrastructure as code, pin and scan dependencies, sign or checksum artifacts, separate environments, and apply least-privilege identity.
  • Keep secrets out of images, repositories, notebooks, logs, and model artifacts.
  • Test restoration and rollback. Track infrastructure drift and provider/runtime changes.
  • Choose batch, online, streaming, embedded, or edge serving from latency, freshness, privacy, resilience, and cost requirements.

9. Make collaboration auditable

Use code review, reproducible scripts, experiment records, decision logs, shared metric definitions, role-based access, and explicit handoffs. Notebooks can support exploration, but production transformations and training should be testable and automation-friendly. Preserve why a model was selected, not only the winning metric.

10. Plan rollback and retirement

Every production model needs a runbook, escalation path, fallback, rollback target, data-retention plan, and retirement criteria. Decommission unused endpoints, credentials, features, dashboards, and artifacts according to policy. A model that no longer has an accountable owner should not remain in production.

A minimum production-readiness record

  • intended and excluded uses;
  • owners and approvers;
  • artifact and data lineage;
  • evaluation datasets, metrics, uncertainty, and subgroup results;
  • security, privacy, and compliance review;
  • deployment strategy, stop conditions, fallback, and rollback;
  • monitoring signals, thresholds, and response owners;
  • retraining, reapproval, and retirement policy.

Conclusion

MLOps is effective when it creates evidence and control around change. Version everything needed for lineage, test data and software, evaluate a candidate before release, limit rollout risk, monitor the whole decision system, and make retraining subject to validation. Add platforms such as registries, feature stores, Kubernetes, or managed services only when the workload and organization justify them.