AI model management coordinates models and the systems around them from proposal through retirement. A model artifact is not the complete AI system: production behavior also depends on data, prompts, retrieval, tools, policies, interfaces, infrastructure, people, and operating conditions.

The goal is controlled evidence, not perfect recall or guaranteed trust. Management should make identity, ownership, intended use, evaluation, approvals, deployment state, incidents, and retirement decisions discoverable and reviewable.

Maintain an inventory

Inventory internally trained, fine-tuned, open-weight, embedded, and third-party models, including pilots and shadow systems. For each entry, record an owner, purpose, prohibited uses, affected population, risk classification, provider or weight source, immutable version, license and terms, data boundary, dependencies, deployments, approval status, and retirement trigger.

A registry can store versions, aliases, lineage, and metadata, but it does not automatically provide governance, reproducibility, approval, or an immutable audit trail. Define required evidence and enforce controls around the registry.

Connect lineage and reproducibility

Link model or adapter weights to the base revision, tokenizer, prompt or chat template, datasets and checksums, feature definitions, code commit, environment or container, dependencies, configuration, seeds, training runs, evaluation harness, and approvals. Exact reproducibility can still be limited by nondeterministic hardware, distributed execution, unavailable services, or mutable dependencies; document the tolerance and restoration test.

For adaptation-specific evidence, see how to fine-tune an LLM.

Evaluate before approval

Prespecify representative and challenge cases, baselines, metrics, error severity, subgroup analysis, uncertainty, and acceptance criteria. Evaluate validity, reliability, safety, security, privacy, bias, explainability, accessibility, robustness, and cost proportionate to context. A single aggregate score or validation loss is insufficient.

Record who reviewed the evidence, unresolved limitations, conditions of approval, expiration or reassessment date, and who can accept residual risk. Use LLM evaluation metrics only where appropriate to the task.

Deploy as a controlled change

Separate model version, endpoint version, and full system release. Use immutable artifacts, integrity verification, environment-specific configuration, least-privilege identities, staged rollout, health checks, rollback targets, and change records. Canary or A/B exposure estimates effects only when assignment, interference, sample size, duration, guardrails, and analysis are valid; it is not a substitute for risk review.

Monitor systems, data, outcomes, and misuse

  • System: availability, latency distributions, throughput, errors, dependency failures, resources, and cost.
  • Data: schema, missingness, range, freshness, source changes, feature availability, distribution shift, and training-serving skew.
  • Outcomes: task performance with uncertainty, calibration where relevant, subgroup behavior, process outcomes, complaints, overrides, and incidents.
  • Security and misuse: anomalous access, integrity failures, abuse, extraction, prompt injection, tool misuse, and response evidence.

Distribution change does not inevitably harm performance, and stable distributions do not prove a model remains valid. Ground truth can arrive late. Thresholds should reflect baseline variation, sample size, uncertainty, consequence, and subgroup risk; an alert should open an investigation rather than trigger automatic retraining.

Retrain and retire deliberately

Retrain only when evidence supports it and approved data is available. Repeat independent evaluation, security checks, approval, controlled deployment, and rollback preparation. Automation can orchestrate evidence collection, but accountable people retain authority for consequential releases.

Retire models when their purpose ends, evidence expires, a provider withdraws support, risk becomes unacceptable, or a replacement is approved. Remove endpoints and credentials, preserve required records, apply retention and deletion rules, notify dependencies, and verify that traffic has stopped.

Protect the supply chain

Protect training data, repositories, registries, feature stores, artifacts, endpoints, secrets, signing keys, CI/CD, and administrative paths. Use strong authentication, least privilege, short-lived workload identity, integrity checks, vulnerability management, logging, separation of duties, incident response, and recovery based on risk.

Logs need defined events, actor and workload identity, integrity protection, time synchronization, retention, privacy controls, review, export, and links to change evidence. Apply the wider controls described in AI governance best practices.

Originally published June 25, 2025; technically reviewed and substantially updated September 4, 2026.