AI Model Card Best Practices: What to Document and Why

An AI model card is a short, standardized document that records a model's intended use, performance benchmarks, ethical considerations, and known limitations. The best practices are: write one card per model version, document intended and out-of-scope uses explicitly, report performance broken down by demographic or data segment (not just an aggregate score), and treat the card as a living artifact updated on every retrain. Model cards were introduced by Mitchell et al. (2019) at Google and are now an industry-standard format used by OpenAI, Hugging Face, and AWS.

What goes in a model card (the required fields)

A useful model card is structured, not a free-text README. At minimum, include these sections drawn from the original Mitchell et al. (2019) template:

• Model details — owner, version, date, model type (see the five families below), training framework, and license.
• Intended use — the primary use case and the intended users. Be concrete: "ranking support tickets by urgency for internal agents," not "text classification."
• Out-of-scope use — the single most-skipped and most-valuable field. State what the model must not be used for (e.g., "not validated for medical triage").
• Performance — metrics disaggregated by relevant subgroup (age, geography, language, data source), not one headline accuracy number.
• Training data — provenance, collection window, known gaps, and consent/licensing basis.
• Ethical considerations & limitations — known failure modes, bias findings, and conditions under which performance degrades.

A worked example

Here is a compact example card for a fraud-detection model, showing the level of specificity to aim for:

• Model: fraud-scorer v3.2, gradient-boosted trees (XGBoost), trained 2024-Q4, owner: Payments Risk team.
• Intended use: real-time scoring of card-not-present transactions to flag for manual review.
• Out-of-scope: auto-blocking without human review; use on ACH or wire transactions.
• Performance: AUC 0.94 overall; 0.91 for transactions under $20; recall drops to 0.78 for merchants onboarded <30 days (thin data).
• Limitations: degrades during holiday spend spikes; no non-English merchant descriptors in training set.

Notice the disaggregated metrics and the explicit weak spots — that is what turns a card from marketing into a risk control. OpenAI's system cards for GPT models follow the same logic at scale, publishing red-team findings and refusal behaviors alongside benchmark scores.

The 30% rule and the five model families

Two things readers frequently conflate with model cards deserve a quick clarification. The "30% rule" is not a formal documentation standard — it's an informal governance heuristic some teams use, meaning roughly that if AI automates or influences more than ~30% of a decision, a human-in-the-loop review and documented card become mandatory. Treat it as a trigger threshold for your policy, not a metric to record.

The five main model families you'll document cards for are: (1) supervised learning models (classification/regression), (2) unsupervised models (clustering, anomaly detection), (3) reinforcement learning models, (4) generative/foundation models (LLMs, diffusion), and (5) deep neural networks spanning the above. Your card's "model type" and "limitations" fields differ sharply by family — a generative model needs prompt-injection and hallucination notes; a supervised classifier needs subgroup fairness metrics.

SageMaker Model Cards vs. AI Service Cards, and generators

If you're on AWS, know the difference between the two products with similar names. SageMaker Model Cards document your own models — you fill in intended use, evaluation results, and risk ratings, and they version alongside the model in the registry. AWS AI Service Cards are AWS-authored documentation for AWS's own managed AI services (like Rekognition or Textract); you consume them, you don't edit them. When you build on a managed service, cite the relevant AI Service Card inside your own model card's "training data" and "limitations" sections.

To scale documentation, use a model card generator rather than writing prose by hand. Options include Google's Model Card Toolkit (which produces cards in the Mitchell et al. format from evaluation metadata), Hugging Face's card templates for any model on the Hub, and SageMaker's built-in card creation. The best practice is to generate the skeleton automatically from your training pipeline so metrics and dataset lineage are captured programmatically, then have a human complete the intended-use and ethics sections that can't be inferred.

Not sure which of your models need a card first, or how much risk each carries? PlayCISO's free AI Model Risk Scanner helps you triage your model inventory and flag the ones that most urgently need documented cards.