Real Model Cards, Reviewed and Scored

We scored five public model cards against one rubric on 29 October 2026. Llama 3.1 8B Instruct and Phi-4 scored 19 out of 20, gpt-oss-20b 18, Gemma 3 12B IT 17, and Mistral-7B-Instruct-v0.3 9. The scores measure how well each card documents its model, not how good or safe the model is.

Who this is for: AppSec and AI-security engineers approving models at intake, GRC leads who need evidence about third-party models, and CISOs setting a minimum documentation bar for AI suppliers.

The rubric

Ten criteria, each scored 0, 1 or 2, for a maximum of 20. It follows the sections of the original Model Cards paper and the Hugging Face template, with one addition security teams need: a named way to report problems. It is the same rubric the Model Card Scorer uses. The scores below were given by a person reading each card; the Scorer applies the rubric automatically by looking for headings, metadata keys and wording, so it can land a point or two away from a careful human read.

Bands: 17 to 20 strong, 12 to 16 adequate, 7 to 11 thin, 0 to 6 poor. Three rules kept the scoring consistent:

• We scored the card: the Hugging Face README.md with its YAML metadata, plus any document the card itself links as its model card or system card. We did not score technical reports, blog posts or the Hub page’s sidebar.
• “Specific” means it names a language, group, task, condition or number for this model. Caveats that fit any language model score 1.
• A policy counts when the card names and links it, such as an acceptable-use or prohibited-use policy.

The scores

Cards are edited after release. These are the revisions we scored, with the Hugging Face last-modified date:

Llama 3.1 8B Instruct: 19/20

What it gets right. The card follows the Model Cards paper closely; it even keeps the paper’s prompt “Where to send questions or comments about the model”. It states the developer (Meta), release date (23 July 2024), a custom licence with a link, intended uses, and out-of-scope uses that include “use in languages beyond those explicitly referenced as supported”. Benchmarks come with shots and metric names. Training compute is given in GPU hours with location-based emissions. The safety section describes red-teaming and uplift testing for CBRNE, child safety and cyber attack enablement, names safeguards (Llama Guard 3, Prompt Guard, Code Shield), and links an output reporting mechanism and a bug bounty.

What it misses. Training data: “a new mix of publicly available online data”, about 15 trillion tokens, cutoff December 2023. Size and cutoff are there; the sources are not.

Phi-4: 19/20

What it gets right. The most compact complete card of the five. A summary table gives developer, architecture (“14B parameters, dense decoder-only Transformer”), 16K context, 1,920 H100-80G GPUs for 21 days, 9.8T training tokens, a June 2024 cutoff and a 12 December 2024 release date. Training data is broken into four source types, including how the synthetic data was made, and links a separate data summary card. Limitations are specific: worse performance outside English, and code training concentrated on Python with common packages. The safety section names the techniques Microsoft’s AI Red Team tried, including jailbreaks, encoding-based attacks and multi-turn attacks.

What it misses. A named place to report problems.

gpt-oss-20b: 18/20

What it gets right. The Hugging Face page is mostly a quick-start. The substance is in the model card it links, a PDF dated 5 August 2025. That document gives the architecture down to layers and experts (20.9B total, 3.6B active parameters), a June 2024 knowledge cutoff, benchmark results, and safety results in tables, including disallowed-content, jailbreak and instruction-hierarchy evaluations. Its limitations are measured: on SimpleQA the 20b model’s hallucination rate was 0.914, and the card warns that the unrestricted chain of thought can contain hallucinated or policy-violating content and should not be shown to users without filtering. It also says the models are “not intended for the diagnosis or treatment of disease”.

What it misses. Training data is described by topic (“trillions of tokens, with a focus on STEM, coding, and general knowledge”), not by source. No feedback channel on either document. And a reviewer who reads only the Hugging Face page would miss almost everything: follow the “Model card” link.

Gemma 3 12B IT: 17/20

What it gets right. Training data is listed by component (web documents in over 140 languages, code, mathematics, images), with 12 trillion tokens for the 12B model and filtering steps for CSAM and personal data. It links its terms of use and the Gemma Prohibited Use Policy. The safety section describes red-teaming by several teams, “assurance evaluations” held apart from the development team, and states that “all testing was conducted without safety filters”.

What it misses. No architecture description, only sizes and context length. The Limitations section is generic; the one specific limitation sits in the safety results: “A limitation of our evaluations was they included only English language prompts.” Misuse reporting is mentioned but not linked or named.

Mistral-7B-Instruct-v0.3: 9/20

What it gets right. The YAML metadata gives the licence (Apache-2.0) and the base model (mistralai/Mistral-7B-v0.3). The card lists what changed from v0.2 (extended vocabulary to 32,768, v3 tokenizer, function calling). And it states the most important fact plainly: the model “does not have any moderation mechanisms”.

What it misses. No training data, no evaluation results, no intended or out-of-scope uses beyond calling the model “a quick demonstration”, no architecture and no feedback channel. Most of the card is installation and code.

What the scores say

• The big-lab cards are strong. Four of five scored 17 or more.
• Feedback channels are the most common gap. Only Llama names one. A security team that finds a jailbreak or a poisoned output needs somewhere to send it.
• Training-data sources stay vague. Every card except Mistral’s gives a size or a cutoff. Only Phi-4 and Gemma name source types clearly enough for a licensing or privacy review.
• Popular is not the same as documented. Mistral-7B-Instruct-v0.3 is a common base for fine-tunes and has the thinnest card here. Every fine-tune built on it inherits those gaps.
• Read the linked documents. gpt-oss-20b would score far lower on its Hugging Face page alone.

Worked example: what Northwind does with a 9/20 base model

Northwind Health (fictional) builds Triage Assistant v2.3 on Mistral-7B-Instruct-v0.3, the lowest-scoring card here. The gaps do not block the project. They decide what Northwind records itself in its AIBOM and what it asks the supplier.

Northwind Health and its Triage Assistant are a fictional example used across the PlayCISO AIBOM Library. All names, figures and evaluation results for it are invented for illustration.

The pattern generalises: copy what the card states, mark what it does not as “not disclosed”, and turn each gap into a question with the AI Supplier Questionnaire. Where the base model sits in the AIBOM is covered in AIBOM identity, provenance and lineage. To practise spotting the gaps, try the Spot the Gaps: AIBOM Lab. For the templates behind these cards, see Model card templates compared, and for every guide by role, the AIBOM Library hub.

Frequently asked questions

What makes a good model card?

It lets a reader decide whether to use the model without asking the developer. In practice: identity and version, an unambiguous licence, intended and out-of-scope uses, specific training-data sources with a cutoff, evaluation numbers, limitations specific to the model, what was safety-tested, technical specifications, machine-readable metadata and a way to report problems.

Which open model has the best model card?

Of the five we scored on 29 October 2026, Meta’s Llama 3.1 8B Instruct and Microsoft’s Phi-4 both scored 19 out of 20, OpenAI’s gpt-oss-20b 18 and Google’s Gemma 3 12B IT 17. Mistral-7B-Instruct-v0.3 scored 9. This measures documentation, not model quality, and cards change after release.

Does a good model card mean the model is safe?

No. The score measures whether the card tells you what you need to know. A model can be well documented and unsuitable for your use, and a thinly documented model can work well. Use the card to decide what to test and what to ask, not as the test.

How do I score a model card?

Use a fixed rubric so different reviewers get the same answer. Ours has ten criteria scored 0, 1 or 2, for a maximum of 20; the full rubric is on this page and in the Model Card Scorer. Record the date and the card revision you scored, because cards are edited after release.

Is a model card enough for AI vendor due diligence?

No. It is the supplier’s own claim. Use the gaps it leaves to drive your questions, verify what you can (hashes, licence, the version you deploy), and record the answers in your own AIBOM so they survive a change of supplier.

Why does the Mistral-7B-Instruct-v0.3 card score low?

Its card is mostly installation and code examples. When we checked it, it gave no training data, no evaluation results, no intended or out-of-scope uses and no feedback channel. It did state the licence, the base model and one important limitation: the model has no moderation mechanisms.

Sources

• meta-llama/Llama-3.1-8B-Instruct
• microsoft/phi-4
• openai/gpt-oss-20b
• OpenAI, gpt-oss-120b & gpt-oss-20b Model Card (arXiv 2508.10925, dated 5 August 2025)
• google/gemma-3-12b-it
• mistralai/Mistral-7B-Instruct-v0.3
• Mitchell et al., Model Cards for Model Reporting (FAT* 2019)
• huggingface_hub: model card template

Checked on 29 October 2026. PlayCISO is not affiliated with any organisation named here.