AIBOM Evaluation, Intended Use and Risk

The evaluation and intended-use fields of an AIBOM tell a buyer, an auditor or your own SOC what a model is fit to do: what it was measured on and how well, who should use it for what, what it must not be used for, and what can go wrong. In CycloneDX they sit in modelCard.quantitativeAnalysis and modelCard.considerations; in SPDX 3.0 in metric, informationAboutApplication, limitation and safetyRiskAssessment. Below is each field, how to fill it so it can be checked, and the fictional Northwind numbers with their gaps.

Who this is for: GRC leads who need this content as evidence, and CISOs who approve or buy models and must decide whether a vendor’s numbers mean anything.

The fields across formats, and where the AI Act asks for them

Field names are from the CycloneDX 1.7 schema, SPDX 3.0.1 and the Hugging Face model card spec and template. The last column shows where the same content appears in the AI Act text for high-risk systems.

CycloneDX is strongest on evaluation: it is the only format with fields for a confidence interval, trade-offs and per-group fairness assessments. SPDX has the only fields for a decision threshold, autonomy and a risk rating. The Hugging Face card is prose with a structured model-index; for public benchmarks, the Hub also has a newer, work-in-progress .eval_results/ format that feeds benchmark leaderboards, which does not fit a private test set like Northwind’s.

Northwind’s evaluation and intended use

Northwind Health (fictional) evaluated Triage Assistant v2.3 on its own test set. Here is the model card block from its CycloneDX file:

{
  "quantitativeAnalysis": {
    "performanceMetrics": [
      { "type": "answer-accuracy", "value": "0.87", "slice": "northwind/triage-eval-v4" },
      { "type": "toxicity-rate", "value": "0.002", "slice": "internal red-team prompt set" }
    ]
  },
  "considerations": {
    "users": ["Northwind Health customer support agents"],
    "useCases": [
      "Draft replies to customer support tickets for a human support agent to review, edit and approve before sending."
    ],
    "technicalLimitations": [
      "Out of scope: Medical advice or clinical triage of symptoms.",
      "Out of scope: Sending any reply to a customer without human approval."
    ],
    "ethicalConsiderations": [
      {
        "name": "A drafted reply could contain medical advice.",
        "mitigationStrategy": "System prompt refuses medical advice; a human agent approves every reply before it is sent."
      },
      {
        "name": "Free-text tickets can contain personal data.",
        "mitigationStrategy": "Training data de-identified before fine-tuning; replies are drafts that never leave without human review."
      }
    ]
  }
}

Excerpt, 28 lines: the model card’s quantitativeAnalysis and considerations from Northwind’s CycloneDX 1.7 file. Keys not shown are left out; every value shown is exactly as in the full file (519 lines pretty-printed), which is in the Pro AIBOM Template Library.

The SPDX file says the same with different fields: ai_metric as key/value pairs, ai_informationAboutApplication, a single ai_limitation string, ai_autonomyType: no and ai_safetyRiskAssessment: medium.

Five rules that make a metric checkable

1. Tie every number to a dataset version

Northwind’s accuracy names its slice, northwind/triage-eval-v4, and that dataset is a component with version 4.0.0 and a hash. Good. Its toxicity rate names “internal red-team prompt set”, which is not in the AIBOM at all. A reader cannot tell what it was measured on.

2. Define the metric

“answer-accuracy” is Northwind’s own name. The AIBOM does not say what makes a draft correct, who graded it, or against which reference answer. Annex IV point 4 asks providers of high-risk systems to describe why their performance metrics are appropriate; you cannot do that without a definition. Put it in the model card and link it.

3. State the uncertainty

An accuracy of 0.87 on 1,200 tickets has a 95% interval of about 0.851 to 0.889 (normal approximation: ±1.96 × √(0.87 × 0.13 ÷ 1,200) ≈ ±0.019). CycloneDX has a field for it:

{
  "performanceMetrics": [
    {
      "type": "answer-accuracy",
      "value": "0.87",
      "slice": "northwind/triage-eval-v4",
      "confidenceInterval": { "lowerBound": "0.851", "upperBound": "0.889" }
    },
    { "type": "toxicity-rate", "value": "0.002", "slice": "internal red-team prompt set" }
  ]
}

Northwind’s metrics with a confidence interval added. We computed the interval for this post from the fictional values; it validates against the CycloneDX 1.7 schema inside the full Northwind BOM.

The toxicity rate cannot get an interval, because the size of the red-team set is not recorded. That matters: 0.2% of 500 prompts is one toxic output; 0.2% of 5,000 is ten. Same number, very different evidence.

4. Slice it

Article 13(3)(b)(v) of the AI Act asks, where appropriate, for performance “regarding specific persons or groups of persons on which the system is intended to be used”, and Annex IV point 3 asks for accuracy for specific persons or groups. Northwind reports one overall number. Its data has a channel field (email, web form or chat) and a category field, so per-channel and per-category accuracy are easy to add as extra slices. Per-customer-group accuracy is harder: the de-identification that protects customers also removed the attributes you would slice on. Write that tension down in fairnessAssessments rather than leaving the field empty.

5. Name the threshold and the judge

A toxicity rate needs a classifier and a cut-off. SPDX has ai_metricDecisionThreshold for exactly this: “each metric might be computed based on a decision threshold”. Northwind records neither the classifier nor the threshold.

Intended use and out of scope: write them to be enforced

Northwind’s intended use is one sentence: “Draft replies to customer support tickets for a human support agent to review, edit and approve before sending.” Its out-of-scope uses are “Medical advice or clinical triage of symptoms” and “Sending any reply to a customer without human approval”. Each one maps to a control you can check:

• No sending without approval → SPDX ai_autonomyType: no, and an Approve button in the agent console.
• No medical advice → a refusal in the system prompt plus human review, recorded as an ethicalConsiderations entry with its mitigation.

If an out-of-scope line cannot be tied to a control, it is a disclaimer, not a limitation.

Inherited limitations. The base model’s card says of Mistral-7B-Instruct-v0.3: “It does not have any moderation mechanisms.” That limitation travels down the lineage to Northwind’s fine-tune. Northwind’s toxicity testing and human approval address it, but its AIBOM never states the inherited limitation, so a reader cannot connect the control to the reason for it. Carry base-model limitations forward with your own mitigation next to each. The identity and lineage section tells you which cards to read.

The risk fields, and one trap

SPDX’s ai_safetyRiskAssessment takes serious, high, medium or low. The spec says the scale follows the EU general risk assessment methodology for product safety, which “implements Article 20 of Regulation (EC) No 765/2008”, and that “this categorization differs from the one proposed in the EU AI Act’s provisional agreement”. So Northwind’s medium is Northwind’s own product-safety rating. It is not an AI Act tier, and it does not say the system is high-risk. Northwind’s AI Act position belongs in the compliance record, worked out with a tool such as the EU AI Act Role & Risk Classifier.

CycloneDX has no overall rating. It records named risks instead: ethicalConsiderations pairs each risk with a mitigation, and fairnessAssessments names the group at risk, the benefits, the harms and the mitigation. For a buyer, named risks with mitigations say more than a single word.

How a buyer reads this section

Six questions for a CISO approving a model, or reading a vendor’s AIBOM:

1. Is every metric tied to a dataset you can identify, or at least describe?
2. Is the metric defined, and does it measure what your use needs?
3. Is there an interval, or at least the size of the test set?
4. Is performance reported for the groups and settings you will use it in?
5. Do the out-of-scope uses include how you plan to use it?
6. Are the base model’s limitations carried forward, and is the scale of any risk rating stated?

The same questions, asked across real public cards, are in Real model cards, reviewed and scored. For the card formats themselves, see Model card templates compared.

Where this lands in the EU AI Act

For providers of high-risk AI systems, this section feeds three duties. Article 13(3)(b) requires the instructions for use to state the intended purpose, the tested level of accuracy “including its metrics”, foreseeable misuse and, where appropriate, performance for specific groups. Article 15(3) requires that “the levels of accuracy and the relevant accuracy metrics … shall be declared in the accompanying instructions of use”. Annex IV points 2(g), 3 and 4 ask for the testing procedures and signed test reports, the capabilities and limitations, and why the metrics are appropriate. Northwind’s assistant is not high-risk, as our Annex IV walkthrough explains, but the same fields would carry the evidence if it were.

Evaluation and intended-use checklist

• Every metric names the dataset and version it was measured on.
• Every metric has a written definition, and any threshold is recorded.
• Every metric has an interval, or the test-set size to compute one.
• Results are sliced by the groups and settings that matter, or the reason they are not is recorded.
• Intended users and uses are specific; each out-of-scope use maps to a control.
• Base-model limitations are carried forward with your mitigation.
• Any risk rating states its scale; named risks have mitigations.

Fictional example. Northwind Health, Triage Assistant and its datasets, services, scores and URLs are invented for illustration; the base model and open-source libraries are real.

The rest of the AIBOM

Evaluation and intended use is one section of four. Identity and lineage says which model and which base. The data section documents the evaluation set behind every metric. Security and runtime covers what the model can reach and how its files are signed. To see these fields across every format, use the AIBOM Format Field Map (Pro). All guides by role are on the AIBOM Library hub.

Frequently asked questions

What is the difference between intended use and out-of-scope use in a model card?

Intended use says who the model is for and what it should do, for example drafting support replies for a human agent. Out-of-scope use names uses the provider rules out, for example medical advice or sending replies without approval. Out-of-scope uses should be specific enough that a control can enforce each one.

Where do evaluation results go in a CycloneDX ML-BOM?

In modelCard.quantitativeAnalysis.performanceMetrics on the model component. Each entry has a type, a value, a slice naming the data it was measured on, and an optional confidenceInterval with lowerBound and upperBound. Values are strings in the schema.

What does safetyRiskAssessment mean in SPDX 3.0?

It records a general safety risk level for the AI system: serious, high, medium or low. The SPDX spec says the scale follows the EU general risk assessment methodology for product safety (Article 20 of Regulation (EC) No 765/2008) and that it differs from the AI Act’s categorisation. It is not an AI Act risk tier.

Do I need confidence intervals for model metrics?

You should give some measure of uncertainty. An accuracy of 0.87 measured on 1,200 examples has a 95% interval of roughly 0.85 to 0.89; the same figure on 100 examples would be far less certain. CycloneDX has a confidenceInterval field for this; SPDX and Hugging Face cards do not, so state it in the metric’s text.

What does the EU AI Act require about model accuracy?

For high-risk AI systems, Article 15(3) requires the levels of accuracy and the relevant accuracy metrics to be declared in the instructions for use, and Article 13(3)(b) requires those instructions to cover the intended purpose, the tested level of accuracy, foreseeable misuse and, where appropriate, performance for specific groups. Annex IV points 2(g), 3 and 4 ask for the testing evidence.

Should a fine-tuned model repeat its base model’s limitations?

Yes, unless you have removed them and can show it. Limitations travel down the lineage like licences do. Record each inherited limitation with what your fine-tune or your system does about it, so a reader does not have to open the base model’s card to find it.

Sources

• CycloneDX 1.7 JSON schema (bom-1.7.schema.json): component, pedigree, modelCard, componentData, service, vulnerability
• SPDX 3.0.1: AIPackage
• SPDX 3.0.1: safetyRiskAssessment
• SPDX 3.0.1: metricDecisionThreshold
• SPDX 3.0.1: autonomyType
• Hugging Face model card template
• Hugging Face model card metadata spec (license, base_model, model-index)
• Hugging Face Hub docs: evaluation results (.eval_results/, work in progress)
• AI Act Article 13 (AI Act Service Desk)
• AI Act Article 15 (AI Act Service Desk)
• AI Act Annex IV (AI Act Service Desk)
• Mitchell et al., Model Cards for Model Reporting (FAT* 2019)
• mistralai/Mistral-7B-Instruct-v0.3 on Hugging Face (model card, Apache-2.0, base_model)

Checked on 29 October 2026. PlayCISO is not affiliated with any standards body, vendor or project named here. This is general information, not legal advice.