AIBOM Maturity Self-Assessment

Do you know every AI model and AI service you run, and does each one have an AIBOM?

1. Is there a register of every AI model and AI service in use?

   • 0: No list. AI use is discovered by accident: an invoice, an incident, a staff question.
   • 1: A one-off spreadsheet from a survey, not updated since.
   • 2: A maintained register of in-house models. AI features in bought products and staff-adopted AI tools are missing or partial.
   • 3: One register covering in-house models, third-party models and APIs, and AI features in bought software, each with a named owner.
   • 4: The register is fed automatically from the model registry, cloud accounts and procurement, and gaps are reviewed every month.

2. What share of production AI systems have an AIBOM?

   • 0: None.
   • 1: One or two, written by hand for a specific request such as a customer or an audit.
   • 2: Some in-house models have one. Third-party and vendor models have none.
   • 3: Every in-house production model has a machine-readable AIBOM (CycloneDX or SPDX). Each vendor model has a supplier AIBOM or a recorded gap.
   • 4: Every production AI system, built or bought, has a current AIBOM, and a release without one is blocked.

3. How are AIBOMs produced?

   • 0: They are not produced.
   • 1: Written by hand in a document or spreadsheet.
   • 2: Generated by a script someone runs on request. The output varies with who runs it.
   • 3: Generated in CI for every model release and stored with the model artefact.
   • 4: Generated in CI, validated against the CycloneDX or SPDX schema, and the build fails when a required field is missing.

4. How fast can you answer “which of our systems use model or library X?”

   • 0: We could not answer with confidence.
   • 1: Days, by asking each team.
   • 2: Within a week, by searching spreadsheets and code repositories.
   • 3: Within a working day, from a searchable inventory.
   • 4: Within an hour, by querying the stored AIBOMs.

5. Do you know which bought products contain AI, and which models they use?

   • 0: No.
   • 1: A few, from vendor marketing.
   • 2: Procurement asks whether a product uses AI, but not which model or provider.
   • 3: Contracts or questionnaires record the model provider, model family and where data is processed for each AI-enabled product.
   • 4: Suppliers must provide an AIBOM or equivalent and notify model changes, and this is checked at every renewal.

NIST AI RMF 1.0 subcategories this domain gives evidence for: GOVERN 1.6, GOVERN 6.1, MAP 4.1.

Can you say exactly which model is deployed and what it was built from?

1. Does every deployed model have an exact, unchangeable identity?

   • 0: Models are known by nickname (“the triage bot”).
   • 1: A name and a loose version (“v2”) in documents.
   • 2: Name and version in a model registry. The weights file is not hashed.
   • 3: Name, version and a SHA-256 hash of the weights are recorded in the AIBOM and the registry.
   • 4: The running service reports the hash it loaded, and deploy checks it against the AIBOM.

2. Is the base model recorded exactly?

   • 0: Not recorded.
   • 1: Family only (“built on Mistral”).
   • 2: Repository id recorded (for example mistralai/Mistral-7B-Instruct-v0.3), but not the exact revision.
   • 3: Repository id, exact revision, licence and source recorded in the AIBOM (CycloneDX pedigree.ancestors or equivalent).
   • 4: As level 3, plus the base model’s own card or AIBOM is linked and the base is re-checked when its publisher changes it.

3. Is the adaptation recorded: fine-tune method, merges, quantisation?

   • 0: Not recorded.
   • 1: Known to the team that did it, not written down.
   • 2: Described in prose, in a wiki page or model card.
   • 3: Method, key parameters (such as LoRA rank), training run id and resulting artefact recorded for each release.
   • 4: Recorded by the training pipeline and linked from the AIBOM. Merged and quantised variants each get their own identity.

4. Can you trace a model version back to the code, data and config that produced it?

   • 0: No.
   • 1: Partly, from people’s memory and chat history.
   • 2: Training code is in version control, but the commit used for a given release is not recorded.
   • 3: Each release records the training code commit, dataset versions and configuration used.
   • 4: The pipeline records the link itself, so it cannot be edited by hand after the fact.

NIST AI RMF 1.0 subcategories this domain gives evidence for: MAP 4.1, MANAGE 3.2.

Are training and evaluation datasets recorded with licence, sensitive content and a card?

1. Are training, fine-tuning and evaluation datasets listed for each model?

   • 0: No.
   • 1: Dataset names appear in prose only.
   • 2: In-house datasets named with a version. The base model’s training data is not mentioned.
   • 3: Each dataset is an AIBOM component with version, size, date range and hash. The base model’s data is recorded as disclosed by its supplier, with a link.
   • 4: Dataset entries come from the data platform and update when a dataset version changes.

2. Is the licence or legal basis for each dataset recorded?

   • 0: Not checked.
   • 1: Assumed: internal data is “ours”, public data is “free”.
   • 2: Checked for some datasets, once, at the time of use.
   • 3: Licence or legal basis recorded per dataset and reviewed by legal or privacy before training.
   • 4: As level 3, and use restrictions (no commercial use, retention end date) are machine-readable and checked before any reuse.

3. Is personal or sensitive data in each dataset recorded, with how it was handled?

   • 0: Unknown.
   • 1: “No personal data” is asserted but not checked.
   • 2: Known for some datasets. Handling such as de-identification is described informally.
   • 3: Each dataset records its sensitive data categories and the treatment applied (for example the de-identification method), signed off by privacy.
   • 4: As level 3, plus an automated scan for identifiers runs before training and the result is attached to the dataset record.

4. Do datasets have dataset cards (Datasheets, Data Cards, Croissant or similar)?

   • 0: None.
   • 1: A readme with a short description.
   • 2: A card for the main training set. Collection method and known gaps or bias are not covered.
   • 3: A card for every training and evaluation set covering source, collection, composition, known gaps and intended use.
   • 4: Cards are machine-readable (for example Croissant with the RAI vocabulary) and linked from the AIBOM.

NIST AI RMF 1.0 subcategories this domain gives evidence for: MAP 4.1, MEASURE 2.10.

Are model files, the inference stack and the tools a model can reach covered by security controls?

1. Are model files checked for unsafe formats and tampering before they are loaded?

   • 0: Models are downloaded and loaded as they are.
   • 1: Informal guidance to prefer safetensors.
   • 2: A policy bans pickle-based formats for third-party models, but nothing enforces it.
   • 3: The pipeline scans model files for unsafe formats and embedded code, and verifies hashes before deploy.
   • 4: Production loads only models whose hash and signature verify. Anything else is refused.

2. Is the software that runs the model covered by vulnerability management?

   • 0: No.
   • 1: Only the application’s SBOM. The inference stack (for example vLLM, PyTorch) is not in it.
   • 2: The inference stack is in the SBOM, but ML libraries are not monitored for advisories.
   • 3: Inference server, ML libraries and their versions are in the AIBOM and checked against vulnerability feeds.
   • 4: As level 3, with alerts routed to owners and patch deadlines tracked like any other production system.

3. Are the tools, agents, MCP servers and retrieval (RAG) sources each model can reach recorded?

   • 0: Not recorded.
   • 1: Known to the team that built it.
   • 2: Listed in design documents. Permissions are not recorded.
   • 3: Each tool, MCP server and retrieval source is a service in the AIBOM with endpoint, data classification and permissions (such as tickets:read, tickets:write).
   • 4: As level 3, and the deployed configuration is compared with the AIBOM so a new tool or permission is flagged.

4. Can incident responders tell which model version was running at a given time?

   • 0: No.
   • 1: By asking the team, from memory.
   • 2: From deploy logs, with effort. The logs are not linked to the AIBOM.
   • 3: Deploy records link each time window to a model version and its AIBOM, kept for the log retention period.
   • 4: Runtime logs carry the model version or hash per request, so responders can scope affected requests directly.

NIST AI RMF 1.0 subcategories this domain gives evidence for: MEASURE 2.7, MANAGE 4.1.

Is each model’s intended use, test results, safety testing and limitations recorded?

1. Is each model’s intended use, out-of-scope use and human oversight recorded?

   • 0: No.
   • 1: Implied in project documents.
   • 2: Stated in prose for some models.
   • 3: Every production model records intended use, out-of-scope uses and the human oversight step in its model card and AIBOM.
   • 4: As level 3, and out-of-scope uses are tested (for example refusal tests) before each release.

2. Are evaluation results recorded with the test set and the metric?

   • 0: No evaluation records.
   • 1: “Performs well” statements, no numbers.
   • 2: Numbers in slides or notebooks, without the test set version.
   • 3: Each release records metric, value and a named, versioned test set.
   • 4: Evaluations run in CI on every release, results are stored in the AIBOM, and a release below its thresholds is blocked.

3. Is safety and security testing (red-teaming, prompt injection, harmful output) recorded?

   • 0: None.
   • 1: Manual testing that nobody wrote down.
   • 2: A one-off red-team exercise before the first launch.
   • 3: Red-team and prompt-injection tests for each release, with findings and mitigations recorded.
   • 4: As level 3, with a maintained attack test set that runs automatically and results compared across versions.

4. Are known limitations and risks recorded with mitigations and owners?

   • 0: No.
   • 1: Known to the team only.
   • 2: Listed in the model card. Mitigations or owners are missing for some.
   • 3: Each limitation has a mitigation and an owner, in the model card and AIBOM (CycloneDX modelCard.considerations or equivalent).
   • 4: As level 3, reviewed at every release and linked to the risk register.

NIST AI RMF 1.0 subcategories this domain gives evidence for: MAP 1.1, MAP 2.2, MEASURE 2.1.

Are models signed, is provenance verifiable, and do changes update the AIBOM?

1. Are model artefacts signed?

   • 0: No.
   • 1: Stored in an access-controlled bucket. No signature.
   • 2: Signed by hand for some releases, with keys held by individuals.
   • 3: Signed in CI for every release (for example Sigstore with OpenSSF model signing), with the signature linked from the AIBOM.
   • 4: As level 3, and deploy verifies the signature and refuses unsigned or mismatched models.

2. Is build provenance produced and verifiable?

   • 0: No.
   • 1: Build logs exist and are kept for a short time.
   • 2: The pipeline writes provenance, but it is not signed or not kept.
   • 3: Signed build provenance (for example SLSA provenance as an in-toto attestation) for every release, linked from the AIBOM.
   • 4: As level 3, verified at deploy, and provenance for third-party models is requested and checked.

3. Do model changes go through change control that updates the AIBOM?

   • 0: Models are replaced without a record.
   • 1: Changes are announced in chat.
   • 2: Changes go through change tickets. The AIBOM is updated later, if at all.
   • 3: A new model version, dataset, tool or prompt template is a change that needs an updated AIBOM before release.
   • 4: As level 3, and the AIBOM difference between versions is reviewed and approved as part of the change.

4. Is it recorded who attests to each claim about a model, and on what evidence?

   • 0: Nobody has been asked.
   • 1: Vendor marketing statements.
   • 2: Self-attestation by the team that built it, unsigned.
   • 3: Each claim has a named attester (team, build platform, auditor) and evidence, kept with the AIBOM or in the evidence store.
   • 4: As level 3, with independent checks by internal audit or a third party on a sample each year.

5. Can you produce regulatory technical documentation from your records?

   • 0: No.
   • 1: It would be written from scratch on request.
   • 2: Partly. Data, evaluation and oversight sections would need research.
   • 3: Documentation for in-scope systems (for example EU AI Act Annex IV for a high-risk system) can be assembled from the AIBOM and cards within days.
   • 4: Documentation is generated from the same records and kept current with each release.

NIST AI RMF 1.0 subcategories this domain gives evidence for: MANAGE 4.1, MANAGE 3.1, GOVERN 1.1.