AI Fairness, Bias & Explainability Self-Assessment
The part of an AI audit most tooling skips: trustworthiness. Work through a checklist across the eight dimensions an AI auditor checks for fairness and explainability — protected attributes and affected groups, fairness metrics chosen and measured, bias sources across the three NIST SP 1270 categories, dataset representativeness, explainability fit to the audience, transparency artifacts, human oversight and contestability and ongoing monitoring. You get a banded 0–100 transparency score, a radar of where you are strong and weak, and the gaps ranked so you close the ones that matter most first — plus a Markdown export structured as a draft model-card. It is an educational self-assessment, not a fairness audit or legal advice; the weighting is a transparent heuristic, nothing is uploaded, and the result is shareable by permalink.
Fairness and explainability are largely unexamined. No group-fairness metric is chosen or measured, bias sources are untracked, and there is no model card or contestability route. An auditor would have nothing to review. Start by naming the affected groups and choosing a fairness metric.
Score per dimension (0–100)
Common group-fairness definitions
They measure different things — and generally cannot all hold at once. Choosing one is a deliberate value judgement for your use case.
The rate of positive decisions is equal across groups, regardless of the true label. Ignores base-rate differences between groups.
True-positive and false-positive rates are equal across groups — the model is equally accurate for each group on both error types.
A relaxation of equalized odds: only the true-positive rate is equalised across groups (equal chance for those who qualify).
For a given score, the probability of the true positive is the same across groups — equal precision / positive predictive value.
These definitions capture different notions of fairness and are generally mathematically incompatible: except in degenerate cases (e.g. equal base rates or a perfect classifier), you cannot satisfy demographic parity, equalized odds and predictive parity at the same time. The choice of which to prioritise is a value judgement for the use case, not a purely technical one.
Prioritised gaps
Highest-weight, lowest-scoring controls first — spend remediation effort here.
- Fairness metrics chosen & measured · unanswered
Pick and justify a fairness metric for the decision at hand. Demographic parity, equalized odds, equal opportunity and predictive parity measure different things and generally cannot all hold at once — choose on purpose.
Standard referenced: ISO/IEC TR 24027
- Fairness metrics chosen & measured · unanswered
Measure the chosen metric per group and intersectional sub-group and report the numbers — a qualitative "we think it is fair" is not a measurement.
- Fairness metrics chosen & measured · unanswered
Report group-wise error rates; a high overall accuracy can hide a much worse false-negative rate for one group (the classic disaggregated-evaluation finding).
- Protected attributes & affected groups · unanswered
List the protected and sensitive attributes relevant to the decision, including likely proxies (postcode, device, name) even where the attribute itself is not collected.
Standard referenced: ISO/IEC TR 24027
- Protected attributes & affected groups · unanswered
Define the affected groups AND their intersections — bias can concentrate in a sub-group that each single-attribute slice looks fine for.
- Bias sources assessed (systemic / statistical / human) · unanswered
Examine systemic bias — whether the historical process that produced your labels already disadvantaged a group, so even a "perfect" model would reproduce it.
Standard referenced: NIST SP 1270
- Bias sources assessed (systemic / statistical / human) · unanswered
Examine statistical/computational bias — non-representative sampling, a label that is a poor proxy for the true target, and objective functions that trade a group off.
Standard referenced: NIST SP 1270
- Explainability approach fits the audience · unanswered
State who needs an explanation and why; the explanation a regulator needs is not the one an affected individual needs.
Standard referenced: ISO/IEC TS 6254
+ 24 more gaps in the full export.
Protected attributes & affected groups · weight 14
You cannot assess fairness toward groups you have not named. Identify the protected and sensitive attributes in play, the people the system decides about, and the harms they could suffer.
The protected and sensitive attributes relevant to this system are explicitly identified (e.g. age, sex, race/ethnicity, disability, and any context-specific proxies).
The affected groups and sub-groups are defined, including intersectional groups (e.g. older women, not only "women" and "older people" separately).
The potential harms to each group are described (e.g. denial of a benefit, higher error rate, exposure to a worse outcome) rather than assumed symmetric.
Where a protected attribute is not collected, there is a documented, lawful approach to estimating or testing fairness without it (and its limits are acknowledged).
Fairness metrics chosen & measured · weight 15
A fairness claim is untested until a metric is chosen, justified and measured per group. Common definitions can be mathematically incompatible, so the choice has to be deliberate.
One or more group-fairness metrics have been explicitly chosen and justified for this use case, with the trade-off against other definitions acknowledged.
The chosen metric(s) are actually measured and reported per group and sub-group, not asserted qualitatively.
Error rates are broken down by group (e.g. false-positive and false-negative rates), not only an aggregate accuracy figure.
A fairness threshold or acceptable disparity range is defined in advance, with the rationale recorded.
Bias sources assessed (systemic / statistical / human) · weight 13
NIST SP 1270 frames AI bias in three categories — systemic, statistical/computational, and human-cognitive. A credible assessment considers all three, not just the data.
SYSTEMIC bias is considered: historical and institutional patterns in how the data was generated or labelled that disadvantage a group, independent of the model.
STATISTICAL / COMPUTATIONAL bias is considered: sampling, measurement, feature selection, label proxies and optimisation choices that skew outcomes.
HUMAN-COGNITIVE bias is considered: how designers, annotators and the humans acting on outputs introduce bias (anchoring, automation bias, confirmation).
Identified bias sources are logged with an owner and a mitigation or acceptance decision, not just noted.
Dataset representativeness & documentation · weight 12
Fairness starts in the data. Know where it came from, who is under-represented, how labels were produced, and write it down so a reviewer can follow it.
The training/validation/test data is documented: source, collection method, time period, known gaps and consent/legal basis.
Representativeness across the affected groups has been checked, and under-represented groups are identified.
The labelling / annotation process is documented, including who labelled, guidelines used, and inter-annotator agreement where relevant.
Evaluation uses a dataset representative of the real deployment population, not only the convenient training distribution.
Explainability approach fits the audience · weight 13
Explainability is relative to a stakeholder and a goal. ISO/IEC TS 6254 frames objectives and approaches; match global vs local and intrinsic vs post-hoc to the audience.
The explainability objective and audience are defined (e.g. a subject contesting a decision, a developer debugging, an auditor assuring) — not a generic "we use SHAP".
The approach distinguishes GLOBAL explanations (overall model behaviour) from LOCAL explanations (why this individual decision), and provides what each audience needs.
The choice of intrinsically interpretable models vs post-hoc explanation methods is deliberate, with the limits of post-hoc methods acknowledged (approximation, instability).
Explanations have been checked for faithfulness and comprehensibility with the intended audience, not just generated.
Transparency artifacts · weight 11
The durable record a reviewer reads: a model card / system card stating intended use, limitations, performance by group and the fairness work done.
A model card or system card exists describing intended use, out-of-scope use and known limitations.
That card reports performance and fairness metrics broken down by group, not just headline accuracy.
The card records the bias assessment, mitigations applied and residual known risks.
The artifacts are versioned and updated when the model or data changes — a point-in-time card is not left to rot.
Human oversight & contestability · weight 12
A person can understand, override and be held accountable for the decision, and an affected individual has a real route to challenge it and reach a human.
A human can review and override the AI decision, and the people doing so are trained for it and not simply rubber-stamping.
Affected individuals are told, in plain language, when an AI system is materially involved in a decision about them.
There is a working route for an individual to contest an outcome and get a human re-review, with a response path.
Accountability for the decision is assigned to a named role, not diffused into "the model decided".
Ongoing monitoring for drift & disparate impact · weight 10
Fairness is not a launch-day property. Populations shift, data drifts, and disparate impact can emerge in production even when the model was fair at release.
Fairness and group-wise performance metrics are monitored in production on a schedule, not only measured once before launch.
Data and concept drift are watched, with thresholds that trigger review or retraining.
Real-world outcomes and complaints are fed back to detect disparate impact that pre-deployment testing missed.
There is a defined trigger and owner for re-running this assessment after significant model, data or context change.
Educational self-assessment. This is a structured self-reflection framework, NOT a fairness audit, conformity assessment, measurement, certification or legal advice. The dimension weights are a transparent judgement framework. This tool references NIST SP 1270, ISO/IEC TR 24027, ISO/IEC TR 24028 and ISO/IEC TS 6254 by name only; it quotes no clause numbers or statistics, performs no scan, and runs entirely in your browser — nothing is uploaded. Your answers are stored only in the shareable link you choose to copy.
What this self-assessment is
- A free, 100% in-browser self-assessment of the fairness, bias and explainability of an AI system — the "trustworthiness" area an AI auditor checks but that most AI-governance tooling leaves empty. Answer a checklist, get a banded 0–100 transparency score.
- Eight weighted dimensions: protected attributes & affected groups, fairness metrics chosen & measured, bias sources across the three NIST SP 1270 categories, dataset representativeness, explainability fit to audience, transparency artifacts, human oversight & contestability, and ongoing monitoring.
- Grounded in well-established public standards — NIST SP 1270, ISO/IEC TR 24027, ISO/IEC TR 24028 and ISO/IEC TS 6254 — referenced by name only; it invents no statistics and quotes no clause numbers.
- Every question maps to a concrete next action, and gaps are prioritised lowest-score, highest-weight first. The Markdown export is structured to double as a draft model-card / transparency summary.
- It is an EDUCATIONAL self-assessment, not a fairness audit, conformity assessment or legal advice. Nothing is uploaded — export to Markdown or CSV, and share the exact assessment with a base64url permalink.
Fairness is a claim until it is measured
It is easy to say an AI system is “fair” and “explainable”. An auditor asks the harder question: show me. Which protected groups did you consider, including the intersectional ones? Which fairness metric did you choose, why that one, and what did it measure per group? Did you look for bias in the history behind your labels, in your sampling and objective, and in the humans acting on the output — the three categories NIST SP 1270 names? Can an affected person understand the decision and contest it? This self-assessment turns those questions into a structured checklist so the gaps are visible before someone else finds them.
The eight dimensions are weighted on purpose — choosing and measuring fairness metrics and identifying who could be harmed carry the most, because without them fairness is an untested assertion. That weighting is a heuristic, shown openly so you can argue with it and adjust it to your own context. The tool references well-established public standards — NIST SP 1270, ISO/IEC TR 24027, ISO/IEC TR 24028 and ISO/IEC TS 6254 — by name, as orientation. It is an educational self-assessment, not a fairness audit, conformity assessment or legal advice, and it quotes no invented statistics or clause numbers.
Frequently asked questions
What does the AI Fairness, Bias & Explainability Self-Assessment measure?
It is a structured self-assessment of the trustworthiness dimensions an AI auditor reviews: whether you have identified the protected attributes and affected groups, chosen and actually measured group-fairness metrics, assessed bias across the three categories in NIST SP 1270, documented dataset representativeness, matched an explainability approach to the audience, produced transparency artifacts like a model card, provided genuine human oversight and contestability, and set up ongoing monitoring for drift and disparate impact. You answer each item as in place, partial, or no, and the tool produces a weighted 0–100 transparency score with a band and a prioritised gap list.
Is this a fairness audit or a certification?
No. It is an educational self-assessment framework for structured self-reflection — not a fairness audit, conformity assessment, measurement, certification or legal advice, and it does not test your model or predict any outcome. The dimension weights are a transparent heuristic you can adjust to your own context. The value is the structured conversation, the prioritised to-do list and the draft model-card it produces, not the exact number.
Which standards does it draw on?
It is organised around widely used public standards: NIST SP 1270 (the three categories of AI bias — systemic, statistical/computational and human-cognitive), ISO/IEC TR 24027 (bias in AI systems and AI aided decision making), ISO/IEC TR 24028 (overview of trustworthiness in AI) and ISO/IEC TS 6254 (objectives and approaches for explainability of ML models and AI systems). It names these for orientation only and does not reproduce or quote clause numbers or figures from them; the question wording is PlayCISO's own plain-language paraphrase.
Why does it say common fairness metrics are "incompatible"?
Group-fairness definitions capture different ideas of fairness: demographic (statistical) parity equalises the rate of positive decisions across groups; equalized odds equalises both true-positive and false-positive rates; equal opportunity equalises only the true-positive rate; and predictive parity equalises precision for a given score. A well-established result is that, except in degenerate cases such as equal base rates or a perfect classifier, these generally cannot all hold at the same time — so which one to prioritise is a deliberate value judgement for the use case, not a purely technical choice. The tool makes you pick on purpose.
Does it scan my model or send my answers anywhere?
No. The assessment runs entirely in your browser. It performs no scan and touches no system or model. Your answers are never uploaded — they live only in the shareable permalink you choose to copy (encoded in the URL, base64url) and in any Markdown or CSV file you export yourself.
Can the output be used as a model card?
The Markdown export is deliberately structured as a draft model-card / transparency summary: the system scope and overall posture, the group-fairness definitions considered and their incompatibility note, a per-dimension table, the prioritised gaps with next actions, and the standards referenced. It is a starting skeleton to adapt, not a finished, verified card — you still have to fill in the real metrics and evidence for your system.
This tool is free and always will be. Would a Pro workspace — saved designs, team sharing and exports — be worth paying for?
Show me Pro →No account needed to use this tool — this just tells us whether Pro is worth building out.