XAI Is Not a Fairness Test: Auditing Model Outcomes with…

Explanations and fairness answer different questions

SHAP and LIME can help investigate how a fitted model responds to features under particular assumptions and reference data. A fairness audit asks broader questions: who is affected, which harms matter, whether data and labels are valid, how outcomes and errors differ, whether differences are justified and lawful, and how the full decision process behaves.

An attribution is not a causal effect, a legal reason, or a fairness verdict. Use explanations as one diagnostic layer alongside group metrics, data review, process review, qualitative evidence, and deployment monitoring.

1. Define the decision, population, and harm

Document the product, decision, applicant population, outcome, model role, human overrides, time horizon, and consequences of false approvals, false denials, delay, or non-decision. Identify relevant groups and intersections with appropriate authority and governance. State why each metric maps to a plausible harm.

2. Audit the data before the model

Describe source, collection process, consent or authority, time period, exclusions, missingness, and label construction.
Report group and intersection counts; small cells require uncertainty controls and privacy review.
Test measurement quality, coverage, missingness, label error, temporal shift, and sample selection by group.
Map potential proxies and downstream process variables rather than assuming that dropping a protected field removes its influence.
Split data by the real deployment boundary—often time, customer, or geography—before fitting preprocessing or resampling.

3. Establish an outcome-and-error baseline

The following code assumes that y_test, binary y_pred, and an audit-only audit_group Series are aligned to the same held-out rows. It reports evidence; it does not declare the system fair or lawful.

from fairlearn.metrics import (
    MetricFrame,
    count,
    false_positive_rate,
    selection_rate,
    true_positive_rate,
)

metrics = {
    "n": count,
    "selection_rate": selection_rate,
    "true_positive_rate": true_positive_rate,
    "false_positive_rate": false_positive_rate,
}

audit = MetricFrame(
    metrics=metrics,
    y_true=y_test,
    y_pred=y_pred,
    sensitive_features=audit_group,
)

print(audit.overall)
print(audit.by_group)
print("max_minus_min_gap")
print(audit.difference(method="between_groups"))

Always publish denominators. Add confidence intervals or a resampling analysis appropriate to the sampling and dependence structure. Do not rank or disclose tiny groups in ways that create privacy risk. Add task-specific utility, calibration, predictive-value, abstention, and severity measures where relevant.

4. Use SHAP for diagnosis, not certification

For a supported tree classifier, a minimal diagnostic pattern is:

import shap

# Use the fitted model and the exact held-out feature matrix.
explainer = shap.TreeExplainer(model)
shap_values = explainer(X_test)

# Inspect the returned shape/output before selecting a class.
print(shap_values.shape)
shap.plots.beeswarm(shap_values)

Pin and record package versions, model output being explained, background/reference data, feature transforms, and class selection. Check explanation stability across reasonable background samples and methods. Investigate whether features or proxies contribute differently across groups, but return to observed outcomes, errors, and process evidence before drawing conclusions.

5. Investigate mechanisms

For each material gap, examine data coverage, labels, missingness, measurement, feature availability, model errors, thresholding, manual review, overrides, appeals, and downstream actions. Review false positives and false negatives with domain experts. Include affected-stakeholder perspectives where feasible.

A matched-pair or counterfactual test must specify which attributes are changed, which must change with them, and why the resulting cases are plausible. Changing a protected attribute while freezing causally related variables can create impossible people and misleading conclusions.

6. Choose a mitigation that targets the mechanism

Possible interventions include improving collection and labels, narrowing the use case, removing unjustified variables, redesigning the target, changing thresholds under an approved policy, adding abstention or human review, improving notices and appeals, or deciding not to automate. Resampling and constrained optimization are only candidates; they do not guarantee fairness.

Predeclare utility and harm constraints. Fit every preprocessing and mitigation step only on training data. Compare candidates on untouched validation/test data, including intersections and rare high-severity cases. Reject a mitigation that merely moves harm to another group, time period, or workflow stage.

7. Validate explanations used in adverse-action notices

For U.S. credit decisions, CFPB guidance states that creditors using complex algorithms must still provide specific and accurate principal reasons for adverse actions. A generic feature-importance chart is not an applicant-specific reason. If a post-hoc explanation is used operationally, validate that it faithfully identifies the factors actually scored and the principal reasons for the decision, and confirm current obligations with counsel.

8. Monitor the deployed decision system

Track input and label drift, missingness, group coverage, outcome and error metrics, explanation stability, overrides, adverse-action reasons, complaints, appeals, incidents, and delayed outcomes. Define alert thresholds, owners, investigation procedures, rollback criteria, and reevaluation triggers. Audit the human-plus-model process, not just the model artifact.

What the audit can conclude

Evidence	Calibrated conclusion
Different group selection rates	A disparity exists in this sample; its uncertainty, causes, relevance, and legal meaning require analysis
Different error rates	The model’s mistakes are distributed differently under the chosen label and threshold
Different SHAP distributions	The model attribution pattern differs under this explainer/reference choice; investigate mechanism and stability
Metric improvement after mitigation	The selected metric improved on held-out data; check uncertainty, utility, other groups/metrics, and process effects

The goal is not to make a fairness chart turn green. It is to build an evidence trail from a defined harm to measured behavior, investigated mechanisms, justified interventions, accountable decisions, and continuous review.