Explainable AI

Start with the decision, not SHAP

Explainability is useful only when an explanation has a named audience, purpose, decision, and validation standard. “Explain the model” is too vague. A model developer debugging leakage, a validator challenging assumptions, an operator deciding whether to escalate, and an applicant receiving an adverse-action notice need different evidence.

NIST’s AI Risk Management Framework begins by mapping context: intended purpose, users, affected people, impacts, risk tolerance, requirements, human oversight, and go/no-go criteria. In a credit setting, model-risk and consumer-protection requirements add institution- and jurisdiction-specific obligations.

1. Write a use-case card

Field	Credit decision-support example
Decision	A trained, authorized employee makes or reviews a defined credit decision under approved policy
Model role	Specify score, recommendation, prioritization, or automation; do not leave this implicit
Target	Define the outcome, time horizon, observation window, exclusions, censoring, and label limitations
Affected people	Applicants, co-applicants, customers, and groups affected by access, price, delay, error, or appeal
Explanation users	Applicant, operator, validator, risk/compliance, model developer, auditor
Harms	False denial, unaffordable approval, disparate errors, delay, privacy loss, inaccurate reason, automation bias
Human oversight	Authority, training, review evidence, override rules, escalation, appeal, and accountability
Stop conditions	Data failure, inability to give accurate reasons, validation failure, excessive harm, drift, or control failure

2. Define each explanation product

Audience	Question	Candidate evidence	Validation
Developer	Is the model using leakage, proxies, or unstable patterns?	Coefficients, partial dependence, SHAP diagnostics, error slices	Reproduction, sensitivity, ablation, domain review
Independent validator	Is design conceptually sound and fit for use?	Data lineage, assumptions, alternatives, performance, limitations	Independent challenge and outcome analysis
Operator	When should this case be reviewed or escalated?	Decision inputs, uncertainty, policy checks, out-of-scope flags	Human-factors and workflow testing
Applicant	What were the specific principal reasons for adverse action?	Accurate reason-generation process tied to factors actually used	Fidelity, specificity, consistency, legal/compliance review

Do not reuse one colorful plot for all four purposes. A post-hoc attribution may be useful for model diagnosis but fail the accuracy, stability, language, or legal requirements of an applicant notice.

3. Create a governed data specification

Before loading a file, document its source, authority and license, collection period, population, sampling, data dictionary, units, missingness meanings, label construction, protected-group audit plan, retention, security classification, and known gaps. Identify variables that would not exist at decision time and variables created by the historical decision process.

Never forward-fill values across unrelated people. Missing-data handling must be justified per field and fitted only on training data. Preserve missingness indicators when they are meaningful and permitted. Split data according to the deployment boundary—often time, applicant, account, household, branch, or geography—before fitting transformations.

4. Use a leakage-safe toy pipeline

This fragment demonstrates software structure on an explicitly synthetic, numeric teaching dataset. It does not establish that the features, target, split, model, or threshold are appropriate for credit.

from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# `toy_X` and `toy_y` must be synthetic teaching data with documented columns.
X_train, X_test, y_train, y_test = train_test_split(
    toy_X,
    toy_y,
    test_size=0.20,
    random_state=42,
    stratify=toy_y,
)

pipeline = Pipeline(
    steps=[
        ("impute", SimpleImputer(strategy="median", add_indicator=True)),
        ("scale", StandardScaler()),
        ("model", LogisticRegression(max_iter=1_000, random_state=42)),
    ]
)

pipeline.fit(X_train, y_train)
y_pred = pipeline.predict(X_test)

print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred, digits=3))

A production experiment must add time-aware or grouped splitting where appropriate, probability and calibration analysis, decision-utility evaluation, group/intersection metrics with uncertainty, robustness tests, data-quality tests, baseline comparisons, and independent validation.

5. Interpret logistic regression precisely

For a logistic model, holding the other modeled variables fixed, a one-unit increase in feature x_j changes the modeled log-odds by coefficient β_j; the corresponding conditional odds ratio is exp(β_j). It is not a probability ratio, a causal effect, or a unit-free importance score.

If a feature is standardized, one unit means one training-set standard deviation after the fitted transformation. Categorical coding, interactions, nonlinear terms, regularization, collinearity, sampling, and omitted variables all affect interpretation. Publish computed coefficients with units, uncertainty or stability evidence, preprocessing, and model version—never an invented example table labeled as a result.

6. Add explanation tools only after defining validation

LIME fits a local surrogate around a selected instance; results depend on perturbation, representation, kernel, sampling, discretization, and random state. SHAP-family methods attribute a selected model output relative to a background/reference distribution; results depend on the explainer, dependence assumptions, background, output, and aggregation.

For either method, record software version, model output, reference data, preprocessing path, instance identity, random seed where applicable, and display transformation. Test fidelity to the model output, stability under reasonable settings, reproducibility, sensitivity to correlated inputs, and usefulness for the intended audience. Do not call an explanation a fairness test or causal account.

7. Validate adverse-action reasons separately

CFPB guidance states that U.S. creditors using complex algorithms must still provide specific and accurate principal reasons for adverse actions. Complexity is not an excuse. If a post-hoc method supports a reason-generation process, validate that the reasons correspond to the factors actually scored and the principal reasons for the individual action. Confirm current requirements and the complete notice process with qualified counsel.

8. Define evidence for a go/no-go decision

documented business purpose, target, affected population, model role, and alternatives;
approved data lineage, quality, representativeness, privacy, security, and permissible-use assessment;
performance and error evidence in deployment-relevant slices, with uncertainty and baseline comparisons;
validated explanation products for each audience, including fidelity and stability;
fairness and harm assessment beyond feature attributions;
independent validation proportionate to use and materiality;
human-oversight, notice, appeal, incident, monitoring, change, rollback, and retirement controls.

Suggested repository structure

explainability-use-case/
├── README.md
├── governance/
│   ├── use-case-card.md
│   ├── explanation-requirements.md
│   ├── risk-and-control-register.md
│   └── decision-record.md
├── data/
│   ├── README.md
│   └── schema.json
├── src/
│   ├── features.py
│   ├── train.py
│   ├── evaluate.py
│   └── explain.py
├── tests/
│   ├── test_data_contract.py
│   ├── test_pipeline.py
│   ├── test_metrics.py
│   └── test_explanation_fidelity.py
├── reports/
│   ├── model-card.md
│   ├── validation-report.md
│   └── monitoring-plan.md
└── requirements.lock

The core lesson is that explainability starts before model training. A trustworthy explanation system is designed around a decision, audience, evidence standard, and accountability process—not added at the end as a plot.