Start with the decision, not SHAP

Explainability is useful only when an explanation has a named audience, purpose, decision, and validation standard. “Explain the model” is too vague. A model developer debugging leakage, a validator challenging assumptions, an operator deciding whether to escalate, and an applicant receiving an adverse-action notice need different evidence.

NIST’s AI Risk Management Framework begins by mapping context: intended purpose, users, affected people, impacts, risk tolerance, requirements, human oversight, and go/no-go criteria. In a credit setting, model-risk and consumer-protection requirements add institution- and jurisdiction-specific obligations.

1. Write a use-case card

FieldCredit decision-support example
DecisionA trained, authorized employee makes or reviews a defined credit decision under approved policy
Model roleSpecify score, recommendation, prioritization, or automation; do not leave this implicit
TargetDefine the outcome, time horizon, observation window, exclusions, censoring, and label limitations
Affected peopleApplicants, co-applicants, customers, and groups affected by access, price, delay, error, or appeal
Explanation usersApplicant, operator, validator, risk/compliance, model developer, auditor
HarmsFalse denial, unaffordable approval, disparate errors, delay, privacy loss, inaccurate reason, automation bias
Human oversightAuthority, training, review evidence, override rules, escalation, appeal, and accountability
Stop conditionsData failure, inability to give accurate reasons, validation failure, excessive harm, drift, or control failure

2. Define each explanation product

AudienceQuestionCandidate evidenceValidation
DeveloperIs the model using leakage, proxies, or unstable patterns?Coefficients, partial dependence, SHAP diagnostics, error slicesReproduction, sensitivity, ablation, domain review
Independent validatorIs design conceptually sound and fit for use?Data lineage, assumptions, alternatives, performance, limitationsIndependent challenge and outcome analysis
OperatorWhen should this case be reviewed or escalated?Decision inputs, uncertainty, policy checks, out-of-scope flagsHuman-factors and workflow testing
ApplicantWhat were the specific principal reasons for adverse action?Accurate reason-generation process tied to factors actually usedFidelity, specificity, consistency, legal/compliance review

Do not reuse one colorful plot for all four purposes. A post-hoc attribution may be useful for model diagnosis but fail the accuracy, stability, language, or legal requirements of an applicant notice.

3. Create a governed data specification

Before loading a file, document its source, authority and license, collection period, population, sampling, data dictionary, units, missingness meanings, label construction, protected-group audit plan, retention, security classification, and known gaps. Identify variables that would not exist at decision time and variables created by the historical decision process.

Never forward-fill values across unrelated people. Missing-data handling must be justified per field and fitted only on training data. Preserve missingness indicators when they are meaningful and permitted. Split data according to the deployment boundary—often time, applicant, account, household, branch, or geography—before fitting transformations.

4. Use a leakage-safe toy pipeline

This fragment demonstrates software structure on an explicitly synthetic, numeric teaching dataset. It does not establish that the features, target, split, model, or threshold are appropriate for credit.

from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# `toy_X` and `toy_y` must be synthetic teaching data with documented columns.
X_train, X_test, y_train, y_test = train_test_split(
    toy_X,
    toy_y,
    test_size=0.20,
    random_state=42,
    stratify=toy_y,
)

pipeline = Pipeline(
    steps=[
        ("impute", SimpleImputer(strategy="median", add_indicator=True)),
        ("scale", StandardScaler()),
        ("model", LogisticRegression(max_iter=1_000, random_state=42)),
    ]
)

pipeline.fit(X_train, y_train)
y_pred = pipeline.predict(X_test)

print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred, digits=3))

A production experiment must add time-aware or grouped splitting where appropriate, probability and calibration analysis, decision-utility evaluation, group/intersection metrics with uncertainty, robustness tests, data-quality tests, baseline comparisons, and independent validation.

5. Interpret logistic regression precisely

For a logistic model, holding the other modeled variables fixed, a one-unit increase in feature x_j changes the modeled log-odds by coefficient β_j; the corresponding conditional odds ratio is exp(β_j). It is not a probability ratio, a causal effect, or a unit-free importance score.

If a feature is standardized, one unit means one training-set standard deviation after the fitted transformation. Categorical coding, interactions, nonlinear terms, regularization, collinearity, sampling, and omitted variables all affect interpretation. Publish computed coefficients with units, uncertainty or stability evidence, preprocessing, and model version—never an invented example table labeled as a result.

6. Add explanation tools only after defining validation

LIME fits a local surrogate around a selected instance; results depend on perturbation, representation, kernel, sampling, discretization, and random state. SHAP-family methods attribute a selected model output relative to a background/reference distribution; results depend on the explainer, dependence assumptions, background, output, and aggregation.

For either method, record software version, model output, reference data, preprocessing path, instance identity, random seed where applicable, and display transformation. Test fidelity to the model output, stability under reasonable settings, reproducibility, sensitivity to correlated inputs, and usefulness for the intended audience. Do not call an explanation a fairness test or causal account.

7. Validate adverse-action reasons separately

CFPB guidance states that U.S. creditors using complex algorithms must still provide specific and accurate principal reasons for adverse actions. Complexity is not an excuse. If a post-hoc method supports a reason-generation process, validate that the reasons correspond to the factors actually scored and the principal reasons for the individual action. Confirm current requirements and the complete notice process with qualified counsel.

8. Define evidence for a go/no-go decision

  • documented business purpose, target, affected population, model role, and alternatives;
  • approved data lineage, quality, representativeness, privacy, security, and permissible-use assessment;
  • performance and error evidence in deployment-relevant slices, with uncertainty and baseline comparisons;
  • validated explanation products for each audience, including fidelity and stability;
  • fairness and harm assessment beyond feature attributions;
  • independent validation proportionate to use and materiality;
  • human-oversight, notice, appeal, incident, monitoring, change, rollback, and retirement controls.

Suggested repository structure

explainability-use-case/
├── README.md
├── governance/
│   ├── use-case-card.md
│   ├── explanation-requirements.md
│   ├── risk-and-control-register.md
│   └── decision-record.md
├── data/
│   ├── README.md
│   └── schema.json
├── src/
│   ├── features.py
│   ├── train.py
│   ├── evaluate.py
│   └── explain.py
├── tests/
│   ├── test_data_contract.py
│   ├── test_pipeline.py
│   ├── test_metrics.py
│   └── test_explanation_fidelity.py
├── reports/
│   ├── model-card.md
│   ├── validation-report.md
│   └── monitoring-plan.md
└── requirements.lock

The core lesson is that explainability starts before model training. A trustworthy explanation system is designed around a decision, audience, evidence standard, and accountability process—not added at the end as a plot.