An explanation method produces an artifact under assumptions. It may describe a local approximation, feature attribution, attention pattern, decision path, counterfactual candidate, similar example, or concept sensitivity. None automatically reveals a model’s true causal reasoning, proves safety, or satisfies a legal duty.

Choose a method only after naming the audience, decision, model, failure cost, and question. Evaluate fidelity, stability, robustness, human understanding, and task usefulness separately.

1. LIME: a local surrogate

LIME perturbs an input, obtains model predictions, weights nearby samples, and fits an interpretable surrogate. Its coefficients explain that surrogate in the sampled neighborhood—not the original model globally. Results depend on perturbations, locality kernel, representation, feature selection, and seed.

Test repeated-run stability, neighborhood realism, and local fidelity. Report configuration and uncertainty.

2. SHAP: additive attribution under a value function

SHAP methods assign additive feature attributions based on a chosen value function and background/reference distribution. Different explainers make different assumptions; correlated features and unrealistic feature coalitions can change the result. A SHAP value is not a causal effect.

Document explainer, background data, output scale, feature dependence assumptions, and version. Test stability across defensible references and compare reconstructed with actual outputs.

3. Attention weights

Attention weights show how a particular architecture distributed attention at a layer and head. They can support analysis, but they are not automatically a faithful explanation of a prediction. Alternative attention patterns can sometimes yield similar outputs, and downstream computation matters.

Label the artifact “attention weights,” not “the reason.” Use intervention or ablation tests when making a fidelity claim.

4. Integrated Gradients and Grad-CAM

Integrated Gradients attributes output change along a path from a baseline to the input. Results depend on the baseline, path, target output, and differentiability. Grad-CAM uses gradients and convolutional feature maps to produce a coarse localization map for a chosen target.

Neither heatmap validates clinical or safety reasoning. Test sensitivity to baselines, model and input perturbations, randomization, localization criteria, and qualified human interpretation.

5. Decision paths

A decision-tree path exactly records the split conditions used by that fitted tree for one input. That is transparent computation, but it does not show causality or guarantee that splits are stable, fair, or sensible. Ensembles require aggregate or per-tree summaries and can be much harder to inspect.

See ensemble methods in machine learning.

6. Counterfactual candidates

A counterfactual explanation proposes changes that would alter a model output under an optimization objective. The change may be infeasible, unactionable, causally invalid, or unstable. Sensitive or immutable attributes require careful constraints; changing a proxy can still encode an unacceptable recommendation.

Report distance metric, constraints, diversity, feasibility checks, robustness, and whether the candidate represents recourse or only model sensitivity.

7. Example-based explanations

Prototypes, nearest neighbors, and influential examples connect an output to selected records. Their meaning depends on representation, similarity metric, candidate pool, and data quality. They can expose personal or copyrighted data and may reinforce unrepresentative examples.

Test retrieval stability and representativeness, and apply privacy, rights, and access controls.

8. TCAV concept sensitivity

TCAV estimates sensitivity of a class score to a direction learned from user-supplied concept examples in an internal representation. It does not directly measure social bias. Results depend on concept and random sets, layer, class, classifier, and replication.

Audit concept purity, repeat across sets and seeds, use significance and multiplicity controls, and avoid treating a concept score as causal evidence.

Build an explanation evaluation plan

  1. Name the audience, decision, possible harm, and recourse.
  2. State exactly what the artifact claims to explain.
  3. Test fidelity, stability, robustness, comprehension, and usefulness independently.
  4. Expose uncertainty, conflicting results, references, and knowledge limits.
  5. Store explanation evidence with model, data, and decision-policy documentation.

Review a transparency use case, bias analysis with XAI, and evaluation design.

Requirements depend on jurisdiction, actor, system classification, and decision. The EU AI Act includes risk-specific transparency and human-oversight duties and a right to explanation for certain individual decisions involving high-risk systems. In U.S. credit decisions, adverse-action reasons must be specific and accurate. No regulation says that running LIME or SHAP is sufficient compliance.

Reviewed and substantially updated September 4, 2026. Original publication date preserved.