Why Did The Model
Do That?

Debugging the Ghost
in the Machine

Cosima Meyer

Who am I?

👩🏼‍🎓 PhD in Politicial Science @ University of Mannheim

👩🏼‍💻 Transitioned into Data Science

Husky classified as wolf

… because of snow

Images taken from Ribeiro, M. T., Singh, S., Guestrin, C., Lundberg, S. M., & Lee, S. I. 2016. “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. arXiv

How Do We Detect This?

You trained a model.

It looks good.

But something feels a bit off.

There are many different approaches and Python libraries.

How do you choose?

An XAI Decision Tree…

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    START(("🔍 Explain?"))
    DTYPE{"Data type?"}
    START --> DTYPE

    TAB["📊 Tabular"]
    TAB_SCOPE{"Scope?"}
    TAB_G{"Insight?"}
    TAB_GR["Ranking"]
    TAB_GE["Effects"]
    TAB_GI["Interactions"]
    TAB_L{"Type?"}
    TAB_LW["Why this?"]
    TAB_LC["What if?"]

    DTYPE -->|Tabular| TAB
    TAB --> TAB_SCOPE
    TAB_SCOPE -->|Global| TAB_G
    TAB_G --> TAB_GR
    TAB_G --> TAB_GE
    TAB_G --> TAB_GI
    TAB_SCOPE -->|Local| TAB_L
    TAB_L --> TAB_LW
    TAB_L --> TAB_LC

    IMG["🖼️ Image"]
    IMG_S{"Scope?"}
    IMG_G["Global"]
    IMG_L{"Access?"}
    IMG_BB["Black-Box"]
    IMG_WB["White-Box"]
    %% IMG_A{"Architecture?"}
    %% IMG_CNN["CNN"]
    %% IMG_VIT["ViT"]

    DTYPE -->|Image| IMG
    IMG --> IMG_S
    IMG_S -->|Global| IMG_G
    IMG_S -->|Local| IMG_L
    IMG_L --> IMG_BB
    IMG_L --> IMG_WB
    %% IMG_A -->|CNN| IMG_CNN
    %% IMG_A -->|ViT| IMG_VIT

    TXT["📝 Text"]
    TXT_S{"Scope?"}
    TXT_G["Global"]
    TXT_L{"Access?"}
    TXT_BB["Black-Box"]
    TXT_WB["White-Box"]

    DTYPE -->|Text| TXT
    TXT --> TXT_S
    TXT_S -->|Global| TXT_G
    TXT_S -->|Local| TXT_L
    TXT_L --> TXT_BB
    TXT_L --> TXT_WB

    style START fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2.5px,color:#1E3A4C
    style DTYPE fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style TAB fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TAB_SCOPE fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TAB_G fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TAB_GR fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_GE fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_GI fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_L fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TAB_LW fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_LC fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style IMG fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style IMG_S fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style IMG_G fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style IMG_L fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style IMG_BB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style IMG_WB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TXT fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TXT_S fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TXT_G fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TXT_L fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TXT_BB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TXT_WB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af

… As Well As Context Info

Key Distinctions

Interpretability vs Explainability

Can you read the model's logic directly, or do you need a separate method to approximate it?

Black-box vs White-box

Can you only query the model, or do you have full access to weights and gradients?

Global vs Local

Explains overall model behavior, or explains a single prediction?

Let’s Build the Decision Tree Together 🌳

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    START(("🔍 Explain?"))

    style START fill:#ffffff,stroke:#1E3A4C,stroke-width:2.5px,color:#1E3A4C

What’s Your Data Type?

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    START(("🔍 Explain?"))
    DTYPE{"Data type?"}
    START --> DTYPE

    style START fill:#ffffff,stroke:#1E3A4C,stroke-width:2.5px,color:#1E3A4C
    style DTYPE fill:#EBF0F3,stroke:#ADC0CA,stroke-width:5px,color:#1E3A4C

What’s Your Data Type?

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    START(("🔍 Explain?"))
    DTYPE{"Data type?"}
    START --> DTYPE

    TAB["<br>📊 Tabular<br>"]
    IMG["<br>🖼️ Image<br>"]
    TXT["<br>📝 Text"<br>]

    DTYPE --> TAB
    DTYPE --> IMG
    DTYPE --> TXT

    style START fill:#ffffff,stroke:#1E3A4C,stroke-width:2.5px,color:#1E3A4C
    style DTYPE fill:#EBF0F3,stroke:#ADC0CA,stroke-width:5px,color:#1E3A4C
    style TAB fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style IMG fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TXT fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C

Tabular

First Question: Is It Already Interpretable?

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    TAB["📊 Tabular"]
    TAB_INTERP{"Inherently<br/>interpretable?"}
    TAB --> TAB_INTERP

    TAB_GLASS["<br>🎉 Use built-in explanations<br>"]
    TAB_INTERP -->|"Yes — linear, tree,<br/>GAM, EBM"| TAB_GLASS

    TAB_SCOPE{"Scope?"}
    TAB_INTERP -->|"No — black-box<br/>model"| TAB_SCOPE

    style TAB fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style TAB_INTERP fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_GLASS fill:#d1fae5,stroke:#059669,stroke-width:5px,color:#065f46
    style TAB_SCOPE fill:#EBF0F3,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af

Tabular → Scope

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    TAB["📊 Tabular"]
    TAB_SCOPE{"Scope?"}
    TAB --> TAB_SCOPE

    TAB_G{"What insight?"}
    TAB_SCOPE -->|"Global"| TAB_G
    TAB_GR["Ranking"]
    TAB_GE["Effects"]
    TAB_GI["Interactions"]
    TAB_G --> TAB_GR
    TAB_G --> TAB_GE
    TAB_G --> TAB_GI

    TAB_L{"Type?"}
    TAB_SCOPE -->|"Local"| TAB_L
    TAB_LW["Why this?"]
    TAB_LC["What if?"]
    TAB_L --> TAB_LW
    TAB_L --> TAB_LC

    style TAB fill:#EBF0F3,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TAB_SCOPE fill:#EBF0F3,stroke:#ADC0CA,stroke-width:5px,color:#9ca3af
    style TAB_G fill:#EBF0F3,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TAB_GR fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_GE fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_GI fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_L fill:#EBF0F3,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TAB_LW fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_LC fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af

Tabular → Scope

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    TAB["📊 Tabular"]
    TAB_SCOPE{"Scope?"}
    TAB --> TAB_SCOPE

    TAB_G{"What insight?"}
    TAB_SCOPE -->|"Global"| TAB_G
    TAB_GR["Ranking"]
    TAB_GE["Effects"]
    TAB_GI["Interactions"]
    TAB_G --> TAB_GR
    TAB_G --> TAB_GE
    TAB_G --> TAB_GI

    TAB_L{"Type?"}
    TAB_SCOPE -->|"Local"| TAB_L
    TAB_LW["Why this?"]
    TAB_LC["What if?"]
    TAB_L --> TAB_LW
    TAB_L --> TAB_LC

    style TAB fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style TAB_SCOPE fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_G fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_GR fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_GE fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_GI fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_L fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_LW fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_LC fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af

Tabular → Global → 3 Questions

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    TAB["📊 Tabular"]
    TAB_SCOPE{"Scope?"}
    TAB --> TAB_SCOPE

    TAB_G{"What insight?"}
    TAB_SCOPE -->|"Global"| TAB_G
    TAB_GR["Ranking"]
    TAB_GE["Effects"]
    TAB_GI["Interactions"]
    TAB_G --> TAB_GR
    TAB_G --> TAB_GE
    TAB_G --> TAB_GI

    TAB_L{"Type?"}
    TAB_SCOPE -->|"Local"| TAB_L
    TAB_LW["Why this?"]
    TAB_LC["What if?"]
    TAB_L --> TAB_LW
    TAB_L --> TAB_LC

    style TAB fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style TAB_SCOPE fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_G fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_GR fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_GE fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_GI fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_L fill:#EBF0F3,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TAB_LW fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_LC fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af

Tabular → Global

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    TAB["📊 Tabular"]
    TAB_SCOPE{"Scope?"}
    TAB --> TAB_SCOPE

    TAB_G{"What insight?"}
    TAB_SCOPE -->|"Global"| TAB_G
    TAB_GR["Ranking"]
    TAB_GE["Effects"]
    TAB_GI["Interactions"]
    TAB_G --> TAB_GR
    TAB_G --> TAB_GE
    TAB_G --> TAB_GI

    TAB_L{"Type?"}
    TAB_SCOPE -->|"Local"| TAB_L
    TAB_LW["Why this?"]
    TAB_LC["What if?"]
    TAB_L --> TAB_LW
    TAB_L --> TAB_LC

    style TAB fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style TAB_SCOPE fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_G fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_GR fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_GE fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_GI fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_L fill:#EBF0F3,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TAB_LW fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_LC fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af

Tabular → Global

Which Features Matter Most?

SHAP Summary Plot

Ranks features by importance and shows how each feature’s values push predictions higher or lower

shap

Image Reference

Tabular → Global

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    TAB["📊 Tabular"]
    TAB_SCOPE{"Scope?"}
    TAB --> TAB_SCOPE

    TAB_G{"What insight?"}
    TAB_SCOPE -->|"Global"| TAB_G
    TAB_GR["Ranking"]
    TAB_GE["Effects"]
    TAB_GI["Interactions"]
    TAB_G --> TAB_GR
    TAB_G --> TAB_GE
    TAB_G --> TAB_GI

    TAB_L{"Type?"}
    TAB_SCOPE -->|"Local"| TAB_L
    TAB_LW["Why this?"]
    TAB_LC["What if?"]
    TAB_L --> TAB_LW
    TAB_L --> TAB_LC

    style TAB fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style TAB_SCOPE fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_G fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_GR fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_GE fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_GI fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_L fill:#EBF0F3,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TAB_LW fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_LC fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af

Tabular → Global

How Does a Feature Affect Predictions?

Partial Dependence Plots (PDP)

How a feature affects the model’s prediction on average, across the dataset

dalex effector PiML sklearn pdpbox

Image Reference

Accumulated Local Effects (ALE) Plots

How a feature affects predictions by accumulating local changes (good when features are correlated)

alibi dalex effector PyALE ALEPython

Image Reference

Tabular → Global

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    TAB["📊 Tabular"]
    TAB_SCOPE{"Scope?"}
    TAB --> TAB_SCOPE

    TAB_G{"What insight?"}
    TAB_SCOPE -->|"Global"| TAB_G
    TAB_GR["Ranking"]
    TAB_GE["Effects"]
    TAB_GI["Interactions"]
    TAB_G --> TAB_GR
    TAB_G --> TAB_GE
    TAB_G --> TAB_GI

    TAB_L{"Type?"}
    TAB_SCOPE -->|"Local"| TAB_L
    TAB_LW["Why this?"]
    TAB_LC["What if?"]
    TAB_L --> TAB_LW
    TAB_L --> TAB_LC

    style TAB fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style TAB_SCOPE fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_G fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_GR fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_GE fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_GI fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_L fill:#EBF0F3,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TAB_LW fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_LC fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af

Tabular → Global

Which Features Work Together?

SHAP Interaction Values

Decomposes predictions into main effects and pairwise feature interactions

shap shapiq

Image Reference

Tabular → Local → Type?

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    TAB["📊 Tabular"]
    TAB_SCOPE{"Scope?"}
    TAB --> TAB_SCOPE

    TAB_G{"What insight?"}
    TAB_SCOPE -->|"Global"| TAB_G
    TAB_GR["Ranking"]
    TAB_GE["Effects"]
    TAB_GI["Interactions"]
    TAB_G --> TAB_GR
    TAB_G --> TAB_GE
    TAB_G --> TAB_GI

    TAB_L{"Type?"}
    TAB_SCOPE -->|"Local"| TAB_L
    TAB_LW["Why this?"]
    TAB_LC["What if?"]
    TAB_L --> TAB_LW
    TAB_L --> TAB_LC

    style TAB fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style TAB_SCOPE fill:#EBF0F3,stroke:#ADC0CA,stroke-width:5px,color:#9ca3af
    style TAB_G fill:#EBF0F3,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TAB_GR fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_GE fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_GI fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_L fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_LW fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_LC fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C

Tabular → Local

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    TAB["📊 Tabular"]
    TAB_SCOPE{"Scope?"}
    TAB --> TAB_SCOPE

    TAB_G{"What insight?"}
    TAB_SCOPE -->|"Global"| TAB_G
    TAB_GR["Ranking"]
    TAB_GE["Effects"]
    TAB_GI["Interactions"]
    TAB_G --> TAB_GR
    TAB_G --> TAB_GE
    TAB_G --> TAB_GI

    TAB_L{"Type?"}
    TAB_SCOPE -->|"Local"| TAB_L
    TAB_LW["Why this?"]
    TAB_LC["What if?"]
    TAB_L --> TAB_LW
    TAB_L --> TAB_LC

    style TAB fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style TAB_SCOPE fill:#EBF0F3,stroke:#ADC0CA,stroke-width:5px,color:#9ca3af
    style TAB_G fill:#EBF0F3,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TAB_GR fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_GE fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_GI fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_L fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_LW fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_LC fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af

Tabular → Local

Why This Prediction?

SHAP Waterfall Plot

Shows how each feature pushes a single prediction up or down from the baseline

shap dalex

Image Reference

LIME

Shows how features influence a single prediction by slighting changing the input and fitting a local surrogate model

dalex lime

Image Reference

Tabular → Local

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    TAB["📊 Tabular"]
    TAB_SCOPE{"Scope?"}
    TAB --> TAB_SCOPE

    TAB_G{"What insight?"}
    TAB_SCOPE -->|"Global"| TAB_G
    TAB_GR["Ranking"]
    TAB_GE["Effects"]
    TAB_GI["Interactions"]
    TAB_G --> TAB_GR
    TAB_G --> TAB_GE
    TAB_G --> TAB_GI

    TAB_L{"Type?"}
    TAB_SCOPE -->|"Local"| TAB_L
    TAB_LW["Why this?"]
    TAB_LC["What if?"]
    TAB_L --> TAB_LW
    TAB_L --> TAB_LC

    style TAB fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style TAB_SCOPE fill:#EBF0F3,stroke:#ADC0CA,stroke-width:5px,color:#9ca3af
    style TAB_G fill:#EBF0F3,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TAB_GR fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_GE fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_GI fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_L fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_LW fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_LC fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C

Tabular → Local

What Would Change?

Counterfactuals

Not why, but what needs to change in the input to achieve a different outcome

dice-ml alibi

Image Reference

Tabular ✅

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    START(("🔍 Explain?"))
    DTYPE{"Data type?"}
    START --> DTYPE

    TAB["📊 Tabular"]
    TAB_SCOPE{"Scope?"}
    TAB_G{"Insight?"}
    TAB_GR["Ranking"]
    TAB_GE["Effects"]
    TAB_GI["Interactions"]
    TAB_L{"Type?"}
    TAB_LW["Why this?"]
    TAB_LC["What if?"]

    DTYPE -->|Tabular| TAB
    TAB --> TAB_SCOPE
    TAB_SCOPE -->|Global| TAB_G
    TAB_G --> TAB_GR
    TAB_G --> TAB_GE
    TAB_G --> TAB_GI
    TAB_SCOPE -->|Local| TAB_L
    TAB_L --> TAB_LW
    TAB_L --> TAB_LC

    IMG["🖼️ Image"]
    IMG_S{"Scope?"}
    IMG_G["Global"]
    IMG_L{"Access?"}
    IMG_BB["Black-Box"]
    IMG_WB["White-Box"]
    
    %% IMG_A{"Architecture?"}
    %% IMG_CNN["CNN"]
    %% IMG_VIT["ViT"]

    DTYPE -->|Image| IMG
    IMG --> IMG_S
    IMG_S -->|Global| IMG_G
    IMG_S -->|Local| IMG_L
    IMG_L --> IMG_BB
    IMG_L --> IMG_WB
    %% IMG_A -->|CNN| IMG_CNN
    %% IMG_A -->|ViT| IMG_VIT

    TXT["📝 Text"]
    TXT_S{"Scope?"}
    TXT_G["Global"]
    TXT_L{"Access?"}
    TXT_BB["Black-Box"]
    TXT_WB["White-Box"]

    DTYPE -->|Text| TXT
    TXT --> TXT_S
    TXT_S -->|Global| TXT_G
    TXT_S -->|Local| TXT_L
    TXT_L --> TXT_BB
    TXT_L --> TXT_WB

    style START fill:#ffffff,stroke:#1E3A4C,stroke-width:2.5px,color:#1E3A4C
    style DTYPE fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style TAB fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_SCOPE fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_G fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_GR fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_GE fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_GI fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_L fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_LW fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TAB_LC fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style IMG fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style IMG_S fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style IMG_G fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style IMG_L fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style IMG_BB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style IMG_WB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TXT fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TXT_S fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TXT_G fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TXT_L fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TXT_BB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TXT_WB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af

Image

Image

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    IMG["🖼️ Image"]
    IMG_S{"Scope?"}
    IMG --> IMG_S

    IMG_G["Global"]
    IMG_S -->|"Global"| IMG_G

    IMG_L{"Access?"}
    IMG_S -->|"Local"| IMG_L

    IMG_BB["Black-Box"]
    IMG_L -->  IMG_BB

    IMG_WB["White-Box"]
    IMG_L -->  IMG_WB

    style IMG fill:#EBF0F3,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style IMG_S fill:#EBF0F3,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style IMG_G fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style IMG_L fill:#EBF0F3,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style IMG_BB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style IMG_WB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af

Image: Scope

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    IMG["🖼️ Image"]
    IMG_S{"Scope?"}
    IMG --> IMG_S

    IMG_G["Global"]
    IMG_S -->|"Global"| IMG_G

    IMG_L{"Access?"}
    IMG_S -->|"Local"| IMG_L

    IMG_BB["Black-Box"]
    IMG_L -->  IMG_BB

    IMG_WB["White-Box"]
    IMG_L -->  IMG_WB
    %% IMG_A{"Architecture?"}
    %% IMG_L -->  IMG_A
    %% IMG_CNN["CNN"]
    %% IMG_VIT["ViT"]
    %% IMG_A -->|"CNN"| IMG_CNN
    %% IMG_A -->|"ViT"| IMG_VIT

    style IMG fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style IMG_S fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style IMG_G fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style IMG_L fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style IMG_BB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style IMG_WB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af

Image → Global

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    IMG["🖼️ Image"]
    IMG_S{"Scope?"}
    IMG --> IMG_S

    IMG_G["Global"]
    IMG_S -->|"Global"| IMG_G

    IMG_L{"Access?"}
    IMG_S -->|"Local"| IMG_L

    IMG_BB["Black-Box"]
    IMG_L -->  IMG_BB

    IMG_WB["White-Box"]
    IMG_L -->  IMG_WB
    %% IMG_A{"Architecture?"}
    %% IMG_L -->  IMG_A
    %% IMG_CNN["CNN"]
    %% IMG_VIT["ViT"]
    %% IMG_A -->|"CNN"| IMG_CNN
    %% IMG_A -->|"ViT"| IMG_VIT

    style IMG fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style IMG_S fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style IMG_G fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style IMG_L fill:#EBF0F3,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style IMG_BB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style IMG_WB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af

Image → Global

What Concepts Has the Model Learned?

Testing with Concept Activation Vectors (TCAV)

Tests whether a human-defined concept (e.g., “striped texture”) influences a model’s predictions using directional derivatives

captum tcav

Image Reference

Feature Visualization

Reveals what a neuron has learned by generating images that maximally activate it

tf-keras-vis lucid lucent flashtorch

Image Reference

Image → Local → Access

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    IMG["🖼️ Image"]
    IMG_S{"Scope?"}
    IMG --> IMG_S

    IMG_G["Global"]
    IMG_S -->|"Global"| IMG_G

    IMG_L{"Access?"}
    IMG_S -->|"Local"| IMG_L

    IMG_BB["Black-Box"]
    IMG_L -->  IMG_BB

    IMG_WB["White-Box"]
    IMG_L -->  IMG_WB
    %% IMG_A{"Architecture?"}
    %% IMG_L -->  IMG_A
    %% IMG_CNN["CNN"]
    %% IMG_VIT["ViT"]
    %% IMG_A -->|"CNN"| IMG_CNN
    %% IMG_A -->|"ViT"| IMG_VIT

    style IMG fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style IMG_S fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style IMG_G fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style IMG_L fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style IMG_BB fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style IMG_WB fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C

Image → Local → Black-Box

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    IMG["🖼️ Image"]
    IMG_S{"Scope?"}
    IMG --> IMG_S

    IMG_G["Global"]
    IMG_S -->|"Global"| IMG_G

    IMG_L{"Access?"}
    IMG_S -->|"Local"| IMG_L

    IMG_BB["Black-Box"]
    IMG_L -->  IMG_BB

    IMG_WB["White-Box"]
    IMG_L -->  IMG_WB
    %% IMG_A{"Architecture?"}
    %% IMG_L -->  IMG_A
    %% IMG_CNN["CNN"]
    %% IMG_VIT["ViT"]
    %% IMG_A -->|"CNN"| IMG_CNN
    %% IMG_A -->|"ViT"| IMG_VIT

    style IMG fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style IMG_S fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style IMG_G fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style IMG_L fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style IMG_BB fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style IMG_WB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af

Image → Local → Black-Box

Which Regions Matter?

SHAP Partition Explainer

Attributes importance to image regions using hierarchical decomposition of superpixels

shap

Image Reference

Image Segmentation

Explains predictions by segmenting the image into superpixels and measuring each region’s influence

lime

Image Reference

Image → Local → White-Box

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    IMG["🖼️ Image"]
    IMG_S{"Scope?"}
    IMG --> IMG_S

    IMG_G["Global"]
    IMG_S -->|"Global"| IMG_G

    IMG_L{"Access?"}
    IMG_S -->|"Local"| IMG_L

    IMG_BB["Black-Box"]
    IMG_L -->  IMG_BB

    IMG_WB["White-Box"]
    IMG_L -->  IMG_WB
    %% IMG_A{"Architecture?"}
    %% IMG_L -->  IMG_A
    %% IMG_CNN["CNN"]
    %% IMG_VIT["ViT"]
    %% IMG_A -->|"CNN"| IMG_CNN
    %% IMG_A -->|"ViT"| IMG_VIT

    style IMG fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style IMG_S fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style IMG_G fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style IMG_L fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style IMG_BB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style IMG_WB fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C

Image → Local → White-Box

Grad-CAM & Integrated Gradients

Gradient-Based Heatmaps

Highlights which spatial regions of the image drive the prediction using class-specific gradient signals

captum grad-cam pytorch-grad-cam

Image Reference

Integrated Gradients

Highlights which pixels drive a prediction by accumulating gradients from a neutral baseline to the actual input

captum

Image Reference

Image ✅

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    START(("🔍 Explain?"))
    DTYPE{"Data type?"}
    START --> DTYPE

    TAB["📊 Tabular"]
    TAB_SCOPE{"Scope?"}
    TAB_G{"Insight?"}
    TAB_GR["Ranking"]
    TAB_GE["Effects"]
    TAB_GI["Interactions"]
    TAB_L{"Type?"}
    TAB_LW["Why this?"]
    TAB_LC["What if?"]

    DTYPE -->|Tabular| TAB
    TAB --> TAB_SCOPE
    TAB_SCOPE -->|Global| TAB_G
    TAB_G --> TAB_GR
    TAB_G --> TAB_GE
    TAB_G --> TAB_GI
    TAB_SCOPE -->|Local| TAB_L
    TAB_L --> TAB_LW
    TAB_L --> TAB_LC

    IMG["🖼️ Image"]
    IMG_S{"Scope?"}
    IMG_G["Global"]
    IMG_L{"Access?"}
    IMG_BB["Black-Box"]
    IMG_WB["White-Box"]

    DTYPE -->|Image| IMG
    IMG --> IMG_S
    IMG_S -->|Global| IMG_G
    IMG_S -->|Local| IMG_L
    IMG_L --> IMG_BB
    IMG_L --> IMG_WB

    TXT["📝 Text"]
    TXT_S{"Scope?"}
    TXT_G["Global"]
    TXT_L{"Access?"}
    TXT_BB["Black-Box"]
    TXT_WB["White-Box"]

    DTYPE -->|Text| TXT
    TXT --> TXT_S
    TXT_S -->|Global| TXT_G
    TXT_S -->|Local| TXT_L
    TXT_L --> TXT_BB
    TXT_L --> TXT_WB

    style START fill:#ffffff,stroke:#1E3A4C,stroke-width:2.5px,color:#1E3A4C
    style DTYPE fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style TAB fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TAB_SCOPE fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TAB_G fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TAB_GR fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_GE fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_GI fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_L fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TAB_LW fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_LC fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style IMG fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style IMG_S fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style IMG_G fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style IMG_L fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style IMG_BB fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style IMG_WB fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C  
    style TXT fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TXT_S fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TXT_G fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TXT_L fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TXT_BB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TXT_WB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af

Text

Text

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    TXT["📝 Text"]
    TXT_S{"Scope?"}
    TXT --> TXT_S

    TXT_G["Global"]
    TXT_S -->|"Global"| TXT_G

    TXT_L{"Access?"}
    TXT_S -->|"Local"| TXT_L

    TXT_BB["Black-Box"]
    TXT_L -->  TXT_BB

    TXT_WB["White-Box"]
    TXT_L -->  TXT_WB

    style TXT fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style TXT_S fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style TXT_G fill:#ffffff,stroke:#1E3A4C,stroke-width:1.5px,color:#1E3A4C
    style TXT_L fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style TXT_BB fill:#ffffff,stroke:#1E3A4C,stroke-width:1.5px,color:#1E3A4C
    style TXT_WB fill:#ffffff,stroke:#1E3A4C,stroke-width:1.5px,color:#1E3A4C

Text: Scope

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    TXT["📝 Text"]
    TXT_S{"Scope?"}
    TXT --> TXT_S

    TXT_G["Global"]
    TXT_S -->|"Global"| TXT_G

    TXT_L{"Access?"}
    TXT_S -->|"Local"| TXT_L

    TXT_BB["Black-Box"]
    TXT_L -->  TXT_BB

    TXT_WB["White-Box"]
    TXT_L -->  TXT_WB

    style TXT fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style TXT_S fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TXT_G fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TXT_L fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TXT_BB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TXT_WB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af

Text → Global

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    TXT["📝 Text"]
    TXT_S{"Scope?"}
    TXT --> TXT_S

    TXT_G["Global"]
    TXT_S -->|"Global"| TXT_G

    TXT_L{"Access?"}
    TXT_S -->|"Local"| TXT_L

    TXT_BB["Black-Box"]
    TXT_L -->  TXT_BB

    TXT_WB["White-Box"]
    TXT_L -->  TXT_WB

    style TXT fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style TXT_S fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TXT_G fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TXT_L fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TXT_BB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TXT_WB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af

Text → Global

SHAP Partition Explainer

SHAP Partition Explainer

Measures token and phrase importance across a dataset using hierarchical masking that respects token boundaries

shap

Image Reference

Text → Local → Access

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    TXT["📝 Text"]
    TXT_S{"Scope?"}
    TXT --> TXT_S

    TXT_G["Global"]
    TXT_S -->|"Global"| TXT_G

    TXT_L{"Access?"}
    TXT_S -->|"Local"| TXT_L

    TXT_BB["Black-Box"]
    TXT_L -->  TXT_BB

    TXT_WB["White-Box"]
    TXT_L -->  TXT_WB

    style TXT fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style TXT_S fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TXT_G fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TXT_L fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TXT_BB fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TXT_WB fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C

Text → Local → Black-Box

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    TXT["📝 Text"]
    TXT_S{"Scope?"}
    TXT --> TXT_S

    TXT_G["Global"]
    TXT_S -->|"Global"| TXT_G

    TXT_L{"Access?"}
    TXT_S -->|"Local"| TXT_L

    TXT_BB["Black-Box"]
    TXT_L -->  TXT_BB

    TXT_WB["White-Box"]
    TXT_L -->  TXT_WB

    style TXT fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style TXT_S fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TXT_G fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TXT_L fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TXT_BB fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TXT_WB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af

Text → Local → Black-Box

Token-Level Attributions Without Gradients

SHAP Partition Explainer

Attributes importance to individual tokens by masking and measuring prediction changes

shap lime

Image Reference

Text → Local → White-Box

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    TXT["📝 Text"]
    TXT_S{"Scope?"}
    TXT --> TXT_S

    TXT_G["Global"]
    TXT_S -->|"Global"| TXT_G

    TXT_L{"Access?"}
    TXT_S -->|"Local"| TXT_L

    TXT_BB["Black-Box"]
    TXT_L -->  TXT_BB

    TXT_WB["White-Box"]
    TXT_L -->  TXT_WB

    style TXT fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style TXT_S fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TXT_G fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TXT_L fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TXT_BB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TXT_WB fill:#ffffff,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C

Text → Local → White-Box

Integrated Gradients

Integrated Gradients

Highlights which tokens drive a prediction by accumulating gradients from a neutral baseline to the actual input

captum inseq transformers-interpret

Image Reference

Text ✅

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#EBF0F3', 'primaryBorderColor': '#1E3A4C', 'lineColor': '#ADC0CA'}, 'flowchart': {'htmlLabels': false, 'nodeSpacing': 15, 'rankSpacing': 25, 'curve': 'basis', 'padding': 12}}}%%
flowchart TD
    START(("🔍 Explain?"))
    DTYPE{"Data type?"}
    START --> DTYPE

    TAB["📊 Tabular"]
    TAB_SCOPE{"Scope?"}
    TAB_G{"Insight?"}
    TAB_GR["Ranking"]
    TAB_GE["Effects"]
    TAB_GI["Interactions"]
    TAB_L{"Type?"}
    TAB_LW["Why this?"]
    TAB_LC["What if?"]

    DTYPE -->|Tabular| TAB
    TAB --> TAB_SCOPE
    TAB_SCOPE -->|Global| TAB_G
    TAB_G --> TAB_GR
    TAB_G --> TAB_GE
    TAB_G --> TAB_GI
    TAB_SCOPE -->|Local| TAB_L
    TAB_L --> TAB_LW
    TAB_L --> TAB_LC

    IMG["🖼️ Image"]
    IMG_S{"Scope?"}
    IMG_G["Global"]
    IMG_L{"Access?"}
    IMG_BB["Black-Box"]
    IMG_WB["White-Box"]
    %% IMG_A{"Architecture?"}
    %% IMG_CNN["CNN"]
    %% IMG_VIT["ViT"]

    DTYPE -->|Image| IMG
    IMG --> IMG_S
    IMG_S -->|Global| IMG_G
    IMG_S -->|Local| IMG_L
    IMG_L --> IMG_BB
    IMG_L --> IMG_WB
    %% IMG_A -->|CNN| IMG_CNN
    %% IMG_A -->|ViT| IMG_VIT

    TXT["📝 Text"]
    TXT_S{"Scope?"}
    TXT_G["Global"]
    TXT_L{"Access?"}
    TXT_BB["Black-Box"]
    TXT_WB["White-Box"]

    DTYPE -->|Text| TXT
    TXT --> TXT_S
    TXT_S -->|Global| TXT_G
    TXT_S -->|Local| TXT_L
    TXT_L --> TXT_BB
    TXT_L --> TXT_WB

    style START fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2.5px,color:#1E3A4C
    style DTYPE fill:#EBF0F3,stroke:#1E3A4C,stroke-width:2px,color:#1E3A4C
    style TAB fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TAB_SCOPE fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TAB_G fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TAB_GR fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_GE fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_GI fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_L fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style TAB_LW fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TAB_LC fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style IMG fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style IMG_S fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style IMG_G fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style IMG_L fill:#ffffff,stroke:#ADC0CA,stroke-width:2px,color:#9ca3af
    style IMG_BB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style IMG_WB fill:#ffffff,stroke:#ADC0CA,stroke-width:1.5px,color:#9ca3af
    style TXT fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TXT_S fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TXT_G fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TXT_L fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TXT_BB fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C
    style TXT_WB fill:#EBF0F3,stroke:#1E3A4C,stroke-width:5px,color:#1E3A4C

Observed Patterns and What’s Next

• SHAP appears in every single branch — it’s the Swiss army knife

• Integrated Gradients feels like a universal white-box method for unstructured data

• “Why this prediction?” and “What would change it?” are fundamentally different questions

• Start with interpretable models when you can

Find an interactive (tabular) version of the decision tree here 👇

More Resources

Links

• Meyer, Cosima. Explainable ML (GitHub Repository), 2026
• Küpfer, Andreas & Meyer, Cosima. BERT and Explainable AI, 2023

Books

• Masís, Serg. Interpretable Machine Learning with Python: Build Explainable, Fair, and Robust High-performance Models With Hands-on, Real-world Examples, 2nd ed., 2023
• Molnar, Christoph. Interpretable Machine Learning: A Guide for Making Black Box Models Explainable. 3rd ed., 2025

Thank You!

cosimameyer.com
cosimameyer