29  Causal Decision-Making

Status: Draft

v0.4

29.1 Introduction

Decision-making requires answering “what if” questions: what happens under different policies? (Murphy 2003; Schulam and Saria 2017; Tennenholtz et al. 2020) This chapter formalises causal decision-making and shows how to determine CDM scope using Markov boundary and Markov blanket.

A fundamental question in causal decision-making is: which variables (nodes) should be included or excluded in a CDM? This question is answered by the concepts of Markov boundary (introduced in Graph Theory and Causal Patterns) and Markov blanket (introduced in From Structure to Time: FEP and Attractors), which provide principled criteria for determining the minimal set of variables needed for causal reasoning and decision-making.

29.2 Determining CDM Scope: Markov Boundary and Markov Blanket

29.2.1 The Problem: What to Include in a CDM?

When constructing a CDM for decision-making, we must decide which variables (nodes) to include. Including too many variables makes the model unnecessarily complex and computationally expensive. Including too few variables risks omitting important causal relationships, leading to biased policy evaluations.

The Markov boundary (introduced in Graph Theory and Causal Patterns) and Markov blanket (introduced in From Structure to Time: FEP and Attractors) provide complementary, principled answers to this question.

29.2.2 Markov Boundary: Graph-Theoretic Minimal Set

The Markov boundary of a target variable \(Y\) is the minimal set of variables needed for causal reasoning about \(Y\) at all three levels of Reason (Pearl 1988, 2014):

  • Level 1 (Association): Given \(\text{MB}(Y)\), no other variables provide additional information about \(Y\)
  • Level 2 (Intervention): The Markov boundary identifies which variables must be included to properly adjust for confounding
  • Level 3 (Counterfactual): The Markov boundary determines which variables are needed to compute counterfactual outcomes

Inclusion principle: When building a CDM for decision-making about outcome \(Y\), we should include at minimum the Markov boundary of \(Y\). This ensures we have all necessary information while avoiding unnecessary complexity.

29.2.3 Markov blanket: organism–environment interface

Following Chapter 9, the Markov blanket is the FEP formalisation of the cut between organism and environment (Friston 2010, 2013):

Blanket role Symbol CDM variable Gloss
Organism \(\mu\) \(X_t\) Internal state evolving under \(f\)
Environment \(\eta\) (outside model) Unmodelled world beyond \(\mu\)
Sensory / observation \(s\) \(Y_t\) What measurement gives
Active / intervention \(a\) \(A_t\) Policy / \(do(\cdot)\) on actions

\(\mu ⫫ \eta \mid (s,a)\): the environment is not observed directly, only through \(s\); the organism acts on the environment only through \(a\).

Boundary principle: Include \(s\) and \(a\) in the CDM; treat \(\eta\) as external unless its parents enter via the blanket. Specifying \((f,h,G)\) posits a society (defining characteristic across time steps).

29.2.4 CDM scope: boundary plus Markov boundary

The Markov boundary (Ch. 3) lists which variables a query about \(Y\) needs; the blanket fixes where the organism ends and environment \(\eta\) begins.

Decision rule: (1) draw the blanket (\(\mu\), \(s\), \(a\) vs \(\eta\)); (2) take the Markov boundary of each target \(Y\); (3) include their union. Distinguish population (sampling frame) from society (shared mechanism class). Environment shift (new site, protocol) is a transport problem (Ch. 8): does the same society-type still apply?

29.3 Integrating Markov Boundary and Markov Blanket

29.3.1 Unified Principle for CDM Construction

When constructing a CDM for causal decision-making, combine both perspectives:

  1. Identify the target variable(s): What outcomes are we interested in? (e.g., \(Y_T\) for policy evaluation)

  2. Determine the Markov boundary: For each target variable, identify its Markov boundary (see Graph Theory and Causal Patterns), this is the minimal set of variables needed for causal reasoning at all three levels of Reason

  3. Define the Markov blanket: Identify internal states (\(X_t\)), sensory states (\(Y_t\)), and active states (\(A_t\)) (see From Structure to Time: FEP and Attractors), this defines the system boundary from an FEP perspective

  4. Include the union: Include all variables in the Markov boundary of target variables, plus all variables in the Markov blanket. This ensures:

  • We have all necessary information for causal reasoning (Markov boundary, Structural stratum)
  • We have a principled system boundary (Markov blanket, Structural/Dynamical stratum)
  1. Exclude everything else: Variables not in the Markov boundary (for target variables) and not part of the Markov blanket can be excluded, as they are either irrelevant for causal reasoning or conditionally independent given the blanket

29.3.2 Implementation: Determining CDM Scope

Here’s an example of using Markov boundary and Markov blanket to determine CDM scope:

# Find project root and include ensure_packages.jl
project_root = let
    current = pwd()
    while !isfile(joinpath(current, "Project.toml")) && !isfile(joinpath(current, "_quarto.yml"))
        parent = dirname(current)
        parent == current && break
        current = parent
    end
    current
end
include(joinpath(dirname(Base.active_project()), "scripts", "book_bootstrap.jl"))

@auto_using DAGMakie CairoMakie CausalDynamics Graphs

# Example: Decision-making about outcome Y
# Graph: Z → X → Y ← W, with Z → W
# Target: Y (outcome we care about)

g = SimpleDiGraph(4)
add_edge!(g, 1, 2)  # Z → X
add_edge!(g, 2, 3)  # X → Y
add_edge!(g, 4, 3)  # W → Y
add_edge!(g, 1, 4)  # Z → W

# Step 1: Compute Markov boundary of Y (target variable)
function markov_boundary(g::AbstractGraph, node::Integer)
    """
    Compute the Markov boundary of a node.
    
    Returns the minimal set of nodes needed for causal reasoning about the target node.
    """
    # Pre-allocate for type stability
    parents = Vector{Int}()
    for src in 1:nv(g)
        if has_edge(g, src, node)
            push!(parents, src)
        end
    end
    
    children = Vector{Int}()
    for dst in 1:nv(g)
        if has_edge(g, node, dst)
            push!(children, dst)
        end
    end
    
    spouses = Set{Int}()
    for child in children
        for parent in 1:nv(g)
            if has_edge(g, parent, child) && parent != node
                push!(spouses, parent)
            end
        end
    end
    return sort(unique([parents; children; collect(spouses)]))
end

mb_Y = markov_boundary(g, 3)  # Markov boundary of Y (node 3)
println("Markov boundary of Y: ", mb_Y)  # Should be [2, 4] = {X, W}

# Step 2: Identify Markov blanket components
# Internal states (X): Latent process states
# Sensory states (Y): Observations
# Active states (A): Actions/interventions (if we add them)
# External states: Everything else

internal_states = [2]  # X (latent)
sensory_states = [3]  # Y (observed)
# Active states would be added if we have interventions

markov_blanket = sort(unique([internal_states; sensory_states]))
println("Markov blanket: ", markov_blanket)

# Step 3: Combine for CDM scope
cdm_scope = sort(unique([mb_Y; markov_blanket]))
println("CDM scope (union): ", cdm_scope)
println("  Includes: Markov boundary of Y + Markov blanket components")

# Visualise
let
    node_colors = [
        RGBf(0.30, 0.55, 0.70),
        RGBf(0.85, 0.65, 0.05),
        RGBf(0.30, 0.55, 0.70),
        RGBf(0.85, 0.65, 0.05),
    ]

    fig, ax, p = dagplot(g;
        figure_size = (800, 600),
        layout_mode = :acyclic,
        node_color = node_colors,
        nlabels = ["Z", "X", "Y", "W"],
        node_size = 40,
        edge_width = 2.0,
        arrow_size = 15,
        nlabels_fontsize = 16,
        nlabels_color = :white,
        nlabels_align = (:center, :center),
        nlabels_distance = 0,
        auto_align_labels = false,
        padding = 0.40,
    )
    
    fig  # Only this gets displayed
end
Markov boundary of Y: [2, 4]
Markov blanket: [2, 3]
CDM scope (union): [2, 3, 4]
  Includes: Markov boundary of Y + Markov blanket components

Determining CDM scope using Markov boundary and Markov blanket

29.4 Policies as Interventions

29.4.1 Definition

A policy (dynamic treatment strategy) is a function that maps history to action:

\[ A_t = \pi(H_t) \]

where \(H_t = (Y_{1:t}, A_{1:t-1}, L_{1:t})\) is the observed history.

29.4.2 Policy Intervention

Under policy \(\pi\), we set: \[ do(\pi): \quad A_t = \pi(H_t) \]

Interpretation: Structural intervention that sets treatment according to policy rule.

29.5 Causal Decision-Making Framework

29.5.1 The Decision Problem

Question: Which action should we take to maximise expected outcome?

Framework:

  1. Define objective: What do we want to maximise/minimise?
  2. Model interventions: How do actions affect outcomes?
  3. Optimize: Choose action that maximises expected outcome

29.5.2 Optimal Policy

The optimal policy maximises expected outcome:

\[ \pi^* = \arg\max_\pi \mathbb{E}^{do(\pi)}[Y_T] \]

Methods (Murphy 2003; Schulam and Saria 2017):

29.5.3 Uncertainty in Decision-Making

Decision-making must account for:

  • Model uncertainty: Which model is correct?
  • Parameter uncertainty: What are the parameter values?
  • Structural uncertainty: What is the causal structure?
  • Process uncertainty: Intrinsic stochasticity
  • Observation uncertainty: Measurement error

Robust decisions: Choose actions that perform well across model/parameter uncertainty.

Robust policy: Find policy that performs well under uncertainty:

\[ \pi^* = \arg\max_\pi \min_{\theta \in \Theta} \mathbb{E}^{do(\pi)}_\theta[Y_T] \]

Interpretation: Policy that works well even in worst-case scenario.

29.5.4 Active Inference and the Free Energy Perspective

So far we have treated decision-making in terms of policies that maximise expected outcomes under explicit CDMs. The Free Energy Principle (FEP) and active inference offer a complementary view in which agents are modelled as possessing internal generative models of their environment and acting to minimise expected surprise (variational free energy) under those models (Friston et al. 2006; Friston 2010, 2013). In this framing, policies are chosen not only to optimise a terminal outcome \(Y_T\), but to maintain the agent within a preferred region of state space, a homeostatic or viable set that defines what it is for the agent (or system) to persist.

Markov blankets provide the structural interface between agents and environments: internal states encode beliefs (and preferences) about external states, sensory states report what is happening at the boundary, and active states change the environment so that future sensory inputs remain within expected bounds. This can be read as a formalisation of goal-directed control under uncertainty: an agent acts to keep future observations within a preferred range (a viable region of state space), given its generative model (Whitehead 1929).

We will not develop active inference in full generality here, but it is worth noting that many of the objects we have already introduced (policies \(\pi\), Markov blankets, CDMs, and robustness notions) have natural counterparts in the FEP literature. In particular, robust policies can be interpreted as those that keep the system within a low-free-energy region of its state space despite uncertainty and perturbations, maintaining both causal coherence and functional viability.

29.5.5 Minimal Active Inference Example in Julia (Sketch)

To illustrate how these ideas can be implemented, we sketch a minimal active inference setup in Julia using a discrete-time, discrete-state model. The goal is for an agent to keep a scalar hidden state \(x_t\) near a preferred value (homeostasis) by choosing actions \(a_t \in \{-1, 0, +1\}\) that adjust \(x_t\). We use a simple tabular generative model and an approximate free-energy-based policy evaluation scheme.

# Find project root and include ensure_packages.jl
project_root = let
    current = pwd()
    while !isfile(joinpath(current, "Project.toml")) && !isfile(joinpath(current, "_quarto.yml"))
        parent = dirname(current)
        parent == current && break
        current = parent
    end
    current
end
include(joinpath(project_root, "scripts", "ensure_packages.jl"))
@auto_using Distributions CairoMakie

# Activate SVG output for responsive figures

# Discrete hidden states x ∈ {1,2,3,4,5}, observations y = x + noise, actions a ∈ {-1,0,+1}
states = 1:5
actions = [-1, 0, 1]
observ_noise = Normal(0.0, 0.3)

# Preferred state distribution: prefer x=3
prior_pref = pdf.(Distributions.Categorical([0.05, 0.1, 0.7, 0.1, 0.05]), states)

# Transition model p(x_{t+1} | x_t, a_t)
function transition_probs(x, a)
    x_next = clamp(x + a, first(states), last(states))
    probs = zeros(length(states))
    probs[x_next] = 1.0  # deterministic for simplicity
    return probs
end

# Likelihood p(y_t | x_t)
function likelihood_y_given_x(y, x)
    pdf(observ_noise, y - x)
end

# One-step expected free energy for action a from belief q(x_t)
function expected_free_energy(q_x, a;
        n_samples = 21, y_min = 0.5, y_max = 5.5)
    # Predictive distribution over x_{t+1}
    q_x_next = zeros(length(states))
    for (i, x) in enumerate(states)
        trans = transition_probs(x, a)
        q_x_next .+= q_x[i] .* trans
    end

    # Approximate predictive distribution over observations y_{t+1}
    ys = range(y_min, y_max; length = n_samples)
    q_y = zeros(length(ys))
    for (j, y) in enumerate(ys)
        for (i, x) in enumerate(states)
            q_y[j] += q_x_next[i] * likelihood_y_given_x(y, x)
        end
    end
    q_y ./= sum(q_y)  # normalise

    # Expected "risk" term: divergence from preferred states (via observations as proxy)
    # Here we use a simple surrogate: map y back to nearest state and compare to prior_pref
    risk = 0.0
    for (j, y) in enumerate(ys)
        x_idx = clamp(round(Int, y), first(states), last(states))
        p_pref = prior_pref[x_idx]
        if q_y[j] > 0 && p_pref > 0
            risk += q_y[j] * log(q_y[j] / p_pref)
        end
    end

    return risk
end

# Simulate active inference control
T = 30
x_true = zeros(Int, T)
y_obs = zeros(Float64, T)
q_hist = Vector{Vector{Float64}}(undef, T)
a_hist = zeros(Int, T)

# Initial true state and belief
x_true[1] = 2
q_x = fill(1.0 / length(states), length(states))  # initial uniform belief

for t in 1:T
    # Observe y_t
    if t == 1
        y_obs[t] = x_true[t] + rand(observ_noise)
    else
        y_obs[t] = x_true[t] + rand(observ_noise)
    end

    # Bayesian update of q(x_t | y_{1:t}) via simple likelihood multiplication
    for (i, x) in enumerate(states)
        q_x[i] *= likelihood_y_given_x(y_obs[t], x)
    end
    q_x ./= sum(q_x)
    q_hist[t] = copy(q_x)

    # Evaluate candidate actions by approximate expected free energy
    g_vals = [expected_free_energy(q_x, a) for a in actions]
    # Choose action that minimises expected free energy
    a_idx = argmin(g_vals)
    a_t = actions[a_idx]
    a_hist[t] = a_t

    # Evolve true state
    if t < T
        x_next_probs = transition_probs(x_true[t], a_t)
        x_true[t + 1] = rand(Distributions.Categorical(x_next_probs))
    end
end

# Visualise true state, observations, and actions
let
    fig = Figure(size = (900, 500))
    ax1 = Axis(fig[1, 1],
        xlabel = "Time",
        ylabel = "State / observation",
        title = "Active-inference-style control towards preferred state"
    )

    lines!(ax1, 1:T, x_true,
        label = "True state x_t",
        color = :blue,
        linewidth = 2
    )
    scatter!(ax1, 1:T, y_obs,
        label = "Observations y_t",
        color = :black,
        markersize = 4,
        alpha = 0.6
    )
    hlines!(ax1, [3.0],
        color = :green,
        linestyle = :dash,
        label = "Preferred state"
    )

    ax2 = Axis(fig[2, 1],
        xlabel = "Time",
        ylabel = "Action a_t",
        title = "Selected actions"
    )
    stem!(ax2, 1:T, a_hist,
        color = :red,
        markersize = 6
    )

    axislegend(ax1, position = :rb)
    fig  # Only this gets displayed
end

Minimal active-inference-style control: maintaining a latent state near a preferred value

This toy example is not a full implementation of ActiveInference.jl, but it shows the core ideas: a generative model over hidden states and observations, a preference distribution, approximate Bayesian updating of beliefs, and action selection via an expected-free-energy-like objective. The same belief-update step can be run with RxInfer message passing (conjugate Gaussian head below); full policy search under FEP remains the province of ActiveInference.jl for larger models.

29.5.6 RxInfer belief update (conjugate latent)

Suppose the agent’s preferred state is a latent mean \(x\) observed through noisy assays \(y_i = x + v_i\). Variational message passing updates \(q(x)\) in milliseconds:

include(joinpath(dirname(Base.active_project()), "scripts", "book_bootstrap.jl"))
@auto_using CairoMakie Random Statistics
using RxInfer
using Distributions

Random.seed!(24)
n_obs = 40
x_true = 0.8
y = x_true .+ 0.5 .* randn(n_obs)

# Qualify the macro: Turing also exports @model
RxInfer.@model function latent_mean(y, n)
    x ~ Normal(mean = 0.0, var = 1.0)
    for i in 1:n
        y[i] ~ Normal(mean = x, var = 0.25)
    end
end

t_bel = @elapsed begin
    global bel = infer(
        model = latent_mean(n = n_obs),
        data = (y = y,),
        iterations = 20,
        returnvars = (x = KeepLast(),),
        showprogress = false,
    )
end
qx = bel.posteriors[:x]
println("q(x): mean≈", round(mean(qx); digits = 3),
    "  var≈", round(var(qx); digits = 4),
    "  (", round(t_bel; digits = 4), " s)")

# One-step action: move toward preferred state x* = 1 with cost ∝ (E[x]-x*)² + Var(x)
x_star = 1.0
a = x_star - mean(qx)   # open-loop correction toward preference
println("Preferred x*=", x_star, "  → suggested shift a≈", round(a; digits = 3))
q(x): mean≈0.799  var≈0.0062  (6.983 s)
Preferred x*=1.0  → suggested shift a≈0.201

RxInfer posterior \(q(x)\) after \(n=40\) noisy assays (density of the conjugate Gaussian marginal). Vertical lines: true latent (blue) and preferred state (amber).

29.5.7 Implementation: Decision-Making Under Uncertainty

Here’s an example of making robust decisions under parameter uncertainty:

# Find project root and include ensure_packages.jl
project_root = let
    current = pwd()
    while !isfile(joinpath(current, "Project.toml")) && !isfile(joinpath(current, "_quarto.yml"))
        parent = dirname(current)
        parent == current && break
        current = parent
    end
    current
end
include(joinpath(project_root, "scripts", "ensure_packages.jl"))

@auto_using Random Distributions OrdinaryDiffEq CairoMakie

Random.seed!(42)

# Example: Treatment decision under parameter uncertainty
# We're uncertain about treatment effectiveness α ∈ [0.2, 0.4]

β = 0.2  # Disease progression (known)
α_values = [0.2, 0.3, 0.4]  # Uncertain treatment effectiveness

# Policies to consider
policy1 = (X, t) -> X > 0.6 ? 1.0 : 0.0  # Conservative
policy2 = (X, t) -> X > 0.4 ? 1.0 : 0.0  # Moderate
policy3 = (X, t) -> 1.0  # Aggressive

function disease_uncertain!(du, u, p, t)
    """Disease model with policy and uncertain treatment effect α."""
    X = u[1]
    policy, α = p
    A = policy(X, t)
    du[1] = β * X - α * A * X
end

u0 = [0.3]
tspan = (0.0, 20.0)

# Evaluate each policy under each parameter value
policies = [("Conservative", policy1), ("Moderate", policy2), ("Aggressive", policy3)]
results = Dict()

for (name, policy) in policies
    outcomes = Float64[]
    for α in α_values
        prob = ODEProblem(disease_uncertain!, u0, tspan, (policy, α))  # Using tuple for better performance
        sol = solve(prob, Tsit5())
        push!(outcomes, sol.u[end][1])
    end
    results[name] = outcomes
end

# Robust decision: minimise worst-case outcome
worst_case = Dict(name => maximum(outcomes) for (name, outcomes) in results)
robust_policy = first(keys(worst_case))  # Initialize
for (name, worst) in worst_case
    if worst < worst_case[robust_policy]
        global robust_policy = name
    end
end

println("Decision-making under uncertainty:")
for (name, outcomes) in results
    println("  ", name, ": outcomes = ", [round(o, digits=3) for o in outcomes])
    println("    worst-case = ", round(worst_case[name], digits=3))
end
println("\nRobust policy (minimises worst-case): ", robust_policy)
Decision-making under uncertainty:
  Moderate: outcomes = [0.404, 0.402, 0.4]
    worst-case = 0.404
  Aggressive: outcomes = [0.3, 0.041, 0.005]
    worst-case = 0.3
  Conservative: outcomes = [0.61, 0.603, 0.6]
    worst-case = 0.61

Robust policy (minimises worst-case): Aggressive
Figure 29.1: Decision-making under uncertainty: robust policy selection

29.6 Stratum context

This chapter addresses Doing in the Observable stratum: how can we make decisions using causal models? Causal decision-making uses observable data and causal models to choose actions that maximise expected outcomes, bridging the Observable stratum (what we observe) with the Structural stratum (what would happen under interventions).

29.7 Summary

Policies are interventional programmes; robust choice accounts for parameter and structural uncertainty. Active-inference glosses (preferred states, expected free energy) sit alongside CDMs; RxInfer supplies fast conjugate belief updates when the generative model is message-passing friendly.

  1. Markov boundary: Minimal set for causal reasoning about target variables
  2. Markov blanket: Principled system boundary from FEP perspective
  3. Unified principle: Combine both to determine CDM scope
  4. Causal decision-making: Choose actions to maximise expected outcomes
  5. Uncertainty matters: Robust decisions account for model/parameter uncertainty

29.8 Further Reading