28  Causal Decision-Making

Status: Draft

v0.4

28.1 Introduction

Decision-making requires answering “what if” questions: what happens under different policies? (Murphy 2003; Schulam and Saria 2017; Tennenholtz et al. 2020) This chapter formalises causal decision-making. Markov boundaries and blankets can be diagnostic descriptions, but neither supplies an automatic rule for CDM scope. A decision model must first declare the target entity, outcome support, information available to the policy, and observation process.

28.2 Multi-agent causal decisions

In a multi-agent system, the decision problem includes the way in which each agent selects an action. A normal-form payoff table usually averages over the agents’ natural responses and therefore hides a distinction between an action generated by a mechanism, an intervention that replaces that mechanism, and a counterfactual policy that conditions on an agent’s latent context. Causal game theory makes these alternatives explicit by organising action spaces according to Pearl’s hierarchy (Maiti et al. 2025).

For CDCS, the useful point is methodological rather than terminological. A policy should declare its information set and whether it is a natural assignment, intervention, or counterfactual response rule. If agents can choose which class of rule to use, the layer choice becomes part of the dynamical decision problem and may change the attainable equilibria. This extension is optional: ordinary policy evaluation remains well defined without a game-theoretic layer-selection model.

A fundamental question in causal decision-making is: which variables (nodes) should be included or excluded in a CDM? The answer depends on the estimand, measurement design, causal assumptions and anticipated interventions. A Markov boundary is conditional on a graph or distribution; a Markov blanket is a conditional-independence partition. Neither proves causal relevance, organismal individuality, cognition, or an admissible variable-selection rule.

28.3 Determining CDM Scope: Markov Boundary and Markov Blanket

28.3.1 The Problem: What to Include in a CDM?

When constructing a CDM for decision-making, we must decide which variables (nodes) to include. Including too many variables makes the model unnecessarily complex and computationally expensive. Including too few variables risks omitting important causal relationships, leading to biased policy evaluations.

The Markov boundary (introduced in Graph Theory and Causal Patterns) and Markov blanket (introduced in From Structure to Time: FEP and Attractors) are conditional modelling summaries, not principled automatic answers to this question.

28.3.2 Markov Boundary: Graph-Theoretic Minimal Set

The Markov boundary of a target variable \(Y\) is a minimal graph-theoretic set that d-separates \(Y\) from the remaining variables under the stated graph or distribution (Pearl 1988, 2014). It is a local conditional-dependence summary, not a causal adjustment or counterfactual-sufficiency theorem:

  • Level 1 (Association): Given \(\text{MB}(Y)\), other variables are screened off for the graph’s local predictive factorisation
  • Level 2 (Intervention): Confounder adjustment requires a separate identification analysis for the estimand
  • Level 3 (Counterfactual): Counterfactual computation requires the structural assignments, latent-state posterior and declared cross-world coupling

Inclusion principle: Consider the Markov boundary as one diagnostic input to CDM scope. Include it only when it is relevant to the estimand, measurement design and anticipated interventions; it does not guarantee causal sufficiency.

28.3.3 Markov blanket: optional conditional-independence partition

Following Chapter 9, a Markov blanket is a conditional-independence partition that can be useful in an explicitly assumed FEP model (Friston 2010, 2013). It is not, by itself, a formalisation of organismal boundaries or a criterion for including variables:

Blanket role Symbol CDM variable Gloss
Organism \(\mu\) \(X_t\) Internal state evolving under \(f\)
Environment \(\eta\) (outside model) Unmodelled world beyond \(\mu\)
Sensory / observation \(s\) \(Y_t\) What measurement gives
Active / action \(a\) Endogenous action state Generated within the model; distinct from an investigator’s \(do(\cdot)\)

\(\mu ⫫ \eta \mid (s,a)\): the environment is not observed directly, only through \(s\); the organism acts on the environment only through \(a\).

The partition is a modelling hypothesis to assess alongside domain knowledge and the estimand. An endogenous active state is not an investigator intervention: \(do(\cdot)\) replaces an assignment, while an action remains generated by the declared system.

28.3.4 CDM scope: an estimand-led specification

The Markov boundary (Ch. 3) can describe conditional dependence around \(Y\) under a selected graph; a blanket can describe an assumed partition. Neither fixes where an organism ends or which variables a causal analysis requires.

Specify scope from the estimand, temporal ordering, plausible confounding, measurement process and feasible interventions. Use boundary or blanket checks as sensitivity analyses, not as a union rule. Environment shift (new site, protocol) is a transport problem (Ch. 8), not a conclusion from a blanket.

28.4 Integrating Markov Boundary and Markov Blanket

28.4.1 Unified Principle for CDM Construction

When constructing a CDM for causal decision-making, begin with the estimand and use graph-derived summaries only as checks:

  1. Identify the target variable(s): What outcomes are we interested in? (e.g., \(Y_T\) for policy evaluation)

  2. Assess conditional-dependence summaries: For each target variable, inspect a Markov boundary under the stated graph or distribution; it is not a universal adjustment or counterfactual set.

  3. State any blanket assumption separately: If an FEP interpretation is intended, declare the conditional-independence partition and distinguish endogenous active states from external \(do(\cdot)\) interventions.

  4. Justify the analysis variables: Include variables required by the estimand, temporal ordering, measurement model and confounding assumptions; document exclusions and test sensitivity where possible.

28.4.2 Implementation: Determining CDM Scope

The following example uses the Markov boundary and Markov blanket to determine CDM scope:

# Find project root and include ensure_packages.jl
project_root = let
    current = pwd()
    while !isfile(joinpath(current, "Project.toml")) && !isfile(joinpath(current, "_quarto.yml"))
        parent = dirname(current)
        parent == current && break
        current = parent
    end
    current
end
include(joinpath(dirname(Base.active_project()), "scripts", "book_bootstrap.jl"))

@auto_using DAGMakie CairoMakie CausalDynamics Graphs

# Example: Decision-making about outcome Y
# Graph: Z → X → Y ← W, with Z → W
# Target: Y (outcome we care about)

g = SimpleDiGraph(4)
add_edge!(g, 1, 2)  # Z → X
add_edge!(g, 2, 3)  # X → Y
add_edge!(g, 4, 3)  # W → Y
add_edge!(g, 1, 4)  # Z → W

# Step 1: Compute Markov boundary of Y (target variable)

mb_Y = CausalDynamics.markov_boundary(g, 3)  # Markov boundary of Y (node 3)
println("Markov boundary of Y: ", mb_Y)  # Should be [2, 4] = {X, W}

# Step 2: Identify Markov blanket components
# Internal states (X): Latent process states
# Sensory states (Y): Observations
# Active states (A): Actions/interventions (if we add them)
# External states: Everything else

internal_states = [2]  # X (latent)
sensory_states = [3]  # Y (observed)
# Active states would be added if we have interventions

markov_blanket = sort(unique([internal_states; sensory_states]))
println("Markov blanket: ", markov_blanket)

# Step 3: Report these as conditional-dependence summaries, not an automatic scope.
println("Candidate Markov-boundary summary: ", sort!(collect(mb_Y)))
println("Candidate blanket summary: ", markov_blanket)
println("Specify CDM scope from the estimand, temporal order, measurement, and intervention assumptions.")

# Visualise
let
    node_colors = [
        RGBf(0.30, 0.55, 0.70),
        RGBf(0.85, 0.65, 0.05),
        RGBf(0.30, 0.55, 0.70),
        RGBf(0.85, 0.65, 0.05),
    ]

    fig, ax, p = dagplot(g;
        figure_size = (800, 600),
        layout_mode = :acyclic,
        node_color = node_colors,
        labels = ["Z", "X", "Y", "W"],
        node_size = 40,
        edge_width = 2.0,
        arrow_size = 15,
        nlabels_fontsize = 16,
        nlabels_color = :white,
        nlabels_align = (:center, :center),
        nlabels_distance = 0,
        padding = 0.40,
    )
    
    fig  # Only this gets displayed
end
Markov boundary of Y: Set([2, 4])
Markov blanket: [2, 3]
Candidate Markov-boundary summary: [2, 4]
Candidate blanket summary: [2, 3]
Specify CDM scope from the estimand, temporal order, measurement, and intervention assumptions.

Determining CDM scope using Markov boundary and Markov blanket

28.5 Policies as Interventions

28.5.1 Definition

A policy (dynamic treatment strategy) is a function that maps history to action:

\[ A_t = \pi(H_t) \]

where \(H_t = (Y_{1:t}, A_{1:t-1}, L_{1:t})\) is the observed history.

28.5.2 Policy Intervention

Under policy \(\pi\), we set: \[ do(\pi): \quad A_t = \pi(H_t) \]

Interpretation: Structural intervention that sets treatment according to policy rule.

28.6 Causal Decision-Making Framework

28.6.1 The Decision Problem

Question: Which action should we take to maximise expected outcome?

Framework:

  1. Define objective: What do we want to maximise/minimise?
  2. Model interventions: How do actions affect outcomes?
  3. Optimize: Choose action that maximises expected outcome

28.6.2 Optimal Policy

The optimal policy maximises expected outcome:

\[ \pi^* = \arg\max_\pi \mathbb{E}^{do(\pi)}[Y_T] \]

Methods (Murphy 2003; Schulam and Saria 2017):

28.6.3 Uncertainty in Decision-Making

Decision-making must account for:

  • Model uncertainty: Which model is correct?
  • Parameter uncertainty: What are the parameter values?
  • Structural uncertainty: What is the causal structure?
  • Process uncertainty: Intrinsic stochasticity
  • Observation uncertainty: Measurement error

Robust decisions: Choose actions that perform well across model/parameter uncertainty.

Robust policy: Find policy that performs well under uncertainty:

\[ \pi^* = \arg\max_\pi \min_{\theta \in \Theta} \mathbb{E}^{do(\pi)}_\theta[Y_T] \]

Interpretation: Policy that works well even in worst-case scenario.

28.6.4 Active Inference and the Free Energy Perspective

So far we have treated decision-making in terms of policies that maximise expected outcomes under explicit CDMs. The Free Energy Principle (FEP) and active inference offer an optional, assumption-dependent interpretation in which an explicitly modelled agent has a generative model and a variational objective (Friston et al. 2006; Friston 2010, 2013; Friston and Ao 2012). That interpretation is not entailed by a CDM, a Markov blanket, a robust policy, or a preferred state set.

Under those additional assumptions, internal, sensory and active states can be given that interpretation. A Markov blanket alone establishes only a conditional-independence relation; it does not establish beliefs, preferences, cognition, agency or active inference. Endogenous active states also remain distinct from investigator interventions.

We will not develop active inference in full generality here. In a fitted FEP model, a preferred state density, expected free energy, and policy-selection rule can represent a system’s prospective orientation and support empirical comparison. Their fit does not, by itself, settle the metaphysical status of final causation. A basin of attraction belongs to the geometry of the declared dynamics, whereas a preference distribution belongs to the model’s objective; each offers a partial empirical handle on organised orientation.1 See Chapter 9 for the operational distinction.

Evidence for an agent-level interpretation should instead combine explicit model comparison with perturbation results: return or reorganisation after damage, adaptive flexibility across routes to a viable state, and counterfactual robustness under stated interventions. Endogenous active states \(A_t\) remain distinct from an investigator’s \(do(A_t=a)\), which replaces their assignment rather than describing the system’s own action.

Cybernetics and enactivism suggest a more demanding practical standard. A claim of autonomous regulation should identify the viable variables, the disturbances that threaten them, the information and action repertoire available to the system, and the mechanism network through which that repertoire is maintained (Ashby 1956; Thompson 2007). The relevant variety is always scale- and task-dependent: a controller may regulate one declared variable well while lacking the means to preserve a wider organisation. This specification turns a broad claim of agency into a programme of perturbation, prediction, and decision tests.

28.6.5 Minimal Active Inference Example in Julia (Sketch)

To illustrate how these ideas can be implemented, we sketch a minimal active inference setup in Julia using a discrete-time, discrete-state model. The goal is for an agent to keep a scalar hidden state \(x_t\) near a preferred value (homeostasis) by choosing actions \(a_t \in \{-1, 0, +1\}\) that adjust \(x_t\). We use a simple tabular generative model and an approximate free-energy-based policy evaluation scheme.

# Find project root and include ensure_packages.jl
project_root = let
    current = pwd()
    while !isfile(joinpath(current, "Project.toml")) && !isfile(joinpath(current, "_quarto.yml"))
        parent = dirname(current)
        parent == current && break
        current = parent
    end
    current
end
include(joinpath(project_root, "scripts", "ensure_packages.jl"))
@auto_using Distributions CairoMakie

# Activate SVG output for responsive figures

# Discrete hidden states x ∈ {1,2,3,4,5}, observations y = x + noise, actions a ∈ {-1,0,+1}
states = 1:5
actions = [-1, 0, 1]
observ_noise = Normal(0.0, 0.3)

# Preferred state distribution: prefer x=3
prior_pref = pdf.(Distributions.Categorical([0.05, 0.1, 0.7, 0.1, 0.05]), states)

# Transition model p(x_{t+1} | x_t, a_t)
function transition_probs(x, a)
    x_next = clamp(x + a, first(states), last(states))
    probs = zeros(length(states))
    probs[x_next] = 1.0  # deterministic for simplicity
    return probs
end

# Likelihood p(y_t | x_t)
function likelihood_y_given_x(y, x)
    pdf(observ_noise, y - x)
end

# One-step expected free energy for action a from belief q(x_t)
function expected_free_energy(q_x, a;
        n_samples = 21, y_min = 0.5, y_max = 5.5)
    # Predictive distribution over x_{t+1}
    q_x_next = zeros(length(states))
    for (i, x) in enumerate(states)
        trans = transition_probs(x, a)
        q_x_next .+= q_x[i] .* trans
    end

    # Approximate predictive distribution over observations y_{t+1}
    ys = range(y_min, y_max; length = n_samples)
    q_y = zeros(length(ys))
    for (j, y) in enumerate(ys)
        for (i, x) in enumerate(states)
            q_y[j] += q_x_next[i] * likelihood_y_given_x(y, x)
        end
    end
    q_y ./= sum(q_y)  # normalise

    # Expected "risk" term: divergence from preferred states (via observations as proxy)
    # Here we use a simple surrogate: map y back to nearest state and compare to prior_pref
    risk = 0.0
    for (j, y) in enumerate(ys)
        x_idx = clamp(round(Int, y), first(states), last(states))
        p_pref = prior_pref[x_idx]
        if q_y[j] > 0 && p_pref > 0
            risk += q_y[j] * log(q_y[j] / p_pref)
        end
    end

    return risk
end

# Simulate active inference control
T = 30
x_true = zeros(Int, T)
y_obs = zeros(Float64, T)
q_hist = Vector{Vector{Float64}}(undef, T)
a_hist = zeros(Int, T)

# Initial true state and belief
x_true[1] = 2
q_x = fill(1.0 / length(states), length(states))  # initial uniform belief

for t in 1:T
    # Observe y_t
    if t == 1
        y_obs[t] = x_true[t] + rand(observ_noise)
    else
        y_obs[t] = x_true[t] + rand(observ_noise)
    end

    # Bayesian update of q(x_t | y_{1:t}) via simple likelihood multiplication
    for (i, x) in enumerate(states)
        q_x[i] *= likelihood_y_given_x(y_obs[t], x)
    end
    q_x ./= sum(q_x)
    q_hist[t] = copy(q_x)

    # Evaluate candidate actions by approximate expected free energy
    g_vals = [expected_free_energy(q_x, a) for a in actions]
    # Choose action that minimises expected free energy
    a_idx = argmin(g_vals)
    a_t = actions[a_idx]
    a_hist[t] = a_t

    # Evolve true state
    if t < T
        x_next_probs = transition_probs(x_true[t], a_t)
        x_true[t + 1] = rand(Distributions.Categorical(x_next_probs))
    end
end

# Visualise true state, observations, and actions
let
    fig = Figure(size = (900, 500))
    ax1 = Axis(fig[1, 1],
        xlabel = "Time",
        ylabel = "State / observation",
        title = "Active-inference-style control towards preferred state"
    )

    lines!(ax1, 1:T, x_true,
        label = "True state x_t",
        color = :blue,
        linewidth = 2
    )
    scatter!(ax1, 1:T, y_obs,
        label = "Observations y_t",
        color = :black,
        markersize = 4,
        alpha = 0.6
    )
    hlines!(ax1, [3.0],
        color = :green,
        linestyle = :dash,
        label = "Preferred state"
    )

    ax2 = Axis(fig[2, 1],
        xlabel = "Time",
        ylabel = "Action a_t",
        title = "Selected actions"
    )
    stem!(ax2, 1:T, a_hist,
        color = :red,
        markersize = 6
    )

    axislegend(ax1, position = :rb)
    fig  # Only this gets displayed
end

Minimal active-inference-style control: maintaining a latent state near a preferred value

This toy example is not a full implementation of ActiveInference.jl, but it shows the core ideas: a generative model over hidden states and observations, a preference distribution, approximate Bayesian updating of beliefs, and action selection via an expected-free-energy-like objective. The same belief-update step can be run with RxInfer message passing (conjugate Gaussian head below); full policy search under FEP remains the province of ActiveInference.jl for larger models.

28.6.6 RxInfer belief update (conjugate latent)

Suppose the agent’s preferred state is a latent mean \(x\) observed through noisy assays \(y_i = x + v_i\). Variational message passing updates \(q(x)\) in milliseconds:

include(joinpath(dirname(Base.active_project()), "scripts", "book_bootstrap.jl"))
@auto_using CairoMakie Random Statistics
using RxInfer
using Distributions

Random.seed!(24)
n_obs = 40
x_true = 0.8
y = x_true .+ 0.5 .* randn(n_obs)

# Qualify the macro: Turing also exports @model
RxInfer.@model function latent_mean(y, n)
    x ~ Normal(mean = 0.0, var = 1.0)
    for i in 1:n
        y[i] ~ Normal(mean = x, var = 0.25)
    end
end

t_bel = @elapsed begin
    global bel = infer(
        model = latent_mean(n = n_obs),
        data = (y = y,),
        iterations = 20,
        returnvars = (x = KeepLast(),),
        showprogress = false,
    )
end
qx = bel.posteriors[:x]
println("q(x): mean≈", round(mean(qx); digits = 3),
    "  var≈", round(var(qx); digits = 4),
    "  (", round(t_bel; digits = 4), " s)")

# One-step action: move toward preferred state x* = 1 with cost ∝ (E[x]-x*)² + Var(x)
x_star = 1.0
a = x_star - mean(qx)   # open-loop correction toward preference
println("Preferred x*=", x_star, "  → suggested shift a≈", round(a; digits = 3))
q(x): mean≈0.799  var≈0.0062  (6.7196 s)
Preferred x*=1.0  → suggested shift a≈0.201

RxInfer posterior \(q(x)\) after \(n=40\) noisy assays (density of the conjugate Gaussian marginal). Vertical lines: true latent (blue) and preferred state (amber).

28.6.7 Implementation: Decision-Making Under Uncertainty

The following example compares decisions under parameter uncertainty:

# Find project root and include ensure_packages.jl
project_root = let
    current = pwd()
    while !isfile(joinpath(current, "Project.toml")) && !isfile(joinpath(current, "_quarto.yml"))
        parent = dirname(current)
        parent == current && break
        current = parent
    end
    current
end
include(joinpath(project_root, "scripts", "ensure_packages.jl"))

@auto_using Random Distributions OrdinaryDiffEq CairoMakie

Random.seed!(42)

# Example: Treatment decision under parameter uncertainty
# We're uncertain about treatment effectiveness α ∈ [0.2, 0.4]

β = 0.2  # Disease progression (known)
α_values = [0.2, 0.3, 0.4]  # Uncertain treatment effectiveness

# Policies to consider
policy1 = (X, t) -> X > 0.6 ? 1.0 : 0.0  # Conservative
policy2 = (X, t) -> X > 0.4 ? 1.0 : 0.0  # Moderate
policy3 = (X, t) -> 1.0  # Aggressive

function disease_uncertain!(du, u, p, t)
    """Disease model with policy and uncertain treatment effect α."""
    X = u[1]
    policy, α = p
    A = policy(X, t)
    du[1] = β * X - α * A * X
end

u0 = [0.3]
tspan = (0.0, 20.0)

# Evaluate each policy under each parameter value
policies = [("Conservative", policy1), ("Moderate", policy2), ("Aggressive", policy3)]
results = Dict()

for (name, policy) in policies
    outcomes = Float64[]
    for α in α_values
        prob = ODEProblem(disease_uncertain!, u0, tspan, (policy, α))  # Using tuple for better performance
        sol = solve(prob, Tsit5())
        push!(outcomes, sol.u[end][1])
    end
    results[name] = outcomes
end

# Robust decision: minimise worst-case outcome
worst_case = Dict(name => maximum(outcomes) for (name, outcomes) in results)
robust_policy = first(keys(worst_case))  # Initialize
for (name, worst) in worst_case
    if worst < worst_case[robust_policy]
        global robust_policy = name
    end
end

println("Decision-making under uncertainty:")
for (name, outcomes) in results
    println("  ", name, ": outcomes = ", [round(o, digits=3) for o in outcomes])
    println("    worst-case = ", round(worst_case[name], digits=3))
end
println("\nRobust policy (minimises worst-case): ", robust_policy)
Decision-making under uncertainty:
  Aggressive: outcomes = [0.3, 0.041, 0.005]
    worst-case = 0.3
  Conservative: outcomes = [0.61, 0.603, 0.6]
    worst-case = 0.61
  Moderate: outcomes = [0.404, 0.402, 0.4]
    worst-case = 0.404

Robust policy (minimises worst-case): Aggressive
Figure 28.1: Decision-making under uncertainty: robust policy selection

28.7 Stratum context

Causal decision-making combines observable data with structural assumptions to choose actions that maximise expected outcomes under a declared decision problem.

28.8 Summary

Policies are interventional programmes; robust choice accounts for parameter and structural uncertainty. Markov boundaries and blankets set CDM scope. Active-inference glosses (preferred states, expected free energy) sit alongside CDMs; RxInfer supplies fast conjugate belief updates when the generative model is message-passing friendly.

28.9 Further Reading

  1. In Whiteheadian usage, a lure names the relevance of a possibility to a becoming. CDCS treats constraints, preferences, policies, and reachable-set geometry as possible empirical correlates of such orientation. They leave open the lure’s intensive relevance and do not turn it into a state variable or efficient cause (Whitehead 1978; Segall 2025, 2026).↩︎