27  Policy Evaluation and Dynamic Treatment Strategies

Status: Draft

v0.5

27.1 Introduction

Dynamic treatment strategies can be evaluated from data even when the full mechanism is not known. Mechanistic policy simulation belongs to Part II; here L2 uses identified functionals and estimators. A dynamic treatment strategy is a rule that maps a declared information history to treatment. This is Doing from the Observable role.

27.1.1 When the behaviour policy sees more than the learner

Offline policy learning adds a measurement problem to the usual overlap problem. A demonstrator may choose actions using a latent or high-dimensional state, while the learner receives only a partial observation of that state. For example, an expert may act on a structured scene representation while the stored dataset pairs the action with an occluded image. Treating the image as a complete Markov state can then induce implicit confounding: the action carries information about an unrecorded state that also affects reward and transition.

This is not repaired by calling the recorded observation a state. The analysis must declare the observation map, the latent state, and the behavioural mechanism. When those assumptions are insufficient for point identification, a robust policy objective can optimise a lower bound over compatible confounded environments rather than report a spurious point value (Li et al. 2026). The practical lesson for CDCS is that policy evaluation belongs at the Observable–Dynamical boundary: the policy acts on an information set, while the transition law may depend on latent state and shared exogenous variation.

27.2 Dynamic Treatment Strategies

27.2.1 Definition

A dynamic treatment strategy (policy) is a rule that maps history to treatment:

\[ A_t = \pi(H_t) \]

where \(H_t = (Y_{1:t}, A_{1:t-1}, L_{1:t})\) is the observed history.

27.2.2 Examples

  • “Treat if severity > threshold”: \(A_t = \mathbb{1}(L_t > \tau)\)
  • “Stop if recovered”: \(A_t = \mathbb{1}(Y_{t-1} = 0)\)
  • “Adaptive dosing”: \(A_t = f(Y_{1:t})\)

27.3 Evaluating Policies

27.3.1 The Policy Evaluation Problem

Question: What is the expected outcome under policy \(\pi\), for the declared target population and horizon?

\[ \Psi(\pi) := \mathbb{E}[Y_T^{\pi}] \]

The right-hand side is not generally equal to an observed conditional mean among units whose actions happen to resemble \(\pi\). Equality requires the relevant identification assumptions, including consistency, exchangeability or a valid randomisation design, positivity, and a specified observation process.

27.3.2 Answer Using CDM

From a fitted CDM, compute: \[ \mathbb{E}^{do(\pi)}[Y_T] = \int Y_T \, P^{do(\pi)}(Y_T \mid Y_1) \, dY_T \]

Computation:

  1. Simulate trajectories under \(do(\pi)\)
  2. Average over trajectories
  3. Account for uncertainty

27.3.3 Methods for Policy Evaluation

G-computation: Simulate outcomes under policy

  • Fit outcome and confounder models
  • Simulate trajectories under policy
  • Average over simulated outcomes

IPTW: Weight observations by inverse probability of following policy

  • Fit treatment models
  • Compute weights: \(w_i = \prod_{t=1}^{T} \frac{1}{P(A_t = \pi(H_t) \mid H_t)}\)
  • Fit outcome model on weighted data

TMLE: Targeted estimation of policy effects

  • Fit initial outcome model
  • Fit treatment model for weights
  • Targeted update to estimate policy effect
  • Provides valid inference

27.3.4 Implementation: Policy Evaluation

The following example evaluates different policies using G-computation:

# Find project root and include ensure_packages.jl
project_root = let
    current = pwd()
    while !isfile(joinpath(current, "Project.toml")) && !isfile(joinpath(current, "_quarto.yml"))
        parent = dirname(current)
        parent == current && break
        current = parent
    end
    current
end
include(joinpath(dirname(Base.active_project()), "scripts", "book_bootstrap.jl"))

@auto_using Random Distributions OrdinaryDiffEq CairoMakie

Random.seed!(42)

# Example: Disease management policies
# Policy 1: Treat if severity > 0.6
# Policy 2: Treat if severity > 0.4
# Policy 3: Always treat

β = 0.2  # Disease progression
α = 0.3  # Treatment effectiveness

# Policy definitions
policy1 = (X, t) -> X > 0.6 ? 1.0 : 0.0  # Treat if severe
policy2 = (X, t) -> X > 0.4 ? 1.0 : 0.0  # Treat if moderate
policy3 = (X, t) -> 1.0  # Always treat

function disease_policy!(du, u, p, t)
    """Disease model with policy function: treatment determined by policy(X, t)."""
    X = u[1]
    policy = p[1]
    A = policy(X, t)
    du[1] = β * X - α * A * X
end

u0 = [0.3]
tspan = (0.0, 20.0)

# Evaluate each policy
policies = [("Treat if X > 0.6", policy1), 
            ("Treat if X > 0.4", policy2),
            ("Always treat", policy3)]
outcomes = Float64[]
trajectories = []

for (name, policy) in policies
    prob = ODEProblem(disease_policy!, u0, tspan, (policy,))  # Using tuple for better performance
    sol = solve(prob, Tsit5())
    push!(outcomes, sol.u[end][1])
    push!(trajectories, (name=name, sol=sol))
end

println("Policy evaluation results:")
for (i, (name, _)) in enumerate(policies)
    println("  ", name, ": Final severity = ", round(outcomes[i], digits=3))
end
best_policy = argmin(outcomes)
println("\nBest policy: ", policies[best_policy][1], " (lowest final severity)")
Policy evaluation results:
  Treat if X > 0.6: Final severity = 0.603
  Treat if X > 0.4: Final severity = 0.402
  Always treat: Final severity = 0.041

Best policy: Always treat (lowest final severity)
Figure 27.1: Policy evaluation: comparing different treatment strategies

27.3.5 Value Function

The value function of policy \(\pi\) is: \[ V^\pi = \mathbb{E}^{do(\pi)}\left[\sum_{t=1}^T \gamma^t R_t\right] \]

where \(R_t\) is the reward at time \(t\) and \(\gamma\) is the discount factor.

27.3.6 Comparing Policies

Compare expected outcomes under different policies:

\[ \mathbb{E}[Y^{do(\pi_1)}] \quad \text{vs} \quad \mathbb{E}[Y^{do(\pi_2)}] \]

This allows us to choose the best policy for a given objective.

27.4 Optimal Policy

27.4.1 Definition

The optimal policy maximises expected outcome:

\[ \pi^* = \arg\max_\pi \mathbb{E}^{do(\pi)}[Y_T] \]

27.4.2 Finding Optimal Policies

Methods (Murphy 2003; Schulam and Saria 2017):

27.5 Off-Policy Evaluation

27.5.1 The Problem

Evaluate policy \(\pi\) using data collected under different policy \(\pi_0\).

27.5.2 Methods (Tennenholtz et al. 2020; Sutton and Barto 2018):

  • Importance sampling: Reweight observations by \(\frac{\pi(A \mid H)}{\pi_0(A \mid H)}\)
  • Doubly robust methods: Combine model-based and importance sampling (e.g., TMLE, augmented IPTW)
  • Causal methods: Use CDM to simulate under \(\pi\)

27.6 Off-Policy Evaluation

27.6.1 The Problem

Evaluate policy \(\pi\) using data collected under different policy \(\pi_0\).

27.6.2 Methods (Tennenholtz et al. 2020; Sutton and Barto 2018):

  • Importance sampling: Reweight observations by \(\frac{\pi(A \mid H)}{\pi_0(A \mid H)}\)
  • Doubly robust methods: Combine model-based and importance sampling (e.g., TMLE, augmented IPTW)
  • Causal methods: Use CDM to simulate under \(\pi\)

27.6.3 Targeted Maximum Likelihood Estimation (TMLE) for Off-Policy Evaluation

TMLE provides a robust approach to off-policy evaluation that combines the strengths of importance sampling and model-based methods (Laan and Rubin 2006; Laan and Rose 2011).

27.6.3.1 TMLE Algorithm for Off-Policy Evaluation

Goal: Estimate \(E[Y^{\pi}]\) (expected outcome under policy \(\pi\)) using data collected under policy \(\pi_0\).

Steps:

  1. Fit initial outcome model: \(Q_0(A, H) = E[Y \mid A, H]\) using data from \(\pi_0\)
  2. Fit behaviour policy model: \(g_0(A \mid H) = \pi_0(A \mid H)\) (known or estimated)
  3. Targeted update: Update outcome model to target \(E[Y^{\pi}]\):
  • Compute clever covariate: \(H(A, H) = \frac{\pi(A \mid H)}{g_0(A \mid H)}\)
  • Fit logistic regression: \(\text{logit}(Q_1(A, H)) = \text{logit}(Q_0(A, H)) + \epsilon H(A, H)\)
  • Update: \(Q_1(A, H) = Q_0(A, H) + \epsilon H(A, H)\)
  1. Compute policy value: \[ \hat{E}[Y^{\pi}] = \frac{1}{n} \sum_{i=1}^n \sum_a \pi(a \mid H_i) Q_1(a, H_i) \]

Advantages:

  • Double robust: Consistent if either outcome model or behaviour policy model is correct
  • Semiparametric efficient: Achieves optimal variance
  • Valid inference: Provides confidence intervals
  • Handles policy mismatch: Works even when \(\pi\) and \(\pi_0\) differ substantially

Limitations:

  • Requires overlap: \(\pi(a \mid H) > 0\) implies \(\pi_0(a \mid H) > 0\) (positivity)
  • More complex than simple importance sampling

27.6.4 Implementation: Off-Policy Evaluation with Importance Sampling

The following example evaluates an off-policy policy using importance sampling:

# Find project root and include ensure_packages.jl
project_root = let
    current = pwd()
    while !isfile(joinpath(current, "Project.toml")) && !isfile(joinpath(current, "_quarto.yml"))
        parent = dirname(current)
        parent == current && break
        current = parent
    end
    current
end
include(joinpath(project_root, "scripts", "ensure_packages.jl"))

@auto_using Random Distributions GLM DataFrames CairoMakie

Random.seed!(42)

# Simulate data collected under behaviour policy π₀
# π₀: Treat if severity > 0.5
n = 500
L = rand(Uniform(0, 1), n)  # Severity
A_behavior = Int.(L .> 0.5)  # Behavior policy: treat if L > 0.5
Y = 0.5 .* A_behavior .- 0.3 .* L .+ rand(Normal(0, 0.1), n)  # Outcome

df = DataFrame(L = L, A = A_behavior, Y = Y)

# Target policy π: Treat if severity > 0.3 (more aggressive)
π_target = (L) -> L > 0.3 ? 1 : 0
π_behavior = (L) -> L > 0.5 ? 1 : 0

# Fit behaviour policy model g₀(A | L) = π₀(A | L)
g0_model = glm(@formula(A ~ L), df, Binomial(), LogitLink())

# Compute importance sampling weights
# w = π(A | L) / π₀(A | L)
weights = Float64[]
for i in 1:n
    L_i = L[i]
    A_i = A_behavior[i]
    
    # π₀(A | L) from fitted model
    p_behavior = GLM.predict(g0_model, DataFrame(L = [L_i]))[1]
    π0_A = A_i == 1 ? p_behavior : (1 - p_behavior)
    
    # π(A | L) from target policy
    # Target policy is deterministic: π_target(L) returns 0 or 1
    # Probability of observed action A_i under target policy
    π_target_prob = π_target(L_i)  # This is 0 or 1 (deterministic policy)
    π_A = A_i == 1 ? π_target_prob : (1 - π_target_prob)
    
    # Weight (with stabilisation to avoid extreme values)
    # Avoid division by zero
    if π0_A > 1e-10
        w = π_A / π0_A
    else
        w = 0.0  # If behaviour policy assigns zero probability, weight is zero
    end
    push!(weights, w)
end

# Truncate extreme weights
weights = min.(weights, 10.0)  # Cap at 10

# Estimate policy value: E[Y^π] = E[w * Y]
policy_value = sum(weights .* Y) / sum(weights)

# Compare to naive estimate (ignoring policy mismatch)
naive_value = mean(Y)

println("Off-policy evaluation:")
println("  Behavior policy π₀: Treat if L > 0.5")
println("  Target policy π: Treat if L > 0.3")
println("  Naive estimate (ignores policy): ", round(naive_value, digits=3))
println("  IS-weighted estimate: ", round(policy_value, digits=3))
Off-policy evaluation:
  Behavior policy π₀: Treat if L > 0.5
  Target policy π: Treat if L > 0.3
  Naive estimate (ignores policy): 0.109
  IS-weighted estimate: 0.158
Figure 27.2: Off-policy evaluation: evaluating a new policy using data from a different policy

27.6.5 Advantages of Causal Approach

CDMs enable:

  • Interventional simulation: Simulate under any policy
  • No positivity issues: Don’t need policy overlap (can simulate policies not observed)
  • Uncertainty quantification: Full uncertainty propagation
  • Mechanistic understanding: Understand why policies work, not just that they work

27.7 Continuous modified treatment policies (LMTP)

When the exposure is continuous, a scientifically useful intervention is often a modified treatment policy (a shift of the natural value \(A\) (e.g. add one standardised unit, subject to quantile clamps)) rather than \(do(A=a^*)\) for an arbitrary \(a^*\) (Díaz Muñoz and Laan 2012; Díaz et al. 2023). Estimation of δ-indexed total-effect curves (cross-fitted Super Learner nuisances, optional TMLE fluctuation) is illustrated with CausalTargeted; identification certificates come from CausalDynamics (Section 24.4). Mediated contrasts (NDE/NIE and related) live in CausalMediation and are worked through in TMLE and Doubly Robust Estimation, not in this chapter’s LMTP grid. Multi-time sequential LMTP from a CDM panel (simulate_panel → execute_estimand) is illustrated in CDMs unified.

Baseline adjustment need not be pre-dummied. CausalTargeted fits an internal covariate schema (StatsModels DummyCoding) on the cleaned analysis frame and reuses it across folds, so String, Bool, and categorical arrays may sit beside numeric confounders. Treatments for continuous MTP remain numeric. Finite-support exposures use DiscreteTreatmentPolicy / run_discrete_lmtp at a single time (Díaz–Williams classification density ratios with dummy-coded \(A\)), matching R lmtp for factor shifts (Díaz et al. 2023; Williams and Díaz 2023). Wide panels with from-onset grid arms (R / SS / SC style) use TemporalNodeSpec / TemporalDAGSpec → plan_targeted_estimation → run_estimation_plan at the same represented time (panel path snippet; CausalDynamics #20, #29). The grid assignment declares single-node support, an attribute representation, and its referent; the faecal outcome declares pointwise state support for the mouse. Multi-time factor recodes reuse the same policy type on SequentialPolicy.policies (CDMs unified); mixed continuous/discrete \(A_t\) is rejected. Point-treatment categorical ATE remains TMLE.jl. CausalMediation interventional recodes of a factor \(A\) (continuous \(M\)) use MediationSpec with both arms discrete (Ch. 20).

NotePanel path: identification → wide columns → discrete LMTP

For static categorical treatment on a wide observational panel (unit id, time-varying columns fec1, fec2, …), CausalDynamics unrolls a temporal DAG, identifies a same-time arm contrast, and returns an EstimationPlan. CausalTargeted executes it with run_estimation_plan (cross-fitted discrete LMTP nuisances). The grid assignment is already present at baseline, so declare temporal_support = FromOnsetSupport(0), value_representation = :attribute, and a stable referent_id; declare fec with pointwise support and value_representation = :state. Ontology is optional here: ordinary estimation needs referent, representation, and support, not an occasion/enduring claim on every node. The planner then maps the from-onset arm to its bare column and pointwise outcomes to fec1, fec2, … without a unit_level override. The pattern mirrors field designs such as the Apodemus superspreader sentinel grid (R / SS / SC arms); a private-data stress notebook lives in CausalTargeted apodemus_panel_stress.qmd.

using CausalDynamics
using CausalDynamics:
    TemporalDAGSpec, TemporalNodeSpec, LaggedEdge, unroll_temporal_dag,
    TemporalEffectQuery, plan_targeted_estimation, panel_column_name
using CausalTargeted
using DataFrames
using Random
using StableRNGs

T = 4
rng = StableRNG(20260828)
n = 120
arms = rand(rng, ["R", "SS", "SC"], n)
wide = DataFrame(mouse_id = string.(1:n), grid_type = arms)
for t in 1:T
    col = panel_column_name(:fec, t)
    effect = 0.25 .* (arms .== "SS") .- 0.08 .* (arms .== "SC")
    wide[!, col] = effect .+ 0.05 .* randn(rng, n)
end

spec = TemporalDAGSpec(
    nodes = [
        TemporalNodeSpec(
            :grid_type;
            temporal_support = FromOnsetSupport(0),
            value_representation = :attribute,
            referent_id = :grid_assignment,
            causal_role = :assigned,
        ),
        TemporalNodeSpec(
            :fec;
            temporal_support = PointwiseSupport(),
            value_representation = :state,
            referent_id = :mouse,
        ),
    ],
    edges = [LaggedEdge(:grid_type, :fec, 0)],
    graph_kind = TimeUnrolledGraph(),
)
u = unroll_temporal_dag(spec, T)
query = TemporalEffectQuery(:grid_type, :fec, 2, 2)
plan = plan_targeted_estimation(u, query, propertynames(wide); data = wide)
@assert plan.engine === :discrete_lmtp
@assert plan.outcome === panel_column_name(:fec, 2)

res = run_estimation_plan(
    wide, plan;
    arm_hi = "SS",
    arm_ref = "R",
    levels = ["R", "SS", "SC"],
    folds = 3,
    learners_outcome = (:glm, :mean),
    rng = StableRNG(1),
)
res.estimate, res.se, res.positivity.ok

Hurdle, binary, and count outcomes route through NodeOutcomeSpec on the planner (:two_part_discrete_lmtp, binomial, negative-binomial nuisances). See CausalDynamics panel bridge docs and CausalTargeted run_estimation_plan.

NoteFrom CausalTargeted.jl

The scientific object in continuous MTP estimation is the shift policy, not a point \(do(A=a^*)\). Additive z-shifts are the default; multiplicative and threshold policies share the same ShiftPolicy type.

# packages/CausalTargeted.jl/src/shift_policies.jl (excerpt)
additive_shift_policy(; scale = "z", lower_q = 0.01, upper_q = 0.99) =
    ShiftPolicy(scale, lower_q, upper_q)
# Realised map (additive): A ↦ clamp(A + δ · s, L, U)

multiplicative_shift_policy(; lower_q = 0.01, upper_q = 0.99) =
    ShiftPolicy("multiplicative", lower_q, upper_q)
# A ↦ clamp(A * (1 + δ), L, U)

threshold_shift_policy(; lower_q = 0.01, upper_q = 0.99) =
    ShiftPolicy("threshold", lower_q, upper_q)
# Raise low exposures toward the median scale (clamped)

run_lmtp_grid estimates a δ-indexed TE curve under such a policy (with cross-fitted nuisances). Baseline columns may be numeric, String, Bool, or categorical; CausalTargeted encodes them with a fold-stable CovariateSchema (StatsModels DummyCoding). Continuous treatments stay numeric. Finite recodes use DiscreteTreatmentPolicy / run_discrete_lmtp (T=1) or SequentialPolicy.policies (multi-time). Identification of the adjustment set remains upstream in CausalDynamics. Visualise the returned grid with plot_mtp_curve(grid) after loading CairoMakie.

project_root = let
    current = pwd()
    while !isfile(joinpath(current, "Project.toml")) && !isfile(joinpath(current, "_quarto.yml"))
        parent = dirname(current)
        parent == current && break
        current = parent
    end
    current
end
include(joinpath(project_root, "scripts", "ensure_packages.jl"))

@auto_using CausalDynamics CausalTargeted DataFrames Random StableRNGs
using CausalDynamics:
    TemporalDAGSpec, TemporalNodeSpec, LaggedEdge, unroll_temporal_dag,
    TemporalEffectQuery, plan_targeted_estimation, panel_column_name

T = 4
rng = StableRNG(20260828)
n = 120
arms = rand(rng, ["R", "SS", "SC"], n)
wide = DataFrame(mouse_id = string.(1:n), grid_type = arms)
for t in 1:T
    col = panel_column_name(:fec, t)
    effect = 0.25 .* (arms .== "SS") .- 0.08 .* (arms .== "SC")
    wide[!, col] = effect .+ 0.05 .* randn(rng, n)
end

spec = TemporalDAGSpec(
    nodes = [
        TemporalNodeSpec(
            :grid_type;
            temporal_support = FromOnsetSupport(0),
            value_representation = :attribute,
            referent_id = :grid_assignment,
            causal_role = :assigned,
        ),
        TemporalNodeSpec(
            :fec;
            temporal_support = PointwiseSupport(),
            value_representation = :state,
            referent_id = :mouse,
        ),
    ],
    edges = [LaggedEdge(:grid_type, :fec, 0)],
    graph_kind = TimeUnrolledGraph(),
)
u = unroll_temporal_dag(spec, T)
query = TemporalEffectQuery(:grid_type, :fec, 2, 2)
plan = plan_targeted_estimation(u, query, propertynames(wide); data = wide)
@assert plan.engine === :discrete_lmtp

res = run_estimation_plan(
    wide, plan;
    arm_hi = "SS",
    arm_ref = "R",
    levels = ["R", "SS", "SC"],
    folds = 3,
    learners_outcome = (:glm, :mean),
    rng = StableRNG(1),
)
(res.estimate, res.se, res.positivity.ok)
(0.24749841057216196, 0.018635470340947216, true)
TipSmall-n checklist (CausalTargeted)

For continuous MTP and mediation with tens to low hundreds of units (Díaz et al. 2023; Laan and Rose 2011):

  1. Start from recommend_run_options(n; engine, n_mediators) (lean Super Learner when \(n < 80\); parallel=false by default).
  2. Pass mixed-type baselines as-is (String / Bool / categorical are encoded with a fold-stable CovariateSchema in LMTP, sequential, survival, and mediation grids).
  3. Inspect positivity / support (positivity_report or grid positivity=true) (Petersen et al. 2012).
  4. For mediation, sweep nested MC (mediation_n_mc_sweep) until signs and SEs stabilise (Liu et al. 2024).
  5. Report tipping-point / partial-\(R^2\) sensitivity (sensitivity_report) (Cinelli and Hazlett 2020).
  6. Treat discovery graphs as sensitivity only (merge_discovery_sensitivity!), never as silent DAG replacement.

Package prose and DOIs: CausalTargeted methods · references. Worked δ-grid: Chapter 23 (Policy Evaluation).

# Find project root and include ensure_packages.jl
project_root = let
    current = pwd()
    while !isfile(joinpath(current, "Project.toml")) && !isfile(joinpath(current, "_quarto.yml"))
        parent = dirname(current)
        parent == current && break
        current = parent
    end
    current
end
include(joinpath(project_root, "scripts", "ensure_packages.jl"))

@auto_using CausalTargeted DataFrames StableRNGs

rng = StableRNG(23)
df, truth_mix = simulate_mixed_baseline_mtp(120; rng = rng)
baseline = truth_mix.baseline
opts = recommend_run_options(nrow(df); engine = :lmtp)
println("Recommended learners (lean default): ", opts.learners_outcome)
println("Package DEFAULT_SL_LEARNERS: ", DEFAULT_SL_LEARNERS)
println("Baseline eltypes: ",
    (; W = eltype(df.W), site = eltype(df.site), vaccinated = eltype(df.vaccinated),
       breed = eltype(df.breed)))

lmtp_grid = run_lmtp_grid(
    df, :A, :Y;
    baseline = baseline,
    deltas = [-1.0, -0.5, 0.0, 0.5, 1.0],
    folds = opts.folds,
    learners_outcome = opts.learners_outcome,
    parallel = false,
    positivity = true,
)
pos = positivity_report(df, :A; deltas = [-1.0, 0.0, 1.0])
sens = sensitivity_report(lmtp_grid.est[end], lmtp_grid.se[end]; n = nrow(df))
println("LMTP TE at δ = ", lmtp_grid.delta[end], ": ",
    round(lmtp_grid.est[end]; digits = 3), " (SE ",
    round(lmtp_grid.se[end]; digits = 3), ")")
println("Positivity rows: ", nrow(pos), "; sensitivity kinds: ", unique(sens.kind))
Recommended learners (lean default): (:glm, :mean)
Package DEFAULT_SL_LEARNERS: (:glm, :mean)
Baseline eltypes: (W = Float64, site = String, vaccinated = Bool, breed = String)
LMTP TE at δ = 1.0: 0.473 (SE 0.04)
Positivity rows: 3; sensitivity kinds: ["tipping_point", "partial_r2"]
NoteLean vs rich SuperLearner libraries

Production grids default to DEFAULT_SL_LEARNERS = (:glm, :mean) so Quarto renders stay fast and dependencies stay lean. For recovery / larger \(n\), pass RICH_SL_LEARNERS (GLM expansions, :glmnet* via MLJLinearModels, :randomforest via MLJDecisionTreeInterface, :evotree* when EvoTrees is loaded). Activate those candidates with the matching using lines before the grid; :xgboost remains an explicit opt-in (not in the rich tuple) because EvoTrees already covers boosting. Mixed-type baselines still encode to a fold-stable Float64 matrix before any of those learners run.

Parallel LMTP TE at δ = 0.5: 0.269
RICH_SL_LEARNERS (opt-in): (:glm, :glm_interact, :glm_quad, :glmnet, :glmnet_lasso, :glmnet_ridge, :randomforest, :evotree, :evotree_deep, :mean)
Figure 27.3: Continuous MTP total-effect curve under additive z-shifts (run_lmtp_grid → plot_mtp_curve). Ribbons use pointwise Wald intervals from the grid. (Díaz et al. 2023)

Figure 27.3 shows a small synthetic MTP curve with positivity and Cinelli–Hazlett-style sensitivity helpers attached (Cinelli and Hazlett 2020; Petersen et al. 2012). For multi-time exposures with a continuous terminal outcome, use SequentialPolicy / run_sequential_lmtp with a CausalDynamics TemporalEffectQuery certificate (CDMs unified). estimand_from_query keeps a numeric LongitudinalPolicy unless you pass factor policies and wide treatment columns.

A recode on a three-level string \(A\) is a one-contrast MTP, not a δ-grid:

@auto_using CausalTargeted DataFrames StableRNGs

df_cat, truth_cat = simulate_categorical_treatment_mtp(180; rng = StableRNG(51))
res_cat = run_discrete_lmtp(
    df_cat, :A, :Y;
    policy = discrete_recode_policy(truth_cat.recode),
    baseline = [:W],
    folds = 2,
    learners_outcome = (:glm, :mean),
    rng = StableRNG(52),
)
println("Categorical MTP TE (recode 2→1): ",
    round(res_cat.estimate; digits = 3), " (SE ", round(res_cat.se; digits = 3),
    "; oracle ", round(truth_cat.te; digits = 3), ")")
Categorical MTP TE (recode 2→1): 0.416 (SE 0.166; oracle 0.458)

27.7.1 Discrete-time survival under MTP

When the scientific target is remaining event-free through a horizon (infection clearance, apparent survival to last sighting, device failure), the outcome is a monotone indicator \(S_t\) rather than a continuous \(Y\). CausalTargeted’s thin survival path estimates \(E[S_h^{d}]\) under a common MTP shift \(d\) on time-ordered treatments, with at-risk sequential regression and optional IPCW for censoring (Díaz et al. 2024). Competing risks stay out of scope here.

@auto_using CausalTargeted DataFrames StableRNGs

rng = StableRNG(26)
df_surv, truth_surv = simulate_discrete_survival_mtp(
    160; T = 3, β_a = -0.55, rng = rng,
)
df_surv.site = [rand(rng, ("A", "B", "C")) for _ in 1:nrow(df_surv)]
df_surv.vaccinated = rand(rng, Bool, nrow(df_surv))
shift_raw = additive_shift_policy(; scale = "raw", lower_q = 0.0, upper_q = 1.0)
est_surv = SurvivalPolicy(
    truth_surv.treatments, truth_surv.surv, [:W, :site, :vaccinated];
    shift = shift_raw, horizon = 3,
)
# execute_estimand uses one δ per call (last of `deltas`); contrast two shifts explicitly
res0 = run_survival_lmtp(
    df_surv, est_surv;
    delta = 0.0, folds = 2, learners = SMALL_N_SL_LEARNERS, rng = StableRNG(27),
)
res1 = run_survival_lmtp(
    df_surv, est_surv;
    delta = 0.75, folds = 2, learners = SMALL_N_SL_LEARNERS, rng = StableRNG(28),
)
ψ0 = truth_surv.survival(0.0)
ψ1 = truth_surv.survival(0.75)
println("Survival LMTP E[S₃] at δ=0: ",
    round(res0.estimate; digits = 3), " (oracle ", round(ψ0; digits = 3), ")")
println("Survival LMTP E[S₃] at δ=0.75: ",
    round(res1.estimate; digits = 3), " (oracle ", round(ψ1; digits = 3), ")")
Survival LMTP E[S₃] at δ=0: 0.364 (oracle 0.426)
Survival LMTP E[S₃] at δ=0.75: 0.472 (oracle 0.549)

With \(\beta_a < 0\) in this synthetic DGP, raising every \(A_t\) lowers the discrete-time hazard, so event-free probability at the horizon rises. Pair identification with TemporalEffectQuery (outcome = event-free node at the horizon) and survival_identification_certificate when a lag DAG is available; the full certificate → panel pattern is in CDMs unified.

27.7.2 Count outcomes under LMTP

Many ecological and epidemiological outcomes are nonnegative integer counts: parasite egg burden, lesion number, daily case incidence. A Gaussian nuisance model is mis-specified for these data, because it ignores the support constraint and the mean–variance relationship. CausalTargeted provides count-aware Super Learner candidates and log-link TMLE targeting for the families :poisson, :negbin, and :zeroinflated_nb.

The machinery is the same run_discrete_lmtp_contrast interface used for continuous outcomes; only the family_outcome kwarg changes. When the outcome is overdispersed (variance exceeds the mean), suggest_family_outcome recommends :negbin; otherwise :poisson. The count Super Learner library includes a Poisson GLM (:glm_poisson), a profiled negative-binomial GLM (:glm_nb, grid over dispersion \(\theta\)), a zero-inflated negative-binomial composite (:zeroinflated_nb), and the intercept-only :mean.

@auto_using CausalTargeted DataFrames StableRNGs Distributions

rng = StableRNG(42)
n = 200
arms = rand(rng, ["control", "low", "high"], n)
W = randn(rng, n)

# Overdispersed count DGP: NB2 with arm effect on the mean
μ = @. exp(1.2 + 0.3 * W + 0.6 * (arms == "high") + 0.25 * (arms == "low"))
θ = 2.0
y = [rand(rng, NegativeBinomial(θ, θ / (θ + m))) for m in μ]

df_count = DataFrame(; id = 1:n, arm = arms, W, y)

# suggest_family_outcome detects overdispersion
family = suggest_family_outcome(df_count.y)
println("Suggested family: ", family)

res_count = run_discrete_lmtp_contrast(
    df_count, :arm, :y;
    arm_hi = "high",
    arm_ref = "control",
    levels = ["control", "low", "high"],
    baseline = [:W],
    family_outcome = family,
    learners_outcome = recommend_count_learners(n; family),
    folds = 3,
    rng = StableRNG(43),
)
println("Count LMTP (high vs control): ",
    round(res_count.estimate; digits = 3),
    " (SE ", round(res_count.se; digits = 3), ")")
Suggested family: negbin
Count LMTP (high vs control): 3.264 (SE 0.399)

The TMLE fluctuation step uses a log-link submodel on the count scale, so the plug-in update respects the nonnegative support. For small samples, recommend_count_learners drops the ZINB candidate to avoid convergence issues.

27.7.3 Hurdle models: two-part discrete LMTP

Biological count data are often zero-inflated: many units have zero burden, and those with positive burden follow a right-skewed distribution. A single count model conflates two distinct processes: whether infection occurs at all (presence) and, given infection, how intense it is (intensity). The two-part hurdle estimand separates these.

CausalTargeted’s run_two_part_discrete_lmtp_contrast fits:

  • a binomial presence LMTP (family_outcome = :binomial) on an indicator \(Y_{\mathrm{pres}} = \mathbf{1}(Y > 0)\), and
  • a Gaussian intensity LMTP on \(Y_{\mathrm{int}} = \log Y\) among the subsample where \(Y_{\mathrm{pres}} > 0\).

Each part returns its own doubly robust estimate and standard error. On the structural side, CausalDynamics marks hurdle nodes via NodeOutcomeSpec(hurdle, :y_bin, :y_logpos), and the planner routes to engine :two_part_discrete_lmtp automatically.

@auto_using CausalTargeted DataFrames StableRNGs Distributions

rng = StableRNG(50)
n = 180
arms = rand(rng, ["R", "SS", "SC"], n)
W = randn(rng, n)

# Hurdle DGP: binomial presence + lognormal intensity among positives
p_inf = @. 0.28 + 0.34 * (arms == "SS") + 0.14 * (arms == "SC")
infected = [rand(rng, Bernoulli(p)) for p in p_inf]
burden = [
    inf == 1 ? exp(0.4 + 0.25 * w + (a == "SS" ? 0.55 : a == "SC" ? 0.15 : 0.0) + 0.3 * randn(rng)) : 0.0
    for (inf, w, a) in zip(infected, W, arms)
]

df_hurdle = DataFrame(;
    id = 1:n, arm = arms, W,
    fec_bin = Float64.(burden .> 0),
    fec_logpos = [b > 0 ? log(b) : missing for b in burden],
)

res_hurdle = run_two_part_discrete_lmtp_contrast(
    df_hurdle, :arm;
    presence = :fec_bin,
    intensity = :fec_logpos,
    arm_hi = "SS",
    arm_ref = "R",
    levels = ["R", "SS", "SC"],
    baseline = [:W],
    folds = 3,
    learners_outcome = (:glm, :mean),
    family_presence = :binomial,
    family_intensity = :gaussian,
    rng = StableRNG(51),
)
println("Presence (SS vs R): ",
    round(res_hurdle.presence.estimate; digits = 3),
    " (SE ", round(res_hurdle.presence.se; digits = 3), ")")
println("Intensity | positive (SS vs R): ",
    round(res_hurdle.intensity.estimate; digits = 3),
    " (SE ", round(res_hurdle.intensity.se; digits = 3), ")")
Presence (SS vs R): 0.31 (SE 0.044)
Intensity | positive (SS vs R): 0.555 (SE 0.057)

The presence part answers “does treatment increase the probability of any infection?”, while the intensity part answers “among those infected, does treatment increase burden?”. This decomposition is natural for parasite egg counts, lesion scores, and other zero-inflated ecological measurements.

When the planner receives a NodeOutcomeSpec with OutcomeKind.hurdle, it populates the EstimationPlan with the appropriate presence_col and intensity_col, and run_estimation_plan dispatches to the two-part runner automatically. The planner also handles OutcomeKind.binary (routes to :binomial family on a standard discrete LMTP) and OutcomeKind.count_outcome (routes to :negbin or :poisson via the count_family kwarg).

27.8 Stratum context

Policy evaluation uses observable data, structural intervention assumptions, and a declared policy information set to compare dynamic treatment strategies.

27.9 Key Takeaways

  1. Dynamic treatment strategies: Policies map history to treatment
  2. Policy evaluation: Estimate expected outcomes under policies
  3. Methods: G-computation, IPTW, TMLE, continuous LMTP, factor recodes (run_discrete_lmtp / sequential policies), discrete-time survival LMTP, count LMTP (:negbin / :poisson), and two-part hurdle LMTP (CausalTargeted) all apply to policy evaluation
  4. Comparing policies: Choose best policy for given objective; check positivity before interpreting MTP curves
  5. Mediation: Path-specific and NDE/NIE contrasts use CausalMediation (Ch. 20), not run_lmtp_grid

27.10 Further Reading