diff --git a/README.md b/README.md index 73775080..82f796fc 100644 --- a/README.md +++ b/README.md @@ -28,9 +28,11 @@ research [[1]](#Literature). It provides a standard interface that allows user t # Documentation -Documentation is available at: +Documentation is available at https://causalml.readthedocs.io/. Good places to start: -https://causalml.readthedocs.io/en/latest/about.html +* [Estimating and Validating Heterogeneous Treatment Effects](https://causalml.readthedocs.io/en/latest/tutorial.html): an end-to-end walkthrough that trains one estimator per family and shows how to decide which one to believe +* [Choosing an Estimator](https://causalml.readthedocs.io/en/latest/choosing_an_estimator.html): a decision path and capability matrix over the estimators +* [FAQ](https://causalml.readthedocs.io/en/latest/faq.html): answers to common installation and usage questions # Installation @@ -54,6 +56,11 @@ Example notebooks are available at: https://causalml.readthedocs.io/en/latest/examples.html +# Benchmark Datasets and Leaderboard + +CausalML ships loaders for the standard causal inference benchmarks (LaLonde, IHDP, Twins) with SHA256-verified downloads, plus the ground-truth metrics (PEHE, ATE error, policy risk) they enable. See the [benchmark datasets](https://causalml.readthedocs.io/en/latest/datasets.html) page for each dataset's provenance and terms, and the [leaderboard notebook](https://causalml.readthedocs.io/en/latest/examples/benchmark_leaderboard.html), which regenerates every published number end to end. + + # Contributing We welcome community contributors to the project. Before you start, please read our [code of conduct](https://github.com/uber/causalml/blob/master/CODE_OF_CONDUCT.md) and check out [contributing guidelines](./CONTRIBUTING.md) first. @@ -72,7 +79,7 @@ This project is licensed under the Apache 2.0 License - see the [LICENSE](https: # References ## Documentation -* [Causal ML API documentation](https://causalml.readthedocs.io/en/latest/about.html) +* [Causal ML API reference](https://causalml.readthedocs.io/en/latest/causalml.html) ## Workshops, Talks, and Publications * (Workshop) [3rd Workshop on Causal Inference and Machine Learning in Practice](https://causal-machine-learning.github.io/kdd2025-workshop/) at KDD 2025 diff --git a/docs/_static/img/intro_confounding_sim.png b/docs/_static/img/intro_confounding_sim.png new file mode 100644 index 00000000..9ed6deb0 Binary files /dev/null and b/docs/_static/img/intro_confounding_sim.png differ diff --git a/docs/_static/img/tutorial_gain.png b/docs/_static/img/tutorial_gain.png new file mode 100644 index 00000000..40b4d6e7 Binary files /dev/null and b/docs/_static/img/tutorial_gain.png differ diff --git a/docs/_static/img/tutorial_overlap.png b/docs/_static/img/tutorial_overlap.png new file mode 100644 index 00000000..a3d608de Binary files /dev/null and b/docs/_static/img/tutorial_overlap.png differ diff --git a/docs/_static/img/tutorial_toc.png b/docs/_static/img/tutorial_toc.png new file mode 100644 index 00000000..0f546c3a Binary files /dev/null and b/docs/_static/img/tutorial_toc.png differ diff --git a/docs/choosing_an_estimator.rst b/docs/choosing_an_estimator.rst new file mode 100644 index 00000000..deb6cc60 --- /dev/null +++ b/docs/choosing_an_estimator.rst @@ -0,0 +1,139 @@ +===================== +Choosing an Estimator +===================== + +CausalML implements many estimators because no single one dominates: they +differ in the outcome and treatment types they accept, the data they are +designed for, and what they report. This page maps a problem to a shortlist. +The mathematics of each method lives in the :doc:`methodology`; the API details +live in the :doc:`API Reference `. + +Start from the data +=================== + +**Was the treatment randomized?** + +* **Yes, and compliance was perfect.** Any estimator below applies, and the + assignment probability is known -- pass it as ``p`` instead of estimating it. + For a binary conversion outcome where the goal is targeting segments or + interpretable rules, start with the uplift trees. For per-unit CATE + estimates with confidence intervals, start with the meta-learners. + +* **Yes, but some units did not comply.** The randomized assignment is an + instrument for the treatment actually received. Use the + :ref:`DRIV learner ` + (``BaseDRIVLearner``) to estimate the effect on compliers, or + :ref:`2SLS ` for a linear model. + +* **No -- the data are observational.** Estimation requires that every + confounder (a variable driving both treatment and outcome) is measured, and + that treated and untreated units overlap (see + :ref:`Checking Overlap `). Prefer the + estimators that model the treatment assignment explicitly: the + :ref:`X-Learner `, + :ref:`R-Learner ` and + :ref:`DR learner ` all accept a + propensity score ``p`` and estimate one internally when it is omitted. + Validate with :doc:`sensitivity analysis ` afterwards. + +* **No, and an important confounder is unmeasured.** With an instrument, use + the IV estimators above. With proxy variables for the hidden confounder, + :ref:`CEVAE ` models it as a latent variable. Without + either, no estimator in this package (or any other) identifies the effect. + +**What do you need out of the model?** + +* **Only the average effect (ATE)** -- :ref:`TMLE + `, + :ref:`IPTW `, + :ref:`matching `, or any meta-learner's + ``estimate_ate()``. +* **Per-unit effects (CATE)** -- meta-learners, causal trees and forests, or + the neural models. +* **Segments and rules you can read** -- uplift trees, with + :ref:`visualization `. +* **Who to treat under constraints** -- estimate CATE first, then use + ``PolicyLearner`` or the + :ref:`value optimization methods `. + +Capability matrix +================= + +.. list-table:: + :header-rows: 1 + :widths: 26 17 15 16 16 10 + + * - Estimator (classes) + - Outcome type + - Treatment + - Observational data + - Uncertainty + - Extra install + * - S/T/X/R meta-learners (``BaseSRegressor`` ... ``BaseRClassifier``) + - continuous (``*Regressor``) or binary (``*Classifier``) + - binary or multiple discrete + - yes; X/R use a propensity score + - ATE CI; bootstrap CATE CI + - -- + * - DR learner (``BaseDRRegressor``/``BaseDRClassifier``) + - continuous or binary + - binary or multiple discrete + - yes, doubly robust + - ATE CI; bootstrap CATE CI + - -- + * - DRIV learner (``BaseDRIVLearner``) + - continuous + - binary, with an instrument + - yes, given an instrument + - ATE CI; bootstrap CATE CI + - -- + * - Uplift trees (``UpliftTreeClassifier``, ``UpliftRandomForestClassifier``) + - binary or multi-class + - binary or multiple discrete + - designed for randomized data + - -- + - -- + * - Causal trees (``CausalTreeRegressor``, ``CausalRandomForestRegressor``) + - continuous + - binary + - yes + - ATE CI (tree); per-prediction variance (forest) + - -- + * - ``DragonNet`` + - continuous or binary + - binary + - yes + - -- + - ``tf`` or ``jax`` + * - ``CEVAE`` + - continuous or binary + - binary + - yes, with proxies for a hidden confounder + - -- + - ``torch`` or ``jax`` + * - 2SLS (``IVRegressor``) + - continuous + - continuous or binary, with an instrument + - yes, given an instrument + - coefficient SE + - -- + * - ``TMLELearner`` (ATE only) + - continuous + - binary + - yes, with a propensity score + - ATE CI + - -- + +How the uncertainty is computed, per estimator, is cataloged in +:doc:`inference`. + +Two rules of thumb +================== + +* **Start simple, then justify complexity.** A T-learner with a linear base + learner is a transparent baseline; adopt a more flexible estimator when + held-out evaluation (:doc:`validation`) shows it ranks units better. +* **Do not choose by in-sample fit.** CATE models cannot be scored against an + observed label. Compare candidates with the validation losses and ranking + metrics on held-out data, as described in + :ref:`Model Selection with Validation Losses `. diff --git a/docs/datasets.rst b/docs/datasets.rst index 55a84bc6..079271e9 100644 --- a/docs/datasets.rst +++ b/docs/datasets.rst @@ -77,7 +77,13 @@ Results on IHDP are reported as a mean and standard error across replications:: scores = [pehe(fetch_ihdp(replication=r).tau, predict(r)) for r in range(100)] -A single replication is not comparable to a published IHDP number. +A single replication is not comparable to a published IHDP number. Neither, +exactly, is a mean over this file: the same release also exists in a +1,000-replication version, and that is what the published tables of the CEVAE +:cite:`louizos2017causal` and DragonNet :cite:`shi2019adapting` papers +aggregate over (with the 672 further split 63/27 into train and validation). +The data-generating process and split geometry match; the replication count +does not. Source: the `clinicalml/cfrnet `_ lineage (MIT), files hosted at ``fredjo.com``. diff --git a/docs/faq.rst b/docs/faq.rst new file mode 100644 index 00000000..c367dd5b --- /dev/null +++ b/docs/faq.rst @@ -0,0 +1,93 @@ +========================== +Frequently Asked Questions +========================== + +Importing CausalML fails with an XGBoost error on macOS +------------------------------------------------------- + +``xgboost`` and ``lightgbm`` need the OpenMP runtime (``libomp``), which macOS +does not ship. Since v0.16, the import error names the fix directly: install it +with ``brew install libomp`` or ``conda install -c conda-forge llvm-openmp``, +then retry. See :ref:`Installation `. + +Downloading a benchmark dataset fails with ``CERTIFICATE_VERIFY_FAILED`` +------------------------------------------------------------------------ + +The dataset loaders (:doc:`datasets`) download over HTTPS with Python's +standard library, which needs a CA certificate bundle. Some Python +installations -- notably the python.org macOS installers -- do not wire one up +by default. Run the ``Install Certificates.command`` that ships with the +python.org installer, or point Python at the ``certifi`` bundle: + +.. code-block:: bash + + export SSL_CERT_FILE=$(python -c "import certifi; print(certifi.where())") + +My causal tree results changed after upgrading +---------------------------------------------- + +``CausalTreeRegressor`` and ``CausalRandomForestRegressor`` estimate leaves +honestly by default since #584: the tree structure is grown on one half of the +sample and leaf values are re-estimated on the other, which changes fitted +trees, leaf values and CATE estimates without any code edit. Pass +``honesty=False`` to reproduce the previous behavior, and see +:ref:`Honest estimation ` for why the new +default is better on most data. + +``UpliftRandomForestClassifier`` got slower after upgrading +----------------------------------------------------------- + +Its default changed from ``n_jobs=-1`` (all cores) to ``n_jobs=None`` (one +worker) in #991, because each concurrent tree fit holds its own working set: +peak memory grew with the machine's core count -- 8.8x the input array at +``n_jobs=-1`` on 10 cores versus 1.4x single-threaded, in the benchmark that +motivated the change. Fitted models are identical either way. Pass +``n_jobs=-1`` explicitly to restore the previous speed if you have the memory. + +My Qini or AUUC score is negative +--------------------------------- + +A negative score means the model's ranking performed *worse* than treating +units in random order on that data -- units it ranked as high-benefit gained +less than average. Before concluding the model is bad, compute the score on +held-out data with ``return_ci=True``: on small samples the confidence +interval is often wide enough that a negative point estimate is +indistinguishable from zero. The :doc:`validation` page gives the full +evaluation workflow, and Step 6 of the :doc:`tutorial` shows a real example. + +My propensity scores pile up near 0 or 1 +---------------------------------------- + +That signals an overlap (positivity) problem: some units essentially always or +never receive treatment given their covariates, so their counterfactual is not +represented in the data. Estimators that weight by inverse propensity become +unstable there. Common responses are trimming the non-overlapping region or +clipping the scores away from the boundaries -- and reconsidering whether the +treatment is really variable for those units. See +:ref:`Checking Overlap `. + +The Twins dataset's outcome looks like earnings, not mortality +-------------------------------------------------------------- + +The Twins benchmark encodes survival as ``9999``, so mortality is +``outcome < 9999`` -- reading the column as a number produces a mean near +8,000 and every downstream statistic is garbage. The loader's docstring and +:doc:`datasets` record this; the loader's tests pin it. + +My IHDP results differ across replications more than expected +------------------------------------------------------------- + +Each IHDP replication draws its own train/test split of the same 747 units, +so rows are **not** aligned across replications -- averaging +predictions row-wise across replications compares different children. Evaluate +each replication independently and aggregate the metric, as the +:doc:`benchmark leaderboard ` does. + +Do I need TensorFlow, PyTorch or JAX? +------------------------------------- + +Only for the neural estimators: ``DragonNet`` needs the ``tf`` or ``jax`` +extra and ``CEVAE`` needs ``torch`` or ``jax``. Everything else -- meta-learners, +trees, IV, evaluation -- runs without any of them. Install via +``pip install causalml[tf]`` etc.; see +:ref:`Installation `. diff --git a/docs/getting_started.rst b/docs/getting_started.rst index 8e02d9b3..8feb8470 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -3,7 +3,9 @@ Getting Started =============== What CausalML is, an introduction to causal machine learning for readers new to -it, how to install the package, and a tour of the API in runnable snippets. +it, how to install the package, a tour of the API in runnable snippets, an +end-to-end walkthrough of estimating and validating heterogeneous treatment +effects on a benchmark, and answers to common questions. .. toctree:: :maxdepth: 2 @@ -12,3 +14,5 @@ it, how to install the package, and a tour of the API in runnable snippets. intro installation quickstart + tutorial + faq diff --git a/docs/inference.rst b/docs/inference.rst new file mode 100644 index 00000000..ba7abd51 --- /dev/null +++ b/docs/inference.rst @@ -0,0 +1,86 @@ +========================== +Uncertainty Quantification +========================== + +Every number CausalML produces is an estimate, and most of them can carry a +confidence interval. This page catalogs, per estimator family and per metric, +what uncertainty measure is available, how it is computed, and how to request +it. It consolidates an API surface that is otherwise spread across the +individual classes. + +Average treatment effects +========================= + +**Meta-learners.** ``estimate_ate()`` returns the triple ``(ate, lb, ub)``. +The default interval is analytic, based on the lower-bound formula (7) of +:cite:`imbens2009recent`; passing ``bootstrap_ci=True`` (with ``n_bootstraps`` +and ``bootstrap_size``) replaces it with a bootstrap interval. The +:ref:`DRIV learner ` +follows the same interface. + +**TMLE.** ``TMLELearner.estimate_ate(..., return_ci=True)`` reports the ATE +with a confidence interval; :ref:`the methodology section +` describes +the estimator. + +**Causal trees.** ``CausalTreeRegressor.estimate_ate()`` also returns +``(ate, lb, ub)``. + +CATE estimates +============== + +**Meta-learners.** ``fit_predict(..., return_ci=True)`` returns per-unit CATE +estimates with lower and upper bounds from a bootstrap over refits: +``n_bootstraps`` controls the number of refits and ``bootstrap_size`` the +resample size, so the cost is roughly ``n_bootstraps`` times the single fit. + +**Causal forests.** ``CausalRandomForestRegressor.calculate_error(X_train, +X_test)`` returns an unbiased sampling variance for each prediction, computed +with the infinitesimal jackknife of `Wager, Hastie and Efron (2014) +`_ as implemented in `forestci +`_. + +**Uplift trees.** ``UpliftTreeClassifier`` and +``UpliftRandomForestClassifier`` do not report uncertainty for their +estimates. + +Evaluation metrics +================== + +The evaluation functions accept ``return_ci=True`` and then return a DataFrame +with the score, its bootstrap standard error and confidence interval bounds +per model column (``n_bootstrap``, ``alpha`` and ``random_state`` control the +bootstrap). Pass ``random_state`` whenever the numbers will be reported: the +bootstrap otherwise draws from the global NumPy state and will not reproduce. + +.. list-table:: + :header-rows: 1 + :widths: 30 40 30 + + * - Function + - Uncertainty reported + - Notes + * - ``auuc_score`` + - bootstrap SE and CI + - no p-value by design; compare models by CI overlap + * - ``qini_score`` + - bootstrap SE, CI and p-value + - p-value tests ranking better than random + * - ``rate_score`` + - half-sample bootstrap SE, CI and p-value + - the heterogeneity test; see :ref:`RATE ` + * - ``dr_score`` / ``plug_in_t_score`` + - bootstrap SE and CI on the loss + - lower loss is better; see + :ref:`Model Selection with Validation Losses ` + +What an interval does and does not cover +======================================== + +These intervals quantify sampling uncertainty under each estimator's +assumptions -- most importantly unconfoundedness. They say nothing about bias +from an unmeasured confounder; that question belongs to +:ref:`sensitivity analysis `. +And a per-unit CATE interval that excludes zero is not yet evidence of +targetable heterogeneity across units -- that claim needs the held-out tests +in :doc:`validation`. diff --git a/docs/intro.rst b/docs/intro.rst index 99efed7e..41541a8f 100644 --- a/docs/intro.rst +++ b/docs/intro.rst @@ -28,6 +28,49 @@ Measuring Causal Effects If an RCT is available and the treatment effects are heterogeneous across covariates, measuring the conditional average treatment effect (CATE) can be of interest. The CATE is an estimate of the treatment effect conditioned on the available covariates. We call these Heterogeneous Treatment Effects (HTEs). +Why a Predictive Model Is Not Enough +------------------------------------ + +Outside an RCT, the units that received treatment differ from those that did +not, and any variable that drives both the treatment and the outcome -- a +*confounder* -- makes the naive comparison wrong. The simulation below makes +this concrete. Data are drawn from the first synthetic mechanism of +:cite:`nie2017quasi` (see :ref:`Validation with Synthetic Data Sets +`), where the same covariates +drive both the probability of treatment and the outcome, and the true average +treatment effect is 0.5. Two hundred datasets are simulated; on each, the +treatment-minus-control difference in means and a T-learner's ATE estimate +(:ref:`T-Learner ` with gradient boosting) are recorded: + +.. code-block:: python + + import numpy as np + from sklearn.ensemble import GradientBoostingRegressor + from causalml.dataset import synthetic_data + from causalml.inference.meta import BaseTRegressor + + naive, tlearn = [], [] + for i in range(200): + np.random.seed(i) + y, X, treatment, tau, b, e = synthetic_data(mode=1, n=2000, p=5, sigma=1.0) + naive.append(y[treatment == 1].mean() - y[treatment == 0].mean()) + learner = BaseTRegressor(learner=GradientBoostingRegressor(random_state=i)) + ate, _, _ = learner.estimate_ate(X=X, treatment=treatment, y=y) + tlearn.append(float(ate)) + +.. image:: ./_static/img/intro_confounding_sim.png + :width: 629 + :alt: Histograms of 200 ATE estimates: the difference in means centers far from the true ATE, the T-learner centers close to it. + +The difference in means centers at 0.90 -- a bias of +0.40, nearly the size of +the true effect itself -- because treated units would have had higher outcomes +even without treatment. The T-learner, which models the outcome separately in +each group and differences the predictions, centers at 0.59. Its remaining bias +comes from regularization in the outcome models; estimators designed to be +robust to it, such as the :ref:`R-Learner ` and +:ref:`Doubly Robust (DR) learner `, +are covered in the :doc:`methodology`. + Example Use Cases ----------------- diff --git a/docs/methodology.rst b/docs/methodology.rst index 3142a158..825046bd 100755 --- a/docs/methodology.rst +++ b/docs/methodology.rst @@ -6,6 +6,9 @@ In this section we dive more deeply into the algorithms implemented in CausalML. We use the Neyman-Rubin potential outcomes framework and assume Y represents the outcome, W represents the treatment assignment, and X_i the observed covariates. +If you are deciding which of these methods to use on a given problem, start +with :doc:`choosing_an_estimator`; this page documents how each method works. + Supported Algorithms -------------------- @@ -38,6 +41,14 @@ CausalML currently supports the following methods: Meta-Learner Algorithms ----------------------- +*Relevant classes:* ``BaseSRegressor``/``BaseSClassifier``, +``BaseTRegressor``/``BaseTClassifier``, ``BaseXRegressor``/``BaseXClassifier``, +``BaseRRegressor``/``BaseRClassifier`` and +``BaseDRRegressor``/``BaseDRClassifier`` in ``causalml.inference.meta``, each +taking any scikit-learn-compatible model as its base learner. Preconfigured +variants (``XGBTRegressor``, ``LRSRegressor``, ...) are listed in the +:doc:`API Reference `. + A meta-algorithm (or meta-learner) is a framework to estimate the Conditional Average Treatment Effect (CATE) using any machine learning estimators (called base learners) :cite:`kunzel2019metalearners`. A meta-algorithm uses either a single base learner while having the treatment indicator as a feature (e.g. S-learner), or multiple base learners separately for each of the treatment and control groups (e.g. T-learner, X-learner and R-learner). @@ -298,6 +309,14 @@ doubly-robust estimate rather than a raw difference in means. Tree-Based Algorithms --------------------- +*Relevant classes:* the uplift trees ``UpliftTreeClassifier`` and +``UpliftRandomForestClassifier``, which split on the divergence criteria below, +and the causal trees ``CausalTreeRegressor`` and +``CausalRandomForestRegressor``, regression trees that split on +treatment-effect heterogeneity and estimate leaves honestly by default (see +:ref:`Honest estimation `). All live in +``causalml.inference.tree``. + Uplift Tree ~~~~~~~~~~~ @@ -487,6 +506,10 @@ that pruning targets. Neural Network Algorithms ------------------------- +*Relevant classes:* ``DragonNet`` in ``causalml.inference.tf`` or +``causalml.inference.jax``, and ``CEVAE`` in ``causalml.inference.torch`` or +``causalml.inference.jax``. + Both methods below are optional backends. Install them with the ``tf``, ``torch`` or ``jax`` extras, e.g. ``pip install causalml[tf]``. @@ -583,6 +606,10 @@ a JAX/``flax.nnx`` backend. Value optimization methods -------------------------- +*Relevant classes:* ``CounterfactualUnitSelector`` and +``CounterfactualValueEstimator`` in ``causalml.optimize``. ``PolicyLearner`` +in the same module learns a treatment-assignment policy directly. + The package supports methods for assigning treatment groups when treatments are costly. To understand the problem, it is helpful to divide the population into four categories according to how a unit's *outcome* responds to being treated: * **Persuadables**. Those who will have a favourable outcome if and only if they are treated. @@ -678,6 +705,11 @@ They use a similar routine to find the bounds for PS and PN. The `get_pns_bounds Selected traditional methods ---------------------------- +*Relevant classes:* ``NearestNeighborMatch`` and ``MatchOptimizer`` in +``causalml.match``, the propensity models in ``causalml.propensity``, +``IVRegressor`` in ``causalml.inference.iv`` for 2SLS, and ``TMLELearner`` in +``causalml.inference.meta``. + The package supports selected traditional causal inference methods. These are usually used to conduct causal inference with observational (non-experimental) data. In these types of studies, the observed difference between the treatment and the control is in general not equal to the difference between "potential outcomes" :math:`\mathbb{E}[Y(1) - Y(0)]`. Thus, the methods below try to deal with this problem in different ways. diff --git a/docs/tutorial.rst b/docs/tutorial.rst new file mode 100644 index 00000000..c4a28b25 --- /dev/null +++ b/docs/tutorial.rst @@ -0,0 +1,476 @@ +========================================================= +Estimating and Validating Heterogeneous Treatment Effects +========================================================= + +This walkthrough trains four CATE estimators from different families -- two +meta-learners, a causal forest, and a neural network -- on the same data and +then does the part that is genuinely hard in causal ML: deciding which of them +to believe. The data is the IHDP benchmark, where the per-unit ground truth is +known -- so the validation methods you would use on real data can themselves +be checked against the right answer. Each step names the User Guide page that +covers it in depth. The dataset downloads on first use and is cached locally +(see :doc:`datasets`). + +Step 1: State the question +========================== + +The Infant Health and Development Program (IHDP) benchmark +:cite:`hill2011bayesian` starts from a real randomized trial of home visits +for premature infants, with a child's cognitive test score as the outcome. Two +modifications made it the standard testbed for heterogeneous-effect +estimation: a nonrandom subset of the treated group was removed, so treatment +assignment is confounded the way observational data is, and the outcomes are +simulated from the real covariates -- so both potential outcomes, and +therefore every unit's true effect, are known. Each of its 100 replications +simulates new outcomes for the same 747 units and draws its own 672/75 +train-test split. + +Step 2: Load the data +===================== + +.. code-block:: python + + import numpy as np + import pandas as pd + from causalml.dataset import fetch_ihdp + + RS = 42 + np.random.seed(RS) + + train = fetch_ihdp(replication=0, split="train") + test = fetch_ihdp(replication=0, split="test") + X_tr = pd.DataFrame(train.data, columns=train.feature_names) + w_tr, y_tr, tau_tr = train.treatment, train.target, train.tau + X_te = pd.DataFrame(test.data, columns=test.feature_names) + w_te, y_te, tau_te = test.treatment, test.target, test.tau + + print(X_tr.shape, w_tr.sum(), X_te.shape, w_te.sum()) + print("true ATE: %.3f | sd of tau: %.3f" % (tau_tr.mean(), tau_tr.std())) + +.. code-block:: text + + (672, 25) 123 (75, 25) 16 + true ATE: 4.012 | sd of tau: 0.866 + +672 training units with 123 treated, 25 covariates, and -- because this is a +benchmark -- the true effect of every unit, held aside for scoring. + +Step 3: State the identification and check overlap +================================================== + +Before any estimation, state what makes the effect identifiable at all: we +assume treatment assignment is *unconfounded* given the 25 observed covariates +-- adjusting for them blocks every path by which assignment and outcome are +jointly determined, so the causal effect can be recovered from observational +data. Every estimator below stands on this assumption. On this benchmark it +holds by construction, because the outcomes were simulated from the observed +covariates alone; on real data it is untestable, which is why Step 8 stresses +it rather than verifying it. + +Identification also needs *overlap*: treated and untreated units throughout +the covariate space (see +:ref:`Checking Overlap `). Estimate each unit's +probability of treatment -- the propensity score -- and compare its +distribution across groups. One practical note: on this data the model's +default cross-validation grid selects a penalty that collapses every score to +the treated share, so the grid is widened explicitly. + +.. code-block:: python + + from causalml.propensity import ElasticNetPropensityModel + + pm = ElasticNetPropensityModel(Cs=np.logspace(0, 3, 8), random_state=RS) + pm.fit(X_tr, w_tr) + p_tr, p_te = pm.predict(X_tr), pm.predict(X_te) + +.. image:: ./_static/img/tutorial_overlap.png + :width: 629 + :alt: Propensity-score distributions with the treatment group shifted right and a mass of control units near zero. + +The two distributions overlap over most of the range -- estimation is possible +-- but they are far from identical: the confounding introduced by removing +part of the treated group is exactly what this picture shows, and a spike of +near-zero-propensity controls marks a region with almost no treated +counterparts. On a randomized experiment this plot is flat. + +Step 4: Train four estimators +============================= + +One estimator from each corner of the library: an +:ref:`X-Learner ` and an +:ref:`R-Learner ` wrapping the same gradient-boosted +base learner, a :ref:`causal random forest ` +with its default honest estimation, and :ref:`DragonNet +`, a neural network built for exactly this kind of +problem. (DragonNet needs an optional extra -- ``pip install causalml[jax]`` +for the implementation used here, or ``causalml[tf]`` for the TensorFlow one; +see :ref:`Installation `.) + +.. code-block:: python + + from xgboost import XGBRegressor + from causalml.inference.meta import BaseXRegressor, BaseRRegressor + from causalml.inference.tree import CausalRandomForestRegressor + from causalml.inference.jax import DragonNet + + learners = { + "X-learner": BaseXRegressor(learner=XGBRegressor(random_state=RS)), + "R-learner": BaseRRegressor(learner=XGBRegressor(random_state=RS)), + "Causal RF": CausalRandomForestRegressor(random_state=RS), + "DragonNet": DragonNet(verbose=False, seed=RS), + } + + def fit_all(learners, X, w, y, p): + for name, m in learners.items(): + if name in ("X-learner", "R-learner"): + m.fit(X=X, treatment=w, y=y, p=p) + else: + m.fit(X=X, treatment=w, y=y) + + def predict_all(learners, X, p): + out = {} + for name, m in learners.items(): + if name == "X-learner": + out[name] = m.predict(X=X, p=p).flatten() + elif name == "DragonNet": + out[name] = np.asarray(m.predict_tau(X)).flatten() + else: + out[name] = m.predict(X).flatten() + return out + + fit_all(learners, X_tr, w_tr, y_tr, p_tr) + + print("ATE (truth: %.3f):" % tau_tr.mean()) + ate_x = learners["X-learner"].estimate_ate(X=X_tr, treatment=w_tr, y=y_tr, + p=p_tr, pretrain=True) + print(" X-learner %.2f (%.2f, %.2f)" % ate_x) + ate_r = learners["R-learner"].estimate_ate(X=X_tr, treatment=w_tr, y=y_tr, + p=p_tr, pretrain=True) + print(" R-learner %.2f (%.2f, %.2f)" % ate_r) + cate_tr = predict_all(learners, X_tr, p_tr) + print(" Causal RF %.2f (point estimate)" % cate_tr["Causal RF"].mean()) + print(" DragonNet %.2f (point estimate)" % cate_tr["DragonNet"].mean()) + +.. code-block:: text + + ATE (truth: 4.012): + X-learner 4.16 (4.07, 4.24) + R-learner 4.14 (4.13, 4.15) + Causal RF 3.99 (point estimate) + DragonNet 3.87 (point estimate) + +All four land near the truth. What each family reports differs -- the +meta-learners return an ATE interval from ``estimate_ate()``, the forest +offers per-prediction variances instead, and DragonNet reports a point +estimate (the full catalog is :doc:`inference`). Note the R-learner: the +narrowest interval on the table, and the only one that excludes the true +value. Precision is not accuracy, and nothing here says which estimator to +trust -- that takes the next two steps. + +Step 5: Evaluate against the ground truth +========================================= + +Predict each unit's effect on the held-out split and score it against the +truth: PEHE (precision in estimating heterogeneous effects -- the root mean +squared error of the per-unit estimates) and :func:`~causalml.metrics.ate_error`: + +.. code-block:: python + + from causalml.metrics import pehe, ate_error + + cate_te = predict_all(learners, X_te, p_te) + for name, c in cate_te.items(): + print(" %-10s PEHE %.3f ate_error %+.3f" + % (name, pehe(tau_te, c, squared=False), ate_error(tau_te, c))) + +.. code-block:: text + + X-learner PEHE 0.883 ate_error +0.164 + R-learner PEHE 2.132 ate_error +0.395 + Causal RF PEHE 0.797 ate_error +0.064 + DragonNet PEHE 0.455 ate_error +0.034 + +With ground truth, evaluation is just measurement: DragonNet -- a network +whose architecture was designed against benchmarks like this one -- is the +most accurate, the forest and the X-learner follow, and the R-learner (with +this base learner and these defaults) is failing. On real data there is no +such measurement, which is the situation the next step simulates. + +Step 6: Validate as if the truth were unknown +============================================= + +Everything in this step uses only what real data provides: covariates, +treatment, outcome, and the models' predictions. The validation losses score +each model's predictions against a proxy for the true effect built by +cross-fitting on the held-out data -- the doubly robust (DR) pseudo-outcome +loss and the plug-in T-learner loss (see +:ref:`Model Selection with Validation Losses `): + +.. code-block:: python + + from causalml.metrics import dr_score, plug_in_t_score, rate_score + + df = pd.DataFrame({"y": y_te, "w": w_te, **cate_te}) + print(dr_score(df, X=X_te, outcome_col="y", treatment_col="w", p=p_te, + learner=XGBRegressor(random_state=RS), + return_ci=True, random_state=RS).round(3)) + print(plug_in_t_score(df, X=X_te, outcome_col="y", treatment_col="w", + learner=XGBRegressor(random_state=RS), + return_ci=True, random_state=RS).round(3)) + +.. code-block:: text + + dr_loss se ci_lower ci_upper + model + X-learner 37.947 22.495 -6.142 82.036 + R-learner 44.465 24.535 -3.622 92.553 + Causal RF 38.482 22.476 -5.570 82.534 + DragonNet 40.687 24.687 -7.700 89.074 + + plug_in_t_loss se ci_lower ci_upper + model + X-learner 1.773 0.272 1.241 2.305 + R-learner 5.400 0.934 3.569 7.231 + Causal RF 1.396 0.244 0.917 1.875 + DragonNet 1.654 0.451 0.769 2.538 + +The plug-in loss, knowing nothing of the truth, reproduces its main verdict: +the R-learner is worst by a wide margin, and the other three sit within each +other's uncertainty. (It cannot resolve the top group's internal order -- it +puts the forest first where the truth puts DragonNet -- but it reliably +catches the failing model. The DR loss agrees on the ordering's tail but is +too noisy at this sample size to separate anything.) + +The ranking metrics tell a sharper cautionary tale: + +.. code-block:: python + + print(rate_score(df, outcome_col="y", treatment_col="w", + return_ci=True, random_state=RS).round(3)) + +.. code-block:: text + + rate se ci_lower ci_upper p_value + model + X-learner -0.166 0.191 -0.541 0.208 0.385 + R-learner 0.612 0.301 0.022 1.201 0.042 + Causal RF -0.143 0.291 -0.713 0.428 0.624 + DragonNet -0.080 0.183 -0.439 0.279 0.662 + +On 75 validation rows, the only nominally significant +:ref:`RATE ` belongs to the *worst* model in the lineup -- +with four models tested at once, one accidental p < 0.05 is exactly what +noise produces. Small validation sets do not merely weaken rank-based +metrics; they can hand a significant-looking verdict to the wrong model. The +next step gives these metrics the sample size they need. + +Step 7: Visualize the ranking with gain and TOC curves +====================================================== + +To see what the ranking metrics measure, pool the replication's 747 units and +re-split them in half, so the validation side has 374 rows instead of 75, and +refit the four learners on the other half: + +.. code-block:: python + + import matplotlib.pyplot as plt + from sklearn.model_selection import train_test_split + from causalml.metrics import auuc_score, plot_gain, plot_toc + + X_all = pd.DataFrame(np.vstack([train.data, test.data]), + columns=train.feature_names) + w_all = np.concatenate([train.treatment, test.treatment]) + y_all = np.concatenate([train.target, test.target]) + + fit_idx, val_idx = train_test_split(np.arange(len(y_all)), test_size=0.5, + random_state=RS, stratify=w_all) + pm_v = ElasticNetPropensityModel(Cs=np.logspace(0, 3, 8), random_state=RS) + pm_v.fit(X_all.iloc[fit_idx], w_all[fit_idx]) + p_fit = pm_v.predict(X_all.iloc[fit_idx]) + p_val = pm_v.predict(X_all.iloc[val_idx]) + + learners_v = { + "X-learner": BaseXRegressor(learner=XGBRegressor(random_state=RS)), + "R-learner": BaseRRegressor(learner=XGBRegressor(random_state=RS)), + "Causal RF": CausalRandomForestRegressor(random_state=RS), + "DragonNet": DragonNet(verbose=False, seed=RS), + } + fit_all(learners_v, X_all.iloc[fit_idx], w_all[fit_idx], + y_all[fit_idx], p_fit) + cate_val = predict_all(learners_v, X_all.iloc[val_idx], p_val) + + df_val = pd.DataFrame({"y": y_all[val_idx], "w": w_all[val_idx], **cate_val}) + print(auuc_score(df_val, outcome_col="y", treatment_col="w", + return_ci=True, random_state=RS).round(3)) + +.. code-block:: text + + auuc se ci_lower ci_upper + model + X-learner 0.527 0.014 0.500 0.554 + R-learner 0.525 0.013 0.499 0.552 + Causal RF 0.499 0.017 0.465 0.532 + DragonNet 0.540 0.013 0.514 0.566 + +The AUUC (area under the cumulative gain curve, normalized so 0.5 is random +targeting) now separates a little: DragonNet holds the largest margin over +random and is the only one whose interval clears 0.5, while the causal +forest's ranking is indistinguishable from random ordering. The gain curves +show how thin these margins are: + +.. code-block:: python + + fig, ax = plt.subplots(figsize=(7, 4.2)) + plot_gain(df_val, outcome_col="y", treatment_col="w", ax=ax) + +.. image:: ./_static/img/tutorial_gain.png + :width: 629 + :alt: Cumulative gain curves with DragonNet slightly above the random diagonal and the causal forest tracking it. + +Every curve stays close to the random diagonal, because most units share a +similar effect: ranking cannot beat random by much when there is little +spread to exploit. Where the advantage sits matters, and that is what the TOC +curve isolates -- TOC(q) is the excess effect among the top-q fraction over +the overall ATE: + +.. code-block:: python + + fig, ax = plt.subplots(figsize=(7, 4.2)) + plot_toc(df_val, outcome_col="y", treatment_col="w", ax=ax) + +.. image:: ./_static/img/tutorial_toc.png + :width: 629 + :alt: TOC curves with the X-learner spiking early, DragonNet strongest through the middle, and the causal forest near zero. + +The X-learner's curve spikes to about 1.5 in the top few percent and decays; +DragonNet's advantage is smaller there but spread across the middle of the +ranking; the forest's curve sits near zero throughout. :ref:`RATE +` with its default ``autoc`` weighting integrates the TOC +with weight :math:`1/q`, so it rewards exactly the early concentration the +X-learner has: + +.. code-block:: python + + print(rate_score(df_val, outcome_col="y", treatment_col="w", + return_ci=True, random_state=RS).round(3)) + +.. code-block:: text + + rate se ci_lower ci_upper p_value + model + X-learner 0.381 0.191 0.006 0.757 0.046 + R-learner -0.012 0.148 -0.301 0.277 0.935 + Causal RF -0.009 0.277 -0.553 0.534 0.974 + DragonNet 0.277 0.267 -0.246 0.800 0.299 + +With five times the validation rows of Step 6, the spurious R-learner signal +is gone and the X-learner's RATE excludes zero (p = 0.046): its top-ranked +units demonstrably benefit more than average. Note the division of labor the +two metrics just displayed -- AUUC favored DragonNet's broad, thin margin; +RATE favored the X-learner's concentrated one. Neither is wrong; they answer +different targeting questions, which is why both exist. + +Step 8: Stress the assumptions +============================== + +Every estimate above leans on unconfoundedness: that the 25 covariates capture +everything driving both treatment and outcome. Sensitivity analysis (see +:ref:`Validation with Sensitivity Analysis `) +perturbs the analysis and re-estimates. Replacing the treatment with random +noise -- the placebo test -- should destroy the effect; adding a random +covariate or halving the sample should not: + +.. code-block:: python + + from causalml.metrics.sensitivity import Sensitivity + + df_s = X_tr.assign(treatment=w_tr, outcome=y_tr, p=p_tr) + sens = Sensitivity(df=df_s, inference_features=list(X_tr.columns), + p_col="p", treatment_col="treatment", + outcome_col="outcome", + learner=BaseXRegressor(learner=XGBRegressor(random_state=RS))) + print(sens.sensitivity_analysis( + methods=["Placebo Treatment", "Random Cause", "Subset Data"], + sample_size=0.5).to_string()) + +.. code-block:: text + + Method ATE New ATE New ATE LB New ATE UB + 0 Placebo Treatment 4.1561 -0.208613 -0.313350 -0.103875 + 1 Random Cause 4.1561 4.013964 3.933172 4.094756 + 2 Subset Data(sample size @0.5) 4.1561 3.939882 3.817171 4.062592 + +The placebo collapses the estimate by 95% -- not exactly to zero, since a +flexible learner finds some structure even in noise, but to the far side of +negligible -- while the other two perturbations barely move it. The analysis +is behaving the way a real effect should. + +Step 9: The replication protocol +================================ + +IHDP results are published as a mean and standard error *across* replications +-- a single replication is not comparable to a published number. The loop is +the unit of comparison: + +.. code-block:: python + + rows = [] + for rep in range(10): + train = fetch_ihdp(replication=rep, split="train") + test = fetch_ihdp(replication=rep, split="test") + ... # refit the four learners, score PEHE on the test split + print(pd.DataFrame(rows).groupby("model")["pehe"].agg(["mean", "sem"]).round(3)) + +.. code-block:: text + + mean sem + model + Causal RF 5.458 3.343 + DragonNet 0.599 0.054 + R-learner 4.729 1.843 + X-learner 3.524 2.031 + +The loop rewrites the single-replication story. A few replications simulate +heavy-tailed outcomes, and they blow up every estimator except DragonNet, +whose mean stays at 0.6 with a standard error fifty times smaller than the +others' -- while the causal forest, second-best on replication 0, has the +worst mean of all. A ranking read off one replication does not survive the +protocol, which is the reason the protocol exists. The +:doc:`benchmark leaderboard ` is the canonical +version of this loop, running all 100 replications of the file CausalML ships. + +One caution when comparing against the literature: the CEVAE +:cite:`louizos2017causal` and DragonNet :cite:`shi2019adapting` papers report +IHDP as a mean and standard error over the *1,000-replication* release of this +same data-generating process (and DragonNet reports only the ATE error, not +PEHE), so their table values are not directly comparable to numbers computed +on the 100-replication file. + +Summary +======= + +The workflow, in the order this page ran it: state the identification and +check overlap before estimating; estimate the ATE and treat interval width +with suspicion; measure per-unit accuracy against ground truth where it +exists; validate blind with the DR and plug-in losses, which reliably catch a +failing model; give rank-based metrics enough validation data before trusting +them -- at 75 rows RATE crowned the worst model, at 374 it found the real +signal; read gain and TOC curves together, because AUUC and RATE reward +different shapes of heterogeneity; stress unconfoundedness with sensitivity +analysis; and never report a benchmark number from a single replication. + +On this data: DragonNet was the most accurate and most stable per-unit +estimator, the X-learner had the most concentrated targeting signal, the +causal forest was accurate on average but ranked no better than random, and +the R-learner's confident-looking ATE interval concealed the worst per-unit +estimates in the lineup. Every one of those verdicts required a different +tool, and none of them is visible in Step 4's table of four plausible ATEs. + +Where to next +============= + +* Which estimator fits your problem: :doc:`choosing_an_estimator` +* The mathematics of each method: :doc:`methodology` +* The full evaluation workflow: :doc:`validation` +* What uncertainty each estimator reports: :doc:`inference` +* Interpreting a fitted model: :doc:`interpretation` diff --git a/docs/user_guide.rst b/docs/user_guide.rst index cfd1f353..90fe4e9e 100644 --- a/docs/user_guide.rst +++ b/docs/user_guide.rst @@ -2,16 +2,19 @@ User Guide ========== -The methods CausalML implements and the theory behind them, how to interpret -and validate what a fitted model produces, the benchmark datasets with known -ground truth, and worked examples as runnable notebooks. +How to choose among the methods CausalML implements and the theory behind +them, how to interpret and validate what a fitted model produces and with what +uncertainty, the benchmark datasets with known ground truth, and worked +examples as runnable notebooks. .. toctree:: :maxdepth: 2 + choosing_an_estimator methodology interpretation validation + inference datasets examples references