C3. Guarded Resampling
From advice to enforcement
C1 established that data leakage is a structural timing error.
C2 showed that many machine learning pipelines permit such errors by default.
A direct consequence is that methodological correctness cannot rely on user discipline alone.
Guarded Resampling is an architectural response to this limitation.
What “Guarded” means
In guarded resampling, resampling is not treated as a downstream step in a workflow. Instead, it provides the enclosing structure for all operations that learn parameters from the data.
Under this design, any operation whose results depend on estimated quantities is executed separately within each resampling split, including:
- preprocessing steps that estimate statistics from the data,
- model fitting,
- hyperparameter selection,
- performance metric computation.
Operations that would require access to information outside the analysis set, or that would induce backward information flow from assessment data, are restricted along the default execution path.
This is not a usage convention. It is a structural constraint.
Resampling as the primary object
In many applied machine learning workflows, resampling is effectively treated as an external evaluation step, rather than as a structural component of the learning procedure. A common pattern is:
- prepare data,
- fit a model,
- evaluate performance using resampling.
Although widely used, this organization obscures the fact that valid performance estimation requires strict separation between training and assessment data at all stages of model construction.
Guarded resampling reverses this relationship. Instead of attaching resampling post hoc, the workflow is defined by the resampling scheme itself:
- define a resampling plan,
- for each split:
- estimate preprocessing steps,
- fit the model,
- generate predictions,
- aggregate results across splits.
Under this formulation, resampling defines the experimental structure, and all learning steps operate conditionally within it.
Why enclosure matters
By enclosing learning within resampling splits, guarded resampling ensures that:
- assessment data are isolated from training within each split,
- preprocessing parameters are estimated separately for each training set,
- model comparisons are conducted using identical resampling splits,
- performance estimates correspond to resampling-based out-of-sample evaluation.
These properties do not depend on remembering rules or following best practices.
They arise from how the workflow is constructed.
Guards are not warnings
Guarded resampling does not attempt to identify leakage after it has occurred.
Instead, it restricts the construction of workflows in which leakage-prone configurations would arise along the standard execution path.
This distinction is important:
- warnings are advisory and can be ignored,
- structural constraints limit what can be expressed.
Workflows that violate the principles of guarded resampling are therefore not part of the default design space.
What this enables
Because resampling is treated as an enclosing structure:
- preprocessing is resampling-aware by default,
- resampling splits are applied consistently across models,
- evaluation remains coupled to model fitting,
- reported metrics reflect resampling-based estimates.
These properties hold independently of model class, algorithmic complexity, or user expertise, subject to the assumptions of the resampling procedure.
Methodological correctness becomes a property of system design rather than of individual user choices.
What this does not do
Guarded resampling does not:
- guarantee strong predictive performance,
- identify a statistically optimal model,
- replace careful study design or domain expertise,
- eliminate all possible sources of bias.
It addresses a specific and well-defined failure mode: invalid evaluation arising from data leakage.
Why this is restrictive by design
Enforcing structural guards necessarily reduces flexibility.
Certain actions that are technically possible in more permissive frameworks are intentionally restricted along the default execution path.
This is not a limitation to be worked around.
It is a deliberate design choice.
The consequences of this choice are addressed in the next concept.
Summary
- Guarded resampling treats resampling as the enclosing structure.
- Data-dependent learning occurs within resampling splits.
- Leakage-prone configurations are restricted by design.
- Enforcement replaces advisory guidance.
This is the core idea behind fastml.