A model evaluation is only useful when its information boundary matches the question being asked. If the training procedure learns from information that should belong to evaluation, an apparently strong result can answer a different and easier question. A leakage review examines that boundary before interpreting the score. This guide proposes a practical review of data preparation, splitting and experiment records. It is independent editorial material, not a report that the research projects in this portfolio contain leakage or were evaluated using this procedure.
The scikit-learn guide to common pitfalls explains that evaluation data must not influence model fitting or learned preprocessing. It recommends splitting data before preprocessing and using pipelines to keep transformations associated with the appropriate training process. The examples below are hypothetical reasoning exercises. They are intended to help reviewers ask precise questions, not to establish a universal splitting strategy for every scientific dataset or application.
State what the model is meant to predict
Begin with the prediction task and the intended use of the result. Write down what information would be available at the moment a prediction is made. This can be more important than the choice of algorithm. A feature that is legitimate in a retrospective description may be unavailable in a prospective prediction. Ask where each input comes from, when it becomes available and whether its construction uses the outcome the model is supposed to predict.
Keep this task statement with the experiment record. If the intended use changes, revisit the information boundary rather than reusing an old score without qualification. For example, an experiment about recognizing examples from a known collection differs from one about generalizing to a new collection. Neither question should be silently substituted for the other. The review should identify the actual claim supported by the evaluation, including which forms of generalization were not examined.
Map the unit of separation
List the entities represented in the dataset and how rows relate to them. Several rows may belong to the same person, object, experiment, document or time series. A row-level split can place related observations on both sides of a boundary even when the files themselves are different. Whether that is appropriate depends on the question. Do not assume that a random split establishes independence simply because every row receives a unique assignment.
Create a small diagram or table for your own review showing the relevant identifiers and relationships. This is a reasoning aid, not a statistical proof. Mark uncertain relationships for investigation. If the dataset lacks the identifiers needed to check a proposed separation, record that limitation before drawing a broad conclusion. A reviewer should be able to see which unit was separated, why that unit was chosen and which kinds of overlap could remain undetected.
Trace transformations back to their fit data
For every transformation, ask whether it learns anything from the supplied data. Scaling, imputation, feature selection and other preparation steps may have fitted parameters or choices. Record which split supplied that information. A transformation that looks like routine housekeeping can still cross the evaluation boundary if it is fitted before the split or uses evaluation outcomes to select features. The review should follow the actual computation, not rely on a reassuring label such as preprocessing.
Inspect the code path used for the reported result rather than a cleaned-up notebook prepared later. Check whether cached intermediate files were created with a different data boundary. A pipeline can make the intended sequence clearer, but its presence does not answer every question about how inputs were assembled. If a feature file was produced upstream using the complete dataset, wrapping later steps in a pipeline does not erase that earlier information flow.
Separate model selection from final evaluation
Record which results influenced choices about features, parameters, model families or stopping. An evaluation set repeatedly consulted during development may become part of the decision process even when it is never passed directly to a fitting function. Describe that use honestly. The relevant question is not just whether a variable named test appears in training code, but whether information from those results helped determine the final procedure being evaluated.
Keep a decision log that links changes with the evidence used to make them. If a final evaluation was inspected and then the procedure was revised, do not continue describing it as untouched evidence for the revised model. Explain the sequence and plan an appropriate further evaluation if the intended claim requires one. The correct design depends on the study, but the record should never conceal how many times an apparently final result informed another decision.
Inspect duplicates and near-duplicates in context
Check for exact duplicate records and consider whether near-duplicates are meaningful for the task. Identical content with different filenames can evade a superficial file-level check. Similarity alone, however, does not automatically prove leakage; related observations may be a deliberate part of the evaluation question. Document how duplicates were defined, what was found and what action was taken. Avoid reporting a clean dataset merely because one narrow duplicate detector returned no matches.
Use examples to make the review understandable. A hypothetical document classifier might contain several versions of the same article. If versions are split across training and evaluation, the score may reflect recognition of shared content rather than the intended generalization to unrelated articles. The reviewer should examine that relationship and its relevance to the claim. This example illustrates an information boundary; it does not imply that every similar pair must be removed from every dataset.
Review time and provenance together
For tasks involving time, distinguish the date an event occurred from the date information became available. A record created later may contain retrospective corrections or summaries unavailable at the intended prediction time. Write down which timestamp the experiment uses and why. If a source was revised after the evaluation period, preserve enough provenance to understand whether the experiment used the historical version or a later version containing additional information.
Data provenance also includes manual curation. Ask whether examples were selected, labelled or filtered after someone inspected model performance. Those steps can affect the interpretation of the result even when they are not represented as code. Keep the rationale and selection process in the experiment notes. A reproducible script is valuable, but it is not a complete account if important choices occurred outside the script and are absent from the record.
Work through a hypothetical experiment
Imagine a research collection with multiple measurements for each physical sample. The stated goal is prediction on previously unseen samples, but the initial split assigns individual measurements randomly. The review should ask whether the same sample appears in both subsets and whether that overlap is compatible with the goal. It should also inspect how summary features were computed. A sample-level summary calculated from all measurements could introduce another information path that the split alone does not address.
A revised experiment might separate samples and recompute learned preparation within the intended training boundary. Its score could differ from the original, but the review should not invent the direction or size of that difference. Record both procedures, explain which question each answers and report only results actually obtained. A leakage audit is not a promise that a corrected model will perform badly or well; it is an effort to make the evaluation claim match the experiment.
Leave an auditable decision record
Finish with a concise account of the task, split unit, transformation fit data, selection process, provenance and unresolved issues. Include references to the code and immutable data versions used for the reported run. Distinguish checks actually performed from checks proposed for later work. A checklist with every box marked is less useful than a clear note explaining an important limitation and how it affects the interpretation of the result.
The site's research portfolio provides the original project context, while the blog contains separately authored reading guides. Our editorial policy explains the separation. For an experiment review, keep returning to one concrete question: what information crossed the boundary before the reported prediction was made? Answering it carefully makes the score easier to interpret and the next experiment easier to design.