Two batches of computational experiments finish overnight. By morning, one produces a more convincing plot: lower prediction error, a narrower spread, or more candidates meeting a chosen criterion.
Before interpreting that improvement, look at the number of results behind it. Did both batches evaluate the same inputs? Did the more promising batch lose a larger fraction of its calculations? Were difficult cases more likely to disappear before a score could be computed?
These questions matter when machine learning experiments depend on computational chemistry or materials calculations. An apparently stronger result can describe a different subset of the intended workload. A useful comparison makes that subset visible and accounts for the runs that never reached the plot.
Forty planned calculations, two different stories
Consider a hypothetical screening exercise with two configurations, A and B. Each is assigned the same 40 candidate inputs. For this example, a result is called qualifying if it meets a criterion defined before the batch begins.
Configuration A produces 36 usable outputs, of which 18 qualify. Configuration B produces 24 usable outputs, of which 15 qualify.
Among usable outputs, the qualifying shares are 50% for A and 62.5% for B. Looking only at those percentages, B appears more promising.
Across the 40 planned candidates, however, A has produced 18 qualifying outputs and B has produced 15. That is 45% and 37.5% of their respective planned workloads. The remaining cases include both usable outputs that did not qualify and calculations with no usable output.
These percentages answer different questions. The first describes the subset that produced results. The second describes the qualifying outputs obtained from the planned batch by the reporting cutoff. Neither percentage tells us what the missing calculations would have shown.
The immediate task is to account for four missing outputs in A and 16 in B. Choosing a preferred configuration before doing that would leave a large part of the comparison unexplained.
Build the denominator before submitting the jobs
A run directory can only tell you about work that left something behind. A separate batch manifest makes the intended workload explicit before execution starts.
For a small experiment, this can be a simple record of candidate identifiers, configuration identifiers, input versions, and planned runs. Give the batch an identifier and retain the submission record. This establishes which candidates were meant to be processed even if a job never starts.
Decide what one row in the final comparison represents. It might be a candidate, a candidate under one configuration, or one independent repetition. That choice determines how retries and repeated calculations should be counted.
For example, three execution attempts for the same candidate do not create three independent candidates. If the research design includes three repetitions with distinct random seeds, those repetitions need their own identities and a defined aggregation rule. Otherwise, the same count can mix recovery attempts with planned experimental variation.
Record the reporting cutoff as well. A calculation still running at noon belongs in a different category from one that has exhausted its allowed execution time. Comparisons made at different stages of completion can change simply because more outputs arrive.
Separate execution outcomes from research outcomes
The word “failed” can conceal several unrelated situations. In the example batch, an input might fail validation before submission, a worker might never start, a calculation might stop without satisfying its convergence criterion, or a parser might reject an otherwise available output.
A practical outcome record distinguishes at least:
- Planned work that has not started.
- Work that started and is still running at the cutoff.
- Execution interrupted by an infrastructure or scheduling problem.
- A calculation that ended without meeting its declared completion criteria.
- An output that exists but could not be validated or parsed.
- A usable output that can be assessed against the research criterion.
Keep the research assessment in a separate field. A usable result that does not meet the screening criterion is still an observed result. It needs to remain in the analysis with its actual value.
Some outcomes require more detail. For a calculation that did not converge, retain the relevant stopping condition and output reference. Avoid assigning a scientific interpretation to that status without examining the method, input, and calculation history. The event record tells a reviewer where to look next.
A retry needs its own identity
Suppose B's missing outputs include eight jobs interrupted by a worker restart. Rerunning those jobs may recover results, but the recovery must remain visible in the experiment history.
Keep the planned run identifier stable and give each attempt a separate identifier. Record when the attempt started, how it ended, and which output belongs to it. If the retry changes a parameter, input, software version, or resource limit, record that change too.
Then define how the analysis chooses an output when several attempts produce one. The rule should be explicit and applied consistently. Selecting whichever attempt happens to yield the most attractive research result would change the meaning of the comparison.
Duplicated event delivery creates another counting problem. A worker might resend a completion record after losing an acknowledgement. Assigning an event identifier lets the analysis recognize that the two records describe one completion. Counting stored rows without checking their meaning can inflate the apparent number of completed experiments.
Query the history behind the result table
Configuration B's 16 missing outputs are the starting point for this check. Take their planned run identifiers and follow them through submission, execution, and output validation. The evidence may be scattered across scheduler messages, calculation outputs, and parser reports, so those identifiers need to survive each stage.
Suppose the records confirm that eight jobs were interrupted by the worker restart, five calculations ended without meeting their completion criteria, and three outputs failed validation. That accounts for all 16, while leaving three different follow-up tasks. Keeping these execution records in a queryable database such as RawTree would let the person reviewing the batch inspect the individual attempts behind each count as recovery work progresses.
Reconcile those outcomes with the original manifest before updating the result table. For each planned run, determine its state at the cutoff and the attempt selected for analysis. If an interrupted job later produces a usable output, preserve the earlier attempt and update the batch assessment according to the stated selection rule. An unexplained gap should remain visible until there is enough evidence to classify it.
The resulting table should let a reviewer move from a missing result to the relevant attempt and its supporting output. Store references to large scientific artifacts where appropriate, with enough version information to retrieve the correct file. A status label becomes much more useful when the underlying evidence is accessible.
Write the result so someone else can challenge it
For the hypothetical batch, a useful report would state that A produced 36 usable outputs from 40 planned candidates, with 18 meeting the criterion. It would give the corresponding figures for B, explain the unresolved outcomes, and identify the shared set used for a direct comparison.
It would also record the cutoff, retry selection rule, configuration versions, and location of the outcome table. Those details let another researcher check the denominator and understand why an input was included or excluded.
The next action then follows from the missing evidence. Interrupted jobs may need another attempt. Unusable outputs may need a parser investigation. Persistent calculation failures may require a closer methodological review. Once those cases are understood, the comparison can say much more precisely what the experiment actually established.