An editor can look excellent on the photographs used to tune it and still surprise you on the next shoot. Hundreds of slider exports from a few RAWs do not provide hundreds of independent tests of that risk.
Chiado’s evaluation groups related scenes, keeps an unopened block for final assessment, and moves inspected failures into the development pool. Synthetic selection probes also test the transitions that a large but poorly sampled chart can miss. This process has already exposed a fit with about 0.04 degrees of training error but 3.25 degrees on the other hue columns, and helped reject a Dehaze/curve candidate that did not transfer.
Split by the experience, not just the filename
Two different files can show almost the same scene under almost the same light. Randomly assigning frames from a burst to training and testing can make a model appear to generalize when it has mostly seen the same conditions twice.
We now group related dates conservatively. A new drive supplied photographs from two September dates and two October dates. The September scenes, inspected after evaluation, included a beach ceremony and warm indoor or shaded outdoor event photographs. We treat those adjacent dates as a related block, rather than claiming six selected frames represent six independent shoots.
Keep one block genuinely unopened
The October block is reserved for final evaluation. Selection used metadata and a deterministic hash ordering, rather than choosing photographs that looked promising. Content hashes checked for exact overlap with the existing calibration pool.
That addresses exact duplicates, not visual similarity. A hash cannot tell us that two different RAWs depict the same subject a second apart. Scene provenance and conservative grouping still matter. Importing files into a catalog is also separate from opening them for model selection: the reserved block remains excluded from calibration inspection and fitting.
A chart can miss the question, too
Synthetic color charts helped us isolate Point Color behavior, but they introduced a different coverage problem. One experiment trained on alternating hue columns. Almost none of the training samples landed where the selection faded out. A model achieved about 0.04 degrees of training error, then missed the other columns by about 3.25 degrees. The split kept samples separate, but it did not give the model enough information to locate the transition.
We found the same problem at a brightness boundary. A fitted curve had an extremely small error, yet neither subset contained a sample in the transition itself. Several different boundaries could explain the observed plateau equally well. We marked the boundary as unidentified instead of treating the low error as success.
Denser charts, including a second source color, supplied the missing transition samples. That helped explain failures on yellow and blue without inventing separate correction coefficients for those colors. It also changed the question we ask of a test: how many examples exercise the behavior we are trying to estimate? A large total patch count cannot answer that on its own. These experiments remain development evidence, and Point Color is still unfinished.
A test set has a limited life
Once we inspect a failure and adjust the model because of it, that example is useful development evidence. It should not keep its old label as an untouched test. We retain it for regression checks, while being explicit that it no longer offers an independent final assessment.
This changes how results should be reported. “171 cases” describes photo-and-setting combinations, not 171 independent photographs. Improvements on reused scenes are encouraging, but they do not establish performance on unfamiliar lighting, another camera or a different kind of shoot.
A practical evaluation needs both: a growing library of known failures that must not return, and a smaller reserve that has not helped choose the answer. The discipline is to say which one produced each result.
What this changes for Chiado
The concrete protection is a distinction between cases that helped choose the implementation and cases reserved to assess it. Known failures remain regression checks; they do not keep counting as fresh evidence. At the time of these experiments the October reserve remained unopened, so this note describes the evaluation safeguards rather than claiming a successful final independent evaluation.