1. Turn the model's expectations into a data contract
Start with the object your program expects, before choosing a validator. For tabular input, name the columns, types, and units. For an image, specify dimensions, channels, and orientation handling. For text, define the encoding, required fields, and the policy on empty inputs. Data can be readable without being suitable for the computation.
Separate three decisions: reject, accept as is, or transform according to a documented rule. Converting a string to a number, replacing a missing value, and truncating an input change the content processed. These operations should not happen simply because a tool picks a default type.
The example in this guide is for illustration and is not executed. It concerns objects containing an identifier and three numeric values between −100 and 100. These bounds are made up to illustrate a contract, with no physical unit or connection to a Kernodeck dataset. They must be replaced with the rules of the actual project.
Scroll the table to read all columns.| Level | Rule | Detectable failure |
|---|---|---|
| Schema | Exactly id and values | Missing or unexpected field. |
| Type | id string; values list of numbers | Number presented as text, boolean, or missing value. |
| Shape | Three values per object | Vector too short or too long. |
| Value | Finite numbers in [−100, 100] | NaN, infinity, or value outside the illustrative range. |
| Corpus | Unique identifiers | Two objects share the same identifier. |
2. Check the reading before conversions
Fix the format and its dialect. For a CSV, document the delimiter, encoding, and presence of the header. Python's standard CSV reader normally returns strings; it does not decide that your column is an integer. An identifier like 0012 can lose its meaning if a conversion turns it into 12. So keep identifiers in their intended type.
Check the number of fields and the column names before creating the business objects. A row shifted by an unexpected delimiter must not pass just because some values remain convertible. For binary files or images, also perform the actual read: a correct extension does not guarantee decodable content.
A decoded JSON is not yet a validated contract. The Python module accepts certain non-finite values and duplicate names in an object by default. If your format forbids them, configure that rejection at decode time, then apply your schema rules. Also set an appropriate size limit before loading an entire file into memory.
3. Isolate one error per rule type
Build a small set where each invalid entry violates only one important rule. That way you will know what the check detects. If the only incorrect example combines a bad identifier, a wrong dimension, and an infinite number, its rejection does not prove that all three rules work.
In the table, the notations represent teaching objects that have already been read. Infinity is a non-finite numeric value, not a JSON syntax to adopt. The verdicts are expected by reasoning and are not the output of an executed program. A second object carrying a is tested after the valid object a to verify uniqueness.
Keep these cases alongside your contract as it evolves. If you decide to accept numeric strings, create an explicit conversion step and keep a record of that decision. Do not silently change the checks to make the corpus's first rejection disappear.
Scroll the table to read all columns.| Identifier and values | Expected verdict | Rule exercised |
|---|---|---|
| a · [1, 2, 3] | Accepted | Valid reference. |
| b · ["4", 5, 6] | Rejected: type | A string is not a number in this contract. |
| c · [7, 8] | Rejected: shape | Two values instead of three. |
| d · [0, infinity, 1] | Rejected: value | A non-finite value cannot enter the computation. |
| a · [4, 5, 6], after the first a | Rejected: duplicate | Uniqueness across the corpus. |
4. Keep an explicit validator and actionable messages
The following excerpt shows the checks on an object, after decoding. It stops at the first failing rule and handles neither the entire file nor all of its possible formats. The seen container belongs to the corpus traversal: recreating it on each line would make the duplicate check useless.
A useful message contains the rule, the logical file, and the position of the object. Avoid copying its entire content into it. When collecting multiple errors, limit the details retained while maintaining complete counters. A multi-gigabyte report is no more helpful for locating the first cause.
On a NumPy array, an element-by-element finiteness check can complement the type and shape checks. It does not replace business bounds: a finite number can still be a negative length or a value expressed in the wrong unit.
import math
def validate_object(item, seen):
if type(item) is not dict or set(item) != {"id", "values"}:
raise ValueError("SCHEMA")
identifier = item["id"]
if type(identifier) is not str or not identifier.strip():
raise ValueError("IDENTIFIER")
if identifier in seen:
raise ValueError("DUPLICATE")
values = item["values"]
if type(values) is not list or len(values) != 3:
raise ValueError("SHAPE")
for value in values:
if type(value) not in (int, float):
raise ValueError("TYPE")
if not (-100 <= value <= 100) or not math.isfinite(value):
raise ValueError("VALUE")
seen.add(identifier)
return item5. Move from the sample to the full corpus
A short sample lets you fix the reader and the contract quickly. Choose ordinary cases and boundaries: empty input, maximum size, an unusual character, the first and last partition. Selecting only the first lines may miss an anomaly located in a later file or in a rare category.
Full validation traverses all relevant entries and applies the global rules. For large volumes, process the files incrementally and record their identity. A set of all identifiers in memory works for the small example, but can become too costly; in that case, choose a uniqueness strategy suited to the volume, without giving up the check.
The report must state its scope: the tuning sample, an entire partition, or the entire defined corpus. Keep the number read, accepted, and rejected, along with the rules applied. If files change afterward, the old report does not automatically validate the new input.
6. Decide what to do with rejected data
Stop the work when errors invalidate the meaning of the computation: an essential column missing, incompatible units, or lost identifier matching. If your task allows excluding isolated items, define that policy before launch, keep the rejections, and compute the results on the scope actually accepted.
Setting something aside is not a correction. If you replace missing values or normalize inputs, produce a new identifiable version and revalidate it. Keep the transformation and its parameters with the experiment. Otherwise, two runs with the same name may use different data.
Before the GPU, check the batch as it is actually built: dimension order, numeric type, any mask, and matching with the targets. File validation comes before the transformations; it does not prove that the pipeline preserves those properties afterward. A representative case lets you verify this last boundary.
7. Produce an understandable launch authorization
The expected output is a short report that answers four questions: which input, which rules, which scope, and which decision. A valid status must refer to a precise corpus identity. A partial status must name what remains to be checked. A rejection must make it possible to find the objects concerned without needlessly disclosing their content.
Add a check of the validator itself: a correct case passes, each incorrect case is rejected for the right reason, and the counters reconcile. Then check a small passage in the application. This double check avoids confusing data conformity with model quality or GPU availability.
Conforming inputs can still be biased, mislabeled, or unsuitable for the question under study. This guide covers structural conformity and explicit rules; it certifies neither representativeness nor usage rights. Those decisions complete the file before a long run.