What you will run
The Kernodeck mini-project includes a small synthetic dataset, a network with dropout, a training loop, and a verifier. The protocol forces the CPU to isolate the save and resume logic. It is not a CUDA, ROCm, or multi-GPU qualification, nor a performance measurement of a rented GPU.
The verifier opens fresh processes for the continuous run, the interruption, the full resume, and a negative case that does not restore the random generators. The point of the latter is to verify that the check can detect an incomplete resume, even when the weights and step number look correct.
Scroll the table to read all columns.| Run | Execution | Question verified |
|---|---|---|
| Continuous | 10 updates from the initial state. | What state is reached without interruption? |
| Interruption | 5 updates, then save and stop. | Does the intermediate point contain the expected states? |
| Full resume | New process, load point 5, then 5 updates. | Do we get the same sequence of inputs, rates, and parameters within the chosen tolerance? |
| Resume without RNG | New process, same resume point but randomness restoration omitted. | Does the test detect drift that a simple weights load would let through? |
Prerequisites and how to launch the protocol
Download the archive, extract it into a working folder, then move into the folder containing train.py and verify_resume.py. Use a Python environment that has PyTorch and NumPy. The archive contains the code and the synthetic data; it downloads no model and requires no Kernodeck account to run the exercise.
The provided proof was run with Python 3.14.6, PyTorch 2.11.0+cu128, and NumPy 2.4.4. The program forces the CPU, float64 precision, and a single PyTorch thread. The package suffix therefore does not mean that the resume used CUDA. On another environment, run your own verification.
Choose an output directory that does not exist yet. Each run produces checkpoint.pt, its checkpoint.pt.sha256 hash, and summary.json. The verifier gathers the comparison into verification.json. The --steps option counts additional steps: after the interruption at 5, the resume command runs 5 more to reach 10. The Python option -B avoids bytecode caches in the exercise folder.
python -B verify_resume.py --output runs/preuve-cpupython -B train.py --steps 10 --output runs/continu
python -B train.py --steps 5 --output runs/coupure
python -B train.py --steps 5 --resume runs/coupure/checkpoint.pt --output runs/repriseWeight exports and resume checkpoints do not play the same role
Start by deciding what you want to recover. An export for inference is meant to produce predictions with a trained model. A training resume must also recover the state that determines the next updates. File-based inference processing, for its part, needs a reliable list of items already completed. These three needs produce different saves.
Do not confuse this persistent save with activation checkpointing. That technique reduces certain activations kept in memory by recomputing them during backpropagation; on its own, it does not create a file that lets you resume after a stop. So specify in your project whether the word checkpoint means a memory optimization or a resume point.
The states to keep together
The model state_dict contains the registered parameters and buffers; the optimizer has its own state. Here, Adam, StepLR, dropout and three random generators influence the next updates. The checkpoint must represent the same instant for all of these elements.
Also document the code version, the experiment parameters and the identity of the data. In the middle of an epoch, knowing only its number is not enough: you need to be able to recover the order of the examples and the next batch to consume. A mistake here can skip entries or process them twice.
The set of 24 rows describes a synthetic relationship between two variables and a target. The network has 33 parameters, with a layer of eight neurons and a dropout of 0.25. The batch contains four rows. After five updates, the cursor is at 20 out of 24: the cut-off falls in the middle of an epoch. The ten updates consume 40 observations, which forces the check to go through a new permutation of the data.
Scroll the table to read all columns.| State | Role | Check to perform |
|---|---|---|
| Model | Keep weights and buffers. | Compare the final parameters and an evaluation output. |
| Optimizer | Keep the states used by the next update. | Verify that it reloads, not just its hyperparameters. |
| Scheduler | Continue the learning rate sequence. | Compare the next rate applied, then the following rates. |
| Python, NumPy and PyTorch RNG | Continue the draws actually used. | Verify that the negative exercise without restoration diverges. |
| Data | Resume the permutation and the cursor. | Compare the entry identifiers after the cut-off. |
| Progress | Interpret the steps and epochs. | Reach 10 updates in total, without redoing or omitting any. |
| Configuration | Reconstruct the same experiment. | Keep dimensions, precision, settings and versions. |
Restore in the right order
Rebuild the model, the optimizer and the scheduler before loading their states. The scheduler must be created before optimizer.load_state_dict(): its construction can otherwise overwrite the restored learning rates. Reload its own state too, then verify the rate actually used at the next step.
Restore the random generators after building the objects that consume draws, right before continuing the work. Simply resetting the initial seed would restart the sequence from the beginning; that is not recovering the state reached after the fifth update. In your project, identify all the generators used, including those of the transforms and of the data loading.
In this exercise, Python applies a slight gain to the inputs, a NumPy PCG64 generator produces noise and the permutations, and PyTorch produces the dropout. The checkpoint keeps their states reached at the cut-off. The verifier also observes their next draws, immediately restoring the state so as not to disturb the rest of the computation.
optimizer = torch.optim.Adam(model.parameters(), lr=0.03)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=3, gamma=0.5)
# In the resume walkthrough, after the objects are built:
state = load_checkpoint(resume)
model.load_state_dict(state["model"])
scheduler.load_state_dict(state["scheduler"])
optimizer.load_state_dict(state["optimizer"])
progress = state["progress"]
history = state["history"]
restore_rng(state["rng"], generator)
model.train()Choose a consistent save boundary
Set an explicit boundary, for example after a complete optimizer update. If you accumulate several microbatches before that update, saving in the middle also forces you to handle the intermediate state. A first implementation is easier to verify when it saves at a boundary where the accumulated gradients have already been consumed.
Keep several backup generations. Write the new file under a distinct name, wait for the write to finish, verify that it is readable, then mark it as usable. Do not replace your only valid checkpoint before that check. The frequency depends on the work you are willing to redo and the write time you observe; it cannot be derived from the rental duration alone.
The mini-project saves after a completed iteration, then exports the file and its fingerprint. It uses a new folder for each run and does not replace a previous proof. If your training uses mixed precision with a GradScaler, its state is also part of the resume. This variant is not covered by the CPU exercise.
Read the comparison and its tolerance
The protocol compares the continuation after step 5: data consumed, learning rate, losses and parameters reached. A match of the step number alone is not enough. A reset optimizer can continue the loop while producing different updates.
The tolerance chosen for this exercise is absolute: 1e-12, with a relative tolerance of 0. This threshold is part of the provided CPU protocol; it is not a universal rule for your models. The comparison must flag non-finite values and structural differences, instead of silently accepting an unusable output.
PyTorch does not guarantee identical results across versions, platforms, CPU and GPU. If you port the exercise, redo the proof on the target and explain the tolerance you chose. Do not widen the threshold simply to make a failure disappear when you have not understood its cause.
In the provided proof, all deviations of the full run are zero: parameters, optimizer state, losses, rates and MSE. The line order, progress, scheduler state and next draws also match. The next learning rate after step 10 is 0.00375 in both runs. The result therefore does not depend solely on a final metric that could mask intermediate differences.
Scroll the table to read all columns.| Comparison | Full resume | Resume without RNG restoration |
|---|---|---|
| Maximum weight deviation | 0 | 0,011669328447718508 |
| Final MSE | 0,09538858591775097 | 0,0936034144665111 |
| MSE deviation from the continuous run | 0 | 0,001785171451239867 |
| Verdict of the consistency sub-test | Consistent within the 1e-12 tolerance | Divergence detected |
Why keep the negative case without restored randomness
A check is more useful when you know which error it detects. The negative variant reloads the same weights, optimizer states, scheduler and progress, but deliberately omits the RNG restoration. The process can finish without a Python exception while pursuing a different trajectory.
In the provided proof, this omission produces a maximum weight deviation greater than 0.011 and an MSE difference greater than 0.0017. The negative MSE here is lower than that of the continuous run: this does not make the resume correct. The goal is to reproduce the same experiment, not to rank two models by their final error.
The verifier succeeds only when the full run is consistent and the negative case diverges. It then displays all_checks_passed: true, positive: true and negative_divergence_detected: true. Its exit code is 0 when the protocol succeeds, 1 if the comparison fails and 2 if the verification could not be completed.
python -B train.py --steps 5 --resume runs/coupure/checkpoint.pt --omit-rng-restore --output runs/reprise-incompleteLoad the exercise file without loosening the safeguards
The project loads only the checkpoint you created with this exercise and kept under your control. It explicitly uses torch.load(..., map_location="cpu", weights_only=True). The Python state contains primitives, the NumPy PCG64 generator state contains integers and strings, and the PyTorch CPU state contains a byte tensor. No arbitrary NumPy array is placed in the saved RNG state.
The loader verifies the associated hash, size, schema, progress, versions, and the identity of the code and data. It rejects an inconsistent state instead of silently resetting a missing element. The hash detects a modification; it does not authenticate the sender of a file.
Do not add weights_only=False simply to silence a loading error. The saved format and its reconstruction must be consistent. Restricted loading reduces the possibilities for deserialization, but it does not make an unknown file trustworthy.
What changes for distributed training
With multiple processes or states split across GPUs, check who writes what. A file produced by a single process is not necessarily a complete backup of the distributed work. Use the backup procedure defined by your strategy and wait for it to complete on the relevant participants. Clearly identify which fragments belong to the same checkpoint.
A change in the number of GPUs may require redistributing states and may change how data is partitioned. Distributed checkpoint mechanisms can handle some changes, but this must be verified for your format and configuration. Do a load test on the intended target. Adding batches to the order does not automatically turn a single-GPU backup into a distributed program.
Finish with an export that can actually be recovered
Before the deadline, export the useful checkpoints along with their configuration, metrics, loading instructions, and data identifiers. Verify the size and a hash of the copied files, then load at least one backup from its destination. An identical hash verifies the copy; reloading verifies that the content is actually sufficient to reconstruct the work.
Only load files whose origin you know, and choose an appropriate format and deserialization options. Keep the last validated checkpoint until the new one has passed your checks. The expected output is a recoverable folder and a short proof of recovery: command executed, step recovered, check passed, and result exported. Plan for this time within your 3, 7, or 30 days.
Scope of the proof and choice of rental
The proof of September 24, 2026 compares four fresh processes on CPU, with an absolute tolerance of 1e-12 and no relative tolerance. It covers neither CUDA, nor ROCm, nor AMP, nor distributed training, nor data loading workers. It validates the resume logic of the provided version, in the described environment, and does not measure the capabilities of a rented GPU.
After this small exercise, transpose the same protocol to your model, your data, and your backend. An 80 GB card or a 192 GB card does not fix an incomplete checkpoint: first choose the compatible chain, then size the memory for a real step. The offers linked below are not presented as hardware tested for this proof.
Within your 3, 7, or 30-day period, plan for an initial backup–stop–resume cycle and the time for the final export. The useful output is a folder whose versions, checkpoint, comparison check, and limitations you can explain; the mere existence of a .pt file does not provide this assurance.