Write the hypothesis before launching the computation
Describe the behavior to improve: classifying documents in a domain, following a response format, or extracting structured information. Also set what must not degrade. For extraction, this can be format validity and the presence of required fields; for classification, a per-category metric rather than a single overall average.
First evaluate the starting model on a set separate from training. Keep the outputs and the configuration of this evaluation. You will then be able to compare the adaptation against a concrete starting point and spot an improvement limited to certain examples. Reserve the final test data: using it to successively choose all the settings ends up weakening its value as a control.
Prepare the data and its split
Version the examples, the cleaning rules, and the transformations. Look for duplicates between training and evaluation, then inspect a small sample after the exact preprocessing of the program. For text, check the tokenizer, the separators, the truncation, and the positions on which the loss is computed. For images, check the dimensions and the transformations applied to the categories.
Sample preparation: take a few representative examples from each category, display their shape after transformation, and manually verify the expected target. Then do a full pass through the training and evaluation loop. This method looks for data or wiring errors; it does not allow you to conclude on the final quality of the model.
Choose the parameters you train
A full fine-tuning updates all the parameters provided by your model. A method like LoRA keeps the base weights and learns additional low-rank matrices in selected modules. This choice reduces the number of trainable parameters, but does not remove the need to load the base model and process its activations.
Record the targeted modules, the trainable parameters, and any additional layers saved. For LoRA, the rank is part of the configuration to compare; it is not enough to predict quality. Verify from the start that an update actually modifies the expected parameters. On export, the adapter must remain associated with the base model and its exact version.
Size a complete training step
Validate a step that includes loss computation, backpropagation, and optimizer update. A model that fits in memory during loading may exceed available capacity during this step. Measure with a representative input length and microbatch. Precision, optimizer states, and the parameters actually being trained are all part of the estimate.
Gradient accumulation lets you organize an update from several microbatches; document their number and the loss normalization. Activation checkpointing trades some extra computation for fewer retained activations. Verify these options separately before combining them. They change the course of the experiment and must appear in the results record.
Keep CUDA, ROCm, and distributed training in the protocol
Check the compatibility of the model, extensions, and adaptation method with the chosen backend. On NVIDIA, prepare the CUDA stack; on AMD, the ROCm stack. Changing platforms means redoing the launch and quality checks. Keep the versions actually used rather than assuming that an environment with the same name produces the same execution.
With DistributedDataParallel, each process works with a replica of the model and gradients are synchronized. This strategy does not automatically share weights across GPU memories; data distribution must also be configured. If your goal is to fit a larger state, consider a strategy that shards that state and verify its constraints before increasing the number of batches.
Organizing variants and the final decision
Give each experiment an identifier and change only one set of parameters that you can explain. Keep the same evaluation procedure across variants, with the seed, training budget, and data used. Also record interrupted or invalid runs: excluding them without explanation makes the comparison hard to interpret.
Plan the 3-, 7-, or 30-day rental around distinct phases: initial check, experiment, evaluation, resumption, and export. Leave a margin to reload a checkpoint in a new process. At the end, deliver the model or adapter, its configuration, the comparative results, and the observed limitations. You remain autonomous in choosing your workloads; Kernodeck does not inspect their content.