Keep the first failure and its context
This guide begins after a successful launch: PyTorch sees the device, then your application fails on a batch or an operator. If no small computation works, start over from the initial diagnostic. Otherwise, keep the first error, the iteration number, and the last completed step. A succession of messages after the first failure may describe its consequences rather than several independent causes.
Record the code revision, the Python and PyTorch versions, the backend, the numeric type, and the input shapes. For the data, prefer an internal identifier and the dimensions over a full copy of the content. Look for what sets the faulty batch apart: length, missing target, incomplete last batch, augmentation, or a rarely used branch. This record lets you reproduce the case without rerunning an entire campaign.
Locate the faulty launch despite asynchrony
On CUDA, operations are queued and may complete after the Python function returns. An error raised during a copy to CPU or a scalar read may therefore come from a previous computation. The PyTorch documentation explains this asynchronous execution. The line indicated by the trace is an observation point to examine, not always the cause.
For a short reproduction on NVIDIA/CUDA, try a separate launch with CUDA_LAUNCH_BLOCKING=1. This option makes calls synchronous and can bring the error closer to its origin. It is for diagnosis, not for timing. You can also temporarily place synchronizations between major steps to narrow the suspect interval. Then remove this instrumentation: it changes the usual scheduling.
The following command is for illustration and has not been executed. It assumes a POSIX terminal and an existing train.py script. The assignment applies to this launch only; adapt the syntax to your shell. Do not generalize this NVIDIA variable to a ROCm stack.
CUDA_LAUNCH_BLOCKING=1 python train.pyRead an error family without concluding too quickly
The message narrows the search space; it does not replace a reproducible case. An out-of-domain index, a tensor on the wrong device, and an impossible allocation call for different checks. Keep the distinction between invalid data, operator contract, and binary environment. Changing the batch, the precision, and the libraries at the same time eliminates that distinction.
After an assertion runs on the device, do not try to continue the same training in the same process. NVIDIA states that cudaErrorAssert invalidates existing allocations and requires terminating and relaunching the process. In a notebook, this means restarting the kernel before the corrected reproduction. Restarting does not, however, fix a wrong target or an invalid index.
Scroll the table to read all columns.| Observed clue | First check | Conclusion to avoid |
|---|---|---|
| device-side assert | Indices, targets, and operator conditions | The GPU must be faulty |
| Out of memory | Shapes, tensor lifetimes, process memory | Every CUDA error is a lack of VRAM |
| Operator or kernel not available | Versions, extension, backend, and dtype | Reinstalling everything at random |
| Different devices | Placement of the model and each input | Adding a copy without understanding its origin |
Worked example: a class 4 in a four-class problem
Let's take an educational classifier whose output has four columns. Its classes are indexed from 0 to 3. An annotation file containing the value 4 may reveal a 1-to-4 encoding or an unexpected fifth class. Simply increasing the output size would remove a constraint without resolving the meaning of the annotations.
The check proposed below applies before the transfer of CPU targets. It has not been executed. It illustrates the CrossEntropyLoss contract for long-type class indices, with ignore_index=-100 explicitly chosen. It does not cover targets consisting of probability distributions. In this scenario, [0, 2, 4] should be rejected; this expected result is deduced from the rule, not presented as a measurement.
Then fix the mapping in the data preparation and check its one-to-one correspondence with the class names. Do not subtract 1 everywhere until you know whether all sources use the same convention. Add the faulty case to a small validation set kept with the project.
import torch
classes = 4
ignore_index = -100
target = torch.tensor([0, 2, 4], dtype=torch.long)
if target.ndim != 1 or target.dtype != torch.long:
raise ValueError("Targets: expected vector of indices")
valid = target[target != ignore_index]
if valid.numel() == 0:
raise ValueError("No usable target in this batch")
if bool(((valid < 0) | (valid >= classes)).any()):
raise ValueError("Class index out of range")Reduce the program without erasing the trigger
First replay a single input or a single batch with the same transformations. Remove remote tracking, result writing and branches unrelated to the failure. Keep the suspect dtype, shapes and operator. If the error depends on a particular length or memory layout, an arbitrary small tensor may no longer reproduce it.
Compare one change at a time: optional extension disabled, reference operator, usual precision or the same operation on CPU when it exists there. A CPU success is a clue, not a CUDA validation. For a custom function, also record the assumptions about strides, contiguity and sizes. Look for an example that fails before the fix and succeeds after, with an output check rather than just the absence of an exception.
Verify the fix on the initial scope
An acceptable fix must pass the minimal case, neighboring cases and a representative portion of the original run. In particular, retake the last batch, a short input, a long input and the boundary values of the mapping. Check that rejected elements are identifiable and that the number of processed inputs remains the expected one. Silently ignoring exceptions can turn a visible crash into an incomplete result.
Remove the diagnostic mode, start from a fresh process and confirm the behavior with the normal configuration. Keep the cause, the change applied and the non-regression check. If you had interrupted a training run, restart from a consistent checkpoint validated before the error; the presence of a file written during a crash is not enough to guarantee its resumption.
Know when to ask for a more targeted analysis
If the same minimal case fails with valid inputs, prepare a precise request: operation, shapes, dtypes, backend, versions and the first relevant message. Remove personal identifiers and unnecessary paths. A binary extension may require its own compatibility matrix; general PyTorch support does not automatically validate that extension.
On ROCm, PyTorch keeps the torch.cuda interface and the cuda device names. Check torch.version.hip to identify this stack before applying an NVIDIA procedure. Messages, tools and diagnostic options may differ. None of the checks described here proves the compatibility of prepared environments with a Kernodeck offering; use these criteria to clarify your needs before choosing the GPU.