Choose a question before opening a trace
A trace answers a precise question better: is the computation waiting for data, is a copy repeated, or is a small operator called too often? Define the input, the output, and the boundaries of your step. For training, specify whether it includes backward, optimizer update, and batch read. For inference, separate model loading and request.
Keep a short scenario whose correctness you know. Note shapes, batch, precision, model mode, any compilation, and data source. A simplified input can help isolate a behavior, but it no longer necessarily represents the full workload. Mention this difference in the record.
Set the final unit: milliseconds per step or examples processed per second. Sums of events do not replace elapsed time. Compare windows with the same read and transfer boundaries.
Distinguish CPU time, GPU work, and waiting
The CPU prepares and launches operations; the GPU may execute them later. A Python interval can therefore contain launching, waiting, or both. PyTorch's CUDA documentation states that accurate measurements must account for this asynchronism, notably through synchronization or events suited to the scope.
For a targeted GPU duration measurement, CUDA events may be suitable. For end-to-end latency, wait for the work included in that latency and measure the entire request. Do not mix these two units. A synchronization added between each operation can remove real overlap and transform the program you are trying to understand.
In the profiler, an operator's self times and its times including sub-operations also answer different questions. Look at the timeline before adding up the table rows. Simultaneous or nested activities do not represent disjoint portions of elapsed time.
Reserve a startup phase and an active window
The first pass may include initialization, loading, or compilation. Decide whether your question concerns this startup or an already stabilized phase. Keep both observations when they matter for usage, instead of erasing the initial cost in an average presented as overall.
The schedule function makes it possible to distinguish waiting, collection preparation, and the active window. The profiler warm-up is not proof that your model has reached a steady state. Also check the shapes encountered and the state of the caches. An application with variable inputs may encounter new paths after several iterations.
The instructional schedule proposed below waits one step, prepares one, and captures two. Four steps are enough to explain the mechanism, not to establish a performance distribution. In a real campaign, choose a justified window and repeat the measurement outside the profiler after analysis.
Proposed example: four steps with readable boundaries
The following snippet was not executed. It assumes that model, optimizer, loss_fn, loader and device exist, that the model is on device and that the loader provides at least four batches of x, y pairs. It performs updates: use an experimental state intended for this, not a session whose weights you need to preserve.
The labels separate CPU reading, transfer and training. The iterator is created before the window, which excludes part of its startup from the scope. The file is chosen fresh to avoid overwriting a trace. The example refuses unavailable GPU collection instead of quietly presenting a CPU trace as GPU analysis.
The step() signal advances the schedule after each step. The official recipe describes this link between iterations and collection. On a ROCm stack, the PyTorch device is still named cuda; the availability of accelerator collection nevertheless depends on the build and its tools. Check which activities are actually present in the result.
from pathlib import Path
import torch
from torch.profiler import (
profile, schedule, record_function,
ProfilerActivity, supported_activities,
)
trace = Path("trace-etape.json")
if trace.exists():
raise FileExistsError("Choose a new trace name")
activities = [ProfilerActivity.CPU]
if device.type == "cuda":
if ProfilerActivity.CUDA not in supported_activities():
raise RuntimeError("GPU collection unavailable in this build")
activities.append(ProfilerActivity.CUDA)
iterator = iter(loader)
model.train()
with profile(
activities=activities,
schedule=schedule(wait=1, warmup=1, active=2, repeat=1),
record_shapes=False, profile_memory=False, with_stack=False,
on_trace_ready=lambda p: p.export_chrome_trace(str(trace)),
) as prof:
for _ in range(4):
with record_function("lecture_batch_cpu"):
x, y = next(iterator)
with record_function("transfert"):
x, y = x.to(device), y.to(device)
with record_function("entrainement"):
optimizer.zero_grad(set_to_none=True)
loss = loss_fn(model(x), y)
loss.backward()
optimizer.step()
prof.step()
print(prof.key_averages().table(
sort_by="self_cpu_time_total", row_limit=8,
))Turning an observation into a verifiable hypothesis
Start with the active window and check that the expected labels appear. Then examine the gaps between activities, the copies and the repetitions. A long wait in lecture_batch_cpu points to the input pipeline; it does not directly measure storage. A frequent copy invites you to examine tensor placement, without proving that it is unnecessary.
State a single hypothesis, then propose a controlled change. For example, if a constant is rebuilt and transferred at every step, check whether its lifetime can span several steps without changing the result. If an operator appears dominant, inspect its shapes and call count before looking for a replacement.
The lines below are possible readings, not findings drawn from an executed trace. No duration or speedup is claimed. The useful conclusion is a next experiment that can confirm or refute the proposed cause.
Scroll the table to read all columns.| Possible observation | Hypothesis | Next check |
|---|---|---|
| GPU idle during reading | The input pipeline cannot keep up | Same computation with the batch already prepared |
| Recurring copies of a constant | Unsuitable placement or lifetime | Move once, verify the outputs |
| Many small launches | Fragmented work | Examine grouping and overall cost |
| One long operator | Shape or algorithm is the deciding factor | Compare the same operator and its inputs |
Limit the cost and information of instrumentation
Start with a short collection and few options. Enable shapes, stacks or memory only if a question requires it. The PyTorch API states that this information adds a cost; shape collection can even retain references to tensors. A detailed profile can therefore change the program's durations or memory usage.
A trace can contain operator names, shapes and, depending on the options, code paths. Inspect it before sharing. A region label must describe the step without embedding an email, token, private path or input content. Choose a trace viewer suited to your environment and keep collection under your control.
If the expected GPU events are absent, do not fill their times with zero. State that they were not observed and examine the collection support. An absence of events in the tool is not proof that no computation occurred.
Validate the optimization outside the profiler
Start again from the same relevant state and compare correctness before and after the change. For training, an extra step changes the weights; two captures launched in succession are not necessarily an equivalent comparison. Keep the code, parameters and starting point so you can explain the difference.
Then measure the scenario without detailed collection, with the same warm-up and several passes. Report the scope, the raw values and their dispersion. A local improvement may disappear in the full loop or degrade quality: both must remain part of the decision.
Finally, use the observations to pin down the resources actually needed. A trace from your workstation does not rank Kernodeck offerings and proves neither the host processor nor the network of a rented server. Comparing GPUs requires a comparable workload, conditions and results on the resources concerned.