1. Distinguish four states that the screen conflates
An open connection only proves you can still talk to the machine. A visible terminal may host a shell whose compute has already finished. Conversely, losing the connection doesn't let you conclude that the process has stopped. So start by giving the run a name and a place to find its traces.
Track the process's existence, its business progress, and the acceptance of the result separately. The number of lines in a log is not a counter of successful data: a program can repeat the same warning. A partial output can be readable while still being incomplete.
This method assumes you have received and verified the necessary access credentials. It promises neither a particular access protocol nor automatic file retention. The available tools and destinations must be checked in the environment actually provided.
Scroll the table to read all columns.| Observation | What it indicates | What it doesn't prove |
|---|---|---|
| Active connection | The channel responds. | The compute is progressing. |
| Process present | A run still exists. | It's processing the right items. |
| Validated counter increasing | Some scheduled units are finished. | The entire corpus is finished. |
| Exit code equal to zero | The program reports a normal termination. | The result meets your contract. |
| Outputs checked and retrieved | The chosen criteria have been verified. | A quality beyond those criteria. |
2. Set up an identifiable run before detaching it
Check the data, the effective configuration, and a short initial run of the application. The GPU diagnostic answers a different question: can the backend run a compute job? It doesn't validate the full loading of the corpus or your program's logic. Do these checks before launching for a significant duration.
Choose a run identifier and a fresh folder. Keep the command without secrets, the code version, the identity of the inputs, and the expected result. Plan where to write the logs and where to retrieve the files. Two launches must not write to the same folder simultaneously.
For work divided into independent partitions, decide when a partition becomes finished: compute complete, file closed, content checked, and state saved. A file being written must not mean the same thing as an accepted result. Also check the actually usable space before starting.
3. Keep a findable terminal when the context allows
If the environment provides a Unix shell and tmux, this multiplexer lets you detach a terminal and find it again after reconnecting. It protects this workflow against loss of the connection client; it is not a recovery mechanism for a machine reboot or a destroyed process.
The commands below are illustrative and are not executed. They assume Bash, tmux, and your own traitement.py program, with the options shown; this file is not a provided resource. First check its short command and use a distinct session name so you do not confuse two computations.
Create the session, then run the second block inside it. Creating the directory fails if it already exists, which avoids silently reusing its logs. The return code is preserved if the shell reaches the write step; a sudden stop can prevent this file from being created. So a missing code is not an implicit success.
With the default shortcuts, detach with Ctrl-b then d. After reconnecting, list the sessions, then attach to the right one. The shell may remain visible while the program has finished: check the log and the recorded code. Do not immediately start a second copy just because your old terminal disappeared.
tmux new -s campagne-amkdir -p runs
mkdir runs/campagne-a && (
code_retour=0
python -u traitement.py --config config.toml --output runs/campagne-a \
> runs/campagne-a/execution.log 2>&1 || code_retour=$?
printf '%s\n' "$code_retour" > runs/campagne-a/exit-code.txt
exit "$code_retour"
)tmux ls
tmux attach -t campagne-a4. Count accepted work, not just activity
The Python -u option removes buffering from its standard and error output. It helps you see emitted messages, but it does not create progress events in the application. A library or a silent step may still require proper observation.
Define understandable phases: read, prepare, compute, write, verify. Add a counter with a stable unit, as well as a total when it is known. If you are counting accepted partitions, do not switch to a counter of lines read in the middle of the log without changing its name.
The following illustrative example concerns eight partitions of five hundred items, or four thousand items. At the moment shown, only five partitions are accepted. The sixth is partial and must not inflate the total. The numbers show a counting rule; they do not describe any Kernodeck run.
Scroll the table to read all columns.| Partitions | State | Items counted as accepted | Decision |
|---|---|---|---|
| 1 to 5 | Verified | 2 500 | Keep their identities and results. |
| 6 | Partial write | 0 | Do not report this partition as finished. |
| 7 and 8 | To process | 0 | Stay on the work list. |
| Overall | Incomplete | 2,500 of 4,000 | Do not accept the final folder. |
5. Investigate a silence or interruption without creating a duplicate
When no counter moves, identify the last known phase and its last completed unit. Check whether the process exists, whether an error message has appeared, and whether the destination is still usable. A slow model startup and a stuck loop can produce the same motionless screen; the context determines the next check.
Prepare graceful shutdown in your application: a stop request, completion of a safe unit, state saving, then exit. Signals and interruptions depend on the system. Python cannot intercept SIGKILL, and a Python handler may wait for a long native call to finish before it runs. Therefore, a shutdown save does not replace periodic backups.
After a connection loss, first find the existing session and run. After a confirmed stop, determine what is complete and what is partial. For training, resuming requires the detailed model and optimizer states; the dedicated guide verifies this case in a new process.
6. Resume only what the contract permits
In our example, per-partition resumption assumes that each partition is independent and that the input, code, and configuration remain identical. It can keep the five validated results and fully recompute the sixth before continuing with the next two. If these assumptions do not hold, this shortcut is not justified.
Do not simply append the new lines to the partial file. You risk producing duplicates or mixing two configurations. Use the expected identifiers to distinguish complete, incomplete, and absent. Keep the old state as an explanatory element, with a separate destination for the new attempt.
Test your strategy on a controlled interruption before relying on it for a large workload. The criterion is the equivalence of the useful outputs under your contract, not the identity of the progress messages. Training runs, computations with external side effects, and distributed processing require other guarantees than this example of independent partitions.
7. Accept and retrieve the result before closing out the work
Start with the exit code, then reconcile the outputs against the input manifest. For our example, expect the eight partitions and the four thousand planned identifiers, with no gaps or duplicates. Check the format, dimensions, and relevant values; reading a file does not prove that it contains the right result.
Retrieve the accepted outputs, the authorized configuration, the versions, and the verification report. Compare the sizes and, if necessary, the hashes between the source and the copy. An identical hash helps verify the transfer of bytes; it proves neither the quality of the model nor the legitimate provenance of the file.
Finally, open a result from its storage destination, using the tool that will consume it. Plan this step before your access to the compute ends. The work is finished when the verified result can be retrieved and interpreted, not when the last percentage reaches one hundred.