How PyTorch DDP cut our training run from 55 hours to 5
Our model-training service ran every job on a single GPU while the rest of the cluster sat idle. Here's how we moved it to distributed training on Kubernetes, and what changed when a 55-hour run became a 5-hour one.
pytorchkubernetesmlopsgpu
For a while, training a model on our platform meant starting a job and coming back two and a half days later.
One production run took 55 hours. If something was off (a bad label batch, the wrong augmentation, a learning rate that was too aggressive), you found out on day three and started again. One iteration every couple of days doesn't work when a client is waiting on a model.
Today the same run finishes in about 5 hours. Nothing about the model changed. We stopped training on one GPU at a time.
Where the 55 hours came from
I work on Baseer Builder, Nabeh's no-code computer-vision platform. Users label their data, pick a task (detection, classification or segmentation) and press train. I built the service behind that button.
Each training run is an on-demand Kubernetes Job. That gave us a lot for free: jobs survive pod restarts, they're scheduled wherever there's capacity, and I could add pause, resume and priority on top. Progress goes to the backend over Kafka, and live metrics go to MLflow, so users can watch the loss curve instead of guessing.
The weak spot was that every job ran on one GPU.
Meanwhile our largest deployment, for the Eastern Province Municipality in Saudi Arabia, covers 200+ computer-vision use cases on a cluster sized for 50+ A100-class machines. During a long training run, most of that capacity sat idle while one GPU did all the work.
What DDP actually does
PyTorch's DistributedDataParallel (DDP) is the simplest way to put more GPUs on one training job:
- Every GPU gets a full copy of the model.
- The dataset is split, so each GPU trains on a different slice of every epoch.
- After each backward pass, the GPUs average their gradients (an all-reduce over NCCL), so every copy of the model takes exactly the same step.
With N GPUs, each one processes roughly 1/N of the data per epoch. The model is identical to single-GPU training; you just finish each epoch faster.
The training loop barely changes:
import os, torch, torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data.distributed import DistributedSampler
dist.init_process_group(backend="nccl")
local_rank = int(os.environ["LOCAL_RANK"])
torch.cuda.set_device(local_rank)
model = DDP(build_model().cuda(), device_ids=[local_rank])
sampler = DistributedSampler(train_set, shuffle=True)
loader = DataLoader(train_set, batch_size=batch_size, sampler=sampler, num_workers=8)
for epoch in range(epochs):
sampler.set_epoch(epoch) # reshuffle differently every epoch
train_one_epoch(model, loader)
if dist.get_rank() == 0: # only one worker logs and checkpoints
log_metrics(epoch)
save_checkpoint(model.module, epoch)
That's a simplified version, but it's the shape of the change.
Making it work on Kubernetes
The hard part wasn't the training code. It was running several cooperating pods, possibly on different machines, as one training job.
We used Kubeflow Trainer for this. It launches the workers, gives each one its rank and the address of the master, and treats the whole group as one job. If a node reboots mid-run, the job comes back instead of quietly dying, which mattered because pause and resume were already part of the product.
A few decisions made the rest of the platform keep working:
- Only rank 0 talks to the outside world. It sends MLflow metrics and Kafka progress events. Otherwise users would see every epoch reported once per GPU.
- Checkpoints go to shared storage. They're written to NFS-backed storage, so a paused or restarted job resumes from the last epoch on any node.
- The user doesn't see any of this. They still press "train". The service builds the distributed job behind the scenes.
The result
| Before | After | |
|---|---|---|
| Training time | 55 hours | ~5 hours |
| GPUs per run | 1 | several, across nodes |
| Live metrics | MLflow | MLflow, from rank 0 |
| Survives a node reboot | Job restarts | Resumes via Kubeflow Trainer |
That's roughly 11× faster.
The bigger change was in how people could work. When a run takes 5 hours you can start it in the morning, check it after lunch and try something else before the end of the day.
If you're about to do the same
Things worth checking before you flip the switch:
- Scale the learning rate with the global batch size. Eight GPUs at batch 16 is an effective batch of 128. Keeping the old learning rate can quietly hurt convergence.
- Call
sampler.set_epoch(epoch). Without it, every epoch sees the data in the same order. - Watch the data loader. Once the GPUs are fast, reading and decoding images often becomes the bottleneck. More DataLoader workers and faster storage help.
- Make NCCL use the right network. On multi-NIC machines, set
NCCL_SOCKET_IFNAMEexplicitly, or the all-reduce may pick a slow interface. - Pick one rank for side effects. Logging, checkpoints and progress events from every worker cause duplicates and corrupted files.
Distributed training sounds like research-lab territory, but for us it was mostly plumbing. The GPUs were already there. The job just had to learn to use more than one of them.