Blog
Notes on Kubernetes, GPU infrastructure, MLOps and the production bugs behind them.
Eight rules for an on-prem office network, and a test for each
We host everything ourselves, from the registry to the internal tools. After rebuilding our office network, access and tooling from the ground up, these are the eight rules I'd insist on anywhere, each with a test you can run on your own office this week.
networkingsecurityon-premkubernetesdevops · 4 min readHow PyTorch DDP cut our training run from 55 hours to 5
Our model-training service ran every job on a single GPU while the rest of the cluster sat idle. Here's how we moved it to distributed training on Kubernetes, and what changed when a 55-hour run became a 5-hour one.
pytorchkubernetesmlopsgpu