Eight rules for an on-prem office network, and a test for each
We host everything ourselves, from the registry to the internal tools. After rebuilding our office network, access and tooling from the ground up, these are the eight rules I'd insist on anywhere, each with a test you can run on your own office this week.
networkingsecurityon-premkubernetesdevops
Our office runs on its own hardware. The container registry, the build server, the storage and every internal tool sit on machines down the corridor instead of in someone's cloud.
For a long time, this is what that looked like:
- Two internet providers, with each developer tied to one of them. When a provider dropped, the people on it got up, joined the other network and started again.
- Shared workstations reachable on every port from anywhere on the network.
- Internal tools running on those same workstations, next to everything else the workstations were busy doing.
- One Nginx box holding all the routing, edited by hand by whoever needed a change.
None of it was anyone's fault. It's what an office network turns into when it grows one quick fix at a time.
I lead infrastructure here, so I rebuilt it. What came out is eight rules I'd now apply to any on-prem office. Each one has a test. If you can't pass the test today, you've found your next project.
1. No one's internet should depend on one provider
Having two providers is not the same as having redundancy. We had two, and an outage still stopped work, because each person's connection belonged to one of them.
The fix was a firewall router with every provider plugged into it. The office talks to the router, and the router decides which line the traffic leaves on. Addresses inside the office are handed out by the router, so they don't belong to any provider.
An outage used to be an event people reacted to. Now the router handles it and the office carries on.
The test: unplug your main line at 11am on a working day. Count how many people notice.
2. Split the network by who is on it
A flat network treats everything on it as equally trusted: a visitor's phone, a build server and the box holding your backups. Whatever gets onto one of them can see all the others.
We run five separate networks (VLANs) over one trunk:
| Network | Who is on it |
|---|---|
| Developers | Laptops of the people who work here |
| Workstations | The shared compute machines |
| Internal tools | Registry, build server, storage, proxy |
| NAS | Storage devices |
| Guests | Anyone from outside |
Wi-Fi follows the same split, so each access point puts you on exactly one of these networks. Which network you're on comes from what you connected to, not from someone remembering to configure your laptop.
The test: take a laptop you've never seen before and join the guest Wi-Fi. List everything it can reach. The right answer is the internet and nothing else.
3. Between networks, the default answer is no
Splitting the network does nothing if every network can still talk to every other one. The firewall rules between them are where the protection comes from.
The rule that matters is the one at the bottom of the list: anything not explicitly allowed is refused. Then every allowed path is a line someone wrote on purpose.
Write the rules down as a grid before you touch the firewall. For example:
| From \ To | Internal tools | Workstations | NAS | Internet |
|---|---|---|---|---|
| Developers | yes | through the VPN | no | yes |
| Workstations | yes | yes | yes | yes |
| Internal tools | yes | no | yes | yes |
| Guests | no | no | no | yes |
A grid like this fits on one screen, and a new joiner can read it in a minute. If yours doesn't fit on one screen, you have too many exceptions.
The test: pick any two networks. Without opening the firewall, say whether they can talk and why. Then open it and check.
4. Nothing is reachable until you've logged in to the VPN
Our workstations used to accept connections on every port from every address on the network. That's convenient, right up to the day one laptop on that network is compromised.
Now there is one way in. You connect to an SSL VPN, you prove who you are, and only then does SSH to a workstation work at all.
The VPN gives you two things. There is far less to attack, because a machine that doesn't answer can't be probed. And there is one door, so there is one place to see who came in and one place to switch someone off.
The test: from a network that isn't your office, scan your own public address.
nmap -Pn -p 22,80,443,3389,5000,8080 <your-office-public-ip>
Every port that answers is something you're offering to the whole internet. Make sure each one is on purpose.
5. Shared machines get real users, not a shared login
A shared workstation with one login that everyone knows has no access control. It has a password and a group chat.
Every team now has its own users on the workstations, and each user can only reach what that team needs. It's the least glamorous change on this list, and it's the one that makes the next rule possible, because a log that says "the shared account did it" tells you nothing.
It also changes what happens when someone leaves. You remove one user. You don't rotate a password that a dozen people have saved somewhere.
The test: someone left the team last Friday. How many passwords do you need to change? The right answer is zero.
6. Record who did what, not only how the machine is doing
We were already collecting metrics from every machine, so we could see CPU, memory and disk. Metrics tell you a machine is unwell. They don't tell you that someone logged in at 2am and changed a config file.
So we added auditing on top. A Wazuh agent on each machine collects the system journal and the kernel logs and sends them to one place, where they can be searched and where the machine's own users can't quietly edit them.
Monitoring answers "is it healthy?". Auditing answers "what happened, and who did it?". You need the second one on the worst day of the year, and by then it's too late to switch it on.
The test: a file disappeared from a workstation last Tuesday. Can you say who deleted it, and when, in under ten minutes?
7. Internal tools get their own home
Our registry, build server and storage used to run on the workstations. The workstations do a lot of other work, so the tools were only as reliable as the busiest and most-rebooted machines in the office.
They now run on a dedicated three-node Kubernetes cluster: Harbor for images, Jenkins for builds, S3-compatible storage and the Nginx proxy. Three nodes is the smallest number that keeps working when one of them dies.
The bigger win is that the two jobs are separate now. Workstations can be rebooted, reinstalled or maxed out without taking the registry down with them. And when the tools need more room, we add a node. Nobody has to negotiate for space on a machine someone else is using.
The test: restart the busiest machine in your office in the middle of the day. What stops working that has nothing to do with that machine?
8. Every change to shared config goes through Git
Nginx is the front door to everything internal, and ours lived on one machine. Changes were made by logging in and editing files. There was no record of who changed what, no way to review a change, and if that machine went down, every internal address went with it.
Two things changed.
It no longer depends on one machine. Nginx runs on more than one node, the config is replicated between them, and keepalived holds a single shared IP address that moves to a healthy node if the active one stops answering.
The config lives in a Git repository. Nobody edits it on the server any more. A team that needs a new route opens a pull request, automated checks run against it, and only a change that passes gets deployed.
This is the rule we had to insist on, because a pull request feels slower than editing a file. It is slower, by a few minutes. In exchange, a typo gets caught by a check before it takes down every internal site, and "who changed this, and why?" has an answer you can link to.
The test: who changed your proxy config last month, and what was the reason? If finding out means asking around, you don't have a record.
Where to start
You don't need all eight at once. If I were starting again from a flat network, the order would be:
- The firewall router and the split networks (rules 1 to 3). Everything else sits on top of them.
- The VPN (rule 4). It closes the most exposure for the least work.
- Real users and auditing (rules 5 and 6), together, because each is only half useful without the other.
- Moving the tools and the config (rules 7 and 8). They take the longest, and they're safe to do slowly.
None of this is exotic. It's ordinary equipment and open-source software, set up on purpose. What separates an office network that grew by accident from one you can rely on is that every "yes" in it was a decision someone made.
Run the eight tests. The ones you fail are your roadmap.