Eight rules for an on-prem office network, and a test for each

We host everything ourselves, from the registry to the internal tools. After rebuilding our office network, access and tooling from the ground up, these are the eight rules I'd insist on anywhere, each with a test you can run on your own office this week.

networkingsecurityon-premkubernetesdevops

Our office runs on its own hardware. The container registry, the build server, the storage and every internal tool sit on machines down the corridor instead of in someone's cloud.

For a long time, this is what that looked like:

  • Two internet providers, with each developer tied to one of them. When a provider dropped, the people on it got up, joined the other network and started again.
  • Shared workstations reachable on every port from anywhere on the network.
  • Internal tools running on those same workstations, next to everything else the workstations were busy doing.
  • One Nginx box holding all the routing, edited by hand by whoever needed a change.

None of it was anyone's fault. It's what an office network turns into when it grows one quick fix at a time.

I lead infrastructure here, so I rebuilt it. What came out is eight rules I'd now apply to any on-prem office. Each one has a test. If you can't pass the test today, you've found your next project.

1. No one's internet should depend on one provider

Having two providers is not the same as having redundancy. We had two, and an outage still stopped work, because each person's connection belonged to one of them.

The fix was a firewall router with every provider plugged into it. The office talks to the router, and the router decides which line the traffic leaves on. Addresses inside the office are handed out by the router, so they don't belong to any provider.

Provider A up down Provider B up Firewall router every line ends here Office same addresses both lines up provider A drops, and nobody has to do anything Provider A up down Provider B up Firewall router every line ends here Office same addresses both lines up provider A drops, and nobody has to do anything
One provider drops and the router keeps sending traffic out through the other. Nothing inside the office changes.

An outage used to be an event people reacted to. Now the router handles it and the office carries on.

The test: unplug your main line at 11am on a working day. Count how many people notice.

2. Split the network by who is on it

A flat network treats everything on it as equally trusted: a visitor's phone, a build server and the box holding your backups. Whatever gets onto one of them can see all the others.

We run five separate networks (VLANs) over one trunk:

Network Who is on it
Developers Laptops of the people who work here
Workstations The shared compute machines
Internal tools Registry, build server, storage, proxy
NAS Storage devices
Guests Anyone from outside

Wi-Fi follows the same split, so each access point puts you on exactly one of these networks. Which network you're on comes from what you connected to, not from someone remembering to configure your laptop.

The test: take a laptop you've never seen before and join the guest Wi-Fi. List everything it can reach. The right answer is the internet and nothing else.

3. Between networks, the default answer is no

Splitting the network does nothing if every network can still talk to every other one. The firewall rules between them are where the protection comes from.

The rule that matters is the one at the bottom of the list: anything not explicitly allowed is refused. Then every allowed path is a line someone wrote on purpose.

Firewall default: deny allow deny Developers Guests Internal tools Workstations NAS Firewall default: deny allow deny Developers Guests Internal tools Workstations NAS
A developer reaching the internal tools is allowed by a written rule. A guest reaching the storage has no rule, so the firewall refuses it.

Write the rules down as a grid before you touch the firewall. For example:

From \ To Internal tools Workstations NAS Internet
Developers yes through the VPN no yes
Workstations yes yes yes yes
Internal tools yes no yes yes
Guests no no no yes

A grid like this fits on one screen, and a new joiner can read it in a minute. If yours doesn't fit on one screen, you have too many exceptions.

The test: pick any two networks. Without opening the firewall, say whether they can talk and why. Then open it and check.

4. Nothing is reachable until you've logged in to the VPN

Our workstations used to accept connections on every port from every address on the network. That's convenient, right up to the day one laptop on that network is compromised.

Now there is one way in. You connect to an SSL VPN, you prove who you are, and only then does SSH to a workstation work at all.

office network direct SSH SSL VPN Laptop anywhere VPN who are you? Workstations SSH office network direct SSH SSL VPN Laptop anywhere VPN who are you? Workstations SSH
Connecting straight to a workstation fails at the edge of the network. The only path in goes through the VPN, which asks who you are first.

The VPN gives you two things. There is far less to attack, because a machine that doesn't answer can't be probed. And there is one door, so there is one place to see who came in and one place to switch someone off.

The test: from a network that isn't your office, scan your own public address.

nmap -Pn -p 22,80,443,3389,5000,8080 <your-office-public-ip>

Every port that answers is something you're offering to the whole internet. Make sure each one is on purpose.

5. Shared machines get real users, not a shared login

A shared workstation with one login that everyone knows has no access control. It has a password and a group chat.

Every team now has its own users on the workstations, and each user can only reach what that team needs. It's the least glamorous change on this list, and it's the one that makes the next rule possible, because a log that says "the shared account did it" tells you nothing.

It also changes what happens when someone leaves. You remove one user. You don't rotate a password that a dozen people have saved somewhere.

The test: someone left the team last Friday. How many passwords do you need to change? The right answer is zero.

6. Record who did what, not only how the machine is doing

We were already collecting metrics from every machine, so we could see CPU, memory and disk. Metrics tell you a machine is unwell. They don't tell you that someone logged in at 2am and changed a config file.

So we added auditing on top. A Wazuh agent on each machine collects the system journal and the kernel logs and sends them to one place, where they can be searched and where the machine's own users can't quietly edit them.

Monitoring answers "is it healthy?". Auditing answers "what happened, and who did it?". You need the second one on the worst day of the year, and by then it's too late to switch it on.

The test: a file disappeared from a workstation last Tuesday. Can you say who deleted it, and when, in under ten minutes?

7. Internal tools get their own home

Our registry, build server and storage used to run on the workstations. The workstations do a lot of other work, so the tools were only as reliable as the busiest and most-rebooted machines in the office.

They now run on a dedicated three-node Kubernetes cluster: Harbor for images, Jenkins for builds, S3-compatible storage and the Nginx proxy. Three nodes is the smallest number that keeps working when one of them dies.

node 1 Harbor Nginx node 2 down Jenkins node 3 S3 storage Jenkins three nodes, all healthy node 2 dies, Jenkins comes back on node 3 node 1 node 2 node 3 Harbor Nginx down Jenkins S3 storage Jenkins three nodes, all healthy node 2 dies, Jenkins comes back on node 3
Lose a node and the cluster restarts its workloads on the two that are left.

The bigger win is that the two jobs are separate now. Workstations can be rebooted, reinstalled or maxed out without taking the registry down with them. And when the tools need more room, we add a node. Nobody has to negotiate for space on a machine someone else is using.

The test: restart the busiest machine in your office in the middle of the day. What stops working that has nothing to do with that machine?

8. Every change to shared config goes through Git

Nginx is the front door to everything internal, and ours lived on one machine. Changes were made by logging in and editing files. There was no record of who changed what, no way to review a change, and if that machine went down, every internal address went with it.

Two things changed.

It no longer depends on one machine. Nginx runs on more than one node, the config is replicated between them, and keepalived holds a single shared IP address that moves to a healthy node if the active one stops answering.

The config lives in a Git repository. Nobody edits it on the server any more. A team that needs a new route opens a pull request, automated checks run against it, and only a change that passes gets deployed.

Pull request a team asks Checks run config is tested Merge only if it passes Live deployed Pull request a team asks Checks run config is tested Merge only if it passes Live deployed
Nobody edits the proxy by hand. A change is proposed, tested and only then deployed.

This is the rule we had to insist on, because a pull request feels slower than editing a file. It is slower, by a few minutes. In exchange, a typo gets caught by a check before it takes down every internal site, and "who changed this, and why?" has an answer you can link to.

The test: who changed your proxy config last month, and what was the reason? If finding out means asking around, you don't have a record.

Where to start

You don't need all eight at once. If I were starting again from a flat network, the order would be:

  1. The firewall router and the split networks (rules 1 to 3). Everything else sits on top of them.
  2. The VPN (rule 4). It closes the most exposure for the least work.
  3. Real users and auditing (rules 5 and 6), together, because each is only half useful without the other.
  4. Moving the tools and the config (rules 7 and 8). They take the longest, and they're safe to do slowly.

None of this is exotic. It's ordinary equipment and open-source software, set up on purpose. What separates an office network that grew by accident from one you can rely on is that every "yes" in it was a decision someone made.

Run the eight tests. The ones you fail are your roadmap.