the Vimoire

recent

See 100 more →

Separating Proxmox Management and VM interfaces into different subnets

Nov 14, 20255 min read

I’ve long wanted to separate my Proxmox web UI and server interfaces into different subnets. I’ve got the VLAN 99 / 10.0.99.0/24 network locked down, only accessible via Tailscale-connected devices, but the SERVER VLAN 20 / 10.0.20.0/24 network much more open. Since I haven’t known how to separate Proxmox and the VM subnets (until today), I’ve had to settle for either 1) a much more locked down server, only accessible via Tailscale-connected devices, or 2) an accessible server, with a much less locked down Proxmox interface. I ended up deciding that it was more important to have the server accessible so I didn’t have to connect every device I owned to Tailscale. And since it’s on my home network, I’m a little less concerned about having everything 100% locked down immediately - I’m willing to revisit and batten down the hatches over a longer period of time, settling for a short-term less-than-ideal solution.

It’s bothered me for a while, but I’ve just let it lie. This attempt was spurred on by trying to set up wake-on-LAN via a KVM, and realizing that it’s not going to work across a subnet - and the KVM I use to send a WOL packet is staying in the MGMT VLAN/subnet.

I originally thought this to be impossible. I’d set up my WAP similarly in the past, and I knew you could only set one VLAN as an access VLAN in any given switchport setup.

I ended up sending this to a coworker of mine (who taught me what I know about networking) to have him sanity-check it.

(beginning of write-up)

Devices

  • Cisco switch - L2 only (L3-capable, but not implemented), VLAN aware (relevant: SERVER VLAN 20 @ 10.0.20.0/24 and MGMT VLAN 99 @ 10.0.99.0/24)
  • Mini PC running Proxmox-hosted VMs with one switchport/cable/NIC available.

Current configuration

  • Proxmox host at 10.0.20.39/24
  • VM hosts at 10.0.20.40/24 and 10.0.20.41/24 (same subnet as Proxmox host)
  • Switchport assigned to VLAN 20, mode access

Desired configuration

  • Proxmox host/management interface at 10.0.99.39/24
  • VM hosts same - on 10.0.20.0/24 network
  • Switchport ??? (needs to enable the above)

Notes

I don’t think this is possible without the switch being L3 aware or using two physical NICs with two physical switchports assigned to two different access VLANs (20 and 99).

For context I have a WAP as a trunk in General mode. It trunks 3 other VLANs (tagged VLANs) but it has one VLAN (99) that is untagged as the access VLAN. I access the WAP over 10.0.99.254. I cannot access the WAP over its other interface addresses (e.g. 10.0.20.254) as those are trunk instead of access.

My perceived issue here is that I want both VLAN 99 and VLAN 20 as access interfaces because the server is an end device (instead of acting as sort of a wireless switch like the WAP), and the management interface is also an end device. I need two-way communication with both. As far as I understand I cannot set multiple access / untagged VLANs on the same L2 switchport.

(end of write-up)

As it turned out, my line of reasoning here was mostly correct. I either needed to 1) make my switch L3 aware - which it is capable of doing - or 2) I needed to use a separate physical NIC + cable + switchport and assign it to a different access VLAN. However, 1) I didn’t want to dive into L3 on the switch, as I wasn’t ready to figure out how to pass all traffic back through my router’s firewall to avoid inter-VLAN routing that shouldn’t be happening. And 2) I also use the second NIC on my mini PC server to connect to my NAS for storage, and I wasn’t about to get rid of that.

I ended up mulling this over for a while after the discussion - because I really felt like it should be possible. Glad I did, because as it turns out, I was just looking in the wrong place! The bulk of the configuration should be taking place on the Proxmox side, not the switch side.

Configuration

Switch

Set the switchport to trunk mode with VLAN 20 tagged, VLAN 99 native. That’s it.

Proxmox

  • Create a VLAN-aware bridge with the physical NIC as the bridge port. Don’t assign an IP address to this bridge.
  • Create VLAN 20 by creating another bridge but then naming it vmbrX.20 (where X is your previously-created bridge device’s number - e.g. vmbr0 would mean you would name this vmbr0.20). Don’t assign an IP address to it.
  • Create VLAN 99 by the same method, but assign 10.0.99.39/24 with default gateway 10.0.99.1. (This is the management interface for the Proxmox web UI.)
  • In any VMs, go into their Hardware and set the VLAN tag to 20 on the bridge interfaces.

That’s it!

  • Proxmox is accessible via 10.0.99.39:8006.
  • VMs are accessible via their 10.0.20.X addresses.
  • Proxmox is NOT accessible via any 10.0.20.X address.
  • One switchport. Dumb L2 switch.

/etc/network/interfaces file

auto lo
iface lo inet loopback

iface enp2s0 inet manual

auto vmbr0
iface vmbr0 inet static
        bridge-ports enp2s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        bridge-pvid 99 # I don't know if this line is strictly necessary

auto vmbr0.20
iface vmbr0.20 inet manual

auto vmbr0.99
iface vmbr0.99 inet static
        address 10.0.99.39/24
        gateway 10.0.99.1

# some other stuff for storage network

EOF

Backlinks