the Vimoire

recent

See 69 more →

Moving iSCSI share to dedicated storage network

Jan 17, 20256 min read

Overview

This was an interesting exercise, but I’m still trying to figure out in my head what benefit, if any, this has. It doesn’t do much if anything for security except through obscurity - I tested this by setting a static IP in the 10.20.99.0/24 network on my laptop and I could communicate directly with the devices (to no one’s surprise). It does keep the storage traffic isolated from the regular network traffic…I think? At least, the share shouldn’t be directly reachable except through its storage network IP. Except, it is, according to the iSCSI discovery command. There’s no practical isolation - it’s not using different devices nor different physical interfaces, just a different subnet. The only things preventing access are the initiator restrictions and allowed networks (just a single IP) on the NAS iSCSI share settings. And the authentication password, but that doesn’t really count.

This setup seems like it would go hand-in-hand with firewall rules, but I’m not there yet.

A thought for the future: I wonder if I had a separate physical NIC (I currently don’t have a second port on the NAS), I could run a cable directly from the NAS to the mini PC, and have a completely isolated network. That seems like it would accomplish my goals of keeping the traffic totally isolated / inaccessible. I still don’t know how to limit discoverability on the NAS iSCSI share to just a specific network, if that’s possible in the first place, but that shouldn’t matter as much. The main thing is keeping the storage network inaccessible, which doesn’t seem possible using sub-interfaces. Having the 10.20.99.0/24 network exist only on those two interfaces, which will not be connected to the router (where my problem currently lies, I think), should accomplish my goals, unless I’m misunderstanding something basic - which is entirely possible.

Basically, this was a learning and repetition exercise, not much else.

Current setup vs end goal setup

Current:

  • One subnet of 10.0.99.0/24
  • TrueNAS at 10.0.99.37
  • Mini PC at 10.0.99.40 running all my services, connected to TrueNAS via iSCSI share
  • All devices are on this subnet

Goal:

  • Dedicated storage network of 10.20.99.0/24
  • Only iSCSI share and target are on this network

Changing subnet to 10.20.99.0/24

Important

Do not set a default gateway, just an IP and netmask.

Note

I used the same last octet of the IP address on the single subnet, just to make the IPs easier to remember.

Configuring TrueNAS

Again, we’ll be setting sub-interfaces given the fact that I only have one physical interface on the NAS.

export EDITOR=vim
sudoedit /etc/network/interfaces.d/enp6s0:1
# /etc/network/interfaces.d/enp6s0:1

auto enp6s0:1
allow-hotplug enp6s0:1
iface enp6s0:1 inet static
	address 10.20.99.37
	netmask 255.255.255.0

sudo systemctl restart networking

Configuring mini PC

sudoedit /etc/network/interfaces.d/ens18:1
# /etc/network/interfaces.d/ens18:1

auto ens18:1
allow-hotplug ens18:1
iface ens18:1 inet static
	address 10.20.99.40
	netmask 255.255.255.0

sudo systemctl restart networking

Re-configuring iSCSI share

Making sure the drive is no longer in use and unmounted

On the mini PC:

# stop all docker services
cd ~/docker
for dir in $(ls .); do
	cd $dir
	sudo docker compose stop
	cd ..
done
 
# check if any other programs are using the drive
sudo lsof /media/shared
 
# turns out mpd was using the drive...
sudo systemctl stop mpd
 
# unmount the iSCSI share
sudo umount -R /media/shared

Changing permissions on iSCSI share in TrueNAS

In the TrueNAS web interface:

  • Shares → iSCSI → Edit (pencil icon)
  • Change network to 10.20.99.40/32, Save

Fixing iSCSI configuration on mini PC

See Network shares with iSCSI for commands and initial setup, just reconfiguring here. I haven’t changed the authentication settings, so I really just need to edit one configuration file from manual connection to automatic connection.

  • Logout of existing share.
  • Discover target. (Note that 4 different targets show up: one on the 10.0.99.* network, two on a 172 network which I don’t quite understand. Make sure to use the one on the 10.20.99.* network.)
  • Login to new target (10.20.99.37).
  • Edit configuration file to connect automatically.
  • Verify old configuration file is set to connect only manually.

Restricting initiators on TrueNAS

  • Shares → iSCSI → Configure
  • Initiators Groups tab → Edit
  • Uncheck “Allow all initiators”
  • Add the connected initiator to the allowed list
  • Save

Bringing everything back online

TrueNAS should already be good.

Previous for loop but run sudo docker compose up -d instead of stop.

Corrections…

I was misunderstanding something with how TrueNAS created network interfaces. I had the idea from this video (timestamp about 6:15, can’t figure out timestamping with nocookie links, if it’s even possible) . But when I went to create a new network interface, the only options I was given were Bridge, VLAN, and Link Aggregation. None of these made sense, I wanted a separate (albeit virtual not physical) network interface. So, I went into the /etc/network/interfaces.d/* directory and created the sub-interface manually.

This worked…to some extent. TrueNAS saw the IP and updated the web UI accordingly. However, neither the 10.0.99.37/24 address nor the 10.20.99.37/24 address show up anywhere else in the web GUI. Also, based on the file timestamps - over a month ago on the main interface, and yesterday on the sub-interface - I can only conclude that TrueNAS does not use the systemd networking service primarily. It stores its settings somewhere else - where, I am not sure at the moment.

I decided to try editing the interface directly in the web GUI and adding aliases for both the main network IP (for the web GUI) and the storage network IP. Now I was getting somewhere. Now going to System → General Settings → GUI Settings, I can update the Web Interface IPv4 address to something other than 0.0.0.0. Both aliases that I set showed up there!

Now, in the iSCSI share (Shares → Configure under iSCSI), Portals tab, I can update the Portal listen address to 10.20.99.37 instead of the default 0.0.0.0. Discovery is now only available via the storage network.

This still does not create a fully inaccessible network - I’m still able to do the static IP “trick” and communicate with the two devices on the storage network - but it’s better. The share is not accessible except via its 10.20.99.37 address; the output of the iscsiadm --mode discovery ... command only lists that one address instead of the 4 previously mentioned.

I’d still like to test with a physically isolated connection, but it’s not as priority now. I also want to figure out where TrueNAS is setting these network interface aliases. The conflicting sub-interface is still there untouched, and I don’t want a restart of the networking service to impact my configuration.

EOF

Backlinks