the Vimoire

recent

See 48 more →

Network shares with iSCSI

Nov 22, 20246 min read

Sources

The impetus

To set the stage, my goal setup is twofold.

  1. TrueNAS box (at 10.0.0.7). The only purpose of this box is to make storage available via the network, and maybe run Syncthing. It’s a decently powerful machine - Intel i5 8500T, 32 gigs of RAM, NVMe SSD boot drive, and 4x 12.73TiB disks running RAIDz1 for a total of about 37TiB worth of actual storage.
  2. Mini PC (AceMagic AM06 Pro AMD Ryzen) running Proxmox (mostly because I already had Proxmox already set up, and I’d like to continue to learn from maintaining it) with most of the capacity dedicated to a Debian 12 virtual machine (at 10.0.0.4). Services include Jellyfin and Navidrome for media streaming, Transmission, FileBrowser, among others.

The goal is to have the NAS live up to its name - network attached storage - not be the one-stop shop for all services on the LAN. It needs to do one thing and do it very well: store and serve my data. (This might change in the future, but for now it makes a lot of sense. I’m still in the requirement-defining phase for a lot of what I actually want to do with the NAS and how to set it up to accomplish this. Mainly the structure of the shares, as well as what exact services make more sense to run on the NAS itself, such as Syncthing.)

To accomplish this, I’ve been fighting with NFS for the past couple weeks (Setting up NFS share on TrueNAS), and haven’t had the wherewithal to dive into NFSv4. Theoretically I think that could solve my issues with file ownership / permissions and authentication, but it still doesn’t seem like a great solution. I got NFS to work, but I was not satisfied with the results. Permissions seem like a lost cause, and I hadn’t yet figured out how to mount at boot or automount.

SMB is an alternative, but I really don’t want to use it - from what I can gather it’s been retrofitted to work with Linux machines, and while fully compatible at this point, I don’t want to rely on it.

Enter stage left: iSCSI. I had absolutely no idea this existed. In short, while NFS and SMB share at the file level, iSCSI shares at the block level - e.g. it shares the disk itself across the network. This immediately clicked as the answer to my solution, at least in theory. Instead of having to wrangle file ownership and permissions on both machines, or just squash them and forget about them entirely, I can use the virtual disk served by the NAS as a normal lettered drive (/dev/sdX) on my mini PC. This means I can format with a filesystem, mount, mount automatically at boot via /etc/fstab, and set file ownership / permissions on the mini PC - the TrueNAS machine doesn’t care.

Some ramifications

  • VIrtual disks seem to be pre-allocated, as they are zvols instead of datasets (up until this point, I have set all datasets to have the maximum capacity available instead of trying to pre-partition).
  • The same issue applies when it comes to multiple machines: if I want to share this disk to multiple machines, I will have to manage users/groups/permissions on both machines.
    • I am not yet sure if I even can share this disk with more than one machine.
  • The target must support iSCSI. Mounting on demand from another device may not work; it may only be accessible through the mini PC.

The process

Creating the zvol on the TrueNAS machine

  • Datasets → Add Zvol
  • Name: share
  • Size: 1TiB
  • Remainder at defaults
  • Save

Creating the iSCSI share

  • Shares → “Wizard” under “Block (iSCSI) Shares Targets”
  • Create or Choose Block Device
    • Name: share
    • Extent type: Device
    • Device: previously created zvol (tank/share)
    • Sharing platform: Modern OS
    • Target: Create New
  • Portal
    • Create New
    • Discovery Authentication Method: CHAP
    • Discover Authentication Group: Create New
    • Group ID: 1
    • User: root
    • Enter secret and confirm secret (no more than 16 characters)
    • IP Addresses: Add, then 0.0.0.0
  • Initiators: blank
  • Save

When prompted to start iSCSI service, make sure “Enable this service to start automatically” is checked, and confirm.

Info

After creation, click the pencil “edit” icon next to the newly created iSCSI share.
Enforce authentication:

  • Authentication method: CHAP
  • Authentication group number: 1
  • Save

Secure the connection to a single device - stopgap for more robust security:

  • Add network
  • Enter IP: 10.0.0.4 / 32

Connecting to the shared drive from the initiator machine

Discovering device

sudo apt install open-iscsi
 
# get available drives
sudo iscsiadm --mode discovery --type sendtargets --portal "10.0.0.7"
# output:
# 10.0.0.7:3260,1 iqn.2005-10.org.freenas.ctl:share

Configuring authentication and automatic startup

# /etc/iscsi/iscsid.conf
...
discovery.sendtargets.auth.authmethod = CHAP
discovery.sendtargets.auth.username = root
discovery.sendtargets.auth.password = password
...
node.session.auth.authmethod = CHAP
node.session.auth.username = root
node.session.auth.password = password
...

# /etc/iscsi/nodes/<node-name>/<ip-address:port,options>/default
...
node.startup = automatic
...

# connect to available drive
sudo iscsiadm --mode node --targetname "iqn.2005-10.org.freenas.ctl:share" --portal "10.0.0.7:3260" --login
 
# verify connectivity
sudo iscsiadm --mode session --print=1
 
# output
# Target: iqn.2005-10.org.freenas.ctl:share (non-flash)
#        Current Portal: 10.0.0.7:3260,1
#        Persistent Portal: 10.0.0.7:3260,1
#                **********
#                Interface:
#                **********
#                Iface Name: default
#                Iface Transport: tcp
#                Iface Initiatorname: iqn.1993-08.org.debian:01:728c6fc6d774
#                Iface IPaddress: 10.0.0.4
#                Iface HWaddress: default
#                Iface Netdev: default
#                SID: 1
#                iSCSI Connection State: LOGGED IN
#                iSCSI Session State: LOGGED_IN
#                Internal iscsid Session State: NO CHANGE
 
# verify drive shows up in /dev/
lsblk
 
# disconnect from drive
sudo iscsiadm --mode node --targetname "iqn.2005-10.org.freenas.ctl:share" --portal "10.0.0.7:3260" --logout
sudo systemctl start open-iscsi.service
sudo systemctl enable open-iscsi.service
sudo systemctl start iscsid.service
sudo systemctl enable iscsid.service

Partitioning and formatting the disk

My disk showed up as /dev/sdc.

sudo fdisk /dev/sdc
# create a single partition, type Linux, full size of the disk, and write
 
sudo mkfs.ext4 /dev/sdc1
 
sudo mount /dev/sdc1 /mnt

Mounting the disk automatically at startup

# get UUID of partition
sudo blkid /dev/sdc1 | cut -d' ' -f2 | cut -d\" -f2
# edit /etc/fstab
...
UUID=<uuid> /<mountpoint> ext4 _netdev 0 0

sudo systemctl daemon-reload
sudo mount -a

Final thoughts

  • Still haven’t figured out the overall structure/types of libraries to share - I’m pretty sure that I don’t want everything on an iSCSI share.
  • Idea: could I expose a subfolder of this as an NFS share? I don’t see why I couldn’t…it would just be from the mini PC IP instead directly from the TrueNAS server. I’m not sure if that solves any of my issues.
  • Security is sketchy. I enforced to only one client for the time being. Another solution would be to just run all services on the NAS itself, and not have to expose the storage device over the network. Something to look into in the future.
  • Related to the foregoing: might try out encryption?

EOF

Backlinks