Client-side encrypted cloud backups with TrueNAS, Backblaze, and rclone

I’ve kicked this can down the road far too long…as usual, analysis paralysis. I didn’t want to dive in. I’d done some preliminary research and scratched up a plan for my ideal backups. Up to this point, though, all my backups are just on-site - Syncthing, VM snapshots synced via SMB, and ZFS snapshots of the datasets (surprisingly recent on the ZFS snapshots). If my NAS were to die or if there was a fire, I’d be SOL.

My goal backup strategy is as follows:

• ZFS and VM snapshots as a first line of defense
• Secondary NAS as hot on-site (way in the future when I have more money to blow…)
• Encrypted hard drive as cool on-site
• Cloud storage as warm off-site
• Encrypted hard drive as cold/archive off-site, with a trusted friend

This should cover most scenarios. The cloud storage one will probably fill my biggest gap for the time being - disaster recovery - but I do also plan to implement the off-site hard drive soon too.

I had a day off, so I decided to dive in. Not half as bad as I made it out to be!

Choosing a provider

I’d heard good things about Hetzner when doing prior research, and they’d been the top one on my list. I signed up this morning, filled in all my personal information, gave them my payment information - and I got hit with some KYC bullshit. Apparently I had to upload a picture of my ID before I could give them my money. In the end, I decided not to move forward with them, and I went back to the drawing board.

I settled on a couple alternatives - Backblaze and rsync.net. I ultimately ended up going with Backblaze B2, mainly due to pricing - and it was my plan from the beginning to encrypt locally before pushing, so I wasn’t too concerned about location or privacy, as long as I didn’t have to submit my ID to go who-knows-where for AI scans. (I will say, I did appreciate rsync.net’s no-nonsense website, privacy policy, and direct access via SSH…I will consider them as an alternative in the future.) So far, I haven’t run into any issues with that aspect.

Restructuring my NAS (again…)

I’ll keep this section short, as I’ve already written at length about this, but I had to do this before actually backing something up.

One of the reasons I kept kicking this can down the road was the sheer volume of data I had, and how to separate what I really needed to back up. I wanted to jump in and get my critical files backed up, but my separation was not great (albeit better than it was when I was using a monolithic iSCSI share for literally everything).

So initially, what I planned to do was create sub-datasets and mount them all as individual SMB shares on the target machine. I realized halfway through creating all the new datasets and shares that I didn’t need to do this, as when I created a sub-dataset, it showed up as a subfolder on the mounted parent dataset. So I could have the granular control of an individual dataset, but just mount the parent dataset as the SMB share. Nice! (Also, in future, I could create a separate SMB share on the child dataset for more granular access.)

I still did have to rename the existing folders, create the new datasets, and move the contents of the renamed folder into the new dataset’s folder, which took some time. My new folder structure looks like the following. Each folder is an individual dataset; the folders one level under tank are all SMB shares.

tank
├── bups
│   ├── mini
│   ├── proxmox
│   └── syncthing
├── dump
├── files
│   ├── documents
│   └── notes
└── media
    ├── books
    ├── downloads
    ├── music
    ├── notes
    ├── pictures
    └── videos

Setting up backup task on TrueNAS

There are plenty of good articles out there covering this, I’ll just write the gist for future me.

• Backblaze admin panel
  • Buckets → Create a bucket - private, unencrypted (I haven’t tested server-side encryption yet, as I plan to do client-side)
  • Application Keys → Add a new application key - access the new bucket you created, RW
  • You’ll also need to go into Caps & Alerts to add payment information if you have more than 10GB to upload (Backblaze’s free tier)
• TrueNAS admin panel
  • Credentials → Backup credentials → Add a cloud credential - provider Backblaze B2, paste your API key and key ID here
  • Data Protection → Add a cloud sync tasks, select Advanced options
    • Description: your choosing
    • Credential: B2 you just created
    • Direction: push
    • Bucket: what you just created
    • Transfer mode: you choose, I went with sync for now
    • Folder (on B2): directory structure in the bucket is up to you, I went with a folder for each dataset under a truenas folder in the root
    • Directory (on NAS): whatever you want to back up
    • Schedule: I went with daily, 2:01am (I’m trying to remember this)
    • Use snapshot: checked
    • Use --fast-list: probably should be checked, as it reduces API queries significantly. I forgot to check it and just corrected.
    • Remote encryption: checked. Filename encryption: checked. Enter a password and a salt value.
      • A note on filename encryption… the TrueNAS GUI says that it is an “experimental” feature of rclone, and I have no idea where this came from. Sure, I did my research and there is a filename length limit, but there is nothing in the official rclone documentation that I see that says this feature is somehow experimental. I’m using this, because I don’t want to worry about PII in my filenames being visible to the storage provider. (I have several gripes with TrueNAS’s GUI design; this is just the most recent.)
    • Rest can be defaults. Go through a dry-run to test everything.

Untested backups are the same as not having backups…

I think that’s how the saying goes.

At this point, what I wanted to do was actually test my access to these files. I have no experience with rclone. Since I used filename encryption, I see a bunch of gibberish when I log into Backblaze’s storage viewer (which is the idea!). I needed to make sure that if my NAS went belly-up, I still had access to the files through old faithful, the CLI.

Here’s the setup with rclone.

• Install rclone with your favorite package manager.
• rclone config - add a new remote
  • n for new remote
  • enter a name (let’s say b2)
  • b2 for storage provider
  • paste your API key ID
  • paste your API key itself
  • hard_delete at default
  • You can skip advanced config
• rclone config again - add an encrypted remote
  • n for new remote
  • enter a name (let’s say b2-crypt)
  • crypt to encrypt a remote
  • b2:<bucket>/path/to/dataset for the remote to encrypt/decrypt
  • 1 or enter to encrypt filenames
  • 1 or enter to encrypt directory names
  • Important: select y to type in the same password you set in the TrueNAS GUI portal
  • Important: select y to type in the same salt you set in the TrueNAS GUI portal
  • You can skip advanced config

Now you should be able to use rclone ls b2:<bucket>/path/to/dataset and see your files not as gibberish! I won’t go through other rclone commands here - they have a decent-looking documentation page.

A couple important notes:

• Entering the path (the remote to encrypt/decrypt) in the second config is the step I got hung up on. You need to make sure it matches the backup directory on the B2 side that you set when creating it in the TrueNAS GUI. So if you set the subfolder to truenas/syncthing, the remote should be specified as b2:truenas/syncthing.
• Add export RCLONE_FAST_LIST=true to your bashrc or zshrc. It adds the --fast-list option to rclone commands by default, which helps limit the API calls. I haven’t done enough testing to see if I would exceed the 2500 API call free-tier maximum (my minimal testing ended up landing at about 150 calls), but I’d rather be safe than get charged. They are cheap…less than half a cent per so many calls, but I imagine that can pile up quickly if you’re not careful. (I’ve made similar mistakes in the past.)

EOF