Mirrored Root on ZFS on Linux on iSCSI on Root on ZFS on FreeBSD …yo dawg! - Act 1: iSCSI

Dear diary…

…today I tried to configure a live offsite-backup for my laptop with ZFS, a remote server and iSCSI - this is (a rough schematic of) what came out:

zroot (Linux-Laptop)
  mirror-0
    TRION150
      cryptsetup
        /dev/sda (internal SSD)
    SDSSDHP-256G
      cryptsetup
        /dev/sdb (USB3 SSD)
    offsite-backup
      cryptsetup
        iSCSI
          zvol
            zroot (FreeBSD-Server)
              mirror-0
                ada0p2
                ada1p2

In words this means:
I have a local RAID1 made out of 3 backing disks:

TRION150 - my internal SATA3 240GB SSD
SDSSDHP-256G - a SATA3 256GB SSD, connected via a USB3.0 SATA adapter
offsite-backup - a “virtual” disk, brought to me by iSCSI

On every raw device is a cryptsetup-layer to encrypt my precious data.

offsite-backup is a block device connected via iSCSI, the Internet Small Computer Systems Interface.
It’s real backing device is a zvol, a ZFS Volume, residing on a FreeBSD server, on another RAID1 made out of two 2TB HDDs (which also stores the servers root-filesystem)

In this (and the following posts…) I will show you how I did this and maybe how it performs.

Disclaimer: I have no idea what I’m doing!

I don’t know if this works good enough to have it enabled most of the time.
I don’t know if it breaks as soon as your internet is flaky.
I don’t even know if it’s secure (enough).

Please don’t blindly copy anything from the web!

Let’s get to the meat of this right away, the other parts are less interesting but will/may follow in seperate posts…

Act 1 Part 1: Linux

On the linux side we only need to install open-iscsi, since I’m on Archlinux (yeah yeah, hate me for it. I do it too. :c) it’s as simple as:

pacman -S open-iscsi

on Ubuntu/Debian it’s also really simple:

apt install open-iscsi

for all other distributions: I have no clue. What the actual fuck even is “Hannah Montana Linux” and why do you use it as a daily driver?!

…uhm, after installing open-iscsi you need to edit /etc/iscsi/iscsid.conf to contain the following things (please read the examples/manpages/etc. for your distribution for more details):

iscsid.startup = /bin/systemctl start open-iscsi.service

node.session.auth.authmethod = CHAP
node.session.auth.username = __USERNAME__
node.session.auth.password = __SECRET_PASSWORD_1__

discovery.sendtargets.auth.authmethod = CHAP
discovery.sendtargets.auth.username = __USERNAME__
discovery.sendtargets.auth.password = __SECRET_PASSWORD_2__

Line 1 is totally optional AND probably different on other distros than Archlinux. It allows you to open the iSCSI-connection without manually starting iscsid first.

And with that, we are done with the linux-stuff. 🎉 Yay! 🎉

Act 1 Part 2: FreeBSD

Now over to our FreeBSD server, first of all we need to create a zvol that we want to serve over iSCSI:

zfs create -V 300G -s -b 4096 -o volmode=dev zroot/example-zvol

This creates a 300GB sparse zvol with a 4K blocksize (which is the most performant blocksize on modern disks), identifying itself like a real harddisk.
Sparse zvol means it is allowed to grow until it reaches 300GB but won’t pre-allocate that space.

Now we can configure the server, just create /etc/ctl.conf with the following content (and replace with your own values…):

auth-group dag0 {
    initiator-name __INITIATIOR_NAME__ # replace with the name in /etc/iscsi/initiatorname.iscsi on your linux-host
    chap __USERNAME__ __SECRET_PASSWORD_2__ # replace with a password, up to 16 characters
}

auth-group tag0 {
    initiator-name __INITIATIOR_NAME__ # replace with the name in /etc/iscsi/initiatorname.iscsi on your linux-host
    chap __USERNAME__ __SECRET_PASSWORD_1__ # replace with a DIFFERENT password, up to 16 characters
}

# this listens on all IP adresses, you probably don't want this because
# iSCSI is unencrypted, don't use it over the big bad internet without a VPN
portal-group pg0 {
    discovery-auth-group dag0
    listen 0.0.0.0
    listen [::]
}

# the target is a bit tricky, let's split it up:
# iqn - a constant, don't change this!
# 2018-07 - year and month of "when you gained controll over this target" …no idea what this is for, just set it to current date
# com.example - a FQDN of this host, in reverse domain name notation: https://en.wikipedia.org/wiki/Reverse_domain_name_notation
# target0 - arbitrary name, choose freely or leave it as is
target iqn.2018-07.com.example:target0 {
    auth-group tag0
    portal-group pg0

    lun 0 {
        path /dev/zvol/zroot/example-zvol # replace with your zvol name, see above "creating the zvol"
        blocksize 4096 # should be the same as when you created the zvol
    }
}

after this add the line

ctld_enable="YES"

to your /etc/rc.conf and finally run:

service ctld start

(Make sure tcp port 3260 is opened in any firewall in front of this server.)

and we are also done with FreeBSD. 🎉🎉🎉

Act 1 Part 3: Back on Linux. Using the stuff.

After configuring all of this, we still need to connect the disk.

First you need to ask the server what disks they serve:

iscsiadm -m discovery -t sendtargets -p $serverFQDN:3260

Then you can connect the disk (replace $targetstring with one of the targets from previous command):

iscsiadm -m node -T $targetstring -p $serverFQDN:3260 --login

Now you have an additional block device in /dev.
Use it to your needs/liking. 🎉🎉🎉

When you are done with the device you can disconnect (like hot-swapping SATA) it with:

iscsiadm -m node -T $targetstring -p $serverFQDN:3260 --logout

(again, replace $targetstring with one of the targets from previous command)

PS: the resilvering on my 200/50 MBit/s connection will be finished in…

zpool status

  pool: zroot
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Wed Jul 11 17:41:08 2018
	3,52G scanned out of 146G at 408K/s, 101h23m to go
	3,49G resilvered, 2,41% done
config:

	NAME                STATE     READ WRITE CKSUM
	zroot               ONLINE       0     0     0
	  mirror-0          ONLINE       0     0     0
	    SDSSDHP-256G    ONLINE       0     0     0
	    TRION150        ONLINE       0     0     0
	    offsite-backup  ONLINE       0   243     0  (resilvering)

errors: No known data errors

One hundred and one hours! Yep. 😐