Jump to content

# NVMe over Fabrics

From ArchWiki

NVMe over Fabrics (NVMe-oF) allows sending NVMe commands over Ethernet or Fibre Channel. This can be used for remote access to block devices similar to iSCSI. An NVMe-oF host can access devices exposed by an NVMe-oF controller. This is not limited to NVMe devices, you can also expose block devices such as ZFS zvols.

## Host configuration

To access a block device exposed by a remote NVMe-oF controller, install the nvme-cli package.

First, make sure the necessary kernel module is loaded:

    # modprobe nvme-fabrics

Now you can use the CLI to connect to the device:

    # nvme connect --transport=tcp --traddr=192.168.0.5 --trsvcid=4420 --nqn=nqn.2024-08.com.example:my-disk

--transport=tcp
  Specifies the transport protocol. TCP is the most simple to set up.
--traddr=192.168.0.5
  The address of the NVMe-oF target, i.e. the server that exposes the block device.
--trsvcid=4420
  The port the target is listening on. 4420 is the recommended port (IANA assignment).
--nqn=nqn.2024-08.com.examplemy-disk
  The NVMe Qualified Name (NQN) of the device to connect to. The device must be configured on the controller.

After running this command, the device will be available as a normal NVMe block device, e.g. under `/dev/nvme0n1` with partitions under `/dev/nvme0n1p1`. It is recommended to refer to these devices by UUID, however.

## Controller configuration

Install the nvmetcli package.

First, make sure the necessary kernel module is loaded:

    # modprobe nvmet

Controller configuration happens through a file system located at `/sys/kernel/config/nvmet`. nvmetcli provides a convenient interface to modify that file system. It also provides a way to load and save the settings. `nvmetcli save` will save the current state to `/etc/nvmet/config.json`, and `nvmetcli restore` will load from that location. You can enable the `nvmet.service` unit to load the configuration from that file automatically.

Warning: nvmetcli will only save, load and edit settings that it recognizes. If you configure unsupported settings (such as DHCHAP) they will be silently dropped. You will have to write them manually to the above filesystem.

Inside `nvmetcli`, you can navigate using `cd` and show the config tree using `ls`. There are three top-level directories.

`hosts`
  contains the NQNs of hosts that are referenced in other parts of the tree, e.g. for access control. You can add an NQN here and then make the client use that NQN using the `nvme connect --hostnqn=...` parameter.
`ports`
  configures the protocols used to access the devices, e.g. a TCP port.
`subsystems`
  configures the individual accessible devices.

### Adding a device

First you'll want to create a device to expose:

    /> cd subsystems
    /subsystems> create nqn.2024-08.com.example:my-device
    /subsystems> cd nqn.2024-08.com.example:my-device
    /subsystems/n...ple:my-device>

Configure access for the device:

    /subsystems/n...ple:my-device> set attr allow_any_host=1

Warning: This will allow any host with network access to this controller to access the block device. There is no further access control by default.

Configure a namespace for the device and set the backing block device:

    /subsystems/n...ple:my-device> cd namespaces
    /subsystems/n...ce/namespaces> create 1
    /subsystems/n...ce/namespaces> cd 1
    /subsystems/n.../namespaces/1> set device path=/dev/path/to/block/device
    /subsystems/n.../namespaces/1> enable

### Configuring the port

Next, create a port. Navigate back to the top using `cd /`.

    /> cd ports
    /ports> create 1
    /ports> cd 1
    /ports/1> set addr trtype=tcp traddr=192.168.0.5 trsvcid=4420 adrfam=ipv4

trtype
  The transport protocol
traddr
  The listener address
trsvcid
  The listener TCP port ("service ID")

Add the device you created above to the network port:

    /ports/1> cd subsystems
    /ports/1/subsystems> create nqn.2024-08.com.example:my-device

That's it! Now the device is accessible to the host using the above instructions. To make this permanent, give the `saveconfig` command and ensure `nvmet.service` is enabled/started.

## Authentication using DHCHAP

It is prudent to not allow just any device access to the exposed block device. To authenticate the host, NVMe-oF offers a DHCHAP handshake using a secret shared between host and controller. Unfortunately, as of writing, nvmetcli does not support this out of the box, so you have to modify the config FS manually. The host `nvme` command does support DHCHAP.

First, use `nvmetcli` to limit access to the device to only a single host, identified by NQN.

    /> cd hosts
    /hosts> create nqn.2024-08.com.example.host
    /hosts> cd /subsystems/nqn.2024-08.com.example:my-device
    /subsystems/n...ple:my-device> set attr allow_any_host=0
    /subsystems/n...ple:my-device> cd allowed_hosts
    /subsystems/n...allowed_hosts> create nqn.2024-08.com.example.host

The NVMe controller now knows that only the host with the NQN `nqn.2024-08.com.example.host` may access the device. You can test this (without DHCHAP, for now) using `nvme connect --hostnqn=nqn.2024-08.com.example.host`. With the wrong NQN, the connection should fail, with the right NQN, it should succeed.

The next step is to configure DHCHAP. First you need to generate a secret like so:

    $ nvme gen-dhchap-key --nqn=nqn.2024-08.com.example.host
    DHHC-1:00:znDcb37R200FNlZkIOkv37idpu/notvalid!!si1VQ09KhKv2g:

Because nvmetcli does not support DHCHAP, you must configure it manually:

    # echo 'DHHC-1:00:znDcb37R200FNlZkIOkv37idpu/notvalid!!si1VQ09KhKv2g:' > /sys/kernel/config/nvmet/hosts/nqn.2024-08.com.example.host/dhchap_key

Now, you can connect using the DHCHAP secret:

    # nvme connect --transport=tcp --traddr=192.168.0.5 --trsvcid=4420 --nqn=nqn.2024-08.com.example:my-disk --hostnqn=nqn.2024-08.com.example.host --dhchap-secret="DHHC-1:00:znDcb37R200FNlZkIOkv37idpu/notvalid!!si1VQ09KhKv2g:"

Warning: DHCHAP will authenticate the host but not encrypt the block device data on the network.

## Booting from NVMe-oF

It is possible to connect to an NVMe-oF device during the boot process inside initramfs and use the device for the root file system. The procedure is very similar to that for iSCSI. Only the NVMe-specific parts are listed here for now.

The `initcpio` hook is different for NVMe-oF. The install script adds different modules, and adds the `nvme` binary instead of `iscsistart`:

    /etc/initcpio/install/nvme-of

    build () {
    	map add_module nvme-fabrics nvme-tcp nvme-keyring
    	add_checked_modules "/drivers/net"
    	add_binary nvme
    	add_runscript
    }
    help () {
    	cat <<HELPEOF
    	This hook allows you to boot from an NVMe-oF target.
    HELPEOF
    }

The actual hook uses the CLI to connect to the device:

    /etc/initcpio/hooks/nvme-of

    run_hook () {
    	msg "Mounting NVMe-oF target"
    	nvme connect --transport=tcp --nqn=nqn.2024-08.com.example:my-device --traddr=192.168.0.5 --trsvcid=4420
    }

Tip: If you configured DHCHAP, make sure to add the required arguments here.

Modify the `mkinitcpio` configuration:

    /etc/mkinitcpio.conf

    MODULES=(... nvme-fabrics)
    HOOKS=(... net nvme-of block ...)

## Troubleshooting

### Page allocation failure

If you are seeing this failure on the host:

    nvme nvme0: Connect command failed, error wo/DNR bit: 6
    nvme nvme0: failed to connect queue: 3 ret=6
    could not add new controller: failed to write to nvme-fabrics device

Check on the controller:

    dmesg

    kworker/5:1H: page allocation failure: order:6, mode:0x40dc0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null),cpuset=/,mems_allowed=0
    CPU: 5 PID: ... Comm: kworker/5:1H Tainted: P           OE      ...
    Hardware name: ...
    Workqueue: nvmet_tcp_wq nvmet_tcp_io_work [nvmet_tcp]
    Call Trace:
     <TASK>
     dump_stack_lvl+0x4d/0x70
     warn_alloc+0x165/0x1e0
     ? __alloc_pages_direct_compact+0x163/0x390
     __alloc_pages_slowpath.constprop.0+0xce9/0xde0
     __alloc_pages+0x320/0x340
     ? nvmet_tcp_install_queue+0x50/0x140 [nvmet_tcp ...]
     __kmalloc_large_node+0x71/0x130
     __kmalloc+0xc4/0x130
     nvmet_tcp_install_queue+0x50/0x140 [nvmet_tcp ...]
     nvmet_install_queue+0xa6/0x1f0 [nvmet ...]
     nvmet_execute_io_connect+0xd1/0x1a0 [nvmet ...]
     nvmet_tcp_io_work+0x811/0x880 [nvmet_tcp ...]
     process_one_work+0x180/0x350
     worker_thread+0x315/0x450
     ? __pfx_worker_thread+0x10/0x10
     kthread+0xe8/0x120
     ? __pfx_kthread+0x10/0x10
     ret_from_fork+0x34/0x50
     ? __pfx_kthread+0x10/0x10
     ret_from_fork_asm+0x1b/0x30
     </TASK>

If the stack traces match, you are running into a linux-nvme bug. To serve a new TCP connection, the NVMe target / controller allocates a relatively large contiguous buffer in kernel space. If kernel memory is full (or fragmented), such a large contiguous region may not be available. This used to cause a kernel crash that was fixed in recent kernels, but as of February 2025 this still prevents new TCP connections to the controller. A patch for that is pending but stuck in review.

To work around this issue, you can use the following command to free up kernel memory on the controller prior to connection:

    # echo 3 > /proc/sys/vm/drop_caches

Another option that seems to work is to create and delete a large file in tmpfs:

    # dd if=/dev/urandom of=/tmp/random-buffer bs=1M count=4096 status=progress; rm -f /tmp/random-buffer

If that does not help, a reboot might be necessary.

Categories:

- Storage
- Networking