17 NVMe-oF #

To use NVMe-oF, you need the nvme command line tool. Install it with zypper:

> sudo zypper in nvme-cli

Use nvme --help to list all available subcommands. Man pages are available for nvme subcommands. Consult them by executing man nvme-SUBCOMMAND. For example, to view the man page for the discover subcommand, execute man nvme-discover.

To list available NVMe subsystems on the NVMe-oF target, you need the discovery controller address and service ID.

> sudo nvme discover -t TRANSPORT -a DISCOVERY_CONTROLLER_ADDRESS -s SERVICE_ID

Replace TRANSPORT with the underlying transport medium: loop, rdma, tcp, or fc. Replace DISCOVERY_CONTROLLER_ADDRESS with the address of the discovery controller. For RDMA and TCP, this should be an IPv4 address. Replace SERVICE_ID with the transport service ID. If the service is IP based, like RDMA or TCP, service ID specifies the port number. For Fibre Channel, the service ID is not required.

The NVMe hosts only see the subsystems they are allowed to connect to.

Example:

> sudo nvme discover -t tcp -a 10.0.0.3 -s 4420

For the FC, the example looks as follows:

> sudo nvme discover --transport=fc \
                --traddr=nn-0x201700a09890f5bf:pn-0x201900a09890f5bf \
                --host-traddr=nn-0x200000109b579ef6:pn-0x100000109b579ef6

For more details, see man nvme-discover.

After you have identified the NVMe subsystem, you can connect it with the nvme connect command.

> sudo nvme connect -t transport -a DISCOVERY_CONTROLLER_ADDRESS -s SERVICE_ID -n SUBSYSTEM_NQN

Replace TRANSPORT with the underlying transport medium: loop, rdma, tcp or fc. Replace DISCOVERY_CONTROLLER_ADDRESS with the address of the discovery controller. For RDMA and TCP this should be an IPv4 address. Replace SERVICE_ID with the transport service ID. If the service is IP based, like RDMA or TCP, this specifies the port number. Replace SUBSYSTEM_NQN with the NVMe qualified name of the desired subsystem as found by the discovery command. NQN is the abbreviation for NVMe Qualified Name. The NQN must be unique.

Example:

> sudo nvme connect -t tcp -a 10.0.0.3 -s 4420 -n nqn.2014-08.com.example:nvme:nvm-subsystem-sn-d78432

For the FC, the example looks as follows:

> sudo nvme connect --transport=fc \
             --traddr=nn-0x201700a09890f5bf:pn-0x201900a09890f5bf \
             --host-traddr=nn-0x200000109b579ef6:pn-0x100000109b579ef6 \
             --nqn=nqn.2014-08.org.nvmexpress:uuid:1a9e23dd-466e-45ca-9f43-a29aaf47cb21

Alternatively, use nvme connect-all to connect to all discovered namespaces. For advanced usage, see man nvme-connect and man nvme-connect-all.

In case of a path loss, the NVMe subsystem tries to reconnect for a time period, defined by the ctrl-loss-tmo option of the nvme connect command. After this time (default value is 600s), the path is removed and the upper layers of the block layer (file system) are notified. By default, the file system is then mounted read-only, which usually is not the expected behavior. Therefore, it is recommended to set the ctrl-loss-tmo option so that the NVMe subsystem keeps trying to reconnect without a limit. To do so, run the following command:

> sudo nvme connect --ctrl-loss-tmo=-1

To make an NVMe over Fabrics subsystem available at boot, create a /etc/nvme/discovery.conf file on the host with the parameters passed to the discover command (as described in Section 17.2.2, “Discovering NVMe-oF targets”. For example, if you use the discover command as follows:

> sudo nvme discover -t tcp -a 10.0.0.3 -s 4420

Add the parameters of the discover command to the /etc/nvme/discovery.conf file:

echo "-t tcp -a 10.0.0.3 -s 4420" | sudo tee -a /etc/nvme/discovery.conf

Then enable the nvmf-autoconnect service:

> sudo systemctl enable nvmf-autoconnect.service

NVMe native multipathing is enabled by default. If the CMIC option in the controller identity settings is set, the NVMe stack recognizes an NVME drive as a multipathed device by default.

To manage the multipathing, you can use the following:

To configure an NVMe-oF target, you need the nvmetcli command line tool. Install it with zypper:

> sudo zypper in nvmetcli

The current documentation for nvmetcli is available at https://git.infradead.org/users/hch/nvmetcli.git/blob_plain/HEAD:/Documentation/nvmetcli.txt.

The following procedure provides an example of how to set up an NVMe-oF target.

The configuration is stored in a tree structure. Use the command cd to navigate. Use ls to list objects. You can create new objects with create.

You can save the target configuration in a JSON file with the following commands:

> sudo nvmetcli
(nvmetcli)> saveconfig nvme-target-backup.json

To restore the configuration, use:

(nvmetcli)> restore nvme-target-backup.json

You can also wipe the current configuration:

(nvmetcli)> clear

Some hardware needs special configuration to work correctly. Skim the titles of the following sections to see if you are using any of the mentioned devices or vendors.

If you are using the Broadcom Emulex LightPulse Fibre Channel SCSI driver, add a Kernel configuration parameter on the target and host for the lpfc module:

> sudo echo "options lpfc lpfc_enable_fc4_type=3" > /etc/modprobe.d/lpfc.conf

Make sure that the Broadcom adapter firmware has at least version 11.4.204.33. Also make sure that you have the current versions of nvme-cli, nvmetcli and the Kernel installed.

To enable a Fibre Channel port as an NVMe target, an additional module parameter needs to be configured: lpfc_enable_nvmet= COMMA_SEPARATED_WWPNS. Enter the WWPN with a leading 0x, for example lpfc_enable_nvmet=0x2000000055001122,0x2000000055003344. Only listed WWPNs will be configured for target mode. A Fibre Channel port can either be configured as target or as initiator.

FC-NVMe is supported on QLE269x and QLE27xx adapters. FC-NVMe support is enabled by default in the Marvell® QLogic® QLA2xxx Fibre Channel driver.

To confirm NVMe is enabled, run the following command:

> cat /sys/module/qla2xxx/parameters/ql2xnvmeenable

A resulting 1 means NVMe is enabled, a 0 indicates it is disabled.

Next, ensure that the Marvell adapter firmware is at least version 8.08.204 by checking the output of the following command:

> cat /sys/class/scsi_host/host0/fw_version

Last, ensure that the latest versions available for SUSE Linux Enterprise Server of nvme-cli, QConvergeConsoleCLI, and the Kernel are installed. You may, for example, run

# zypper lu && zypper pchk

to check for updates and patches.

For more details on installation, please refer to the FC-NVMe sections in the following Marvell user guides:

To boot the system from NVMe-oF over TCP, the following requirements must be met:

To install SLES from NVMe-oF over TCP, proceed as follows:

After installation, the system should boot from NVMe-oF over TCP automatically. If it does not, check if the boot priority is set correctly in the BIOS setup.

The network interfaces used for booting are named nbft0, nbft1 and so on. To get information about the NVMe-oF boot, run the command:

# nvme nbft show