Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
Applies to SUSE Enterprise Storage 6

12 Installation of NFS Ganesha Edit source

NFS Ganesha provides NFS access to either the Object Gateway or the CephFS. In SUSE Enterprise Storage 6, NFS versions 3 and 4 are supported. NFS Ganesha runs in the user space instead of the kernel space and directly interacts with the Object Gateway or CephFS.

Warning: Cross Protocol Access

Native CephFS and NFS clients are not restricted by file locks obtained via Samba, and vice versa. Applications that rely on cross protocol file locking may experience data corruption if CephFS backed Samba share paths are accessed via other means.

12.1 Preparation Edit source

12.1.1 General Information Edit source

To successfully deploy NFS Ganesha, you need to add a role-ganesha to your /srv/pillar/ceph/proposals/policy.cfg. For details, see Section 5.5.1, “The policy.cfg File”. NFS Ganesha also needs either a role-rgw or a role-mds present in the policy.cfg.

Although it is possible to install and run the NFS Ganesha server on an already existing Ceph node, we recommend running it on a dedicated host with access to the Ceph cluster. The client hosts are typically not part of the cluster, but they need to have network access to the NFS Ganesha server.

To enable the NFS Ganesha server at any point after the initial installation, add the role-ganesha to the policy.cfg and re-run at least DeepSea stages 2 and 4. For details, see Section 5.3, “Cluster Deployment”.

NFS Ganesha is configured via the file /etc/ganesha/ganesha.conf that exists on the NFS Ganesha node. However, this file is overwritten each time DeepSea stage 4 is executed. Therefore we recommend to edit the template used by Salt, which is the file /srv/salt/ceph/ganesha/files/ganesha.conf.j2 on the Salt master. For details about the configuration file, see Section 30.2, “Configuration”.

12.1.2 Summary of Requirements Edit source

The following requirements need to be met before DeepSea stages 2 and 4 can be executed to install NFS Ganesha:

  • At least one node needs to be assigned the role-ganesha.

  • You can define only one role-ganesha per minion.

  • NFS Ganesha needs either an Object Gateway or CephFS to work.

  • The kernel based NFS needs to be disabled on minions with the role-ganesha role.

12.2 Example Installation Edit source

This procedure provides an example installation that uses both the Object Gateway and CephFS File System Abstraction Layers (FSAL) of NFS Ganesha.

  1. If you have not done so, execute DeepSea stages 0 and 1 before continuing with this procedure.

    root@master # salt-run state.orch ceph.stage.0
    root@master # salt-run state.orch ceph.stage.1
  2. After having executed stage 1 of DeepSea, edit the /srv/pillar/ceph/proposals/policy.cfg and add the line


    Replace NODENAME with the name of a node in your cluster.

    Also make sure that a role-mds and a role-rgw are assigned.

  3. Execute at least stages 2 and 4 of DeepSea. Running stage 3 in between is recommended.

    root@master # salt-run state.orch ceph.stage.2
    root@master # salt-run state.orch ceph.stage.3 # optional but recommended
    root@master # salt-run state.orch ceph.stage.4
  4. Verify that NFS Ganesha is working by checking that the NFS Ganesha service is running on the minion node:

    root@master # salt -I roles:ganesha service.status nfs-ganesha

12.3 Active-Active Configuration Edit source

This section provides an example of simple active-active NFS Ganesha setup. The aim is to deploy two NFS Ganesha servers layered on top of the same existing CephFS. The servers will be two Ceph cluster nodes with separate addresses. The clients need to be distributed between them manually. Failover in this configuration means manually unmounting and remounting the other server on the client.

12.3.1 Prerequisites Edit source

For our example configuration, you need the following:

  • Running Ceph cluster. See Section 5.3, “Cluster Deployment” for details on deploying and configuring Ceph cluster by using DeepSea.

  • At least one configured CephFS. See Chapter 11, Installation of CephFS for more details on deploying and configuring CephFS.

  • Two Ceph cluster nodes with NFS Ganesha deployed. See Chapter 12, Installation of NFS Ganesha for more details on deploying NFS Ganesha.

    Tip: Use Dedicated Servers

    Although NFS Ganesha nodes can share resources with other Ceph related services, we recommend to use dedicated servers to improve performance.

After you deploy the NFS Ganesha nodes, verify that the cluster is operational and the default CephFS pools are there:

cephadm@adm > rados lspools

12.3.2 Configure NFS Ganesha Edit source

Check that both NFS Ganesha nodes have the file /etc/ganesha/ganesha.conf installed. Add the following blocks, if they do not exist yet, to the configuration file in order to enable RADOS as the recovery backend of NFS Ganesha.

    Enable_NLM = false;
    Enable_RQUOTA = false;
    Protocols = 4;
    RecoveryBackend = rados_cluster;
    Minor_Versions = 1,2;
    Dir_Chunk = 0;
    NParts = 1;
    Cache_Size = 1;
    pool = "rados_pool";
    namespace = "pool_namespace";
    nodeid = "fqdn"
    UserId = "cephx_user_id";
    Ceph_Conf = "path_to_ceph.conf"

You can find out the values for rados_pool and pool_namespace by checking the already existing line in the configuration of the form:

%url rados://rados_pool/pool_namespace/...

The value for nodeid option corresponds to the FQDN of the machine, and UserId and Ceph_Conf options value can be found in the already existing RADOS_URLS block.

Because legacy versions of NFS prevent us from lifting the grace period early and therefore prolong a server restart, we disable options for NFS prior to version 4.2. We also disable most of the NFS Ganesha caching as Ceph libraries do aggressive caching already.

The 'rados_cluster' recovery back-end stores its info in RADOS objects. Although it is not a lot of data, we want it highly available. We use the CephFS metadata pool for this purpose, and declare a new 'ganesha' namespace in it to keep it distinct from CephFS objects.

Note: Cluster Node IDs

Most of the configuration is identical between the two hosts, however the nodeid option in the 'RADOS_KV' block needs to be a unique string for each node. By default, NFS Ganesha sets nodeid to the host name of the node.

If you need to use different fixed values other than host names, you can for example set nodeid = 'a' on one node and nodeid = 'b' on the other one.

12.3.3 Populate the Cluster Grace Database Edit source

We need to verify that all of the nodes in the cluster know about each other. This done via a RADOS object that is shared between the hosts. NFS Ganesha uses this object to communicate the current state with regard to a grace period.

The nfs-ganesha-rados-grace package contains a command line tool for querying and manipulating this database. If the package is not installed on at least one of the nodes, install it with

root # zypper install nfs-ganesha-rados-grace

We will use the command to create the DB and add both nodeids. In our example, the two NFS Ganesha nodes are named ses6min1.example.com and ses6min2.example.com On one of the NFS Ganesha hosts, run

cephadm@adm > ganesha-rados-grace -p cephfs_metadata -n ganesha add ses6min1.example.com
cephadm@adm > ganesha-rados-grace -p cephfs_metadata -n ganesha add ses6min2.example.com
cephadm@adm > ganesha-rados-grace -p cephfs_metadata -n ganesha
cur=1 rec=0
ses6min1.example.com     E
ses6min2.example.com     E

This creates the grace database and adds both 'ses6min1.example.com' and 'ses6min2.example.com' to it. The last command dumps the current state. Newly added hosts are always considered to be enforcing the grace period so they both have the 'E' flag set. The 'cur' and 'rec' values show the current and recovery epochs, which is how we keep track of what hosts are allowed to perform recovery and when.

12.3.4 Restart NFS Ganesha Services Edit source

On both NFS Ganesha nodes, restart the related services:

root # systemctl restart nfs-ganesha.service

After the services are restarted, check the grace database:

cephadm@adm > ganesha-rados-grace -p cephfs_metadata -n ganesha
cur=3 rec=0
Note: Cleared the 'E' Flag

Note that both nodes have cleared their 'E' flags, indicating that they are no longer enforcing the grace period and are now in normal operation mode.

12.3.5 Conclusion Edit source

After you complete all the preceding steps, you can mount the exported NFS from either of the two NFS Ganesha servers, and perform normal NFS operations against them.

Our example configuration assumes that if one of the two NFS Ganesha servers goes down, you will restart it manually within 5 minutes. After 5 minutes, the Metadata Server may cancel the session that the NFS Ganesha client held and all of the state associated with it. If the session’s capabilities get cancelled before the rest of the cluster goes into the grace period, the server’s clients may not be able to recover all of their state.

12.4 More Information Edit source

More information can be found in Chapter 30, NFS Ganesha: Export Ceph Data via NFS.