Applies to SUSE Enterprise Storage 7.1

8 Troubleshooting NFS Ganesha #

8.1 Debugging NFS Ganesha logs #

NFS Ganesha has the following log levels: NULL, FATAL, MAJ, CRIT, WARN, EVENT, INFO, DEBUG, MID_DEBUG, M_DBG, FULL_DEBUG and F_DBG.

Find the logs by running the following command:

cephuser@adm > cephadm logs --name nfs.SERVICE_ID.hostname

For example:

  INFO:cephadm:Inferring fsid e4d48df0-fc2a-11ea-8734-525400e03ff1 
  -- Logs begin at Mon 2020-09-21 18:29:25 CEST. -- 
  Sep 22 04:59:08 node1 bash[29705]: 22/09/2020 02:59:08 : epoch 5f6960b1 : node1 : ganesha.nfsd-1[svc_6] nfs_rpc_process_request :DISP :F_DBG :About to authenticate Prog=100003, vers=4, proc=1, xid=770234021, SVCXPRT=0x7> 
  Sep 22 04:59:08 node1 bash[29705]: 22/09/2020 02:59:08 : epoch 5f6960b1 : node1 : ganesha.nfsd-1[svc_6] nfs_rpc_process_request :DISP :F_DBG :Before SVCAUTH_CHECKSUM on SVCXPRT 0x7f081c0030c0 fd 53 
  Sep 22 04:59:08 node1 bash[29705]: 22/09/2020 02:59:08 : epoch 5f6960b1 : node1 : ganesha.nfsd-1[svc_6] export_check_access :RW LOCK :F_DBG :Got read lock on 0x7f0872a64660 (&export_opt_lock) at /home/abuild/rpmbuild/BU> 
  Sep 22 04:59:08 node1 bash[29705]: 22/09/2020 02:59:08 : epoch 5f6960b1 : node1 : ganesha.nfsd-1[svc_6] export_check_access :EXPORT :M_DBG :EXPORT_DEFAULTS (options=03303002/00080000 , , , > 
  Sep 22 04:59:08 node1 bash[29705]: 22/09/2020 02:59:08 : epoch 5f6960b1 : node1 : ganesha.nfsd-1[svc_6] export_check_access :EXPORT :M_DBG :default options (options=03303002/ffffffff root_squash , ----, 34-, UDP, TCP,> 
  Sep 22 04:59:08 node1 bash[29705]: 22/09/2020 02:59:08 : epoch 5f6960b1 : node1 : ganesha.nfsd-1[svc_6] export_check_access :EXPORT :M_DBG :Final options (options=03303002/ffffffff root_squash , ----, 34-, UDP, TCP,> 
  Sep 22 04:59:08 node1 bash[29705]: 22/09/2020 02:59:08 : epoch 5f6960b1 : node1 : ganesha.nfsd-1[svc_6] export_check_access :RW LOCK :F_DBG :Unlocked 0x7f0872a64660 (&export_opt_lock) at /home/abuild/rpmbuild/BUILD/nfs-> 
  Sep 22 04:59:08 node1 bash[29705]: 22/09/2020 02:59:08 : epoch 5f6960b1 : node1 : ganesha.nfsd-1[svc_6] get_gsh_client :RW LOCK :F_DBG :Got read lock on 0x7f0872a5e5d0 (&client_by_ip.lock) at /home/abuild/rpmbuild/BUILD> 
  Sep 22 04:59:08 node1 bash[29705]: 22/09/2020 02:59:08 : epoch 5f6960b1 : node1 : ganesha.nfsd-1[svc_6] get_gsh_client :HT CACHE :DEBUG :client_mgr cache hit slot 859

8.1.1 Setting the default log level #

To change the default log level, you can use a RADOS object to add a LOG configuration block.

To enable debug logging, change the log level to FULL_DEBUG to increase the log verbosity. Keep in mind that this can produce a large amount of log data and adversely affect performance.

For example:

  LOG {
      # The components block contains one or more logging components
      # and the setting to be used.
      components {
          # The ALL component is special. When set it defines the level
          # for all components and overrides any other setting in this block.
          ALL = FULL_DEBUG; # this will likely kill performance
      }
  }

Put the file into a RADOS configuration object. In this example, the object is named debugconf-nfs.foo:

cephuser@adm > rados --pool nfs-ganesha --namespace foo put debugconf-nfs.foo debugconf-nfs.foo

Add a reference to the debug configuration object in the RADOS common configuration file:

%url "rados://nfs-ganesha/foo/export-1" 
%url "rados://nfs-ganesha/foo/userconf-nfs-foo" 
%url "rados://nfs-ganesha/foo/debugconf-nfs.foo"

Put the updated common configuration file into the RADOS common configuration object. In this example, the common configuration object is named conf-nfs.foo:

cephuser@adm > rados --pool nfs-ganesha --namespace foo put conf-nfs.foo conf-nfs.foo

Once both configuration objects are placed into the RADOS recovery pool, notify the NFS Ganesha daemons of the change:

cephuser@adm > rados --pool nfs-ganesha --namespace foo notify conf-nfs.foo conf-nfs.foo 
reply client.35746 cookie 94143722285376 : 0 bytes 
reply client.35749 cookie 94618743368000 : 0 bytes

8.2 Changing the default port #

To change the default port for NFS Ganesha you can use a user defined RADOS configuration object to add an NFS_CORE_PARAM block.

Set the NFS_Port configuration value to the port you want to change. This example uses port 12345.

  NFS_CORE_PARAM { 
       Enable_NLM = false; 
       Enable_RQUOTA = false; 
       Protocols = 4; 
       NFS_Port = 12345; 
  }

Put the file into a user defined RADOS configuration object. In this example, the object is named userconf-nfs.foo:

cephuser@adm > rados --pool nfs-ganesha --namespace foo put userconf-nfs.foo userconf-nfs.foo

Add a reference to the parameter in your RADOS common configuration file:

%url "rados://nfs-ganesha/foo/export-1" 
%url "rados://nfs-ganesha/foo/userconf-nfs.foo"

Note

This action is similar to how exports are included in the common configuration file.

Put the updated common configuration file into the RADOS common configuration object. In this example, the common configuration object is named conf-nfs.foo:

cephuser@adm > rados --pool nfs-ganesha --namespace foo put conf-nfs.foo conf-nfs.foo

Once both configuration objects are placed into the RADOS recovery pool, restart the NFS Ganesha daemons using the Ceph orchestrator:

cephuser@adm > ceph orch restart nfs.foo 
restart nfs.foo.node1 from host 'node1' 
restart nfs.foo.node3 from host 'node3'