Administration Guide

48.1 Finding and gathering information #

Linux reports things in a very detailed way. There are several places to look when you encounter problems with your system, most of which are standard to Linux systems in general, and some are relevant to SUSE Linux Enterprise Server systems. Most log files can be viewed with YaST (Miscellaneous › Start-Up Log).

YaST offers the possibility to collect all system information needed by the support team. Use Other › Support and select the problem category. When all information is gathered, attach it to your support request.

A list of the most frequently checked log files follows with the description of their typical purpose. Paths containing ~ refer to the current user's home directory.

Table 48.1: Log files #

Log File	Description
`~/.xsession-errors`	Messages from the desktop applications currently running.
`/var/log/apparmor/`	Log files from AppArmor, see Part V, “Confining privileges with AppArmor” for detailed information.
`/var/log/audit/audit.log`	Log file from Audit to track any access to files, directories, or resources of your system, and trace system calls. See Part VII, “The Linux Audit Framework” for detailed information.
`/var/log/mail.*`	Messages from the mail system.
`/var/log/NetworkManager`	Log file from NetworkManager to collect problems with network connectivity
`/var/log/samba/`	Directory containing Samba server and client log messages.
`/var/log/warn`	All messages from the kernel and system log daemon with the “warning” level or higher.
`/var/log/wtmp`	Binary file containing user login records for the current machine session. View it with `last`.
`/var/log/Xorg.*.log`	Start-up and runtime log files from the X Window System. It is useful for debugging failed X start-ups.
`/var/log/YaST2/`	Directory containing YaST's actions and their results.
`/var/log/zypper.log`	Log file of Zypper.

Apart from log files, your machine also supplies you with information about the running system. See Table 48.2: System information with the /proc file system

Table 48.2: System information with the /proc file system #

File	Description
`/proc/cpuinfo`	Contains processor information, including its type, make, model, and performance.
`/proc/dma`	Shows which DMA channels are currently being used.
`/proc/interrupts`	Shows which interrupts are in use, and how many of each have been in use.
`/proc/iomem`	Displays the status of I/O (input/output) memory.
`/proc/ioports`	Shows which I/O ports are in use at the moment.
`/proc/meminfo`	Displays memory status.
`/proc/modules`	Displays the individual modules.
`/proc/mounts`	Displays devices currently mounted.
`/proc/partitions`	Shows the partitioning of all hard disks.
`/proc/version`	Displays the current version of Linux.

Apart from the /proc file system, the Linux kernel exports information with the sysfs module, an in-memory file system. This module represents kernel objects, their attributes and relationships. For more information about sysfs, see the context of udev in Chapter 29, Dynamic kernel device management with udev. Table 48.3 contains an overview of the most common directories under /sys.

Table 48.3: System information with the /sys file system #

File	Description
`/sys/block`	Contains subdirectories for each block device discovered in the system. Generally, these are mostly disk type devices.
`/sys/bus`	Contains subdirectories for each physical bus type.
`/sys/class`	Contains subdirectories grouped together as a functional types of devices (like graphics, net, printer, etc.)
`/sys/device`	Contains the global device hierarchy.

Linux comes with several tools for system analysis and monitoring. See Chapter 2, System monitoring utilities for a selection of the most important ones used in system diagnostics.

Each of the following scenarios begins with a header describing the problem followed by a paragraph or two offering suggested solutions, available references for more detailed solutions, and cross-references to other scenarios that are related.

48.2 Boot problems #

Boot problems are situations when your system does not boot properly (does not boot to the expected target and login screen).

48.2.1 The GRUB 2 boot loader fails to load #

If the hardware is functioning properly, it is possible that the boot loader is corrupted and Linux cannot start on the machine. In this case, it is necessary to repair the boot loader. To do so, you need to start the Rescue System as described in Section 48.5.2, “Using the rescue system” and follow the instructions in Section 48.5.2.4, “Modifying and re-installing the boot loader”.

Alternatively, you can use the Rescue System to fix the boot loader as follows. Boot your machine from the installation media. In the boot screen, choose More › Boot Linux System. Select the disk containing the installed system and kernel with the default kernel options.

When the system is booted, start YaST and switch to System › Boot Loader. Make sure that the Write generic Boot Code to MBR option is enabled, and click OK. This fixes the corrupted boot loader by overwriting it, or installs the boot loader if it is missing.

Other reasons for the machine not booting may be BIOS-related:

BIOS settings: Check your BIOS for references to your hard disk. GRUB 2 may simply not be started if the hard disk itself cannot be found with the current BIOS settings.
BIOS boot order: Check whether your system's boot order includes the hard disk. If the hard disk option was not enabled, your system may install properly, but fails to boot when access to the hard disk is required.

48.2.2 No graphical login #

If the machine starts, but does not boot into the graphical login manager, anticipate problems either with the choice of the default systemd target or the configuration of the X Window System. To check the current systemd default target run the command sudo systemctl get-default. If the value returned is not graphical.target, run the command sudo systemctl isolate graphical.target. If the graphical login screen starts, log in and start YaST › System › Services Manager and set the Default System Target to Graphical Interface. From now on the system should boot into the graphical login screen.

If the graphical login screen does not start even if having booted or switched to the graphical target, your desktop or X Window software is probably misconfigured or corrupted. Examine the log files at /var/log/Xorg.*.log for detailed messages from the X server as it attempted to start. If the desktop fails during start, it may log error messages to the system journal that can be queried with the command journalctl (see Chapter 21, journalctl: Query the systemd journal for more information). If these error messages hint at a configuration problem in the X server, try to fix these issues. If the graphical system still does not come up, consider reinstalling the graphical desktop.

48.2.3 Root Btrfs partition cannot be mounted #

If a btrfs root partition becomes corrupted, try the following options:

Mount the partition with the -o recovery option.
If that fails, run btrfs-zero-log on your root partition.

48.2.4 Force checking root partitions #

If the root partition becomes corrupted, use the parameter forcefsck on the boot prompt. This passes the option -f (force) to the fsck command.

48.2.5 Disable swap to enable booting #

When a swap device is not available and the system cannot enable it during boot, booting may fail. Try disabling all swap devices by appending the following options to the kernel command line:

systemd.device_wants_unit=off systemd.mask=swap.target

You may also try disabling specific swap devices:

systemd.mask=dev-sda1.swap

48.2.6 GRUB 2 fails during reboot on a dual-boot system #

If GRUB 2 fails during reboot, disable the Fast Boot setting in the BIOS.

48.3 Login problems #

Login problems occur when your system refuses to accept the user name and password, or accepts them but then fails to start the graphic desktop, produces errors, or drops to a command line, for example.

48.3.1 Valid user name and password combinations fail #

This often occurs when the system is configured to use network authentication or directory services and cannot retrieve results from its configured servers. The root user is the only local user that can still log in to these machines. The following are common reasons a machine appears functional but cannot process logins correctly:

The network is not working. For further directions on this, turn to Section 48.4, “Network problems”.
DNS is not working at the moment (which prevents GNOME from working and the system from making validated requests to secure servers). One indication that this is the case is that the machine takes a long time to respond to any action. Find more information about this topic in Section 48.4, “Network problems”.
If the system is configured to use Kerberos, the system's local time may have drifted past the accepted variance with the Kerberos server time (this is typically 300 seconds). If NTP (network time protocol) is not working properly or local NTP servers are not working, Kerberos authentication ceases to function because it depends on common clock synchronization across the network.
The system's authentication configuration is misconfigured. Check the PAM configuration files involved for any typographical errors or misordering of directives. For additional background information about PAM and the syntax of the configuration files involved, refer to Chapter 2, Authentication with PAM.
The home partition is encrypted. Find more information about this topic in Section 48.3.3, “Login to encrypted home partition fails”.

In cases that do not involve external network problems, the solution is to log in as root and repair the configuration. If you cannot log in to the running system, reboot it into the rescue mode as outlined in Procedure 18.3, “Entering rescue mode”.

48.3.2 Valid user name and password not accepted #

This is by far the most common problem users encounter, because there are many reasons this can occur. Depending on whether you use local user management and authentication or network authentication, login failures occur for different reasons.

Local user management can fail for the following reasons:

The user may have entered the wrong password.
The user's home directory containing the desktop configuration files is corrupted or write protected.
There may be problems with the X Window System authenticating this particular user, especially if the user's home directory has been used with another Linux distribution before installing the current one.

To locate the reason for a local login failure, proceed as follows:

Check whether the user remembered their password correctly before you start debugging the whole authentication mechanism. If the user may have not have remembered their password correctly, use the YaST User Management module to change the user's password. Pay attention to the Caps Lock key and unlock it, if necessary.
Log in as root and check the system journal with journalctl -e for error messages of the login process and of PAM.
Try to log in from a console (using Ctrl–Alt–F1). If this is successful, the blame cannot be put on PAM, because it is possible to authenticate this user on this machine. Try to locate any problems with the X Window System or the GNOME desktop. For more information, refer to Section 48.3.4, “GNOME desktop has issues”.
If the user's home directory has been used with another Linux distribution, remove the Xauthority file in the user's home. Use a console login via Ctrl–Alt–F1 and run rm .Xauthority as this user. This should eliminate X authentication problems for this user. Try graphical login again.
If the desktop could not start because of corrupt configuration files, proceed with Section 48.3.4, “GNOME desktop has issues”.

In the following, common reasons a network authentication for a particular user may fail on a specific machine are listed:

The user may have entered the wrong password.
The user name exists in the machine's local authentication files and is also provided by a network authentication system, causing conflicts.
The home directory exists but is corrupt or unavailable. Perhaps it is write protected or is on a server that is inaccessible at the moment.
The user does not have permission to log in to that particular host in the authentication system.
The machine has changed host names, for whatever reason, and the user does not have permission to log in to that host.
The machine cannot reach the authentication server or directory server that contains that user's information.
There may be problems with the X Window System authenticating this particular user, especially if the user's home has been used with another Linux distribution before installing the current one.

To locate the cause of the login failures with network authentication, proceed as follows:

Check whether the user remembered their password correctly before you start debugging the whole authentication mechanism.
Determine the directory server which the machine relies on for authentication and make sure that it is up and running and properly communicating with the other machines.
Determine that the user's user name and password work on other machines to make sure that their authentication data exists and is properly distributed.
See if another user can log in to the misbehaving machine. If another user can log in without difficulty or if root can log in, log in and examine the system journal with the journalctl -e > file. Locate the time stamps that correspond to the login attempts and determine if PAM has produced any error messages.
Try to log in from a console (using Ctrl–Alt–F1). If this is successful, the problem is not with PAM or the directory server on which the user's home is hosted, because it is possible to authenticate this user on this machine. Try to locate any problems with the X Window System or the GNOME desktop. For more information, refer to Section 48.3.4, “GNOME desktop has issues”.
If the user's home directory has been used with another Linux distribution, remove the Xauthority file in the user's home. Use a console login via Ctrl–Alt–F1 and run rm .Xauthority as this user. This should eliminate X authentication problems for this user. Try graphical login again.
If the desktop could not start because of corrupt configuration files, proceed with Section 48.3.4, “GNOME desktop has issues”.

48.3.3 Login to encrypted home partition fails #

It is recommended to use an encrypted home partition for laptops. If you cannot log in to your laptop, the reason might be that your partition could not be unlocked.

During the boot time, you need to enter the passphrase to unlock your encrypted partition. If you do not enter it, the boot process continues, leaving the partition locked.

To unlock your encrypted partition, proceed as follows:

Switch to the text console with Ctrl–Alt–F1.
Become root.
Restart the unlocking process again with:
```
# systemctl restart home.mount
```
Enter your passphrase to unlock your encrypted partition.
Exit the text console and switch back to the login screen with Alt–F7.
Log in as usual.

48.3.4 GNOME desktop has issues #

If you are experiencing issues with the GNOME desktop, there are several ways to troubleshoot the misbehaving graphical desktop environment. The recommended procedure described below offers the safest option to fix a broken GNOME desktop.

Procedure 48.1: Troubleshooting GNOME #

Launch YaST and switch to Security and Users.
Open the User and Group Management dialog and click Add.
Fill out the required fields and click OK to create a new user.
Log out and log in as the new user. This gives you a fresh GNOME environment.
Copy individual subdirectories from the ~/.local/ and ~/.config/ directories of the old user account to the respective directories of the new user account.
Log out and log in again as the new user after every copy operation to check whether GNOME still works correctly.
Repeat the previous step until you find the configuration file that breaks GNOME.
Log in as the old user, and move the offending configuration file to a different location. Log out and log in again as the old user.
Delete the previously created user.

48.4 Network problems #

Many problems of your system may be network-related, even though they do not seem to be at first. For example, the reason for a system not allowing users to log in may be a network problem of some kind. This section introduces a simple checklist you can apply to identify the cause of any network problem encountered.

Procedure 48.2: How to identify network problems #

When checking the network connection of your machine, proceed as follows:

If you use an Ethernet connection, check the hardware first. Make sure that your network cable is properly plugged into your computer and router (or hub, etc.). The control lights next to your Ethernet connector are normally both be active.
If the connection fails, check whether your network cable works with another machine. If it does, your network card causes the failure. If hubs or switches are included in your network setup, they may be faulty, as well.
If using a wireless connection, check whether the wireless link can be established by other machines. If not, contact the wireless network's administrator.
When you have checked your basic network connectivity, try to find out which service is not responding. Gather the address information of all network servers needed in your setup. Either look them up in the appropriate YaST module or ask your system administrator. The following list gives some typical network servers involved in a setup together with the symptoms of an outage.
DNS (name service)
A broken or malfunctioning name service affects the network's functionality in many ways. If the local machine relies on any network servers for authentication and these servers cannot be found because of name resolution issues, users would not even be able to log in. Machines in the network managed by a broken name server would not be able to “see” each other and communicate.
NTP (time service)
A malfunctioning or completely broken NTP service could affect Kerberos authentication and X server functionality.
NFS (file service)
If any application needs data stored in an NFS mounted directory, it cannot start or function properly if this service was down or misconfigured. In the worst case scenario, a user's personal desktop configuration would not come up if their home directory containing the .gconf subdirectory could not be found because of a faulty NFS server.
Samba (file service)
If any application needs data stored in a directory on a faulty Samba server, it cannot start or function properly.
NIS (user management)
If your SUSE Linux Enterprise Server system relies on a faulty NIS server to provide the user data, users cannot log in to this machine.
LDAP (user management)
If your SUSE Linux Enterprise Server system relies on a faulty LDAP server to provide the user data, users cannot log in to this machine.
Kerberos (authentication)
Authentication will not work and login to any machine fails.
CUPS (network printing)
Users cannot print.
Check whether the network servers are running and whether your network setup allows you to establish a connection:
Important: Limitations
The debugging procedure described below only applies to a simple network server/client setup that does not involve any internal routing. It assumes both server and client are members of the same subnet without the need for additional routing.
1. Use ping IP_ADDRESS/HOSTNAME (replace with the host name or IP address of the server) to check whether each one of them is up and responding to the network. If this command is successful, it tells you that the host you were looking for is up and running and that the name service for your network is configured correctly.
  If ping fails with destination host unreachable, either your system or the desired server is not properly configured or down. Check whether your system is reachable by running ping IP address or YOUR_HOSTNAME from another machine. If you can reach your machine from another machine, it is the server that is not running or not configured correctly.
  If ping fails with unknown host, the name service is not configured correctly or the host name used was incorrect. For further checks on this matter, refer to Step 4.b. If ping still fails, either your network card is not configured correctly or your network hardware is faulty.
2. Use host HOSTNAME to check whether the host name of the server you are trying to connect to is properly translated into an IP address and vice versa. If this command returns the IP address of this host, the name service is up and running. If the host command fails, check all network configuration files relating to name and address resolution on your host:
  /var/run/netconfig/resolv.conf
  This file is used to keep track of the name server and domain you are currently using. It is a symbolic link to /run/netconfig/resolv.conf and is usually automatically adjusted by YaST or DHCP. Make sure that this file has the following structure and all network addresses and domain names are correct:
  search FULLY_QUALIFIED_DOMAIN_NAME nameserver IPADDRESS_OF_NAMESERVER
  This file can contain more than one name server address, but at least one of them must be correct to provide name resolution to your host. If needed, adjust this file using the YaST Network Settings module (Hostname/DNS tab).
  If your network connection is handled via DHCP, enable DHCP to change host name and name service information by selecting Set Hostname via DHCP (can be set globally for any interface or per interface) and Update Name Servers and Search List via DHCP in the YaST Network Settings module (Hostname/DNS tab).
  /etc/nsswitch.conf
  This file tells Linux where to look for name service information. It should look like this:
  ... hosts: files dns networks: files dns ...
  The dns entry is vital. It tells Linux to use an external name server. Normally, these entries are automatically managed by YaST, but it would be prudent to check.
  If all the relevant entries on the host are correct, let your system administrator check the DNS server configuration for the correct zone information. For detailed information about DNS, refer to Chapter 39, The domain name system. If you have made sure that the DNS configuration of your host and the DNS server are correct, proceed with checking the configuration of your network and network device.
3. If your system cannot establish a connection to a network server and you have excluded name service problems from the list of possible culprits, check the configuration of your network card.
  Use the command ip addr show NETWORK_DEVICE to check whether this device was properly configured. Make sure that the inet address with the netmask (/MASK) is configured correctly. An error in the IP address or a missing bit in your network mask would render your network configuration unusable. If necessary, perform this check on the server as well.
4. If the name service and network hardware are properly configured and running, but certain external network connections still get long timeouts or fail entirely, use traceroute FULLY_QUALIFIED_DOMAIN_NAME (executed as root) to track the network route these requests are taking. This command lists any gateway (hop) that a request from your machine passes on its way to its destination. It lists the response time of each hop and whether this hop is reachable. Use a combination of traceroute and ping to track down the culprit and let the administrators know.

When you have identified the cause of your network trouble, you can resolve it yourself (if the problem is located on your machine) or let the system administrators of your network know about your findings so they can reconfigure the services or repair the necessary systems.

48.4.1 NetworkManager problems #

If you have a problem with network connectivity, narrow it down as described in Procedure 48.2, “How to identify network problems”. If NetworkManager seems to be the culprit, proceed as follows to get logs providing hints on why NetworkManager fails:

Open a shell and log in as root.
Restart the NetworkManager:
```
> sudo systemctl restart NetworkManager
```
Open a Web page, for example, http://www.opensuse.org as normal user to see, if you can connect.
Collect any information about the state of NetworkManager in /var/log/NetworkManager.

For more information about NetworkManager, refer to Chapter 31, Using NetworkManager.

48.5 Data problems #

Data problems are when the machine may or may not boot properly but, in either case, it is clear that there is data corruption on the system and that the system needs to be recovered. These situations call for a backup of your critical data, enabling you to recover the system state from before your system failed.

48.5.1 Managing partition images #

Sometimes you need to perform a backup from an entire partition or even hard disk. Linux comes with the dd tool which can create an exact copy of your disk. Combined with gzip you save some space.

Procedure 48.3: Backing up and restoring hard disks #

Start a Shell as user root.
Select your source device. Typically this is something like /dev/sda (labeled as SOURCE).
Decide where you want to store your image (labeled as BACKUP_PATH). It must be different from your source device. In other words: if you make a backup from /dev/sda, your image file must not to be stored under /dev/sda.

Run the commands to create a compressed image file:

# dd if=/dev/SOURCE | gzip > /BACKUP_PATH/image.gz

Restore the hard disk with the following commands:

# gzip -dc /BACKUP_PATH/image.gz | dd of=/dev/SOURCE

If you only need to back up a partition, replace the SOURCE placeholder with your respective partition. In this case, your image file can lie on the same hard disk, but on a different partition.

48.5.2 Using the rescue system #

There are several reasons a system could fail to come up and run properly. A corrupted file system following a system crash, corrupted configuration files, or a corrupted boot loader configuration are the most common ones.

To help you to resolve these situations, SUSE Linux Enterprise Server contains a rescue system that you can boot. The rescue system is a small Linux system that can be loaded into a RAM disk and mounted as root file system, allowing you to access your Linux partitions from the outside. Using the rescue system, you can recover or modify any important aspect of your system.

Manipulate any type of configuration file.
Check the file system for defects and start automatic repair processes.
Access the installed system in a “change root” environment.
Check, modify, and re-install the boot loader configuration.
Recover from a badly installed device driver or unusable kernel.
Resize partitions using the parted command. Find more information about this tool at the GNU Parted Web site http://www.gnu.org/software/parted/parted.html.

The rescue system can be loaded from various sources and locations. The simplest option is to boot the rescue system from the original installation medium.

Note: IBM Z: starting the rescue system

On IBM Z the installation system can be used for rescue purposes. To start the rescue system follow the instructions in Section 48.6, “IBM Z: using initrd as a rescue system”.

Insert the installation medium into your DVD drive.
Reboot the system.
At the boot screen, press F4 and choose DVD-ROM. Then choose Rescue System from the main menu.
Enter root at the Rescue: prompt. A password is not required.

If your hardware setup does not include a DVD drive, you can boot the rescue system from a network source. The following example applies to a remote boot scenario—if using another boot medium, such as a DVD, modify the info file accordingly and boot as you would for a normal installation.

Enter the configuration of your PXE boot setup and add the lines install=PROTOCOL://INSTSOURCE and rescue=1. If you need to start the repair system, use repair=1 instead. As with a normal installation, PROTOCOL stands for any of the supported network protocols (NFS, HTTP, FTP, etc.) and INSTSOURCE for the path to your network installation source.
Boot the system using “Wake on LAN”, as described in Section 17.5, “Using wake-on-LAN for remote wakeups”.
Enter root at the Rescue: prompt. A password is not required.

When you have entered the rescue system, you can use the virtual consoles that can be reached with Alt–F1 to Alt–F6.

A shell and other useful utilities, such as the mount program, are available in the /bin directory. The /sbin directory contains important file and network utilities for reviewing and repairing the file system. This directory also contains the most important binaries for system maintenance, such as fdisk, mkfs, mkswap, mount, and shutdown, ip and ss for maintaining the network. The directory /usr/bin contains the vi editor, find, less, and SSH.

To see the system messages, either use the command dmesg or view the system log with journalctl.

48.5.2.1 Checking and manipulating configuration files #

As an example for a configuration that might be fixed using the rescue system, imagine you have a broken configuration file that prevents the system from booting properly. You can fix this using the rescue system.

To manipulate a configuration file, proceed as follows:

Start the rescue system using one of the methods described above.
To mount a root file system located under /dev/sda6 to the rescue system, use the following command:
```
> sudo mount /dev/sda6 /mnt
```
All directories of the system are now located under /mnt
Change the directory to the mounted root file system:
```
> sudo cd /mnt
```
Open the problematic configuration file in the vi editor. Adjust and save the configuration.
Unmount the root file system from the rescue system:
```
> sudo umount /mnt
```
Reboot the machine.

48.5.2.2 Repairing and checking file systems #

Generally, file systems cannot be repaired on a running system. If you encounter serious problems, you may not even be able to mount your root file system and the system boot may end with a “kernel panic”. In this case, the only way is to repair the system from the outside. The system contains the fsck utility to check and repair multiple file system types, such as ext2, ext3, ext4, msdos, and vfat. Use the -t option to specify which file system to check.

The following command checks all ext4 file systems found in the /etc/fstab specification:

> sudo fsck -t ext4 -A

Tip

For Btrfs, you can use the btrfs check command found in the btrfsprogs package.

Find topics about the Btrfs file system in the following places:

The Storage Administration Guide includes https://documentation.suse.com/sles/html/SLES-all/cha-filesystems.html#sec-filesystems-major-btrfs and https://documentation.suse.com/sles/15-SP5/html/SLES-all/cha-resize-fs.html#sec-resize-fs-btrfs sections.
The following article https://www.suse.com/support/kb/doc/?id=000018769 describes how to recover from Btrfs errors.
The following article includes links to multiple Btrfs related topics https://www.suse.com/support/kb/doc/?id=000018779.
The man 8 btrfs-check man page details all options of the btrfs check command.

48.5.2.3 Accessing the installed system #

If you need to access the installed system from the rescue system, you need to do this in a change root environment. For example, to modify the boot loader configuration, or to execute a hardware configuration utility.

To set up a change root environment based on the installed system, proceed as follows:

Tip: Import LVM volume groups
If you are using an LVM setup (refer to Part II, “Logical volumes (LVM)” for more general details), import all existing volume groups to be able to find and mount the device(s):
```
rootvgimport -a
```
Run lsblk to check which node corresponds to the root partition. It is /dev/sda2 in our example:
```
> lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda           8:0    0 149,1G  0 disk
├─sda1        8:1    0     2G  0 part  [SWAP]
├─sda2        8:2    0    20G  0 part  /
└─sda3        8:3    0   127G  0 part
  └─cr_home 254:0    0   127G  0 crypt /home
```
Mount the root partition from the installed system:
```
> sudo mount /dev/sda2 /mnt
```

Mount /proc, /dev, and /sys partitions:

> sudo mount -t proc none /mnt/proc
> sudo mount --rbind /dev /mnt/dev
> sudo mount --rbind /sys /mnt/sys

Now you can “change root” into the new environment, keeping the bash shell:
```
> chroot /mnt /bin/bash
```
Finally, mount the remaining partitions from the installed system:
```
> mount -a
```
Now you have access to the installed system. Before rebooting the system, unmount the partitions with umount -a and leave the “change root” environment with exit.

Warning: Limitations

Although you have full access to the files and applications of the installed system, there are some limitations. The kernel that is running is the one that was booted with the rescue system, not with the change root environment. It only supports essential hardware and it is not possible to add kernel modules from the installed system unless the kernel versions are identical. Always check the version of the currently running (rescue) kernel with uname -r and then find out if a matching subdirectory exists in the /lib/modules directory in the change root environment. If yes, you can use the installed modules, otherwise you need to supply their correct versions on other media, such as a flash disk. Most often the rescue kernel version differs from the installed one — then you cannot simply access a sound card, for example. It is also not possible to start a graphical user interface.

Also note that you leave the “change root” environment when you switch the console with Alt–F1 to Alt–F6.

48.5.2.4 Modifying and re-installing the boot loader #

Sometimes a system cannot boot because the boot loader configuration is corrupted. The start-up routines cannot, for example, translate physical drives to the actual locations in the Linux file system without a working boot loader.

To check the boot loader configuration and re-install the boot loader, proceed as follows:

Perform the necessary steps to access the installed system as described in Section 48.5.2.3, “Accessing the installed system”.
Check that the GRUB 2 boot loader is installed on the system. If not, install the package grub2 and run
```
> sudo grub2-install /dev/sda
```
Check whether the following files are correctly configured according to the GRUB 2 configuration principles outlined in Chapter 18, The boot loader GRUB 2 and apply fixes if necessary.
- /etc/default/grub
- /boot/grub2/device.map
- /boot/grub2/grub.cfg (this file is generated, do not edit)
- /etc/sysconfig/bootloader
Re-install the boot loader using the following command sequence:
```
> sudo grub2-mkconfig -o /boot/grub2/grub.cfg
```
Unmount the partitions, log out of the “change root” environment, and reboot the system:
```
> umount -a
exit
reboot
```

48.5.2.5 Fixing kernel installation #

A kernel update may introduce a new bug which can impact the operation of your system. For example a driver for a piece of hardware in your system may be faulty, which prevents you from accessing and using it. In this case, revert to the last working kernel (if available on the system) or install the original kernel from the installation media.

Tip: How to keep last kernels after update

To prevent failures to boot after a faulty kernel update, use the kernel multiversion feature and tell libzypp which kernels you want to keep after the update.

For example to always keep the last two kernels and the currently running one, add

multiversion.kernels = latest,latest-1,running

to the /etc/zypp/zypp.conf file. See Chapter 27, Installing multiple kernel versions for more information.

A similar case is when you need to re-install or update a broken driver for a device not supported by SUSE Linux Enterprise Server. For example when a hardware vendor uses a specific device, such as a hardware RAID controller, which needs a binary driver to be recognized by the operating system. The vendor typically releases a Driver Update Disk (DUD) with the fixed or updated version of the required driver.

In both cases you need to access the installed system in the rescue mode and fix the kernel related problem, otherwise the system may fail to boot correctly:

Boot from the SUSE Linux Enterprise Server installation media.
If you are recovering after a faulty kernel update, skip this step. If you need to use a driver update disk (DUD), press F6 to load the driver update after the boot menu appears, and choose the path or URL to the driver update and confirm with Yes.
Choose Rescue System from the boot menu and press Enter. If you chose to use DUD, you will be asked to specify where the driver update is stored.
Enter root at the Rescue: prompt. A password is not required.
Manually mount the target system and “change root” into the new environment. For more information, see Section 48.5.2.3, “Accessing the installed system”.
If using DUD, install/re-install/update the faulty device driver package. Always make sure the installed kernel version exactly matches the version of the driver you are installing.
If fixing faulty kernel update installation, you can install the original kernel from the installation media with the following procedure.
1. Identify your DVD device with hwinfo --cdrom and mount it with mount /dev/sr0 /mnt.
2. Navigate to the directory where your kernel files are stored on the DVD, for example cd /mnt/suse/x86_64/.
3. Install required kernel-*, kernel-*-base, and kernel-*-extra packages of your flavor with the rpm -i command.
Update configuration files and reinitialize the boot loader if needed. For more information, see Section 48.5.2.4, “Modifying and re-installing the boot loader”.
Remove any bootable media from the system drive and reboot.

48.6 IBM Z: using initrd as a rescue system #

If the kernel of the SUSE® Linux Enterprise Server for IBM Z is upgraded or modified, it is possible to reboot the system accidentally in an inconsistent state, so standard procedures of IPLing the installed system fail. In such a case, you may use the installation system for rescue purposes.

IPL the SUSE Linux Enterprise Server for IBM Z installation system as described in Section 5.3, “Preparing for installation”. Choose Start Installation and enter all required parameters. After the installation system has loaded and you are asked which display type to use to control the installation, select SSH. Now you can log in to the system with SSH as root without a password.

In this state, no disks are configured. You need to configure them before you can proceed.

Procedure 48.4: Configuring DASDs #

Configure DASDs with the following command:
```
dasd_configure 0.0.0150 1 0
```
0.0.0150 is the channel to which the DASD is connected. The 1 means activate the disk (a 0 at this place would deactivate the disk). The 0 stands for “no DIAG mode” for the disk (a 1 here would enable DAIG access to the disk).
Now the DASD is online (check with cat /proc/partitions) and can used for subsequent commands.

Procedure 48.5: Configuring a zFCP disk #

To configure a zFCP disk, it is necessary to first configure the zFCP adapter. Do this with the following command:
```
zfcp_host_configure 0.0.4000 1
```
0.0.4000 is the channel to which the adapter is attached and 1 stands for activate (a 0 here would deactivate the adapter).
After the adapter is activated, a disk can be configured. Do this with the following command:
```
zfcp_disk_configure 0.0.4000 1234567887654321 8765432100000000  1
```
0.0.4000 is the previously-used channel ID, 1234567887654321 is the WWPN (World wide Port Number), and 8765432100000000 is the LUN (logical unit number). The 1 stands for activating the disk (a 0 here would deactivate the disk).
Now the zFCP disk is online (check with cat /proc/partitions) and can used for subsequent commands.

Now the rescue system is fully set up and you can start repairing the installed system. See Section 48.5.2, “Using the rescue system” for instructions on how to repair the most common issues.

48.7 IBM Z: After a kernel update the system boots into the previous kernel #

Installing a new kernel version on an IBM Z system does not automatically update the "stage 1" zipl loader. This means that after a restart the system boots into the old kernel. And if secure boot is enabled, booting fails when the old kernel is signed with a signing key that has been retracted by, for example, a shim update at the same time.

To solve the problem, refresh zipl to make it aware of the new kernel version. To do this, run the following command after you install the new kernel:

grub2-emu --kexec

In the grub2 boot menu, choose the new kernel to reboot. Run the command above the second time for the changes to take effect. Finally, run the following command to reinstall the boot loader:

update-bootloader --reinit