5 Troubleshooting #
Abstract
This document describes situations which leads to issues during build or boot time of the image build with KIWI NG. The suggested solutions are considered best practice but are just one out of other possible solution candidates.
5.1 Build Host Constraints #
For building images a host system is required that runs the build process. Tools to create the image are used from that host and this creates an indirect dependency to the target image. For example; Building an Ubuntu image requires the apt and dpkg tools and metadata to be available and functional on the host to build an Ubuntu image. There are many more of those host vs. image dependencies and not all of them can be resolved in a clear and clean way.
The most compatible environment is provided if the build host is of the same distribution than the target image. In other cases our recommendation is that the build host is of the same distribution than the target and near to the major version (+-1) compared to the target. Such an environment can be found in:
The Open Build Service OBS.
The KIWI NG boxed plugin: Section 6.1, “Building in a Self-Contained Environment”
In general, our goal is to support any major distribution with KIWI NG. However for building images we rely on core tools which are not under our control. Also several design aspects of distributions like secure boot and working with upstream projects are different and not influenced by us. There are many side effects that can be annoying especially if the build host is not of the same distribution vendor than the image target.
5.1.1 Package Manager Behavior #
One of the key requirements for KIWI NG is that the target distribution
consists out of a collection of software packages provided through
software repositories. As of today this applies to every Linux
distribution in some shape or form. For the installation of this
software packages there are tools available called package manager
and KIWI NG implements an API for a variety of package managers to
support the different Linux distributions.
In KIWI NG the package manager tools are called in a way that allows for a non interactive installation of the packages specified in the image description. However, KIWI NG does intentionally not configure other features of the package manager to keep their distribution default behavior as much as possible. There are many options that can be set for a package manager to behave differently. This part of the troubleshooting exists to inform about the most common surprises or unexpected behavior of certain package managers and solutions how to address them.
All of the following information applies to package manager
calls performed inside of the image root tree. Meaning after
the image bootstrap phase. During the image bootstrap phase
all package manager calls happens on the build host and their
behavior can only be influenced by changing the setup of the
build host. An isolated build environment is required to
address issues in this area. See Section 6.1, “Building in a Self-Contained Environment” for
details.
5.1.1.1 zypper: modalias matching #
A module alias is an identifier that exists if the respective hardware was found by the kernel on this system. The package manager can use this information to match package(s) that holds a reference to the alias name and installs it automatically. For example: Automatically install a driver package if the respective hardware is present. In zypper this behavior is by default enabled. When building an image this feature can be unwanted and lead to interesting side effects. As it’s required to build the image on some host, there is also some hardware available during build. This must not necessarily be the same or compatible hardware the image is expected to run on later. For example: The buildhost has a nvidia graphics card. An image build process would install the nvidia driver package due to the modalias match. The image is expected to run on a system without a nvidia card. The build process would install unneeded software. To switch off modalias matching in zypper follow these steps:
Add a
post_bootstrap.shhook script to your image description with the following code:echo 'ZYPP_MODALIAS_SYSFS=""' > .kiwi.package_manager.envOptionally add a
config.shhook script to your image description with the following code:rm .kiwi.package_manager.envThis will delete the custom environment file such that it does not appear in the final image
Rebuild your image. The
kiwi.package_manager.envenvironment file is used by kiwi and added to the execution environment of the package manager. TheZYPP_MODALIAS_SYSFSvariable can be used to switch off the modalias matching. For more details on zypper setting please refer to: https://doc.opensuse.org/projects/libzypp/HEAD/zypp-envars.html
5.2 Architectures #
With regards to the information in Section 5.1, “Build Host Constraints” one requirement between the build host and the image when it comes to architecture support is, that the image architecture should match the build host architecture. Cross arch building would require any core tool that is used to build an image to be cross arch capable.
To patch e.g an x86_64 system such that it can build an aarch64 image would require some work on binutils and hacks as well as performance tweaks which is all not worth the effort and still can lead to broken results. Thus we recommend to provide native systems for the target architecture and build there. One possible alternative is to use the kiwi boxed plugin as mentioned above together with a box created for the desired architecture. However keep in mind the performance problematic when running a VM of a different architecture.
The majority of the image builds are based on the x86 architecture. As mentioned KIWI NG also supports other architectures, shown in the table below:
|
Architecture |
Supported |
|---|---|
|
x86_64 |
yes |
|
ix86 |
yes note:distro |
|
s390/s390x |
yes note:distro |
|
arm/aarch64 |
yes note:distro |
|
ppc64 |
yes note:distro |
5.3 Host Security Settings Conflicts with KIWI #
Abstract
This page provides further information how to solve
image build problems caused by selinux security
policies.
Linux systems are protected against write/read or other
operations depending on the application which wants to
access or modify data. The rules for this protection are
provided in security policies. There are several applications
enforcing these security settings, e.g apparmor or selinux.
In this troubleshooting chapter the focus is set on selinux
Protecting files, process groups, kernel filesystems, device nodes and more from unauthorized access and restrict it to a certain set of applications is a nice concept. However, if taken serious no other application except the ones configured in the security policy will function properly.
When building an appliance, the appliance builder has to have access to a wide range of services. It must be able to create a new package database elsewhere in the system. It must be able to create, read and write device nodes, create filesystems, partitions, bootloader configurations etc etc. The list is very long and no security policy could cover this in a way that it would not be open to everything which in the end leads to a pointless exercise and no security at all.
This means for users who would like to keep the security settings
of the system enforced and unchanged, the only way to allow KIWI NG
to do its job is to run it through boxbuild as explained in
Section 6.1, “Building in a Self-Contained Environment”
For users who can afford to open the system security policy, the following procedure will make KIWI NG to work:
sudo setenforce 0This action disables selinux temporary. To disable selinux permanently perform the following steps:
Open the SELinux configuration file:
/etc/selinux/configLocate the following line:
SELINUX=enforcingChange the value to disabled:
SELINUX=disabledOn the next reboot, SELinux is permanently disabled.
similar instructions applies to other application security
subsystems like apparmor. Due to the complexity of these
systems this article just mentions the most common issue
people run into when building images on systems protected
through selinux.
5.4 Incompatible Filesystem Settings on Host vs. Image #
Abstract
This page provides further information how to solve image boot problems if the filesystem tool chain on the image build host is incompatible with the image target distribution
When KIWI NG builds an image which requests the creation of a
filesystem, the required filesystem creation tool, for
example mkfs.xfs, is called from the host on which KIWI NG
gets called. It is expected that the generated filesystem
is compatible with the image target distribution. This
expectation is not always correct and depends on the
compatibility of the filesystem default settings between
build host and image target. We know about the following
settings that causes an incompatible filesystem which
will not be able to be used on boot:
- Ext[2,3,4]
Check
/etc/mke2fs.confon the build host and make sure the configuredinode_sizeis the same as the setting used for the target image. To solve an issue of this type use the following filesystem creation option in your KIWI NG image configuration:<type fscreateoptions="-I inode-size"/>- XFS
Check the XFS metadata setup on the build host and make sure the settings are compatible with the target image. XFS has the default settings compiled in, thus it might be needed to build the image first and use the
xfs_infotool in adisk.shscript to fetch the settings at build time of the image. We know from community reports that the settingsparse=1will cause issues on older versions of grub’s xfs module, which does not know how to handle this setting properly. To solve an issue of this type use the following filesystem creation option in your KIWI NG image configuration:<type fscreateoptions="-i sparse=0"/>- btrfs
btrfs and default page sizes (4k vs 64k). By default btrfs autodetects the sectorsize according to the used kernel page size. If the sectorsize differs from the page size, the created filesystem can not be mounted by the image target kernel. If there is a different kernel page size between the kernel on the system the image is build on and the later kernel used for the image, it’s required to specify the filesystem sector size to match with the kernel page size of the kernel used for the image. This can be done like in the following example:
<type fscreateoptions="--sectorsize 4k"/>
There can be more inconsistencies in the area of filesystems
which we haven’t listed here. In general it’s advisable to
build the image in a compatible environment. At best the
build host distribution is of the same major Linux version
than the image target. For this purpose KIWI NG provides the
so called boxed-plugin. Further details can be found
in Section 6.1, “Building in a Self-Contained Environment”
5.5 Boxbuild Tweaks #
Abstract
This document describes a few ways to modify box build VMs for testing/debugging.
5.5.1 Increase Box Build Image Size #
In particularly large builds, you may find that the upstream build boxes aren’t quite large enough, and fail to build during the final few steps. While it is a bit of a kludge, it is possible to increase the size of the build box.
To do so, follow these steps:
For this example, we will assume the box increase question is an Ubuntu box, located in ~/.kiwi_boxes/ubuntu
While the VM is offline, locate the VM you want to modify and resize with
qemu-img. Here we will increase the size by 20G. The VM will have to be told to utilize this space in the following steps.
$ qemu-img resize Ubuntu-Box.x86_64-1.22.04-System-BuildBox.qcow2 +20GWhen relaunching your
kiwi-ngbox build, make sure you use--no-snapshotand--box-debugoptions within your build command/script. Example:
$ kiwi --debug --profile="Disk" --type oem system boxbuild --no-snapshot \
--box-memory=32G --box-smp-cpus=16 --box-debug --box ubuntu -- \
--description ./ubuntu-jammy --target-dir /build/kiwi/outputs/When the build fails and drops you into the VM console, you will need to extend the partition of the VM rootfs, then resize with
resize2fs. In this example,partedwas used and the partition in question was /dev/vda3.
$ parted
# Can run parted print to check for relevant partitions if needed.
(parted) $ print
(parted) $ resizepart 3 100%
# Exit from parted
(parted) $ quit
# Run resize2fs to grow the filesystem to fill the space
$ resize2fs /dev/vda3From this point, depending on where your build failed, it may be possible to continue your build from inside the box, using the existing 9p mount points defined by your build command. Using the command above as an example,
/resultwithin the box maps up to/build/kiwi/outputson the host, and it’s possible to run
$ kiwi-ng --profile="Disk" --type oem system create \
--root=/result/build/image-root/ --target-dir=/resultIf the rebuild from within was successful, you can copy the files from
/resultto/bundle, from within the VM, where/bundlemaps to yourtarget-diron the host.