Keg is a tool which helps to create and manage image descriptions for use with the KIWI appliance builder. A KIWI image description consists of a single XML document that specifies type, configuration, and content of the image to build. Optionally there can be configuration scripts and overlay archives added to an image description, which allow for further configuration and additional content.

Since KIWI image descriptions are monolithic, maintaining a number of image descriptions that have considerable overlap with respect to content and setup can be cumbersome and error-prone. Keg attempts to alleviate that by allowing image descriptions to be broken into modules. Those modules can be composed in different ways in so called image definitions, and modules can inherit from parent modules which allows for fine-tuning for specific image setups. Configuration scripts and overlay archives can also be generated in a modular fashion.

The collection of source data required for keg to produce image descriptions is called recipes. Keg recipes are typically kept in a git repository, and keg has support for producing change logs from git commit history, but this is not a requirement. A recipes repository provides keg with the information how an image description is to be composed as well as the content of the components.

The basic principle of operation is that when keg is executed, it is pointed to a directory within the recipes repository and it reads any YAML files in that directory and any parent directories and merges their contents into a dictionary. How the image definition data is structured and composed is not relevant, as long as the resulting dictionary represents a valid image definition. This allows for a lot of flexibility in the layout of a recipes repository. The SUSE Public Cloud Keg Recipes repository provides an example of a highly modular one with strong use of inheritance.

For more details on what constitutes a recipes repository, see section Chapter 4, Recipes basics (ff).

To create an image description, keg needs to be installed, as well as KIWI, as the latter is used by keg to validate the final image description. See Chapter 2, Installation for information about how to install keg, and KIWI Installation about how to install KIWI.

Additionally, a recipes repository is required. The following example uses the aforementioned SUSE Public Cloud keg recipes:

$ git clone https://github.com/SUSE-Enceladus/keg-recipes.git

$ mkdir sles15-sp4-byos

$ keg --recipes-root keg-recipes --dest-dir sles15-sp4-byos \
      cross-cloud/sles/byos/15-sp4

After the keg command completes the destination directory specified with --dest-dir contains a description for a SUSE Linux Enterprise Server 15 SP4 image for use in the Public Clouds. It can be processed with KIWI to build an image. For more details about KIWI image descriptions, see https://osinside.github.io/kiwi/image_description.html.

Recipes used to generate an image description can be spread over multiple repositories. For that purpose, the --recipes-root command line argument may be given multiple times, with each one specifying a different recipes repository. Repositories will be searched in the order they are specified, and for any dictionary key, config scriptlet, or overlay archive module that exists in multiple repositories, the one that is read last will be used.

Using multiple repositories for recipes can be useful in some situations. For example, if some parts of recipes data are public and some private, they can be kept in different repositories. It could also be used to base recipes on an upstream repository and only maintain additional image definitions or modifications in a separate repository.

Keg also provides support for producing image descriptions for use with the Open Build Service. It can generate _multibuild files that are required by OBS for image descriptions with multiple profiles, and it comes with an OBS Source Service plug-in for automating generating image descriptions. See Chapter 8, Keg OBS source service for details.

Keg is available for various openSUSE distributions from the Cloud:Tools repository in the openSUSE Build Service. Package name is python[py_version]-kiwi-keg.

Keg can be obtained from the Python Package Index (PyPi) via Python’s package manager pip:

$ pip install kiwi_keg

Essentially, a keg recipes repository conists of three top-level directories which contain different types of configuration data. Those three are:

See Chapter 6, Data modules for details on data modules.

Keg uses Jinja2 templates to produce the headers for config.sh and images.sh. Both are optional and keg will write a fallback header if they are missing. Additionally, a Jinja2 template can be used to generate config.kiwi instead of using the internal XML generator.

This section contains some general information about how keg handles its source data.

An image description is internally represented by a data dictionary with a certain structure. This dictionary gets composed by parsing source image definition and data files referenced by the image definition and merging them into a dictionary.

Image definitions as well as data modules are used by referencing a directory (under images or data respectively), which may be several layers of directories under the root directory. When parsing those, keg will also read any .yaml file that is in a directory above the referenced one, and merge all source data into one dictionary, with the lower (i.e. more specific) layers taking precedence over upper (i.e. more generic) ones. This inheritance mechanism is intended to reduce data duplication.

Keg uses namespaces in the image definition to group certain bits of information (for instance, a list of packages) which can be overwritten in derived modules, allowing for creating specialized versions of data modules for specific use case or different image description versions.

Once everything is merged, the resulting dictionary is validated against the image definition schema, to ensure its structure is correct and all required keys are present. If that is the case, keg runs the image dictionary through its XML generator to produce a config.kiwi file. In case the image definition contains configuration scripts or overlay archives specifications, keg will generate those as well.

An image definition dictionary is composed of several parts that define different parts of the image. The actual image description, configuration scripts, overlay archives. All parts are defined under a top-level key in the dictionary. There are additional top-level keys that affect data parsing and generator selection.

The top-level keys are as follows:

image:
  _attributes:
    schemaversion: "<schema_maj>.<schema_min>"
    name: <image_name>
    displayname: <image_boot_title>
  description:
    _attributes:
      type: <system_type>
    author: <author_name>
    contact: <author_email>
  preferences:
    - version: <version_string>
    - _attributes:
        profiles:
          - <profile_name>
          ...
      type:
        _attributes:
          image: <image_type>
            kernelcmdline:
            <kernel_param>: <kernel_param_value>
            ...
          ...
        size:
          _attributes:
            unit: <size_unit>
          _text: <disk_size>
    ...
  users:
    user:
      - _attributes:
          name: <user_name>
          groups: <user_groups>
          home: <user_home>
          password: <user_password>
      ...
  packages:
    - _attributes:
        type: image|bootstrap
        profiles:
          - <profile>
          ...
      archive:
        _attributes:
          name: <archive_filename>
      <namespace>:
        package:
          - _attributes:
              name: <package_name>
              arch: <package_arch>
          ...
      ...
    ...
  profiles:
    profile:
      - _attributes:
          name: <profile_name>
          description: <profile_description>
      ...

type:
  _attributes:
    image: vmx
    kernelcmdline:
      console: ttyS0
      debug: []

<type image="vmx" kernelcmdline="console=ttyS0 debug"/>

packages:
  _map_attribute: name
  _namespace_some_pkgs:
    package:
      - pkg1
      - pkg2

packages:
  _namespace_some_pkgs:
  package:
    - _attribute:
        name: pkg1
    - _attribute:
        name: pkg1
  archive:
    - _attributes:
        name: archive1.tar.gz

5.1.2 config #

 

The config dictionary defines the content of the config.sh file keg should generate. config.sh is a script that KIWI runs during the image prepare step and can be used to modify the image’s configuration. The config dictionary structure is as follows:

config:
  - profiles:
      - <profile_name>
      ...
    files:
      <namespace>:
        - path: <file>
          append: bool (defaults to False if missing)
          content: string
        ...
      ...
    scripts:
      <namespace>:
        - <script>
        ...
      ...
    services:
      <namespace>:
        - <service_name>
        - name: <service_name>
          enable: bool
        ...
      ...
    sysconfig:
      <namespace>:
        - file: <sysconfig_file>
          name: <sysconfig_variable>
          value: string
        ...
      ...
  ...

Each list item in config produces a section in config.sh, with the optional profiles key defining for which image profile that section should apply. Each item can have the following keys (all are optional, but there has to be at least one):

files defines files that should be created (or overwritten if existing) with the given content or have content appended to in config.sh.

scripts defines which scriptlets should be included. <script> refers to a file data/scripts/<script>.sh in the recipes tree.

services defines which systemd services and timers should be enabled or disabled in the image. The short version (just a string) means the string is the service name and it should be enabled.

defines which existing sysconfig variables should the altered.

Note

<namespace> defines a namespace with the same purpose as in the image dictionary, but config namespaces don’t have to start with _, but are allowed to.

5.1.4 archive #

 

The archive dictionary defines the content of overlay tar archives, that can be included in the image via the archive sub-section of the packages section of the image dictionary. The structure is as follows:

archive:
  - name: <archive_filename>
    <namespace>:
      _include_overlays:
        - <overlay_module>
        ...
  ...

When generating the image description, keg will produce a tar archive for each entry in archive with the given file name, with its contents being composed of all files that are in the listed overlay modules. Each module references a directory in data/overlayfiles.

Keg automatically compresses the archive based on the file name extension. Supported are gz, , xz, or no extension for uncompressed archive.

Note

The archive name root.tar (regardless of compression extension) is automatically included in all profiles (if there are any) by KIWI. It is not necessary to include it explicitly in the image definition.

Keg supports importing parts of the image definition from other directory trees within the recipes to allow for modularization. For that purpose, a key in the image dictionary may have a sub-key called _include. Its value is a list of strings, each of which points to a directory in the data sub-directory of the recipes root. To process the instruction, keg generates another dictionary from all YAML files in the referenced directory trees (the same mechanism as when parsing the images tree applies). It then looks up the key in that dictionary that is equal to the parent key of the _include key, and replaces the _include key with its contents. That means, if the _include statement is below a key called packages, only data under packages in the include dictionary will be copied into the image definition dictionary. This allows for having different types of configuration data in the same directory and including them in different places in the image definition. See Chapter 6, Data modules for details on data modules.

There are three additional optional top-level image definition sections that affect how the image definition dictionary is composed and the image description is generated:

5.3.2 image-config-comments #

 

This section allows to add top-level comments in the produced KIWI file. The format is as follows:

image-config-comments:
  <comment_name>: <comment>
  ...

<comment_name> is just a name and is not included in the generated output. Comments can be used to include arbitrary information in the image description. Some comments have a special meaning for processing image descriptions by the Open Build Service, for instance the OBS-Profiles directive that is required to process multi-profile image descriptions. See https://osinside.github.io/kiwi/working_with_images/build_in_buildservice.html for details.

Note

Keg generates some comments automatically. In case the image definition has multiple profiles and the --disable-multibuild command line switch is not set, it will add an OBS-Profiles: @BUILD_FLAVOR@ comment. In case the image description is generated for one or more specific architectures via the -a command line option, the apprpriate OBS-ExclusiveArch comment is added.

xmlfiles:
  - name: <filename>
    content:
      <content_dictionary>
  ...

5.3.4 schema #

 

Keg starting with version 2.0.0 has an internal XML generator to produce KIWI image descriptions. Previously, a Jinja2 template was used to convert the image dictionary that keg constructed into a KIWI image description. Using a Jinja2 template is still supported and can be configured as follows in the image definition:

schema: <template>

In this case, instead of running the XML generator, keg would read the file <template>.kiwi.templ from the schemas directory in the recipes root directory and run it trough the Jinja2 engine.

Note

While using a Jinja2 template would in theory allow to operate on different input data structures, the internal schema validator requires the image definition to comply with what keg expects.

Any directory under data that is not scripts or overlayfiles is considered an image definition data module and may be included in the main image definition using the _include statement.

Inheritance rules apply similarly to the image definition tree, but additionally, keg supports sub-versions of data modules. This can be used for instance to create slightly different versions of modules for use with different image versions while still sharing most of the image definition between those versions.

For this purpose, keg supports the include-paths directive in the image definition. Include paths are paths that get appended to any source path and those get scanned for input files as well. See the following image definition as an example:

include-paths:
  leap15/1
  leap15/2
image:
  preferences:
    - _include:
        - base/common
  packages:
    - _include:
        - base/common
  config:
    - _include:
        - base/common

This tells keg, when adding data from directory data/base/common to the image data dictionary, to also look into sub directories leap15/2, leap15/1, and leap15 (through inheritance). This would lead to the following directories being scanned:

data
data/common
data/common/base
data/common/base/leap15
data/common/base/leap15/1
data/common/base/leap15/2

This allows for example to put generic configuration bits in data/common/base, Leap 15 specific configuration in data/common/base/leap15, and adjust the configuration for minor versions, if necessary.

When merging the included dictionaries into the main dictionary, keg only copies the dictionary under the top level key that matches the key under which the _include statement is. That means, assuming the YAML files collected from the above trees resulted in the following data structure:

preferences:
  locale: en_US
  timezone: UTC
  type:
    _attributes:
      firmware: efi
      image: vmx
packages:
  _namespace_base_packages:
    package:
      - bash
      - glibc
      - kernel-default
config:
  _namespace_base_services:
    services:
      - sshd

Would result in a data structure like this:

include-paths:
  leap15/1
  leap15/2
image:
  preferences:
    locale: en_US
    timezone: UTC
    type:
      _attributes:
        firmware: efi
        image: vmx
  packages:
    _namespace_base_packages:
      package:
        - bash
        - glibc
        - kernel-default
config:
  _namespace_base_services:
    services:
      - sshd

Merging based on the parent key allows for grouping of different types of configuration data in one data module.

Configuration scriptlets are individual script snippets that can be used to generate image configuration scripts. KIWI runs those scripts at certain points in the image build process. They can be used to do changes to the system’s configuration.

The scriptlets are located in data/scripts and are required to have a .sh suffix. These are referenced in the scripts lists of the config or setup sections in the image definition (without the .sh suffix). See Section 5.1.2, “config” for details on the config section.

KIWI image descriptions can contain optional overlay archives, which will be extracted into the system’s root directory before the image is created. Overlay files are located in sub-directories in data/overlayfiles, with each sub-directory representing an overlay files module. Any directory structure under the module’s top directory is preserved.

Overlay files modules can be referenced in the archive section of the image definition using the _include_overlays directive. See Section 5.1.4, “archive” for details.

With source info tracking enabled, keg will write one or more source info files in addition to the image description in the output directory. In case the image description at hand is single-build, a single file log_sources is written, in case it is multi-build, a file log_sources_PROFILE is written for each profile. This allows for generating individual change logs for the resulting image binaries.

The source info logs contain detailed information about which bits from the keg-recipes tree was used to generate the image description. The source info log files will contain several lines of the following format:

root:/path/to/repository
range:start:end:/path/to/repository/file
/path/to/repository/file_or_dir

The first line specifies the repository location. There will be one for each keg-recipes directory given to keg. Lines starting with range: specify a part of a file in a repository. This is used to track the source location of each key that was in the final image dictionary. The third line format simply specifies a file or a directory in the repository that was used in the image description, and is used for configuration script snippets and overlay files.

This enables the change log generator to produce a change log using the git commit history, selecting only commits that apply to the generated image description.

The generated source info log files, together with the keg-recipes in the place and state they were used to generate the image description, can be used to generate change logs. The keg distribution contains a tool generate_recipes_changelog for that purpose. When called with a source log file as argument, generate_recipes_changelog will use the source information to select matching git messages and produce a change log in chronological order. There are parameters to to narrow down the applicable commit range as well as some formatting options. Refer to Section 3.2, “generate_recipes_changelog” command overview for details.

The keg distribution contains a module for integrating with the Open Build Service, an implementation of a so-called OBS Source Service. It supports automatic handling of change log generation. See Chapter 8, Keg OBS source service for details.