Managing Modular KIWI Image Descriptions with Keg #

Keg is available for various openSUSE distributions from the Cloud:Tools repository in the openSUSE Build Service. The package name is pythonpy_version-kiwi-keg.

Keg can be obtained from PyPI via Python’s package manager pip:

> pip install kiwi_keg

keg [options] <source>

The Keg tool helps create and manage image descriptions suitable for the KIWI appliance builder. While Keg can handle single definitions, it provides no considerable advantage in such use cases. Its strategic advantage lies in managing multiple image descriptions with significant content overlap.

Keg requires source data called recipes, which provides all the information necessary for Keg to create KIWI image descriptions. See Section 5, “Structuring Keg recipes for image definitions” for more information about recipes.

The recipes used for generating SUSE Public Cloud image descriptions can be found in the Public Cloud Keg Recipes repository.

> git clone https://github.com/SUSE-Enceladus/keg-recipes.git
> keg --recipes-root keg-recipes --dest-dir leap_description leap/jeos/15.2

generate_recipes_changelog [options] <logfile>

generate_recipes_changelog generates a changelog from the git commit history of one or more keg-recipes repositories. The input file is a source info log that is generated by Keg when run with source tracking enabled (-s command line switch).

The exit status is 0 if a changelog was generated successfully, 1 in case an error occurred, or 2 if no error occurred but generated changelog is empty.

> generate_recipes_changelog -r /path/to/repo:12345678.. -t 1.1.8 log_sources

This will produce a YAML changelog, namespaced with 1.1.8, intended as a version number, only considering commits after hash 12345678. If 12345678 was the hash of the HEAD commit in the checked-out branch at /PATH/TO/REPO at the previous run of generate_recipes_changelog on the same data set, this would limit the changelog to only contain newer changes. This method can be used to produce incremental change logs.

The changelog will have the following format:

1.1.8:
  - change: git message subject
    date: git commit UTC timestamp
    details: |-
      git message body
  ...

Essentially, a keg recipes repository consists of three top-level directories, which contain different types of configuration data. Those three are:

An image description is internally represented by a data dictionary with a certain structure. This dictionary is composed by parsing source image definition and data files referenced by the image definition and merging them into a dictionary.

Image definitions and data modules are used by referencing a directory (under images or data respectively), which may be several layers of directories under the root directory. When parsing those, Keg will also read any .yaml file that is in a directory above the referenced one, and merge all source data into one dictionary, with the lower (i.e. more specific) layers taking precedence over upper (i.e. more generic) ones. This inheritance mechanism is intended to reduce data duplication.

Keg uses namespaces in the image definition to group certain bits of information (for instance, a list of packages) which can be overwritten in derived modules, allowing for the creation of specialized versions of data modules for specific use case or different image description versions.

Once everything is merged, the resulting dictionary is validated against the image definition schema to ensure its structure is correct and all required keys are present. If that is the case, Keg runs the image dictionary through its XML generator to produce a config.kiwi file. In case the image definition contains configuration scripts or overlay archive specifications, Keg will generate those as well.

An image definition dictionary is composed of several parts that define different parts of the image: the actual image description, configuration scripts and overlay archives. All parts are defined under a top-level key in the dictionary. There are additional top-level keys that affect data parsing and generator selection.

The top-level keys are as follows:

image:
  _attributes:
    schemaversion: "SCHEMA_MAJ.SCHEMA_MIN"
    name: IMAGE_NAME
    displayname: IMAGE_BOOT_TITLE
  description:
    _attributes:
      type: SYSTEM_TYPE
    author: AUTHOR_NAME
    contact: AUTHOR_EMAIL
  preferences:
    - version: VERSION_STRING
    - _attributes:
        profiles:
          - PROFILE_NAME
          ...
      type:
        _attributes:
          image: IMAGE_TYPE
            kernelcmdline:
            KERNEL_PARAM: KERNEL_PARAM_VALUE
            ...
          ...
        size:
          _attributes:
            unit: SIZE_UNIT
          _text: DISK_SIZE
    ...
  users:
    user:
      - _attributes:
          name: USER_NAME
          groups: USER_GROUPS
          home: USER_HOME
          password: USER_PASSWORD
      ...
  packages:
    - _attributes:
        type: image|bootstrap
        profiles:
          - PROFILE
          ...
      archive:
        _attributes:
          name: ARCHIVE_FILENAME
      NAMESPACE:
        package:
          - _attributes:
              name: PACKAGE_NAME
              arch: PACKAGE_ARCH
          ...
      ...
    ...
  profiles:
    profile:
      - _attributes:
          name: PROFILE_NAME
          description: PROFILE_DESCRIPTION
      ...

6.1.2 config #

 

The config dictionary defines the content of the config.sh file Keg should generate. config.sh is a script that KIWI runs during the image preparation step and can be used to modify the image’s configuration. The config dictionary structure is as follows:

config:
  - profiles:
      - PROFILE_NAME
      ...
    files:
      NAMESPACE:
        - path: FILE
          append: bool
          content: string
        ...
      ...
    scripts:
      NAMESPACE:
        - SCRIPT
        ...
      ...
    services:
      NAMESPACE:
        - SERVICE_name
        - name: SERVICE_name
          enable: bool
        ...
      ...
    sysconfig:
      NAMESPACE:
        - file: SYSCONFIG_FILE
          name: SYSCONFIG_VARIABLE
          value: string
        ...
      ...
  ...

Each list item in config produces a section in config.sh, with the optional profiles key defining for which image profile that section should apply. Each item can have the following keys (all are optional, but there has to be at least one):

files

Defines files that should be created (or overwritten if existing) with the given content or have content appended to in config.sh (default on append is false).

scripts

Defines which scriptlets should be included. script refers to a file data/scripts/SCRIPT.sh in the recipes tree.

services

Defines which systemd services and timers should be enabled or disabled in the image. The short version (just a string) means the string is the service name and it should be enabled.

sysconfig

Defines which existing sysconfig variables should be altered.

Note

NAMESPACE defines a namespace with the same purpose as in the image dictionary, but config namespaces do not have to start with _ (although this would be allowed, too).

6.1.4 archive #

 

The archive dictionary defines the content of overlay tar archives that can be included in the image via the archive sub-section of the packages section of the image dictionary. The structure is as follows:

archive:
  - name: archive_filename
    namespace:
      _include_overlays:
        - overlay_module
        ...
  ...

When generating the image description, Keg will produce a tar archive for each entry in archive with the given file name, with its contents being composed of all files that are in the listed overlay modules. Each module references a directory in data/overlayfiles.

Keg automatically compresses the archive based on the file name extension. Supported are gz, , xz, or no extension for uncompressed archive.

Note

The archive name root.tar (regardless of compression extension) is automatically included in all profiles (if there are any) by KIWI. It is not necessary to include it explicitly in the image definition.

Keg supports importing parts of the image definition from other directory trees within the recipes to allow for modularization. For that purpose, a key in the image dictionary may have a subkey called _include. Its value is a list of strings, each of which points to a directory in the data subdirectory of the recipes root. To process the instruction, Keg generates another dictionary from all YAML files in the referenced directory trees (the same mechanism as when parsing the images tree applies). It then looks up the key in that dictionary that is equal to the parent key of the _include key, and replaces the _include key with its contents. That means, if the _include statement is below a key called packages, only data under packages in the include dictionary will be copied into the image definition dictionary. This allows for having different types of configuration data in the same directory and including them in different places in the image definition. See Section 7, “Using data modules for modular configuration” for details on data modules.

There are three additional optional top-level image definition sections that affect how the image definition dictionary is composed and the image description is generated:

6.3.2 image-config-comments #

 

The image-config-comments element allows you to add top-level comments to the generated KIWI file. The format is as follows:

image-config-comments:
  COMMENT_NAME: string
  ...

COMMENT_NAME is just a name and is not included in the generated output. Comments can be used to include arbitrary information in the image description. Some comments have a special meaning for processing image descriptions by the Open Build Service; for instance, the OBS-Profiles directive that is required to process multi-profile image descriptions. See https://osinside.github.io/kiwi/working_with_images/build_in_buildservice.html for details.

Note

Keg generates some comments automatically. If the image definition has multiple profiles and the --disable-multibuild command-line switch is not set, it will add an OBS-Profiles: @BUILD_FLAVOR@ comment. If the image description is generated for one or more specific architectures via the -a command-line option, the appropriate OBS-ExclusiveArch comment is added.

xmlfiles:
  - name: FILENAME
    content:
      CONTENT_DICTIONARY
  ...

6.3.4 schema #

 

Keg, starting with version 2.0.0, has an internal XML generator to produce KIWI image descriptions. Previously, a Jinja2 template was used to convert the image dictionary that Keg constructed into a KIWI image description. Using a Jinja2 template is still supported and can be configured as follows in the image definition:

schema: TEMPLATE

In this case, instead of running the XML generator, Keg would read the file TEMPLATE.kiwi.templ from the schemas directory in the recipes root directory and run it through the Jinja2 engine.

Note

While using a Jinja2 template would in theory allow to operate on different input data structures, the internal schema validator requires the image definition to comply with what Keg expects.

Any directory under data that is not scripts or overlayfiles is considered an image definition data module and may be included in the main image definition using the _include statement.

Inheritance rules apply similarly to the image definition tree, but Keg supports subversions of data modules. This can be used, for instance, to create slightly different versions of modules for use with different image versions while still sharing most of the image definition between those versions.

For this purpose, Keg supports the include-paths directive in the image definition. Include paths are paths that get appended to any source path, and these paths get scanned for input files as well. See the following image definition as an example:

include-paths:
  leap15/1
  leap15/2
image:
  preferences:
    - _include:
        - base/common
  packages:
    - _include:
        - base/common
  config:
    - _include:
      - base/common

This tells Keg, when adding data from directory data/base/common to the image data dictionary, to also look into subdirectories leap15/2, leap15/1, and leap15 (through inheritance). This would lead to the following directories being scanned:

data
data/common
data/common/base
data/common/base/leap15
data/common/base/leap15/1
data/common/base/leap15/2

This allows, for example, to put generic configuration bits in data/common/base, Leap 15 specific configuration in data/common/base/leap15, and adjust the configuration for minor versions.

When merging the included dictionaries into the main dictionary, Keg only copies the dictionary under the top-level key that matches the key under which the _include statement is. That means, assuming the YAML files collected from the above trees resulted in the following data structure:

preferences:
  locale: en_US
  timezone: UTC
  type:
    _attributes:
      firmware: efi
      image: vmx
packages:
  _namespace_base_packages:
    package:
      - bash
      - glibc
      - kernel-default
config:
  _namespace_base_services:
    services:
      - sshd

Would result in a data structure like this:

include-paths:
  leap15/1
  leap15/2
image:
  preferences:
    locale: en_US
    timezone: UTC
    type:
      _attributes:
        firmware: efi
        image: vmx
  packages:
    _namespace_base_packages:
      package:
        - bash
        - glibc
        - kernel-default
config:
  _namespace_base_services:
    services:
      - sshd

Merging based on the parent key allows for grouping different types of configuration data in one data module.

Configuration scriptlets are individual script snippets that can be used to generate image configuration scripts. KIWI runs those scripts at certain points in the image build process. They can be used to change the system’s configuration.

The scriptlets are located in data/scripts and are required to have a .sh suffix. These are referenced in the scripts list of the config or setup sections in the image definition (without the .sh suffix). See Section 6.1.2, “config” for details on the config section.

KIWI image descriptions can contain optional overlay archives, which will be extracted into the system’s root directory before the image is created. Overlay files are located in subdirectories in data/overlayfiles, with each subdirectory representing an overlay files module. Any directory structure under the module’s top directory is preserved.

Overlay files modules can be referenced in the archive section of the image definition using the _include_overlays directive. See Section 6.1.4, “archive” for details.

With source info tracking enabled, Keg will write one or more source info files in addition to the image description in the output directory. If the image description at hand is single-build, a single file log_sources is written. If it is multi-build, a file log_sources_PROFILE is written for each profile. This allows for generating individual changelogs for the resulting image binaries.

The source info logs contain detailed information about which bits from the keg-recipes tree were used to generate the image description. The source info log files will contain several lines of the following format:

root:/path/to/repository
range:start:end:/path/to/repository/file
/path/to/repository/file_or_dir

The first line specifies the repository location. There will be one for each keg-recipes directory given to Keg. Lines starting with range: specify a part of a file in a repository. This is used to track the source location of each key that was in the final image dictionary. The third line format simply specifies a file or a directory in the repository that was used in the image description, and is used for configuration script snippets and overlay files.

This enables the changelog generator to produce a changelog using the git commit history, selecting only commits that apply to the generated image description.

The generated source info log files, together with the keg-recipes in the place and state they were used to generate the image description, can be used to generate changelogs. The Keg distribution contains the tool generate_recipes_changelog for that purpose. When called with a source log file as argument, generate_recipes_changelog will use the source information to select matching git messages and produce a changelog in chronological order. There are parameters to narrow down the applicable commit range and some formatting options. Refer to Section 4.2, “The generate_recipes_changelog command” command overview for details.

The Keg distribution contains a module for integrating with the Open Build Service, an implementation of a so-called OBS Source Service. It supports automatic changelog generation. See Section 9, “Automating image descriptions with the OBS Source Service” for details.