Keg Reference Guide #

Keg is available for various openSUSE distributions from the Cloud:Tools repository in the openSUSE Build Service. Package name is pythonpy_version-kiwi-keg.

Keg can be obtained from the Python Package Index (PyPi) via Python’s package manager pip:

> pip install kiwi_keg

keg [options] <source>

Keg is a tool which helps to create and manage image descriptions suitable for the KIWI appliance builder. While Keg can be used to manage a single image definition the tool provides no considerable advantage in such a use case. The primary use case for Keg are situations where many image descriptions must be managed and the image descriptions have considerable overlap with respect to content and setup.

Keg requires source data called recipes which provides all information necessary for Keg to create KIWI image descriptions. See Section 5, “Recipes basics” for more information about recipes.

The recipes used for generating SUSE Public Cloud image descriptions can be found in the Public Cloud Keg Recipes repository.

> git clone https://github.com/SUSE-Enceladus/keg-recipes.git
> keg --recipes-root keg-recipes --dest-dir leap_description leap/jeos/15.2

generate_recipes_changelog [options] <logfile>

generate_recipes_changelog generates a change log from the git commit history of one or more keg-recipes repositories. The input file is a source info log that is generated by Keg when run with source tracking enabled (-s command line switch).

The exit status is 0 in case a change log was generated successfully, 1 in case an error occurred, or 2 in case no error occurred but generated change log is empty.

> generate_recipes_changelog -r /path/to/repo:12345678.. -t 1.1.8 log_sources

This will produce a YAML change log, namespaced with 1.1.8, intended as a version number, only considering commits after hash 12345678. If 12345678 was the hash of the HEAD commit in the checked out branch at /PATH/TO/REPO at the previous run of generate_recipes_changelog on the same data set, this would limit the change log to only contain newer changes. This method can be used to produce incremental change logs.

The change log will have the following format:

1.1.8:
  - change: git message subject
    date: git commit UTC timestamp
    details: |-
      git message body
  ...

Essentially, a keg recipes repository conists of three top-level directories which contain different types of configuration data. Those three are:

This section contains some general information about how Keg handles its source data.

An image description is internally represented by a data dictionary with a certain structure. This dictionary gets composed by parsing source image definition and data files referenced by the image definition and merging them into a dictionary.

Image definitions as well as data modules are used by referencing a directory (under images or data respectively), which may be several layers of directories under the root directory. When parsing those, Keg will also read any .yaml file that is in a directory above the referenced one, and merge all source data into one dictionary, with the lower (i.e. more specific) layers taking precedence over upper (i.e. more generic) ones. This inheritance mechanism is intended to reduce data duplication.

Keg uses namespaces in the image definition to group certain bits of information (for instance, a list of packages) which can be overwritten in derived modules, allowing for creating specialized versions of data modules for specific use case or different image description versions.

Once everything is merged, the resulting dictionary is validated against the image definition schema, to ensure its structure is correct and all required keys are present. If that is the case, Keg runs the image dictionary through its XML generator to produce a config.kiwi file. In case the image definition contains configuration scripts or overlay archives specifications, Keg will generate those as well.

An image definition dictionary is composed of several parts that define different parts of the image. The actual image description, configuration scripts, overlay archives. All parts are defined under a top-level key in the dictionary. There are additional top-level keys that affect data parsing and generator selection.

The top-level keys are as follows:

image:
  _attributes:
    schemaversion: "SCHEMA_MAJ.SCHEMA_MIN"
    name: IMAGE_NAME
    displayname: IMAGE_BOOT_TITLE
  description:
    _attributes:
      type: SYSTEM_TYPE
    author: AUTHOR_NAME
    contact: AUTHOR_EMAIL
  preferences:
    - version: VERSION_STRING
    - _attributes:
        profiles:
          - PROFILE_NAME
          ...
      type:
        _attributes:
          image: IMAGE_TYPE
            kernelcmdline:
            KERNEL_PARAM: KERNEL_PARAM_VALUE
            ...
          ...
        size:
          _attributes:
            unit: SIZE_UNIT
          _text: DISK_SIZE
    ...
  users:
    user:
      - _attributes:
          name: USER_NAME
          groups: USER_GROUPS
          home: USER_HOME
          password: USER_PASSWORD
      ...
  packages:
    - _attributes:
        type: image|bootstrap
        profiles:
          - PROFILE
          ...
      archive:
        _attributes:
          name: ARCHIVE_FILENAME
      NAMESPACE:
        package:
          - _attributes:
              name: PACKAGE_NAME
              arch: PACKAGE_ARCH
          ...
      ...
    ...
  profiles:
    profile:
      - _attributes:
          name: PROFILE_NAME
          description: PROFILE_DESCRIPTION
      ...

6.1.2 config #

 

The config dictionary defines the content of the config.sh file Keg should generate. config.sh is a script that KIWI runs during the image prepare step and can be used to modify the image’s configuration. The config dictionary structure is as follows:

config:
  - profiles:
      - PROFILE_NAME
      ...
    files:
      NAMESPACE:
        - path: FILE
          append: bool
          content: string
        ...
      ...
    scripts:
      NAMESPACE:
        - SCRIPT
        ...
      ...
    services:
      NAMESPACE:
        - SERVICE_name
        - name: SERVICE_name
          enable: bool
        ...
      ...
    sysconfig:
      NAMESPACE:
        - file: SYSCONFIG_FILE
          name: SYSCONFIG_VARIABLE
          value: string
        ...
      ...
  ...

Each list item in config produces a section in config.sh, with the optional profiles key defining for which image profile that section should apply. Each item can have the following keys (all are optional, but there has to be at least one):

files

Defines files that should be created (or overwritten if existing) with the given content or have content appended to in config.sh (default on append is false>).

scripts

Defines which scriptlets should be included. script refers to a file data/scripts/SCRIPT.sh in the recipes tree.

services

Defines which systemd services and timers should be enabled or disabled in the image. The short version (just a string) means the string is the service name and it should be enabled

sysconfig

Defines which existing sysconfig variables should the altered.

Note

NAMESPACE defines a namespace with the same purpose as in the image dictionary, but config namespaces does not have to start with _ (although this would be allowed, too).

6.1.4 archive #

 

The archive dictionary defines the content of overlay tar archives, that can be included in the image via the archive sub-section of the packages section of the image dictionary. The structure is as follows:

archive:
  - name: archive_filename
    namespace:
      _include_overlays:
        - overlay_module
        ...
  ...

When generating the image description, Keg will produce a tar archive for each entry in archive with the given file name, with its contents being composed of all files that are in the listed overlay modules. Each module references a directory in data/overlayfiles.

Keg automatically compresses the archive based on the file name extension. Supported are gz, , xz, or no extension for uncompressed archive.

Note

The archive name root.tar (regardless of compression extension) is automatically included in all profiles (if there are any) by KIWI. It is not necessary to include it explicitly in the image definition.

Keg supports importing parts of the image definition from other directory trees within the recipes to allow for modularization. For that purpose, a key in the image dictionary may have a sub-key called _include. Its value is a list of strings, each of which points to a directory in the data sub-directory of the recipes root. To process the instruction, Keg generates another dictionary from all YAML files in the referenced directory trees (the same mechanism as when parsing the images tree applies). It then looks up the key in that dictionary that is equal to the parent key of the _include key, and replaces the _include key with its contents. That means, if the _include statement is below a key called packages, only data under packages in the include dictionary will be copied into the image definition dictionary. This allows for having different types of configuration data in the same directory and including them in different places in the image definition. See Section 7, “Data modules” for details on data modules.

There are three additional optional top-level image definition sections that affect how the image definition dictionary is composed and the image description is generated:

6.3.2 image-config-comments #

 

This section allows to add top-level comments in the produced KIWI file. The format is as follows:

image-config-comments:
  COMMENT_NAME: string
  ...

COMMENT_NAME is just a name and is not included in the generated output. Comments can be used to include arbitrary information in the image description. Some comments have a special meaning for processing image descriptions by the Open Build Service, for instance the OBS-Profiles directive that is required to process multi-profile image descriptions. See https://osinside.github.io/kiwi/working_with_images/build_in_buildservice.html for details.

Note

Keg generates some comments automatically. In case the image definition has multiple profiles and the --disable-multibuild command line switch is not set, it will add an OBS-Profiles: @BUILD_FLAVOR@ comment. In case the image description is generated for one or more specific architectures via the -a command line option, the apprpriate OBS-ExclusiveArch comment is added.

xmlfiles:
  - name: FILENAME
    content:
      CONTENT_DICTIONARY
  ...

6.3.4 schema #

 

Keg starting with version 2.0.0 has an internal XML generator to produce KIWI image descriptions. Previously, a Jinja2 template was used to convert the image dictionary that Keg constructed into a KIWI image description. Using a Jinja2 template is still supported and can be configured as follows in the image definition:

schema: TEMPLATE

In this case, instead of running the XML generator, Keg would read the file TEMPLATE.kiwi.templ from the schemas directory in the recipes root directory and run it trough the Jinja2 engine.

Note

While using a Jinja2 template would in theory allow to operate on different input data structures, the internal schema validator requires the image definition to comply with what Keg expects.

Any directory under data that is not scripts or overlayfiles is considered an image definition data module and may be included in the main image definition using the _include statement.

Inheritance rules apply similarly to the image definition tree, but additionally, Keg supports sub-versions of data modules. This can be used for instance to create slightly different versions of modules for use with different image versions while still sharing most of the image definition between those versions.

For this purpose, Keg supports the include-paths directive in the image definition. Include paths are paths that get appended to any source path and those get scanned for input files as well. See the following image definition as an example:

include-paths:
  leap15/1
  leap15/2
image:
  preferences:
    - _include:
        - base/common
  packages:
    - _include:
        - base/common
  config:
    - _include:
      - base/common

This tells Keg, when adding data from directory data/base/common to the image data dictionary, to also look into sub directories leap15/2, leap15/1, and leap15 (through inheritance). This would lead to the following directories being scanned:

data
data/common
data/common/base
data/common/base/leap15
data/common/base/leap15/1
data/common/base/leap15/2

This allows for example to put generic configuration bits in data/common/base, Leap 15 specific configuration in data/common/base/leap15, and adjust the configuration for minor versions, if necessary.

When merging the included dictionaries into the main dictionary, Keg only copies the dictionary under the top level key that matches the key under which the _include statement is. That means, assuming the YAML files collected from the above trees resulted in the following data structure:

preferences:
  locale: en_US
  timezone: UTC
  type:
    _attributes:
      firmware: efi
      image: vmx
packages:
  _namespace_base_packages:
    package:
      - bash
      - glibc
      - kernel-default
config:
  _namespace_base_services:
    services:
      - sshd

Would result in a data structure like this:

include-paths:
  leap15/1
  leap15/2
image:
  preferences:
    locale: en_US
    timezone: UTC
    type:
      _attributes:
        firmware: efi
        image: vmx
  packages:
    _namespace_base_packages:
      package:
        - bash
        - glibc
        - kernel-default
config:
  _namespace_base_services:
    services:
      - sshd

Merging based on the parent key allows for grouping of different types of configuration data in one data module.

Configuration scriptlets are individual script snippets that can be used to generate image configuration scripts. KIWI runs those scripts at certain points in the image build process. They can be used to do changes to the system’s configuration.

The scriptlets are located in data/scripts and are required to have a .sh suffix. These are referenced in the scripts lists of the config or setup sections in the image definition (without the .sh suffix). See Section 6.1.2, “config” for details on the config section.

KIWI image descriptions can contain optional overlay archives, which will be extracted into the system’s root directory before the image is created. Overlay files are located in sub-directories in data/overlayfiles, with each sub-directory representing an overlay files module. Any directory structure under the module’s top directory is preserved.

Overlay files modules can be referenced in the archive section of the image definition using the _include_overlays directive. See Section 6.1.4, “archive” for details.

With source info tracking enabled, Keg will write one or more source info files in addition to the image description in the output directory. In case the image description at hand is single-build, a single file log_sources is written, in case it is multi-build, a file log_sources_PROFILE is written for each profile. This allows for generating individual change logs for the resulting image binaries.

The source info logs contain detailed information about which bits from the keg-recipes tree was used to generate the image description. The source info log files will contain several lines of the following format:

root:/path/to/repository
range:start:end:/path/to/repository/file
/path/to/repository/file_or_dir

The first line specifies the repository location. There will be one for each keg-recipes directory given to Keg. Lines starting with range: specify a part of a file in a repository. This is used to track the source location of each key that was in the final image dictionary. The third line format simply specifies a file or a directory in the repository that was used in the image description, and is used for configuration script snippets and overlay files.

This enables the change log generator to produce a change log using the git commit history, selecting only commits that apply to the generated image description.

The generated source info log files, together with the keg-recipes in the place and state they were used to generate the image description, can be used to generate change logs. The Keg distribution contains a tool generate_recipes_changelog for that purpose. When called with a source log file as argument, generate_recipes_changelog will use the source information to select matching git messages and produce a change log in chronological order. There are parameters to to narrow down the applicable commit range as well as some formatting options. Refer to Section 4.2, “generate_recipes_changelog” command overview for details.

The Keg distribution contains a module for integrating with the Open Build Service, an implementation of a so-called OBS Source Service. It supports automatic handling of change log generation. See Section 9, “Keg OBS source service” for details.