Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]

12 Contributing

Note
Note

Abstract

This document describes the development process of KIWI NG and how you can be part of it. This description applies to version 9.23.60.

12.1 Using KIWI NG in a Python Project

Note
Note

Abstract

KIWI NG is provided as python module under the kiwi namespace. It is available for the python 3 version. The following description applies for KIWI NG version 9.23.60.

KIWI NG can also function as a module for other Python projects. The following example demonstrates how to read an existing image description, add a new repository definition and export the modified description on stdout.

import sys
import logging

from kiwi.xml_description import XMLDescription
from kiwi.xml_state import XMLState

description = XMLDescription('path/to/kiwi/XML/config.xml')

xml_data = description.load()

xml_state = XMLState(
    xml_data=xml_data, profiles=[], build_type='iso'
)

xml_state.add_repository(
    repo_source='http://repo',
    repo_type='rpm-md',
    repo_alias='myrepo',
    repo_prio=99
)

xml_data.export(
    outfile=sys.stdout, level=0
)

All classes are written in a way to care for a single responsibility in order to allow for re-use on other use cases. Therefore it is possible to use KIWI NG outside of the main image building scope to manage e.g the setup of loop devices, filesystems, partitions, etc…

This means KIWI NG provides you a way to describe a system but you are free to make use of the kiwi description format or not. The following example shows how to use kiwi to create a simple filesystem image which contains your host tmp directory.

import logging

from kiwi.storage.loop_device import LoopDevice
from kiwi.filesystem import FileSystem

loop_provider = LoopDevice(
    filename='my_tmp.ext4', filesize_mbytes=100
)
loop_provider.create()

filesystem = FileSystem.new(
    'ext4', loop_provider, '/tmp/'
)
filesystem.create_on_device(
    label='TMP'
)
filesystem.sync_data()

12.2 Plugin Architecture

Each command provided by KIWI NG is written as a task plugin under the kiwi.tasks namespace. As a developer you can extend KIWI NG with custom task plugins if the following conventions are taken into account:

12.2.1 Naming Conventions

Task Plugin File Name

The file name of a task plugin must follow the pattern <service>_<command>.py. This allows to invoke the task with kiwi-ng service command ...

Task Plugin Option Handling

KIWI NG uses the docopt module to handle options. Each task plugin must use docopt to allow option handling.

Task Plugin Class

The implementation of the plugin must be a class that matches the naming convention: <Service><Command>Task. The class must inherit from the CliTask base class. On startup of the plugin, KIWI NG expects an implementation of the process method.

Task Plugin Entry Point

Registration of the plugin must be done in setup.py using the entry_points concept from Python’s setuptools.

'packages': ['kiwi_plugin'],
'entry_points': {
    'kiwi.tasks': [
        'service_command=kiwi_plugin.tasks.service_command'
    ]
}

12.2.2 Example Plugin

Note
Note

The following example assumes an existing Python project which was set up according to the Python project rules and standards.

  1. Assuming the project namespace is kiwi_relax_plugin.

    Create the task plugin directory kiwi_relax_plugin/tasks

  2. Create the entry point in setup.py.

    Assuming we want to create the service named relax providing the command justdoit this would be the following entry point definition in setup.py:

    'packages': ['kiwi_relax_plugin'],
    'entry_points': {
        'kiwi.tasks': [
            'relax_justdoit=kiwi_relax_plugin.tasks.relax_justdoit'
        ]
    }
  3. Create the plugin code in the file kiwi_relax_plugin/tasks/relax_justdoit.py with the following content:

    """
    usage: kiwi-ng relax justdoit -h | --help
           kiwi-ng relax justdoit --now
    
    commands:
        justdoit
            time to relax
    
    options:
        --now
            right now. For more details about docopt
            see: http://docopt.org
    """
    # These imports requires kiwi to be part of your environment
    # It can be either installed from pip into a virtual development
    # environment or from the distribution package manager
    from kiwi.tasks.base import CliTask
    from kiwi.help import Help
    
    class RelaxJustdoitTask(CliTask):
        def process(self):
            self.manual = Help()
            if self.command_args.get('help') is True:
                # The following will invoke man to show the man page
                # for the requested command. Thus for the call to
                # succeed a manual page needs to be written and
                # installed by the plugin
                return self.manual.show('kiwi::relax::justdoit')
    
            print(
                'https://genius.com/Frankie-goes-to-hollywood-relax-lyrics'
            )
  4. Test the plugin

    $ ./setup.py develop
    $ kiwi-ng relax justdoit --now

12.3 Write Integration Tests for the Scripts

Kiwi ships a set of helper functions that can be used in config.sh (see also: Section 7.6, “User Defined Scripts”). These utilize containers to run the individual functions and verify that they resulted in the desired state.

Ensure that you have either podman or docker installed and configured on your system. The integration tests will use podman in rootless mode by default, if it is installed on your system. You can select docker instead by setting the environment variable CONTAINER_RUNTIME to docker. Then you can run the integration tests via tox:

$ tox -e scripts -- -n NUMBER_OF_THREADS

It is recommended to leverage testinfra and the shared_container and container_per_test fixtures for writing these integration tests. The fixtures give your test functions a connection to a running container (by default that will be opensuse/tumbleweed) functions.sh copied to /bin/functions.sh inside the container. You can then use the connection to perform some setup, tear down and the actual tests as follows:

def test_RmWorks(shared_container):
    # create the file /root/foobar
    shared_container.run_expect([0], "touch /root/foobar")
    assert shared_container.file("/root/foobar").exists

    # source the functions and execute our function under test
    shared_container.run_expect([0], ". /bin/functions.sh && Rm /root/foobar")

    # verify the result
    assert not shared_container.file("/root/foobar").exists

In this example we used the shared_container fixture: it creates a podman container at the start of the test session and gives each function using this fixture the same connection. Therefore you must only use it for tests where you do not perform any mutation of the container that you are not undoing afterwards! If you need to perform extensive mutation that you cannot or do not want to undo yourself, then resort to the container_per_test fixture. It will give you a fresh container for each test function. While this makes writing tests simpler, it also increases the runtime significantly, thus only use it when necessary.

12.3.1 Running Tests for multiple container images

It is important to test certain functions on multiple operating systems & versions, to e.g. ensure that older tools behave the same way that you expect them to.

This can be achieved by leveraging pytest’s fixture parametrization as follows:

@pytest.mark.parametrize(
    "shared_container",
    (
        "Tumbleweed",
        "Leap-15.3",
    ),
    indirect=True,
)
def test_something(shared_container):
    pass

Where we pass multiple image names to the container images to the shared_container fixture. Pytest will then look for the image with the given name in the predefined list of containers in conftest.py.

To add a new container, simply add a new Container class to the CONTAINERS list and give it appropriate values for name and url.

12.4 Extending KIWI NG with Custom Operations

Note
Note

Abstract

Users building images with KIWI NG need to implement their own infrastructure if the image description does not provide a way to embed custom information which is outside of the scope of the general schema as it is provided by KIWI NG today.

This document describes how to create an extension plugin for the KIWI NG schema to add and validate additional information in the KIWI NG image description.

Such a schema extension can be used in an additional KIWI NG task plugin to provide a new subcommand for KIWI NG. As of today there is no other plugin interface except for providing additional KIWI NG commands implemented.

Depending on the demand for custom plugins, the interface to hook in code into other parts of the KIWI NG processing needs to be extended.

This description applies for version 9.23.60.

12.4.1 The <extension> Section

The main KIWI NG schema supports an extension section which allows to specify any XML structure and attributes as long as they are connected to a namespace. According to this any custom XML structure can be implemented like the following example shows:

<image>
    ...
    <extension xmlns:my_plugin="http://www.my_plugin.com">
        <my_plugin:my_feature>
            <my_plugin:title name="cool stuff"/>
        </my_plugin:my_feature>
    </extension>
</image>
  • Any toplevel namespace must exist only once

  • Multiple different toplevel namespaces are allowed, e.g my_plugin_a, my_plugin_b

12.4.2 RELAX NG Schema for the Extension

If an extension section is found, KIWI NG looks up its namespace and asks the main XML catalog for the schema file to validate the extension data. The schema file must be a RELAX NG schema in the .rng format. We recommend to place the schema as /usr/share/xml/kiwi/my_plugin.rng

For the above example the RELAX NG Schema in the compressed format my_plugin.rnc would look like this:

namespace my_plugin = "http://www.my_plugin.com"

start =
    k.my_feature

div {
    k.my_feature.attlist = empty
    k.my_feature =
        element my_plugin:my_feature {
            k.my_feature.attlist &
            k.title
        }
}

div {
    k.title.name.attribute =
        attribute name { text }
    k.title.attlist = k.title.name.attribute
    k.title =
        element my_plugin:title {
            k.title.attlist
        }
}

In order to convert this schema to the .rng format just call:

$ trang -I rnc -O rng my_plugin.rnc /usr/share/xml/kiwi/my_plugin.rng

12.4.3 Extension Schema in XML catalog

As mentioned above the mapping from the extension namespace to the correct RELAX NG schema file is handled by a XML catalog file. The XML catalog for the example use here looks like this:

<?xml version="1.0"?>
<catalog xmlns="urn:oasis:names:tc:entity:xmlns:xml:catalog">
    <system
        systemId="http://www.my_plugin.com"
        uri="file:////usr/share/xml/kiwi/my_plugin.rng"/>
</catalog>

For resolving the catalog KIWI NG uses the xmlcatalog command and the main XML catalog from the system which is /etc/xml/catalog.

Note
Note

It depends on the distribution and its version how the main catalog gets informed about the existence of the KIWI NG extension catalog file. Please consult the distribution manual about adding XML catalogs.

If the following command provides the information to the correct RELAX NG schema file you are ready for a first test:

$ xmlcatalog /etc/xml/catalog http://www.my_plugin.com

12.4.4 Using the Extension

In order to test your extension place the example extension section from the beginning of this document into one of your image description’s config.xml file

The following example will read the name attribute from the title section of the my_feature root element and prints it:

import logging

from kiwi.xml_description import XMLDescription

description = XMLDescription('path/to/kiwi/XML/config.xml')
description.load()

my_plugin = description.get_extension_xml_data('my_plugin')

print(my_plugin.getroot()[0].get('name'))

12.5 The Basics

The core appliance builder is developed in Python and follows the test driven development rules.

If you want to implement a bigger feature, consider opening an issue on GitHub first to discuss the changes. Or join the discussion in the #kiwi channel on Riot.im.

12.6 Fork the upstream repository

  1. On GitHub, navigate to: https://github.com/OSInside/kiwi

  2. In the top-right corner of the page, click Fork.

12.7 Create a local clone of the forked repository

$ git clone https://github.com/YOUR-USERNAME/kiwi

$ git remote add upstream https://github.com/OSInside/kiwi.git

12.8 Install Required Operating System Packages

KIWI NG requires the following additional packages which are not provided by pip:

XML processing libraries

libxml2 and libxslt (for lxml)

Python header files, GCC compiler and glibc-devel header files

Required for python modules that hooks into shared library context

Spell Checking library

Provided by the enchant library

ShellCheck

ShellCheck script linter.

ISO creation program

One of xorriso (preferred) or genisoimage.

LaTeX documentation build environment

A full LaTeX installation is required to build the PDF documentation .

Host Requirements To Build Images

A full set of tools needed to build images and provided by the kiwi-systemdeps package

The above mentioned system packages will be installed by calling the install_devel_packages.sh helper script from the checked out Git repository as follows:

$ sudo helper/install_devel_packages.sh
Note
Note

The helper script checks for the package managers zypper and dnf and associates a distribution with it. If you use a distribution that does not use one of those package managers the script will not install any packages and exit with an error message. In this case we recommend to take a look at the package list encoded in the script and adapt to your distribution and package manager as needed.

12.9 Create a Python Virtual Development Environment

The following commands initializes and activates a development environment for Python 3:

$ tox -e devel
$ source .tox/3/bin/activate

The commands above automatically creates the application script called kiwi-ng, which allows you to run KIWI NG from the Python sources inside the virtual environment:

$ kiwi-ng --help
Warning
Warning

The virtualenv’s $PATH will not be taken into account when calling KIWI NG via sudo! Use the absolute path to the KIWI NG executable to run an actual build using your local changes:

$ sudo $PWD/.tox/3/bin/kiwi-ng system build ...

To leave the development mode, run:

$ deactivate

To resume your work, cd into your local Git repository and call:

$ source .tox/3/bin/activate

Alternatively, you can launch single commands inside the virtualenv without sourcing it directly:

$ tox -e devel -- kiwi-ng --version

12.10 Running the Unit Tests

We use tox to run the unit tests. Tox sets up its own virtualenvs inside the .tox directory for multiple Python versions and should thus not be invoked from inside your development virtualenv.

Before submitting your changes via a pull request, ensure that all tests pass and that the code has the required test coverage via the command:

$ tox

We also include pytest-xdist in the development virtualenv which allows to run the unit tests in parallel. It is turned off by default but can be enabled via:

$ tox "-n NUMBER_OF_PROCESSES"

where you can insert an arbitrary number as NUMBER_OF_PROCESSES (or a shell command like $(nproc)). Note that the double quotes around -n NUMBER_OF_PROCESSES are required (otherwise tox will consume this command line flag instead of forwarding it to pytest).

The previous call would run the unit tests for different Python versions, check the source code for errors and build the documentation.

If you want to see the available targets, use the option -l to let tox print a list of them:

$ tox -l

To only run a special target, use the -e option. The following example runs the test cases for the Python 3.6 interpreter only:

$ tox -e unit_py3_6

12.11 Create a Branch for each Feature or Bugfix

Code changes should be done in an extra Git branch. This allows for creating GitHub pull requests in a clean way. See also: Collaborating with issues and pull requests

$ git checkout -b my-topic-branch

Make and commit your changes.

Note
Note

You can make multiple commits which is generally useful to give your changes a clear structure and to allow us to better review your work.

Note
Note

Your work is important and must be signed to ensure the integrity of the repository and the code. Thus we recommend to setup a signing key as documented in Signing Git Patches.

$ git commit -S -a

Run the tests and code style checks. All of these are also performed by GitLab CI when a pull request is created.

$ tox

Once everything is done, push your local branch to your forked repository and create a pull request into the upstream repository.

$ git push origin my-topic-branch

Thank you much for contributing to KIWI NG. Your time and work effort is very much appreciated!

12.12 Coding Style

KIWI NG follows the general PEP8 guidelines with the following exceptions:

  • We do not use free functions at all. Even utility functions must be part of a class, but should be either prefixed with the @classmethod or @staticmethod decorators (whichever is more appropriate).

  • Do not set module and class level variables, put these into the classes’ __init__ method.

  • The names of constants are not written in all capital letters.

12.12.1 Documentation

KIWI NG uses Sphinx for the API and user documentation.

In order to build the HTML documentation call:

tox -e doc

or to build the full documentation (including a PDF generated by LaTeX ):

tox -e packagedoc

Document all your classes, methods, their parameters and their types using the standard reStructuredText syntax as supported by Sphinx, an example class is documented as follows:

class Example:
    """
    **Example class**

    :param str param: A parameter
    :param bool : Source file name to compress
    :param list supported_zipper: List of supported compression tools
    :attr Optional[str] attr: A class attribute
    """
    def __init__(self, param, param_w_default=False):
        self.attr = param if param_w_default else None

    def method(self, param):
        """
        A method that takes a parameter.

        :param list param: a parameter
        :return: whether param is very long
        :rtype: bool
        """
        return len(param) > 50

Try to stick to the following guidelines when documenting source code:

  • Classes should be documented directly in their main docstring and not in __init__.

  • Document every function parameter and every public attribute including their types.

  • Only public methods should be documented, private methods don’t have to, unless they are complex and it is not easy to grasp what they do (which should be avoided anyway).

Please also document any user-facing changes that you implementing (e.g. adding a new build type) in the user documentation, which can be found in doc/source. General documentation should be put into the working_with_kiwi/ subfolder, whereas documentation about more specialized topics would belong into the building/ subfolder.

Adhere to a line limit of 75 characters when writing the user facing documentation .

12.13 Additional Information

The following sections provides further information about the repository integrity, version, package and documentation management.

12.13.1 Signing Git Patches

To ensure the integrity of the repository and the code base, patches sent for inclusion should be signed with a GPG key.

To prepare Git to sign commits, follow these instructions:

  1. Create a key suitable for signing (it is not recommended to use existing keys to not mix it with your email environment):

    $ gpg2 --expert --full-gen-key
  2. Either choose a RSA key for signing (option (4)) or an ECC key for signing (option (10)). For a RSA key choose a key size of 4096 bits and for a ECC key choose Curve 25519 (option (1)). Enter a reasonable validity period (we recommend 2 to 5 years). Complete the key generation by entering your name and email address.

  3. Add the key ID to your git configuration, by running the following git config commands:

    $ git config --local user.signingkey $YOUR_SIGN_KEY_ID
    $ git config --local commit.gpgSign true

    Omitting the flag --local will make these settings global for all repositories (they will be added to ~/.gitconfig). You can find your signkey’s ID via:

    $ gpg2 --list-keys --keyid-format long $YOUR_EMAIL
    pub   rsa4096/AABBCCDDEEFF0011 2019-04-26 [S] [expires: 2021-04-16]
    AAAAAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBB
    uid                 [ultimate] YOU <$YOUR_EMAIL>

    The key’s ID in this case would be AABBCCDDEEFF0011. Note that your signkey will have only a [S] after the creation date, not a [SC] (then you are looking at your ordinary GPG key that can also encrypt).

12.13.2 Bumping the Version

The KIWI NG project follows the Semantic Versioning scheme. We use the bumpversion tool for consistent versioning.

Follow these instructions to bump the major, minor, or patch part of the KIWI NG version. Ensure that your repository is clean (i.e. no modified and unknown files exist) beforehand running bumpversion.

  • For backwards-compatible bug fixes:

    $ bumpversion patch
  • For additional functionality in a backwards-compatible manner. When changed, the patch level is reset to zero:

    $ bumpversion minor
  • For incompatible API changes. When changed, the patch and minor levels are reset to zero:

    $ bumpversion major

12.13.3 Creating a RPM Package

We provide a template for a RPM spec file in package/python-kiwi-spec-template alongside with a rpmlint configuration file and an automatically updated python-kiwi.changes.

To create the necessary files to build a RPM package via rpmbuild, run:

$ make build

The sources are collected in the dist/ directory. These can be directly build it with rpmbuild, fedpkg, or submitted to the Open Build Service using osc.

Print this page