My first Yocto layer

Posted on Tue 27 November 2018 in BSP

Now that we have built a first bootable image with Yocto we can start customizing it. I will demonstrate how to create a simple layer to add a first custom library and an application depending on it, that should cover most of typical the use cases and should give you some pointers to start well Yocto.

Why creating a meta layer?

Despite most of the customization can be done with the local.conf configuration file, it is not possible to:

  • Add a new application in the build system
  • Add a new custom kernel
  • Add a new machine definition
  • Add a new image definition

This is why I strongly recommend creating a layer that can be customized to your specific needs.

What is a recipe?

A recipe is a small description file that indicate how to build an artifact that can be:

  • A library
  • An application
  • A root file system
  • A bootloader
  • A kernel
  • ...

Each instruction is splitted into tasks where the most commonly used are:

  • fetch: Get the source files
  • patch: Patch the sources
  • configure: Prepare/configure the sources for the build
  • compile: Cross-compile the sources
  • install: Install the binary in a temporary directory
  • package: Create a package that can be installed in the final file system or used in a package repository

Yocto has default implementation for each task and most of the time, there is nothing to do. However it is possible to customize them.

A minimalistic layer

First we will create a folder to contain the meta layer and add the bare minimal to integrate it succesfully to Yocto. The most important file is a configuration that gives the name of your layer, depedencies and compatibiliy with other layers.

$ mkdir -p my-layer/conf
$ touch my-layer/conf/layer.conf

Now here is what the layer.conf file looks like:

# Inform bitbake about this layer
BBPATH .= ":${LAYERDIR}"

# Regex that will include all recipes (.bb) and existing recipes
# customization (.bbappend)
BBFILES += "${LAYERDIR}/recipes-*/*/*.bb ${LAYERDIR}/recipes-*/*/*.bbappend"

# The name of the layer
BBFILE_COLLECTIONS += "my_layer"

# Inform bitbake about recipes belonging to this meta layer. The
# BBFILE_PATTERN variable is appended with "_" and the layer name
BBFILE_PATTERN_my_layer = "^${LAYERDIR}/"

# Priority of this layer when other recipes are also available in other
# existing layers. Also appendedn with "_" and the layer name
BBFILE_PRIORITY_my_layer = "1"

# Version of this layer
LAYERVERSION_my_layer = "1"

# Depedency to other layers, here we only need the core layer that is
# part of poky meta layer
LAYERDEPENDS_my_layer = "core"

# Compatibility with other layers version, we use here the latest stable
LAYERSERIES_COMPAT_my_layer = "sumo"

To add this layer to yocto, we need to add it to the bblayers.conf file as described in the previous article. To make sure that it is well integrated in the build system, we can start a build and it should be listed in the output:

build$ bitbake core-image-minimal
# SNIP
Build Configuration:
BB_VERSION           = "1.38.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "arm-poky-linux-gnueabi"
MACHINE              = "raspberrypi3"
DISTRO               = "poky"
DISTRO_VERSION       = "2.5.1"
TUNE_FEATURES        = "arm armv7ve vfp thumb neon vfpv4 callconvention-hard cortexa7"
TARGET_FPU           = "hard"
meta
meta-poky
meta-yocto-bsp       = "sumo:eebbc00b252a84d2502c3f5c7acd5a622dbd6e31"
meta-my-layer        # <-- Iiiyieah!

Recipes naming convetion

By convention, recipes are splitted into categories and application name, that corresponds to the BBFILES variable that we defined in the configuration. The most difficult part is deciding in which category your recipe will go but by checking what was already done in the official layers should give you a good idea of what you should do.

The name of the recipe is very important, by convention, it should be named as follow:

name_version.bb

Yocto will then automatically fill some internal variables where:

  • PN: The of the library/application (Package Name)
  • PV: The of the library/application (Package Version)

The version can be any alpha numeric values, git for the latest git version, or svn for the latest SVN version respectively. It is important that the version number goes incrementally, when more than one version of the recipe is available, Yocto will pick up automatically the latest one, where git has the highest priority.

My first library recipe

For this example, I will add a library called rglib in the support category and provide the latest git version, so create the parent folder and the recipe file:

my-layer$ mkdir -p recipes-support/rglib
my-layer$ touch recipes-support/rglib/rglib_git.bb

All recipes start with the same variables that gives a small description of what it is and mostly used for the packages creation and gives an idea of what is the package:

SUMMARY = "R4nd0m6uy library"
DESCRIPTION = "Reusable C++ objects for platforms abstraction"
AUTHOR = "R4nd0m6uy <r4nd0m6uy@r4nd0m6uy.ch>"
HOMEPAGE = "https://gitlab.com/morandg/lib-r4nd0m6uy"

Then there is the licensing information that is very important as Yocto tries make sure that all packages are open sources and fails in case you are using a commercial license. It is still possible to use a commercial license but this must be explicitely allowed. Moreover, we must provide a file containing the licensing with a MD5 checksum. When the checksum doesn't correspond, Yocto will pop up an error to make sure that we notice it.

LICENSE = "GPLv3"
LIC_FILES_CHKSUM = "file://LICENSE;md5=1ebbd3e34237af26da5dc08a4e440464"

In case we have depedenciy to other packages, the DEPENDS variable is used to list them. In our case we will need that libevent to be available before starting the compliation. Moreover, we need the pkg-config utility for the host system in order to find flags when compiling the package, that is used within the Makefile. The -native extension informs Yocto that it must be run on the building host during the compilation and will not be available in the final image:

DEPENDS = " \
  pkgconfig-native \
  libevent \
  "

The following is specific to git packages. SRCREV should point to a valid revision within the repository. With AUTOREV, Yocto will fill the variable with the latest git revision. We also change the PV variable to also include the revision within the version:

SRCREV = "${AUTOREV}"
PV = "git-${SRCPV}"

Then we need to indicate where Yocto must download the sources. Yocto is smart enough to atomatically get sources from any kind such as archive, SVN repositories, git, local files and many more. Basically, this indicate to the fetch task what, where and how sources must be downloaded, a gitlab repository in our case:

SRC_URI = " \
  git://gitlab.com/morandg/lib-r4nd0m6uy.git;protocol=https;branch=master \
  "

Here we change the S variable built by Yocto, that is the source directory. In case of a git repository, the sources are cloned within a folder called git. The WORKDIR variable contains the path to the working directory used during the build of the recipe:

S = "${WORKDIR}/git"

Then we customize the compile task by calling make. It is important to use oe_runmake as Yocto performs some tasks when invoking make. We force a debug mode compilation as Yocto will provide a debug mode package and will automatically strip the binaries for a release version. In case of errors, we enable the verbose build to have more information when reading the logs:

do_compile() {
  oe_runmake LDFLAGS="${LDFLAGS}" static DEBUG=1 V=1
  oe_runmake LDFLAGS="${LDFLAGS}" shared DEBUG=1 V=1
}

Finally, installing the library is done by calling make with the install target. The D variable contains the destination folder where the files will be installed, that is within the working directory and the prefix that is usually /usr:

do_install() {
  oe_runmake install DESTDIR=${D} PREFIX=${prefix} DEBUG=1 V=1
}

Now we are done! Notice that for other tasks, nothing is defined as we use the generic implementation for downloading the sources and creating the package. Now we can check that the build is done correctly by invoking bitbake and the recipe name:

build$ bitbake rglib

My first application

For the next step, I will create a recipe for an application using the library created previously that will be called charguychess_git.bb:

SUMMARY = "Software for the Charguychess DIY hardware"
DESCRIPTION = "Software for the Charguychess DIY hardware"
AUTHOR = "R4nd0m6uy and Charly"
HOMEPAGE = "https://gitlab.com/morandg/charguychess2"
SECTION = "games"
LICENSE = "GPLv3"
LIC_FILES_CHKSUM = "file://LICENSE;md5=d32239bcb673463ab874e80d47fae504"

In the DEPENDS variable, we will also add the libraries we need, Yocto will then make sure it will be available during the cross-compilation:

DEPENDS = " \
  pkgconfig-native \
  libconfig \
  rglib \
  "

This RRECOMMENDS variable indicate other packages that need to be installed in order to extend the functionnality of the application. The difference with DEPENDS is that it is not needed for the compilation but will be shipped when installing the package or building the root file system:

RRECOMMENDS_${PN} = " \
  stockfish \
  fruit \
  "

Other than the git URL, we will also provide a configuration file config that will be available in the files subdirectory of the recipe:

SRC_URI = " \
  git://gitlab.com/morandg/charguychess2.git;protocol=https;branch=master \
  file://config \
  "

The rest is already known, except that we also install the configuration file in /etc/charguychess/ during install task:

S = "${WORKDIR}/git"

do_compile() {
  oe_runmake LDFLAGS="${LDFLAGS}" DEBUG=1 V=1
}

do_install() {
  oe_runmake install DESTDIR=${D} DEBUG=1 V=1 PREFIX=${prefix}

  install -D -m 0664 ${WORKDIR}/config ${D}${sysconfdir}/charguychess/config
}

Now we can test that the recipe builds correctly:

build$ bitbake charguychess

Adding the application to the final image

Adding the recipe and compiling the application manually is not enough to add it to the image. We must indicate bitbake that it must be installed in the root file system.

The fastest and most convenient is to add the package in the build by editing the local.conf file and adding the following line:

IMAGE_INSTALL_append = "charguychess"

However, this will be added to every built image that is probably not what we want. This is why I recommend to write a recipe to build a custom image with the minimal set of packages. Simply create a file in recipes-core/images/charguychess.bb that looks like this:

DESCRIPTION = "Base image for charguychess DIY hardware"

# Indicate that this is an image recipe and we want to add users
inherit core-image extrausers

# Packages to install, packagegroup-core-boot contains the bare minimal
# to boot the system.
IMAGE_INSTALL = " \
  packagegroup-core-boot \
  charguychess \
  "

# Add dropbear ssh server
IMAGE_FEATURES = "ssh-server-dropbear"

# Set a root password to "charguychess"
EXTRA_USERS_PARAMS = "usermod -P charguychess root;"

Notice that rglib was not included to the image. Yocto is able to guess which packes will be included with the DEPENDS, RDEPENDS, RRECOMMEND variables. Now we can build the image with bitbake:

build$ MACHINE="raspberrypi3" bitbake charguychess-image

Now just need to deploy the image on the target, boot and that's it!

Using classes

Did you notice the inherit keyword in the image recipe? This means that it inherit some behavior already implemented by the Yocto people. There are classes for many other builds system and common tasks such as:

Before doing something on your own, check if there is a class that already does what you are trying to achieve.

Where getting help

At first glance, everything look a bit magic but basically Yocto works by inheriting classes and filling variables. For a list of all available variables, please refer to the manual.

When the build fails, it might be very frustrating fir the beginners and requires a bit of experience to dig into the working directory to make changes, rebuild and update the recipe, that would probably require another article.

However, there are thousend of recipes available out there and there is a good chance that some are doing something similare to what you need, reading the source of the existing might help most of the time!

People on themailing list and IRC are very kind and helpful, searching in the archive might also give you some pointers and if you don't find an answer to your question, don't hesitate to subscribe and ask.

What to do next

Now we are able to build a custom image but this is still not very convenient for a developper who doesn't want to run bitbake to compile his application!

This is why building an SDK is a more convinient way of providing a cross toolchain during the development stage and is made very easy to do with Yocto. Once the applications is well tested and works with the SDK, we can then add recipe in our custom meta-layer.

Material of this article

Of course, you are free to get ideas out of my work that I was using while writing this article. Other than what I discussed here, you might find some useful tips:

Conclusion

Yocto is quiet a beast and some experience is required to adopt it. However I hope that you found in this article the basic to start, where to get help and information about all the magic behind the hood.