R4nd0m's Bloghttps://r4nd0m6uy.ch/2018-11-27T18:00:00+01:00Software EngineerMy first Yocto layer2018-11-27T18:00:00+01:002018-11-27T18:00:00+01:00R4nd0m 6uytag:r4nd0m6uy.ch,2018-11-27:/my-first-yocto-layer.html<p>Now that <a href="https://r4nd0m6uy.ch/my-first-yocto-target.html">we have built a first bootable image with Yocto</a>
we can start customizing it. I will demonstrate how to create a simple
layer to add a first custom library and an application depending on it,
that should cover most of typical the use cases and should give you …</p><p>Now that <a href="https://r4nd0m6uy.ch/my-first-yocto-target.html">we have built a first bootable image with Yocto</a>
we can start customizing it. I will demonstrate how to create a simple
layer to add a first custom library and an application depending on it,
that should cover most of typical the use cases and should give you some
pointers to start well Yocto.</p>
<h1>Why creating a meta layer?</h1>
<p>Despite most of the customization can be done with the <em>local.conf</em>
configuration file, it is not possible to:</p>
<ul>
<li>Add a new application in the build system</li>
<li>Add a new custom kernel</li>
<li>Add a new machine definition</li>
<li>Add a new image definition</li>
</ul>
<p>This is why I strongly recommend creating a layer that can be customized
to your specific needs.</p>
<h1>What is a recipe?</h1>
<p>A recipe is a small description file that indicate how to build an
artifact that can be:</p>
<ul>
<li>A library</li>
<li>An application</li>
<li>A root file system</li>
<li>A bootloader</li>
<li>A kernel</li>
<li>...</li>
</ul>
<p>Each instruction is splitted into tasks where the most commonly used
are:</p>
<ul>
<li><em>fetch</em>: Get the source files</li>
<li><em>patch</em>: Patch the sources</li>
<li><em>configure</em>: Prepare/configure the sources for the build</li>
<li><em>compile</em>: Cross-compile the sources</li>
<li><em>install</em>: Install the binary in a temporary directory</li>
<li><em>package</em>: Create a package that can be installed in the final file
system or used in a package repository</li>
</ul>
<p>Yocto has default implementation for each task and most of the time,
there is nothing to do. However it is possible to customize them.</p>
<h1>A minimalistic layer</h1>
<p>First we will create a folder to contain the meta layer and add the bare
minimal to integrate it succesfully to Yocto. The most important file is
a configuration that gives the name of your layer, depedencies and
compatibiliy with other layers.</p>
<div class="highlight"><pre><span></span><code>$ mkdir -p my-layer/conf
$ touch my-layer/conf/layer.conf
</code></pre></div>
<p>Now here is what the <em>layer.conf</em> file looks like:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Inform bitbake about this layer</span>
BBPATH .<span class="o">=</span> <span class="s2">":</span><span class="si">${</span><span class="nv">LAYERDIR</span><span class="si">}</span><span class="s2">"</span>
<span class="c1"># Regex that will include all recipes (.bb) and existing recipes</span>
<span class="c1"># customization (.bbappend)</span>
<span class="nv">BBFILES</span> <span class="o">+=</span> <span class="s2">"</span><span class="si">${</span><span class="nv">LAYERDIR</span><span class="si">}</span><span class="s2">/recipes-*/*/*.bb </span><span class="si">${</span><span class="nv">LAYERDIR</span><span class="si">}</span><span class="s2">/recipes-*/*/*.bbappend"</span>
<span class="c1"># The name of the layer</span>
<span class="nv">BBFILE_COLLECTIONS</span> <span class="o">+=</span> <span class="s2">"my_layer"</span>
<span class="c1"># Inform bitbake about recipes belonging to this meta layer. The</span>
<span class="c1"># BBFILE_PATTERN variable is appended with "_" and the layer name</span>
<span class="nv">BBFILE_PATTERN_my_layer</span> <span class="o">=</span> <span class="s2">"^</span><span class="si">${</span><span class="nv">LAYERDIR</span><span class="si">}</span><span class="s2">/"</span>
<span class="c1"># Priority of this layer when other recipes are also available in other</span>
<span class="c1"># existing layers. Also appendedn with "_" and the layer name</span>
<span class="nv">BBFILE_PRIORITY_my_layer</span> <span class="o">=</span> <span class="s2">"1"</span>
<span class="c1"># Version of this layer</span>
<span class="nv">LAYERVERSION_my_layer</span> <span class="o">=</span> <span class="s2">"1"</span>
<span class="c1"># Depedency to other layers, here we only need the core layer that is</span>
<span class="c1"># part of poky meta layer</span>
<span class="nv">LAYERDEPENDS_my_layer</span> <span class="o">=</span> <span class="s2">"core"</span>
<span class="c1"># Compatibility with other layers version, we use here the latest stable</span>
<span class="nv">LAYERSERIES_COMPAT_my_layer</span> <span class="o">=</span> <span class="s2">"sumo"</span>
</code></pre></div>
<p>To add this layer to yocto, we need to add it to the <em>bblayers.conf</em>
file as <a href="https://r4nd0m6uy.ch/my-first-yocto-target.html">described in the previous article</a>.
To make sure that it is well integrated in the build system, we can
start a build and it should be listed in the output:</p>
<div class="highlight"><pre><span></span><code>build$ bitbake core-image-minimal
<span class="c1"># SNIP</span>
Build Configuration:
<span class="nv">BB_VERSION</span> <span class="o">=</span> <span class="s2">"1.38.0"</span>
<span class="nv">BUILD_SYS</span> <span class="o">=</span> <span class="s2">"x86_64-linux"</span>
<span class="nv">NATIVELSBSTRING</span> <span class="o">=</span> <span class="s2">"universal"</span>
<span class="nv">TARGET_SYS</span> <span class="o">=</span> <span class="s2">"arm-poky-linux-gnueabi"</span>
<span class="nv">MACHINE</span> <span class="o">=</span> <span class="s2">"raspberrypi3"</span>
<span class="nv">DISTRO</span> <span class="o">=</span> <span class="s2">"poky"</span>
<span class="nv">DISTRO_VERSION</span> <span class="o">=</span> <span class="s2">"2.5.1"</span>
<span class="nv">TUNE_FEATURES</span> <span class="o">=</span> <span class="s2">"arm armv7ve vfp thumb neon vfpv4 callconvention-hard cortexa7"</span>
<span class="nv">TARGET_FPU</span> <span class="o">=</span> <span class="s2">"hard"</span>
meta
meta-poky
meta-yocto-bsp <span class="o">=</span> <span class="s2">"sumo:eebbc00b252a84d2502c3f5c7acd5a622dbd6e31"</span>
meta-my-layer <span class="c1"># <-- Iiiyieah!</span>
</code></pre></div>
<h1>Recipes naming convetion</h1>
<p>By convention, recipes are splitted into categories and application
name, that corresponds to the <em>BBFILES</em> variable that we defined in the
configuration. The most difficult part is deciding in which category
your recipe will go but by checking what was already done in the
official layers should give you a good idea of what you should do.</p>
<p>The name of the recipe is very important, by convention, it should be
named as follow:</p>
<blockquote>
<p><em>name</em>_<em>version</em>.bb</p>
</blockquote>
<p>Yocto will then automatically fill some internal variables where:</p>
<ul>
<li><strong>PN</strong>: The <name/> of the library/application (<strong>P</strong>ackage <strong>N</strong>ame)</li>
<li><strong>PV</strong>: The <version/> of the library/application (<strong>P</strong>ackage
<strong>V</strong>ersion)</li>
</ul>
<p>The version can be any alpha numeric values, <em>git</em> for the latest git
version, or <em>svn</em> for the latest SVN version respectively. It is
important that the version number goes incrementally, when more than one
version of the recipe is available, Yocto will pick up automatically the
latest one, where <em>git</em> has the highest priority.</p>
<h1>My first library recipe</h1>
<p>For this example, I will add a library called <em>rglib</em> in the <em>support</em>
category and provide the latest <em>git</em> version, so create the parent
folder and the recipe file:</p>
<div class="highlight"><pre><span></span><code>my-layer$ mkdir -p recipes-support/rglib
my-layer$ touch recipes-support/rglib/rglib_git.bb
</code></pre></div>
<p>All recipes start with the same variables that gives a small description
of what it is and mostly used for the packages creation and gives an
idea of what is the package:</p>
<div class="highlight"><pre><span></span><code><span class="nv">SUMMARY</span> <span class="o">=</span> <span class="s2">"R4nd0m6uy library"</span>
<span class="nv">DESCRIPTION</span> <span class="o">=</span> <span class="s2">"Reusable C++ objects for platforms abstraction"</span>
<span class="nv">AUTHOR</span> <span class="o">=</span> <span class="s2">"R4nd0m6uy <r4nd0m6uy@r4nd0m6uy.ch>"</span>
<span class="nv">HOMEPAGE</span> <span class="o">=</span> <span class="s2">"https://gitlab.com/morandg/lib-r4nd0m6uy"</span>
</code></pre></div>
<p>Then there is the licensing information that is very important as Yocto
tries make sure that all packages are open sources and fails in case
you are using a commercial license. It is still possible to use a
commercial license but this must be explicitely allowed. Moreover, we
must provide a file containing the licensing with a MD5 checksum. When
the checksum doesn't correspond, Yocto will pop up an error to make sure
that we notice it.</p>
<div class="highlight"><pre><span></span><code><span class="nv">LICENSE</span> <span class="o">=</span> <span class="s2">"GPLv3"</span>
<span class="nv">LIC_FILES_CHKSUM</span> <span class="o">=</span> <span class="s2">"file://LICENSE;md5=1ebbd3e34237af26da5dc08a4e440464"</span>
</code></pre></div>
<p>In case we have depedenciy to other packages, the <em>DEPENDS</em> variable is
used to list them. In our case we will need that <em>libevent</em> to be
available before starting the compliation. Moreover, we need the
<em>pkg-config</em> utility for the host system in order to find flags when
compiling the package, that is used within the Makefile. The <em>-native</em>
extension informs Yocto that it must be run on the building host during
the compilation and will not be available in the final image:</p>
<div class="highlight"><pre><span></span><code><span class="nv">DEPENDS</span> <span class="o">=</span> <span class="s2">" \</span>
<span class="s2"> pkgconfig-native \</span>
<span class="s2"> libevent \</span>
<span class="s2"> "</span>
</code></pre></div>
<p>The following is specific to git packages. <em>SRCREV</em> should point to a
valid revision within the repository. With <em>AUTOREV</em>, Yocto will fill
the variable with the latest git revision. We also change the <em>PV</em>
variable to also include the revision within the version:</p>
<div class="highlight"><pre><span></span><code>SRCREV = "<span class="cp">${</span><span class="n">AUTOREV</span><span class="cp">}</span>"
PV = "git-<span class="cp">${</span><span class="n">SRCPV</span><span class="cp">}</span>"
</code></pre></div>
<p>Then we need to indicate where Yocto must download the sources. Yocto is
smart enough to atomatically get sources from any kind such as archive,
SVN repositories, git, local files and many more. Basically, this
indicate to the <em>fetch</em> task what, where and how sources must be
downloaded, a gitlab repository in our case:</p>
<div class="highlight"><pre><span></span><code><span class="nv">SRC_URI</span> <span class="o">=</span> <span class="s2">" \</span>
<span class="s2"> git://gitlab.com/morandg/lib-r4nd0m6uy.git;protocol=https;branch=master \</span>
<span class="s2"> "</span>
</code></pre></div>
<p>Here we change the <em>S</em> variable built by Yocto, that is the source
directory. In case of a git repository, the sources are cloned within a
folder called <em>git</em>. The <em>WORKDIR</em> variable contains the path to the
working directory used during the build of the recipe:</p>
<div class="highlight"><pre><span></span><code><span class="nv">S</span> <span class="o">=</span> <span class="s2">"</span><span class="si">${</span><span class="nv">WORKDIR</span><span class="si">}</span><span class="s2">/git"</span>
</code></pre></div>
<p>Then we customize the <em>compile</em> task by calling make. It is important to
use <em>oe_runmake</em> as Yocto performs some tasks when invoking <em>make</em>. We
force a debug mode compilation as Yocto will provide a debug mode
package and will automatically strip the binaries for a release version.
In case of errors, we enable the verbose build to have more information
when reading the logs:</p>
<div class="highlight"><pre><span></span><code>do_compile<span class="o">()</span> <span class="o">{</span>
oe_runmake <span class="nv">LDFLAGS</span><span class="o">=</span><span class="s2">"</span><span class="si">${</span><span class="nv">LDFLAGS</span><span class="si">}</span><span class="s2">"</span> static <span class="nv">DEBUG</span><span class="o">=</span><span class="m">1</span> <span class="nv">V</span><span class="o">=</span><span class="m">1</span>
oe_runmake <span class="nv">LDFLAGS</span><span class="o">=</span><span class="s2">"</span><span class="si">${</span><span class="nv">LDFLAGS</span><span class="si">}</span><span class="s2">"</span> shared <span class="nv">DEBUG</span><span class="o">=</span><span class="m">1</span> <span class="nv">V</span><span class="o">=</span><span class="m">1</span>
<span class="o">}</span>
</code></pre></div>
<p>Finally, installing the library is done by calling <em>make</em> with the
<em>install</em> target. The <em>D</em> variable contains the destination folder where
the files will be installed, that is within the working directory and
the prefix that is usually <em>/usr</em>:</p>
<div class="highlight"><pre><span></span><code>do_install<span class="o">()</span> <span class="o">{</span>
oe_runmake install <span class="nv">DESTDIR</span><span class="o">=</span><span class="si">${</span><span class="nv">D</span><span class="si">}</span> <span class="nv">PREFIX</span><span class="o">=</span><span class="si">${</span><span class="nv">prefix</span><span class="si">}</span> <span class="nv">DEBUG</span><span class="o">=</span><span class="m">1</span> <span class="nv">V</span><span class="o">=</span><span class="m">1</span>
<span class="o">}</span>
</code></pre></div>
<p>Now we are done! Notice that for other tasks, nothing is defined as we
use the generic implementation for downloading the sources and creating
the package. Now we can check that the build is done correctly by
invoking <em>bitbake</em> and the recipe name:</p>
<div class="highlight"><pre><span></span><code>build$ bitbake rglib
</code></pre></div>
<h1>My first application</h1>
<p>For the next step, I will create a recipe for an application using the
library created previously that will be called <em>charguychess_git.bb</em>:</p>
<div class="highlight"><pre><span></span><code><span class="nt">SUMMARY</span> <span class="o">=</span> <span class="s2">"Software for the Charguychess DIY hardware"</span>
<span class="nt">DESCRIPTION</span> <span class="o">=</span> <span class="s2">"Software for the Charguychess DIY hardware"</span>
<span class="nt">AUTHOR</span> <span class="o">=</span> <span class="s2">"R4nd0m6uy and Charly"</span>
<span class="nt">HOMEPAGE</span> <span class="o">=</span> <span class="s2">"https://gitlab.com/morandg/charguychess2"</span>
<span class="nt">SECTION</span> <span class="o">=</span> <span class="s2">"games"</span>
<span class="nt">LICENSE</span> <span class="o">=</span> <span class="s2">"GPLv3"</span>
<span class="nt">LIC_FILES_CHKSUM</span> <span class="o">=</span> <span class="s2">"file://LICENSE;md5=d32239bcb673463ab874e80d47fae504"</span>
</code></pre></div>
<p>In the <em>DEPENDS</em> variable, we will also add the libraries we need, Yocto
will then make sure it will be available during the cross-compilation:</p>
<div class="highlight"><pre><span></span><code><span class="nv">DEPENDS</span> <span class="o">=</span> <span class="s2">" \</span>
<span class="s2"> pkgconfig-native \</span>
<span class="s2"> libconfig \</span>
<span class="s2"> rglib \</span>
<span class="s2"> "</span>
</code></pre></div>
<p>This <em>RRECOMMENDS</em> variable indicate other packages that need to be
installed in order to extend the functionnality of the application. The
difference with <em>DEPENDS</em> is that it is not needed for the compilation
but will be shipped when installing the package or building the root
file system:</p>
<div class="highlight"><pre><span></span><code>RRECOMMENDS_<span class="si">${</span><span class="nv">PN</span><span class="si">}</span> <span class="o">=</span> <span class="s2">" \</span>
<span class="s2"> stockfish \</span>
<span class="s2"> fruit \</span>
<span class="s2"> "</span>
</code></pre></div>
<p>Other than the git URL, we will also provide a configuration file
<em>config</em> that will be available in the <em>files</em> subdirectory of the
recipe:</p>
<div class="highlight"><pre><span></span><code><span class="nv">SRC_URI</span> <span class="o">=</span> <span class="s2">" \</span>
<span class="s2"> git://gitlab.com/morandg/charguychess2.git;protocol=https;branch=master \</span>
<span class="s2"> file://config \</span>
<span class="s2"> "</span>
</code></pre></div>
<p>The rest is already known, except that we also install the configuration
file in <em>/etc/charguychess/</em> during <em>install</em> task:</p>
<div class="highlight"><pre><span></span><code><span class="nv">S</span> <span class="o">=</span> <span class="s2">"</span><span class="si">${</span><span class="nv">WORKDIR</span><span class="si">}</span><span class="s2">/git"</span>
do_compile<span class="o">()</span> <span class="o">{</span>
oe_runmake <span class="nv">LDFLAGS</span><span class="o">=</span><span class="s2">"</span><span class="si">${</span><span class="nv">LDFLAGS</span><span class="si">}</span><span class="s2">"</span> <span class="nv">DEBUG</span><span class="o">=</span><span class="m">1</span> <span class="nv">V</span><span class="o">=</span><span class="m">1</span>
<span class="o">}</span>
do_install<span class="o">()</span> <span class="o">{</span>
oe_runmake install <span class="nv">DESTDIR</span><span class="o">=</span><span class="si">${</span><span class="nv">D</span><span class="si">}</span> <span class="nv">DEBUG</span><span class="o">=</span><span class="m">1</span> <span class="nv">V</span><span class="o">=</span><span class="m">1</span> <span class="nv">PREFIX</span><span class="o">=</span><span class="si">${</span><span class="nv">prefix</span><span class="si">}</span>
install -D -m <span class="m">0664</span> <span class="si">${</span><span class="nv">WORKDIR</span><span class="si">}</span>/config <span class="si">${</span><span class="nv">D</span><span class="si">}${</span><span class="nv">sysconfdir</span><span class="si">}</span>/charguychess/config
<span class="o">}</span>
</code></pre></div>
<p>Now we can test that the recipe builds correctly:</p>
<div class="highlight"><pre><span></span><code>build$ bitbake charguychess
</code></pre></div>
<h1>Adding the application to the final image</h1>
<p>Adding the recipe and compiling the application manually is not enough
to add it to the image. We must indicate bitbake that it must be
installed in the root file system.</p>
<p>The fastest and most convenient is to add the package in the build by
editing the <em>local.conf</em> file and adding the following line:</p>
<div class="highlight"><pre><span></span><code><span class="nv">IMAGE_INSTALL_append</span> <span class="o">=</span> <span class="s2">"charguychess"</span>
</code></pre></div>
<p>However, this will be added to every built image that is probably not
what we want. This is why I recommend to write a recipe to build a
custom image with the minimal set of packages. Simply create a file in
<em>recipes-core/images/charguychess.bb</em> that looks like this:</p>
<div class="highlight"><pre><span></span><code><span class="nv">DESCRIPTION</span> <span class="o">=</span> <span class="s2">"Base image for charguychess DIY hardware"</span>
<span class="c1"># Indicate that this is an image recipe and we want to add users</span>
inherit core-image extrausers
<span class="c1"># Packages to install, packagegroup-core-boot contains the bare minimal</span>
<span class="c1"># to boot the system.</span>
<span class="nv">IMAGE_INSTALL</span> <span class="o">=</span> <span class="s2">" \</span>
<span class="s2"> packagegroup-core-boot \</span>
<span class="s2"> charguychess \</span>
<span class="s2"> "</span>
<span class="c1"># Add dropbear ssh server</span>
<span class="nv">IMAGE_FEATURES</span> <span class="o">=</span> <span class="s2">"ssh-server-dropbear"</span>
<span class="c1"># Set a root password to "charguychess"</span>
<span class="nv">EXTRA_USERS_PARAMS</span> <span class="o">=</span> <span class="s2">"usermod -P charguychess root;"</span>
</code></pre></div>
<p>Notice that rglib was not included to the image. Yocto is able to guess
which packes will be included with the <em>DEPENDS</em>, <em>RDEPENDS</em>,
<em>RRECOMMEND</em> variables. Now we can build the image with bitbake:</p>
<div class="highlight"><pre><span></span><code>build$ <span class="nv">MACHINE</span><span class="o">=</span><span class="s2">"raspberrypi3"</span> bitbake charguychess-image
</code></pre></div>
<p>Now just need to deploy the image on the target, boot and that's it!</p>
<h1>Using classes</h1>
<p>Did you notice the <em>inherit</em> keyword in the image recipe? This means
that it inherit some behavior already implemented by the Yocto people.
There are classes for many other builds system and common tasks such as:</p>
<ul>
<li><a href="https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#ref-classes-autotools">Building with autotools</a></li>
<li><a href="https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#ref-classes-cmake">Building with cmake</a></li>
<li><a href="https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#ref-classes-extrausers">Adding users</a></li>
<li><a href="https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#ref-classes-update-rc.d">Installing init scripts</a></li>
<li><a href="https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#ref-classes-systemd">Installing systemd units</a>
(<strong>No flameware here but please
<a href="https://suckless.org/sucks/systemd/">don't</a>
<a href="http://without-systemd.org/wiki/index.php/Arguments_against_systemd">do</a>
<a href="http://ewontfix.com/14/">that</a> in an embbedded system! I listed here
as informational purpose!</strong>)</li>
<li>... and many more!</li>
</ul>
<p>Before doing something on your own, check if there is a class that
already does what you are trying to achieve.</p>
<h1>Where getting help</h1>
<p>At first glance, everything look a bit magic but basically Yocto works
by inheriting classes and filling variables. For a list of all available
variables, please <a href="https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#ref-variables-glossary">refer to the manual</a>.</p>
<p>When the build fails, it might be very frustrating fir the beginners and
requires a bit of experience to dig into the working directory to make
changes, rebuild and update the recipe, that would probably require
another article.</p>
<p>However, there are
<a href="https://layers.openembedded.org/">thousend of recipes</a> available out
there and there is a good chance that some are doing something similare
to what you need, reading the source of the existing might help most of
the time!</p>
<p>People on the<a href="https://www.yoctoproject.org/community/mailing-lists/">mailing list and IRC</a>
are very kind and helpful, searching in the archive might also give you
some pointers and if you don't find an answer to your question, don't
hesitate to subscribe and ask.</p>
<h1>What to do next</h1>
<p>Now we are able to build a custom image but this is still not very
convenient for a developper who doesn't want to run bitbake to compile
his application!</p>
<p>This is why building an SDK is a more convinient way of providing a
cross toolchain during the development stage and is made very easy to do
with Yocto. Once the applications is well tested and works with the SDK,
we can then add recipe in our custom meta-layer.</p>
<h1>Material of this article</h1>
<p>Of course, you are free to get ideas out of my work that I was using
while writing this article. Other than what I discussed here, you might
find some useful tips:</p>
<ul>
<li><a href="https://gitlab.com/morandg/yocto-build-script">My Yocto build helper script</a></li>
<li><a href="https://gitlab.com/morandg/meta-r4nd0m6uy">My personnal Yocto layer</a></li>
</ul>
<h1>Conclusion</h1>
<p>Yocto is quiet a beast and some experience is required to adopt it.
However I hope that you found in this article the basic to start, where
to get help and information about all the magic behind the hood.</p>My first Yocto target2018-11-21T18:00:00+01:002018-11-21T18:00:00+01:00R4nd0m 6uytag:r4nd0m6uy.ch,2018-11-21:/my-first-yocto-target.html<p>Due to its flexibility, Yocto might seem a bit complex the first time we
use it and can be very scary. In this article I would like to
demonstrate the basic usage of this wonderful build system by building a
first image for your specific target, that is the first …</p><p>Due to its flexibility, Yocto might seem a bit complex the first time we
use it and can be very scary. In this article I would like to
demonstrate the basic usage of this wonderful build system by building a
first image for your specific target, that is the first thing to do when
starting with Yocto.</p>
<h1>Host distribution</h1>
<p>For this article, I decided to use <em>Debian 9 (stretch)</em> but any
compatible distribution should do the trick, for more information,
please <a href="https://www.yoctoproject.org/docs/2.5/mega-manual/mega-manual.html#detailed-supported-distros">refer to the manual</a>.</p>
<h1>Preparing the host system</h1>
<p>First make sure that you have all the tools required by Yocto. It tries
to compile everything from scratch but some tools are still required
during the boostrap process. Moreover, it is written with python3 that
must be available on your system.</p>
<p>Under Debian, the following packages need to be installed:</p>
<div class="highlight"><pre><span></span><code>sudo apt-get install <span class="se">\</span>
git <span class="se">\</span>
python3 <span class="se">\</span>
build-essential <span class="se">\</span>
chrpath <span class="se">\</span>
diffstat <span class="se">\</span>
gawk <span class="se">\</span>
texinfo
</code></pre></div>
<h1>Download Yocto sources</h1>
<p>We will prepare a working directory to host all sources and build
artifacts that I will call <em>YOCTO-DIR</em> for convenience, just adapt it to
a path that makes sense for you and make sure that you have at least
<em>100Gb</em> available on the partition:</p>
<div class="highlight"><pre><span></span><code>$ mkdir -p <YOCTO-DIR> <span class="o">&&</span> <span class="nb">cd</span> <YOCTO-DIR>
</code></pre></div>
<p>We will then download the <em>poky</em> meta layers that contains the base
distribution definition and minimal recipes to build a bootable image.</p>
<div class="highlight"><pre><span></span><code><YOCTO-DIR>$ mkdir meta-layers
<YOCTO-DIR>$ git clone https://git.yoctoproject.org/git/poky meta-layers/poky
</code></pre></div>
<h1>Add a meta-layer for your target</h1>
<p>We also need a machine configuration file for our target. To find a
layer containing your machine, refer to the <a href="https://layers.openembedded.org/layerindex/branch/sumo/machines/">OpenEmbedded recipe
index</a>
and search for the machine you want to build for. In my case, I have
found <a href="https://layers.openembedded.org/layerindex/branch/sumo/layer/meta-raspberrypi/">this meta-layer</a>
that as machine definition file for the raspberry PI. If you don't find
an exact match for your machine, try to find a similar one as a starting
point.</p>
<p>Now clone the meta layer in the folder we prepared previously:</p>
<div class="highlight"><pre><span></span><code><YOCTO-DIR>$ git clone git://git.yoctoproject.org/meta-raspberrypi meta-layers/meta-raspberrypi
</code></pre></div>
<h1>Checkout a stable branch</h1>
<p>To avoid headaches I don't recommend using the development branch that
might break often. It is also discouraged to use meta-layers of
different releases that should not be compatible with each other. This
is why we will checkout that latest stable branch of each layer that is
<em>sumo</em> the day this article was written:</p>
<div class="highlight"><pre><span></span><code><YOCTO-DIR>$ <span class="nb">cd</span> meta-layers/poky
<YOCTO-DIR>/poky$ git checkout sumo
<YOCTO-DIR>/poky$ <span class="nb">cd</span> ../meta-raspberrypi
<YOCTO-DIR>/meta-raspberrypi$ git checkout sumo
</code></pre></div>
<p>There is a new release every six monthes. Ususally upgrading to the next
stable is not so painful as long as this is done regularly to avoid a
big gap between two stable releases. This is why I recommend testing the
build with the new stable once a release is done, even though you decide
to use an previous release for your product.</p>
<h1>Preparing the build directory</h1>
<p>The next step is to prepare a build directory that will contain a
configuration file for the build and the list of meta-layers you want to
use. Yocto is able to generate a default configuration that works for
most of the use cases. This can be done by sourcing a shell script and
giving a path to the build directory. Once this is done, you will be
inside the build directory given as argument:</p>
<div class="highlight"><pre><span></span><code><YOCTO-DIR>$ . ./meta-layers/poky/oe-init-build-env build
<YOCTO-DIR>/build$
</code></pre></div>
<p>Now we also need to add the raspberry PI meta-layer in the
<em><YOCTO-DIR>/build/conf/bblayer.conf</em> file. Simply edit and add the
layer cloned previously:</p>
<div class="highlight"><pre><span></span><code> <YOCTO-DIR>/meta-layers/poky/meta \
<YOCTO-DIR>/meta-layers/poky/meta-poky \
<YOCTO-DIR>/meta-layers/poky/meta-yocto-bsp \
<span class="gi">+ <YOCTO-DIR>/meta-layers/meta-raspberrypi \</span>
"
</code></pre></div>
<p>It is possible to tweak the build configuration by editing the
<em><YOCTO-DIR>/build/conf/local.conf</em> configuration file. The one
provided by default is self explanatory and should be easy to adapt to
your needs. For this article, I'm not changing anything.</p>
<h1>Start the build</h1>
<p>Now everything is ready to start a build. This can be done by calling
the <em>bitbake</em> command. Bitbake needs a recipe to build, in our case we
will build an image called <em>core-image-minimal</em>. We can also configure
the target machine by editing the <em>MACHINE</em> variable in the
<em><YOCTO-DIR>/build/conf/local.conf</em> configuration file. It is also
possible to overwrite this variable on the command line as follow:</p>
<div class="highlight"><pre><span></span><code><YOCTO-DIR>/build$ <span class="nv">MACHINE</span><span class="o">=</span>raspberrypi3 bitbake core-image-minimal
</code></pre></div>
<p>Now be patient, the first time the build takes a lot of time. However
Yocto is very smart to keep trace of what was built that makes
incremental build much faster, consequently future builds will take much
less time.</p>
<h1>Writing the image on an SD card</h1>
<p>Once the build is done, built artifacts such as device trees, kernels,
root file systems and so on are available in the
<em><YOCTO-DIR>/build/tmp/deploy/images/<MACHINE></em> folder. For the
raspberry pi, you should find a file with the <em>rpi-sdimg</em> extension that
is an image than can be written on an SD card with the <em>dd</em> utility.
After inserting the SD card in my computer, I could write the image as
follow:</p>
<div class="highlight"><pre><span></span><code><YOCTO-DIR>$ sudo dd <span class="k">if</span><span class="o">=</span>build/tmp/deploy/images/raspberrypi3/core-image-minimal-raspberrypi3.rpi-sdimg <span class="nv">of</span><span class="o">=</span>/dev/sdd
</code></pre></div>
<p>Now eject the SD card, insert it in the raspberry PI, power it on and
voilà!</p>
<p>For other platforms, it is possible that you have other images types
such as ubifs, ext4, u-boot, and many others. In that case, flashing the
image might be less convenient, in the worst case you might need to use
a JTAG interface. For further details, refer to the board vendor
documentation.</p>
<h1>Going further</h1>
<p>Now we have just built an image with the bare minimal to boot but there
are still many things we need to do in order to have something
customized to our needs such as:</p>
<ul>
<li>Writing a custom meta layer</li>
<li>Writing a custom distribution</li>
<li>Writing a custom machine definition file</li>
<li>Writing a custom image recipes</li>
<li>Building a custom kernel</li>
<li>Adapting existing recipes</li>
<li>Adding other applications to the image</li>
<li>Adding custom applications to the build system</li>
<li>Prividing an SDK</li>
<li>Providing a package repository</li>
</ul>
<p>Moreover, we don't want to clone each recipe manually each time we start
a new build. This is why I recommend automatising this with a <a href="https://gitlab.com/morandg/yocto-build-script">shell
script like this one</a>, a
Makefile, a python script or anything that suits to you.</p>
<h1>Conclusion</h1>
<p>You now know all the basics you need to build an existing image for
your target, that is the starting point when using Yocto. Customizing
the image is the fun part but I will keep this for another article, stay
tuned!</p>The quest to the perfect Makefile2018-10-17T19:00:00+02:002018-10-17T19:00:00+02:00R4nd0m 6uytag:r4nd0m6uy.ch,2018-10-17:/the-quest-to-the-perfect-makefile.html<p>I have been using <a href="https://cmake.org/">cmake</a> for a while because I
think writing good Makefile is a very tricky task. Unfortunately, cmake
is not always available and writing a Makefile is the only solution.
I decided to spend some time on writing a small Makefile structure that
I could use for …</p><p>I have been using <a href="https://cmake.org/">cmake</a> for a while because I
think writing good Makefile is a very tricky task. Unfortunately, cmake
is not always available and writing a Makefile is the only solution.
I decided to spend some time on writing a small Makefile structure that
I could use for new projects and would share my journey to this quest
with you.</p>
<p>The material of this article is also available on
<a href="https://gitlab.com/morandg/perfect-makefile">gitlab</a></p>
<h1>Requirements</h1>
<p>Here is what I expect from a well written Makefile:</p>
<ul>
<li>Takes well care of parallel builds</li>
<li>Makes easy to add a new source files</li>
<li>Makes cross compilation easy</li>
<li>Has fancy and debug output</li>
<li>Rebuilds only and everything that is required on changes</li>
<li>Can enable debug or release compilation mode</li>
<li>Builds the unit tests when requested</li>
<li>Finds depedencies to other libraries automatically</li>
<li>Generates sources automatically</li>
</ul>
<p>That may sound a lot but it should be the bare minimal to make it usable
in complex projects.</p>
<h1>The project</h1>
<p>To illustrate the usage of this Makefile, I will implement an
overengineered Fibonnaci numbers computation. I won't spend too much
time on the details here but this will give us some source files to
compile.</p>
<h1>The base structure</h1>
<p>Let's start writing the basic structure of our project with a entry
point and some interfaces. The tree looks like this:</p>
<div class="highlight"><pre><span></span><code>├── Makefile
└── src
├── FibonacciNumbersRecursed.cpp
├── FibonacciNumbersRecursed.hpp
├── IFibonacciNumbers.cpp
├── IFibonacciNumbers.hpp
└── main.cpp
</code></pre></div>
<p>We have an entry point in main.cpp, an interface to compute fibonacci
numbers and a first implementation that does nothing for now. Let's now
write a basic Makefile.</p>
<h1>The first Makefile</h1>
<p>First we define some useful variables that can be reused later:</p>
<div class="highlight"><pre><span></span><code><span class="nv">PROJECT</span> <span class="o">=</span> fibonacci
<span class="nv">BUILD_DIR</span> <span class="o">?=</span> build
</code></pre></div>
<p>The binary and the list of source files will also be saved in a
variable:</p>
<div class="highlight"><pre><span></span><code><span class="nv">APP_BIN</span> <span class="o">=</span> <span class="k">$(</span>BUILD_DIR<span class="k">)</span>/<span class="k">$(</span>PROJECT<span class="k">)</span>
<span class="nv">APP_SOURCES</span> <span class="o">=</span> src/IFibonacciNumbers.cpp <span class="se">\</span>
src/FibonacciNumbersRecursed.cpp <span class="se">\</span>
src/main.cpp
</code></pre></div>
<p>Here we create a list of compiled <em>.o</em> files from the list of sources,
that can be used as a prerequist list.</p>
<div class="highlight"><pre><span></span><code><span class="nv">APP_OBJS</span> <span class="o">=</span> <span class="k">$(</span>patsubst %.cpp,<span class="k">$(</span>BUILD_DIR<span class="k">)</span>/%.o,<span class="k">$(</span>APP_SOURCES<span class="k">))</span>
</code></pre></div>
<p>Define here all the compiler flags:</p>
<div class="highlight"><pre><span></span><code><span class="nv">COMMON_CFLAGS</span> <span class="o">=</span> -Wall -Werror -Wextra
<span class="nv">CFLAGS</span> <span class="o">+=</span> <span class="k">$(</span>COMMON_CFLAGS<span class="k">)</span>
<span class="nv">CXXFLAGS</span> <span class="o">+=</span> <span class="k">$(</span>COMMON_CFLAGS<span class="k">)</span> -std<span class="o">=</span>c++14
</code></pre></div>
<p>Now we have all the variables we need, we can start writing a default
target that is by convention <em>all</em>. It will simply build the main
binary:</p>
<div class="highlight"><pre><span></span><code><span class="nf">all</span><span class="o">:</span> <span class="k">$(</span><span class="nv">APP_BIN</span><span class="k">)</span>
<span class="nf">.PHONY</span><span class="o">:</span> <span class="n">all</span>
</code></pre></div>
<p>Now how to link the binary:</p>
<div class="highlight"><pre><span></span><code><span class="nf">$(APP_BIN)</span><span class="o">:</span> <span class="k">$(</span><span class="nv">APP_OBJS</span><span class="k">)</span>
<span class="k">$(</span>CXX<span class="k">)</span> -o <span class="nv">$@</span> <span class="k">$(</span>APP_OBJS<span class="k">)</span>
</code></pre></div>
<p>How each source will be built:</p>
<div class="highlight"><pre><span></span><code><span class="nf">$(BUILD_DIR)/%.o</span><span class="o">:</span> %.<span class="n">cpp</span>
mkdir -p <span class="k">$(</span>dir <span class="nv">$@</span><span class="k">)</span>
<span class="k">$(</span>CXX<span class="k">)</span> <span class="k">$(</span>CXXFLAGS<span class="k">)</span> -c $< -o <span class="nv">$@</span>
</code></pre></div>
<p>Finally a small clean target:</p>
<div class="highlight"><pre><span></span><code><span class="nf">clean</span><span class="o">:</span>
rm -rf <span class="k">$(</span>BUILD_DIR<span class="k">)</span>
<span class="nf">.PHONY</span><span class="o">:</span> <span class="n">clean</span>
</code></pre></div>
<h1>First make commands</h1>
<p>Let's give a first attempt to make to build our application:</p>
<div class="highlight"><pre><span></span><code>$ make
mkdir -p build/src/
g++ -Wall -Werror -Wextra -std<span class="o">=</span>c++14 -c src/IFibonacciNumbers.cpp -o build/src/IFibonacciNumbers.o
mkdir -p build/src/
g++ -Wall -Werror -Wextra -std<span class="o">=</span>c++14 -c src/FibonacciNumbersRecursed.cpp -o build/src/FibonacciNumbersRecursed.o
mkdir -p build/src/
g++ -Wall -Werror -Wextra -std<span class="o">=</span>c++14 -c src/main.cpp -o build/src/main.o
g++ -o build/fibonacci build/src/IFibonacciNumbers.o build/src/FibonacciNumbersRecursed.o build/src/main.o
$ ./build/fibonacci
Computing using implementation recursive
Fibonacci number of <span class="m">3</span> is ... <span class="m">3</span>
</code></pre></div>
<p>The application is not implemented but at least it generates a working
executable. Let's see what goals we have achieved until now.</p>
<h1>Takes well care of parallel builds</h1>
<p>This is important to build the project faster by making use of all CPUs.
We can check if make behaves correctly with the <em>-j</em> option. After
cleaning and rebuilding the project with different jobs, it always
succeeds. We can also see that the sources files are built in parallel:</p>
<div class="highlight"><pre><span></span><code>$ make
$ make clean
$ make -j2
$ make clean
$ make -j3
$ ...
</code></pre></div>
<h1>Makes easy to add a new source file</h1>
<p>When adding a new source file, the Makefile should have as least changes
as possible. To demonstrate this, we simply add a new Fibonacci number
implementation and update the source list:</p>
<div class="highlight"><pre><span></span><code> APP_BIN = $(BUILD_DIR)/$(PROJECT)
APP_SOURCES = src/IFibonacciNumbers.cpp \
src/FibonacciNumbersRecursed.cpp \
<span class="gi">+ src/FibonacciNumbersDynamic.cpp \</span>
src/main.cpp
APP_OBJS = $(patsubst %.cpp,$(BUILD_DIR)/%.o,$(APP_SOURCES))
</code></pre></div>
<p>The application is updated in order to be able to choose the
implementation at runtime. We can now check that it works as expected:</p>
<div class="highlight"><pre><span></span><code>$ make
<span class="c1"># ...</span>
$ ./build/fibonacci -t <span class="m">0</span>
Computing using implementation recursive
Fibonacci number of <span class="m">3</span> is ... <span class="m">3</span>
$ ./build/fibonacci -t <span class="m">1</span>
Computing using implementation dynamic
Fibonacci number of <span class="m">3</span> is ... <span class="m">3</span>
</code></pre></div>
<p>Some poeple would prefere having a function that automatically searches
for all the source files with something like this:</p>
<div class="highlight"><pre><span></span><code><span class="nv">SOURCES</span> <span class="o">=</span> <span class="k">$(</span>shell find src -name <span class="s1">'*.cpp'</span><span class="k">)</span>
</code></pre></div>
<p>However, we lose a bit of control on which source file we want to
include in the build in case we support different platforms which don't
use the same sources files. This is why I'm fine with adding a line in
the Makefile when I add a new source file in the project for more
control on what is being built.</p>
<h1>Makes cross compilation easy</h1>
<p>When using multiple hardware, it should be easy to build for another
architecture with the minimal effort. In our case, cross compiling can
be done by defining another build directory and compiler on the command
line:</p>
<div class="highlight"><pre><span></span><code>$ make <span class="nv">BUILD_DIR</span><span class="o">=</span>build_arm <span class="nv">CXX</span><span class="o">=</span>arm-linux-g++
<span class="c1"># ...</span>
</code></pre></div>
<h1>Has fancy and debug output</h1>
<p>We don't want make to print the executed commands that is very verbose,
this can be done by adding the <em>@</em> character at the beginning of the
command. When something goes wrong, we need to see what is done. This
should be enabled with an environment variable.</p>
<p>First we define an environement variable to enable the verbose mode:</p>
<div class="highlight"><pre><span></span><code><span class="cp">ifneq ($(V),)</span>
<span class="nv">SILENCE</span> <span class="o">=</span>
<span class="cp">else</span>
<span class="nv">SILENCE</span> <span class="o">=</span> @
<span class="cp">endif</span>
</code></pre></div>
<p>Update all the recipes consequently:</p>
<div class="highlight"><pre><span></span><code> $(APP_BIN): $(APP_OBJS)
<span class="gd">- $(CXX) -o $@ $(APP_OBJS)</span>
<span class="gi">+ $(SILENCE)$(CXX) -o $@ $(APP_OBJS)</span>
$(BUILD_DIR)/%.o: %.cpp
<span class="gd">- mkdir -p $(dir $@)</span>
<span class="gd">- $(CXX) $(CXXFLAGS) -c $< -o $@</span>
<span class="gi">+ $(SILENCE)mkdir -p $(dir $@)</span>
<span class="gi">+ $(SILENCE)$(CXX) $(CXXFLAGS) -c $< -o $@</span>
clean:
<span class="gd">- rm -rf $(BUILD_DIR)</span>
<span class="gi">+ $(SILENCE)rm -rf $(BUILD_DIR)</span>
.PHONY: clean
</code></pre></div>
<p>Now we can enable the verbose mode by setting the <em>V</em> environment
variable to any value:</p>
<div class="highlight"><pre><span></span><code>make <span class="nv">V</span><span class="o">=</span><span class="m">1</span>
<span class="c1"># ...</span>
</code></pre></div>
<p>Unfortunately, in silent mode, we don't see anything:</p>
<div class="highlight"><pre><span></span><code>$ make
$
</code></pre></div>
<p>Some feedback is always good in order to know what happens behind the
hood. However, we can print something nicer than the build command and
use for that some <em>printf</em> that can be reused:</p>
<div class="highlight"><pre><span></span><code><span class="nv">SHOW_COMMAND</span> <span class="o">:=</span> @printf <span class="s2">"%-15s%s\n"</span>
<span class="nv">SHOW_CXX</span> <span class="o">:=</span> <span class="k">$(</span>SHOW_COMMAND<span class="k">)</span> <span class="s2">"[ </span><span class="k">$(</span>CXX<span class="k">)</span><span class="s2"> ]"</span>
<span class="nv">SHOW_CLEAN</span> <span class="o">:=</span> <span class="k">$(</span>SHOW_COMMAND<span class="k">)</span> <span class="s2">"[ CLEAN ]"</span>
</code></pre></div>
<p>Update the recipe to print the status of the build:</p>
<div class="highlight"><pre><span></span><code> $(APP_BIN): $(APP_OBJS)
<span class="gi">+ $(SHOW_CXX) $@</span>
$(SILENCE)$(CXX) -o $@ $(APP_OBJS)
$(BUILD_DIR)/%.o: %.cpp
<span class="gi">+ $(SHOW_CXX) $@</span>
$(SILENCE)mkdir -p $(dir $@)
$(SILENCE)$(CXX) $(CXXFLAGS) -c $< -o $@
clean:
<span class="gi">+ $(SHOW_CLEAN) $(BUILD_DIR)</span>
$(SILENCE)rm -rf $(BUILD_DIR)
.PHONY: clean
</code></pre></div>
<p>Now the bulid is more verbose but still very comfortable to watch:</p>
<div class="highlight"><pre><span></span><code>$ make
<span class="o">[</span> g++ <span class="o">]</span> build/src/IFibonacciNumbers.o
<span class="o">[</span> g++ <span class="o">]</span> build/src/FibonacciNumbersRecursed.o
<span class="o">[</span> g++ <span class="o">]</span> build/src/FibonacciNumbersDynamic.o
<span class="o">[</span> g++ <span class="o">]</span> build/src/main.o
<span class="o">[</span> g++ <span class="o">]</span> build/fibonacci
$ make clean
<span class="o">[</span> CLEAN <span class="o">]</span> build
</code></pre></div>
<p>Nice!</p>
<h1>Rebuilds only and everything that is required on changes</h1>
<p>This is very important for big projects, we don't want to recompile
everything if we changed only one source file. When using unit tests, we
want to have a feedback aboout our changes as fast as possible, to
achieve this we need to recompile only the files that changed.</p>
<p>As a first test, we will check what happens if we update a source file:</p>
<div class="highlight"><pre><span></span><code>$ touch src/IFibonacciNumbers.cpp
$ make
<span class="o">[</span> g++ <span class="o">]</span> build/src/IFibonacciNumbers.o
<span class="o">[</span> g++ <span class="o">]</span> build/fibonacci
</code></pre></div>
<p>The file gets compiled and the binary linked again, this seems correct.
However, having a nice depedency of headers is more difficult to
maintain as demonstrated here:</p>
<div class="highlight"><pre><span></span><code>$ touch src/IFibonacciNumbers.hpp
$ make
make: Nothing to be <span class="k">done</span> <span class="k">for</span> <span class="s1">'all'</span>.
</code></pre></div>
<p>We would expect <em>main.cpp</em> and <em>IFibonacciNumbers.cpp</em> to be compiled
again as those include the modified header. We could manually list the
header depedency for each source file like this:</p>
<div class="highlight"><pre><span></span><code><span class="nf">main.o</span><span class="o">:</span> <span class="n">main</span>.<span class="n">cpp</span> <span class="n">IFibonacciNumbers</span>.<span class="n">hpp</span>
<span class="nf">IFibonacciNumbers.o</span><span class="o">:</span> <span class="n">IFibonacciNumbers</span>.<span class="n">cpp</span> <span class="n">IFibonacciNumbers</span>.<span class="n">hpp</span>
<span class="c"># ...</span>
</code></pre></div>
<p>But this is a nightmare to maintain and is source of mistakes when the
project evolve! Fortunately, the compiler is able to generate this list
of depedencies automatically with the <em>-MMD</em> argument that will generate
a file with the .d extension. First we add this flags:</p>
<div class="highlight"><pre><span></span><code><span class="gd">-COMMON_CFLAGS = -Wall -Werror -Wextra</span>
<span class="gi">+COMMON_CFLAGS = -Wall -Werror -Wextra -MMD</span>
</code></pre></div>
<p>Out of curiosity, let's have a look at the list of generated files:</p>
<div class="highlight"><pre><span></span><code>$ make
<span class="c1"># ...</span>
$ find build/ <span class="p">|</span> grep <span class="se">\\</span>.d<span class="se">\$</span>
build/src/main.d
build/src/FibonacciNumbersRecursed.d
build/src/FibonacciNumbersDynamic.d
build/src/IFibonacciNumbers.d
$ cat build/src/main.d
build/src/main.o: src/main.cpp src/IFibonacciNumbers.hpp
</code></pre></div>
<p>A rule is automatically generated with the list of headers, exactly what
we need! We must now include those rules in our Makfile by defining a
list of generated depedency files, that is the list of objects with the
.d extension:</p>
<div class="highlight"><pre><span></span><code><span class="nv">DEPS</span> <span class="o">=</span> <span class="k">$(</span>APP_OBJS:.o<span class="o">=</span>.d<span class="k">)</span>
</code></pre></div>
<p>And include them right after the default target:</p>
<div class="highlight"><pre><span></span><code><span class="nf">all</span><span class="o">:</span> <span class="k">$(</span><span class="nv">APP_BIN</span><span class="k">)</span>
<span class="nf">.PHONY</span><span class="o">:</span> <span class="n">all</span>
<span class="cp">-include $(DEPS)</span>
</code></pre></div>
<p>Now verify that it works as expected when modifying different header:</p>
<div class="highlight"><pre><span></span><code>$ touch src/IFibonacciNumbers.hpp
$ make
<span class="o">[</span> g++ <span class="o">]</span> build/src/IFibonacciNumbers.o
<span class="o">[</span> g++ <span class="o">]</span> build/src/FibonacciNumbersRecursed.o
<span class="o">[</span> g++ <span class="o">]</span> build/src/FibonacciNumbersDynamic.o
<span class="o">[</span> g++ <span class="o">]</span> build/src/main.o
<span class="o">[</span> g++ <span class="o">]</span> build/fibonacci
$ touch src/FibonacciNumbersDynamic.hpp
$ make
<span class="o">[</span> g++ <span class="o">]</span> build/src/IFibonacciNumbers.o
<span class="o">[</span> g++ <span class="o">]</span> build/src/FibonacciNumbersDynamic.o
<span class="o">[</span> g++ <span class="o">]</span> build/fibonacci
</code></pre></div>
<p>Much better!</p>
<h1>Can enable debug or release compilation mode</h1>
<p>We often need a debug version during the development but the
application should be shipped in release mode. To switch from a mode to
another, we will use an environement variable and change the flags
consequently. We also change the build directory to not mix objects
compiled with different flags:</p>
<div class="highlight"><pre><span></span><code><span class="nv">COMMON_CFLAGS</span> <span class="o">=</span> -Wall -Werror -Wextra -MMD
<span class="cp">ifneq ($(DEBUG),)</span>
<span class="nv">COMMON_CFLAGS</span> <span class="o">+=</span> -g
BUILD_DIR :<span class="o">=</span> <span class="k">$(</span>BUILD_DIR<span class="k">)</span>/debug
<span class="cp">else</span>
<span class="nv">COMMON_CFLAGS</span> <span class="o">+=</span> -DNDEBUG -O3
BUILD_DIR :<span class="o">=</span> <span class="k">$(</span>BUILD_DIR<span class="k">)</span>/release
<span class="cp">endif</span>
<span class="nv">CFLAGS</span> <span class="o">+=</span> <span class="k">$(</span>COMMON_CFLAGS<span class="k">)</span>
<span class="nv">CXXFLAGS</span> <span class="o">+=</span> <span class="k">$(</span>COMMON_CFLAGS<span class="k">)</span> -std<span class="o">=</span>c++14
</code></pre></div>
<p>To verify that the flags are passed correctly we run make in verbose
mode:</p>
<div class="highlight"><pre><span></span><code>$ make <span class="nv">V</span><span class="o">=</span><span class="m">1</span>
g++ -Wall -Werror -Wextra -MMD -DNDEBUG -O3 -std<span class="o">=</span>c++14 -c src/IFibonacciNumbers.cpp -o build/release/src/IFibonacciNumbers.o
$ make <span class="nv">V</span><span class="o">=</span><span class="m">1</span> <span class="nv">DEBUG</span><span class="o">=</span><span class="m">1</span>
g++ -Wall -Werror -Wextra -MMD -g -std<span class="o">=</span>c++14 -c src/IFibonacciNumbers.cpp -o build/debug/src/IFibonacciNumbers.o
</code></pre></div>
<h1>Builds the unit tests when requested</h1>
<p>During the development, it makes a lot of sense to build and execute the
unit tests. For an end user, this is probably not required and should
be skippable.</p>
<p>I like the <a href="https://cpputest.github.io/">CPPUTest</a> test framework and
will use it here. If you use another testing framework, the idea is the
same, you would need to adapt the flags consequently. First we write
some tests source files that contains an entry point and a test that
fails:</p>
<div class="highlight"><pre><span></span><code>.
└── tests
├── FibonacciNumbersRecursedTest.cpp
└── main.cpp
</code></pre></div>
<p>Now we need to remove <em>main.cpp</em> from the applications sources list to
not have two entry points when linking the tests:</p>
<div class="highlight"><pre><span></span><code> APP_SOURCES = src/IFibonacciNumbers.cpp \
src/FibonacciNumbersRecursed.cpp \
<span class="gd">- src/FibonacciNumbersDynamic.cpp \</span>
<span class="gd">- src/main.cpp</span>
<span class="gd">-APP_OBJS = $(patsubst %.cpp,$(BUILD_DIR)/%.o,$(APP_SOURCES))</span>
<span class="gi">+ src/FibonacciNumbersDynamic.cpp</span>
<span class="gi">+APP_MAIN = src/main.cpp</span>
<span class="gi">+APP_OBJS = $(patsubst %.cpp,$(BUILD_DIR)/%.o,$(APP_SOURCES) $(APP_MAIN))</span>
</code></pre></div>
<p>We list the files required for the test the same manner as for the
application. We recompile the application sources because CppUTest adds
some flags in order to enable the memory leak detection:</p>
<div class="highlight"><pre><span></span><code><span class="nv">TEST_BIN</span> <span class="o">=</span> <span class="k">$(</span>BUILD_DIR<span class="k">)</span>/<span class="k">$(</span>PROJECT<span class="k">)</span>_tests
<span class="nv">TEST_SOURCES</span> <span class="o">=</span> <span class="k">$(</span>APP_SOURCES<span class="k">)</span> <span class="se">\</span>
tests/FibonacciNumbersRecursedTest.cpp <span class="se">\</span>
tests/main.cpp
<span class="nv">TEST_OBJS</span> <span class="o">=</span> <span class="k">$(</span>patsubst %.cpp,<span class="k">$(</span>BUILD_DIR<span class="k">)</span>/tests/%.o,<span class="k">$(</span>TEST_SOURCES<span class="k">))</span>
</code></pre></div>
<p>Do not forget to add the tests source to the depedency list:</p>
<div class="highlight"><pre><span></span><code><span class="gd">-DEPS = $(APP_OBJS:.o=.d)</span>
<span class="gi">+DEPS = $(APP_OBJS:.o=.d) \</span>
<span class="gi">+ $(TEST_OBJS:.o=.d)</span>
</code></pre></div>
<p>We need to add the flags required for CppUTest. We let the user defining
a directory where cpputest is installed, otherwise we use
<a href="https://www.freedesktop.org/wiki/Software/pkg-config/">pkg-config</a> to
detect the flags automatically.</p>
<div class="highlight"><pre><span></span><code><span class="cp">ifneq ($(CPPUTEST_HOME),)</span>
CPPUTEST_FLAGS :<span class="o">=</span> -I<span class="k">$(</span>CPPUTEST_HOME<span class="k">)</span>/include
CPPUTEST_LDFLAGS :<span class="o">=</span> -L<span class="k">$(</span>CPPUTEST_HOME<span class="k">)</span>/lib -lCppUTest -lCppUTestExt
<span class="cp">else</span>
CPPUTEST_FLAGS :<span class="o">=</span> <span class="k">$(</span>shell pkg-config --cflags cpputest <span class="m">2</span>>/dev/null<span class="k">)</span>
CPPUTEST_LDFLAGS :<span class="o">=</span> <span class="k">$(</span>shell pkg-config --libs cpputest <span class="m">2</span>>/dev/null<span class="k">)</span>
<span class="cp">endif</span>
</code></pre></div>
<p>Add a target to link the test application with the right flags:</p>
<div class="highlight"><pre><span></span><code> <span class="k">$(</span>TEST_BIN<span class="k">)</span>: <span class="k">$(</span>TEST_OBJS<span class="k">)</span>
<span class="k">$(</span>SHOW_CXX<span class="k">)</span> <span class="nv">$@</span>
<span class="k">$(</span>SILENCE<span class="k">)$(</span>CXX<span class="k">)</span> <span class="k">$(</span>TEST_OBJS<span class="k">)</span> <span class="k">$(</span>CPPUTEST_LDFLAGS<span class="k">)</span> -o <span class="nv">$@</span>
<span class="nf">$(BUILD_DIR)/tests/%.o</span><span class="o">:</span> %.<span class="n">cpp</span>
<span class="k">$(</span>SHOW_CXX<span class="k">)</span> <span class="nv">$@</span>
<span class="k">$(</span>SILENCE<span class="k">)</span>mkdir -p <span class="k">$(</span>dir <span class="nv">$@</span><span class="k">)</span>
<span class="k">$(</span>SILENCE<span class="k">)$(</span>CXX<span class="k">)</span> <span class="k">$(</span>CXXFLAGS<span class="k">)</span> <span class="k">$(</span>CPPUTEST_FLAGS<span class="k">)</span> -c $< -o <span class="nv">$@</span>
</code></pre></div>
<p>For convenience, add a phony target to build and run the tests. If the
test application fails, it should be removed and the make command should
fail. We want to keep the test application for debugging, in that case
we can use the <em>tests</em> target with make, the <em>run_tests</em> target will run
them automatically and can be used 99% of the time:</p>
<div class="highlight"><pre><span></span><code><span class="nf">tests</span><span class="o">:</span> <span class="k">$(</span><span class="nv">TEST_BIN</span><span class="k">)</span>
<span class="nf">.PHONY</span><span class="o">:</span> <span class="n">tests</span>
<span class="nf">run_tests</span><span class="o">:</span> <span class="k">$(</span><span class="nv">BUILD_DIR</span><span class="k">)</span>/.<span class="n">tests_passed</span>
<span class="nf">.PHONY</span><span class="o">:</span> <span class="n">run_tests</span>
<span class="nf">$(BUILD_DIR)/.tests_passed</span><span class="o">:</span> <span class="k">$(</span><span class="nv">TEST_BIN</span><span class="k">)</span>
<span class="k">$(</span>SILENCE<span class="k">)</span>./$< <span class="o">||</span> rm $<
<span class="k">$(</span>SILENCE<span class="k">)</span>touch <span class="nv">$@</span>
</code></pre></div>
<p>Now we can check that the tests are built and run correctly:</p>
<div class="highlight"><pre><span></span><code>$ make tests
<span class="o">[</span> g++ <span class="o">]</span> build/release/tests/src/IFibonacciNumbers.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/tests/src/FibonacciNumbersRecursed.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/tests/src/FibonacciNumbersDynamic.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/tests/tests/FibonacciNumbersRecursedTest.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/tests/tests/main.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/fibonacci_tests
$ make run_tests
tests/FibonacciNumbersRecursedTest.cpp:54: error: Failure <span class="k">in</span> TEST<span class="o">(</span>FibonacciRecursed, Fibo0Is0<span class="o">)</span>
CHECK<span class="o">(</span><span class="nb">false</span><span class="o">)</span> failed
.
Errors <span class="o">(</span><span class="m">1</span> failures, <span class="m">1</span> tests, <span class="m">1</span> ran, <span class="m">1</span> checks, <span class="m">0</span> ignored, <span class="m">0</span> filtered out, <span class="m">0</span> ms<span class="o">)</span>
</code></pre></div>
<p>Now that we have a structure that works, we can implement our fibonacci
numbers with TDD and check that it works as expected:</p>
<div class="highlight"><pre><span></span><code>$ make run_tests
<span class="o">[</span> g++ <span class="o">]</span> build/release/fibonacci_tests
.......
OK <span class="o">(</span><span class="m">7</span> tests, <span class="m">7</span> ran, <span class="m">7</span> checks, <span class="m">0</span> ignored, <span class="m">0</span> filtered out, <span class="m">0</span> ms<span class="o">)</span>
$ ./build/release/fibonacci -n <span class="m">45</span> -t0
Computing using implementation recursive
Fibonacci number of <span class="m">45</span> is ... <span class="m">1134903170</span>
$ ./build/release/fibonacci -n <span class="m">45</span> -t1
Computing using implementation dynamic
Fibonacci number of <span class="m">45</span> is ... <span class="m">1134903170</span>
</code></pre></div>
<h1>Finds depedencies automatically</h1>
<p>When using external libraries, compiler flags should be added. If it is
not found, a clear error message should be printed to inform the user
what is wrong.</p>
<p>The CppUTest depedency is a good point to start and we can improve the
current implementation by building the tests automatically if CppUTest
is found.</p>
<p>We will change a bit the Makefile to assume that CppUTest is found when
the user gives a path:</p>
<div class="highlight"><pre><span></span><code><span class="cp">ifneq ($(CPPUTEST_HOME),)</span>
<span class="nv">HAS_CPPUTEST</span> <span class="o">=</span> <span class="m">1</span>
<span class="nv">CPPUTEST_FLAGS</span> <span class="o">=</span> -I<span class="k">$(</span>CPPUTEST_HOME<span class="k">)</span>/include
<span class="nv">CPPUTEST_LDFLAGS</span> <span class="o">=</span> -L<span class="k">$(</span>CPPUTEST_HOME<span class="k">)</span>/lib -lCppUTest -lCppUTestExt
</code></pre></div>
<p>Otherwise we use pkg-config return code to check if the package was
found on the system:</p>
<div class="highlight"><pre><span></span><code><span class="cp">else</span>
<span class="nv">HAS_CPPUTEST</span> <span class="o">=</span> <span class="k">$(</span>shell pkg-config cpputest <span class="o">&&</span> <span class="nb">echo</span> <span class="m">1</span><span class="k">)</span>
<span class="cp"> ifeq ($(HAS_CPPUTEST),1)</span>
<span class="nv">CPPUTEST_FLAGS</span> <span class="o">=</span> <span class="k">$(</span>shell pkg-config --cflags cpputest <span class="m">2</span>>/dev/null<span class="k">)</span>
<span class="nv">CPPUTEST_LDFLAGS</span> <span class="o">=</span> <span class="k">$(</span>shell pkg-config --libs cpputest <span class="m">2</span>>/dev/null<span class="k">)</span>
<span class="cp"> endif</span>
<span class="cp">endif</span>
</code></pre></div>
<p>We will change the defaut target to run the tests automatically if
CppUTest is present:</p>
<div class="highlight"><pre><span></span><code><span class="nv">DEFAULT_TARGET</span> <span class="o">=</span> <span class="k">$(</span>APP_BIN<span class="k">)</span>
<span class="cp">ifeq ($(HAS_CPPUTEST),1)</span>
<span class="nv">DEFAULT_TARGET</span> <span class="o">+=</span> run_tests
<span class="cp">endif</span>
<span class="nf">all</span><span class="o">:</span> <span class="k">$(</span><span class="nv">DEFAULT_TARGET</span><span class="k">)</span>
<span class="nf">.PHONY</span><span class="o">:</span> <span class="n">all</span>
</code></pre></div>
<p>To avoid the user trying to build the test when CppUTest is not found,
we will throw an error message instead:</p>
<div class="highlight"><pre><span></span><code><span class="nf">$(BUILD_DIR)/tests/%.o</span><span class="o">:</span> %.<span class="n">cpp</span>
<span class="cp">ifneq ($(HAS_CPPUTEST),1)</span>
<span class="k">$(</span>error CppUTest not found, cannot build the tests<span class="k">)</span>
<span class="cp">endif</span>
<span class="k">$(</span>SHOW_CXX<span class="k">)</span> <span class="nv">$@</span>
<span class="k">$(</span>SILENCE<span class="k">)</span>mkdir -p <span class="k">$(</span>dir <span class="nv">$@</span><span class="k">)</span>
<span class="k">$(</span>SILENCE<span class="k">)$(</span>CXX<span class="k">)</span> <span class="k">$(</span>CXXFLAGS<span class="k">)</span> <span class="k">$(</span>CPPUTEST_FLAGS<span class="k">)</span> -c $< -o <span class="nv">$@</span>
</code></pre></div>
<p>Let's compile the application without CppUTest support:</p>
<div class="highlight"><pre><span></span><code>$ make
<span class="o">[</span> g++ <span class="o">]</span> build/release/src/IFibonacciNumbers.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/src/FibonacciNumbersRecursed.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/src/FibonacciNumbersDynamic.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/src/main.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/fibonacci
$ make run_tests
Makefile:110: *** CppUTest not found, cannot build the tests. Stop.
</code></pre></div>
<p>Now the same test with a <em>CPPUTEST_HOME</em> defined:</p>
<div class="highlight"><pre><span></span><code>$ <span class="nb">export</span> <span class="nv">CPPUTEST_HOME</span><span class="o">=</span>/path/to/cpputest/
$ make
<span class="o">[</span> g++ <span class="o">]</span> build/release/src/IFibonacciNumbers.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/src/FibonacciNumbersRecursed.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/src/FibonacciNumbersDynamic.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/src/main.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/fibonacci
<span class="o">[</span> g++ <span class="o">]</span> build/release/tests/src/IFibonacciNumbers.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/tests/src/FibonacciNumbersRecursed.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/tests/src/FibonacciNumbersDynamic.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/tests/tests/FibonacciNumbersRecursedTest.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/tests/tests/FibonacciNumbersDynamicTest.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/tests/tests/main.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/fibonacci_tests
.......
OK <span class="o">(</span><span class="m">7</span> tests, <span class="m">7</span> ran, <span class="m">7</span> checks, <span class="m">0</span> ignored, <span class="m">0</span> filtered out, <span class="m">0</span> ms<span class="o">)</span>
</code></pre></div>
<p>And with pkg-config method:</p>
<div class="highlight"><pre><span></span><code>$ <span class="nb">export</span> <span class="nv">PKG_CONFIG_PATH</span><span class="o">=</span>/path/to/cpputest.pc/
$ make
<span class="o">[</span> g++ <span class="o">]</span> build/release/src/IFibonacciNumbers.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/src/FibonacciNumbersRecursed.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/src/FibonacciNumbersDynamic.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/src/main.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/fibonacci
<span class="o">[</span> g++ <span class="o">]</span> build/release/tests/src/IFibonacciNumbers.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/tests/src/FibonacciNumbersRecursed.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/tests/src/FibonacciNumbersDynamic.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/tests/tests/FibonacciNumbersRecursedTest.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/tests/tests/FibonacciNumbersDynamicTest.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/tests/tests/main.o
<span class="o">[</span> g++ <span class="o">]</span> build/release/fibonacci_tests
.......
OK <span class="o">(</span><span class="m">7</span> tests, <span class="m">7</span> ran, <span class="m">7</span> checks, <span class="m">0</span> ignored, <span class="m">0</span> filtered out, <span class="m">0</span> ms<span class="o">)</span>
</code></pre></div>
<h2>Depedencies with other libraries</h2>
<p>This can be handled the same manner using pkg-config, here for example
detecting if zlib is available:</p>
<div class="highlight"><pre><span></span><code><span class="nv">HAS_ZLIB</span> <span class="o">=</span> <span class="k">$(</span>shell pkg-config zlib <span class="o">&&</span> <span class="nb">echo</span> <span class="m">1</span><span class="k">)</span>
<span class="cp">ifeq ($(HAS_ZLIB),1)</span>
<span class="nv">ZLIB_FLAGS</span> <span class="o">=</span> <span class="k">$(</span>shell pkg-config --cflags zlib <span class="m">2</span>>/dev/null<span class="k">)</span>
<span class="nv">ZLIB_LDFLAGS</span> <span class="o">=</span> <span class="k">$(</span>shell pkg-config --libs zlib <span class="m">2</span>>/dev/null<span class="k">)</span>
<span class="cp">endif</span>
</code></pre></div>
<p>In case the library is required, we can fail and write a clear error
message:</p>
<div class="highlight"><pre><span></span><code><span class="cp">ifeq ($(HAS_ZLIB),1)</span>
<span class="c"> #...</span>
<span class="cp">else</span>
<span class="k">$(</span>error Cannot build application, zlib support not found!<span class="k">)</span>
<span class="cp">endif</span>
</code></pre></div>
<p>Or maybe the depedency is optional and new sources need to be added. It
is possible to define a macro that can be used in the sources to detect
if zlib is supported:</p>
<div class="highlight"><pre><span></span><code><span class="cp">ifeq ($(HAS_ZLIB),1)</span>
<span class="nv">ZLIB_FLAGS</span> <span class="o">=</span> <span class="k">$(</span>shell pkg-config --cflags zlib <span class="m">2</span>>/dev/null<span class="k">)</span> -D_HAS_ZLIB_
<span class="nv">ZLIB_LDFLAGS</span> <span class="o">=</span> <span class="k">$(</span>shell pkg-config --libs zlib <span class="m">2</span>>/dev/null
<span class="nv">APP_SOURCES</span> <span class="o">+=</span> src/Sources.cpp <span class="se">\</span>
src/Using.cpp <span class="se">\</span>
src/Zlib.cpp
<span class="cp">endif</span>
</code></pre></div>
<p>zilb detection can be used as follow in a source file:</p>
<div class="highlight"><pre><span></span><code><span class="cp">#ifdef _HAS_ZLIB_</span>
<span class="cp">#include</span> <span class="cpf"><zlib.h></span><span class="cp"></span>
<span class="cp">#endif</span>
<span class="c1">// ...</span>
<span class="kt">void</span> <span class="nf">foo</span><span class="p">()</span>
<span class="p">{</span>
<span class="cp">#ifdef _HAS_ZLIB_</span>
<span class="c1">// Something with zlib</span>
<span class="cp">#endif</span>
<span class="p">}</span>
</code></pre></div>
<p>Just keep in mind that using macros decreases the readibility, it should
be concentrated in one place like factory methods and not spread around
the projects.</p>
<h1>Generates other sources automatically</h1>
<p>This can be required when you are using a source code generator such as
<a href="https://developers.google.com/protocol-buffers/">protocol buffer</a>, if
you want to generate .pc files for pkg-config when building a library
and many more. In this article, we will simply generate a version file
that will contain the usual major, minor, bugfix version with also the
git hash.</p>
<p>First we create a template called <em>Version.hpp.in</em> that looks like this:</p>
<div class="highlight"><pre><span></span><code><span class="cp">#ifndef _FIBO_VERSION_HPP_</span>
<span class="cp">#define _FIBO_VERSION_HPP_</span>
<span class="cp">#include</span> <span class="cpf"><string></span><span class="cp"></span>
<span class="k">namespace</span> <span class="nn">fibo</span> <span class="p">{</span>
<span class="k">static</span> <span class="k">const</span> <span class="kt">unsigned</span> <span class="kt">int</span> <span class="n">VERSION_MAJOR</span> <span class="o">=</span> <span class="o">%%</span><span class="n">VERSION_MAJOR</span><span class="o">%%</span><span class="p">;</span>
<span class="k">static</span> <span class="k">const</span> <span class="kt">unsigned</span> <span class="kt">int</span> <span class="n">VERSION_MINOR</span> <span class="o">=</span> <span class="o">%%</span><span class="n">VERSION_MINOR</span><span class="o">%%</span><span class="p">;</span>
<span class="k">static</span> <span class="k">const</span> <span class="kt">unsigned</span> <span class="kt">int</span> <span class="n">VERSION_BUGFIX</span> <span class="o">=</span> <span class="o">%%</span><span class="n">VERSION_BUGFIX</span><span class="o">%%</span><span class="p">;</span>
<span class="k">static</span> <span class="k">const</span> <span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="n">VERSION_GIT</span> <span class="o">=</span> <span class="s">"%%VERSION_GIT%%"</span><span class="p">;</span>
<span class="p">}</span> <span class="c1">// namespace</span>
<span class="cp">#endif </span><span class="c1">// _FIBO_VERSION_HPP_</span>
</code></pre></div>
<p>The version variables will be defined inside the Makefile</p>
<div class="highlight"><pre><span></span><code><span class="c"># Version information</span>
<span class="nv">VERSION_MAJOR</span> <span class="o">=</span> <span class="m">1</span>
<span class="nv">VERSION_MINOR</span> <span class="o">=</span> <span class="m">2</span>
<span class="nv">VERSION_BUGFIX</span> <span class="o">=</span> <span class="m">3</span>
<span class="nv">VERSION_GIT</span> <span class="o">:=</span> <span class="k">$(</span>shell git describe --tag --always --abbrev<span class="o">=</span><span class="m">5</span> --dirty<span class="k">)</span>
</code></pre></div>
<p>Then with <em>sed</em>, we can replace the placeholders of the variables:</p>
<div class="highlight"><pre><span></span><code><span class="nf">src/Version.hpp</span><span class="o">:</span> <span class="n">src</span>/<span class="n">Version</span>.<span class="n">hpp</span>.<span class="n">in</span> <span class="n">Makefile</span> <span class="k">$(</span><span class="nv">APP_SOURCES</span><span class="k">)</span>
<span class="k">$(</span>GEN_INFO<span class="k">)</span> <span class="nv">$@</span>
<span class="k">$(</span>SILENCE<span class="k">)</span>sed -e<span class="s1">'s/%%VERSION_MAJOR%%/$(VERSION_MAJOR)/g'</span> <span class="se">\</span>
-e <span class="s1">'s/%%VERSION_MINOR%%/$(VERSION_MINOR)/g'</span> <span class="se">\</span>
-e <span class="s1">'s/%%VERSION_BUGFIX%%/$(VERSION_BUGFIX)/g'</span> <span class="se">\</span>
-e <span class="s1">'s/%%VERSION_GIT%%/$(VERSION_GIT)/g'</span> <span class="se">\</span>
$< > <span class="nv">$@</span>
</code></pre></div>
<p>Don't forget to add depedency to the Version.hpp file so it gets
generated:</p>
<div class="highlight"><pre><span></span><code><span class="gd">-$(APP_BIN): $(APP_OBJS)</span>
<span class="gi">+$(APP_BIN): src/Version.hpp $(APP_OBJS)</span>
</code></pre></div>
<p>and remove the generated file in the clean target:</p>
<div class="highlight"><pre><span></span><code> clean:
$(SHOW_CLEAN) $(BUILD_DIR)
<span class="gd">- $(SILENCE)rm -rf $(BUILD_DIR)</span>
<span class="gi">+ $(SILENCE)rm -rf $(BUILD_DIR) src/Version.hpp</span>
</code></pre></div>
<p>Finally, we update the application to also display the version
information with the <em>-v</em> argument and check if it works:</p>
<div class="highlight"><pre><span></span><code>$ ./build/release/fibonacci -v
<span class="m">1</span>.2.3 <span class="o">(</span>3a5e3-dirty<span class="o">)</span>
</code></pre></div>
<h1>Conlusion</h1>
<p>There is still room for improvements, maybe this would deserve another
article as this one is getting slightly long... At least we have reach
our goals and most of the usual tasks are covered to build a complex
project with make. Stay tuned for further improvements!</p>
<p>As a reminder, the source is available on
<a href="https://gitlab.com/morandg/perfect-makefile">gitlab</a> do not hesitate to
take, share, improve, ...</p>Managing manually compiled packages with stow2018-07-18T12:00:00+02:002018-07-18T12:00:00+02:00R4nd0m 6uytag:r4nd0m6uy.ch,2018-07-18:/managing-manually-compiled-packages-with-stow.html<p>I often compile applications (mostly game) manually because I love using
the very last features and seeing the progress of the development! I
usually install everything in <em>/usr/local/</em> to avoid polluting my base
system as much as possible. Unfortunately it is difficult to make sur
that all files are …</p><p>I often compile applications (mostly game) manually because I love using
the very last features and seeing the progress of the development! I
usually install everything in <em>/usr/local/</em> to avoid polluting my base
system as much as possible. Unfortunately it is difficult to make sur
that all files are correcly removed when uninstalling the package
manually and miserably fail keeping a sane system!</p>
<p>While surfing the net, I've found this beast and though "why the hell
didn't I know this before?!". That's why I would like to present
<a href="https://www.gnu.org/software/stow/">stow</a> that is a small package
management system that permits installing applications from a separated
folder using symlinks.</p>
<h1>Creating a separated sysroot</h1>
<p>To have a better overview of the packages I compiled myself, I create an
independent sysroot in <em>/opt/</em>. Then configure the system to make sure
that libraries and binaries are correcly found. It also makes sure that
nothing is installed somewhere else as I can use a standard user at
installation, if I have to type "<em>sudo make install</em>" to solve
permission issues, it shows that something is wrong.</p>
<p>First I create my sysroot folder and give the right permissions:</p>
<div class="highlight"><pre><span></span><code>sudo mkdir -p /opt/my-sysroot/
sudo chown <span class="k">$(</span>whoami<span class="k">)</span>.<span class="k">$(</span>whoami<span class="k">)</span> /opt/my-sysroot/
</code></pre></div>
<p>Then configure the ldloader to find the shared object of the manually
installed libraries:</p>
<div class="highlight"><pre><span></span><code><span class="nb">echo</span> /opt/my-sysroot/lib <span class="p">|</span> sudo tee /etc/ld.so.conf.d/my-sysroot.conf
</code></pre></div>
<p>After installing a new library, running <em>ldconfig</em> makes sure that they
are correctly loaded at runtime.</p>
<p>Finally I export the <em>PATH</em> environement variable to find binaries using
my local <em>.bashrc</em> configuration:</p>
<div class="highlight"><pre><span></span><code><span class="nb">echo</span> <span class="s2">"export PATH=/opt/my-sysroot/bin:</span><span class="nv">$PATH</span><span class="s2">"</span> >> ~/.bashrc
</code></pre></div>
<p>Now the separated sysroot is ready to host a new package!</p>
<h1>Installing a package from sources</h1>
<p>Doing this is dependent of the build system used but the idea is to use
a different prefix at installation. I just give the separated sysroot we
created by specifying a <em>stow</em> subfolder. I also append the verison of
the package to its name, usually using <em>git describe</em>. Then it is easy
to have an overview of all available packages and their versions by
listing the content of this directory.</p>
<h2>With autoconf</h2>
<p>When using autoconf, the easiest is to pass the <em>--prefix</em> argument when
invoking the <em>./configure</em> script, for example when configuring
<em>package</em>:</p>
<div class="highlight"><pre><span></span><code>./configure --prefix<span class="o">=</span>/opt/my-sysroot/stow/package-<span class="k">$(</span>git describe --tags<span class="k">)</span>
make
make install
</code></pre></div>
<p>Notice that here, <em>sudo</em> is not used when installing the package to make
sure that all files will be installed in the sysroot we defined and not
in other folders where root permissions are required.</p>
<h2>With cmake</h2>
<p>The process is very similar with <em>cmake</em>, just use the
<em>CMAKE_INSTALL_PREFIX</em> variable on the command line:</p>
<div class="highlight"><pre><span></span><code>cmake -DCMAKE_INSTALL_PREFIX<span class="o">=</span>/opt/my-sysroot/stow/package-<span class="k">$(</span>git describe --tags<span class="k">)</span>
make
make install
</code></pre></div>
<h2>Other build system</h2>
<p>Most of the Makefile based applications use the <em>DESTDIR</em> and <em>PREFIX</em>
variables, but that is not always the case. Other build systems might
use different mechanism or variables.</p>
<p>Morality, refer to the application specific documentation in order to
change the installation directory!</p>
<h1>Installing a compiled package with stow</h1>
<p>Let's go back to our topic and install an application we have just
compiled! When installing a package with stow, there are three important
parameters to give:</p>
<ul>
<li>The directory containing our packages (stow directory)</li>
</ul>
<p>By default, stow use the the <em>STOW_DIR</em> environement variable, if unset,
uses the current directory. It is also possible to specify it when
invoking stow with <em>-d/--dir</em>.</p>
<ul>
<li>The destination directory</li>
</ul>
<p>When not specified, the destination directory is the parent of the stow
directory. Also here it is possible to change it on the command line
using the <em>-t/--target</em> argument.</p>
<ul>
<li>The package to install or remove</li>
</ul>
<p>Finally the package name is a folder existing in the stow direction, in
our case <em>package-version</em>. The <em>-S/--stow</em> argument must be given in
order to install a package or/and the <em>-D/--delete</em> to remove the
package.</p>
<p>Let's <em>stow</em> the cpputest stable version 3.5 package that is avaible in
my stow directory:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># Check available packages</span>
ls /opt/my-sysroot/stow/ <span class="p">|</span> grep cpputest
cpputest-v3.8
cpputest-v3.8-214-gf3563d53
<span class="c1"># Install stable v3.8 with stow</span>
stow --dir<span class="o">=</span>/opt/my-sysroot/stow/ <span class="se">\</span>
--target<span class="o">=</span>/opt/my-sysroot/ <span class="se">\</span>
--stow cpputest-v3.8
</code></pre></div>
<p>Let's see the effect of command in our sysroot</p>
<div class="highlight"><pre><span></span><code>ls -l /opt/my-sysroot/
total <span class="m">4</span>
lrwxrwxrwx <span class="m">1</span> random random <span class="m">26</span> Feb <span class="m">18</span> <span class="m">12</span>:52 include -> stow/cpputest-v3.8/include/
lrwxrwxrwx <span class="m">1</span> random random <span class="m">22</span> Feb <span class="m">18</span> <span class="m">12</span>:52 lib -> stow/cpputest-v3.8/lib/
drwxr-xr-x <span class="m">4</span> random random <span class="m">4096</span> Feb <span class="m">18</span> <span class="m">12</span>:35 stow/
</code></pre></div>
<p>Now it is easy to switch from a cpputest version to another</p>
<div class="highlight"><pre><span></span><code>stow --dir<span class="o">=</span>/opt/my-sysroot/stow/ <span class="se">\</span>
--target<span class="o">=</span>/opt/my-sysroot/ <span class="se">\</span>
--delete cpputest-v3.8 <span class="se">\</span>
--stow cpputest-v3.8-214-gf3563d53
</code></pre></div>
<p>We can notice that the symlinks have been changed consequently:</p>
<div class="highlight"><pre><span></span><code>ls -l /opt/my-sysroot/
total 4total <span class="m">4</span>
lrwxrwxrwx <span class="m">1</span> random random <span class="m">40</span> Feb <span class="m">18</span> <span class="m">12</span>:54 include -> stow/cpputest-v3.8-214-gf3563d53/include/
lrwxrwxrwx <span class="m">1</span> random random <span class="m">36</span> Feb <span class="m">18</span> <span class="m">12</span>:54 lib -> stow/cpputest-v3.8-214-gf3563d53/lib/
</code></pre></div>
<h2>Stow tree folding/unfolding</h2>
<p>You can notice that <em>include</em> is a direct symbolic link to the include
directory of cpputest package. That is not good if another package also
has a directory called "<em>include"</em>!</p>
<p>Because stow does its best effort to create as less symbolic as
possible, it is not necessary to create the <em>include</em> folder. Once
another package must install this folder, stow will remove the
<em>include</em> symlink, create a directory and create symlinks again within
the newly created directory.</p>
<h1>Cleaning up an old package</h1>
<p>Let's say that a package is getting really old and you want to remove it
completely from your system, it is just a matter of uninstalling it with
stow and removing the folder in the stow directory.</p>
<p>Despite the package is probably not installed, I always perform the
<em>delete</em> operation to make sure that there will be no dangling symbolic
links left after erasing the package folder.</p>
<div class="highlight"><pre><span></span><code>stow --dir<span class="o">=</span>/opt/my-sysroot/stow/ <span class="se">\</span>
--target<span class="o">=</span>/opt/my-sysroot/ <span class="se">\</span>
--delete cpputest-v3.8
rm -rf /opt/my-sysroot/stow/cpputest-v3.8
</code></pre></div>
<p>No sudo and my system is totally cleanup from the older version!</p>
<h1>Where stow can make your life easier</h1>
<p>Of course, staw is useful if you always want to work with the latest
package or play with the latest game version on your every day machine.
It makes also very convenient to switch back to an older version when
official server of a given game only work with stable release but want
to play locally with the latest for testing.</p>
<p>If you are working in a company and have applications and libraries that
depend on each other to be built, you can use an alternate sysroot where
application will be updated very frequently, when using a build machine
to test each commit when working in a contunously integrated development
cycle.</p>
<h1>Conclusion</h1>
<p>Stow changed the way I handle manually compiled packages, not only for
my hobbies but also in my daily job! I hope this small introduction will
also help you keeping a clean system while using manually compiled
packages!</p>Interrupt driven user space application with the uio driver2017-09-07T20:00:00+02:002017-09-07T20:00:00+02:00R4nd0m 6uytag:r4nd0m6uy.ch,2017-09-07:/interrupt-driven-user-space-application-with-the-uio-driver.html<p>I would like to present here a simple solution to write an interrupt driven user
space application with the help of the generic user IO kernel driver. It permits
sharing a part of the memory to the user space and catch a given interrupt
without the need of programming a …</p><p>I would like to present here a simple solution to write an interrupt driven user
space application with the help of the generic user IO kernel driver. It permits
sharing a part of the memory to the user space and catch a given interrupt
without the need of programming a specific kernel driver.</p>
<p>We will go through an example on a zynq platform and a programmable logic that
raises an interrupt after filling some memory area with a counter.</p>
<h1>The programmable logic</h1>
<p>Unfortunately, the programmable logic was not written by myself because this is
not my domain of action and I have colleagues who are good at writing VHDL. I
just had to make sure that the FPGA is programmed correctly.</p>
<p>On the zynq there are different flavors of doing this, but it's not the purpose
of this article to explain this, you can refer to the
<a href="http://www.wiki.xilinx.com/Programming+the+Programmable+Logic">Xilinx documentation</a>
instead.</p>
<h1>Kernel configuration</h1>
<p>Here is how you can enable the uio driver in the kernel configuration menu:</p>
<div class="highlight"><pre><span></span><code><span class="n">Device</span> <span class="n">Drivers</span> <span class="o">---></span>
<span class="o"><*></span> <span class="n">Userspace</span> <span class="n">I</span><span class="o">/</span><span class="n">O</span> <span class="n">drivers</span> <span class="o">---></span>
<span class="o"><*></span> <span class="n">Userspace</span> <span class="n">I</span><span class="o">/</span><span class="n">O</span> <span class="n">platform</span> <span class="n">driver</span> <span class="n">with</span> <span class="n">generic</span> <span class="n">IRQ</span> <span class="n">handling</span>
<span class="o"><*></span> <span class="n">Userspace</span> <span class="n">platform</span> <span class="n">driver</span> <span class="n">with</span> <span class="n">generic</span> <span class="n">irq</span> <span class="kr">and</span> <span class="n">dynamic</span> <span class="n">memory</span>
</code></pre></div>
<p>This should activate the following options in your .config:</p>
<div class="highlight"><pre><span></span><code>CONFIG_UIO=y
CONFIG_UIO_PDRV_GENIRQ=y
CONFIG_UIO_DMEM_GENIRQ=y
</code></pre></div>
<h1>Device tree</h1>
<p>To activate the driver, you have to update the device tree with the IRQ
information and memory space that you want to share. In our example, the FPGA
will write at address <em>0x100000</em> and use the IRQ <em>61</em>.</p>
<p>An common mistake is not defining the right address size, <strong>it must be aligned
with the page size</strong>. On the zynq, it must be a multiple of <em>0x1000</em> (4Kb).</p>
<p>On zynq platform, computing the IRQ number is a bit special ... For more details
please read <a href="http://billauer.co.il/blog/2012/08/irq-zynq-dts-cortex-a9/">this excellent article</a>.
Basically what we need to do here is <em>61 - 32</em> that gives use the interrupt
<em>29</em>.</p>
<div class="highlight"><pre><span></span><code><span class="o">&</span><span class="n">amba</span> <span class="p">{</span>
<span class="n">counters</span><span class="mi">@100000</span> <span class="p">{</span>
<span class="n">compatible</span> <span class="o">=</span> <span class="s">"fpga-counter"</span><span class="p">;</span>
<span class="n">reg</span> <span class="o">=</span> <span class="o"><</span> <span class="mh">0x100000</span> <span class="mh">0x1000</span> <span class="o">></span><span class="p">;</span>
<span class="n">interrupts</span> <span class="o">=</span> <span class="o"><</span> <span class="mi">0</span> <span class="mi">29</span> <span class="mi">1</span> <span class="o">></span><span class="p">;</span>
<span class="n">interrupt</span><span class="o">-</span><span class="n">parent</span> <span class="o">=</span> <span class="o"><&</span><span class="n">intc</span><span class="o">></span><span class="p">;</span>
<span class="p">};</span>
<span class="p">};</span>
</code></pre></div>
<h1>Command line arguments</h1>
<p>The uio driver need some command line argument in order to know on which
compatible driver is our generic uio driver mapped (<em>compatible</em>). In our case,
this is what need to be added:</p>
<div class="highlight"><pre><span></span><code>uio_pdrv_genirq.of_id="fpga-counter"
</code></pre></div>
<h1>Verifying</h1>
<p>Once we have uploaded the new kernel, device tree and updated the command line
arguments, we can verify that the kernel implementation works as expected.</p>
<p>The first thing to check is if the command line argument was given correctly:</p>
<div class="highlight"><pre><span></span><code># cat /proc/cmdline
console=ttyPS0,115200 quiet uio_pdrv_genirq.of_id=fpga-counter root=/dev/mmcblk0p2 rw rootwait rootfstype=ext4
</code></pre></div>
<p>A new uio char device should be available in <em>/dev</em> and <em>/sys/class/uio</em></p>
<div class="highlight"><pre><span></span><code># ls /sys/class/uio/
uio0
# cat /sys/class/uio/uio0/name
counters
# ls -l /dev/uio0
crw------- 1 root root 245, 0 Sep 7 14:17 /dev/uio0
</code></pre></div>
<p>Also make sure that the correct interrupt is registered:</p>
<div class="highlight"><pre><span></span><code># cat /proc/interrupts
CPU0 CPU1
...
167: 0 0 GIC-0 61 Edge counters
...
</code></pre></div>
<p>If ereything look good, we can now write the user space application to catch
interrupts raised by the FPGA.</p>
<h1>uio user space interface</h1>
<p>Once the char device is available, you can access it with the standard C library
calls.</p>
<p><strong>open(...)</strong></p>
<p>To open the char device and get a file descriptor.</p>
<p><strong>read(...)</strong></p>
<p>Blocking read until an interrupt is raised. The result will contain the amount
of interrupts that occured. <strong>It is important to read 32 bits</strong>, otherwise you
will get an error from the driver.</p>
<p>If you don't want to block on the read and do something else in background, a
<em>select</em> call can be used as well.</p>
<p><strong>write(...)</strong></p>
<p>The interrupt must be acknowledged with a write. Also here, <strong>make sure that you
write a 32 bits value</strong>, to avoid an error from the driver. The written value
must be bigger or equal to 1, otherwise the interrupt will not be acknowledged
and not be raised again.</p>
<p><strong>mmap(...)</strong></p>
<p>To map the memory region to the user space. As in the device tree, the <strong>length
argument must be aligned to the page size</strong>, otherwise you will get an error
from the driver.</p>
<h1>Example application</h1>
<p>Once we know everything we need to start coding, we can write a small user space
application:</p>
<div class="highlight"><pre><span></span><code><span class="cp">#include</span> <span class="cpf"><sys/types.h></span><span class="cp"></span>
<span class="cp">#include</span> <span class="cpf"><sys/stat.h></span><span class="cp"></span>
<span class="cp">#include</span> <span class="cpf"><sys/mman.h></span><span class="cp"></span>
<span class="cp">#include</span> <span class="cpf"><fcntl.h></span><span class="cp"></span>
<span class="cp">#include</span> <span class="cpf"><errno.h></span><span class="cp"></span>
<span class="cp">#include</span> <span class="cpf"><string.h></span><span class="cp"></span>
<span class="cp">#include</span> <span class="cpf"><stdint.h></span><span class="cp"></span>
<span class="cp">#include</span> <span class="cpf"><unistd.h></span><span class="cp"></span>
<span class="cp">#include</span> <span class="cpf"><stdio.h></span><span class="cp"></span>
<span class="cp">#define UIO_DEVICE "/dev/uio0"</span>
<span class="cp">#define MMAP_SIZE 0x1000</span>
<span class="kt">int</span> <span class="nf">main</span><span class="p">(</span><span class="kt">int</span> <span class="n">argc</span><span class="p">,</span> <span class="kt">char</span><span class="o">*</span> <span class="n">argvp</span><span class="p">[])</span>
<span class="p">{</span>
<span class="kt">int</span> <span class="n">retCode</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">uioFd</span><span class="p">;</span>
<span class="k">volatile</span> <span class="kt">uint32_t</span><span class="o">*</span> <span class="n">counters</span><span class="p">;</span>
<span class="c1">// Open uio device</span>
<span class="n">uioFd</span> <span class="o">=</span> <span class="n">open</span><span class="p">(</span><span class="n">UIO_DEVICE</span><span class="p">,</span> <span class="n">O_RDWR</span><span class="p">);</span>
<span class="k">if</span><span class="p">(</span><span class="n">uioFd</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">fprintf</span><span class="p">(</span><span class="n">stderr</span><span class="p">,</span> <span class="s">"Cannot open %s: %s</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">UIO_DEVICE</span><span class="p">,</span> <span class="n">strerror</span><span class="p">(</span><span class="n">errno</span><span class="p">));</span>
<span class="k">return</span> <span class="mi">-1</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1">// Mmap memory region containing counters value</span>
<span class="n">counters</span> <span class="o">=</span> <span class="n">mmap</span><span class="p">(</span><span class="nb">NULL</span><span class="p">,</span> <span class="n">MMAP_SIZE</span><span class="p">,</span> <span class="n">PROT_READ</span><span class="p">,</span> <span class="n">MAP_SHARED</span><span class="p">,</span> <span class="n">uioFd</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
<span class="k">if</span><span class="p">(</span><span class="n">counters</span> <span class="o">==</span> <span class="n">MAP_FAILED</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">fprintf</span><span class="p">(</span><span class="n">stderr</span><span class="p">,</span> <span class="s">"Cannot mmap: %s</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">strerror</span><span class="p">(</span><span class="n">errno</span><span class="p">));</span>
<span class="n">close</span><span class="p">(</span><span class="n">uioFd</span><span class="p">);</span>
<span class="k">return</span> <span class="mi">-1</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1">// Interrupt loop</span>
<span class="k">while</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">uint32_t</span> <span class="n">intInfo</span><span class="p">;</span>
<span class="kt">ssize_t</span> <span class="n">readSize</span><span class="p">;</span>
<span class="c1">// Acknowldege interrupt</span>
<span class="n">intInfo</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="k">if</span><span class="p">(</span><span class="n">write</span><span class="p">(</span><span class="n">uioFd</span><span class="p">,</span> <span class="o">&</span><span class="n">intInfo</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">intInfo</span><span class="p">))</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">fprintf</span><span class="p">(</span><span class="n">stderr</span><span class="p">,</span> <span class="s">"Cannot acknowledge uio device interrupt: %s</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span>
<span class="n">strerror</span><span class="p">(</span><span class="n">errno</span><span class="p">));</span>
<span class="n">retCode</span> <span class="o">=</span> <span class="mi">-1</span><span class="p">;</span>
<span class="k">break</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1">// Wait for interrupt</span>
<span class="n">readSize</span> <span class="o">=</span> <span class="n">read</span><span class="p">(</span><span class="n">uioFd</span><span class="p">,</span> <span class="o">&</span><span class="n">intInfo</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">intInfo</span><span class="p">));</span>
<span class="k">if</span><span class="p">(</span><span class="n">readSize</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">fprintf</span><span class="p">(</span><span class="n">stderr</span><span class="p">,</span> <span class="s">"Cannot wait for uio device interrupt: %s</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span>
<span class="n">strerror</span><span class="p">(</span><span class="n">errno</span><span class="p">));</span>
<span class="n">retCode</span> <span class="o">=</span> <span class="mi">-1</span><span class="p">;</span>
<span class="k">break</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1">// Display counter value</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"We got %lu interrupts, counter value: 0x%08x</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span>
<span class="n">intInfo</span><span class="p">,</span> <span class="n">counters</span><span class="p">[</span><span class="mi">0</span><span class="p">]);</span>
<span class="p">}</span>
<span class="c1">// Should never reach</span>
<span class="n">munmap</span><span class="p">((</span><span class="kt">void</span><span class="o">*</span><span class="p">)</span><span class="n">counters</span><span class="p">,</span> <span class="n">MMAP_SIZE</span><span class="p">);</span>
<span class="n">close</span><span class="p">(</span><span class="n">uioFd</span><span class="p">);</span>
<span class="k">return</span> <span class="n">retCode</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>If everything works fine, you should get some event from the driver:</p>
<div class="highlight"><pre><span></span><code>...
We got 63 interrupts, counter value: 0x9e9fffa6
We got 64 interrupts, counter value: 0x9e9fffa7
...
</code></pre></div>
<p>The amount of interrupts should correspond to what is in <em>/proc/interrupts</em></p>
<div class="highlight"><pre><span></span><code># cat /proc/interrupts
CPU0 CPU1
167: 64 3 GIC-0 61 Edge counters
</code></pre></div>
<h1>Further reading</h1>
<ul>
<li><a href="https://www.kernel.org/doc/html/latest/driver-api/uio-howto.html">Kernel documentation</a></li>
</ul>
<h1>Conclusion</h1>
<p>The uio driver is good for quick tests and prototyping. Unfortunately, it is not
as fast as a kernel space dedicated driver. If we have real time constraints,
the uio driver should not be used.</p>Configuring WLAN NAT for lxc2017-02-22T12:00:00+01:002017-02-22T12:00:00+01:00R4nd0m 6uytag:r4nd0m6uy.ch,2017-02-22:/configuring-wlan-nat-for-lxc.html<p>After having successfully installed an <a href="https://r4nd0m6uy.ch/unpriviledged-containers-in-void-linux.html">unprivildged linux
container</a>, I wanted to use my
WLAN interface for the network but I could not create a bridge due to
security restricition of the protocol. So I had to configure NAT over my
wireless to make it working. I want to show here …</p><p>After having successfully installed an <a href="https://r4nd0m6uy.ch/unpriviledged-containers-in-void-linux.html">unprivildged linux
container</a>, I wanted to use my
WLAN interface for the network but I could not create a bridge due to
security restricition of the protocol. So I had to configure NAT over my
wireless to make it working. I want to show here the steps to achieve
this.</p>
<h1>The problem</h1>
<p>My first idea was to bridge the WLAN interface to my lxc bridge but this
doesn't work:</p>
<div class="highlight"><pre><span></span><code>sudo brctl addif lxcbr0 wlp2s0
can't add wlp2s0 to bridge lxcbr0: Operation not supported
</code></pre></div>
<p>At first glance it might look weird but here is some <a href="http://web.archive.org/web/20110925231256/http://kerneltrap.org/mailarchive/linux-ath5k-devel/2010/3/21/6871733">more detailed
explanation</a>:</p>
<div class="highlight"><pre><span></span><code><span class="nv">It</span><span class="s1">'</span><span class="s">s no longer possible to add an interface in the managed mode to a</span>
<span class="nv">bridge</span>. <span class="nv">You</span> <span class="nv">should</span> <span class="nv">run</span> <span class="nv">hostapd</span> <span class="nv">first</span> <span class="nv">to</span> <span class="nv">pure</span> <span class="nv">the</span> <span class="nv">interface</span> <span class="nv">to</span> <span class="nv">the</span>
<span class="nv">master</span> <span class="nv">mode</span>.
<span class="nv">Bridging</span> <span class="nv">doesn</span><span class="s1">'</span><span class="s">t work on the station side anyway because the 802.11</span>
<span class="nv">header</span> <span class="nv">has</span> <span class="nv">three</span> <span class="nv">addresses</span> <span class="ss">(</span><span class="nv">except</span> <span class="nv">when</span> <span class="nv">WDS</span> <span class="nv">is</span> <span class="nv">used</span><span class="ss">)</span> <span class="nv">omitting</span> <span class="nv">the</span>
<span class="nv">address</span> <span class="nv">that</span> <span class="nv">would</span> <span class="nv">be</span> <span class="nv">needed</span> <span class="k">for</span> <span class="nv">a</span> <span class="nv">station</span> <span class="nv">to</span> <span class="k">send</span> <span class="nv">or</span> <span class="nv">receive</span> <span class="nv">a</span> <span class="nv">packet</span>
<span class="nv">on</span> <span class="nv">behalf</span> <span class="nv">of</span> <span class="nv">another</span> <span class="nv">system</span>.
<span class="o">--</span>
<span class="nv">Regards</span>,
<span class="nv">Pavel</span> <span class="nv">Roskin</span>
</code></pre></div>
<p>I didn't want to use hostap to solve this problem, I choosed the NAT
alternative.</p>
<h1>Preparing the host system</h1>
<p>First you need prepare your host system to activate NAT. We will need to
configure a bit the firewal, install iptables if you don't have it
already:</p>
<div class="highlight"><pre><span></span><code>sudo xbps-install iptables
</code></pre></div>
<p>Then I activate the NAT at the kernel level where <em>wlp2s0</em> is the name
of my wireless interface:</p>
<div class="highlight"><pre><span></span><code><span class="nb">echo</span> <span class="m">1</span> <span class="p">|</span> sudo tee /proc/sys/net/ipv4/ip_forward
sudo iptables -t nat -A POSTROUTING -o wlp2s0 -j MASQUERADE
</code></pre></div>
<p>Now you have to set an IP to your bridge so it can communicate, I will
use the range 10.0.0/24 and assign the first IP available:</p>
<div class="highlight"><pre><span></span><code>sudo brctl addbr lxcbr0
sudo ifconfig lxcbr0 <span class="m">10</span>.0.0.1/24 up
</code></pre></div>
<p>Now your host system is ready for NATing!</p>
<h1>Configuring the container</h1>
<p>Now we still will have to configure the newtork within the container.
Some would prefer installing a DHCP server on the host but I like
setting a static IP to each container, so I know how to reach them.</p>
<p>First you will need to know the IP of your DNS server, that is your
router most of the time but to be sure, check the <em>/etc/resolv.conf</em>
file of your host, that is <em>192.168.0.1</em> for me:</p>
<div class="highlight"><pre><span></span><code>cat /etc/resolv.conf
<span class="c1"># Generated by dhcpcd from wlp2s0.dhcp</span>
<span class="c1"># /etc/resolv.conf.head can replace this line</span>
domain <span class="nb">local</span>
nameserver <span class="m">192</span>.168.0.1
<span class="c1"># /etc/resolv.conf.tail can replace this line</span>
</code></pre></div>
<p>Now the network configuration is dependent on the distribution you have
in your container but the Debian way that also work with Ubuntu is to
edit the <em>/etc/network/interfaces</em> file:</p>
<div class="highlight"><pre><span></span><code>lxc-attach -n ubuntu-xenial
root@ubuntu-xenial:/# vim /etc/network/interfaces
</code></pre></div>
<p>Now assign a new IP in the range defined previously, I choosed
<em>10.0.0.100</em> for this container where the gateway is the IP of your
bridge and the name server the one found in <em>/etc/resolv.conf</em> of your
host system:</p>
<div class="highlight"><pre><span></span><code>auto eth0
iface eth0 inet static
address 10.0.0.100
netmask 255.255.255.0
gateway 10.0.0.1
dns-nameservers 192.168.0.1
</code></pre></div>
<p>Now apply the configuration and test with ping:</p>
<div class="highlight"><pre><span></span><code>root@ubuntu-xenial:/# /sbin/ifdown eth0
root@ubuntu-xenial:/# /sbin/ifup eth0
root@ubuntu-xenial:/# ping www.google.com
PING www.google.com <span class="o">(</span><span class="m">216</span>.58.213.196<span class="o">)</span> <span class="m">56</span><span class="o">(</span><span class="m">84</span><span class="o">)</span> bytes of data.
<span class="m">64</span> bytes from ham02s15-in-f196.1e100.net <span class="o">(</span><span class="m">216</span>.58.213.196<span class="o">)</span>: <span class="nv">icmp_seq</span><span class="o">=</span><span class="m">1</span> <span class="nv">ttl</span><span class="o">=</span><span class="m">51</span> <span class="nv">time</span><span class="o">=</span><span class="m">38</span>.6 ms
^C
--- www.google.com ping statistics ---
<span class="m">1</span> packets transmitted, <span class="m">1</span> received, <span class="m">0</span>% packet loss, <span class="nb">time</span> 0ms
rtt min/avg/max/mdev <span class="o">=</span> <span class="m">38</span>.678/38.678/38.678/0.000 ms
</code></pre></div>
<p>Now you can open an ssh connection to your container from your host:</p>
<div class="highlight"><pre><span></span><code>ssh random@10.0.0.100
random@10.0.0.100<span class="err">'</span>s password:
Welcome to Ubuntu <span class="m">16</span>.04.2 LTS <span class="o">(</span>GNU/Linux <span class="m">4</span>.9.11_1 x86_64<span class="o">)</span>
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
Last login: Wed Feb <span class="m">22</span> <span class="m">11</span>:24:13 <span class="m">2017</span> from <span class="m">10</span>.0.0.1
$
</code></pre></div>
<p>That was easy!</p>
<h1>Start lxc script</h1>
<p>Now as my network configuraton has changed a bit, here is how my script
to start the container looks like:</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/sh</span>
<span class="nb">echo</span> <span class="m">1</span> <span class="p">|</span> sudo tee /proc/sys/net/ipv4/ip_forward
sudo iptables -t nat -A POSTROUTING -o wlp2s0 -j MASQUERADE
sudo brctl addbr lxcbr0
sudo ifconfig lxcbr0 <span class="m">10</span>.0.0.1/24 up
sudo cgm create all <span class="nv">$USER</span>
sudo cgm chown all <span class="nv">$USER</span> <span class="k">$(</span>id -u<span class="k">)</span> <span class="k">$(</span>id -g<span class="k">)</span>
cgm movepid all <span class="nv">$USER</span> <span class="nv">$$</span>
lxc-start -n ubuntu-xenial
</code></pre></div>
<h1>Further reading</h1>
<p>Some pointers that helped me solving this issue:</p>
<ul>
<li><a href="http://serverfault.com/questions/152363/bridging-wlan0-to-eth0">http://serverfault.com/questions/152363/bridging-wlan0-to-eth0</a></li>
</ul>Unpriviledged containers in Void Linux2017-02-21T19:00:00+01:002017-02-21T19:00:00+01:00R4nd0m 6uytag:r4nd0m6uy.ch,2017-02-21:/unpriviledged-containers-in-void-linux.html<p>It seems that I was not the first one to struggle to make unprivildged
linux containers working in
<a href="http://www.voidlinux.eu/download/">Void Linux</a>. In systemd/Linux
distributions, the process is straight forward as systemd handles
things automagically for you (mostly everthing about cgroup) when in
Void you have to configure few things by …</p><p>It seems that I was not the first one to struggle to make unprivildged
linux containers working in
<a href="http://www.voidlinux.eu/download/">Void Linux</a>. In systemd/Linux
distributions, the process is straight forward as systemd handles
things automagically for you (mostly everthing about cgroup) when in
Void you have to configure few things by yourself.</p>
<p>I want to give some pointers to have the bare minimal to have a
container starting in unpriviledged mode in distribution that use a
minimal init system and more precisely here, Void Linux.</p>
<p><strong>UPDATE: Unfortunately, this doesn't work anymore with the latest lxc
version and only works with lxd version 2</strong></p>
<h1>Prerequists</h1>
<p>First we need few services installed on the host. Of course lxc but also
cgmanager to take care of the cgroups, dbus for IPC communication
between cgmanager and the bridge-utils to create a network interface for
our container:</p>
<div class="highlight"><pre><span></span><code>sudo xbps-install cgmanager dbus bridge-utils lxc
</code></pre></div>
<p>Configure runit to start dbus and cgmanager automatically at startup,
also start them right away:</p>
<div class="highlight"><pre><span></span><code>sudo ln -s /etc/sv/dbus/ /var/service/
sudo ln -s /etc/sv/cgmanager/ /var/service/
sudo sv start dbus
sudo sv start cgmanager
</code></pre></div>
<p>Now we need to configure the subuid, subgid mapping. First create the
two <em>/etc/sub{u,g}id</em> and give the read permission for all others:</p>
<div class="highlight"><pre><span></span><code>sudo touch /etc/sub<span class="o">{</span>u,g<span class="o">}</span>id
sudo chmod o+r /etc/sub<span class="o">{</span>g,u<span class="o">}</span>id
</code></pre></div>
<p>Then map your user to the lxc g/u ids:</p>
<div class="highlight"><pre><span></span><code>sudo usermod --add-subuids <span class="m">100000</span>-165536 <span class="nv">$USER</span>
sudo usermod --add-subgids <span class="m">100000</span>-165536 <span class="nv">$USER</span>
</code></pre></div>
<p>We also need a bridge interface to connect our container, my host
network interface is <em>enp0s25</em> and the bridge will be <em>lxcbr0</em>.</p>
<div class="highlight"><pre><span></span><code>sudo brctl addbr lxcbr0
sudo brctl addif lxcbr0 enp0s25
sudo ifconfig lxcbr0 up
</code></pre></div>
<p>If you don't want to reconfigure your bridge at each boot, you could
create a runit service for that. Now create a default lxc configuration
in <em>~/.config/lxc/default.conf</em> containing everything we have just done:</p>
<div class="highlight"><pre><span></span><code>lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.flags = up
lxc.network.hwaddr = 00:16:3e:BB:CC:DD
lxc.id_map = u 0 100000 65536
lxc.id_map = g 0 100000 65536
</code></pre></div>
<p>Now your host system should be ready to run unpriviledged container.</p>
<h1>Creating and starting a container</h1>
<p>First create a cgroup crontroller for your user and give the right
permission with cgm. <strong>This has to be done at each boot!</strong> For
convenience, I suggest to put it in a script or create a runit service.</p>
<div class="highlight"><pre><span></span><code>sudo cgm create all <span class="nv">$USER</span>
sudo cgm chown all <span class="nv">$USER</span> <span class="k">$(</span>id -u<span class="k">)</span> <span class="k">$(</span>id -g<span class="k">)</span>
</code></pre></div>
<p>Then move your current process to the created controller. <strong>This has to
be done each time you open a new shell!</strong></p>
<div class="highlight"><pre><span></span><code>cgm movepid all <span class="nv">$USER</span> <span class="nv">$$</span>
</code></pre></div>
<p>Now lxc should be able to work correcly with cgroups, you can create a
new container:</p>
<div class="highlight"><pre><span></span><code>lxc-create -t download -n ubuntu-xenial
...
<Fill requested fields>
...
</code></pre></div>
<p>Once the creation process is done, simply start the container and attach
to it:</p>
<div class="highlight"><pre><span></span><code>lxc-start -n ubuntu-xenial
lxc-attach -n ubuntu-xenial
</code></pre></div>
<p>You should now be inside your container, hurray!</p>
<div class="highlight"><pre><span></span><code>root@ubuntu-xenial:/# cat /etc/issue
Ubuntu <span class="m">16</span>.04.2 LTS <span class="se">\n</span> <span class="se">\l</span>
</code></pre></div>
<h1>Starting lxc after a new boot</h1>
<p>Unfortunately the configuration is not persistent and some operation
have to be done manually. As I don't start my container all the time, I
don't want to create a runit service, that's why I use a little script:</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/sh</span>
sudo cgm create all <span class="nv">$USER</span>
sudo cgm chown all <span class="nv">$USER</span> <span class="k">$(</span>id -u<span class="k">)</span> <span class="k">$(</span>id -g<span class="k">)</span>
sudo brctl addbr lxcbr0
sudo brctl addif lxcbr0 enp0s25
sudo ifconfig lxcbr0 up
cgm movepid all <span class="nv">$USER</span> <span class="nv">$$</span>
lxc-start -n ubuntu-xenial
</code></pre></div>
<h1>Further reading</h1>
<p>Some articles that helped me solving this issue:</p>
<ul>
<li><a href="https://s3hh.wordpress.com/2014/03/25/introducing-cgmanager/">https://s3hh.wordpress.com/2014/03/25/introducing-cgmanager/</a></li>
<li><a href="https://stgraber.org/2014/01/17/lxc-1-0-unprivileged-containers/">https://stgraber.org/2014/01/17/lxc-1-0-unprivileged-containers/</a></li>
</ul>
<h1>Next step</h1>
<p>Now you have your container starting, <a href="https://r4nd0m6uy.ch/configuring-wlan-nat-for-lxc.html">you probably want to configure
the network</a>.</p>CMake and CppUTest2017-02-21T12:00:00+01:002017-02-21T12:00:00+01:00R4nd0m 6uytag:r4nd0m6uy.ch,2017-02-21:/cmake-and-cpputest.html<p>When implementing a new feature or fixing bugs, I like having a quick
feedback to make sure that my changes didn't break the code. This is why
I usually write unit tests and execute them during the build process. In
this article I want to give some pointers to make …</p><p>When implementing a new feature or fixing bugs, I like having a quick
feedback to make sure that my changes didn't break the code. This is why
I usually write unit tests and execute them during the build process. In
this article I want to give some pointers to make my favorite tools work
well together that are the <a href="https://cpputest.github.io/">CppUtest</a> unit
testing framework and the <a href="https://cmake.org/">cmake</a> build utility.</p>
<p>I like <strong>cmake</strong> because it creates excellent
<a href="https://www.gnu.org/software/make/manual/make.html">Makefiles</a> that is
usually difficult to get right. As a developper, it compiles only files
that changed that is very efficient when your recompiling your tests
very frequently. cmake is supported by many
<a href="https://r4nd0m6uy.ch/yocto-buildroot-co.html">build systems</a> that facilitates the
deployement on embedded targets and can be easily added to your
contineous integration process where you want to compile the unit tests
automatically.</p>
<p>There are many out the but <strong>CppUTest</strong> is a well proven unit test
framework. I like it because it has a mocking framework, a memory leak
detector and doesn't need other external libraries to compile.</p>
<h1>Installing cmake and CppUTest</h1>
<p>Those tools should be available in your distribution, you should use
your package manager to install them. As long as it is decent enough, I
don't mind much about cmake version but I like working with the last
development version of CppUTest and maybe want to use different version
when compiling a stable release.</p>
<p>There are different ways to compile CppUTest but as long as we are
talking about cmake, I will use this way. The default installation path
is <em>/usr/local</em> but as I want to host different version, I define the
prefix to <em>/opt/cpputest-<GIT_VERSION></em>:</p>
<div class="highlight"><pre><span></span><code>git clone https://github.com/cpputest/cpputest.git <span class="p">;</span> <span class="nb">cd</span> cpputest
mkdir cmake-build <span class="p">;</span> <span class="nb">cd</span> cmake-build
cmake ../ -DCMAKE_INSTALL_PREFIX:PATH<span class="o">=</span>/opt/cpputest-<span class="k">$(</span>git describe<span class="k">)</span>
make -j3
</code></pre></div>
<p>Once the compilation is done, you can run <em>make install</em> and make an
user friendly symbolic link to the last available version:</p>
<div class="highlight"><pre><span></span><code>sudo make install
sudo ln -s /opt/cpputest-<span class="k">$(</span>git describe<span class="k">)</span> /opt/cpputest
</code></pre></div>
<p>To switch from a version to another, I just change the /opt/cpputest
symlink.</p>
<h1>The project structure</h1>
<p>My favorite structure is having the sources and tests in separated
folders. I write a root CMakeLists.txt that can include what I want to
compile, it should be easy to build the application without the tests.
Here is a skeleton showing what your project would look like:</p>
<div class="highlight"><pre><span></span><code>.
├── CMakeLists.txt
├── README.md
├── src
│ ├── CMakeLists.txt
│ ├── main.cpp
│ └── time
│ ├── Bar.cpp
│ ├── Bar.hpp
│ ├── Foo.cpp
│ ├── Foo.hpp
│ ├── IFoo.cpp
│ └── IFoo.hpp
└── tests
├── CMakeLists.txt
├── BarTest.cpp
├── FooTest.cpp
├── main.cpp
└── mocks
├── IFooMock.cpp
└── IFooMOck.hpp
</code></pre></div>
<h1>CMakefile.txt</h1>
<p>Now the intersting part is writing CMakeFiles. Here is what looks like
the root CMakeLists.txt:</p>
<div class="highlight"><pre><span></span><code><span class="c"># (1) CMake definitions, compiler flags and useful variables</span>
<span class="nb">cmake_minimum_required</span><span class="p">(</span><span class="s">VERSION</span> <span class="s">3.7</span><span class="p">)</span>
<span class="nb">project</span><span class="p">(</span><span class="s">cmakeCppUTestDemo</span><span class="p">)</span>
<span class="nb">add_compile_options</span><span class="p">(</span><span class="s">-std=c++11</span> <span class="s">-Wall</span> <span class="s">-Werror</span><span class="p">)</span>
<span class="nb">set</span><span class="p">(</span><span class="s">APP_NAME</span> <span class="s">fooApp</span><span class="p">)</span>
<span class="nb">set</span><span class="p">(</span><span class="s">APP_LIB_NAME</span> <span class="s">fooAppLib</span><span class="p">)</span>
<span class="c"># (2) Include application build instructions</span>
<span class="nb">add_subdirectory</span><span class="p">(</span><span class="s">src</span><span class="p">)</span>
<span class="c"># (3) include tests build instructions </span>
<span class="nb">option</span><span class="p">(</span><span class="s">COMPILE_TESTS</span> <span class="s2">"Compile the tests"</span> <span class="s">OFF</span><span class="p">)</span>
<span class="nb">if</span><span class="p">(</span><span class="s">COMPILE_TESTS</span><span class="p">)</span>
<span class="nb">add_subdirectory</span><span class="p">(</span><span class="s">tests</span><span class="p">)</span>
<span class="nb">endif</span><span class="p">(</span><span class="s">COMPILE_TESTS</span><span class="p">)</span>
</code></pre></div>
<p><strong>(1)</strong></p>
<p>First the usual cmake options, we need to define a project name and
decide which version of cmake we need. I also add some restrictive
compiliation flags and declare some variables that I can share between
both subprojects, the application and the tests.</p>
<p><strong>(2)</strong></p>
<p>Because making sure that the application compiles is an excellent test,
this is done all the time and include the sub cmakefile.</p>
<p><strong>(3)</strong></p>
<p>Compiling the unit tests is probably not a required step for everyone,
this is why I want to give explicitely the option to do it by passing
<em>COMPILE_TESTS=ON</em> to cmake.</p>
<h1>src/CMakefile.txt</h1>
<p>Here is how looks the instruction to build the main application:</p>
<div class="highlight"><pre><span></span><code><span class="c"># (1) Build a library with my application sources</span>
<span class="nb">set</span><span class="p">(</span><span class="s">APP_LIB_SOURCE</span>
<span class="s">IFoo.cpp</span>
<span class="s">Foo.cpp</span>
<span class="s">Bar.cpp</span>
<span class="p">)</span>
<span class="nb">add_library</span><span class="p">(</span><span class="o">${</span><span class="nv">APP_LIB_NAME</span><span class="o">}</span> <span class="o">${</span><span class="nv">APP_LIB_SOURCE</span><span class="o">}</span><span class="p">)</span>
<span class="c"># (2) Add main(..) to the application library to have something we can run</span>
<span class="nb">add_executable</span><span class="p">(</span><span class="o">${</span><span class="nv">APP_NAME</span><span class="o">}</span> <span class="s">main.cpp</span><span class="p">)</span>
<span class="nb">target_link_libraries</span><span class="p">(</span><span class="o">${</span><span class="nv">APP_NAME</span><span class="o">}</span> <span class="o">${</span><span class="nv">APP_LIB_NAME</span><span class="o">}</span><span class="p">)</span>
</code></pre></div>
<p><strong>(1)</strong></p>
<p>The trick is to use a separated main function from the application
sources, so it doesn't conflict with the main from the tests. We simply
compile here a static library that can be used when linking with the
tests or the main from the normal program.</p>
<p><strong>(2)</strong></p>
<p>As we have all the objects we need, we add the main file where we can
provide an entry point.</p>
<h1>tests/CMakefile.txt</h1>
<p>Finally, how to build the unit tests that you wrote using CppUtest:</p>
<div class="highlight"><pre><span></span><code><span class="c"># (1) Look for installed version of CppUTest</span>
<span class="nb">if</span><span class="p">(</span><span class="s">DEFINED</span> <span class="s">ENV{CPPUTEST_HOME}</span><span class="p">)</span>
<span class="nb">message</span><span class="p">(</span><span class="s">STATUS</span> <span class="s2">"Using CppUTest home: $ENV{CPPUTEST_HOME}"</span><span class="p">)</span>
<span class="nb">set</span><span class="p">(</span><span class="s">CPPUTEST_INCLUDE_DIRS</span> <span class="o">$ENV{</span><span class="nv">CPPUTEST_HOME</span><span class="o">}</span><span class="s">/include</span><span class="p">)</span>
<span class="nb">set</span><span class="p">(</span><span class="s">CPPUTEST_LIBRARIES</span> <span class="o">$ENV{</span><span class="nv">CPPUTEST_HOME</span><span class="o">}</span><span class="s">/lib</span><span class="p">)</span>
<span class="nb">set</span><span class="p">(</span><span class="s">CPPUTEST_LDFLAGS</span> <span class="s">CppUTest</span> <span class="s">CppUTestExt</span><span class="p">)</span>
<span class="nb">else</span><span class="p">()</span>
<span class="nb">find_package</span><span class="p">(</span><span class="s">PkgConfig</span> <span class="s">REQUIRED</span><span class="p">)</span>
<span class="nb">pkg_search_module</span><span class="p">(</span><span class="s">CPPUTEST</span> <span class="s">REQUIRED</span> <span class="s">cpputest>=3.8</span><span class="p">)</span>
<span class="nb">message</span><span class="p">(</span><span class="s">STATUS</span> <span class="s2">"Found CppUTest version ${CPPUTEST_VERSION}"</span><span class="p">)</span>
<span class="nb">endif</span><span class="p">()</span>
<span class="c"># (2) Our unit tests sources</span>
<span class="nb">set</span><span class="p">(</span><span class="s">TEST_APP_NAME</span> <span class="o">${</span><span class="nv">APP_NAME</span><span class="o">}</span><span class="s">_tests</span><span class="p">)</span>
<span class="nb">set</span><span class="p">(</span><span class="s">TEST_SOURCES</span>
<span class="s">mocks/IFooMock.cpp</span>
<span class="s">FooTest.cpp</span>
<span class="s">BarTest.cpp</span>
<span class="s">main.cpp</span>
<span class="p">)</span>
<span class="c"># (3) Take care of include directories</span>
<span class="nb">include_directories</span><span class="p">(</span><span class="o">${</span><span class="nv">CPPUTEST_INCLUDE_DIRS</span><span class="o">}</span> <span class="s">../src/</span><span class="p">)</span>
<span class="nb">link_directories</span><span class="p">(</span><span class="o">${</span><span class="nv">CPPUTEST_LIBRARIES</span><span class="o">}</span>
<span class="c"># (4) Build the unit tests objects and link then with the app library</span>
<span class="s">add_executable(</span><span class="o">${</span><span class="nv">TEST_APP_NAME</span><span class="o">}</span> <span class="o">${</span><span class="nv">TEST_SOURCES</span><span class="o">}</span><span class="p">)</span>
<span class="nb">target_link_libraries</span><span class="p">(</span><span class="o">${</span><span class="nv">TEST_APP_NAME</span><span class="o">}</span> <span class="o">${</span><span class="nv">APP_LIB_NAME</span><span class="o">}</span> <span class="o">${</span><span class="nv">CPPUTEST_LDFLAGS</span><span class="o">}</span><span class="p">)</span>
<span class="c"># (5) Run the test once the build is done</span>
<span class="nb">add_custom_command</span><span class="p">(</span><span class="s">TARGET</span> <span class="o">${</span><span class="nv">TEST_APP_NAME</span><span class="o">}</span> <span class="s">COMMAND</span> <span class="s">./</span><span class="o">${</span><span class="nv">TEST_APP_NAME</span><span class="o">}</span> <span class="s">POST_BUILD</span><span class="p">)</span>
</code></pre></div>
<p><strong>(1)</strong></p>
<p>If the user has <em>CPPUTEST_HOME</em> set in his environment, we use it to
locate CppUTest installation. In case it is not provided, we use
pkgconfig to look for it in the system. The cmake process fails here if
CppUTest is not found or its version is too old.</p>
<p><strong>(2)</strong></p>
<p>Then we add all the sources required to compile the unit tests,
including the main function.</p>
<p><strong>(3)</strong></p>
<p>We have to define the include directory of our application source so the
compiler can find our headers files. We also have to take care of
CppUTest include directories variables we set previously.</p>
<p><strong>(4)</strong></p>
<p>Compile the unit test application by linking it with our library and
CppUTest.</p>
<p><strong>(5)</strong></p>
<p>Finally run the tests once it is built! If the unit test fails, the make
command will also fail, that gives us a quick feedback if everything is
fine!</p>
<h1>Compile the whole</h1>
<p>To compile the application, proceed the usual cmake way:</p>
<div class="highlight"><pre><span></span><code>mkdir build <span class="p">;</span> <span class="nb">cd</span> build
cmake ../
make -j3
</code></pre></div>
<p>If you want to compile the unit tests, execute cmake with this
parameter:</p>
<div class="highlight"><pre><span></span><code>cmake -DCOMPILE_TESTS<span class="o">=</span>ON ../
</code></pre></div>
<p>If you haven't installed cpputest or use a too old version, this error
will pop out:</p>
<div class="highlight"><pre><span></span><code><span class="o">--</span> <span class="nv">Checking</span> <span class="k">for</span> <span class="nv">one</span> <span class="nv">of</span> <span class="nv">the</span> <span class="nv">modules</span> <span class="s1">'</span><span class="s">cpputest>=3.8</span><span class="s1">'</span>
<span class="nv">CMake</span> <span class="nv">Error</span> <span class="nv">at</span> <span class="o">/</span><span class="nv">usr</span><span class="o">/</span><span class="nv">share</span><span class="o">/</span><span class="nv">cmake</span><span class="o">-</span><span class="mi">3</span>.<span class="mi">7</span><span class="o">/</span><span class="nv">Modules</span><span class="o">/</span><span class="nv">FindPkgConfig</span>.<span class="nv">cmake</span>:<span class="mi">637</span> <span class="ss">(</span><span class="nv">message</span><span class="ss">)</span>:
<span class="nv">None</span> <span class="nv">of</span> <span class="nv">the</span> <span class="nv">required</span> <span class="s1">'</span><span class="s">cpputest>=3.8</span><span class="s1">'</span> <span class="nv">found</span>
<span class="k">Call</span> <span class="nl">Stack</span> <span class="ss">(</span><span class="nv">most</span> <span class="nv">recent</span> <span class="k">call</span> <span class="nl">first</span><span class="ss">)</span>:
<span class="nv">tests</span><span class="o">/</span><span class="nv">CMakeLists</span>.<span class="nv">txt</span>:<span class="mi">8</span> <span class="ss">(</span><span class="nv">pkg_search_module</span><span class="ss">)</span>
</code></pre></div>
<p>You can install cpputest with your package manager or set the
<em>CPPUTEST_HOME</em> environment variables pointing to your last version:</p>
<div class="highlight"><pre><span></span><code><span class="nv">CPPUTEST_HOME</span><span class="o">=</span>/opt/cpputest/ cmake -DCOMPILE_TESTS<span class="o">=</span>ON ../
</code></pre></div>
<p>Once cmake ran correcly, you just need to run <em>make</em> to (re)build the
changes and run the unit tests:</p>
<div class="highlight"><pre><span></span><code>make -j3
...
<span class="o">[</span><span class="m">100</span>%<span class="o">]</span> Linking CXX executable fooApp_tests
..................................................
..
OK <span class="o">(</span><span class="m">52</span> tests, <span class="m">52</span> ran, <span class="m">112</span> checks, <span class="m">0</span> ignored, <span class="m">0</span> filtered out, <span class="m">7</span> ms<span class="o">)</span>
<span class="o">[</span><span class="m">100</span>%<span class="o">]</span> Built target fooApp_tests
</code></pre></div>
<h1>Conclusion</h1>
<p>Now you have a template that you can use it for your own project and be
efficient by having a quick feedback if something goes wrong!</p>Fishing shells2016-12-26T15:00:00+01:002016-12-26T15:00:00+01:00R4nd0m 6uytag:r4nd0m6uy.ch,2016-12-26:/fishing-shells.html<p>I have been used to bash for years and always thought it was very good but after
trying other alternatives I found out that there are more efficient ways to use
the command line.</p>
<p>I would like to present here one of my favorite interpreter that is the <a href="https://fishshell.com/">fish
shell …</a></p><p>I have been used to bash for years and always thought it was very good but after
trying other alternatives I found out that there are more efficient ways to use
the command line.</p>
<p>I would like to present here one of my favorite interpreter that is the <a href="https://fishshell.com/">fish
shell</a>. <em>Fish</em> stands for <strong>F</strong>riendly <strong>I</strong>nteractive
<strong>Sh</strong>ell and is yet another command line interpreter. Its goal is to be user
friendly while being fast. </p>
<p>The major difference with other standards command line interpreters, is that
fish is not POSIX and has some subtle differences. People behind fish think
POSIX is not adapted for modern usage and decided to not follow it to invent a
better, less error prone and more user friendly syntax.</p>
<p>To make the switch less painful, I will try to show the majors difference
between the well known bash and give some tips on how to customize it.</p>
<h1>Installation</h1>
<p>You can install fish shell from the source but it is recommended to use your
favorite package manager:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># .deb</span>
apt-get install fish
<span class="c1"># .rpm</span>
yum install fish
<span class="c1"># Arch</span>
pacman -S fish
<span class="c1"># Gentoo</span>
emerge fish
<span class="c1"># Void</span>
xbps-install fish-shell
<span class="c1"># Other I forgot</span>
<span class="c1"># Read the fine manual :)</span>
</code></pre></div>
<h1>Starting fish</h1>
<p>On your usual shell prompt, you can simply type:</p>
<div class="highlight"><pre><span></span><code>fish
</code></pre></div>
<p>If after some tries your are happy with it, you can set it as default by editing
the <em>/etc/passwd</em> manually or by using the <em>chsh</em> command:</p>
<div class="highlight"><pre><span></span><code>chsh -s /usr/bin/fish <your-username>
</code></pre></div>
<h1>Common usages</h1>
<p>Here is a list useful daily tasks but using the fish syntax.</p>
<ul>
<li><strong>Return code</strong></li>
</ul>
<p>$? doesn't exist, the <em>status</em> variable is what you are looking for:</p>
<div class="highlight"><pre><span></span><code><span class="nb">fish</span><span class="nv">$ </span><span class="k">true</span> <span class="p">;</span> <span class="k">echo</span> <span class="nv">$status</span>
0
<span class="nb">fish</span><span class="nv">$ </span><span class="k">false</span> <span class="p">;</span> <span class="k">echo</span> <span class="nv">$status</span>
1
</code></pre></div>
<ul>
<li><strong>;and, ;or</strong></li>
</ul>
<p>People from fish uses the <em>and/or</em> keyword instead of &&/||</p>
<div class="highlight"><pre><span></span><code><span class="nb">fish</span><span class="nv">$ </span><span class="k">true</span> <span class="p">;</span> <span class="k">and</span> <span class="k">echo</span> sucess!
sucess!
<span class="nb">fish</span><span class="nv">$ </span><span class="k">false</span> <span class="p">;</span> <span class="k">or</span> <span class="k">echo</span> failed!
failed!
</code></pre></div>
<ul>
<li><strong>Variables</strong></li>
</ul>
<p>When you want to export variables with bash, you would use the <em>export</em> and <em>=</em>
sign. Don't add any space in between otherwise it will not work! Lists are not
well supported either.</p>
<p>With fish shell, the <em>set</em> built-in is used to declare variables command. With
the <em>-x</em> parameter, the variabled is exported. fish supports lists very well and
can be declared by using space between each element:</p>
<div class="highlight"><pre><span></span><code><span class="nb">fish</span><span class="nv">$ </span><span class="k">set</span> -x PATH <span class="nv">$PATH</span> <span class="nv">$HOME</span>/bin /opt/toolchain/bin
<span class="nb">fish</span><span class="nv">$ </span><span class="k">echo</span> <span class="nv">$PATH</span>
/usr/local/bin /bin /usr/bin /home/random/bin/ /opt/toolchain/bin
<span class="nb">fish</span><span class="nv">$ </span><span class="k">set</span> -x ARCH arm
<span class="nb">fish</span><span class="nv">$ </span>make menuconfig
</code></pre></div>
<p>More information about list can be found further.</p>
<ul>
<li><strong>Command substitution</strong></li>
</ul>
<p>Under bash, my favorite flavor is using <em>$(...)</em> but think fish guys think that
the <em>$</em> character is confusing as it is already used for variable expansion, and
is consequently not used:</p>
<div class="highlight"><pre><span></span><code><span class="nv">$ </span><span class="k">echo</span> My usernmae is <span class="o">(</span>whoami<span class="o">)</span>
My usernmae is <span class="nb">random</span>
</code></pre></div>
<ul>
<li><strong>Flow control</strong></li>
</ul>
<p>With fish, you have to use the <em>end</em> keyword that is common for every flow
control mechanism unlike <em>done</em>, <em>fi</em>, <em>esac</em> with other shells. Also where fish
adds a touch of user friendliness is that it takes care of all the indentation
and coloring:</p>
<div class="highlight"><pre><span></span><code><span class="nb">fish</span><span class="nv">$ </span><span class="k">while</span> <span class="k">true</span>
uptime
sleep 1
<span class="k">end</span>
</code></pre></div>
<p>Another if statement example:</p>
<div class="highlight"><pre><span></span><code><span class="nv">$fish</span> <span class="k">if</span> <span class="k">true</span>
<span class="k">echo</span> All good
<span class="k">end</span>
All good
</code></pre></div>
<ul>
<li><strong>Functions</strong></li>
</ul>
<p>When using a function, parameters are stored in the <em>argv</em> list:</p>
<div class="highlight"><pre><span></span><code><span class="nb">fish</span><span class="nv">$ </span><span class="k">function</span> hello
<span class="k">echo</span> Hello <span class="nv">$argv</span><span class="o">[</span>1<span class="o">]</span>
<span class="k">end</span>
<span class="nb">fish</span><span class="nv">$ </span>hello
Hello
<span class="nb">fish</span><span class="nv">$ </span>hello world
Hello world
</code></pre></div>
<p>I love the <a href="https://fishshell.com/docs/current/index.html#expand-index-range">index range expansion</a> feature!</p>
<div class="highlight"><pre><span></span><code><span class="nb">fish</span><span class="nv">$ </span><span class="k">function</span> hello
<span class="k">echo</span> Hello <span class="nv">$argv</span><span class="o">[</span>1..-1<span class="o">]</span>
<span class="k">end</span>
<span class="nb">fish</span><span class="nv">$ </span>hello tux world everyone
Hello tux world everyone
</code></pre></div>
<p>Or if you want to say hello to everyone more personnaly:</p>
<div class="highlight"><pre><span></span><code><span class="nb">fish</span><span class="nv">$ </span><span class="k">function</span> hello
<span class="k">for</span> who <span class="k">in</span> <span class="nv">$argv</span><span class="o">[</span>1..-1<span class="o">]</span>
<span class="k">echo</span> Hello <span class="nv">$who</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="nb">fish</span><span class="nv">$ </span>hello tux world everyone
Hello tux
Hello world
Hello everyone
</code></pre></div>
<ul>
<li><strong>time command</strong></li>
</ul>
<p>It is sometime interesting to know how long it takes to run a command and <em>time</em>
built-in bash implementation does the trick very well. In fish, there is no
<em>time</em> command but the duration is stored in a variable:</p>
<div class="highlight"><pre><span></span><code>fish$ sleep <span class="m">1</span>
fish$ <span class="nb">echo</span> <span class="nv">$CMD_DURATION</span>
<span class="m">1006</span>
</code></pre></div>
<p>You could implement the time function yourself but you can also use some fish
scripts from the communauty. <a href="https://github.com/jichu4n/fish-command-timer">fish_command_timer</a> is my favoriyte, simply clone the
repository and source the required file:</p>
<div class="highlight"><pre><span></span><code><span class="nb">fish</span><span class="nv">$ </span>git clone https://github.com/jichu4n/fish-command-timer.git
<span class="nb">fish</span><span class="nv">$ </span>. ./fish-command-timer/fish_command_timer.fish
<span class="nb">fish</span><span class="nv">$ </span>ls
<span class="nb">fish</span>-command-timer/
<span class="o">[</span> 0s101 <span class="o">|</span> Dec <span class="m">09</span> 04:19PM <span class="o">]</span>
</code></pre></div>
<p>To make this permanent, you can add this your the fish configuration as
explained later.</p>
<h1>Customization</h1>
<p>Of course, fish is highly configurable but again, differs a bit from other
shells.</p>
<ul>
<li><strong>.bashrc</strong></li>
</ul>
<p>If you are looking for something similar to the .bashrc file, you can edit or
create the <em>.config/fish/config.fish</em>. Here is for example how mine looks like:</p>
<div class="highlight"><pre><span></span><code><span class="k">set</span> fish_greeting <span class="o">(</span>uptime<span class="o">)</span>
. /home/random/sources/fish-command-timer/fish_command_timer.fish
</code></pre></div>
<ul>
<li><strong>Prompt</strong></li>
</ul>
<p>Unlike the <em>PS1</em> variable, the fish prompt is defined in the <em>fish_prompt</em>
function:</p>
<div class="highlight"><pre><span></span><code><span class="nb">fish</span><span class="nv">$ </span><span class="k">function</span> <span class="nb">fish_prompt</span>
<span class="nb"> </span><span class="k">echo</span> -n <span class="o">(</span>whoami<span class="o">)</span>@<span class="o">(</span>cat /etc/hostname<span class="o">)</span><span class="se">\$</span><span class="s2">" "</span>
<span class="k">end</span>
<span class="nb">random</span>@random-notebook<span class="err">$</span>
</code></pre></div>
<ul>
<li><strong>GIT prompt</strong></li>
</ul>
<p>Fish is able to show the git branch and repository status on the prompt.
Just enable some built-in configurations by adding this in the fish
configuration:</p>
<div class="highlight"><pre><span></span><code><span class="k">set</span> __fish_git_prompt_showdirtystate <span class="s1">'yes'</span>
<span class="k">set</span> __fish_git_prompt_showstashstate <span class="s1">'yes'</span>
<span class="k">set</span> __fish_git_prompt_showuntrackedfiles <span class="s1">'yes'</span>
<span class="k">set</span> __fish_git_prompt_showupstream <span class="s1">'yes'</span>
<span class="k">set</span> __fish_git_prompt_color_branch yellow
<span class="k">set</span> __fish_git_prompt_color_upstream_ahead green
<span class="k">set</span> __fish_git_prompt_color_upstream_behind red
</code></pre></div>
<p>This will change your prompt as follow when inside a git repository:</p>
<div class="highlight"><pre><span></span><code><span class="nb">fish</span> <span class="o">(</span>master<span class="o">)</span> ➞
</code></pre></div>
<ul>
<li><strong>Oh my fish</strong></li>
</ul>
<p><em>omf</em> is a repository containing different skins for fish. Everyone is welcome
to use it and share his themes. Make sure you have a decent version of fish to
have a better user exeprience, unfortunately it doesn't work well on older LTS
distriubtions.</p>
<p>Here is how you can install it and user my favorite theme:</p>
<div class="highlight"><pre><span></span><code><span class="nb">fish</span><span class="nv">$ </span>curl -L http://get.oh-my.fish <span class="o">|</span> <span class="nb">fish</span>
<span class="c"># ...</span>
<span class="nb">fish</span><span class="nv">$ </span>omf install fisk
Installing theme fisk
✔ theme fisk successfully installed.
<span class="o">[</span>0<span class="o">]</span> <span class="nb">random</span>@random-notebook ~ ➞
</code></pre></div>
<p>For forther information check the <a href="https://github.com/oh-my-fish/oh-my-fish">official git repository</a> and the list of <a href="https://github.com/oh-my-fish/oh-my-fish/blob/master/docs/Themes.md">available themes</a></p>
<p>This would probably remind <a href="http://ohmyz.sh/">oh my zsh</a> for the zsh<em>ish</em>!</p>
<h1>Common shortcuts</h1>
<p>Luckily some shortcuts you learned from bash are also available within fish.
Here is the list of the one I use the most frequently:</p>
<ul>
<li><strong>TAB</strong>: Enable autocompletion for almost everything :p</li>
<li><strong>CTRL+f</strong>: Accept <strong>f</strong>ish autosuggestion</li>
<li><strong>CTRL+e</strong>: Go to the <strong>e</strong>nd of the command</li>
<li><strong>CTRL+a</strong>: Go to the beginning of the command (<strong>A</strong>nfang in German?)</li>
<li><strong>CTRL+p | up arrow </strong>: Go to the <strong>p</strong>revious in history</li>
<li><strong>CTRL+n | down arrow</strong>: Go to the <strong>n</strong>ext in history </li>
<li><strong> < pattern > CTRL+p|CTRL+n</strong>: Look for <em>< pattern ></em> in hostory</li>
</ul>
<h1>Conclusion</h1>
<p>Fish has a very nice out of the box configuration and most people will be very
happy with it! If you like it and think it could improve your productivity, do
not hesitate to have a deeper look at the <a href="https://fishshell.com/docs/current/tutorial.html">introduction page</a> and at the <a href="https://fishshell.com/docs/current/index.html">documentation</a> for advanced usage. </p>
<p>It has the huge drawback that it is not POSIX but this is OK when using it only
in interactive mode. Unfortunately I haven't written many <em>fischripts</em> and would
be interesting to see if it really brings much performance improvement.</p>
<p><a href="http://www.zsh.org/">zsh</a> is a nice alternative, it is very user friendly while
respecting the standards. Those who spent a lot of time customizing their zsh
will prefer sticking with it but fish worth having a look for your own culture!</p>Yocto, Buildroot & Co.2016-11-16T19:00:00+01:002016-11-16T19:00:00+01:00R4nd0m 6uytag:r4nd0m6uy.ch,2016-11-16:/yocto-buildroot-co.html<p>Here are <a href="../documents/YoctoBuildrootAndCo.pdf">the slides</a> about the talk I
gave during the <a href="http://fribourg-linux-seminar.ch">11th Fribourg Linux Seminar</a>. I wanted to explain the different
philosophies behing the two major build systems available in the
comunity.</p>
<p>Maybe my opinion was a bit flawed as I haven't used buildroot for few years and
I know …</p><p>Here are <a href="../documents/YoctoBuildrootAndCo.pdf">the slides</a> about the talk I
gave during the <a href="http://fribourg-linux-seminar.ch">11th Fribourg Linux Seminar</a>. I wanted to explain the different
philosophies behing the two major build systems available in the
comunity.</p>
<p>Maybe my opinion was a bit flawed as I haven't used buildroot for few years and
I know Yocto better. At then end, both projects are good, make sure you thought
about everything you will need during the whole lifetime of your product! The
time was too short, I spoke only about two but don't forget considering other
alternatives!</p>Event driven programming with the reactor pattern2016-10-22T15:00:00+02:002016-10-22T15:00:00+02:00R4nd0m 6uytag:r4nd0m6uy.ch,2016-10-22:/event-driven-programming-with-the-reactor-pattern.html<p>I want to show in this article an example of bad practice and give an elegant
solution with the reactor pattern that helped me and my team writing more
efficient code.</p>
<p>If you use frameworks like QT, SDL, Java Swing, gtklib, ... this article will
give you a rough idea about …</p><p>I want to show in this article an example of bad practice and give an elegant
solution with the reactor pattern that helped me and my team writing more
efficient code.</p>
<p>If you use frameworks like QT, SDL, Java Swing, gtklib, ... this article will
give you a rough idea about what is happening behind the scene and from where
come from some limitations. I will write <a href="https://github.com/
r4nd0m6uy/reactor_example">the source</a> in CPP as it is a language I know well enough and is
not too difficult to understand if you come from the OO world.</p>
<p>[TOC]</p>
<h1>The example application</h1>
<p>Let's say that for this example you are waiting for data to coming from a named
pipe and want to print its content to stdout. Once the application is started,
we wil generate some output by writing in a named pipe:</p>
<blockquote>
<p>echo "Hello app" > /tmp/test_pipe</p>
</blockquote>
<p>That will print the following on the console</p>
<blockquote>
<p>Hello app</p>
</blockquote>
<p>Don't forget to quit gracefully the app on CTRL+C and that's all we need!</p>
<h1>The bad code</h1>
<p>Here is how would look the naive implementation of this program:</p>
<div class="highlight"><pre><span></span><code><span class="kt">bool</span> <span class="n">isRunning</span> <span class="o">=</span> <span class="nb">false</span><span class="p">;</span>
<span class="c1">//--------------------------------------------------------------------------</span>
<span class="kt">void</span> <span class="nf">onSignal</span><span class="p">(</span><span class="kt">int</span> <span class="n">signum</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">isRunning</span> <span class="o">=</span> <span class="nb">false</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1">//--------------------------------------------------------------------------</span>
<span class="kt">int</span> <span class="nf">main</span><span class="p">(</span><span class="kt">int</span> <span class="n">argc</span><span class="p">,</span> <span class="kt">char</span><span class="o">*</span> <span class="n">argv</span><span class="p">[])</span>
<span class="p">{</span>
<span class="cm">/**</span>
<span class="cm"> * Skipped initialization</span>
<span class="cm"> */</span>
<span class="kt">int</span> <span class="n">fifoFd</span> <span class="o">=</span> <span class="n">open</span><span class="p">(</span><span class="s">"/tmp/test_pipe"</span><span class="p">,</span> <span class="n">O_RDONLY</span> <span class="o">|</span> <span class="n">O_NONBLOCK</span><span class="p">);</span>
<span class="k">if</span><span class="p">(</span><span class="n">fifoFd</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span>
<span class="k">return</span> <span class="mi">-1</span><span class="p">;</span>
<span class="n">isRunning</span> <span class="o">=</span> <span class="nb">true</span><span class="p">;</span>
<span class="k">while</span><span class="p">(</span><span class="n">isRunning</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">int</span> <span class="n">buffSize</span> <span class="o">=</span> <span class="mi">256</span><span class="p">;</span>
<span class="kt">char</span> <span class="n">buffer</span><span class="p">[</span><span class="n">buffSize</span><span class="p">];</span>
<span class="kt">ssize_t</span> <span class="n">readSize</span><span class="p">;</span>
<span class="n">readSize</span> <span class="o">=</span> <span class="n">read</span><span class="p">(</span><span class="n">fifoFd</span><span class="p">,</span> <span class="n">buffer</span><span class="p">,</span> <span class="n">buffSize</span><span class="p">);</span>
<span class="k">if</span><span class="p">(</span><span class="n">readSize</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span><span class="p">(</span><span class="n">errno</span> <span class="o">==</span> <span class="n">EAGAIN</span> <span class="o">||</span> <span class="n">errno</span> <span class="o">==</span> <span class="n">EWOULDBLOCK</span><span class="p">)</span>
<span class="k">continue</span><span class="p">;</span>
<span class="k">else</span>
<span class="k">return</span> <span class="mi">-1</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">if</span><span class="p">(</span><span class="n">write</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">buffer</span><span class="p">,</span> <span class="n">readSize</span><span class="p">)</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span>
<span class="k">return</span> <span class="mi">-1</span><span class="p">;</span>
<span class="p">}</span>
<span class="cm">/**</span>
<span class="cm"> * Skipped cleanup</span>
<span class="cm"> */</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>If you don't see what is wrong here, try run the application and check the CPU
usage, it should be 100% as the loop runs continuously despite nothing has to be
read from the pipe. Using sleep should help to spare some resources but this is
not good enough because the answer time will be in the worst case, the time you
sleep. If you have real time constraints, game over!</p>
<p>Another bad point is extending this code, in the case you have more
I/Os to read I'll let you imagine how this can get very messy... Using multi
threading would not help and will bring more complexity when synchronizing them,
don't do that for this specific problem!</p>
<h1>The Reactor pattern</h1>
<p>The reactor pattern gives a simple solution to do what we want using a single
thread. The following diagram explains very nicely its mechanism:</p>
<p><img alt="Reactor sequence" src="../images/reactor/sequence.png" title="source: http://www.cs.wustl.edu/~schmidt/PDF/reactor-siemens.pdf"></p>
<p>Don't worry if you don't understand each detail for now, I will try to explain
them while implementing the pattern.</p>
<h1>Our IO handler</h1>
<p>We will define first a small interface that reflects the <em>Event_Handler</em>. This
object will be implemented by the user and be in charge of reading data from the
<em>handle</em> when requested by the <em>Dispatcher</em>.</p>
<p>In the Linux world, when we speak about <em>Handle</em> we refer to a file descriptor,
that's why we will use an int and call it <em>IoHandle</em>:</p>
<div class="highlight"><pre><span></span><code><span class="k">typedef</span> <span class="kt">int</span> <span class="n">IoHandle</span><span class="p">;</span>
</code></pre></div>
<p>Then we declare an interface:</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">IIoHandler</span>
<span class="p">{</span>
<span class="k">public</span><span class="o">:</span>
<span class="n">IIoHandler</span><span class="p">();</span>
<span class="k">virtual</span> <span class="o">~</span><span class="n">IIoHandler</span><span class="p">();</span>
</code></pre></div>
<p>We will need a function to pass the handler's <em>IoHandle</em> to the <em>Dispatcher</em>:</p>
<div class="highlight"><pre><span></span><code> <span class="k">virtual</span> <span class="n">IoHandle</span> <span class="nf">getIoHandle</span><span class="p">()</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
</code></pre></div>
<p>And a callback function that will be called when data is ready to be read from
the given <em>IoHandle</em>:</p>
<div class="highlight"><pre><span></span><code> <span class="k">virtual</span> <span class="kt">void</span> <span class="nf">readReady</span><span class="p">()</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">};</span>
</code></pre></div>
<p>That's it for the interface but we still need a concrete implementation. For
this example we will create a class that will be called <em>NamedPipeToStdout</em>.
Yes, in programming the biggest difficulty is finding good names:</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">NamedPipeToStdout</span><span class="o">:</span>
<span class="k">public</span> <span class="n">IIoHandler</span>
<span class="p">{</span>
</code></pre></div>
<p>We will declare two private fields, one containing the <em>IoHandle</em> and one the
path to the pipe:</p>
<div class="highlight"><pre><span></span><code><span class="k">private</span><span class="o">:</span>
<span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="n">m_pipePath</span><span class="p">;</span>
<span class="n">IoHandle</span> <span class="n">m_pipeIoHandle</span><span class="p">;</span>
</code></pre></div>
<p>Let's start with a constructor from which we will inject the path to the pipe
and initialize the <em>IoHandle</em> to an invalid value:</p>
<div class="highlight"><pre><span></span><code><span class="k">public</span><span class="o">:</span>
<span class="n">NamedPipeToStdout</span><span class="p">(</span><span class="k">const</span> <span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="o">&</span> <span class="n">name</span><span class="p">)</span><span class="o">:</span>
<span class="n">m_pipePath</span><span class="p">(</span><span class="n">name</span><span class="p">),</span>
<span class="n">m_pipeIoHandle</span><span class="p">(</span><span class="mi">-1</span><span class="p">)</span>
<span class="p">{</span>
<span class="p">}</span>
</code></pre></div>
<p>As usual the destructor will perform some cleanup:</p>
<div class="highlight"><pre><span></span><code> <span class="o">~</span><span class="n">NamedPipeToStdout</span><span class="p">()</span>
<span class="p">{</span>
<span class="n">close</span><span class="p">(</span><span class="n">m_pipeIoHandle</span><span class="p">);</span>
<span class="n">unlink</span><span class="p">(</span><span class="n">m_pipePath</span><span class="p">.</span><span class="n">c_str</span><span class="p">());</span>
<span class="p">}</span>
</code></pre></div>
<p>An init function will be useful to create and open the pipe:</p>
<div class="highlight"><pre><span></span><code> <span class="kt">int</span> <span class="nf">init</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">if</span><span class="p">(</span><span class="n">mkfifo</span><span class="p">(</span><span class="n">m_pipePath</span><span class="p">.</span><span class="n">c_str</span><span class="p">(),</span> <span class="n">S_IRWXU</span><span class="p">)</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span><span class="p">(</span><span class="n">errno</span> <span class="o">!=</span> <span class="n">EEXIST</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">perror</span><span class="p">(</span><span class="s">"mkfifo"</span><span class="p">);</span>
<span class="k">return</span> <span class="mi">-1</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">m_pipeIoHandle</span> <span class="o">=</span> <span class="n">open</span><span class="p">(</span><span class="n">m_pipePath</span><span class="p">.</span><span class="n">c_str</span><span class="p">(),</span> <span class="n">O_RDWR</span> <span class="o">|</span> <span class="n">O_NONBLOCK</span><span class="p">);</span>
<span class="k">if</span><span class="p">(</span><span class="n">m_pipeIoHandle</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">perror</span><span class="p">(</span><span class="s">"Open fifo"</span><span class="p">);</span>
<span class="k">return</span> <span class="mi">-1</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>We still need to implement the functions from the interface, first the
<em>getIoHandle</em> function that simply returns the file descriptor of the pipe:</p>
<div class="highlight"><pre><span></span><code> <span class="n">IoHandle</span> <span class="nf">getIoHandle</span><span class="p">()</span> <span class="k">override</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">m_pipeIoHandle</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>And the callback when data in the pipe are available, we will just print the
content to stdout as described by the class name:</p>
<div class="highlight"><pre><span></span><code> <span class="kt">void</span> <span class="nf">readReady</span><span class="p">()</span> <span class="k">override</span>
<span class="p">{</span>
<span class="kt">int</span> <span class="n">buffSize</span> <span class="o">=</span> <span class="mi">64</span><span class="p">;</span>
<span class="kt">char</span> <span class="n">buffer</span><span class="p">[</span><span class="n">buffSize</span><span class="p">];</span>
<span class="kt">ssize_t</span> <span class="n">readSize</span><span class="p">;</span>
<span class="n">readSize</span> <span class="o">=</span> <span class="n">read</span><span class="p">(</span><span class="n">m_pipeIoandle</span><span class="p">,</span> <span class="n">buffer</span><span class="p">,</span> <span class="n">buffSize</span><span class="p">);</span>
<span class="k">if</span><span class="p">(</span><span class="n">readSize</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">perror</span><span class="p">(</span><span class="s">"Read from pipe"</span><span class="p">);</span>
<span class="k">return</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">if</span><span class="p">(</span><span class="n">write</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">buffer</span><span class="p">,</span> <span class="n">readSize</span><span class="p">)</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span>
<span class="n">perror</span><span class="p">(</span><span class="s">"Write to stdout"</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">};</span>
</code></pre></div>
<p>That's should be it for our <em>IoHandler</em>, now we need a <em>Dispatcher</em> where all
the magic will occur!</p>
<h1>Our event dispatcher</h1>
<p>We will write first the interface for our <em>Dispatcher</em>. An interface is always
wise as it is easier to mock for unit testing. Moreover, under Linux, there are
different flavor of the <em>select</em> call (<a href="http://man7.org/linux/man-pages/
man2/select.2.html">select</a>, <a href="http://man7.org/linux/man-pages/man2/poll.2.html">poll</a>,
<a href="http://man7.org/linux/man-pages/man7/epoll.7.html">epoll</a>, ...). If one day you
want to change, you just need to write another implementation of this interface.
As we want to perform operations on I/Os, we will explicitly call it
<em>IIoDispatcher</em>:</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">IIoDispatcher</span>
<span class="p">{</span>
<span class="k">public</span><span class="o">:</span>
<span class="n">IIoDispatcher</span><span class="p">();</span>
<span class="k">virtual</span> <span class="o">~</span><span class="n">IIoDispatcher</span><span class="p">();</span>
</code></pre></div>
<p>As previously, we will define first an init function:</p>
<div class="highlight"><pre><span></span><code> <span class="k">virtual</span> <span class="kt">int</span> <span class="nf">init</span><span class="p">()</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
</code></pre></div>
<p>The dispatch function that will block as long as the application runs:</p>
<div class="highlight"><pre><span></span><code> <span class="k">virtual</span> <span class="kt">int</span> <span class="nf">dispatch</span><span class="p">()</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
</code></pre></div>
<p>A function to register an <em>IIoHandler</em>:</p>
<div class="highlight"><pre><span></span><code> <span class="k">virtual</span> <span class="kt">int</span> <span class="nf">registerIoHandler</span><span class="p">(</span><span class="n">IIoHandler</span><span class="o">&</span> <span class="n">handler</span><span class="p">)</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
</code></pre></div>
<p>Finally a function to break the loop in case we want to exit:</p>
<div class="highlight"><pre><span></span><code> <span class="k">virtual</span> <span class="kt">int</span> <span class="nf">breakLoop</span><span class="p">()</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">};</span>
</code></pre></div>
<p>Again, we need a concrete implementation. It is not the purpose of this article
to describe all possible system call to peform a <em>select</em>, <a href="http://www.ulduzsoft.com/2014/01/select-poll-epoll-practical-
difference-for-system-architects/">there are other
great articles</a> about this topic. Just be aware of the pro
and contra of each of them to help you decide which one suits you the best.</p>
<p>Despite it is known for being slow, I will stick with the actual <em>select</em> call
that is implemented in most decent modern OS. Let's give it an explicit name:</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">IoDispatcherSelect</span><span class="o">:</span>
<span class="k">public</span> <span class="n">IIoDispatcher</span>
<span class="p">{</span>
</code></pre></div>
<p>We will store the registered <em>IoHandlers</em> in a list, a field to remember the
highest file descriptor value, that is specific to <em>select</em> and also a running
flag in order to break the loop:</p>
<div class="highlight"><pre><span></span><code><span class="k">private</span><span class="o">:</span>
<span class="n">IoHandle</span> <span class="n">m_maxIoHandle</span><span class="p">;</span>
<span class="kt">bool</span> <span class="n">m_isRunning</span><span class="p">;</span>
<span class="n">std</span><span class="o">::</span><span class="n">list</span><span class="o"><</span><span class="n">IIoHandler</span><span class="o">*></span> <span class="n">m_ioHandlers</span><span class="p">;</span>
</code></pre></div>
<p>First the constructor that initializes everything:</p>
<div class="highlight"><pre><span></span><code> <span class="n">IoDispatcherSelect</span><span class="p">()</span><span class="o">:</span>
<span class="n">m_maxIoHandle</span><span class="p">(</span><span class="mi">-1</span><span class="p">),</span>
<span class="n">m_isRunning</span><span class="p">(</span><span class="nb">false</span><span class="p">)</span>
<span class="p">{</span>
<span class="p">}</span>
</code></pre></div>
<p>A destructor that doesn't do much in this implementation:</p>
<div class="highlight"><pre><span></span><code> <span class="o">~</span><span class="n">IoDispatcherSelect</span><span class="p">()</span>
<span class="p">{</span>
<span class="p">}</span>
</code></pre></div>
<p>Despite the init function might be useful for other calls such as <em>epoll</em>, the
select doesn't need to be initialized, so we don't do anything here:</p>
<div class="highlight"><pre><span></span><code> <span class="kt">int</span> <span class="nf">init</span><span class="p">()</span> <span class="k">override</span>
<span class="p">{</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>Finally where things get interesting is in the <em>dispatch</em> function. This
implementation is specific to <em>select</em> but the main idea is the same for the
other calls despite the API differs. I will try to explain step by step what
should be done.</p>
<p>First the main loop and variables declaration:</p>
<div class="highlight"><pre><span></span><code> <span class="kt">int</span> <span class="nf">dispatch</span><span class="p">()</span> <span class="k">override</span>
<span class="p">{</span>
<span class="kt">int</span> <span class="n">ret</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="n">m_isRunning</span> <span class="o">=</span> <span class="nb">true</span><span class="p">;</span>
<span class="k">while</span><span class="p">(</span><span class="n">m_isRunning</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">fd_set</span> <span class="n">read_fds</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">selectRes</span><span class="p">;</span>
<span class="k">struct</span> <span class="nc">timeval</span> <span class="n">timeout</span><span class="p">;</span>
</code></pre></div>
<p>For each call to select, we need to clear the set of file descriptor that is
stored in a <em>fd_set</em> with the <em>FD_ZERO</em> macro:</p>
<div class="highlight"><pre><span></span><code> <span class="n">FD_ZERO</span><span class="p">(</span><span class="o">&</span><span class="n">read_fds</span><span class="p">);</span>
</code></pre></div>
<p>This is where <em>select</em> is not efficient, we have to loop through our registered
<em>IIoHandlers</em> and add their file descriptor in the set with <em>FD_SET</em>:</p>
<div class="highlight"><pre><span></span><code> <span class="k">for</span><span class="p">(</span><span class="k">auto</span><span class="o">&</span> <span class="nl">it</span><span class="p">:</span> <span class="n">m_readIoHandlers</span><span class="p">)</span>
<span class="n">FD_SET</span><span class="p">(</span><span class="n">it</span><span class="o">-></span><span class="n">getIoHandle</span><span class="p">(),</span> <span class="o">&</span><span class="n">read_fds</span><span class="p">);</span>
</code></pre></div>
<p>I will explain later why using a timeout here is not a good idea but for
simplicity, we will use this solution. We don't want to block in the select call
forever if nothing has to be read, we want to check the running flag status as
well, that's why we prepare a timeout of 1 second:</p>
<div class="highlight"><pre><span></span><code> <span class="n">timeout</span><span class="p">.</span><span class="n">tv_usec</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="n">timeout</span><span class="p">.</span><span class="n">tv_sec</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
</code></pre></div>
<p>Finally the call to select. The first argument is the amount of file descriptors
to watch, the second the set of file descriptor to watch for read, then come the
set for write and exception that we don't use in this example. The last argument
is the <em>timeval</em> struct containing the timeout:</p>
<div class="highlight"><pre><span></span><code> <span class="n">selectRes</span> <span class="o">=</span> <span class="n">select</span><span class="p">(</span><span class="n">m_maxIoHandle</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span>
<span class="o">&</span><span class="n">read_fds</span><span class="p">,</span>
<span class="nb">NULL</span><span class="p">,</span>
<span class="nb">NULL</span><span class="p">,</span>
<span class="o">&</span><span class="n">timeout</span><span class="p">);</span>
</code></pre></div>
<p>Once the call returns, we check the return code in case of error. We accept
<em>EINTR</em> and perform the select again otherwise exit with an error:</p>
<div class="highlight"><pre><span></span><code> <span class="k">if</span><span class="p">(</span><span class="n">selectRes</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span><span class="p">(</span><span class="n">errno</span> <span class="o">==</span> <span class="n">EINTR</span><span class="p">)</span>
<span class="k">continue</span><span class="p">;</span>
<span class="k">else</span>
<span class="p">{</span>
<span class="n">perror</span><span class="p">(</span><span class="s">"select"</span><span class="p">);</span>
<span class="n">ret</span> <span class="o">=</span> <span class="mi">-1</span><span class="p">;</span>
<span class="k">break</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>If <em>select</em> returns 0, no file descriptor was ready, this means the timeout was
reached, we perform the select again or exit the loop in case the running flag
changed to false:</p>
<div class="highlight"><pre><span></span><code> <span class="k">else</span> <span class="k">if</span><span class="p">(</span><span class="n">selectRes</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span>
<span class="k">continue</span><span class="p">;</span>
</code></pre></div>
<p>Finally we have to loop again throught the list of regsitered handlers to check
if the file descriptor is ready and call the callback function. Again, this is
where <em>select</em> is inefficient:</p>
<div class="highlight"><pre><span></span><code> <span class="k">for</span><span class="p">(</span><span class="k">auto</span><span class="o">&</span> <span class="nl">it</span><span class="p">:</span> <span class="n">m_readIoHandlers</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span><span class="p">(</span><span class="n">FD_ISSET</span><span class="p">(</span><span class="n">it</span><span class="o">-></span><span class="n">getIoHandle</span><span class="p">(),</span> <span class="o">&</span><span class="n">read_fds</span><span class="p">))</span>
<span class="n">it</span><span class="o">-></span><span class="n">readReady</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">}</span> <span class="c1">// while(m_isRunning)</span>
</code></pre></div>
<p>The loop broke, exit the function with a return code:</p>
<div class="highlight"><pre><span></span><code> <span class="k">return</span> <span class="n">ret</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>That's it for the dispatch function using select!</p>
<p>Now in the <em>registerIoHandler</em> we will store the maximum file descriptor value
for the select call, we also want to make sure that the <em>IIoHandler</em> gives a
valid value!</p>
<div class="highlight"><pre><span></span><code> <span class="kt">int</span> <span class="nf">registerIoHandler</span><span class="p">(</span><span class="n">IIoHandler</span><span class="o">&</span> <span class="n">handler</span><span class="p">)</span> <span class="k">override</span>
<span class="p">{</span>
<span class="n">IoHandle</span> <span class="n">handle</span> <span class="o">=</span> <span class="n">handler</span><span class="p">.</span><span class="n">getIoHandle</span><span class="p">();</span>
<span class="n">assert</span><span class="p">(</span><span class="n">handle</span> <span class="o">>=</span> <span class="mi">0</span><span class="p">);</span>
<span class="k">if</span><span class="p">(</span><span class="n">handle</span> <span class="o">></span> <span class="n">m_maxFd</span><span class="p">)</span>
<span class="n">m_maxIoHandle</span> <span class="o">=</span> <span class="n">handle</span><span class="p">;</span>
<span class="n">m_readIoHandlers</span><span class="p">.</span><span class="n">push_back</span><span class="p">(</span><span class="o">&</span><span class="n">handler</span><span class="p">);</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>To break the loop, it is just a matter of setting the running flag to false:</p>
<div class="highlight"><pre><span></span><code><span class="kt">int</span> <span class="nf">IoDispatcherSelect::breakLoop</span><span class="p">()</span> <span class="k">override</span>
<span class="p">{</span>
<span class="n">m_isRunning</span> <span class="o">=</span> <span class="nb">false</span><span class="p">;</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>Now we have all the components we need to write a nice and elegant solution for
our problem, the final step would be writing a main function!</p>
<h1>Wiring all the thing</h1>
<p>Now let's write a main.cpp file where all the wiring will be done. We will make
the dispatcher local to access it from other functions:</p>
<div class="highlight"><pre><span></span><code><span class="n">IoDispatcherSelect</span> <span class="n">ioDispatcher</span><span class="p">;</span>
</code></pre></div>
<p>We also want to catch signals to break the loop, we define here a signal
callback:</p>
<div class="highlight"><pre><span></span><code><span class="kt">void</span> <span class="nf">onSignal</span><span class="p">(</span><span class="kt">int</span> <span class="n">signum</span><span class="p">,</span> <span class="kt">siginfo_t</span> <span class="o">*</span><span class="n">info</span><span class="p">,</span> <span class="kt">void</span> <span class="o">*</span><span class="n">ptr</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">std</span><span class="o">::</span><span class="n">cout</span> <span class="o"><<</span> <span class="s">"Reived signal "</span> <span class="o"><<</span> <span class="n">signum</span> <span class="o"><<</span> <span class="s">" breaking loop!"</span> <span class="o"><<</span>
<span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span>
<span class="n">ioDispatcher</span><span class="p">.</span><span class="n">breakLoop</span><span class="p">();</span>
<span class="p">}</span>
</code></pre></div>
<p>A small function to register to signals:</p>
<div class="highlight"><pre><span></span><code><span class="kt">int</span> <span class="nf">signalInit</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">struct</span> <span class="nc">sigaction</span> <span class="n">act</span><span class="p">;</span>
<span class="n">memset</span><span class="p">(</span><span class="o">&</span><span class="n">act</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">act</span><span class="p">));</span>
<span class="n">act</span><span class="p">.</span><span class="n">sa_sigaction</span> <span class="o">=</span> <span class="n">onSignal</span><span class="p">;</span>
<span class="n">act</span><span class="p">.</span><span class="n">sa_flags</span> <span class="o">=</span> <span class="n">SA_SIGINFO</span><span class="p">;</span>
<span class="k">if</span><span class="p">(</span><span class="n">sigaction</span><span class="p">(</span><span class="n">SIGTERM</span><span class="p">,</span> <span class="o">&</span><span class="n">act</span><span class="p">,</span> <span class="nb">NULL</span><span class="p">)</span> <span class="o">||</span>
<span class="n">sigaction</span><span class="p">(</span><span class="n">SIGINT</span><span class="p">,</span> <span class="o">&</span><span class="n">act</span><span class="p">,</span> <span class="nb">NULL</span><span class="p">))</span>
<span class="p">{</span>
<span class="n">perror</span><span class="p">(</span><span class="s">"sigaction"</span><span class="p">);</span>
<span class="k">return</span> <span class="mi">-1</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>And finally our main function. We will use two handlers because this is fun and
to show the flexibility of the pattern:</p>
<div class="highlight"><pre><span></span><code><span class="kt">int</span> <span class="nf">main</span><span class="p">(</span><span class="kt">int</span> <span class="n">argc</span><span class="p">,</span> <span class="kt">char</span><span class="o">*</span> <span class="n">argv</span><span class="p">[])</span>
<span class="p">{</span>
<span class="n">NamedPipeToStdout</span> <span class="n">pipe1</span><span class="p">(</span><span class="s">"/tmp/test_pipe1"</span><span class="p">);</span>
<span class="n">NamedPipeToStdout</span> <span class="n">pipe2</span><span class="p">(</span><span class="s">"/tmp/test_pipe2"</span><span class="p">);</span>
<span class="k">if</span><span class="p">(</span><span class="n">signalInit</span><span class="p">())</span>
<span class="k">return</span> <span class="mi">-1</span><span class="p">;</span>
<span class="k">if</span><span class="p">(</span><span class="n">ioDispatcher</span><span class="p">.</span><span class="n">init</span><span class="p">())</span>
<span class="k">return</span> <span class="mi">-1</span><span class="p">;</span>
<span class="k">if</span><span class="p">(</span><span class="n">pipe1</span><span class="p">.</span><span class="n">init</span><span class="p">())</span>
<span class="k">return</span> <span class="mi">-1</span><span class="p">;</span>
<span class="k">if</span><span class="p">(</span><span class="n">pipe2</span><span class="p">.</span><span class="n">init</span><span class="p">())</span>
<span class="k">return</span> <span class="mi">-1</span><span class="p">;</span>
<span class="n">ioDispatcher</span><span class="p">.</span><span class="n">registerIoHandler</span><span class="p">(</span><span class="n">pipe1</span><span class="p">);</span>
<span class="n">ioDispatcher</span><span class="p">.</span><span class="n">registerIoHandler</span><span class="p">(</span><span class="n">pipe2</span><span class="p">);</span>
<span class="k">return</span> <span class="n">mainLoop</span><span class="p">.</span><span class="n">dispatch</span><span class="p">();</span>
<span class="p">}</span>
</code></pre></div>
<h1>The new status</h1>
<p>When running the program, you will notice that 0% of CPU use used while nothing
has to be done. If you do a stress test like this:</p>
<blockquote>
<p>cat /dev/urandom > /tmp/test_pipe1</p>
</blockquote>
<p>You will see that a lot of CPU will be used but most time will be spent in other
tasks, our application will use only few resource and reacts directly when
something has to be done. Moreover, you can register a big amount of
<em>IoHandlers</em> that will be very responsive!</p>
<h1>What can be improved</h1>
<p>I wanted to stay simple and show the mechanism of the pattern but there are few
things missing in this implementation to use it in a production code.</p>
<ul>
<li><strong>Timeout in select</strong></li>
</ul>
<p>Due to the timeout in the <em>select</em> call, we are still doing some kind of polling
to check the running flag.</p>
<p>To remedy this problem we could use <a href="http://man7.org/linux/
man-pages/man2/signalfd.2.html"><em>signalfd</em></a> system call in Linux. This allows registering a
file descriptor to watch for signal. Just know that this system call was
criticized by many people and keep in mind the consequence of using it.</p>
<p>If you don't have this system call, a more portable way would be using the <em>self
pipe trick</em>. It consists of creating a pipe, registering the read end in the
select and write something from the signal handler. The self pipe trick is also
very useful if you want to wake up the main loop from a worker thread.</p>
<p>A nice article about <em>signalfd</em> and the <em>self pipe trick</em> <a href="https://ldpreload.com/blog/signalfd-is-useless">is available here</a>.</p>
<ul>
<li><strong>Deregistering handlers</strong></li>
</ul>
<p>In a normal application, it often happens that an <em>IoHandler</em> wants to
deregister itself from the <em>Dispatcher</em>. Especially when writing network
applications, imagine that a client client disconnects!</p>
<p>I skipped this functionality because this is not so trivial as it might sound.
Make sure this doesn't invalidate your <em>IoHandler</em> iterator and that you are not
performing a call on a handler that was already destroyed! The dirty flag might
be helpful here.</p>
<ul>
<li><strong>Other than read events</strong></li>
</ul>
<p>You probably noticed that two arguments are set to <em>NULL</em> in the <em>select</em> call.
This is because it supports other than <em>read</em> event. A misconception in this
implementation is that the write can potentially be blocking, that will block
the main thread. It would be wiser to use non blocking write and register to
write event on <em>EWOULDBLOCK</em> error. When writing is possible again, you will be
informed by the <em>Dispatcher</em>.</p>
<ul>
<li><strong>Signals and timers handlers</strong></li>
</ul>
<p>I tried to name my things with the <em>IO</em> keyword because I only wanted to handle
I/Os. However a decent main loop would be able to register handlers to catch
signals and timeouts on timers.</p>
<p>How to handle signal was explained previously but how to handle timers? Again
here the self pipe trick can be useful to wake up an object in charge of
managing different timers in the main thread. Another possible implementaiton on
Linux is the <em>timerfd_create</em> system call that, as the <em>signalfd</em> flavor, it
permits watching timers through a file descriptor.</p>
<h1>Going further with the reactor pattern</h1>
<p>I tried to explicitly name my things with the <em>I/O</em> keyword because thanks to
the "<em>Everything is a file</em>" Unix philosphy, there are a lot of I/Os you can
monitor:</p>
<ul>
<li>Mouse, keyboard, joystick from /dev/inputx</li>
<li>Unix socket</li>
<li>Serial ports</li>
<li>GPIOs</li>
<li>Message queues</li>
<li>inotify subsystem</li>
<li>...</li>
</ul>
<p>Moreover, you can write a specific <em>IoHandler</em> to interprete events coming from
the underlying hardware and generate other than I/Os event:</p>
<ul>
<li>Key pressed/released event from a keyboard, GPIO, ...</li>
<li>Mouse moved, button clicked event from mouse</li>
<li>New connection event from a server socket (accept)</li>
<li>Modem event if you are reading AT commands coming from a serial port</li>
<li>File modified, opened, closed, … from inotify</li>
<li>...</li>
</ul>
<p>Possibilites are infinite!</p>
<h1>Don't write it yourself</h1>
<p>This example is purely academic to help understanding what is behind the scene
of many frameworks! If you write an application from scratch, don’t write
the main loop yourself, because it is difficult to do it right and
portable the first time. There are smart people who wrote libraries to solve
specific problem, I specially recommend the <a href="http://libevent.org/"><em>libevent</em></a>
library, it can handle signals, timers, is fast, portable and the bother of
choosing the right select flavor is transparent.</p>
<p>However writing a generic interface is always a good idea for unit testing. It
should be possible to encapsulate <em>libevent</em> instead of the <em>select</em>
implementation we wrote together.</p>
<h1>Pro</h1>
<p>After going through this small example, I hope you will agree with the following
advantages of the reactor pattern</p>
<ul>
<li>It is not very difficult to understand</li>
<li>It spares resources by using the CPU only when required</li>
<li>Well defined separation of concerns with different handlers</li>
<li>Avoid all the thread synchronization headaches</li>
</ul>
<h1>Contra</h1>
<p>There are some limitations but if you are aware of them, there are some
solutions:</p>
<ul>
<li><em>IoHandler</em> tasks should be small and fast</li>
</ul>
<p>If you take too much time to process data, it will block the main thread and all
other registered <em>IoHandler</em> consequently. The solution for that is, using a
worker thread and well defined synchronization mechanisms. This is why also in
many frameworks such as QT or Java Swing, you should not perform long task in
callbacks and is even a good practice implementing worker threads in that case.</p>
<ul>
<li>It brings more complexity in the code</li>
</ul>
<p>That's also the drawback of flexibility... This is why you should define
separations of concerns clearly while doing your architecture.</p>
<ul>
<li>It doesn't scale well on multi processor architecture</li>
</ul>
<p>This is true if you are using a single thread but if you doing background job,
your OS will run them on the different cores! If you have a lot of connections
to handle, you can improve performances by using more than one select call in
different threads but do this only if you have good reasons and you did some
benchmarks that showed big benefits!</p>
<h1>Further reading</h1>
<p>Some interesting reading that are not linked in the article:</p>
<ul>
<li><a href="http://www.cs.wustl.edu/
~schmidt/PDF/reactor-siemens.pdf">A very nice paper about the reactor pattern</a></li>
<li><a href="http://www.kegel.com/c10k.html">The c10k problem</a></li>
<li><a href="https://www.sitepoint.com/
the-self-pipe-trick-explained/">Well explained self pipe trick</a></li>
</ul>
<h1>Conclusion</h1>
<p>I hope this small introduction to the reactor pattern showed its flexibility and
what can be achieved! Now you understand why in Java you should write small
handlers when a button is clicked and understand what is behind
<a href="http://doc.qt.io/qt-5/qapplication.html#exec"><em>QApplication::exec()</em></a> in QT.</p>
<p>The sources I used to write this article are <a href="https://
github.com/r4nd0m6uy/reactor_example">available on github</a>, feel free to
play and extend it to have a better feeling with the different flavor of select!</p>Linux Music Studio2016-08-20T18:00:00+02:002016-08-20T18:00:00+02:00R4nd0m 6uytag:r4nd0m6uy.ch,2016-08-20:/linux-music-studio.html<p>When starting playing music with Linux, it might look scary and very complicated
due to this jungle of softwares available and you can waste a lot of time until
you find the setup that fits you the most.</p>
<p>This guide will try to help you having a basic setup to …</p><p>When starting playing music with Linux, it might look scary and very complicated
due to this jungle of softwares available and you can waste a lot of time until
you find the setup that fits you the most.</p>
<p>This guide will try to help you having a basic setup to start having fun
quickly.</p>
<p>[TOC]</p>
<h1>Hardware</h1>
<p>Before starting playing, you need some different pieces of hardware, like a fast
enough computer, a soundcard with few I/Os, some MIDI controlers and of course
some music instruments!</p>
<p>To avoid <a href="https://en.wikipedia.org/wiki/Buffer_underrun">buffer over/under flow</a>
while recording, you should have a decent machine with a good CPU. Especially
if you have few channels, it takes time to process each of them. As reference,
I'm running an <strong>intel i5 running at 2.5GHz</strong> and this provides good enough
performances using 44KHz sampling rate.</p>
<p>If you have more than one instrument, you should have a sound card with more
than one input. I recommend M-Audio devices that are available for a reasonable
price and have very good Linux compatibility. Mine is an <strong>M-Audio Fast Track
ultra</strong> that has an USB interface, 8 I/O channels and XLR connectors:</p>
<p><img alt="Fast Track Ultra" src="../images/linuxStudio/fastTrackUltra.jpg"></p>
<p>You probably also want some MIDI controlers to start or stop recording, change
volumes on the fly, mute channels, ... You can also find some USB/MIDI
controlers for a reasonable price, I'm using a <strong>KORG nano kontrol</strong> that has
some knobs, sliders, buttons and support four different scenes:</p>
<p><img alt="Korg nano kontrol" src="../images/linuxStudio/korgNanoKontrol.jpg"></p>
<p>At least but not last you should have some instruments. Pick up your favorites
one that you can plug into your sound card such as <strong>synthesizers, guitares,
microphones, ...</strong> You can also use a MIDI keyboard to play with a software
synthesizer.</p>
<h1>Improve your latency</h1>
<p>It is not mandatory to configure your system to be perfect but this will improve
your experience and avoid XRuns. This is pretty much an endless topic and is
dependent on the distribution you are using. It is not the purpose of this
article to explain everything in detail, I will probably write one dedicated to
this topic someday but here some my quick pointers:</p>
<ul>
<li>
<p>Use a <strong>low latency kernel</strong> or better, apply the <strong>RT patch</strong> and compile your
kernel with an higher timer frequency. </p>
</li>
<li>
<p>In case you are using an RT kernel, configure your <strong>limits and RT
permissions</strong> correctly.</p>
</li>
<li>
<p>Make sure also that you <strong>share your IRQs correctly</strong>, avoid using a mouse on
the same IRQ channel as your sound card.</p>
</li>
</ul>
<p>If you have no idea on how to achieve this, use a dedicated Linux distribution
that does it already for you,
<a href="https://r4nd0m6uy.ch/linux-music-studio.html#other-notable-softwares">see below</a></p>
<h1>JACK sound server</h1>
<p>For a Linux studio, I strongly recommend using
<a href="http://www.jackaudio.org/">jackd</a>. It provides the possibility to
<strong>interconnect the audio I/Os</strong> of your different softwares with your sound
card. It is able to provide a <strong>time synchronization mechanism</strong>, this is useful
if you want to syncrhonize your drum machine with a looper as explained further.
You can also configure the sample rate in order to improve the latency in the
cost of the audio quality depending on your computer performances and what you
want to achieve.</p>
<p>Using jackd on the command line can be a bit cumbersome, especially if you have
a lot of channels. I recommend using
<a href="http://qjackctl.sourceforge.net/">qjacktcl</a> that is a nice QT frontend. Here is
how it looks like on my Desktop:</p>
<p><img alt="qjackctl" src="../images/linuxStudio/qjackctl.png"></p>
<h1>Let's start with some rythm</h1>
<p>You will probably need some drum machine to conduct your song. My favorite
software is <a href="http://www.hydrogen-music.org/">hydrogen</a>. It is pretty easy to use,
is highly configurable, has a faire amount of free samples available in the
repository and can serve as <strong>jack time master</strong>. You can also create your sound
library if you already have some samples available. I also map some buttons of
my MIDI controler to start or stop playing and also to mute or unmute the audio
output. In case of live performance, it helps doing breaks or if you want to
play an instrument in solo.</p>
<p><img alt="hydrogen" src="../images/linuxStudio/hydrogen.png"></p>
<h1>Let's start looping</h1>
<p>If like me you don't have any musician friend and want to perform with more than
one instrument, the best way to do this is using a looper.
<a href="http://essej.net/sooperlooper/">sooperlooper</a> is the software that suits me the
best. It allows recording loops in live and can be <strong>synchronized with a jack
time master</strong> like hydrogen. You can map some buttons on your MIDI controler to
start recording, mute, trigger, change the volume on the fly of your loops.</p>
<p>Once you have a basic idea of what you wanna play, you can record a bass line,
some melodies, loop them and let yourself improvising something nice on top of
this!</p>
<p><img alt="sooperlooper" src="../images/linuxStudio/sooperlooper.png"></p>
<h1>Immortalize your song</h1>
<p>If you are happy with your performance and want to share or keep an audio
version as a souvenir, you can start recording it. To do that, I use
<a href="https://ardour.org/">Ardour</a>. This is a pretty complex software, I myself don't
event use 1% of its functionalities but can do my tasks pretty easily.</p>
<p>It is able to record different channels and if you are not happy with one of
your solo, you can record again single track, like in a real studio. It has the
possibility to add sound effects and do some post production like balancing your
channels, apply a compressor, ... and finally export your work into your
favorite audio format.</p>
<p><img alt="ardour" src="../images/linuxStudio/ardour.png"></p>
<h1>Other notable softwares</h1>
<p>Until here I showed the minimal set of software I use but there are a lot
available out there, here are some that worse having a look:</p>
<ul>
<li><a href="http://zynaddsubfx.sourceforge.net/">zynaddsubfx</a>: A software synthesizer</li>
<li><a href="http://jack-rack.sourceforge.net/">jack-rack</a>: An audio effects stack</li>
<li><a href="http://guitarix.org/">guitarix</a>: Like jack-rack but more guitare players
oriented</li>
<li><a href="http://dssi.sourceforge.net/hexter.html">hexter</a>: A Yamaha DX-7 emulator</li>
<li><a href="https://sourceforge.net/projects/bristol/">bristol</a>: A vintage keyboards
emulator</li>
<li><a href="http://www.nongnu.org/lingot/">lingot</a>: A guitare tuner</li>
<li><a href="http://drobilla.net/software/patchage/">patchage</a>: A frontend to patch your
I/Os with jack</li>
</ul>
<p>In case you are not a Linux beast and don't know much how to tweak your system,
there are some <strong>Linux distributions</strong> with everything configured and installed
for you:</p>
<ul>
<li><a href="https://ubuntustudio.org/">Ubuntu studio</a>: Based on Ubuntu</li>
<li><a href="http://kxstudio.linuxaudio.org/">kxstudio</a>: Yet another Ubuntu based distro</li>
</ul>
<h1>Conclusion</h1>
<p>At first glance, playing music under Linux might look very complicated and can
scare out some neophytes. However, once you have found your marks and your
habits, using the jack server and some of your favorite softwares offers a lot
of flexibility! This is the Unix philosophy, different softwares that do a
single task very well!</p>
<p>I hope this small guide will help some of you finding your marks and wake up
some creative spirits!</p>
<p>Enjoy your personal Linux studio!</p>printf("Hello World\n");2016-08-19T19:00:00+02:002016-08-19T19:00:00+02:00R4nd0m 6uytag:r4nd0m6uy.ch,2016-08-19:/printfhello-worldn.html<p>This is my brand new blog where I want to write and speak about nerdy stuff such
as video games, technology, let's say
<a href="https://en.wikipedia.org/wiki/GNU/Linux_naming_controversy">GNU/Linux</a>
softwares more precisly and life in general.</p>
<p>I hope you will enjoy it, happy surfing!</p>