Tinderbox/scratchbox integration
Overview
This guide is intended for helping developers to build a continuous integration environment with the
scratchbox
cross-compilation toolkit and some of the Mozilla Webtools.
The Mozilla tools used are:
- Tinderbox2: an automatic build master
- Bonsai: a web-based interface to CVS repositories
- Bugzilla: a bug tracking system
- LXR: a cross referenced display for source code
The tinderbox is the core of our continuous integration system. This tool builds from scratch some repositories we have at fixed intervals. I would like to point out that all these Mozilla tools are intended to work against CVS repositories. I'll explain later how to integrate the scratchbox in the continuous integration loop of the tinderbox.
The tinderbox updates a web page with the results of builds. It's also capable to show the checkins that took place between builds. The status page has the links to the Bonsai web pages that shows the files checked in and some other info, like developer comments, dates and so on. There is also another nice feature, if you include a number of a registered bug (in the Bugzilla) in any cvs checkin, the tinderbox will show a link to the bug in the status web page.
This guide is not intended to be a tutorial for configuring Mozilla webtools, so I'll assume that we have already an environment with all this stuff running. Our machine is running a Debian Sarge GNU/Linux system. I chose the library
hildon-libs to this example. I'll show how we configured the continuous integration environment in order to compile the
hildon-libs within the scratchbox.
Scratchbox configuration
We just followed the
Maemo tutorial in order to configure the scratchbox in the machine that will run the tinderbox loop. We added a scratchbox user called
buildmaster because it's the user we use for running the tinderbox. We wanted to have two different build environments, one for the libraries that come with the Maemo rootstrap, and another one with libraries built from the SVN trunk. The Maemo tutorial explains quite good how to configure several targets inside the same scratchbox.
Tinderbox configuration
Build script configuration
The tinderbox client configuration is located at this file
/var/local/tinderbox2/buildcf. We defined a target called
constructhildon like this:
$BUILD_DIR = $build_dir;
$SBOX_CMD="/usr/bin/scratchbox";
$SBOX_DIR="/scratchbox/users/$ENV{USER}/home/$ENV{USER}";
$BUILDS = {
'constructhildon' => [
{
'phase_name' => "print_environment",
'error_status' => "build_failed",
'dir' => "$BUILD_DIR",
'script' => [
"/var/local/tinderbox2/show_env",
"/var/local/tinderbox2/show_time",
"$SBOX_CMD rm -rf $module",
],
},
{
'phase_name' => "select_target",
'error_status' => "build_failed",
'dir' => "$BUILD_DIR",
'script' => [
"$SBOX_DIR/wait-if-not-target.sh $sbox_target",
"$SBOX_CMD sbox-config --select-target=$sbox_target",
],
},
{
'phase_name' => "checkout",
'error_status' => "build_failed",
'dir' => "$BUILD_DIR",
'script' => [
"$SBOX_CMD -d $sbox_target cvs -d $ENV{CVSROOT} checkout
$cvs_checkout_flags $module",
],
},
{
'phase_name' => "lint",
'error_status' => "test_failed",
'dir' => "$BUILD_DIR/$module",
'script' => [
"$SBOX_CMD ./maemo-lint.sh $sbox_target/$module",
],
},
{
'phase_name' => "configure",
'error_status' => "build_failed",
'dir' => "$BUILD_DIR/$module",
'script' => [
"$SBOX_CMD -d $sbox_target/$module ./autogen.sh --prefix=/usr",
],
},
{
'phase_name' => "build",
'error_status' => "build_failed",
'dir' => "$BUILD_DIR/$module",
'script' => [
"DISPLAY=:20 $SBOX_CMD -d $sbox_target/$module make",
],
},
{
'phase_name' => "clean",
'error_status' => "build_failed",
'dir' => "$BUILD_DIR/$module",
'script' => [
"$SBOX_CMD -d $sbox_target/$module make uninstall",
"$SBOX_CMD -d $sbox_target/$module make clean",
],
},
], # end construct
The tinderbox loop will execute all the above phases (
print_environment,
select_target,
checkout,
lint,
configure,
build, and
clean) by order. A few other comments about this code:
-
$module is replaced by the tinderbox scripts with all the modules specified in the file /var/local/tinderbox2/lib
- The
-d option of the scratchbox changes the directory in the scratchbox before executing the specified command
- There is a phase called
select_target that changes the current target of the scratchbox. Note that there could be some other active compilations at the same time that were using another target. That's why we call a script called wait-if-not-target.sh that waits a fixed amount of time before changing the target, if the current target is different from the one that the module is trying to set, with the hope that the other compilations have finished yet. In order to know the target of each module, we modified the VC_TREE variable defined into TreeData.pm file. This is an example:
'hildon-libs' => {
root => '/var/cvs',
module => 'hildon-libs',
branch => 'HEAD',
sbox_target => 'SDK_PC_HEAD',
},
In this case we will compile the HEAD of hildon-libs within our SDK_PC_HEAD target of the scratchbox. The
$sbox_target variable is read into the
build_scripts function this way:
my $sbox_target = $TreeData::VC_TREE{$tree_name}{'sbox_target'};
- The lint phase uses an script that executes the command splint over all the source code files. Note that both the lint phase and the unittest phase will return the
test_failed and not build_failed. The tinderbox is not stopped when a test fails.
- It's possible to add another phases to the integration loop. We can for example add phases for regression testing, performance measurements, memory checking, code coverage statistics.... For example if we're building a module with check support we could insert between the build and clean phases a new one called
unittest.
{
'phase_name' => "unittest",
'error_status' => "test_failed",
'dir' => "$BUILD_DIR/$module",
'script' => [
"nohup xvfb-run -n 20 -w 0 -p + xeyes &",
"$SBOX_CMD -d $sbox_target/$module make install",
"DISPLAY=:20 $SBOX_CMD af-sb-init.sh start",
"DISPLAY=:20 $SBOX_CMD $sbox_target/$module make check",
"DISPLAY=:20 $SBOX_CMD af-sb-init.sh stop",
"killall -q Xvfb || true",
],
},
There is also some other stuff to configure in the
buildcf file. In the function
build_environment we changed some environment variables:
# For generating docbook documentation
$ENV{SGML_CATALOG_FILES} = "/etc/sgml/catalog";
# User that will be used for executing the build commands
$ENV{USER} = "buildmaster";
$ENV{HOME} = "/home/buildmaster";
# Use distcc for distributed compiling
$ENV{CC} = "ccache distcc gcc-3.3";
$ENV{CCACHE_DIR} = "/var/tmp";
$ENV{CCACHE_UMASK} = "000";
$ENV{DISTCC_HOSTS} = "a_list_of_host_names_separated_by_blank_spaces";
As you can note we use also the distcc tool to scatter the compilation among several machines.
Testing tinderbox configuration
I want to remark that I used this command:
/usr/bin/perl -- /var/local/tinderbox2/build_shellscript -buildcf /var/local/tinderbox2/buildcf --once
--build constructhildon --tree hildon-libs --noreport
for the tests. The
--once parameter executes a single build. The
--noreport is useful because it sends no data to the tinderbox server (we avoided annoying e-mails). Another interesting option is
--test that only prints the commands that will be executed but without really executing them.
--
SergioVillar - 05 Apr 2006
div class="twikiTopicInfo twikiRevInfo twikiGrayText twikiMoved"<&/div>-->
