Integration of JHBuild and Buildbot
Author:
JoseDapenaPaz
Context of the project
Build brigade
In
Gnome Build Brigade
we want to set up a continuous integration service for all Gnome projects. After a meeting in
Guadec 2006, we established the team and decided to try with Buildbot. There was a discussion about the tools (currently Frederic Peters maintains
JHAutobuild, and I showed
my work with Tinderbox 3).
Since Guadec, I've been doing some integration work among JHbuild and Buildbot, in order to set up a Buildbot for all the Gnome JHBuild moduleset projects.
Buildbot initial status
Buildbot has the following architecture:
- Some slaves: the slaves are the compilation controllers. They run the compilation tools and retrieve the results from the compilation tree.
- Some masters: a master is a piece of software with the following functionalities:
- Control the slaves, scheduling the steps of each compilation.
- Manage the reporting, providing a web interface, issuing mails, or IRC notifications.
The design of Buildbot is intended for holding just one project in each master. This means that, for Gnome project, we should have just one master per project. And also by design, each slave belongs to only one master. In the standard Buildbot, you have to define a server for each master and slave, and the schedulers are associated with projects, with no obvious way to connect them.
Buildbot is implemented in
Twisted framework. It's a
Python event-driven network framework.
Goals
JHBuild should handle compilations and version control interaction
JHbuild is a very good tool to handle builds. The build control should happen in JHbuild as far as we can.
Reasons:
- JHbuild has already implemented the control for Gnome projects (not only standard autotools-based projects, but also Python ones). And this implementation is being actively maintained by developers and integrators.
- JHBuild also handles the Version Control tools used in Gnome projects. This is also actively maintained.
- The JHBuild moduleset defines the Gnome projects list, and gives a lot of metainformation. These modulesets are actively maintained, and maintaining a separate setup would give problems to keep sync.
Then,
a goal is making Buildbot rely exclusively on JHBuild for compilation and version control interactions.
Reduce the impact of multimaster/multislave
In the original architecture of Buildbot, we would have just one master per project (150 servers, more or less), and one slave per project and setup (just another 150 servers). We want to make the masters and slaves share resources.
Reasons:
- If we can make one server maintain just all masters, less resources are used. Same applies to slaves.
- We can write cooperation code among the masters inside this one server. For example this would be used to let schedulers cooperate and avoid running compilation of all projects in one machine at the same time.
Another
goal is making all masters be created in just one Twisted server, and all slaves be created in just another Twisted server.
Improve reporting systems of Buildbot
We want to get:
- A global view of all projects status.
- Rss feeds for getting compilation failures in each project.
JHBuild buildbot
Changes in JHBuild
- Implementation of checkout modes in Buildbot (blog entry, bug and patch.
- Avoid JHbuild breakage running unit tests. (same blog entry, bug and patch. This is not required for my final solution, that requires the "pathrun" patch only (see the following point).
- Add a "pathrun" command that enables you to run specific commands in the module build directory. It's used in my Buildbot changes to run unit tests (bug and patch).
Changes in default Buildbot web UI
- Added an RSS feed for each waterfall view (bug and patch).
- Added a link in each builder view to the waterfall showing only that builder (bug and patch).
Buildbot master and slave scripts
I've created custom scripts for running buildbot for all JHBuild gnome projects. They're in Igalia CVS:
- CVS root:
:pserver:anonymous@cvs.igalia.com:/var/publiccvs
- Module:
jhbuild-buildbot-scripts
- Browse module: bonsai web.
Installation of multimaster
- Install buildbot in the server
- Create a user in the machine where the master will be run. For example,
buildmaster.
- In the buildmaster account, you should copy the files of the
master directory.
- Then you should edit the
multimaster.cfg file. It's commented, you should set up all the significant variables for your setup.
- Install jhbuild in a path of the master account.
Installation of multislave
- Install buildbot in the client. In the client you should also install lcov (it's used for coverage reports).
- Create a user in the machine where the slave will be run. For example,
buildslave.
- In the buildslave account, you should copy the files of the
slave directory.
- Then you should also edit the
multislave.cfg file to make it fit your deployment.
- Install jhbuild in a path of the slave account.
- Copy the slaves/scripts files to the PATH of the slave user. These scripts are used to run unit tests and coverage reports clean. You should tweak at least the module-reports.sh script.
Last steps
- Set up the jhbuildrc and copy it to the places specified in the master and slave accounts config (see multimaster.cfg and multislave.cfg).
- Create the directories in the slave for binaries and sources as they'll be obtained from jhbuild. If you use the "copy" checkout mode of jhbuild, you should also establish the copy directory.
- Run the update-buildmasters script in master account. This will create the buildslaves.list file. This file should be copied to the slaves machine/account (see the multislaves.cfg to know where it expects to find the file). The update-buildmasters should be run everytime the moduleset is expected to change.
Startup and shutdown
To startup the master, you should write "make start" in the master install directory. To startup the slave, you should also write "make start" in the slave install directory.
To shutdown the master and the server, you use "make stop". Sometimes the master takes some time to get stopped. It's normal.
You can view the web interface connecting with a web browser to the host and port you specified in the multimaster.cfg file.
Implementation notes
- These scripts are based in Einar Karttunen ones, published in buildbot-devel list.
- The idea is making one buildbot.tac run each master (in the masters server) and each slave (in the slaves server). This server includes custom schedulers to make the compilations of all the build set happen one after another.
- There's a JHBuild factory to create builders for each jhbuild module. It also adds steps/commands to implement jhbuild interaction. The provided factory runs
- A JHbuild updateone for the module
- A JHbuild buildone for the module
- A make check (through pathrun interface).
- Generation of coverage reports.
- The buildbot.tac includes also a multiproject view. It's the one shown when you connect to the port specified in multimaster.cfg. It includes the current status of each project and builder, and shows links to the standard waterfall view of them.
--
JoseDapenaPaz - 26 Sep 2006
- IRC bot and mailing list easy setup from multimaster.cfg
- Changes view from a custom gnome-cvs-commits mailing list fetcher.
Final notes
- This should be run with current buildbot Head.
To do
- Add configuration for email and IRC reporting.
- Done. It's now set up using multimaster.cfg. Anyway, it would be great if I could get one of those wonderful irc bots from thomasvs for gstreamer. -- JoseDapenaPaz - 26 Sep 2006
- Add configuration for changes reports of modules with Version Control system.
- Partially done. I've implemented a source changes fetcher for gnome-cvs-commits mailing list through a maildir. It works for most of gnome projects. -- JoseDapenaPaz - 26 Sep 2006
- Add a bugzilla service to allow tracking of requests and forget this nasty TODO.
Examples running
--
JoseDapenaPaz - 18 Sep 2006
div class="twikiTopicInfo twikiRevInfo twikiGrayText twikiMoved"<&/div>-->
