<html>
<head>
<link rel="stylesheet" href="style.css" type="text/css">
</head>
<body>
<hr>
<h1>The Hydra Buildfarm User Manual</h1>
<h2>Draft (Version 0.1)</h2>
<p/>
<h3>Eelco Dolstra and Eelco Visser</h3>
<p/>
Delft University of Technology
<p/>
Department of Software Technology
<p/>
Copyright 2008 Eelco Dolstra
<hr>
<h2>Chapter 1. Introduction</h2>
<h3>1.1. About Hydra</h3>
<li> In large projects, developers often work on a particular
component of the project, and do not build and test the composition
of those components (again since this is likely to take too long).
To prevent the phenomenon of ``big bang integration'', where
components are only tested together near the end of the development
process, it is important to test components together as soon as
possible (hence <em>continuous integration</em>).
<p/>
<li> It allows software to be <em>released</em> by automatically
creating packages that users can download and install. To do this
manually represents an often prohibitive amount of work, as one may
want to produce releases for many different platforms: e.g.,
installers for Windows and Mac OS X, RPM or Debian packages for
certain Linux distributions, and so on.
<p/>
</ol>
<p/>
</ol>
<p/>
Examples of continuous integration tools include CruiseControl
Tinderbox, Sisyphus, Anthill and BuildBot. These tools have various
limitations.
<ol>
<li> They do not manage the <em>build environment</em>. The build
environment consists of the dependencies necessary to perform a build
action, e.g., compilers, libraries, etc. Setting up the environment
is typically done manually, and without proper SCM control (so it may
be hard to reproduce a build at a later time). Manual management of
the environment scales poorly in the number of configurations that
must be supported. For instance, suppose that we want to build a
component that requires a certain compiler X. We then have to go to
each machine and install X. If we later need a newer version of X,
the process must be repeated all over again. An ever worse problem
occurs if there are conflicting, mutually exclusive versions of the
dependencies. Thus, simply installing the latest version is not an
option. Of course, we can install these components in different
directories and manually pass the appropriate paths to the build
processes of the various components. But this is a rather tiresome
and error-prone process. <p/>
<li> They do not easily support <em>variability in software
systems</em>. A system may have a great deal of build-time
variability: optional functionality, whether to build a debug or
production version, different versions of dependencies, and so on.
(For instance, the Linux kernel now has over 2,600 build-time
configuration switches.) It is therefore important that a continuous
integration tool can easily select and test different instances from
the configuration space of the system to reveal problems, such as
erroneous interactions between features. In a continuous integration
setting, it is also useful to test different combinations of versions
of subsystems, e.g., the head revision of a component against stable
releases of its dependencies, and vice versa, as this can reveal
various integration problems.
</ol>
<p/>
<em>Hydra</em>, is a continuous integration tool that solves these
problems. It is built on top of the <a href="http://nixos.org">Nix
package manager</a>, which has a purely functional language for
describing package build actions and their dependencies. This allows
the build environment for projects to be produced automatically and
deterministically, and variability in components to be expressed
naturally using functions; and as such is an ideal fit for a
continuous build system.
<p/>
The nixos.org installation of Hydra runs at
<blockquote>
<a href="http://hydra.nixos.org/">http://hydra.nixos.org</a>
</blockquote>
That installation is used to build software components from the
<a href="http://nixos.org">Nix</a>,
<a href="http://nixos.org/nixos">NixOS</a>,
<a href="http://strategoxt.org">Stratego/XT</a>,
and related projects.
If you are one of the developers on those projects, it is likely that
you will be using the NixOS Hydra server in some way. If you need to
administer automatic builds for your project, you should pull the
right strings to get an account on the server. This manual will tell
you how to set up new projects and build jobs within those projects
and write a release.nix file to describe the build process of your
project to Hydra. You can skip Chapter 2.
<p/>
If your project does not yet have automatic builds within the NixOS
Hydra server, it may actually be eligible. We are in the process of
setting up a large buildfarm that should be able to support open
source and academic software projects. Get in touch.
If you need to run your own Hydra installation, Chapter 2 explains
how to download and install the system on your own server.
<h2>Chapter 2. Installation and Configuration</h2>
This chapter explains how to install Hydra on your own buildfarm server.
<h3>2.1. Platform Requirements</h3>
<p/>
Of course we think it is a good idea to use the <a
href="http://nixos.org/nixos">NixOS</a> Linux distribution for your
buildfarm server. But this is not a requirement. The Nix software
deployment system can be installed on any Linux distribution in
parallel to the regular package management system. Thus, you can use
Hydra on a Suse, Fedora, or Ubuntu system.
<h3>2.2. Getting Nix</h3>
If your server runs NixOS you are all set to continue with
installation of Hydra. Otherwise you first need to install Nix.
The <a href="http://nixos.org/releases/nix/nix-0.12">latest stable release</a> is Nix 0.12.
Installation instructions can be found in the
<a href="http://nixos.org/releases/nix/nix-0.12/manual/">Nix User's Guide</a>.
<h3>2.3. Installation</h2>
To install Hydra, get the most recent 'closure' available from
<blockquote>
<a href="http://hydra.nixos.org/releases/hydra/unstable">http://hydra.nixos.org/releases/hydra/unstable</a>
</blockquote>
And follow the instructions that are revealed by clicking [help].
<pre class="programlisting">
$ gunzip < hydra-build.closure.gz | nix-store --import
</pre>
This unpacks the closure and imports its components into the Nix store.
<pre class="programlisting">
$ nix-env -i /nix/store/...-hydra-build
</pre>
This makes the tools in the Hydra package available in your Nix user
environment.
<p/>
Command completion should then reveal a number of tools related to
hydra installed:
<pre class="programlisting">
$ hydra_<tab>
hydra_build.pl hydra_fastcgi.pl hydra_scheduler.pl
hydra_cgi.pl hydra_init.pl hydra_server.pl
hydra_create.pl hydra_queue_runner.pl hydra_test.pl
</pre>
<h3>2.4. Configuration</h3>
The Hydra software is installed in the Nix store, but to run it needs
a directory for storing the database, logs, and session data. In your
<code>.bashrc</code> or similar configuration file define:
<pre class="programlisting">
export HYDRA_DATA=/usr/local/hydra
</pre>
and make sure that you actually create that directory. (Of course, you
can use another directory, but then remember to also substitute that
name in the commands below.)
<p/>
Run <code>hydra_init.pl</code> to initialize the database
<pre class="programlisting">
$ hydra_init.pl
</pre>
Run <code>hydra_server.pl</code> to start the webserver at <a href="http://localhost:3000">http://localhost:3000</a>
<pre class="programlisting">
$ hydra_server.pl
</pre>
Also start the scheduler, which monitors the source repositories and
adds builds to the queue, and the runner, which executes jobs in the
queue.
<pre class="programlisting">
$ hydra_scheduler.pl
$ hydra_queue_runner.pl
</pre>
Now your Hydra server should be up and running and the web interface
operational.
<h3>2.5. User Administration</h3>
To be able to add jobs and create projects you need to register users
in the Hydra database. In the current version, the web interface does
not yet support user administration. Use the following command to add
a new user to the database.
<pre class="programlisting">
$ sqlite3 /usr/local/hydra/hydra.sqlite "insert into Users(userName, emailAddress, password) values('eelco', 'blablah@example.org', '$(echo -n foobar | sha1sum | cut -c1-40)');"
</pre>
where <code>eelco</code> is the username, and <code>foobar</code> the
password. (Make sure to use other values!)
<p/>
To give this user administrator privileges, follow this up by:
<pre class="programlisting">
$ sqlite3 /usr/local/hydra/hydra.sqlite "insert into UserRoles(userName, role) values('eelco', 'admin');"
</pre>
Now you should be able to create a project using the Hydra web interface.
<h2>Chapter 3. Creating Projects</h2>
The next step is to add projects to the buildfarm. We follow the
example of the patchelf project at hydra.nixos.org. Note that the
error messages provided as feedback by the webinterface can be
somewhat unfriendly in the current version.
<p/>
<a href="http://localhost:3000/login">Login</a>
to the webinterface of your Hydra installation using
the username and password you inserted in the database.
Then follow the
'<a href="http://localhost:3000/createproject">Create Project</a>'
link to create a new project.
<h3>3.1. General information</h3>
</pre>
<pre>
I'm very sorry, but an error occurred:
DBIx::Class::ResultSet::create(): DBI Exception: DBD::SQLite::st execute failed: column name is not unique(19) at dbdimp.c line 402
</pre>
So try to create the project after entering just the general
information to figure out if you have chosen a unique name.
Jobsets can be added once the project has been created.
<p/>
A project can consist of multiple `jobsets', separate tasks that can
be built separately, but may depend on each other (without cyclic
dependencies, of course). Go to the
<a href="http://localhost:3000/project/patchelf/edit">Edit</a>
page of the Patchelf project and 'Add a new jobset'
by providing the following 'Information':
Identifier: trunk
Description: Trunk
Nix expression: release.nix in input patchelfSrc
</pre>
This states that in order to build the 'Trunk' jobset, the Nix
expression in the file 'release.nix', which can be obtained from input
'patchelfSrc', should be evaluated. (We'll have a look at release.nix
later.)
<p/>
To realize a job we probably need a number of inputs, which can be
declared in the table below. As many inputs as required can be added.
For patchelf we declare the following inputs.
nixpkgs 'CVS checkout' https://svn.nixos.org/repos/nix/nixpkgs/trunk
</pre>
officialRelease Boolean false
</pre>
system String value "i686-linux"
</pre>
system
there must be one primary job
check the radio button of exactly one job
https://svn.nixos.org/repos/nix/nixpkgs/trunk
- Voorbeelden van Nix expressies voor Hydra:
https://svn.nixos.org/repos/nix/patchelf/trunk/release.nix
https://svn.nixos.org/repos/nix/nix/trunk/release.nix
https://svn.nixos.org/repos/nix/hydra/trunk/release.nix
Overigens zijn die helemaal niet Hydra-specifiek, je kunt ze gewoon vanaf de
command line bouwen, bijv. als je een patchelf checkout hebt (met een nixpkgs
checkout in ../nixpkgs):
$ nix-build release.nix -A rpm_fedora10i386
</body>
</html>
<h3>3.5. Building on the command line</h3>
<h3>3.4. release.nix</h3>
<h3>3.3. Building Jobs</h3>
<h3>3.2. Release Set</h3>
<pre class="programlisting">
officialRelease
nixpkgs
<pre class="programlisting">
<pre class="programlisting">
patchelfSrc
'Subversion checkout' https://svn.nixos.org/repos/nix/patchelf/trunk
</pre>
patchelfSrc
<pre class="programlisting">
<pre class="programlisting">
<h3>3.2. Jobsets</h3>
Once created there should be an entry for the project in the
sidebar. Go to the project page for the <a
href="http://localhost:3000/project/patchelf">Patchelf</a> project.
Only if the project is <strong>enabled</strong> are builds performed.
<pre class="programlisting">
Display name: Patchelf
</pre>
The <strong>display name</strong> is used in menus.
<pre class="programlisting">
Description: A tool for modifying ELF binaries
</pre>
The <strong>description</strong> is used as short documentation of the
nature of the project.
<pre class="programlisting">
Owner: eelco
</pre>
The <strong>owner</strong> of a project can create and edit jobsets.
<pre class="programlisting">
Enabled: Yes
</pre>
The <strong>identifier</strong> is the identity of the project. It is
used in URLs and in the names of build results.
<p/>
The identifier should be a unique name (it is the
primary database key for the project table in the database). If you
try to create a project with an already existing identifier you'd get
an error message such as:
<pre class="programlisting">
Identifier: patchelf
Here's what we fill in for the patchelf project:
A project definition consists of some general information and a set of
jobsets. The general information identifies a project, its owner, and
current state of activity.
To run Hydra you need a Linux server. For small projects, Hydra can
be run on any reasonably modern machine. For individual projects you
can even run Hydra on a laptop. However, the charm of a buildfarm
server is usually that it operates without disturbing the developer's
working environment and can serve releases over the internet. In
conjunction you should typically have your source code administered in
a version management system, such as subversion. Therefore, you will
probably want to install a server that is connected to the
internet. To scale up to large and/or many projects, you will need at
least a considerable amount of diskspace to store builds. Since Hydra
can schedule multiple simultaneous build jobs, it can be useful to
have a multi-core machine, and/or attach multiple build machines in a
network to the central Hydra server.
<h3>1.6. Hydra on your own buildfarm</h3>
<h3>1.4. License</h3>
Hydra is free software; you can redistribute it and/or modify it under
the terms of the GNU Lesser General Public License as published by the
Free Software Foundation; either version 2.1 of the License, or (at
your option) any later version. Hydra is distributed in the hope that
it will be useful, but WITHOUT ANY WARRANTY; without even the implied
warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See
the GNU Lesser General Public License for more details.
<h3>1.5. Hydra at nixos.org</h3>
<h3>1.2. About Us</h3>
Hydra is the successor of the Nix Buildfarm, which was developed in
tandem with the Nix software deployment system. Nix was originally
developed at the Department of Information and Computing Sciences,
Utrecht University by the TraCE project (2003-2008). The project was
funded by the Software Engineering Research Program Jacquard to
improve the support for variability in software systems. Funding for
the development of Nix and Hydra is now provided by the NIRICT LaQuSo
Build Farm project.
<h3>1.3. About this Manual</h3>
This manual tells you how to install the Hydra buildfarm software on
your own server and how to operate that server using its web
interface.
In its simplest form, a continuous integration tool sits in a loop
building and releasing software components from a version management
system. For each component, it performs the following tasks:
<ol>
<li>It obtains the latest version of the component's source code
from the version management system.
<li> It runs the component's build process (which presumably includes
the execution of the component's test set).
<li> It presents the results of the build (such as error logs and
releases) to the developers, e.g., by producing a web page.
<li> Likewise, many projects have very large test sets (e.g.,
regression tests in a compiler, or stress tests in a DBMS) that can
take hours or days to run to completion.
<p/>
<li> Many kinds of static and dynamic analyses can be performed as
part of the tests, such as code coverage runs and static analyses.
<p/>
<li> It may also be necessary to build many different <em>variants</em>
of the software. For instance, it may be necessary to verify that
the component builds with various versions of a compiler.
<p/>
<li> Developers typically use incremental building to test their
changes (since a full build may take too long), but this is
unreliable with many build management tools (such as Make), i.e.,
the result of the incremental build might differ from a full build.
<p/>
<li> It ensures that the software can be built from the sources under
revision control. Users of version management systems such as CVS
and Subversion often forget to place source files under revision
control.
<p/>
<li> The machines on which the continuous integration system runs
ideally provides a clean, well-defined build environment. If this
environment is administered through proper SCM techniques, then
builds produced by the system can be reproduced. In contrast,
developer work environments are typically not under any kind of SCM
control.
<p/>
Hydra is a tool for continuous integration testing and software
release that uses a purely functional language to describe build jobs
and their dependencies. Continuous integration is a simple technique
to improve the quality of the software development process. An
automated system continuously or periodically checks out the source
code of a project, builds it, runs tests, and produces reports for the
developers. Thus, various errors that might accidentally be committed
into the code base are automatically caught. Such a system allows
more in-depth testing than what developers could feasibly do manually:
<p/>
<ol>
<li> <em>Portability testing</em>: The software may need to be built
and tested on many different platforms. It is infeasible for each
developer to do this before every commit.
<p/>