doc: Reintegrate the intro by Visser & Dolstra from `manual.html'.

[?]

Mar 9, 2011, 5:13 PM

CPQQFQMZ3W7ISBF6L6WP2BWQYUZ6HQQI2VANQNRRBWT6JRIKR3KAC

Dependencies

[2] 5AKAE3R6 start of hydra manual
[3] KBW3FDZ2 Merge remote branch 'remotes/origin/master'

Change contents

edit in doc/manual/introduction.xml at line 4

[3.3933]

[3.3933]

<title>Introduction</title>

<section>
<title>About Hydra</title>

<para>
Hydra is a tool for continuous integration testing and software
release that uses a purely functional language to describe build jobs
and their dependencies. Continuous integration is a simple technique
to improve the quality of the software development process. An
automated system continuously or periodically checks out the source
code of a project, builds it, runs tests, and produces reports for the
developers. Thus, various errors that might accidentally be committed
into the code base are automatically caught. Such a system allows
more in-depth testing than what developers could feasibly do manually:

<itemizedlist>
<listitem> <emphasis>Portability testing</emphasis>: The
software may need to be built and tested on many different
platforms. It is infeasible for each developer to do this
before every commit.
</listitem>

<listitem> Likewise, many projects have very large test sets
(e.g., regression tests in a compiler, or stress tests in a
DBMS) that can take hours or days to run to completion.
</listitem>

<listitem> Many kinds of static and dynamic analyses can be
performed as part of the tests, such as code coverage runs and
static analyses.
</listitem>

<listitem> It may also be necessary to build many different
<emphasis>variants</emphasis> of the software. For instance,
it may be necessary to verify that the component builds with
various versions of a compiler.
</listitem>

<listitem> Developers typically use incremental building to
test their changes (since a full build may take too long), but
this is unreliable with many build management tools (such as
Make), i.e., the result of the incremental build might differ
from a full build.
</listitem>

<listitem> It ensures that the software can be built from the
sources under revision control. Users of version management
systems such as CVS and Subversion often forget to place
source files under revision control.
</listitem>

<listitem> The machines on which the continuous integration
system runs ideally provides a clean, well-defined build
environment. If this environment is administered through
proper SCM techniques, then builds produced by the system can
be reproduced. In contrast, developer work environments are
typically not under any kind of SCM control.
</listitem>

<listitem> In large projects, developers often work on a
particular component of the project, and do not build and test
the composition of those components (again since this is
likely to take too long). To prevent the phenomenon of ``big
bang integration'', where components are only tested together
near the end of the development process, it is important to
test components together as soon as possible (hence
<emphasis>continuous integration</emphasis>).
</listitem>

<listitem> It allows software to be
<emphasis>released</emphasis> by automatically creating
packages that users can download and install. To do this
manually represents an often prohibitive amount of work, as
one may want to produce releases for many different platforms:
e.g., installers for Windows and Mac OS X, RPM or Debian
packages for certain Linux distributions, and so on.
</listitem>

</itemizedlist>
</para>

<para>
In its simplest form, a continuous integration tool sits in a
loop building and releasing software components from a version
management system. For each component, it performs the
following tasks:

<itemizedlist>

<listitem>It obtains the latest version of the component's
source code from the version management system.
</listitem>

<listitem> It runs the component's build process (which
presumably includes the execution of the component's test
set).
</listitem>

<listitem> It presents the results of the build (such as error
logs and releases) to the developers, e.g., by producing a web
page.
</listitem>

</itemizedlist>

Examples of continuous integration tools include Jenkins,
CruiseControl Tinderbox, Sisyphus, Anthill and BuildBot. These
tools have various limitations.

<itemizedlist>
replacement in doc/manual/introduction.xml at line 117

[3.3934]→[3.3934:3962](∅→∅)

<title>Introduction</title>

[3.3934]

[3.3962]

<listitem> They do not manage the <emphasis>build
environment</emphasis>. The build environment consists of the
dependencies necessary to perform a build action, e.g.,
compilers, libraries, etc. Setting up the environment is
typically done manually, and without proper SCM control (so it
may be hard to reproduce a build at a later time). Manual
management of the environment scales poorly in the number of
configurations that must be supported. For instance, suppose
that we want to build a component that requires a certain
compiler X. We then have to go to each machine and install X.
If we later need a newer version of X, the process must be
repeated all over again. An ever worse problem occurs if
there are conflicting, mutually exclusive versions of the
dependencies. Thus, simply installing the latest version is
not an option. Of course, we can install these components in
different directories and manually pass the appropriate paths
to the build processes of the various components. But this is
a rather tiresome and error-prone process.
</listitem>

<listitem> They do not easily support <emphasis>variability in software
systems</emphasis>. A system may have a great deal of build-time
variability: optional functionality, whether to build a debug or
production version, different versions of dependencies, and so on.
(For instance, the Linux kernel now has over 2,600 build-time
configuration switches.) It is therefore important that a continuous
integration tool can easily select and test different instances from
the configuration space of the system to reveal problems, such as
erroneous interactions between features. In a continuous integration
setting, it is also useful to test different combinations of versions
of subsystems, e.g., the head revision of a component against stable
releases of its dependencies, and vice versa, as this can reveal
various integration problems.
</listitem>

</itemizedlist>
</para>

<para>
<emphasis>Hydra</emphasis>, is a continuous integration tool
that solves these problems. It is built on top of the <link
xlink:href="http://nixos.org/nix/">Nix package manager</link>,
which has a purely functional language for describing package
build actions and their dependencies. This allows the build
environment for projects to be produced automatically and
deterministically, and variability in components to be expressed
naturally using functions; and as such is an ideal fit for a
continuous build system.
</para>

</section>

<section>
<title>About Us</title>

<para>
Hydra is the successor of the Nix Buildfarm, which was developed
in tandem with the Nix software deployment system. Nix was
originally developed at the Department of Information and
Computing Sciences, Utrecht University by the TraCE project
(2003-2008). The project was funded by the Software Engineering
Research Program Jacquard to improve the support for variability
in software systems. Funding for the development of Nix and
Hydra is now provided by the NIRICT LaQuSo Build Farm project.
</para>
</section>

<section>
<title>About this Manual</title>

<para>
This manual tells you how to install the Hydra buildfarm
software on your own server and how to operate that server using
its web interface.
</para>
</section>

<section>
<title>License</title>

<para>
Hydra is free software: you can redistribute it and/or
modify it under the terms of the GNU General Public License as
published by the Free Software Foundation, either version 3 of
the License, or (at your option) any later version.
</para>

<para>
Hydra is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
<link xlink:href="http://www.gnu.org/licenses/">GNU General
Public License</link> for more details.
</para>
</section>

<section>
<title>Hydra at <literal>nixos.org</literal></title>

<para>
The <literal>nixos.org</literal> installation of Hydra runs at
<link
xlink:href="http://hydra.nixos.org/"><literal>http://hydra.nixos.org/</literal></link>.

That installation is used to build software components from the
<link xlink:href="http://nixos.org">Nix</link>,
<link xlink:href="http://nixos.org/nixos">NixOS</link>,
<link xlink:href="http://www.gnu.org/">GNU</link>,
<link xlink:href="http://strategoxt.org">Stratego/XT</link>,
and related projects.
</para>

<para>
If you are one of the developers on those projects, it is likely
that you will be using the NixOS Hydra server in some way. If
you need to administer automatic builds for your project, you
should pull the right strings to get an account on the
server. This manual will tell you how to set up new projects and
build jobs within those projects and write a release.nix file to
describe the build process of your project to Hydra. You can
skip the next chapter.
</para>

<para>
If your project does not yet have automatic builds within the
NixOS Hydra server, it may actually be eligible. We are in the
process of setting up a large buildfarm that should be able to
support open source and academic software projects. Get in
touch.
</para>
</section>

<section>
<title>Hydra on your own buildfarm</title>

<para>
If you need to run your own Hydra installation, <xref
linkend="chap-installation" /> explains how to download and
install the system on your own server.
</para>
</section>
edit in doc/manual/manual.xml at line 33

[2.894]

[2.894]

<author>
<personname>
<firstname>Eelco</firstname>
<surname>Visser</surname>
</personname>
<affiliation>
<orgname>Delft University of Technology</orgname>
<orgdiv>Department of Software Technology</orgdiv>
</affiliation>
<contrib>Author</contrib>
</author>