ripped intro from the SCP paper and manualized it

[?]

Dec 1, 2008, 11:20 PM

YOC4WW2T4CGI4OV6BRFOEPTBURZSIU7JSRH3B7HAT5DVHV2IPQQQC

Dependencies

[2] 6A6CZ7SC draft manual
[3] Y3UAIWOK general information
[4] EWQC6EZV general information
[5] QPXJVRYK boilerplate, platform stuff

Change contents

edit in doc/manual/manual.html at line 32

[2.430]

[2.430]

Hydra is a tool for continuous integration testing and software
release that uses a purely functional language to describe build jobs
and their dependencies. Continuous integration is a simple technique
to improve the quality of the software development process. An
automated system continuously or periodically checks out the source
code of a project, builds it, runs tests, and produces reports for the
developers. Thus, various errors that might accidentally be committed
into the code base are automatically caught. Such a system allows
more in-depth testing than what developers could feasibly do manually:

<p/>

<ol>

<li> <em>Portability testing</em>: The software may need to be built
and tested on many different platforms. It is infeasible for each
developer to do this before every commit.
<p/>
replacement in doc/manual/manual.html at line 52

[2.431]→[2.431:504](∅→∅)

Hydra is a buildfarm system based on the Nix software deployment
system.

[2.431]

[2.504]

<li> Likewise, many projects have very large test sets (e.g.,
regression tests in a compiler, or stress tests in a DBMS) that can
take hours or days to run to completion.
<p/>

<li> Many kinds of static and dynamic analyses can be performed as
part of the tests, such as code coverage runs and static analyses.
<p/>

<li> It may also be necessary to build many different <em>variants</em>
of the software. For instance, it may be necessary to verify that
the component builds with various versions of a compiler.
<p/>

<li> Developers typically use incremental building to test their
changes (since a full build may take too long), but this is
unreliable with many build management tools (such as Make), i.e.,
the result of the incremental build might differ from a full build.
<p/>

<li> It ensures that the software can be built from the sources under
revision control. Users of version management systems such as CVS
and Subversion often forget to place source files under revision
control.
<p/>

<li> The machines on which the continuous integration system runs
ideally provides a clean, well-defined build environment. If this
environment is administered through proper SCM techniques, then
builds produced by the system can be reproduced. In contrast,
developer work environments are typically not under any kind of SCM
control.
<p/>
edit in doc/manual/manual.html at line 86

[2.505]

[2.505]

<li> In large projects, developers often work on a particular
component of the project, and do not build and test the composition
of those components (again since this is likely to take too long).
To prevent the phenomenon of ``big bang integration'', where
components are only tested together near the end of the development
process, it is important to test components together as soon as
possible (hence <em>continuous integration</em>).
<p/>

<li> It allows software to be <em>released</em> by automatically
creating packages that users can download and install. To do this
manually represents an often prohibitive amount of work, as one may
want to produce releases for many different platforms: e.g.,
installers for Windows and Mac OS X, RPM or Debian packages for
certain Linux distributions, and so on.
<p/>

</ol>
replacement in doc/manual/manual.html at line 107

[2.511]→[2.511:530](∅→∅)

... advantages ...

[2.511]

[2.530]

In its simplest form, a continuous integration tool sits in a loop
building and releasing software components from a version management
system. For each component, it performs the following tasks:

<ol>

<li>It obtains the latest version of the component's source code
from the version management system.

<li> It runs the component's build process (which presumably includes
the execution of the component's test set).

<li> It presents the results of the build (such as error logs and
releases) to the developers, e.g., by producing a web page.
edit in doc/manual/manual.html at line 122

[2.531]

[2.531]

</ol>
edit in doc/manual/manual.html at line 125

[2.537]

[2.537]

Examples of continuous integration tools include CruiseControl
Tinderbox, Sisyphus, Anthill and BuildBot. These tools have various
limitations.

<ol>
edit in doc/manual/manual.html at line 132

[2.538]

[2.538]

<li> They do not manage the <em>build environment</em>. The build
environment consists of the dependencies necessary to perform a build
action, e.g., compilers, libraries, etc. Setting up the environment
is typically done manually, and without proper SCM control (so it may
be hard to reproduce a build at a later time). Manual management of
the environment scales poorly in the number of configurations that
must be supported. For instance, suppose that we want to build a
component that requires a certain compiler X. We then have to go to
each machine and install X. If we later need a newer version of X,
the process must be repeated all over again. An ever worse problem
occurs if there are conflicting, mutually exclusive versions of the
dependencies. Thus, simply installing the latest version is not an
option. Of course, we can install these components in different
directories and manually pass the appropriate paths to the build
processes of the various components. But this is a rather tiresome
and error-prone process. <p/>

<li> They do not easily support <em>variability in software
systems</em>. A system may have a great deal of build-time
variability: optional functionality, whether to build a debug or
production version, different versions of dependencies, and so on.
(For instance, the Linux kernel now has over 2,600 build-time
configuration switches.) It is therefore important that a continuous
integration tool can easily select and test different instances from
the configuration space of the system to reveal problems, such as
erroneous interactions between features. In a continuous integration
setting, it is also useful to test different combinations of versions
of subsystems, e.g., the head revision of a component against stable
releases of its dependencies, and vice versa, as this can reveal
various integration problems.

</ol>
edit in doc/manual/manual.html at line 167

[2.544]

[2.544]

<em>Hydra</em>, is a continuous integration tool that solves these
problems. It is built on top of the <a href="http://nixos.org">Nix
package manager</a>, which has a purely functional language for
describing package build actions and their dependencies. This allows
the build environment for projects to be produced automatically and
deterministically, and variability in components to be expressed
naturally using functions; and as such is an ideal fit for a
continuous build system.
edit in doc/manual/manual.html at line 268

[2.2581]→[2.2581:2608](∅→∅)

<p/>

Hydra on Windows??
replacement in doc/manual/manual.html at line 460

[2.7649]→[2.7649:7666](∅→∅)

<h3>Jobsets</h3>

[2.7649]

[2.7666]

<h3>3.2. Jobsets</h3>
replacement in doc/manual/manual.html at line 513

[2.8844]→[2.8844:8865](∅→∅)

<h3>Release Set</h3>

[2.8844]

[2.8865]

<h3>3.2. Release Set</h3>
replacement in doc/manual/manual.html at line 521

[2.8987]→[2.8987:9010](∅→∅)

<h3>Building Jobs</h3>

[2.8987]

[2.9010]

<h3>3.3. Building Jobs</h3>
replacement in doc/manual/manual.html at line 524

[2.9012]→[2.9012:9033](∅→∅)

<h3>release.nix</h3>

[2.9012]

[2.9033]

<h3>3.4. release.nix</h3>
replacement in doc/manual/manual.html at line 536

[2.9254]→[2.9254:9292](∅→∅)

<h3>Building on the command line</h3>

[2.9254]

[2.9292]

<h3>3.5. Building on the command line</h3>
edit in doc/manual/manual.html at line 545

[2.9520]→[2.9520:9544](∅→∅)

<h3>Release Set</h3>