{"header":{"message":"draft manual","description":null,"timestamp":"2008-12-01T20:03:18Z","authors":[{"email":"visser@acm.org","name":"Eelco Visser"}]},"hash":"6A6CZ7SC2VGEGRC5LSNTDDM3LFWVYO6Z6QNDF74SBJWO5P6T3ARQC","authors":[],"deps":{"hashes":[["6A6CZ7SC2VGEGRC5LSNTDDM3LFWVYO6Z6QNDF74SBJWO5P6T3ARQC","draft manual"],["AA",null],["CLXEECMFKLUIN5QBV3BPPPSU6G5UF6MXRSNWA6LXUBNQGSJY4U5QC","* Start putting build results in a database."]],"deps":["CLXEECMFKLUIN5QBV3BPPPSU6G5UF6MXRSNWA6LXUBNQGSJY4U5QC"],"known":[]},"hunks":[{"FileAdd":{"meta":{"basename":"doc","metadata":576},"context":[{"change":2,"pos":2}],"contents":[]}},{"FileAdd":{"meta":{"basename":"manual","metadata":576},"context":[{"change":0,"pos":1}],"contents":[]}},{"FileAdd":{"meta":{"basename":"manual.html","metadata":0},"context":[{"change":0,"pos":18}],"contents":[{"Add":""},{"Add":""},{"Add":""},{"Add":""},{"Add":""},{"Add":""},{"Add":"

"},{"Add":""},{"Add":"

The Hydra Buildfarm User Manual

"},{"Add":""},{"Add":""},{"Add":"

Draft (Version 0.1)

"},{"Add":""},{"Add":"

Eelco Dolstra and Eelco Visser

"},{"Add":""},{"Add":"

"},{"Add":"Delft University of Technology"},{"Add":"

"},{"Add":"Department of Software Technology"},{"Add":""},{"Add":"

"},{"Add":""},{"Add":"

Chapter 1. Introduction

"},{"Add":""},{"Add":""},{"Add":"

1.1. About Hydra

"},{"Add":""},{"Add":"Hydra is a buildfarm system based on the Nix software deployment"},{"Add":"system."},{"Add":""},{"Add":"

"},{"Add":""},{"Add":"... advantages ..."},{"Add":""},{"Add":""},{"Add":"

"},{"Add":""},{"Add":"

"},{"Add":""},{"Add":""},{"Add":"

"},{"Add":""},{"Add":"

1.2. Hydra at nixos.org

"},{"Add":""},{"Add":"The nixos.org installation of Hydra runs at "},{"Add":""},{"Add":"

"},{"Add":" http://hydra.nixos.org"},{"Add":"

"},{"Add":""},{"Add":"That installation is used to build software components from the "},{"Add":"Nix, "},{"Add":"NixOS, "},{"Add":"Stratego/XT, "},{"Add":"and related projects. "},{"Add":""},{"Add":"If you are one of the developers on those projects, it is likely that"},{"Add":"you will be using the NixOS Hydra server in some way. If you need to"},{"Add":"administer automatic builds for your project, you should pull the"},{"Add":"right strings to get an account on the server. This manual will tell"},{"Add":"you how to set up new projects and build jobs within those projects"},{"Add":"and write a release.nix file to describe the build process of your"},{"Add":"project to Hydra. You can skip Chapter 2."},{"Add":""},{"Add":"

"},{"Add":""},{"Add":"If your project does not yet have automatic builds within the NixOS"},{"Add":"Hydra server, it may actually be eligible. We are in the process of"},{"Add":"setting up a large buildfarm that should be able to support open"},{"Add":"source and academic software projects. Get in touch."},{"Add":""},{"Add":"

1.3. Hydra on your own buildfarm

"},{"Add":""},{"Add":"If you need to run your own Hydra installation, Chapter 2 explains"},{"Add":"how to download and install the system on your own server."},{"Add":""},{"Add":""},{"Add":"

Chapter 2. Installation and Configuration

"},{"Add":""},{"Add":"This chapter explains how to install Hydra on your own buildfarm server."},{"Add":"We assume that you have "},{"Add":""},{"Add":"

2.1. Platform Requirements

"},{"Add":""},{"Add":"To run Hydra you need a Linux server with at least a considerable"},{"Add":"amount of diskspace to store builds. A multi-core machine is not a"},{"Add":"waste since Hydra can schedule multiple simultaneous build jobs."},{"Add":""},{"Add":"

"},{"Add":""},{"Add":"Of course we think it is a good idea to use the NixOS Linux distribution for your"},{"Add":"buildfarm server. But this is not a requirement. The Nix software"},{"Add":"deployment system can be installed on any Linux distribution in"},{"Add":"parallel to the regular package management system. Thus, you can use"},{"Add":"Hydra on a Suse, Fedora, or Ubuntu system."},{"Add":""},{"Add":"

"},{"Add":""},{"Add":"Hydra on Windows??"},{"Add":""},{"Add":""},{"Add":"

2.2. Getting Nix

"},{"Add":""},{"Add":"If your server runs NixOS you are all set to continue with"},{"Add":"installation of Hydra. Otherwise you first need to install Nix."},{"Add":"The latest stable release is Nix 0.12."},{"Add":"Installation instructions can be found in the "},{"Add":"Nix User's Guide."},{"Add":""},{"Add":"

"},{"Add":""},{"Add":"Why Nix ..."},{"Add":""},{"Add":"

2.3. Installation

"},{"Add":""},{"Add":"To install Hydra, get the most recent 'closure' available from"},{"Add":""},{"Add":"

"},{"Add":" http://hydra.nixos.org/releases/hydra/unstable"},{"Add":"

"},{"Add":""},{"Add":"And follow the instructions that are revealed by clicking [help]."},{"Add":""},{"Add":"

"},{"Add":"$ gunzip < hydra-build.closure.gz | nix-store --import"},{"Add":"

"},{"Add":""},{"Add":"This unpacks the closure and imports its components into the Nix store."},{"Add":""},{"Add":"

"},{"Add":"$ nix-env -i /nix/store/...-hydra-build"},{"Add":"

"},{"Add":""},{"Add":"This makes the tools in the Hydra package available in your Nix user"},{"Add":"environment."},{"Add":""},{"Add":"

"},{"Add":""},{"Add":"Command completion should then reveal a number of tools related to"},{"Add":"hydra installed:"},{"Add":""},{"Add":"

"},{"Add":"$ hydra_<tab>"},{"Add":"hydra_build.pl         hydra_fastcgi.pl       hydra_scheduler.pl"},{"Add":"hydra_cgi.pl           hydra_init.pl          hydra_server.pl"},{"Add":"hydra_create.pl        hydra_queue_runner.pl  hydra_test.pl"},{"Add":"

"},{"Add":""},{"Add":""},{"Add":"

2.4. Configuration

"},{"Add":""},{"Add":"The Hydra software is installed in the Nix store, but to run it needs"},{"Add":"a directory for storing the database, logs, and session data. In your"},{"Add":".bashrc or similar configuration file define:"},{"Add":""},{"Add":"

"},{"Add":" export HYDRA_DATA=/usr/local/hydra"},{"Add":"

"},{"Add":""},{"Add":"and make sure that you actually create that directory. (Of course, you"},{"Add":"can use another directory, but then remember to also substitute that"},{"Add":"name in the commands below.)"},{"Add":""},{"Add":"

"},{"Add":""},{"Add":"Run hydra_init.pl to initialize the database"},{"Add":""},{"Add":"

"},{"Add":"$ hydra_init.pl"},{"Add":"

"},{"Add":""},{"Add":"Run hydra_server.pl to start the webserver at http://localhost:3000"},{"Add":""},{"Add":"

"},{"Add":"$ hydra_server.pl"},{"Add":"

"},{"Add":""},{"Add":"Also start the scheduler, which monitors the source repositories and"},{"Add":"adds builds to the queue, and the runner, which executes jobs in the"},{"Add":"queue."},{"Add":""},{"Add":"

"},{"Add":"$ hydra_scheduler.pl"},{"Add":"$ hydra_queue_runner.pl"},{"Add":"

"},{"Add":""},{"Add":"Now your Hydra server should be up and running and the web interface"},{"Add":"operational."},{"Add":""},{"Add":"

2.5. User Administration

"},{"Add":""},{"Add":"To be able to add jobs and create projects you need to register users"},{"Add":"in the Hydra database. In the current version, the web interface does"},{"Add":"not yet support user administration. Use the following command to add"},{"Add":"a new user to the database."},{"Add":""},{"Add":"

"},{"Add":"$ sqlite3 /usr/local/hydra/hydra.sqlite \"insert into Users(userName, emailAddress, password) values('eelco', 'blablah@example.org', '$(echo -n foobar | sha1sum | cut -c1-40)');\""},{"Add":"

"},{"Add":""},{"Add":"where eelco is the username, and foobar the"},{"Add":"password. (Make sure to use other values!)"},{"Add":""},{"Add":"

"},{"Add":""},{"Add":"To give this user administrator privileges, follow this up by:"},{"Add":""},{"Add":"

"},{"Add":"$ sqlite3 /usr/local/hydra/hydra.sqlite \"insert into UserRoles(userName, role) values('eelco', 'admin');\""},{"Add":"

"},{"Add":""},{"Add":"Now you should be able to create a project using the Hydra web interface."},{"Add":""},{"Add":"

Chapter 3. Creating Projects

"},{"Add":""},{"Add":"The next step is to add projects to the buildfarm. We follow the"},{"Add":"example of the patchelf project at hydra.nixos.org. Note that the"},{"Add":"error messages provided as feedback by the webinterface can be"},{"Add":"somewhat unfriendly in the current version."},{"Add":""},{"Add":"

"},{"Add":""},{"Add":"Login "},{"Add":"to the webinterface of your Hydra installation using "},{"Add":"the username and password you inserted in the database. "},{"Add":"Then follow the "},{"Add":"'Create Project' "},{"Add":"link to create a new project."},{"Add":""},{"Add":"

3.1. General information

"},{"Add":""},{"Add":"A project definition consists of some general information "},{"Add":""},{"Add":"The general information of a project are mainly its name and"},{"Add":"owner. Here's what we fill in for the patchelf project:"},{"Add":""},{"Add":"

"},{"Add":"  Identifier:    patchelf"},{"Add":"  Display name:  Patchelf"},{"Add":"  Description:   A tool for modifying ELF binaries"},{"Add":"  Owner:         eelco"},{"Add":"  Enabled:       Yes"},{"Add":"

"},{"Add":""},{"Add":"The Identifier should be a unique name (it is the primary"},{"Add":"database key for the project table in the database). If you try to"},{"Add":"create a project with an already existing identifier you'd get an"},{"Add":"error message such as:"},{"Add":""},{"Add":"

"},{"Add":"I'm very sorry, but an error occurred: "},{"Add":"DBIx::Class::ResultSet::create(): DBI Exception: DBD::SQLite::st execute failed: column name is not unique(19) at dbdimp.c line 402"},{"Add":"

"},{"Add":""},{"Add":"So try to create the project after entering just the general"},{"Add":"information to figure out if you have chosen a unique name. "},{"Add":"Jobsets can be added once the project has been created."},{"Add":""},{"Add":"

"},{"Add":""},{"Add":"Once create there should be an entry for the project in the sidebar. Go to"},{"Add":"the project page for the "},{"Add":"Patchelf"},{"Add":"project."},{"Add":""},{"Add":"

Jobsets

"},{"Add":""},{"Add":"A project can consist of multiple `jobsets', separate tasks that can"},{"Add":"be built separately, but may depend on each other (without cyclic"},{"Add":"dependencies, of course). Go to the "},{"Add":"Edit "},{"Add":"page of the Patchelf project and 'Add a new jobset' "},{"Add":"by providing the following 'Information':"},{"Add":""},{"Add":"

"},{"Add":"  Identifier:     trunk"},{"Add":"  Description:    Trunk"},{"Add":"  Nix expression: release.nix in input patchelfSrc"},{"Add":"

"},{"Add":""},{"Add":"This states that in order to build the 'Trunk' jobset, the Nix"},{"Add":"expression in the file 'release.nix', which can be obtained from input"},{"Add":"'patchelfSrc', should be evaluated. (We'll have a look at release.nix"},{"Add":"later.)"},{"Add":""},{"Add":"

"},{"Add":""},{"Add":"To realize a job we probably need a number of inputs, which can be"},{"Add":"declared in the table below. As many inputs as required can be added. "},{"Add":"For patchelf we declare the following inputs."},{"Add":""},{"Add":"

"},{"Add":"  nixpkgs 'CVS checkout' https://svn.nixos.org/repos/nix/nixpkgs/trunk"},{"Add":"

"},{"Add":""},{"Add":"

"},{"Add":"  officialRelease   Boolean false"},{"Add":"

"},{"Add":""},{"Add":""},{"Add":"

"},{"Add":"  patchelfSrc Subversion checkout https://svn.nixos.org/repos/nix/patchelf/trunk"},{"Add":"

"},{"Add":""},{"Add":""},{"Add":"

"},{"Add":"  system String value \"i686-linux\" "},{"Add":"

"},{"Add":""},{"Add":"nixpkgs"},{"Add":""},{"Add":"officialRelease"},{"Add":""},{"Add":"patchelfSrc"},{"Add":""},{"Add":"system "},{"Add":""},{"Add":"

Release Set

"},{"Add":""},{"Add":"there must be one primary job"},{"Add":""},{"Add":"check the radio button of exactly one job"},{"Add":""},{"Add":"https://svn.nixos.org/repos/nix/nixpkgs/trunk"},{"Add":""},{"Add":"

Building Jobs

"},{"Add":""},{"Add":""},{"Add":"

release.nix

"},{"Add":""},{"Add":""},{"Add":""},{"Add":"- Voorbeelden van Nix expressies voor Hydra:"},{"Add":""},{"Add":""},{"Add":""},{"Add":"https://svn.nixos.org/repos/nix/patchelf/trunk/release.nix"},{"Add":"https://svn.nixos.org/repos/nix/nix/trunk/release.nix"},{"Add":"https://svn.nixos.org/repos/nix/hydra/trunk/release.nix"},{"Add":""},{"Add":"

Building on the command line

"},{"Add":""},{"Add":"Overigens zijn die helemaal niet Hydra-specifiek, je kunt ze gewoon vanaf de"},{"Add":"command line bouwen, bijv. als je een patchelf checkout hebt (met een nixpkgs"},{"Add":"checkout in ../nixpkgs):"},{"Add":""},{"Add":"$ nix-build release.nix -A rpm_fedora10i386"},{"Add":""},{"Add":""},{"Add":"

Release Set

"},{"Add":""},{"Add":""},{"Add":""},{"Add":""},{"Add":""}]}},{"FileAdd":{"meta":{"basename":"style.css","metadata":0},"context":[{"change":0,"pos":18}],"contents":[{"Add":"/* Copied from http://bakefile.sourceforge.net/, which appears"},{"Add":" licensed under the GNU GPL. */"},{"Add":""},{"Add":""},{"Add":"/***************************************************************************"},{"Add":" Basic headers and text:"},{"Add":" ***************************************************************************/"},{"Add":""},{"Add":"body"},{"Add":"{"},{"Add":" font-family: sans-serif;"},{"Add":" background: white;"},{"Add":" margin: 2em 1em 2em 1em;"},{"Add":"}"},{"Add":""},{"Add":"h1,h2,h3"},{"Add":"{"},{"Add":" color: #005aa0;"},{"Add":" text-align: left;"},{"Add":"}"},{"Add":""},{"Add":"h1 /* title */"},{"Add":"{"},{"Add":" font-size: 200%;"},{"Add":"}"},{"Add":""},{"Add":"h2 /* chapters, appendices, subtitle */"},{"Add":"{"},{"Add":" font-size: 180%;"},{"Add":"}"},{"Add":""},{"Add":"/* Extra space between chapters, appendices. */"},{"Add":"div.chapter > div.titlepage h2, div.appendix > div.titlepage h2 "},{"Add":"{ "},{"Add":" margin-top: 1.5em;"},{"Add":"}"},{"Add":""},{"Add":"div.section > div.titlepage h2 /* sections */"},{"Add":"{"},{"Add":" font-size: 150%;"},{"Add":" margin-top: 1.5em;"},{"Add":"}"},{"Add":""},{"Add":"h3 /* subsections */"},{"Add":"{"},{"Add":" font-size: 125%;"},{"Add":"}"},{"Add":""},{"Add":"div.simplesect h2"},{"Add":"{"},{"Add":" font-size: 110%;"},{"Add":"}"},{"Add":""},{"Add":"div.appendix h3"},{"Add":"{"},{"Add":" font-size: 150%;"},{"Add":" margin-top: 1.5em;"},{"Add":"}"},{"Add":""},{"Add":"div.refnamediv h2, div.refsynopsisdiv h2, div.refsection h2 /* refentry parts */"},{"Add":"{"},{"Add":" margin-top: 1.4em;"},{"Add":" font-size: 125%;"},{"Add":"}"},{"Add":""},{"Add":"div.refsection h3"},{"Add":"{"},{"Add":" font-size: 110%;"},{"Add":"}"},{"Add":""},{"Add":""},{"Add":"/***************************************************************************"},{"Add":" Examples:"},{"Add":" ***************************************************************************/"},{"Add":""},{"Add":"div.example"},{"Add":"{"},{"Add":" border: 1px solid #6185a0;"},{"Add":" padding: 6px 6px;"},{"Add":" margin-left: 1.5em;"},{"Add":" margin-right: 1.5em;"},{"Add":" background: #f4f4f8;"},{"Add":"}"},{"Add":""},{"Add":"div.example p.title"},{"Add":"{"},{"Add":" margin-top: 0em;"},{"Add":"}"},{"Add":""},{"Add":""},{"Add":"/***************************************************************************"},{"Add":" Screen dumps:"},{"Add":" ***************************************************************************/"},{"Add":""},{"Add":"pre.screen, pre.programlisting"},{"Add":"{"},{"Add":" border: 1px solid #6185a0;"},{"Add":" padding: 3px 3px;"},{"Add":" margin-left: 1.5em;"},{"Add":" margin-right: 1.5em;"},{"Add":" color: #600000;"},{"Add":" background: #f4f4f8;"},{"Add":" font-family: monospace;"},{"Add":" /* font-size: 90%; */"},{"Add":"}"},{"Add":""},{"Add":"div.example pre.programlisting"},{"Add":"{"},{"Add":" border: 0px;"},{"Add":" padding: 0 0;"},{"Add":" margin: 0 0 0 0;"},{"Add":"}"},{"Add":""},{"Add":""},{"Add":"/***************************************************************************"},{"Add":" Notes, warnings etc:"},{"Add":" ***************************************************************************/"},{"Add":""},{"Add":".note, .warning"},{"Add":"{"},{"Add":" border: 1px solid #6185a0;"},{"Add":" padding: 3px 3px;"},{"Add":" margin-left: 1.5em;"},{"Add":" margin-right: 1.5em;"},{"Add":" margin-bottom: 1em;"},{"Add":" padding: 0.3em 0.3em 0.3em 0.3em;"},{"Add":" background: #fffff5;"},{"Add":"}"},{"Add":""},{"Add":"div.note, div.warning"},{"Add":"{"},{"Add":" font-style: italic;"},{"Add":"}"},{"Add":""},{"Add":"div.note h3, div.warning h3"},{"Add":"{"},{"Add":" color: red;"},{"Add":" font-size: 100%;"},{"Add":"// margin: 0 0 0 0;"},{"Add":" padding-right: 0.5em;"},{"Add":" display: inline;"},{"Add":"}"},{"Add":""},{"Add":"div.note p, div.warning p"},{"Add":"{"},{"Add":" margin-bottom: 0em;"},{"Add":"}"},{"Add":""},{"Add":"div.note h3 + p, div.warning h3 + p"},{"Add":"{"},{"Add":" display: inline;"},{"Add":"}"},{"Add":""},{"Add":"div.note h3"},{"Add":"{"},{"Add":" color: blue;"},{"Add":" font-size: 100%;"},{"Add":"}"},{"Add":""},{"Add":"div.navfooter *"},{"Add":"{"},{"Add":" font-size: 90%;"},{"Add":"}"},{"Add":""},{"Add":""},{"Add":"/***************************************************************************"},{"Add":" Links colors and highlighting: "},{"Add":" ***************************************************************************/"},{"Add":""},{"Add":"a:link { color: #0048b3; }"},{"Add":"a:visited { color: #002a6a; }"},{"Add":"a:hover { background: #ffffcd; }"},{"Add":""},{"Add":""},{"Add":"/***************************************************************************"},{"Add":" Table of contents:"},{"Add":" ***************************************************************************/"},{"Add":""},{"Add":".toc"},{"Add":"{"},{"Add":" font-size: 90%;"},{"Add":"}"},{"Add":""},{"Add":""},{"Add":""},{"Add":"/***************************************************************************"},{"Add":" Special elements:"},{"Add":" ***************************************************************************/"},{"Add":""},{"Add":"tt, code"},{"Add":"{"},{"Add":" color: #400000;"},{"Add":"}"},{"Add":""},{"Add":".term"},{"Add":"{"},{"Add":" font-weight: bold;"},{"Add":" "},{"Add":"}"},{"Add":""},{"Add":"div.variablelist dd p, div.glosslist dd p"},{"Add":"{"},{"Add":" margin-top: 0em;"},{"Add":"}"},{"Add":""},{"Add":"div.variablelist dd, div.glosslist dd"},{"Add":"{"},{"Add":" margin-left: 1.5em;"},{"Add":"}"},{"Add":""},{"Add":"div.glosslist dt"},{"Add":"{"},{"Add":" font-style: italic;"},{"Add":"}"},{"Add":""},{"Add":".default"},{"Add":"{"},{"Add":" font-style: italic;"},{"Add":"}"},{"Add":""},{"Add":".availability"},{"Add":"{"},{"Add":" font-style: italic;"},{"Add":"}"},{"Add":""},{"Add":".varname"},{"Add":"{"},{"Add":" color: #400000;"},{"Add":"}"},{"Add":""},{"Add":""},{"Add":"div.informaltable table"},{"Add":"{"},{"Add":" border: 1px solid #6185a0;"},{"Add":" width: 100%;"},{"Add":"}"},{"Add":""},{"Add":"div.informaltable td"},{"Add":"{"},{"Add":" border: 0;"},{"Add":" padding: 5px;"},{"Add":"}"},{"Add":""},{"Add":"div.informaltable td.default"},{"Add":"{"},{"Add":" text-align: right;"},{"Add":"}"},{"Add":""},{"Add":"div.informaltable th"},{"Add":"{"},{"Add":" text-align: left;"},{"Add":" color: #005aa0;"},{"Add":" border: 0;"},{"Add":" padding: 5px;"},{"Add":" background: #fffff5;"},{"Add":" font-weight: normal;"},{"Add":" font-style: italic;"},{"Add":"}"},{"Add":""},{"Add":"td.varname, td.tagname, td.paramname"},{"Add":"{"},{"Add":" font-weight: bold;"},{"Add":" vertical-align: top;"},{"Add":"}"},{"Add":""},{"Add":"div.epigraph"},{"Add":"{"},{"Add":" font-style: italic;"},{"Add":" text-align: right;"},{"Add":"}"},{"Add":""},{"Add":"table.productionset table.productionset"},{"Add":"{"},{"Add":" font-family: monospace;"},{"Add":"}"},{"Add":""},{"Add":"strong.command"},{"Add":"{"},{"Add":"// font-family: monospace;"},{"Add":"// font-style: italic;"},{"Add":"// font-weight: normal;"},{"Add":" color: #400000;"},{"Add":"}"},{"Add":""},{"Add":"div.calloutlist td"},{"Add":"{"},{"Add":" padding-bottom: 1em;"},{"Add":"}"}]}}],"owner":"grahamc","repo":"hydra2","token":"o66G0Da3gDU/9JlFhvF1KHO5Jpb4QdoYQVNhrLZbtOU=","login":null}

…

Dependencies

Change contents