Provide a command ‘hydra-init’ to initialise/upgrade the database

[?]

Feb 28, 2012, 7:16 PM

AMFMXR52JXHUOWWG4VZ5DHQGLF4F7H7XSJRLX5XNEXOCCWYLDQOAC

Dependencies

[2] AIEKRM6T * Add DBD::SQLite to the list of dependencies.
[3] CFZ6WDUZ
[4] TH674WKJ add log diff to compare logs to previous builds
[5] 46Z25UHS Use CatalystX::Script::Server::Starman
[6] DXBTGOA4 Print out an Apache-style access log
[7] 7DWCXNC7 Use the new Nix Perl bindings
[8] SB2V735V Keep track of the database schema version
[9] GYPHTT4M Manual: Remove tabs, indent consistently
[10] SZYDW2DG hydra: added some user admin
[11] BQYE6R7Y
[12] DO432JDJ * Build Hydra with the Nix Perl bindings.
[13] 6XJH4OFL add datetime to deps
[14] KN3VYE5P * Cleaned up the foreign key constraints.
[15] G2ZB6464 first test, not yet in buildprocess
[16] HN7JDKV3 doc: Mention SQLite.
[17] EDDTVGB4 set BZR_HOME env var so bzr won't fail
[18] BKOIYITR added some json responses
[19] FV2M6MOT hydra: use autoconf/-make
[20] J5YZVALJ Remove duplicate dependency.
[21] A6EKITA6 Update the docs to reflect the renaming
[22] 6US6LEC7 * Add a NarSize field to Hydra manifests. This allows nix-env
[23] SAFVRDTK * Put Hydra's dependencies so that they can easily be installed in a
[24] RBZRCTAL hydra: fixed and re-enabled evaluation tests
[25] EKNK5AHQ doc: Augment the "Installation" section with material by Visser & Dolstra.
[26] N22GPKYT * Put info about logs / build products in the DB.
[27] VOBFCHIX use top_srcdir in stead of srcdir
[28] 3YSK5YXG * Update to Catalyst 5.9.
[29] LZO3C2KI * Hack around those SQLite timeouts: just retry the transaction.
[30] A63IHCMX * Register GC roots properly.
[31] HSVVEKTY * Start of a JSON API to get information about a specific build.
[32] W5GW4E22 Don't install hydra-create
[33] IN272KZW * Automatically keep all builds in the latest successful release in
[34] IJWKUOG4 emailsimplecreator was removed
[35] KBW3FDZ2 Merge remote branch 'remotes/origin/master'
[36] YT25ZGSZ
[37] QUTWJR7P * Include more info in notification emails.
[38] RBGYFVWX use HYDRA_DATA/hydra.conf as default location for hydra.conf
[39] TNLAFBN7 Add missing dependency on Catalyst::Plugin::Session::State::Cookie.
[40] OG7BEM57
[41] CXRCPDSQ * added support for twitter notification
[42] YFPZ46YK * hydra: added variant of build input type, 'build output (same system)' to allow better continous integration in one jobset for multiple system. it makes sure that the system of the build that is passed as input for a job has the same system as the job.
[*] 2GK5DOU7 * Downloading closures.
[*] D5QIOJGP * Move everything up one directory.

Change contents

replacement in deps.nix at line 1

[10.1]→[10.2:10](∅→∅)

{pkgs}:

[10.1]

[10.10]

{ pkgs }:
replacement in deps.nix at line 5

[10.326]→[10.23:52](∅→∅),[10.23]→[10.23:52](∅→∅)

[ perlPackages.CatalystDevel

[10.23]

[10.52]

[ perlPackages.CatalystAuthenticationStoreDBIxClass
perlPackages.CatalystPluginAccessLog
perlPackages.CatalystPluginAuthorizationRoles
perlPackages.CatalystPluginSessionStateCookie
edit in deps.nix at line 11

[10.142]→[10.142:190](∅→∅),[10.190]→[10.0:48](∅→∅),[10.48]→[6.0:39](∅→∅),[6.39]→[10.190:272](∅→∅),[10.48]→[10.190:272](∅→∅),[10.190]→[10.190:272](∅→∅)

perlPackages.CatalystPluginAuthorizationRoles
perlPackages.CatalystPluginSessionStateCookie
perlPackages.CatalystPluginAccessLog
perlPackages.CatalystAuthenticationStoreDBIxClass
perlPackages.CatalystViewTT
edit in deps.nix at line 13

[10.32]

[5.0]

perlPackages.CatalystViewTT
replacement in deps.nix at line 15

[10.32]→[10.349:396](∅→∅),[5.44]→[10.349:396](∅→∅),[10.349]→[10.349:396](∅→∅),[10.396]→[3.0:26](∅→∅),[3.26]→[10.427:451](∅→∅),[10.427]→[10.427:451](∅→∅)

perlPackages.XMLSimple
perlPackages.IPCRun
perlPackages.IOCompress
perlPackages.Readonly

[5.44]

[10.451]

perlPackages.CryptRandPasswd
edit in deps.nix at line 18

[2.25]

[10.0]

perlPackages.DataDump
perlPackages.DateTime
perlPackages.DigestSHA1
edit in deps.nix at line 22

[10.61]→[10.0:25](∅→∅),[10.25]→[4.0:24](∅→∅)

perlPackages.TextTable
perlPackages.TextDiff
edit in deps.nix at line 23

[4.49]

[10.0]

perlPackages.IOCompress
perlPackages.IPCRun
perlPackages.JSONXS
replacement in deps.nix at line 28

[10.25]→[10.25:49](∅→∅),[10.49]→[10.0:22](∅→∅),[10.22]→[10.0:24](∅→∅),[10.24]→[10.0:57](∅→∅),[10.57]→[10.47:71](∅→∅),[10.71]→[10.49:80](∅→∅)

perlPackages.DataDump
perlPackages.JSONXS
perlPackages.DateTime
perlPackages.DigestSHA1
perlPackages.CryptRandPasswd
perlPackages.TestMore
perlPackages.SysHostnameLong

[10.25]

[10.0]

perlPackages.CatalystDevel
perlPackages.Readonly
perlPackages.SQLSplitStatement
edit in deps.nix at line 32

[10.23]

[7.0]

perlPackages.SysHostnameLong
perlPackages.TestMore
perlPackages.TextDiff
perlPackages.TextTable
perlPackages.XMLSimple
edit in doc/manual/installation.xml at line 76

[10.2349]

[9.2697]

<!--
replacement in doc/manual/installation.xml at line 81

[9.2768]→[9.2768:2815](∅→∅)

nix-env -Ai hydra -f /path/to/nixpkgs</screen>

[9.2768]

[10.2463]

nix-env -f /path/to/nixpkgs -iA hydra</screen>
edit in doc/manual/installation.xml at line 86

[9.2951]

[10.2590]

-->
replacement in doc/manual/installation.xml at line 89

[9.2963]→[9.2963:3033](∅→∅)

Alternatively, the latest development snapshot can be installed

[9.2963]

[9.3033]

The latest development snapshot of Hydra can be installed
replacement in doc/manual/installation.xml at line 92

[9.3196]→[9.3196:3262](∅→∅)

and use the one-click install available at one of the build

[9.3196]

[9.3262]

and using the one-click install available at one of the build
replacement in doc/manual/installation.xml at line 106

[9.3493]→[10.0:118](∅→∅),[10.3048]→[10.0:118](∅→∅)

hydra-build hydra-evaluator hydra-server
hydra-eval-jobs hydra-queue-runner hydra-update-gc-roots</screen>

[9.3493]

[9.3494]

hydra-build hydra-init hydra-update-gc-roots
hydra-eval-jobs hydra-queue-runner
hydra-evaluator hydra-server
</screen>
replacement in doc/manual/installation.xml at line 123

[9.3843]→[9.3843:3911](∅→∅)

as database name and user name, issue the following commands:

[9.3843]

[10.232]

as database name and user name, issue the following commands on
the PostgreSQL server:
replacement in doc/manual/installation.xml at line 127

[9.3927]→[10.1698:1920](∅→∅),[10.1698]→[10.1698:1920](∅→∅)

createdb hydra
echo "CREATE USER hydra WITH PASSWORD '<your-password>' ;" | psql hydra
cat $prefix/share/hydra/sql/hydra-postgresql.sql | psql hydra
echo "GRANT ALL ON DATABASE hydra TO hydra;" | psql hydra</screen>

[9.3927]

[9.3928]

createuser -S -D -R -P hydra
createdb -O hydra hydra</screen>
replacement in doc/manual/installation.xml at line 135

[9.4047]→[9.4047:4133](∅→∅)

For SQLite, the following command is all it takes to create the
database:

[9.4047]

[10.591]

Hydra uses an environment variable to know which database should
be used, and a variable which point to a location that holds
some state. To set these variables for a PostgreSQL database,
add the following to the file <filename>~/.profile</filename> of
the user running the Hydra services.
replacement in doc/manual/installation.xml at line 142

[9.4149]→[9.4149:4235](∅→∅)

cat $prefix/share/hydra/sql/hydra-sqlite.sql | sqlite3 /path/to/hydra.sqlite</screen>

[9.4149]

[9.4235]

export HYDRA_DBI="dbi:Pg:dbname=hydra;host=dbserver.example.org;user=hydra;"
export HYDRA_DATA=/var/lib/hydra</screen>

You can provide the username and password in the file
<filename>~/.pgpass</filename>, e.g.

<screen>
dbserver.example.org:*:hydra:hydra:password</screen>

Make sure that the <emphasis>HYDRA_DATA</emphasis> directory
exists and is writable for the user which will run the Hydra
services. For a SQLite database, the
<varname>HYDRA_DBI</varname> should be set to something like
<literal>dbi:SQLite:/path/to/hydra.sqlite</literal>
edit in doc/manual/installation.xml at line 159

[9.4259]

[9.4259]

Having set these environment variables, you can now initialise
the database by doing:
<screen>
hydra-init</screen>
</para>

<para>
replacement in doc/manual/installation.xml at line 177

[10.2434]→[9.4643:4967](∅→∅)

<para>
Hydra uses an environment variable to know which database should
be used, and a variable which point to a location that holds
some state. To set these variables for a PostgreSQL database,
add the following to the <filename>.profile</filename> of the
user running the Hydra services.

[10.2434]

[10.1333]

</section>

<section>
<title>Upgrading</title>
edit in doc/manual/installation.xml at line 182

[10.1334]

[9.4968]

<para>If you're upgrading Hydra from a previous version, you
should do the following to perform any necessary database schema migrations:
replacement in doc/manual/installation.xml at line 185

[9.4983]→[10.2737:2835](∅→∅),[10.2737]→[10.2737:2835](∅→∅)

export HYDRA_DBI="dbi:Pg:dbname=hydra;host=localhost;"
export HYDRA_DATA=/var/lib/hydra</screen>

[9.4983]

[10.2835]

hydra-init</screen>
</para>
edit in doc/manual/installation.xml at line 188

[10.2836]→[9.4984:5299](∅→∅)

Make sure that the <emphasis>HYDRA_DATA</emphasis> directory
exists and is writable for the user which will run the Hydra
services. For a SQLite database, the
<varname>HYDRA_DBI</varname> should be set to something like
<literal>dbi:SQLite:/path/to/hydra.sqlite</literal>
</para>
replacement in src/lib/Hydra/Helper/Nix.pm at line 11

[10.107]→[10.0:64](∅→∅)

getHydraPath getHydraDBPath openHydraDB getHydraConf txn_do

[10.619]

[10.665]

getHydraPath getHydraHome getHydraDBPath openHydraDB getHydraConf txn_do
edit in src/lib/Hydra/Helper/Nix.pm at line 24

[10.145]

[10.145]

sub getHydraHome {
my $dir = $ENV{"HYDRA_HOME"} or die "The HYDRA_HOME directory does not exist!\n";
return $dir;
}
file addition: hydra-init (---r------)

[45.2543]

#! /var/run/current-system/sw/bin/perl -w

use strict;
use Hydra::Schema;
use Hydra::Helper::Nix;
use File::Slurp;
use SQL::SplitStatement;
use List::Util qw(max);

my $db = openHydraDB;
my $dbh = $db->storage->dbh;
$dbh->{RaiseError} = 1;

my $home = getHydraHome;

my $sql_splitter = SQL::SplitStatement->new;

# Figure out the target schema version.
my $maxSchemaVersion = max (map { /.*\/upgrade-(\d.*)\.sql/; $1 } (glob "$home/sql/upgrade-[0-9]*.sql")) || 1;

# Check whether the database has been initialised. If not, load the
# schema.
my @tables = $dbh->tables;
if (! grep { /SchemaVersion/i } @tables) {
print STDERR "initialising the Hydra database schema...\n";
my $schema = read_file(
$dbh->{Driver}->{Name} eq 'SQLite' ? "$home/sql/hydra-sqlite.sql" :
$dbh->{Driver}->{Name} eq 'Pg' ? "$home/sql/hydra-postgresql.sql" :
die "unsupported database type\n");
my @statements = $sql_splitter->split($schema);
eval {
$dbh->begin_work;
$dbh->do($_) foreach @statements;
$db->resultset('SchemaVersion')->create({version => $maxSchemaVersion});
$dbh->commit;
};
die "schema initialisation failed: $@\n" if $@;
exit 0;
}

# Get the current schema version.
my @versions = $db->resultset('SchemaVersion')->all;
die "couldn't get Hydra schema version!" if scalar @versions != 1;
my $schemaVersion = $versions[0]->version;

for (my $n = $schemaVersion; $n < $maxSchemaVersion; $n++) {
my $m = $n + 1;
print STDERR "upgrading Hydra schema from version $n to $m\n";
my $schema = read_file("$home/sql/upgrade-$m.sql");
my @statements = $sql_splitter->split($schema);
eval {
$dbh->begin_work;
$dbh->do($_) foreach @statements;
$db->resultset('SchemaVersion')->update({version => $m});
$dbh->commit;
};
die "schema upgrade failed: $@\n" if $@;
}
replacement in src/sql/Makefile.am at line 3

[10.2199458]→[10.2199458:2199488](∅→∅)

sqldir = $(datadir)/hydra/sql

[10.2199458]

[10.2199488]

sqldir = $(libexecdir)/hydra/sql
edit in src/sql/hydra.sql at line 5

[8.32788]→[8.32788:32837](∅→∅)

insert into SchemaVersion (version) values (1);
replacement in tests/Makefile.am at line 34

[10.51]→[10.51:112](∅→∅)

sqlite3 db.sqlite < $(top_srcdir)/src/sql/hydra-sqlite.sql

[10.51]

[10.0]

perl $(top_srcdir)/src/script/hydra-init