add support for git as jobinput

[?]
Nov 17, 2009, 3:16 PM
JTRG7RDQXKPSO4ESGDLSVAT5WIFGKDL424MN6YYCVTKCOR2FTXRQC

Dependencies

  • [2] 5SHCWE7X * Prevent repeated evaluation of a jobset with the same inputs. This
  • [3] CLJQCY2X * Store info about all the build actions and allow them to be
  • [4] EYNG4EL4 * Regenerate the bindings from a clean sqlite database.
  • [5] L2E6EVE2 * Merged the Build and Job tables.
  • [6] 3XTHEUMP * Implemented the clone feature.
  • [7] ODNCGFQ5 * Improved the navigation bar: don't include all projects (since that
  • [8] KXGOUX7P * Creating releases.
  • [9] FHF6IZJQ * Basic release management: releases are now dynamically computed as
  • [10] IK53RV4V
  • [11] YTZOC7C5 * Editing of jobset inputs.
  • [12] YAPITGB3 * Boolean inputs.
  • [13] 3ZCEPLNO
  • [14] TLZ2SPBR
  • [15] TQKGQ5R3
  • [16] H7CNGK4O * Log evaluation errors etc. in the DB.
  • [17] 7ZSVXUGF sequence fix for postgresql
  • [18] ZEHSSVFG
  • [19] GWCV3TQV * BuildInputs table: link to dependencies, include store paths.
  • [20] S6OISBQ3 * Mark the "current" builds in a jobset, i.e. those corresponding to
  • [21] 67P45PY4
  • [22] VCOSLZRP
  • [23] CMU3YKOU * Store the release name.
  • [24] SJLEZFC4 check getHydraPath in stead of Envvar HYDRA_DBI directly
  • [25] M552HLIA * Support variant builds.
  • [26] R5D7DZPE
  • [27] 3E6IP3R3 * Add the name of the jobset to ReleaseSetJobs, otherwise we can't
  • [28] ECBA3GQO * Make the schema class names match the case of the SQL table names.
  • [29] BVOPAMLS
  • [30] BD3GRK4B * Get rid of "positive failures" and separate log phases.
  • [31] B72GLND4
  • [32] 37R34XJO * Negative caching: don't perform a build if a dependency already
  • [33] KOTB7BKV
  • [34] RU7AQO7U * Role-based access control. Only admins can create projects. Only
  • [35] GNIEG2GC * Disambiguate jobs by jobset name. I.e. jobs with the same name in
  • [36] N22GPKYT * Put info about logs / build products in the DB.
  • [37] TWVSALRL * Allow the maximum number of concurrent builds per platform to be
  • [38] WYN733ST * Store build duration, handle cached builds.
  • [39] DVNWJXWW * Generic declaration of build products.
  • [40] AHTEIK7G * Added a maintainers field to the Builds table.
  • [41] VJHIHMEH * Store the meta.longDescription and meta.license attributes in the
  • [42] BHZXGT2H * Channels: provide an index page that lists all the packages in the
  • [43] S66BOMVU * Added authentication.
  • [44] P5X4P6VK * Renaming "release sets" to "views" (not finished yet). Having
  • [45] SHBLLAVH * More global substitution.
  • [46] V4RNHJNR * Add a link to each project's homepage. Suggested by karltk.
  • [47] X27GNHDV * Basic job info in the database.
  • [48] QNDXPVCI * Store the jobset's nixExprPath and nixExprInput fields in a build to
  • [49] OOQ2D3KC * Refactoring: move fetchInput out of hydra_scheduler into a separate
  • [50] RBNQKATL * Adding persistant releases. A release is a named set of builds.
  • [51] S5PV6IIM * Represent jobs explicitly in the DB.
  • [52] 7YBYT2LQ
  • [53] PHX2HIVG * Store info about the build inputs in the build record.
  • [54] ZVTSOVHN * Support Subversion checkouts.
  • [55] 3HCBU2FA
  • [56] 4D4U5IPY * Allow jobsets to be disabled.
  • [57] 6BLUKEQ2 * Caching of "path" inputs, and fake a revision number for those.
  • [58] J5UVLXOK * Start of a basic Catalyst web interface.
  • [59] D3DIBMOK * For products that are directories (like manuals), allow a default
  • [60] LCKWLQW3 * In Sqlite "release" is now a keyword, so use "release_" instead.
  • [61] KN3VYE5P * Cleaned up the foreign key constraints.
  • [*] T4LLYESZ * Nix expression for building Hydra.
  • [*] HPIJ35SG
  • [*] DFXIN6YW * Include jquery and tablesorter in the build.
  • [*] 3HZY24CX * Make jobsets viewable under

Change contents

  • edit in release.nix at line 91
    [64.425]
    [65.531]
    cp ${"${nixpkgs}/pkgs/build-support/fetchgit/nix-prefetch-git"} $out/bin/nix-prefetch-git
  • replacement in src/lib/Hydra/Controller/Jobset.pm at line 123
    [3.1858][3.1858:1889]()
    $inputType eq "build";
    [3.1858]
    [3.1889]
    $inputType eq "git" || $inputType eq "build";
  • edit in src/lib/Hydra/Helper/AddBuilds.pm at line 65
    [3.4415][3.4415:4416]()
  • replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 66
    [3.4417][3.4417:4434]()
    sub fetchInput {
    [3.4417]
    [3.4434]
    sub fetchInputPath {
  • edit in src/lib/Hydra/Helper/AddBuilds.pm at line 69
    [3.4495][3.4495:4522]()
    if ($type eq "path") {
  • replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 127
    [3.6787][3.6787:6793]()
    }
    [3.6787]
    [3.6793]
    }
    sub fetchInputSVN {
    my ($db, $project, $jobset, $name, $type, $value) = @_;
  • edit in src/lib/Hydra/Helper/AddBuilds.pm at line 132
    [3.6794][3.6794:6823]()
    elsif ($type eq "svn") {
  • replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 181
    [3.8701][3.8701:8707]()
    }
    [3.8701]
    [3.8707]
    }
    sub fetchInputBuild {
    my ($db, $project, $jobset, $name, $type, $value) = @_;
  • edit in src/lib/Hydra/Helper/AddBuilds.pm at line 186
    [3.8708][3.8708:8739]()
    elsif ($type eq "build") {
  • edit in src/lib/Hydra/Helper/AddBuilds.pm at line 216
    [3.10008]
    [3.10008]
    }
    sub fetchInputGit {
    my ($db, $project, $jobset, $name, $type, $value) = @_;
    my $uri = $value;
    my $timestamp = time;
    my $sha256;
    my $storePath;
    # Some simple caching: don't check a path more than once every N seconds.
    (my $cachedInput) = $db->resultset('CachedGitInputs')->search(
    {uri => $uri, lastseen => {">", $timestamp - 3600}},
    {rows => 1, order_by => "lastseen DESC"});
    if (defined $cachedInput && isValidPath($cachedInput->storepath)) {
    $storePath = $cachedInput->storepath;
    $sha256 = $cachedInput->sha256hash;
    $timestamp = $cachedInput->timestamp;
    } else {
    # Then download this revision into the store.
    print STDERR "checking out Git input from $uri";
    $ENV{"NIX_HASH_ALGO"} = "sha256";
    $ENV{"PRINT_PATH"} = "1";
    my $stdout; my $stderr;
    (my $res, $stdout, $stderr) = captureStdoutStderr(
    "nix-prefetch-git", $uri);
    die "Cannot check out Git repository `$uri':\n$stderr" unless $res;
    ($sha256, $storePath) = split ' ', $stdout;
    ($cachedInput) = $db->resultset('CachedGitInputs')->search(
    {uri => $uri, sha256hash => $sha256});
    if (!defined $cachedInput) {
    txn_do($db, sub {
    $db->resultset('CachedGitInputs')->create(
    { uri => $uri
    , timestamp => $timestamp
    , lastseen => $timestamp
    , sha256hash => $sha256
    , storepath => $storePath
    });
    });
    } else {
    $timestamp = $cachedInput->timestamp;
    txn_do($db, sub {
    $cachedInput->update({lastseen => time});
    });
    }
  • edit in src/lib/Hydra/Helper/AddBuilds.pm at line 267
    [3.10014]
    [3.10014]
    return
    { type => $type
    , uri => $uri
    , storePath => $storePath
    , sha256hash => $sha256
    , revision => strftime "%Y%m%d%H%M%S", gmtime($timestamp)
    };
    }
    sub fetchInputCVS {
    my ($db, $project, $jobset, $name, $type, $value) = @_;
    }
    sub fetchInput {
    my ($db, $project, $jobset, $name, $type, $value) = @_;
    if ($type eq "path") {
    return fetchInputPath($db, $project, $jobset, $name, $type, $value);
    }
    elsif ($type eq "svn") {
    return fetchInputSVN($db, $project, $jobset, $name, $type, $value);
    }
    elsif ($type eq "build") {
    return fetchInputBuild($db, $project, $jobset, $name, $type, $value);
    }
    elsif ($type eq "git") {
    return fetchInputGit($db, $project, $jobset, $name, $type, $value);
    }
  • replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 327
    [3.1790][3.1790:1840]()
    when (["svn", "path", "build"]) {
    [3.1790]
    [3.1840]
    when (["svn", "path", "build", "git", "cvs"]) {
  • replacement in src/lib/Hydra/Schema/BuildInputs.pm at line 53
    [3.1271][3.1271:1299]()
    data_type => "integer",
    [3.1271]
    [3.1299]
    data_type => "text",
  • replacement in src/lib/Hydra/Schema/BuildInputs.pm at line 110
    [3.25][3.172:317]()
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-10-23 16:56:03
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:RTeBJCvspGfTdHUR/TxFsg
    [3.25]
    [3.781]
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-11-17 16:04:13
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:skn8HFEXTvEiL4/6Q+ulvw
  • replacement in src/lib/Hydra/Schema/BuildProducts.pm at line 97
    [3.4922][3.405:550]()
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-10-23 16:56:03
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:8FZKxfabZQfchVpHfvnVHA
    [3.4922]
    [3.5064]
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-11-17 16:04:13
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:6wJ0KJAILATio3ELRefU5Q
  • replacement in src/lib/Hydra/Schema/BuildResultInfo.pm at line 92
    [3.2382][3.638:783]()
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-10-23 16:56:03
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:496WClgIZB6hXV0zCsBjbQ
    [3.2382]
    [3.2524]
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-11-17 16:04:13
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:K+kKzTgAlKSXevJWtLIwGA
  • replacement in src/lib/Hydra/Schema/BuildSchedulingInfo.pm at line 49
    [3.4473][3.871:1016]()
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-10-23 16:56:03
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:giLh18QewdxF8Zi/5juWqA
    [3.4473]
    [3.4615]
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-11-17 16:04:13
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:eiaLNgxvTjSZGC4pMDtNWA
  • replacement in src/lib/Hydra/Schema/BuildSteps.pm at line 97
    [3.2959][3.1104:1249]()
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-10-23 16:56:03
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:5tx+fkjjKYUNYBYQS+kSOw
    [3.2959]
    [3.2959]
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-11-17 16:04:13
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:5iUdHsNyEoM3D7i+tSteBQ
  • replacement in src/lib/Hydra/Schema/Builds.pm at line 189
    [3.6194][3.344:489]()
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-10-26 14:22:27
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:qbNPA5NkENlZXILuE3OGkA
    [3.6194]
    [3.62]
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-11-17 16:04:13
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:Un0iCqVS8PTpSdJiTjRXeA
  • replacement in src/lib/Hydra/Schema/CachedPathInputs.pm at line 53
    [3.1497][3.1599:1744]()
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-10-23 16:56:03
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:vMJmHfs180MNWH9+pjS94g
    [3.1497]
    [3.1639]
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-11-17 16:04:13
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:ZEYeoP+fkE3y/cohKLoCLg
  • replacement in src/lib/Hydra/Schema/CachedSubversionInputs.pm at line 46
    [3.2342][3.1832:1977]()
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-10-23 16:56:03
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:3hG0+UX3MJMLjSTN29Le4Q
    [3.2342]
    [3.2484]
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-11-17 16:04:13
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:f3ZEnsrXJUjcpGkLNl24Rw
  • replacement in src/lib/Hydra/Schema/Jobs.pm at line 81
    [3.2613][3.2065:2210]()
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-10-23 16:56:03
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:bY5DYFb1G7JRyNFpGbWXwA
    [3.2613]
    [3.2755]
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-11-17 16:04:13
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:I5DYg9q22g99Mstln/26fw
  • replacement in src/lib/Hydra/Schema/JobsetInputAlts.pm at line 54
    [3.12118][3.12118:12146]()
    data_type => "integer",
    [3.12118]
    [3.12146]
    data_type => "text",
  • replacement in src/lib/Hydra/Schema/JobsetInputAlts.pm at line 75
    [3.1831][3.2298:2443]()
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-10-23 16:56:03
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:XPfPlym2UDd6gUr1aKrXhg
    [3.1831]
    [3.1973]
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-11-17 16:04:13
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:B2BMIuiQ3IAoqEJ18pHCeQ
  • replacement in src/lib/Hydra/Schema/JobsetInputs.pm at line 71
    [3.1681][3.2531:2676]()
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-10-23 16:56:03
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:961DCIj2fAQFYAR6/SnJ8A
    [3.1681]
    [3.1823]
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-11-17 16:04:13
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:srhHJGx+LAdeo++jv1RmMg
  • replacement in src/lib/Hydra/Schema/Jobsets.pm at line 117
    [3.2992][2.1593:1738]()
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-11-17 14:04:55
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:xWsqXneZw90uEw/vcEXc4w
    [3.2992]
    [3.3134]
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-11-17 16:05:10
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:fVXvhb343Zw1625daVz40g
  • replacement in src/lib/Hydra/Schema/Projects.pm at line 97
    [3.3685][2.1867:2012]()
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-11-17 14:04:55
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:dWe2DEsuZuOjVj4IA8TwQg
    [3.3685]
    [3.3827]
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-11-17 16:05:10
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:+HDJ8tIPcvj5+IwgHqTnaw
  • replacement in src/lib/Hydra/Schema/ReleaseMembers.pm at line 56
    [3.2936][3.3230:3375]()
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-10-23 16:56:03
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:DvQQfFaVP0ci1LMKfbl3tg
    [3.2936]
    [3.3081]
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-11-17 16:04:13
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:fu21YcmM1U76pcgFY1qZ5A
  • replacement in src/lib/Hydra/Schema/Releases.pm at line 56
    [3.4209][3.3463:3608]()
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-10-23 16:56:03
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:zG8H+WLuEnvPl9UEJ3yyCQ
    [3.4209]
    [3.4354]
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-11-17 16:04:13
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:cbt+hnRoIN5T6u8qC9uQBg
  • replacement in src/lib/Hydra/Schema/SystemTypes.pm at line 27
    [3.2105][3.3696:3841]()
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-10-23 16:56:03
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:7HmnBjVA60uMDrV2Bc9TmA
    [3.2105]
    [3.2247]
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-11-17 16:04:13
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:QtiooD8Th7dBKknK5IRhMQ
  • replacement in src/lib/Hydra/Schema/UserRoles.pm at line 34
    [3.3736][3.3929:4074]()
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-10-23 16:56:03
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:l4FwcyoQw1cpvc+QKt6W7Q
    [3.3736]
    [3.3878]
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-11-17 16:04:13
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:9oO99uYxlWqY3GxfUWjNzg
  • replacement in src/lib/Hydra/Schema/Users.pm at line 56
    [3.3782][3.4162:4307]()
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-10-23 16:56:03
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:FSfQfOaAf2oeQeFtBI7HoQ
    [3.3782]
    [3.3924]
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-11-17 16:04:13
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:oBiKpCxHGZcGujhY/ZbbxA
  • replacement in src/lib/Hydra/Schema/ViewJobs.pm at line 72
    [3.6070][3.4395:4540]()
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-10-23 16:56:03
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:FqMPRQo5hyHDy6zZIqdR5w
    [3.6070]
    [3.6212]
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-11-17 16:04:13
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:RX9tEuV8mEg13dxEe9SJrQ
  • replacement in src/lib/Hydra/Schema/Views.pm at line 48
    [3.7142][3.4628:4773]()
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-10-23 16:56:03
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:DVmiIWbS8/PBOMvjjmns6g
    [3.7142]
    [3.7284]
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-11-17 16:04:13
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:BnM3PKkdJXAaT4rPR8gJsQ
  • replacement in src/lib/Hydra/Schema.pm at line 14
    [3.6498][3.4861:5006]()
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-10-23 16:56:03
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:s5XNrT7kEVqbLED5g/J/SA
    [3.6498]
    [3.6640]
    # Created by DBIx::Class::Schema::Loader v0.04999_09 @ 2009-11-17 16:04:13
    # DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:Y97NEH6tnsRjsy0/nVgMjQ
  • edit in src/root/common.tt at line 8
    [3.2885]
    [3.2885]
    , "git" = "Git checkout"
  • replacement in src/sql/hydra.sql at line 75
    [3.6536][3.6536:6584]()
    revision integer, -- for type == 'svn'
    [3.6536]
    [3.6584]
    revision text, -- for type == 'svn'
  • replacement in src/sql/hydra.sql at line 242
    [3.2754][3.2754:2781]()
    revision integer,
    [3.2754]
    [3.2781]
    revision text,
  • edit in src/sql/hydra.sql at line 293
    [3.5623]
    [3.2956]
    );
    create table CachedGitInputs (
    uri text not null,
    timestamp integer not null, -- when we first saw this hash
    lastSeen integer not null, -- when we last saw this hash
    sha256hash text not null,
    storePath text not null,
    primary key (uri, sha256hash)
    );
    create table CachedCVSInputs (
    uri text not null,
    module text not null,
    revision integer not null,
    timestamp integer not null, -- when we first saw this hash
    lastSeen integer not null, -- when we last saw this hash
    sha256hash text not null,
    storePath text not null,
    primary key (uri, module, sha256hash)