Support revision control systems via plugins
[?]
May 25, 2013, 7:36 PM
JAH3UPWAVSHXIPNGL6PROQPZBYZHPJNFONWBDZX4HCX646USZXUQCDependencies
- [2]
X3COYEVIAdd missing / - [3]
YRKCK7UBFix Mercurial input fetching if a cached input has been GC'ed - [4]
A43SLRSHFix handling of IPC::Run::run exit status - [5]
6ZB4CIW6Security: Improve checking of build products - [6]
BLVQGJ4LUse OO-style plugins - [7]
NILMMFMYRespect X-Request-Base header coming from a frontend proxy. - [8]
TJK27WSBOpen the DB using Hydra::Model::DB->new - [9]
VJ4BZV5RAdd branch to local clone path of git repositories for now. - [10]
ODNCGFQ5* Improved the navigation bar: don't include all projects (since that - [11]
2WUNXJGWHydra/26: Go back to using "svn export" as default for svn, added svn-checkout type for jobset which need .svn dirs. export is much more efficient - [12]
FXW2UR7Finitial bzr support - [13]
R4MHON3Opass svn/bzr revisions as integers - [14]
HQ4QHAOFmake default branch default, remove quotes - [15]
OXJ4QT6KStrip all whitespace from revision from subversion call. Patch provided by Petr Ročkai - [16]
QMW24O5SAdd support for Guile & Guix. - [17]
HZ2U4QEVno branch info available yet in scmdiff, so disable for now - [18]
3I6BNF4Shydra: do not use refs/heads while checking for latest commit in branch - [19]
Y5DJHUBMAllow Subversion inputs to specify a revision number. - [20]
C7CXMZ66hydra: do not perform git clone every time. in stead work on local clone and pull - [21]
BAOVFU73fix mercurial diffs, should change to cloned repo first - [22]
NFVN7JRBHandle missing "build" inputs - [23]
HPEG2RHVMerge the BuildResultInfo table into the Builds table - [24]
3XTHEUMP* Implemented the clone feature. - [25]
G7XYM6MUuse local clone to limit bandwidth usage - [26]
WVX47J4Eproperly pass bazaar inputs - [27]
6ZSLRO7MfetchInputGit: Populate TopGit branches when on a topic branch. - [28]
IK53RV4V - [29]
TH674WKJadd log diff to compare logs to previous builds - [30]
2WRTOU2ZCleanup - [31]
WWUOQ7V4* hydra: indentation and fixed duplicate key in cachedgitinputs bug - [32]
SGNXIOI4Hydra/32: Add option to force evaluation of a certain jobset via web interface (for admins only) - [33]
KJ5JMWXYIn case of deep clone, make sure there is local branch. - [34]
7ZHHVD6Q* Inputs of type "build" must now be declared explicitly. - [35]
HABC6L4GRevert "Use git fetch --all to update local clone in stead of git pull. Also, do not use --branch in initial clone." - [36]
RXVJFQ5AEvaluator cleanups - [37]
5SHCWE7X* Prevent repeated evaluation of a jobset with the same inputs. This - [38]
OSVLMLCQhydra: factored out build restart and - [39]
VOBOLKL4missing case for hg input type - [40]
VJFLX7B6Fix rendering of jobset inputs - [41]
4WZQW2N6Fix indentation and get rid of some unnecessary whitespace in the output - [42]
UO67NQMRChange `fetchInputGit' to do `git clone -b BRANCH' instead of `git clone'. - [43]
4FWDVNWAPass additional attributes for Git inputs - [44]
POPU2ATH* hydra_scheduler: use eval-jobs. - [45]
O25D52TAinitial support for mercurial - [46]
VH5ZABDRAdd a page to show the latest evaluations for the entire server - [47]
RFE6T5LG* Store jobset evaluations in the database explicitly. This includes - [48]
VXY434VY - [49]
A22P7HCOhydra: at evaluation, check if path is already built, and mark as built in stead of adding to the queue. - [50]
OOQ2D3KC* Refactoring: move fetchInput out of hydra_scheduler into a separate - [51]
2M7J26V4inital version of links to diff in scm - [52]
JZE7DC2FWhitespace - [53]
VLS2QTLMAdd numbered revision id of mercurial as revCount. - [54]
OSTNUSGYFallback for git fetch, which allows specific gits revisions to be used as build inputs. - [55]
JTRG7RDQadd support for git as jobinput - [56]
CHQEG6WYHydra/29: Added timeout to svn/git checkouts, job evaluation - [57]
M73H45NEadd support for mercurial tags/revs - [58]
YFPZ46YK* hydra: added variant of build input type, 'build output (same system)' to allow better continous integration in one jobset for multiple system. it makes sure that the system of the build that is passed as input for a job has the same system as the job. - [59]
KNJTTZ3ZPass the abbreviated Git revision - [60]
7XC62UTYUse git fetch -fu origin +$branch:$branch to fetch all changes for branch. - [61]
EDRUQ4UKDie TABs die - [62]
WQXF2T3Dhydra-evaluator: Don't require $HYDRA_CONFIG - [63]
ODWGRX32check for revisions - [64]
E5DMQRPORemove unnecessary whitespace in the HTML output - [65]
INNOEHO6* Fix getBuildLog for bzip2'd files. - [66]
DAKY4N64Allow use of a single Git clone for several branches. - [67]
TM6XBAG2git branches can share local clones - [68]
ENYUXIFPsupport git branches as input in stead of only master branch - [69]
PNPT3Y7RUse git fetch --all to update local clone in stead of git pull. Also, do not use --branch in initial clone. - [70]
BK24VA6Q* for git inputs, check latest revision of branch (defaults to master for now), if there is change, only use input if last checkout was > hour ago. - [71]
JLDUSNUO* Unify rendering of finished and scheduled builds. - [72]
HE3GX5IPOptimize fetch-git. - [73]
V2MLUZRPRandom indentation fix - [74]
BDSD2JLV* Speed up manifest generation. - [75]
CQTN62OHDie tabs die - [76]
QBQSQOSYhydra: moved getbuildlog - [77]
JI26P3PChydra: export addBuildProducts - [78]
BMSQD2ZHIndentation - [79]
Y6AHH4THRemove the logfile and logSize columns from the database - [80]
KQS7DSKJ* Clean up indentation. - [81]
6KIJX24RGet rid of unnecessary [%- and -%] tags - [82]
ARD6Z67TDo incremental SVN checkouts - [83]
P5XCKTFDFix sysbuild input type handling - [84]
IEXUBVNBallow dots (.) in job names when used as build input - [85]
OX6NYJDVSplit viewing and editing a jobset - [86]
SHYRGAWZhydra: when no external url is given for diff's of git/hg input, revert to a diff done on a local available clone - [87]
5DZPBNDE* hydra: on git ls-remote add refs/heads/ prefix to branchname to avoid other branches called master (or $branchname) - [88]
H7CNGK4O* Log evaluation errors etc. in the DB. - [*]
BKOIYITRadded some json responses - [*]
LBNVQXUB* Build the /build stuff in a separate controller. - [*]
J5UVLXOK* Start of a basic Catalyst web interface. - [*]
EFWN7JBV* Added a status page that shows all the currently executing build steps. - [*]
2GK5DOU7* Downloading closures. - [*]
A63IHCMX* Register GC roots properly. - [*]
LZO3C2KI* Hack around those SQLite timeouts: just retry the transaction. - [*]
PMNWRTGJAdd multiple output support - [*]
AFTXA575* $HYDRA_DATA environment variable. - [*]
5EQYVRWEAdd a plugin mechanism - [*]
LKDWWJLMDoh - [*]
N22GPKYT* Put info about logs / build products in the DB. - [*]
JK2QWPH6 - [*]
OVR2RWBIhydra-evaluator: Always pick the jobset that hasn't been evaluated longest
Change contents
- edit in src/lib/Hydra/Controller/API.pm at line 221
# FIXME: injection danger. - replacement in src/lib/Hydra/Controller/API.pm at line 226
my $clonePath = scmPath . "/" . sha256_hex($uri);my $clonePath = getSCMCacheDir . "/hg/" . sha256_hex($uri); - replacement in src/lib/Hydra/Controller/API.pm at line 232
my $clonePath = scmPath . "/" . sha256_hex($uri);my $clonePath = getSCMCacheDir . "/git/" . sha256_hex($uri); - replacement in src/lib/Hydra/Controller/Build.pm at line 536
$c->model('DB'), $build->project, $build->jobset,$c->hydra_plugins, $c->model('DB'), $build->project, $build->jobset, - edit in src/lib/Hydra/Controller/Root.pm at line 31
# Gather the supported input types.$c->stash->{inputTypes} = {'string' => 'String value','boolean' => 'Boolean','build' => 'Build output','sysbuild' => 'Build output (same system)'};$_->supportedInputTypes($c->stash->{inputTypes}) foreach @{$c->hydra_plugins}; - edit in src/lib/Hydra/Helper/AddBuilds.pm at line 6
use POSIX qw(strftime); - replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 21
fetchInput evalJobs checkBuild inputsToArgs captureStdoutStderrgetReleaseName addBuildProducts restartBuild scmPathfetchInput evalJobs checkBuild inputsToArgsgetReleaseName addBuildProducts restartBuild - edit in src/lib/Hydra/Helper/AddBuilds.pm at line 25[8.63]→[8.2815:2816](∅→∅),[8.98]→[8.2815:2816](∅→∅),[8.110]→[8.2815:2816](∅→∅),[8.127]→[8.2815:2816](∅→∅),[8.159]→[8.2815:2816](∅→∅),[8.307]→[8.2815:2816](∅→∅),[8.774]→[8.2815:2816](∅→∅),[8.1152]→[8.2815:2816](∅→∅),[8.1355]→[8.2815:2816](∅→∅),[8.2815]→[8.2815:2816](∅→∅),[8.2816]→[8.160:161](∅→∅),[8.161]→[8.32:46](∅→∅),[8.2816]→[8.32:46](∅→∅),[8.46]→[8.23:76](∅→∅),[8.76]→[8.111:115](∅→∅),[8.81]→[8.111:115](∅→∅),[8.266]→[8.2817:3151](∅→∅),[8.2817]→[8.2817:3151](∅→∅),[8.3151]→[8.191:209](∅→∅),[8.209]→[8.3173:3175](∅→∅),[8.3173]→[8.3173:3175](∅→∅)
sub scmPath {return Hydra::Model::DB::getHydraPath . "/scm" ;}sub getStorePathHash {my ($storePath) = @_;my $hash = `nix-store --query --hash $storePath`or die "cannot get hash of $storePath";chomp $hash;die unless $hash =~ /^sha256:(.*)$/;$hash = $1;$hash = `nix-hash --to-base16 --type sha256 $hash`or die "cannot convert hash";chomp $hash;return $hash;} - edit in src/lib/Hydra/Helper/AddBuilds.pm at line 72[8.4413]→[8.4413:4415](∅→∅),[8.4415]→[8.0:1](∅→∅),[8.1]→[8.4416:4417](∅→∅),[8.4416]→[8.4416:4417](∅→∅),[8.4417]→[8.156:177](∅→∅),[8.177]→[8.0:53](∅→∅),[8.53]→[8.4494:4495](∅→∅),[8.4494]→[8.4494:4495](∅→∅),[8.4495]→[8.2:24](∅→∅),[8.24]→[8.4548:4549](∅→∅),[8.4548]→[8.4548:4549](∅→∅),[8.4549]→[8.25:86](∅→∅),[8.86]→[8.4622:4623](∅→∅),[8.4622]→[8.4622:4623](∅→∅),[8.4623]→[8.87:347](∅→∅),[8.347]→[8.4899:4900](∅→∅),[8.4899]→[8.4899:4900](∅→∅),[8.4900]→[8.348:569](∅→∅),[8.569]→[8.5141:5142](∅→∅),[8.5141]→[8.5141:5142](∅→∅),[8.5142]→[8.570:678](∅→∅),[8.678]→[4.76:140](∅→∅),[4.140]→[8.742:768](∅→∅),[8.742]→[8.742:768](∅→∅),[8.768]→[8.5356:5357](∅→∅),[8.5356]→[8.5356:5357](∅→∅),[8.5357]→[8.769:816](∅→∅),[8.816]→[8.5408:5409](∅→∅),[8.5408]→[8.5408:5409](∅→∅),[8.5409]→[8.817:941](∅→∅),[8.941]→[8.5541:5542](∅→∅),[8.5541]→[8.5541:5542](∅→∅),[8.5542]→[8.942:1350](∅→∅),[8.1350]→[3.0:70](∅→∅),[3.70]→[8.1410:1653](∅→∅),[8.1410]→[8.1410:1653](∅→∅),[8.1653]→[8.6323:6343](∅→∅),[8.6323]→[8.6323:6343](∅→∅),[8.6343]→[8.1654:1825](∅→∅),[8.1825]→[8.6548:6558](∅→∅),[8.6548]→[8.6548:6558](∅→∅),[8.6558]→[8.1826:1832](∅→∅),[8.1832]→[8.6558:6559](∅→∅),[8.6558]→[8.6558:6559](∅→∅),[8.6559]→[8.1833:1844](∅→∅),[8.1844]→[8.54:76](∅→∅),[8.76]→[8.1890:2033](∅→∅),[8.1890]→[8.1890:2033](∅→∅),[8.2033]→[8.178:181](∅→∅),[8.6787]→[8.178:181](∅→∅),[8.181]→[8.2034:2035](∅→∅),[8.2035]→[8.181:201](∅→∅),[8.181]→[8.181:201](∅→∅),[8.201]→[8.77:141](∅→∅),[8.141]→[8.6793:6794](∅→∅),[8.185]→[8.6793:6794](∅→∅),[8.261]→[8.6793:6794](∅→∅),[8.6793]→[8.6793:6794](∅→∅),[8.6794]→[8.0:110](∅→∅),[8.110]→[8.6849:6850](∅→∅),[8.2058]→[8.6849:6850](∅→∅),[8.6849]→[8.6849:6850](∅→∅),[8.6850]→[8.2059:2094](∅→∅),[8.2274]→[8.2274:2302](∅→∅),[8.2302]→[8.111:145](∅→∅),[8.145]→[8.26:217](∅→∅),[8.217]→[4.141:281](∅→∅),[4.281]→[8.0:66](∅→∅),[8.361]→[8.0:66](∅→∅),[8.66]→[8.477:484](∅→∅),[8.407]→[8.477:484](∅→∅),[8.477]→[8.477:484](∅→∅),[8.484]→[8.2483:2520](∅→∅),[8.2483]→[8.2483:2520](∅→∅),[8.2520]→[8.7351:7352](∅→∅),[8.7351]→[8.7351:7352](∅→∅),[8.7352]→[8.17:190](∅→∅),[8.190]→[8.2521:2642](∅→∅),[8.7352]→[8.2521:2642](∅→∅),[8.2642]→[8.7481:7482](∅→∅),[8.7481]→[8.7481:7482](∅→∅),[8.7482]→[8.2643:2818](∅→∅),[8.2818]→[8.191:192](∅→∅),[8.192]→[8.408:515](∅→∅),[8.515]→[2.0:77](∅→∅),[2.77]→[8.381:493](∅→∅),[8.625]→[8.381:493](∅→∅),[8.381]→[8.381:493](∅→∅),[8.493]→[4.282:475](∅→∅),[4.475]→[8.560:561](∅→∅),[8.560]→[8.560:561](∅→∅),[8.561]→[8.626:1162](∅→∅),[8.1034]→[8.8133:8134](∅→∅),[8.1162]→[8.8133:8134](∅→∅),[8.3238]→[8.8133:8134](∅→∅),[8.8133]→[8.8133:8134](∅→∅),[8.8134]→[8.1035:1104](∅→∅),[8.1104]→[8.8190:8191](∅→∅),[8.3291]→[8.8190:8191](∅→∅),[8.8190]→[8.8190:8191](∅→∅),[8.8191]→[8.3292:3318](∅→∅),[8.3318]→[8.1105:1177](∅→∅),[8.1177]→[8.3380:3552](∅→∅),[8.3380]→[8.3380:3552](∅→∅),[8.3552]→[8.8479:8495](∅→∅),[8.8479]→[8.8479:8495](∅→∅),[8.8495]→[8.3553:3559](∅→∅),[8.3559]→[8.8505:8506](∅→∅),[8.8505]→[8.8505:8506](∅→∅),[8.8506]→[8.210:221](∅→∅),[8.221]→[8.142:164](∅→∅),[8.3572]→[8.142:164](∅→∅),[8.164]→[8.3618:3727](∅→∅),[8.3618]→[8.3618:3727](∅→∅)
}sub fetchInputPath {my ($db, $project, $jobset, $name, $value) = @_;my $uri = $value;my $timestamp = time;my $sha256;my $storePath;# Some simple caching: don't check a path more than once every N seconds.(my $cachedInput) = $db->resultset('CachedPathInputs')->search({srcpath => $uri, lastseen => {">", $timestamp - 30}},{rows => 1, order_by => "lastseen DESC"});if (defined $cachedInput && isValidPath($cachedInput->storepath)) {$storePath = $cachedInput->storepath;$sha256 = $cachedInput->sha256hash;$timestamp = $cachedInput->timestamp;} else {print STDERR "copying input ", $name, " from $uri\n";$storePath = `nix-store --add "$uri"`or die "cannot copy path $uri to the Nix store.\n";chomp $storePath;$sha256 = getStorePathHash $storePath;($cachedInput) = $db->resultset('CachedPathInputs')->search({srcpath => $uri, sha256hash => $sha256});# Path inputs don't have a natural notion of a "revision", so# we simulate it by using the timestamp that we first saw this# path have this SHA-256 hash. So if the contents of the path# changes, we get a new "revision", but if it doesn't change# (or changes back), we don't get a new "revision".if (!defined $cachedInput) {txn_do($db, sub {$db->resultset('CachedPathInputs')->update_or_create({ srcpath => $uri, timestamp => $timestamp, lastseen => $timestamp, sha256hash => $sha256, storepath => $storePath});});} else {$timestamp = $cachedInput->timestamp;txn_do($db, sub {$cachedInput->update({lastseen => time});});}}return{ uri => $uri, storePath => $storePath, sha256hash => $sha256, revision => strftime "%Y%m%d%H%M%S", gmtime($timestamp)};}sub fetchInputSVN {my ($db, $project, $jobset, $name, $value, $checkout) = @_;# Allow users to specify a revision number next to the URI.my ($uri, $revision) = split ' ', $value;my $sha256;my $storePath;my $stdout; my $stderr;unless (defined $revision) {# First figure out the last-modified revision of the URI.my @cmd = (["svn", "ls", "-v", "--depth", "empty", $uri],"|", ["sed", 's/^ *\([0-9]*\).*/\1/']);IPC::Run::run(@cmd, \$stdout, \$stderr);die "cannot get head revision of Subversion repository at `$uri':\n$stderr" if $?;$revision = $stdout; $revision =~ s/\s*([0-9]+)\s*/$1/sm;}die unless $revision =~ /^\d+$/;# Do we already have this revision in the store?# !!! This needs to take $checkout into account! Otherwise "svn"# and "svn-checkout" inputs can get mixed up.(my $cachedInput) = $db->resultset('CachedSubversionInputs')->search({uri => $uri, revision => $revision});if (defined $cachedInput && isValidPath($cachedInput->storepath)) {$storePath = $cachedInput->storepath;$sha256 = $cachedInput->sha256hash;} else {# No, do a checkout. The working copy is reused between# invocations to speed things up.my $wcPath = scmPath . "/svn/" . sha256_hex($uri) . "/svn-checkout";print STDERR "checking out Subversion input ", $name, " from $uri revision $revision into $wcPath\n";(my $res, $stdout, $stderr) = captureStdoutStderr(600, "svn", "checkout", $uri, "-r", $revision, $wcPath);die "error checking out Subversion repo at `$uri':\n$stderr" if $res;if ($checkout) {$storePath = addToStore($wcPath, 1, "sha256");} else {# Hm, if the Nix Perl bindings supported filters in# addToStore(), then we wouldn't need to make a copy here.my $tmpDir = File::Temp->newdir("hydra-svn-export.XXXXXX", CLEANUP => 1, TMPDIR => 1) or die;(system "svn", "export", $wcPath, "$tmpDir/svn-export", "--quiet") == 0 or die "svn export failed";$storePath = addToStore("$tmpDir/svn-export", 1, "sha256");}$sha256 = queryPathHash($storePath); $sha256 =~ s/sha256://;txn_do($db, sub {$db->resultset('CachedSubversionInputs')->update_or_create({ uri => $uri, revision => $revision, sha256hash => $sha256, storepath => $storePath});});}return{ uri => $uri, storePath => $storePath, sha256hash => $sha256, revision => $revision}; - replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 127
return undef;return (); - edit in src/lib/Hydra/Helper/AddBuilds.pm at line 150[8.6139]→[8.870:871](∅→∅),[8.870]→[8.870:871](∅→∅),[8.350]→[8.350:371](∅→∅),[8.371]→[8.365:418](∅→∅),[8.418]→[8.431:432](∅→∅),[8.431]→[8.431:432](∅→∅),[8.432]→[8.0:114](∅→∅),[8.114]→[8.100:101](∅→∅),[8.100]→[8.100:101](∅→∅),[8.101]→[8.454:515](∅→∅),[8.454]→[8.454:515](∅→∅),[8.515]→[8.0:1](∅→∅),[8.103]→[8.103:124](∅→∅),[8.124]→[8.0:54](∅→∅),[8.54]→[8.179:180](∅→∅),[8.61]→[8.179:180](∅→∅),[8.62]→[8.179:180](∅→∅),[8.110]→[8.179:180](∅→∅),[8.1232]→[8.179:180](∅→∅),[8.179]→[8.179:180](∅→∅),[8.180]→[8.115:162](∅→∅),[8.162]→[8.181:208](∅→∅),[8.120]→[8.181:208](∅→∅),[8.208]→[8.163:318](∅→∅),[8.318]→[4.476:658](∅→∅),[4.658]→[8.390:397](∅→∅),[8.390]→[8.390:397](∅→∅),[8.424]→[8.162:227](∅→∅),[8.227]→[8.379:380](∅→∅),[8.380]→[8.0:264](∅→∅),[8.264]→[4.659:721](∅→∅),[4.721]→[8.0:56](∅→∅),[8.328]→[8.0:56](∅→∅),[8.56]→[4.722:854](∅→∅),[8.414]→[8.563:564](∅→∅),[4.854]→[8.563:564](∅→∅),[8.1289]→[8.563:564](∅→∅),[8.563]→[8.563:564](∅→∅),[8.564]→[8.1290:1388](∅→∅),[8.1388]→[4.855:948](∅→∅),[8.151]→[8.515:516](∅→∅),[8.308]→[8.515:516](∅→∅),[8.783]→[8.515:516](∅→∅),[4.948]→[8.515:516](∅→∅),[8.1485]→[8.515:516](∅→∅),[8.515]→[8.515:516](∅→∅),[8.516]→[8.1486:1528](∅→∅),[8.1528]→[4.949:1095](∅→∅),[4.1095]→[8.405:406](∅→∅),[8.1642]→[8.405:406](∅→∅),[8.405]→[8.405:406](∅→∅),[8.406]→[8.1643:1890](∅→∅),[8.1890]→[4.1096:1270](∅→∅),[4.1270]→[8.237:238](∅→∅),[8.1403]→[8.237:238](∅→∅),[8.2082]→[8.237:238](∅→∅),[8.237]→[8.237:238](∅→∅),[8.238]→[8.2083:2308](∅→∅),[8.2308]→[4.1271:1420](∅→∅),[4.1420]→[8.2464:2474](∅→∅),[8.2464]→[8.2464:2474](∅→∅),[8.1496]→[8.323:330](∅→∅),[8.2474]→[8.323:330](∅→∅),[8.323]→[8.323:330](∅→∅),[8.330]→[8.2475:2621](∅→∅),[8.2621]→[8.532:618](∅→∅),[8.532]→[8.532:618](∅→∅),[8.618]→[8.6185:6272](∅→∅),[8.918]→[8.759:832](∅→∅),[8.759]→[8.759:832](∅→∅),[8.832]→[8.0:46](∅→∅),[8.46]→[8.6481:6525](∅→∅),[8.6571]→[8.6571:6615](∅→∅),[8.171]→[8.947:960](∅→∅),[8.956]→[8.947:960](∅→∅),[8.6615]→[8.947:960](∅→∅),[8.947]→[8.947:960](∅→∅),[8.177]→[8.6616:6670](∅→∅),[8.6670]→[8.2622:2690](∅→∅),[8.59]→[8.6727:6803](∅→∅),[8.2690]→[8.6727:6803](∅→∅),[8.6727]→[8.6727:6803](∅→∅),[8.6803]→[8.2691:3441](∅→∅)
sub fetchInputGit {my ($db, $project, $jobset, $name, $value) = @_;(my $uri, my $branch, my $deepClone) = split ' ', $value;$branch = defined $branch ? $branch : "master";my $timestamp = time;my $sha256;my $storePath;mkpath(scmPath);my $clonePath = scmPath . "/" . sha256_hex($uri);my $stdout = ""; my $stderr = ""; my $res;if (! -d $clonePath) {# Clone everything and fetch the branch.# TODO: Optimize the first clone by using "git init $clonePath" and "git remote add origin $uri".($res, $stdout, $stderr) = captureStdoutStderr(600, "git", "clone", "--branch", $branch, $uri, $clonePath);die "error cloning git repo at `$uri':\n$stderr" if $res;}chdir $clonePath or die $!; # !!! urgh, shouldn't do a chdir# This command force the update of the local branch to be in the same as# the remote branch for whatever the repository state is. This command mirror# only one branch of the remote repository.($res, $stdout, $stderr) = captureStdoutStderr(600,"git", "fetch", "-fu", "origin", "+$branch:$branch");($res, $stdout, $stderr) = captureStdoutStderr(600,"git", "fetch", "-fu", "origin") if $res;die "error fetching latest change from git repo at `$uri':\n$stderr" if $res;($res, $stdout, $stderr) = captureStdoutStderr(600,("git", "rev-parse", "$branch"));die "error getting revision number of Git branch '$branch' at `$uri':\n$stderr" if $res;my ($revision) = split /\n/, $stdout;die "error getting a well-formated revision number of Git branch '$branch' at `$uri':\n$stdout"unless $revision =~ /^[0-9a-fA-F]+$/;my $ref = "refs/heads/$branch";# If deepClone is defined, then we look at the content of the repository# to determine if this is a top-git branch.if (defined $deepClone) {# Checkout the branch to look at its content.($res, $stdout, $stderr) = captureStdoutStderr(600, "git", "checkout", "$branch");die "error checking out Git branch '$branch' at `$uri':\n$stderr" if $res;if (-f ".topdeps") {# This is a TopGit branch. Fetch all the topic branches so# that builders can run "tg patch" and similar.($res, $stdout, $stderr) = captureStdoutStderr(600,"tg", "remote", "--populate", "origin");print STDERR "warning: `tg remote --populate origin' failed:\n$stderr" if $res;}}# Some simple caching: don't check a uri/branch/revision more than once.# TODO: Fix case where the branch is reset to a previous commit.my $cachedInput ;($cachedInput) = $db->resultset('CachedGitInputs')->search({uri => $uri, branch => $branch, revision => $revision},{rows => 1});if (defined $cachedInput && isValidPath($cachedInput->storepath)) {$storePath = $cachedInput->storepath;$sha256 = $cachedInput->sha256hash;$revision = $cachedInput->revision;} else {# Then download this revision into the store.print STDERR "checking out Git branch $branch from $uri\n";$ENV{"NIX_HASH_ALGO"} = "sha256";$ENV{"PRINT_PATH"} = "1";$ENV{"NIX_PREFETCH_GIT_LEAVE_DOT_GIT"} = "0";$ENV{"NIX_PREFETCH_GIT_DEEP_CLONE"} = "";if (defined $deepClone) {# Checked out code often wants to be able to run `git# describe', e.g., code that uses Gnulib's `git-version-gen'# script. Thus, we leave `.git' in there. Same for# Subversion (e.g., libgcrypt's build system uses that.)$ENV{"NIX_PREFETCH_GIT_LEAVE_DOT_GIT"} = "1";# Ask for a "deep clone" to allow "git describe" and similar# tools to work. See# http://thread.gmane.org/gmane.linux.distributions.nixos/3569# for a discussion.$ENV{"NIX_PREFETCH_GIT_DEEP_CLONE"} = "1";} - edit in src/lib/Hydra/Helper/AddBuilds.pm at line 151
($res, $stdout, $stderr) = captureStdoutStderr(600, "nix-prefetch-git", $clonePath, $revision);die "cannot check out Git repository branch '$branch' at `$uri':\n$stderr" if $res; - edit in src/lib/Hydra/Helper/AddBuilds.pm at line 152[8.3]→[8.7626:7678](∅→∅),[8.1149]→[8.7626:7678](∅→∅),[8.7678]→[8.267:268](∅→∅),[8.268]→[8.784:1063](∅→∅),[8.1335]→[8.784:1063](∅→∅),[8.1063]→[8.8276:8296](∅→∅),[8.8276]→[8.8276:8296](∅→∅),[8.1991]→[8.8394:8410](∅→∅),[8.1921]→[8.10008:10014](∅→∅),[8.2011]→[8.10008:10014](∅→∅),[8.8420]→[8.10008:10014](∅→∅),[8.10008]→[8.10008:10014](∅→∅),[8.10014]→[8.4:73](∅→∅),[8.73]→[8.0:175](∅→∅),[8.175]→[8.199:420](∅→∅),[8.199]→[8.199:420](∅→∅),[8.420]→[8.176:290](∅→∅),[8.290]→[8.1922:1934](∅→∅),[8.420]→[8.1922:1934](∅→∅),[8.10014]→[8.1922:1934](∅→∅),[8.1934]→[8.419:441](∅→∅),[8.441]→[8.8467:8565](∅→∅),[8.8467]→[8.8467:8565](∅→∅),[8.8565]→[8.421:486](∅→∅),[8.486]→[8.291:323](∅→∅),[8.323]→[8.8565:8578](∅→∅),[8.486]→[8.8565:8578](∅→∅),[8.8565]→[8.8565:8578](∅→∅),[8.8578]→[8.92:93](∅→∅),[8.93]→[8.228:229](∅→∅),[8.229]→[8.93:116](∅→∅),[8.93]→[8.93:116](∅→∅),[8.116]→[8.442:506](∅→∅),[8.506]→[8.187:275](∅→∅),[8.187]→[8.187:275](∅→∅),[8.275]→[8.1233:1234](∅→∅),[8.1234]→[8.294:315](∅→∅),[8.294]→[8.294:315](∅→∅),[8.315]→[8.1235:1289](∅→∅),[8.1289]→[8.366:394](∅→∅),[8.366]→[8.366:394](∅→∅),[8.394]→[4.1618:1788](∅→∅),[4.1788]→[8.582:621](∅→∅),[8.582]→[8.582:621](∅→∅),[8.621]→[4.1789:1945](∅→∅),[4.1945]→[8.791:854](∅→∅),[8.791]→[8.791:854](∅→∅),[8.854]→[4.1946:2019](∅→∅),[4.2019]→[8.942:943](∅→∅),[8.942]→[8.942:943](∅→∅),[8.943]→[4.2020:2144](∅→∅),[4.2144]→[8.1071:1447](∅→∅),[8.1071]→[8.1071:1447](∅→∅),[8.1447]→[8.269:270](∅→∅),[8.270]→[8.1460:1745](∅→∅),[8.1460]→[8.1460:1745](∅→∅),[8.1745]→[8.271:272](∅→∅),[8.272]→[8.1754:1817](∅→∅),[8.1754]→[8.1754:1817](∅→∅),[8.1817]→[4.2145:2272](∅→∅),[4.2272]→[8.1950:2282](∅→∅),[8.1950]→[8.1950:2282](∅→∅),[8.2282]→[8.2117:2118](∅→∅),[8.8578]→[8.2117:2118](∅→∅),[8.2117]→[8.2117:2118](∅→∅),[8.2118]→[8.273:284](∅→∅),[8.284]→[8.507:529](∅→∅),[8.2295]→[8.507:529](∅→∅),[8.529]→[8.2341:2439](∅→∅),[8.2341]→[8.2341:2439](∅→∅),[8.2439]→[8.4:18](∅→∅),[8.18]→[8.285:286](∅→∅),[8.286]→[8.249:268](∅→∅),[8.2456]→[8.249:268](∅→∅),[8.2121]→[8.249:268](∅→∅),[8.268]→[8.530:583](∅→∅),[8.583]→[8.287:288](∅→∅),[8.288]→[8.0:43](∅→∅),[8.274]→[8.0:43](∅→∅),[8.43]→[8.0:41](∅→∅),[8.41]→[8.374:402](∅→∅),[8.43]→[8.374:402](∅→∅),[8.374]→[8.374:402](∅→∅),[8.402]→[8.3503:3541](∅→∅),[8.3541]→[8.1290:1291](∅→∅),[8.430]→[8.1290:1291](∅→∅),[8.1291]→[8.449:470](∅→∅),[8.449]→[8.449:470](∅→∅),[8.470]→[8.1292:1346](∅→∅),[8.1346]→[8.521:612](∅→∅),[8.521]→[8.521:612](∅→∅),[8.612]→[4.2273:2391](∅→∅),[4.2391]→[8.736:801](∅→∅),[8.736]→[8.736:801](∅→∅),[8.801]→[4.2392:2548](∅→∅),[4.2548]→[8.971:972](∅→∅),[8.971]→[8.971:972](∅→∅),[8.972]→[4.2549:2770](∅→∅),[8.258]→[8.1142:1143](∅→∅),[8.1140]→[8.1142:1143](∅→∅),[4.2770]→[8.1142:1143](∅→∅),[8.1142]→[8.1142:1143](∅→∅),[8.1143]→[8.132:193](∅→∅),[8.193]→[8.289:290](∅→∅),[8.290]→[8.1295:1806](∅→∅),[8.1295]→[8.1295:1806](∅→∅),[8.1806]→[8.291:292](∅→∅),[8.292]→[8.1815:1878](∅→∅),[8.1815]→[8.1815:1878](∅→∅),[8.1878]→[4.2771:2904](∅→∅),[4.2904]→[8.2011:2091](∅→∅),[8.2011]→[8.2011:2091](∅→∅),[8.2091]→[3.71:135](∅→∅),[3.135]→[8.2145:2376](∅→∅),[8.2145]→[8.2145:2376](∅→∅),[8.2376]→[8.293:304](∅→∅),[8.304]→[8.584:606](∅→∅),[8.2388]→[8.584:606](∅→∅),[8.606]→[8.2434:2560](∅→∅),[8.2434]→[8.2434:2560](∅→∅),[8.2560]→[8.194:231](∅→∅),[8.231]→[8.2560:2571](∅→∅),[8.2560]→[8.2560:2571](∅→∅),[8.2576]→[8.2201:2203](∅→∅),[8.2201]→[8.2201:2203](∅→∅),[8.2203]→[8.8579:8580](∅→∅),[8.8580]→[8.2203:2204](∅→∅),[8.2203]→[8.2203:2204](∅→∅)
($sha256, $storePath) = split ' ', $stdout;txn_do($db, sub {$db->resultset('CachedGitInputs')->update_or_create({ uri => $uri, branch => $branch, revision => $revision, sha256hash => $sha256, storepath => $storePath});});}# For convenience in producing readable version names, pass the# number of commits in the history of this revision (‘revCount’)# the output of git-describe (‘gitTag’), and the abbreviated# revision (‘shortRev’).my $revCount = `git rev-list $revision | wc -l`; chomp $revCount;die "git rev-list failed" if $? != 0;my $gitTag = `git describe --always $revision`; chomp $gitTag;die "git describe failed" if $? != 0;my $shortRev = `git rev-parse --short $revision`; chomp $shortRev;die "git rev-parse failed" if $? != 0;return{ uri => $uri, storePath => $storePath, sha256hash => $sha256, revision => $revision, revCount => int($revCount), gitTag => $gitTag, shortRev => $shortRev};}sub fetchInputBazaar {my ($db, $project, $jobset, $name, $value, $checkout) = @_;my $uri = $value;my $sha256;my $storePath;my $stdout; my $stderr;mkpath(scmPath);my $clonePath = scmPath . "/" . sha256_hex($uri);if (! -d $clonePath) {(my $res, $stdout, $stderr) = captureStdoutStderr(600, "bzr", "branch", $uri, $clonePath);die "error cloning bazaar branch at `$uri':\n$stderr" if $res;}chdir $clonePath or die $!;(my $res, $stdout, $stderr) = captureStdoutStderr(600, "bzr", "pull");die "error pulling latest change bazaar branch at `$uri':\n$stderr" if $res;# First figure out the last-modified revision of the URI.my @cmd = (["bzr", "revno"], "|", ["sed", 's/^ *\([0-9]*\).*/\1/']);IPC::Run::run(@cmd, \$stdout, \$stderr);die "cannot get head revision of Bazaar branch at `$uri':\n$stderr" if $?;my $revision = $stdout; chomp $revision;die unless $revision =~ /^\d+$/;(my $cachedInput) = $db->resultset('CachedBazaarInputs')->search({uri => $uri, revision => $revision});if (defined $cachedInput && isValidPath($cachedInput->storepath)) {$storePath = $cachedInput->storepath;$sha256 = $cachedInput->sha256hash;} else {# Then download this revision into the store.print STDERR "checking out Bazaar input ", $name, " from $uri revision $revision\n";$ENV{"NIX_HASH_ALGO"} = "sha256";$ENV{"PRINT_PATH"} = "1";$ENV{"NIX_PREFETCH_BZR_LEAVE_DOT_BZR"} = "$checkout";(my $res, $stdout, $stderr) = captureStdoutStderr(600,"nix-prefetch-bzr", $clonePath, $revision);die "cannot check out Bazaar branch `$uri':\n$stderr" if $res;($sha256, $storePath) = split ' ', $stdout;txn_do($db, sub {$db->resultset('CachedBazaarInputs')->create({ uri => $uri, revision => $revision, sha256hash => $sha256, storepath => $storePath});});}return{ uri => $uri, storePath => $storePath, sha256hash => $sha256, revision => $revision};}sub fetchInputHg {my ($db, $project, $jobset, $name, $value) = @_;(my $uri, my $id) = split ' ', $value;$id = defined $id ? $id : "default";# init local hg clonemy $stdout = ""; my $stderr = "";mkpath(scmPath);my $clonePath = scmPath . "/" . sha256_hex($uri);if (! -d $clonePath) {(my $res, $stdout, $stderr) = captureStdoutStderr(600,"hg", "clone", $uri, $clonePath);die "error cloning mercurial repo at `$uri':\n$stderr" if $res;}# hg pull + check revchdir $clonePath or die $!;(my $res, $stdout, $stderr) = captureStdoutStderr(600, "hg", "pull");die "error pulling latest change mercurial repo at `$uri':\n$stderr" if $res;(my $res1, $stdout, $stderr) = captureStdoutStderr(600,"hg", "log", "-r", $id, "--template", "{node|short} {rev} {branch}");die "error getting branch and revision of $id from `$uri':\n$stderr" if $res1;my ($revision, $revCount, $branch) = split ' ', $stdout;my $storePath;my $sha256;(my $cachedInput) = $db->resultset('CachedHgInputs')->search({uri => $uri, branch => $branch, revision => $revision});if (defined $cachedInput && isValidPath($cachedInput->storepath)) {$storePath = $cachedInput->storepath;$sha256 = $cachedInput->sha256hash;} else {print STDERR "checking out Mercurial input from $uri $branch revision $revision\n";$ENV{"NIX_HASH_ALGO"} = "sha256";$ENV{"PRINT_PATH"} = "1";(my $res, $stdout, $stderr) = captureStdoutStderr(600,"nix-prefetch-hg", $clonePath, $revision);die "cannot check out Mercurial repository `$uri':\n$stderr" if $res;($sha256, $storePath) = split ' ', $stdout;txn_do($db, sub {$db->resultset('CachedHgInputs')->update_or_create({ uri => $uri, branch => $branch, revision => $revision, sha256hash => $sha256, storepath => $storePath});});}return{ uri => $uri, branch => $branch, storePath => $storePath, sha256hash => $sha256, revision => $revision, revCount => int($revCount)};} - replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 153
my ($db, $project, $jobset, $name, $type, $value) = @_;my ($plugins, $db, $project, $jobset, $name, $type, $value) = @_; - replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 156[8.2282]→[8.2282:2309](∅→∅),[8.2309]→[8.17:94](∅→∅),[8.94]→[8.2379:2414](∅→∅),[8.695]→[8.2379:2414](∅→∅),[8.8658]→[8.2379:2414](∅→∅),[8.2379]→[8.2379:2414](∅→∅),[8.2414]→[8.95:174](∅→∅),[8.174]→[8.337:381](∅→∅),[8.770]→[8.337:381](∅→∅),[8.337]→[8.337:381](∅→∅),[8.381]→[8.175:254](∅→∅),[8.254]→[8.2483:2520](∅→∅),[8.460]→[8.2483:2520](∅→∅),[8.845]→[8.2483:2520](∅→∅),[8.8735]→[8.2483:2520](∅→∅),[8.2483]→[8.2483:2520](∅→∅),[8.2520]→[8.255:333](∅→∅)
if ($type eq "path") {push @inputs, fetchInputPath($db, $project, $jobset, $name, $value);}elsif ($type eq "svn") {push @inputs, fetchInputSVN($db, $project, $jobset, $name, $value, 0);}elsif ($type eq "svn-checkout") {push @inputs, fetchInputSVN($db, $project, $jobset, $name, $value, 1);}elsif ($type eq "build") {push @inputs, fetchInputBuild($db, $project, $jobset, $name, $value);if ($type eq "build") {@inputs = fetchInputBuild($db, $project, $jobset, $name, $value); - replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 160
push @inputs, fetchInputSystemBuild($db, $project, $jobset, $name, $value);@inputs = fetchInputSystemBuild($db, $project, $jobset, $name, $value); - edit in src/lib/Hydra/Helper/AddBuilds.pm at line 162[8.2597]→[8.2597:2626](∅→∅),[8.2626]→[8.419:495](∅→∅),[8.495]→[8.2695:2701](∅→∅),[8.1071]→[8.2695:2701](∅→∅),[8.8976]→[8.2695:2701](∅→∅),[8.2695]→[8.2695:2701](∅→∅),[8.2701]→[8.0:28](∅→∅),[8.28]→[8.496:571](∅→∅),[8.571]→[8.103:109](∅→∅),[8.1142]→[8.103:109](∅→∅),[8.103]→[8.103:109](∅→∅),[8.109]→[8.2457:2486](∅→∅),[8.2486]→[8.572:654](∅→∅),[8.654]→[8.2568:2612](∅→∅),[8.1220]→[8.2568:2612](∅→∅),[8.2568]→[8.2568:2612](∅→∅),[8.2612]→[8.655:737](∅→∅),[8.737]→[8.305:311](∅→∅)
elsif ($type eq "git") {push @inputs, fetchInputGit($db, $project, $jobset, $name, $value);}elsif ($type eq "hg") {push @inputs, fetchInputHg($db, $project, $jobset, $name, $value);}elsif ($type eq "bzr") {push @inputs, fetchInputBazaar($db, $project, $jobset, $name, $value, 0);}elsif ($type eq "bzr-checkout") {push @inputs, fetchInputBazaar($db, $project, $jobset, $name, $value, 1);} - replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 164
push @inputs, { value => $value };@inputs = { value => $value }; - replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 168
push @inputs, { value => $value };@inputs = { value => $value }; - replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 171
die "input `" . $name . "' has unknown type `$type'.";my $found = 0;foreach my $plugin (@{$plugins}) {@inputs = $plugin->fetchInput($type, $name, $value);if (defined $inputs[0]) {$found = 1;last;}}die "input `$name' has unknown type `$type'." unless $found; - replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 182[8.1378]→[8.326:360](∅→∅),[8.360]→[8.2781:2831](∅→∅),[8.2831]→[8.904:910](∅→∅),[8.904]→[8.904:910](∅→∅)
foreach my $input (@inputs) {$input->{type} = $type if defined $input;}$_->{type} = $type foreach @inputs; - edit in src/lib/Hydra/Helper/AddBuilds.pm at line 203
- edit in src/lib/Hydra/Helper/AddBuilds.pm at line 218
(defined $input->{revNumber} ? "; rev = " . $input->{revNumber} . "" : "") . - replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 246
when (["path", "build", "git", "hg", "sysbuild"]) {default { - edit in src/lib/Hydra/Helper/AddBuilds.pm at line 248
}when (["svn", "svn-checkout", "bzr", "bzr-checkout"]) {push @res, "--arg", $input, ("{ outPath = builtins.storePath " . $alt->{storePath} . "" .(defined $alt->{revision} ? "; rev = " . $alt->{revision} . "" : "") .";}"); - edit in src/lib/Hydra/Helper/AddBuilds.pm at line 255
- edit in src/lib/Hydra/Helper/AddBuilds.pm at line 256[8.20]→[8.2297:2323](∅→∅),[8.2297]→[8.2297:2323](∅→∅),[8.2323]→[8.1347:1377](∅→∅),[8.398]→[8.398:450](∅→∅)
sub captureStdoutStderr {my ($timeout, @cmd) = @_;my $stdin = "";my $stdout;my $stderr; - edit in src/lib/Hydra/Helper/AddBuilds.pm at line 257[8.451]→[8.451:556](∅→∅),[8.556]→[4.2969:3028](∅→∅),[4.3028]→[8.623:640](∅→∅),[8.623]→[8.623:640](∅→∅),[8.640]→[8.0:7](∅→∅),[8.7]→[8.648:649](∅→∅),[8.648]→[8.648:649](∅→∅),[8.649]→[8.8:22](∅→∅),[8.22]→[4.3029:3135](∅→∅),[8.1605]→[8.116:129](∅→∅),[4.3135]→[8.116:129](∅→∅),[8.3583]→[8.116:129](∅→∅),[8.116]→[8.116:129](∅→∅),[8.129]→[4.3136:3175](∅→∅),[8.1647]→[8.163:169](∅→∅),[4.3175]→[8.163:169](∅→∅),[8.163]→[8.163:169](∅→∅),[8.169]→[8.835:837](∅→∅),[8.835]→[8.835:837](∅→∅),[8.837]→[8.21:22](∅→∅),[8.22]→[8.1378:1379](∅→∅)
eval {local $SIG{ALRM} = sub { die "timeout\n" }; # NB: \n requiredalarm $timeout;IPC::Run::run(\@cmd, \$stdin, \$stdout, \$stderr);alarm 0;};if ($@) {die unless $@ eq "timeout\n"; # propagate unexpected errorsreturn (-1, "", "timeout\n");} else {return ($?, $stdout, $stderr);}} - edit in src/lib/Hydra/Helper/Nix.pm at line 14
getSCMCacheDir - replacement in src/lib/Hydra/Helper/Nix.pm at line 22
pathIsInsidePrefix);pathIsInsidePrefixcaptureStdoutStderr); - edit in src/lib/Hydra/Helper/Nix.pm at line 52
}sub getSCMCacheDir {return Hydra::Model::DB::getHydraPath . "/scm" ; - edit in src/lib/Hydra/Helper/Nix.pm at line 451
- edit in src/lib/Hydra/Helper/Nix.pm at line 453
sub captureStdoutStderr {my ($timeout, @cmd) = @_;my $stdin = "";my $stdout;my $stderr; - edit in src/lib/Hydra/Helper/Nix.pm at line 459
eval {local $SIG{ALRM} = sub { die "timeout\n" }; # NB: \n requiredalarm $timeout;IPC::Run::run(\@cmd, \$stdin, \$stdout, \$stderr);alarm 0;};if ($@) {die unless $@ eq "timeout\n"; # propagate unexpected errorsreturn (-1, "", "timeout\n");} else {return ($?, $stdout, $stderr);}} - file addition: BazaarInput.pm[99.1]
package Hydra::Plugin::BazaarInput;use strict;use parent 'Hydra::Plugin';use Digest::SHA qw(sha256_hex);use File::Path;use Hydra::Helper::Nix;use Nix::Store;sub supportedInputTypes {my ($self, $inputTypes) = @_;$inputTypes->{'bzr'} = 'Bazaar export';$inputTypes->{'bzr-checkout'} = 'Bazaar checkout';}sub fetchInput {my ($self, $type, $name, $value) = @_;return undef if $type ne "bzr" && $type ne "bzr-checkout";my $uri = $value;my $sha256;my $storePath;my $stdout; my $stderr;my $cacheDir = getSCMCacheDir . "/bzr";mkpath($cacheDir);my $clonePath = $cacheDir . "/" . sha256_hex($uri);if (! -d $clonePath) {(my $res, $stdout, $stderr) = captureStdoutStderr(600, "bzr", "branch", $uri, $clonePath);die "error cloning bazaar branch at `$uri':\n$stderr" if $res;}chdir $clonePath or die $!;(my $res, $stdout, $stderr) = captureStdoutStderr(600, "bzr", "pull");die "error pulling latest change bazaar branch at `$uri':\n$stderr" if $res;# First figure out the last-modified revision of the URI.my @cmd = (["bzr", "revno"], "|", ["sed", 's/^ *\([0-9]*\).*/\1/']);IPC::Run::run(@cmd, \$stdout, \$stderr);die "cannot get head revision of Bazaar branch at `$uri':\n$stderr" if $?;my $revision = $stdout; chomp $revision;die unless $revision =~ /^\d+$/;(my $cachedInput) = $self->{db}->resultset('CachedBazaarInputs')->search({uri => $uri, revision => $revision});if (defined $cachedInput && isValidPath($cachedInput->storepath)) {$storePath = $cachedInput->storepath;$sha256 = $cachedInput->sha256hash;} else {# Then download this revision into the store.print STDERR "checking out Bazaar input ", $name, " from $uri revision $revision\n";$ENV{"NIX_HASH_ALGO"} = "sha256";$ENV{"PRINT_PATH"} = "1";$ENV{"NIX_PREFETCH_BZR_LEAVE_DOT_BZR"} = $type eq "bzr-checkout" ? "1" : "0";(my $res, $stdout, $stderr) = captureStdoutStderr(600,"nix-prefetch-bzr", $clonePath, $revision);die "cannot check out Bazaar branch `$uri':\n$stderr" if $res;($sha256, $storePath) = split ' ', $stdout;txn_do($self->{db}, sub {$self->{db}->resultset('CachedBazaarInputs')->create({ uri => $uri, revision => $revision, sha256hash => $sha256, storepath => $storePath});});}return{ uri => $uri, storePath => $storePath, sha256hash => $sha256, revision => $revision};}1; - file addition: GitInput.pm[99.1]
package Hydra::Plugin::GitInput;use strict;use parent 'Hydra::Plugin';use Digest::SHA qw(sha256_hex);use File::Path;use Hydra::Helper::Nix;use Nix::Store;sub supportedInputTypes {my ($self, $inputTypes) = @_;$inputTypes->{'git'} = 'Git checkout';}sub fetchInput {my ($self, $type, $name, $value) = @_;return undef if $type ne "git";(my $uri, my $branch, my $deepClone) = split ' ', $value;$branch = defined $branch ? $branch : "master";my $timestamp = time;my $sha256;my $storePath;my $cacheDir = getSCMCacheDir . "/git";mkpath($cacheDir);my $clonePath = $cacheDir . "/" . sha256_hex($uri);my $stdout = ""; my $stderr = ""; my $res;if (! -d $clonePath) {# Clone everything and fetch the branch.# TODO: Optimize the first clone by using "git init $clonePath" and "git remote add origin $uri".($res, $stdout, $stderr) = captureStdoutStderr(600, "git", "clone", "--branch", $branch, $uri, $clonePath);die "error cloning git repo at `$uri':\n$stderr" if $res;}chdir $clonePath or die $!; # !!! urgh, shouldn't do a chdir# This command force the update of the local branch to be in the same as# the remote branch for whatever the repository state is. This command mirror# only one branch of the remote repository.($res, $stdout, $stderr) = captureStdoutStderr(600,"git", "fetch", "-fu", "origin", "+$branch:$branch");($res, $stdout, $stderr) = captureStdoutStderr(600,"git", "fetch", "-fu", "origin") if $res;die "error fetching latest change from git repo at `$uri':\n$stderr" if $res;($res, $stdout, $stderr) = captureStdoutStderr(600,("git", "rev-parse", "$branch"));die "error getting revision number of Git branch '$branch' at `$uri':\n$stderr" if $res;my ($revision) = split /\n/, $stdout;die "error getting a well-formated revision number of Git branch '$branch' at `$uri':\n$stdout"unless $revision =~ /^[0-9a-fA-F]+$/;my $ref = "refs/heads/$branch";# If deepClone is defined, then we look at the content of the repository# to determine if this is a top-git branch.if (defined $deepClone) {# Checkout the branch to look at its content.($res, $stdout, $stderr) = captureStdoutStderr(600, "git", "checkout", "$branch");die "error checking out Git branch '$branch' at `$uri':\n$stderr" if $res;if (-f ".topdeps") {# This is a TopGit branch. Fetch all the topic branches so# that builders can run "tg patch" and similar.($res, $stdout, $stderr) = captureStdoutStderr(600,"tg", "remote", "--populate", "origin");print STDERR "warning: `tg remote --populate origin' failed:\n$stderr" if $res;}}# Some simple caching: don't check a uri/branch/revision more than once.# TODO: Fix case where the branch is reset to a previous commit.my $cachedInput ;($cachedInput) = $self->{db}->resultset('CachedGitInputs')->search({uri => $uri, branch => $branch, revision => $revision},{rows => 1});if (defined $cachedInput && isValidPath($cachedInput->storepath)) {$storePath = $cachedInput->storepath;$sha256 = $cachedInput->sha256hash;$revision = $cachedInput->revision;} else {# Then download this revision into the store.print STDERR "checking out Git branch $branch from $uri\n";$ENV{"NIX_HASH_ALGO"} = "sha256";$ENV{"PRINT_PATH"} = "1";$ENV{"NIX_PREFETCH_GIT_LEAVE_DOT_GIT"} = "0";$ENV{"NIX_PREFETCH_GIT_DEEP_CLONE"} = "";if (defined $deepClone) {# Checked out code often wants to be able to run `git# describe', e.g., code that uses Gnulib's `git-version-gen'# script. Thus, we leave `.git' in there. Same for# Subversion (e.g., libgcrypt's build system uses that.)$ENV{"NIX_PREFETCH_GIT_LEAVE_DOT_GIT"} = "1";# Ask for a "deep clone" to allow "git describe" and similar# tools to work. See# http://thread.gmane.org/gmane.linux.distributions.nixos/3569# for a discussion.$ENV{"NIX_PREFETCH_GIT_DEEP_CLONE"} = "1";}($res, $stdout, $stderr) = captureStdoutStderr(600, "nix-prefetch-git", $clonePath, $revision);die "cannot check out Git repository branch '$branch' at `$uri':\n$stderr" if $res;($sha256, $storePath) = split ' ', $stdout;txn_do($self->{db}, sub {$self->{db}->resultset('CachedGitInputs')->update_or_create({ uri => $uri, branch => $branch, revision => $revision, sha256hash => $sha256, storepath => $storePath});});}# For convenience in producing readable version names, pass the# number of commits in the history of this revision (‘revCount’)# the output of git-describe (‘gitTag’), and the abbreviated# revision (‘shortRev’).my $revCount = `git rev-list $revision | wc -l`; chomp $revCount;die "git rev-list failed" if $? != 0;my $gitTag = `git describe --always $revision`; chomp $gitTag;die "git describe failed" if $? != 0;my $shortRev = `git rev-parse --short $revision`; chomp $shortRev;die "git rev-parse failed" if $? != 0;return{ uri => $uri, storePath => $storePath, sha256hash => $sha256, revision => $revision, revCount => int($revCount), gitTag => $gitTag, shortRev => $shortRev};}1; - file addition: MercurialInput.pm[99.1]
package Hydra::Plugin::MercurialInput;use strict;use parent 'Hydra::Plugin';use Digest::SHA qw(sha256_hex);use File::Path;use Hydra::Helper::Nix;use Nix::Store;sub supportedInputTypes {my ($self, $inputTypes) = @_;$inputTypes->{'hg'} = 'Mercurial checkout';}sub fetchInput {my ($self, $type, $name, $value) = @_;return undef if $type ne "hg";(my $uri, my $id) = split ' ', $value;$id = defined $id ? $id : "default";# init local hg clonemy $stdout = ""; my $stderr = "";my $cacheDir = getSCMCacheDir . "/hg";mkpath($cacheDir);my $clonePath = $cacheDir . "/" . sha256_hex($uri);if (! -d $clonePath) {(my $res, $stdout, $stderr) = captureStdoutStderr(600,"hg", "clone", $uri, $clonePath);die "error cloning mercurial repo at `$uri':\n$stderr" if $res;}# hg pull + check revchdir $clonePath or die $!;(my $res, $stdout, $stderr) = captureStdoutStderr(600, "hg", "pull");die "error pulling latest change mercurial repo at `$uri':\n$stderr" if $res;(my $res1, $stdout, $stderr) = captureStdoutStderr(600,"hg", "log", "-r", $id, "--template", "{node|short} {rev} {branch}");die "error getting branch and revision of $id from `$uri':\n$stderr" if $res1;my ($revision, $revCount, $branch) = split ' ', $stdout;my $storePath;my $sha256;(my $cachedInput) = $self->{db}->resultset('CachedHgInputs')->search({uri => $uri, branch => $branch, revision => $revision});if (defined $cachedInput && isValidPath($cachedInput->storepath)) {$storePath = $cachedInput->storepath;$sha256 = $cachedInput->sha256hash;} else {print STDERR "checking out Mercurial input from $uri $branch revision $revision\n";$ENV{"NIX_HASH_ALGO"} = "sha256";$ENV{"PRINT_PATH"} = "1";(my $res, $stdout, $stderr) = captureStdoutStderr(600,"nix-prefetch-hg", $clonePath, $revision);die "cannot check out Mercurial repository `$uri':\n$stderr" if $res;($sha256, $storePath) = split ' ', $stdout;txn_do($self->{db}, sub {$self->{db}->resultset('CachedHgInputs')->update_or_create({ uri => $uri, branch => $branch, revision => $revision, sha256hash => $sha256, storepath => $storePath});});}return{ uri => $uri, branch => $branch, storePath => $storePath, sha256hash => $sha256, revision => $revision, revCount => int($revCount)};}1; - file addition: PathInput.pm[99.1]
package Hydra::Plugin::PathInput;use strict;use parent 'Hydra::Plugin';use POSIX qw(strftime);use Hydra::Helper::Nix;use Nix::Store;sub supportedInputTypes {my ($self, $inputTypes) = @_;$inputTypes->{'path'} = 'Local path';}sub fetchInput {my ($self, $type, $name, $value) = @_;return undef if $type ne "path";my $uri = $value;my $timestamp = time;my $sha256;my $storePath;# Some simple caching: don't check a path more than once every N seconds.(my $cachedInput) = $self->{db}->resultset('CachedPathInputs')->search({srcpath => $uri, lastseen => {">", $timestamp - 30}},{rows => 1, order_by => "lastseen DESC"});if (defined $cachedInput && isValidPath($cachedInput->storepath)) {$storePath = $cachedInput->storepath;$sha256 = $cachedInput->sha256hash;$timestamp = $cachedInput->timestamp;} else {print STDERR "copying input ", $name, " from $uri\n";$storePath = `nix-store --add "$uri"`or die "cannot copy path $uri to the Nix store.\n";chomp $storePath;$sha256 = (queryPathInfo($storePath, 0))[1] or die;($cachedInput) = $self->{db}->resultset('CachedPathInputs')->search({srcpath => $uri, sha256hash => $sha256});# Path inputs don't have a natural notion of a "revision", so# we simulate it by using the timestamp that we first saw this# path have this SHA-256 hash. So if the contents of the path# changes, we get a new "revision", but if it doesn't change# (or changes back), we don't get a new "revision".if (!defined $cachedInput) {txn_do($self->{db}, sub {$self->{db}->resultset('CachedPathInputs')->update_or_create({ srcpath => $uri, timestamp => $timestamp, lastseen => $timestamp, sha256hash => $sha256, storepath => $storePath});});} else {$timestamp = $cachedInput->timestamp;txn_do($self->{db}, sub {$cachedInput->update({lastseen => time});});}}return{ uri => $uri, storePath => $storePath, sha256hash => $sha256, revision => strftime "%Y%m%d%H%M%S", gmtime($timestamp)};}1; - file addition: SubversionInput.pm[99.1]
package Hydra::Plugin::SubversionInput;use strict;use parent 'Hydra::Plugin';use Digest::SHA qw(sha256_hex);use Hydra::Helper::Nix;use IPC::Run;use Nix::Store;sub supportedInputTypes {my ($self, $inputTypes) = @_;$inputTypes->{'svn'} = 'Subversion export';$inputTypes->{'svn-checkout'} = 'Subversion checkout';}sub fetchInput {my ($self, $type, $name, $value) = @_;return undef if $type ne "svn" && $type ne "svn-checkout";# Allow users to specify a revision number next to the URI.my ($uri, $revision) = split ' ', $value;my $sha256;my $storePath;my $stdout; my $stderr;unless (defined $revision) {# First figure out the last-modified revision of the URI.my @cmd = (["svn", "ls", "-v", "--depth", "empty", $uri],"|", ["sed", 's/^ *\([0-9]*\).*/\1/']);IPC::Run::run(@cmd, \$stdout, \$stderr);die "cannot get head revision of Subversion repository at `$uri':\n$stderr" if $?;$revision = int($stdout); $revision =~ s/\s*([0-9]+)\s*/$1/sm;}die unless $revision =~ /^\d+$/;$revision = int($revision);# Do we already have this revision in the store?# !!! This needs to take $checkout into account! Otherwise "svn"# and "svn-checkout" inputs can get mixed up.(my $cachedInput) = $self->{db}->resultset('CachedSubversionInputs')->search({uri => $uri, revision => $revision});if (defined $cachedInput && isValidPath($cachedInput->storepath)) {$storePath = $cachedInput->storepath;$sha256 = $cachedInput->sha256hash;} else {# No, do a checkout. The working copy is reused between# invocations to speed things up.my $wcPath = getSCMCacheDir . "/svn/" . sha256_hex($uri) . "/svn-checkout";print STDERR "checking out Subversion input ", $name, " from $uri revision $revision into $wcPath\n";(my $res, $stdout, $stderr) = captureStdoutStderr(600, "svn", "checkout", $uri, "-r", $revision, $wcPath);die "error checking out Subversion repo at `$uri':\n$stderr" if $res;if ($type eq "svn-checkout") {$storePath = addToStore($wcPath, 1, "sha256");} else {# Hm, if the Nix Perl bindings supported filters in# addToStore(), then we wouldn't need to make a copy here.my $tmpDir = File::Temp->newdir("hydra-svn-export.XXXXXX", CLEANUP => 1, TMPDIR => 1) or die;(system "svn", "export", $wcPath, "$tmpDir/svn-export", "--quiet") == 0 or die "svn export failed";$storePath = addToStore("$tmpDir/svn-export", 1, "sha256");}$sha256 = queryPathHash($storePath); $sha256 =~ s/sha256://;txn_do($self->{db}, sub {$self->{db}->resultset('CachedSubversionInputs')->update_or_create({ uri => $uri, revision => $revision, sha256hash => $sha256, storepath => $storePath});});}return{ uri => $uri, storePath => $storePath, sha256hash => $sha256, revNumber => $revision};}1; - edit in src/lib/Hydra/Plugin.pm at line 21
}# Called to determine the set of supported input types. The plugin# should add these to the $inputTypes hashref, e.g. $inputTypes{'svn'}# = 'Subversion checkout'.sub supportedInputTypes {my ($self, $inputTypes) = @_; - edit in src/lib/Hydra/Plugin.pm at line 30
# Called to fetch an input of type ‘$type’. ‘$value’ is the input# location, typically the repository URL.sub fetchInput {my ($self, $type, $name, $value) = @_;return undef;} - edit in src/lib/Hydra.pm at line 7
use Hydra::Plugin; - edit in src/lib/Hydra.pm at line 85
my $plugins; - edit in src/lib/Hydra.pm at line 88
has 'hydra_plugins' => (is => 'ro',default => sub { return $plugins; });after setup_finalize => sub {my $class = shift;$plugins = [Hydra::Plugin->plugins(db => $class->model('DB'), config => $class->config)];}; - edit in src/root/common.tt at line 5[8.35]→[8.694:695](∅→∅),[8.53]→[8.694:695](∅→∅),[8.588]→[8.694:695](∅→∅),[8.6935]→[8.694:695](∅→∅),[8.694]→[8.694:695](∅→∅)
- edit in src/root/common.tt at line 6
inputTypes ={ "svn" = "Subversion export", "svn-checkout" = "Subversion checkout", "bzr" = "Bazaar export", "bzr-checkout" = "Bazaar checkout", "git" = "Git checkout", "hg" = "Mercurial checkout", "string" = "String value", "boolean" = "Boolean", "path" = "Local path", "build" = "Build output", "sysbuild" = "Build output (same system)"}; - edit in src/root/edit-jobset.tt at line 170
- edit in src/script/hydra-evaluator at line 6
use Hydra::Plugin; - edit in src/script/hydra-evaluator at line 24[103.1][103.1]
my $plugins = [Hydra::Plugin->plugins(db => $db, config => $config)]; - replacement in src/script/hydra-evaluator at line 35
my @info = fetchInput($db, $project, $jobset, $input->name, $input->type, $alt->value);foreach my $info_el (@info) {push @{$$inputInfo{$input->name}}, $info_el if defined $info_el;}push @{$$inputInfo{$input->name}}, $_foreach fetchInput($plugins, $db, $project, $jobset, $input->name, $input->type, $alt->value);