Move most of AddBuilds to hydra-eval-jobset

[?]

Feb 21, 2017, 3:17 PM

VU2OLHD246DFWL2WPFSKWPLXTN3WRL25TSRU4PJMJBNWDF35OFRAC

Dependencies

[2] MSDSOL5L Fix same system inputs
[3] ZTQ5WWHD Pass project and jobset to fetchInput
[4] E3IEH57U Revert "Make evaluation fail with proper error when a input of type build is not available."
[5] GREPCKGI Die tabs die
[6] 5KWIPCUK UTF-8 fix
[7] O2BEFBIO buildInputToString: Use inputType attribute instead of type attribute
[8] GXKMNJ26 Pass a build's drv path as a store path
[9] BGDZAEU7 Remove unneeded camelcase
[10] KLFSX2R6 hydra-evaluator: Fix input change check
[11] 5TZGO7RJ Store unset descriptions etc. as nulls
[12] 53IMJNBB Add isChannel column and meta attribute.
[13] I6QMKSIS Move getBaseUrl
[14] DKIA7GAD Better fix for dots in jobset names
[15] NY77R3JJ Bump evaluation timeout to 6 hours
[16] MPFSVI5X Pedantry: CLOCK_REALTIME -> CLOCK_MONOTONIC
[17] JYZEGF56 Create Builds with iscurrent set
[18] FM4O2L4M hydra: if evaluator sees cached build, also add the buildproducts
[19] HE3GX5IP Optimize fetch-git.
[20] AFTXA575 * $HYDRA_DATA environment variable.
[21] NFVN7JRB Handle missing "build" inputs
[22] INNOEHO6 * Fix getBuildLog for bzip2'd files.
[23] PHNLYPKB Call buildFinished when a cached build is added
[24] FTPCV25M Store aggregate members in the database
[25] KQS7DSKJ * Clean up indentation.
[26] A43SLRSH Fix handling of IPC::Run::run exit status
[27] 5GPK54IV Log segfaults from the evaluator
[28] DDGBLKEN Update isCurrent properly
[29] CQTN62OH Die tabs die
[30] X5EOJLNA Allow passing a specific build as an input
[31] CPMIKBDT Allow dots in job specifier of input type 'Previous build'
[32] 5SHCWE7X * Prevent repeated evaluation of a jobset with the same inputs. This
[33] R4MHON3O pass svn/bzr revisions as integers
[34] FYWE74AA
[35] RWIBJ5L4 * Autoflush stdout.
[36] 4FWDVNWA Pass additional attributes for Git inputs
[37] PGSSRA7C Add an input type "nix" for passing arbitrary Nix expressions
[38] JK2QWPH6
[39] XDDCO6CH * hydra: add dependency list with links to logfiles
[40] PIMGMGAF Rename hydra_eval_jobs to hydra-eval-jobs
[41] 3XTHEUMP * Implemented the clone feature.
[42] OOQ2D3KC * Refactoring: move fetchInput out of hydra_scheduler into a separate
[43] LBNVQXUB * Build the /build stuff in a separate controller.
[44] X27GNHDV * Basic job info in the database.
[45] QW7GTTXI Don't maintain BuildInputs anymore
[46] PXUCXYZI * Pass `-j 1' to hydra_eval_jobs to ensure that it can make progress
[47] SMCOU72F hydra: add some admin for adding/enabling/etc build machines
[48] N22GPKYT * Put info about logs / build products in the DB.
[49] AKRVETP5 Handle UTF-8 characters in eval error messages
[50] Z52T2BC4 Support passing a jobset evaluation as an input
[51] NP4OLR5T Don't pass an undefined input
[52] YFPZ46YK * hydra: added variant of build input type, 'build output (same system)' to allow better continous integration in one jobset for multiple system. it makes sure that the system of the build that is passed as input for a job has the same system as the job.
[53] 2WRTOU2Z Cleanup
[54] SM5M2J3A Pass inputs to release expressions using -I
[55] CHQEG6WY Hydra/29: Added timeout to svn/git checkouts, job evaluation
[56] FAIJDQKZ
[57] 6GJSWZ4F Copy-paste error
[58] JZE7DC2F Whitespace
[59] JTRG7RDQ add support for git as jobinput
[60] SGNXIOI4 Hydra/32: Add option to force evaluation of a certain jobset via web interface (for admins only)
[61] BKOIYITR added some json responses
[62] RXVJFQ5A Evaluator cleanups
[63] SKQXOQ7T hydra: fixed missing argument to restartbuild function
[64] LZVO64YG Merge in the first bits of the API work
[65] JCJJKRWQ Handle job aliases in AggregateConstituents
[66] 24BMQDZA Start of single-process hydra-queue-runner
[67] ZWCTAZGL added newsitems, added some admin options to clear various caches.
[68] E7M2WP7A Remove unused Jobs columns
[69] MRJCQ4EO Don't use given/when
[70] JAH3UPWA Support revision control systems via plugins
[71] 5SMQ2PLK Fix tests
[72] OSVLMLCQ hydra: factored out build restart and
[73] QT4FO2HP refactored admin controller, using chains, to avoid using requireadmin on each endpoint
[74] IMQRX4MP hydra-eval-jobs: Use JSON instead of XML
[75] 4OXJRURP buildInputToString: Pass along the input's type and urr
[76] D3YETHSB Only pass the drv path if it is still valid
[77] DVLDHOUZ hydra-evaluator: Reduce verbosity
[78] P5XCKTFD Fix sysbuild input type handling
[79] PMNWRTGJ Add multiple output support
[80] ZTQEU5QS Hydra: Add support for maxSilent meta attribute (also already added timeout, but not implemented the actual timeout for the build yet)
[81] MOX7XJ2E Merge the BuildSchedulingInfo table into the Builds table
[82] H7CNGK4O * Log evaluation errors etc. in the DB.
[83] KHYZVPBR Propagate checkresponsible from JobsetInput to BuildInput
[84] WVD3YYON hydra-evaluator: Add some debug code
[85] QS4OX6Z7 Huuuge speedup in the Hydra evaluator
[86] RFE6T5LG * Store jobset evaluations in the database explicitly. This includes
[87] POPU2ATH * hydra_scheduler: use eval-jobs.
[88] A22P7HCO hydra: at evaluation, check if path is already built, and mark as built in stead of adding to the queue.
[89] SHYRGAWZ hydra: when no external url is given for diff's of git/hg input, revert to a diff done on a local available clone
[90] TJK27WSB Open the DB using Hydra::Model::DB->new
[91] HX4QYOYA add first evaluations tests
[92] IEXUBVNB allow dots (.) in job names when used as build input
[93] Y6FFX4YB Make evaluation fail with proper error when a input of type build is not available.
[94] ARD6Z67T Do incremental SVN checkouts
[95] 7DWCXNC7 Use the new Nix Perl bindings
[96] VHV6GI4L Add a jobset eval action to restart all aborted/cancelled builds
[97] 6WRGCITD Enable declarative projects.
[98] BMSQD2ZH Indentation
[99] JOYONH2K Prevent multiple builds with the same (job, outPath) tuple from being added
[100] K3HODXGH Check all inputs for blame but only email selected inputs
[101] WWUOQ7V4 * hydra: indentation and fixed duplicate key in cachedgitinputs bug
[102] H3S3H752 Pass along drvPath and outputName for inputs that are previous builds.
[103] Q24QXGSM * Don't do pretty printing for large logs, because the XSLT processing
[104] 5NO7NCKT * Refactoring.
[105] ZDENAYQI * email notification of evaluation errors to project owner (if desired)
[106] FXW2UR7F initial bzr support
[107] QMW24O5S Add support for Guile & Guix.
[108] JOVVHIJX Remove all entry points to modify machines
[109] RX5IIZMT Use Email::MIME instead of Email::Simple
[110] HPEG2RHV Merge the BuildResultInfo table into the Builds table
[111] TPNHTE5V Remove obsolete Builds columns and provide accurate "Running builds"
[112] EDRUQ4UK Die TABs die
[113] SS4TZXNU Distinguish between permanent evaluation errors and transient input errors
[114] DDMYFZ5X Fix the jobset unchanged check
[115] NS7SND6R hydra-evaluator: Send statistics to statsd
[*] 2GK5DOU7 * Downloading closures.

Change contents

edit in src/lib/Hydra/Controller/API.pm at line 8

[17.141]→[17.0:30](∅→∅)

use Hydra::Helper::AddBuilds;
edit in src/lib/Hydra/Controller/Admin.pm at line 8

[17.154]→[17.0:30](∅→∅)

use Hydra::Helper::AddBuilds;
edit in src/lib/Hydra/Controller/Build.pm at line 9

[17.156]→[17.0:30](∅→∅)

use Hydra::Helper::AddBuilds;
edit in src/lib/Hydra/Helper/AddBuilds.pm at line 7

[17.1945]→[17.1290:1304](∅→∅),[17.2735]→[17.1290:1304](∅→∅)

use IPC::Run;
replacement in src/lib/Hydra/Helper/AddBuilds.pm at line 22

[17.18]→[17.576:624](∅→∅),[17.624]→[17.710:769](∅→∅)

fetchInput evalJobs checkBuild inputsToArgs
restartBuild getPrevJobsetEval updateDeclarativeJobset

[17.18]

[17.769]

updateDeclarativeJobset
edit in src/lib/Hydra/Helper/AddBuilds.pm at line 25

[17.3175]→[17.0:1](∅→∅),[17.1]→[17.3175:3176](∅→∅),[17.3175]→[17.3175:3176](∅→∅),[17.3]→[17.3177:3502](∅→∅),[17.3177]→[17.3177:3502](∅→∅),[17.3502]→[14.0:82](∅→∅),[17.68]→[17.3568:4413](∅→∅),[17.70]→[17.3568:4413](∅→∅),[14.82]→[17.3568:4413](∅→∅),[17.3568]→[17.3568:4413](∅→∅),[17.3727]→[17.262:265](∅→∅),[17.8701]→[17.262:265](∅→∅),[17.265]→[17.3728:3729](∅→∅),[17.3729]→[17.265:287](∅→∅),[17.265]→[17.265:287](∅→∅),[17.287]→[17.165:218](∅→∅),[17.218]→[17.8707:8708](∅→∅),[17.347]→[17.8707:8708](∅→∅),[17.8707]→[17.8707:8708](∅→∅),[17.8708]→[17.110:129](∅→∅),[17.129]→[17.8900:8901](∅→∅),[17.3879]→[17.8900:8901](∅→∅),[17.8900]→[17.8900:8901](∅→∅),[17.8901]→[17.130:783](∅→∅),[17.783]→[17.9287:9288](∅→∅),[17.4242]→[17.9287:9288](∅→∅),[17.9287]→[17.9287:9288](∅→∅),[17.9288]→[4.0:87](∅→∅),[4.87]→[17.9475:9476](∅→∅),[17.87]→[17.9475:9476](∅→∅),[17.126]→[17.9475:9476](∅→∅),[17.4414]→[17.9475:9476](∅→∅),[17.9475]→[17.9475:9476](∅→∅),[17.9476]→[17.4415:4492](∅→∅),[17.4492]→[17.9557:9558](∅→∅),[17.9557]→[17.9557:9558](∅→∅),[17.9558]→[17.4493:4593](∅→∅),[17.4593]→[17.9666:9667](∅→∅),[17.9666]→[17.9666:9667](∅→∅),[17.9667]→[17.4477:4545](∅→∅),[17.4545]→[17.4674:4741](∅→∅),[17.4674]→[17.4674:4741](∅→∅),[17.4741]→[17.0:49](∅→∅),[17.49]→[17.222:223](∅→∅),[17.4741]→[17.222:223](∅→∅),[17.223]→[17.0:17](∅→∅),[17.17]→[17.50:91](∅→∅),[17.234]→[17.50:91](∅→∅),[17.91]→[17.4823:4884](∅→∅),[17.262]→[17.4823:4884](∅→∅),[17.4807]→[17.4823:4884](∅→∅),[17.4823]→[17.4823:4884](∅→∅),[17.4884]→[17.92:134](∅→∅),[17.175]→[17.4884:4895](∅→∅),[17.4884]→[17.4884:4895](∅→∅),[17.4895]→[9.0:94](∅→∅),[17.50]→[17.111:138](∅→∅),[9.94]→[17.111:138](∅→∅),[17.111]→[17.111:138](∅→∅),[17.138]→[17.134:136](∅→∅),[17.4895]→[17.134:136](∅→∅),[17.10008]→[17.134:136](∅→∅),[17.136]→[17.4896:4897](∅→∅),[17.4897]→[17.136:165](∅→∅),[17.136]→[17.136:165](∅→∅),[17.165]→[17.263:316](∅→∅),[17.316]→[17.225:226](∅→∅),[17.225]→[17.225:226](∅→∅),[17.226]→[17.4898:5047](∅→∅),[17.5047]→[17.387:388](∅→∅),[17.387]→[17.387:388](∅→∅),[17.388]→[17.5048:5182](∅→∅),[17.5182]→[17.528:529](∅→∅),[17.528]→[17.528:529](∅→∅),[17.529]→[17.5183:5249](∅→∅),[17.5249]→[2.0:80](∅→∅),[2.80]→[17.5317:5323](∅→∅),[17.4889]→[17.5317:5323](∅→∅),[17.5317]→[17.5317:5323](∅→∅)

sub parseJobName {
# Parse a job specification of the form `<project>:<jobset>:<job>
# [attrs]'. The project, jobset and attrs may be omitted. The
# attrs have the form `name = "value"'.
my ($s) = @_;
our $key;
our %attrs = ();
# hm, maybe I should stop programming Perl before it's too late...
$s =~ / ^ (?: (?: ($projectNameRE) : )? ($jobsetNameRE) : )? ($jobNameRE) \s*
(\[ \s* (
([\w]+) (?{ $key = $^N; }) \s* = \s* \"
([\w\-]+) (?{ $attrs{$key} = $^N; }) \"
\s* )* \])? $
/x
or die "invalid job specifier `$s'";
return ($1, $2, $3, \%attrs);
}

sub attrsToSQL {
my ($attrs, $id) = @_;

my $query = "1 = 1";

foreach my $name (keys %{$attrs}) {
my $value = $attrs->{$name};
$name =~ /^[\w\-]+$/ or die;
$value =~ /^[\w\-]+$/ or die;
# !!! Yes, this is horribly injection-prone... (though
# name/value are filtered above). Should use SQL::Abstract,
# but it can't deal with subqueries. At least we should use
# placeholders.
$query .= " and exists (select 1 from buildinputs where build = $id and name = '$name' and value = '$value')";
}

return $query;
}

sub fetchInputBuild {
my ($db, $project, $jobset, $name, $value) = @_;

my $prevBuild;

if ($value =~ /^\d+$/) {
$prevBuild = $db->resultset('Builds')->find({ id => int($value) });
} else {
my ($projectName, $jobsetName, $jobName, $attrs) = parseJobName($value);
$projectName ||= $project->name;
$jobsetName ||= $jobset->name;

# Pick the most recent successful build of the specified job.
$prevBuild = $db->resultset('Builds')->search(
{ finished => 1, project => $projectName, jobset => $jobsetName
, job => $jobName, buildStatus => 0 },
{ order_by => "me.id DESC", rows => 1
, where => \ attrsToSQL($attrs, "me.id") })->single;
}

return () if !defined $prevBuild || !isValidPath(getMainOutput($prevBuild)->path);

#print STDERR "input `", $name, "': using build ", $prevBuild->id, "\n";

my $pkgNameRE = "(?:(?:[A-Za-z0-9]|(?:-[^0-9]))+)";
my $versionRE = "(?:[A-Za-z0-9\.\-]+)";

my $relName = ($prevBuild->releasename or $prevBuild->nixname);
my $version = $2 if $relName =~ /^($pkgNameRE)-($versionRE)$/;

my $mainOutput = getMainOutput($prevBuild);

my $result =
{ storePath => $mainOutput->path
, id => $prevBuild->id
, version => $version
, outputName => $mainOutput->name
};
if (isValidPath($prevBuild->drvpath)) {
$result->{drvPath} = $prevBuild->drvpath;
}

return $result;
}

sub fetchInputSystemBuild {
my ($db, $project, $jobset, $name, $value) = @_;

my ($projectName, $jobsetName, $jobName, $attrs) = parseJobName($value);
$projectName ||= $project->name;
$jobsetName ||= $jobset->name;

my @latestBuilds = $db->resultset('LatestSucceededForJob')
->search({}, {bind => [$projectName, $jobsetName, $jobName]});

my @validBuilds = ();
foreach my $build (@latestBuilds) {
push(@validBuilds, $build) if isValidPath(getMainOutput($build)->path);
}
edit in src/lib/Hydra/Helper/AddBuilds.pm at line 26

[17.236]→[17.5332:5444](∅→∅),[17.5332]→[17.5332:5444](∅→∅),[17.5444]→[17.674:693](∅→∅),[17.693]→[17.5466:5472](∅→∅),[17.5466]→[17.5466:5472](∅→∅)

if (scalar(@validBuilds) == 0) {
print STDERR "input `", $name, "': no previous build available\n";
return ();
}
edit in src/lib/Hydra/Helper/AddBuilds.pm at line 27

[17.238]→[17.5477:5498](∅→∅),[17.5477]→[17.5477:5498](∅→∅),[17.5498]→[17.239:240](∅→∅),[17.240]→[17.5503:5654](∅→∅),[17.5503]→[17.5503:5654](∅→∅),[17.5654]→[17.241:242](∅→∅),[17.242]→[17.4546:4618](∅→∅),[17.714]→[17.4546:4618](∅→∅),[17.4618]→[17.5739:5810](∅→∅),[17.5739]→[17.5739:5810](∅→∅),[17.5810]→[17.243:244](∅→∅),[17.244]→[17.5827:5847](∅→∅),[17.5827]→[17.5827:5847](∅→∅),[17.5847]→[17.4890:4949](∅→∅),[17.364]→[17.5927:6091](∅→∅),[17.4949]→[17.5927:6091](∅→∅),[17.5927]→[17.5927:6091](∅→∅),[17.6091]→[17.245:266](∅→∅),[17.266]→[17.6137:6139](∅→∅),[17.6137]→[17.6137:6139](∅→∅),[17.6139]→[17.923:1014](∅→∅),[17.1014]→[17.0:1](∅→∅),[17.3441]→[17.0:1](∅→∅),[17.7419]→[17.0:1](∅→∅),[17.1]→[17.1015:2932](∅→∅),[17.2932]→[17.2204:2221](∅→∅),[17.2204]→[17.2204:2221](∅→∅),[17.2221]→[17.111:200](∅→∅),[17.89]→[17.0:16](∅→∅),[17.200]→[17.0:16](∅→∅),[17.764]→[17.0:16](∅→∅),[17.2281]→[17.0:16](∅→∅),[17.16]→[17.2281:2282](∅→∅),[17.622]→[17.2281:2282](∅→∅),[17.2281]→[17.2281:2282](∅→∅),[17.2282]→[17.765:867](∅→∅),[17.333]→[17.1515:1555](∅→∅),[17.867]→[17.1515:1555](∅→∅),[17.919]→[17.1515:1555](∅→∅),[17.8814]→[17.1515:1555](∅→∅),[17.2591]→[17.1515:1555](∅→∅),[17.1555]→[17.868:948](∅→∅),[17.948]→[17.2933:3042](∅→∅),[17.418]→[17.2591:2597](∅→∅),[17.948]→[17.2591:2597](∅→∅),[17.999]→[17.2591:2597](∅→∅),[17.1632]→[17.2591:2597](∅→∅),[17.3042]→[17.2591:2597](∅→∅),[17.8899]→[17.2591:2597](∅→∅),[17.2591]→[17.2591:2597](∅→∅),[17.2597]→[17.36:86](∅→∅),[17.86]→[17.10051:10086](∅→∅),[17.10051]→[17.10051:10086](∅→∅),[17.10086]→[17.949:988](∅→∅),[17.988]→[17.312:318](∅→∅),[17.781]→[17.312:318](∅→∅),[17.318]→[17.10146:10257](∅→∅),[17.2714]→[17.10146:10257](∅→∅),[17.10146]→[17.10146:10257](∅→∅),[17.10257]→[17.989:1028](∅→∅),[17.1028]→[17.319:325](∅→∅),[17.825]→[17.319:325](∅→∅),[17.325]→[17.10317:10328](∅→∅),[17.2725]→[17.10317:10328](∅→∅),[17.10317]→[17.10317:10328](∅→∅),[17.10328]→[17.1029:1095](∅→∅),[17.1095]→[3.0:84](∅→∅),[3.84]→[17.1160:1341](∅→∅),[17.1160]→[17.1160:1341](∅→∅),[17.1341]→[17.10391:10397](∅→∅),[17.2968]→[17.10391:10397](∅→∅),[17.10391]→[17.10391:10397](∅→∅),[17.10397]→[17.1377:1378](∅→∅),[17.1378]→[17.90:156](∅→∅),[17.156]→[17.201:257](∅→∅),[17.257]→[17.212:218](∅→∅),[17.212]→[17.212:218](∅→∅),[17.218]→[17.910:931](∅→∅),[17.1382]→[17.910:931](∅→∅),[17.910]→[17.910:931](∅→∅),[17.931]→[17.633:740](∅→∅),[17.740]→[17.2832:2973](∅→∅),[17.2973]→[17.863:876](∅→∅),[17.863]→[17.863:876](∅→∅),[17.876]→[17.2974:3000](∅→∅),[17.3000]→[17.899:925](∅→∅),[17.899]→[17.899:925](∅→∅),[17.925]→[17.1356:1358](∅→∅),[17.931]→[17.1356:1358](∅→∅),[17.1425]→[17.1356:1358](∅→∅),[17.10397]→[17.1356:1358](∅→∅),[17.1358]→[17.1383:1384](∅→∅),[17.1384]→[17.1358:1359](∅→∅),[17.1358]→[17.1358:1359](∅→∅),[17.1359]→[17.926:1032](∅→∅),[17.1032]→[17.3001:3570](∅→∅),[17.3570]→[17.1552:1565](∅→∅),[17.1552]→[17.1552:1565](∅→∅),[17.1565]→[17.3571:3652](∅→∅),[17.3652]→[7.0:57](∅→∅),[7.57]→[17.52:133](∅→∅),[17.52]→[17.52:133](∅→∅),[17.133]→[17.1385:1474](∅→∅),[17.3652]→[17.1385:1474](∅→∅),[17.1474]→[17.3652:4114](∅→∅),[17.3652]→[17.3652:4114](∅→∅),[17.4114]→[17.176:278](∅→∅),[17.278]→[8.0:108](∅→∅),[8.108]→[17.4114:4132](∅→∅),[17.367]→[17.4114:4132](∅→∅),[17.4114]→[17.4114:4132](∅→∅),[17.4132]→[17.2077:2105](∅→∅),[17.2077]→[17.2077:2105](∅→∅),[17.2105]→[17.4950:4951](∅→∅),[17.2105]→[17.1359:1379](∅→∅),[17.4951]→[17.1359:1379](∅→∅),[17.1359]→[17.1359:1379](∅→∅),[17.1379]→[17.2106:2143](∅→∅),[17.2143]→[17.1405:1424](∅→∅),[17.1405]→[17.1405:1424](∅→∅),[17.1424]→[10.0:50](∅→∅),[10.50]→[17.220:293](∅→∅),[17.1469]→[17.220:293](∅→∅),[17.293]→[17.361:412](∅→∅),[17.412]→[17.345:411](∅→∅),[17.345]→[17.345:411](∅→∅),[17.411]→[17.1469:1521](∅→∅),[17.1469]→[17.1469:1521](∅→∅),[17.1521]→[5.0:269](∅→∅),[5.269]→[17.213:257](∅→∅),[17.213]→[17.213:257](∅→∅),[17.257]→[5.270:452](∅→∅),[5.452]→[17.404:449](∅→∅),[17.404]→[17.404:449](∅→∅),[17.449]→[5.453:988](∅→∅),[17.851]→[17.2259:2295](∅→∅),[5.988]→[17.2259:2295](∅→∅),[17.2259]→[17.2259:2295](∅→∅),[17.2296]→[17.19:20](∅→∅),[17.450]→[17.450:451](∅→∅),[17.1379]→[17.2474:2489](∅→∅),[17.2474]→[17.2474:2489](∅→∅),[17.2489]→[17.2322:2392](∅→∅),[17.2392]→[17.2548:2609](∅→∅),[17.2548]→[17.2548:2609](∅→∅),[17.2609]→[14.83:155](∅→∅),[14.155]→[17.3249:3346](∅→∅),[17.3249]→[17.3249:3346](∅→∅),[17.3346]→[17.2779:2913](∅→∅),[17.2779]→[17.2779:2913](∅→∅),[17.2913]→[17.413:414](∅→∅),[17.414]→[17.2393:2483](∅→∅),[17.2528]→[17.2528:2529](∅→∅),[17.2529]→[17.0:379](∅→∅),[17.379]→[15.0:76](∅→∅),[15.76]→[17.2016:2107](∅→∅),[17.455]→[17.2016:2107](∅→∅),[17.3465]→[17.2016:2107](∅→∅),[17.2107]→[6.13:86](∅→∅),[6.86]→[17.2163:2180](∅→∅),[17.2163]→[17.2163:2180](∅→∅),[17.178]→[17.3181:3211](∅→∅),[17.2180]→[17.3181:3211](∅→∅),[17.3549]→[17.3181:3211](∅→∅),[17.3181]→[17.3181:3211](∅→∅),[17.3211]→[17.2181:2233](∅→∅),[17.1513]→[17.90:94](∅→∅),[17.94]→[17.0:137](∅→∅),[17.137]→[17.216:240](∅→∅),[17.216]→[17.216:240](∅→∅),[17.240]→[17.138:224](∅→∅),[17.224]→[17.423:478](∅→∅),[17.478]→[17.280:327](∅→∅),[17.280]→[17.280:327](∅→∅),[17.327]→[17.372:394](∅→∅),[17.372]→[17.372:394](∅→∅),[17.394]→[17.2316:2319](∅→∅),[17.1513]→[17.2316:2319](∅→∅),[17.2316]→[17.2316:2319](∅→∅),[17.2319]→[17.25:26](∅→∅),[17.26]→[17.3432:3507](∅→∅),[17.2319]→[17.3432:3507](∅→∅),[17.3432]→[17.3432:3507](∅→∅),[17.3507]→[17.1083:1198](∅→∅),[17.248]→[17.7099:7100](∅→∅),[17.1198]→[17.7099:7100](∅→∅),[17.186]→[17.7099:7100](∅→∅),[17.7100]→[17.2234:2292](∅→∅),[17.2292]→[17.7157:7193](∅→∅),[17.7157]→[17.7157:7193](∅→∅),[17.186]→[17.3600:3601](∅→∅),[17.494]→[17.3600:3601](∅→∅),[17.7193]→[17.3600:3601](∅→∅),[17.3600]→[17.3600:3601](∅→∅),[17.3601]→[17.7194:7402](∅→∅),[17.7402]→[17.2293:2362](∅→∅),[17.2362]→[17.7478:7575](∅→∅),[17.7478]→[17.7478:7575](∅→∅),[17.7575]→[17.3724:3725](∅→∅),[17.3724]→[17.3724:3725](∅→∅),[17.3860]→[17.3860:3898](∅→∅),[17.3898]→[17.1199:1272](∅→∅),[17.1272]→[17.4090:4091](∅→∅),[17.4090]→[17.4090:4091](∅→∅),[17.4186]→[17.4186:4677](∅→∅),[17.4677]→[17.7576:7803](∅→∅),[17.7803]→[17.495:528](∅→∅),[17.4892]→[17.495:528](∅→∅),[17.528]→[17.7804:7933](∅→∅),[17.7933]→[17.0:57](∅→∅),[17.528]→[17.0:57](∅→∅),[17.57]→[17.4133:4387](∅→∅),[17.4387]→[17.1273:1368](∅→∅),[17.1368]→[17.8023:8172](∅→∅),[17.8023]→[17.8023:8172](∅→∅),[17.391]→[17.663:701](∅→∅),[17.4550]→[17.663:701](∅→∅),[17.8172]→[17.663:701](∅→∅),[17.663]→[17.663:701](∅→∅),[17.701]→[17.0:93](∅→∅),[17.93]→[17.0:124](∅→∅),[17.793]→[17.0:124](∅→∅),[17.124]→[17.842:880](∅→∅),[17.1450]→[17.842:880](∅→∅),[17.842]→[17.842:880](∅→∅),[17.880]→[17.5165:5175](∅→∅),[17.5165]→[17.5165:5175](∅→∅),[17.5175]→[17.187:279](∅→∅),[17.279]→[17.1451:1523](∅→∅),[17.1523]→[17.345:374](∅→∅),[17.8247]→[17.345:374](∅→∅),[17.345]→[17.345:374](∅→∅),[17.374]→[17.94:168](∅→∅),[17.168]→[17.447:477](∅→∅),[17.447]→[17.447:477](∅→∅),[17.477]→[17.578:579](∅→∅),[17.579]→[17.411:438](∅→∅),[17.411]→[17.411:438](∅→∅),[17.438]→[17.580:581](∅→∅),[17.581]→[11.0:98](∅→∅),[11.98]→[17.9004:9045](∅→∅),[17.690]→[17.9004:9045](∅→∅),[17.690]→[17.5211:5250](∅→∅),[17.1383]→[17.5211:5250](∅→∅),[17.9045]→[17.5211:5250](∅→∅),[17.5211]→[17.5211:5250](∅→∅),[17.5250]→[17.691:724](∅→∅),[17.724]→[11.99:329](∅→∅),[11.329]→[17.261:359](∅→∅),[17.5581]→[17.261:359](∅→∅),[17.359]→[17.5581:5662](∅→∅),[17.5581]→[17.5581:5662](∅→∅),[17.5696]→[17.5696:5741](∅→∅),[17.5770]→[17.5770:5872](∅→∅),[17.5872]→[17.2419:2478](∅→∅),[17.2478]→[17.19661:19689](∅→∅),[17.4638]→[17.0:29](∅→∅),[17.29]→[12.80:131](∅→∅),[12.131]→[17.5872:5888](∅→∅),[17.1504]→[17.5872:5888](∅→∅),[17.4664]→[17.5872:5888](∅→∅),[17.5872]→[17.5872:5888](∅→∅),[17.5888]→[17.9046:9047](∅→∅),[17.9047]→[17.2479:2569](∅→∅),[17.2569]→[17.9144:9178](∅→∅),[17.9144]→[17.9144:9178](∅→∅),[17.9178]→[17.5888:5889](∅→∅),[17.5888]→[17.5888:5889](∅→∅),[17.5889]→[17.125:233](∅→∅),[17.233]→[17.1593:1667](∅→∅),[17.1593]→[17.1593:1667](∅→∅),[17.1667]→[17.725:726](∅→∅),[17.9255]→[17.725:726](∅→∅),[17.546]→[17.725:726](∅→∅),[17.726]→[17.19690:19799](∅→∅),[17.19799]→[17.6965:6996](∅→∅),[17.6965]→[17.6965:6996](∅→∅),[17.3068]→[17.459:460](∅→∅),[17.9374]→[17.459:460](∅→∅),[17.459]→[17.459:460](∅→∅),[17.9432]→[17.588:589](∅→∅),[17.588]→[17.588:589](∅→∅)

my @inputs = ();

foreach my $prevBuild (@validBuilds) {
my $pkgNameRE = "(?:(?:[A-Za-z0-9]|(?:-[^0-9]))+)";
my $versionRE = "(?:[A-Za-z0-9\.\-]+)";

my $relName = ($prevBuild->releasename or $prevBuild->nixname);
my $version = $2 if $relName =~ /^($pkgNameRE)-($versionRE)$/;

my $input =
{ storePath => getMainOutput($prevBuild)->path
, id => $prevBuild->id
, version => $version
, system => $prevBuild->system
};
push(@inputs, $input);
}

return @inputs;
}

sub fetchInputEval {
my ($db, $project, $jobset, $name, $value) = @_;

my $eval;

if ($value =~ /^\d+$/) {
$eval = $db->resultset('JobsetEvals')->find({ id => int($value) });
die "evaluation $eval->{id} does not exist\n" unless defined $eval;
} elsif ($value =~ /^($projectNameRE):($jobsetNameRE)$/) {
my $jobset = $db->resultset('Jobsets')->find({ project => $1, name => $2 });
die "jobset ‘$value’ does not exist\n" unless defined $jobset;
$eval = getLatestFinishedEval($jobset);
die "jobset ‘$value’ does not have a finished evaluation\n" unless defined $eval;
} elsif ($value =~ /^($projectNameRE):($jobsetNameRE):($jobNameRE)$/) {
$eval = $db->resultset('JobsetEvals')->find(
{ project => $1, jobset => $2, hasnewbuilds => 1 },
{ order_by => "id DESC", rows => 1
, where =>
\ [ # All builds in this jobset should be finished...
"not exists (select 1 from JobsetEvalMembers m join Builds b on m.build = b.id where m.eval = me.id and b.finished = 0) "
# ...and the specified build must have succeeded.
. "and exists (select 1 from JobsetEvalMembers m join Builds b on m.build = b.id where m.eval = me.id and b.job = ? and b.buildstatus = 0)"
, [ 'name', $3 ] ]
});
die "there is no successful build of ‘$value’ in a finished evaluation\n" unless defined $eval;
} else {
die;
}

my $jobs = {};
foreach my $build ($eval->builds) {
next unless $build->finished == 1 && $build->buildstatus == 0;
# FIXME: Handle multiple outputs.
my $out = $build->buildoutputs->find({ name => "out" });
next unless defined $out;
# FIXME: Should we fail if the path is not valid?
next unless isValidPath($out->path);
$jobs->{$build->get_column('job')} = $out->path;
}

return { jobs => $jobs };
}

sub fetchInput {
my ($plugins, $db, $project, $jobset, $name, $type, $value, $emailresponsible) = @_;
my @inputs;

if ($type eq "build") {
@inputs = fetchInputBuild($db, $project, $jobset, $name, $value);
}
elsif ($type eq "sysbuild") {
@inputs = fetchInputSystemBuild($db, $project, $jobset, $name, $value);
}
elsif ($type eq "eval") {
@inputs = fetchInputEval($db, $project, $jobset, $name, $value);
}
elsif ($type eq "string" || $type eq "nix") {
die unless defined $value;
@inputs = { value => $value };
}
elsif ($type eq "boolean") {
die unless defined $value && ($value eq "true" || $value eq "false");
@inputs = { value => $value };
}
else {
my $found = 0;
foreach my $plugin (@{$plugins}) {
@inputs = $plugin->fetchInput($type, $name, $value, $project, $jobset);
if (defined $inputs[0]) {
$found = 1;
last;
}
}
die "input `$name' has unknown type `$type'." unless $found;
}

foreach my $input (@inputs) {
$input->{type} = $type;
$input->{emailresponsible} = $emailresponsible;
}

return @inputs;
}

sub booleanToString {
my ($exprType, $value) = @_;
my $result;
if ($exprType eq "guile") {
if ($value eq "true") {
$result = "#t";
} else {
$result = "#f";
}
$result = $value;
} else {
$result = $value;
}
return $result;
}

sub buildInputToString {
my ($exprType, $input) = @_;
my $result;
if ($exprType eq "guile") {
$result = "'((file-name . \"" . ${input}->{storePath} . "\")" .
(defined $input->{revision} ? "(revision . \"" . $input->{revision} . "\")" : "") .
(defined $input->{revCount} ? "(revision-count . " . $input->{revCount} . ")" : "") .
(defined $input->{gitTag} ? "(git-tag . \"" . $input->{gitTag} . "\")" : "") .
(defined $input->{shortRev} ? "(short-revision . \"" . $input->{shortRev} . "\")" : "") .
(defined $input->{version} ? "(version . \"" . $input->{version} . "\")" : "") .
")";
} else {
$result = "{ outPath = builtins.storePath " . $input->{storePath} . "" .
"; inputType = \"" . $input->{type} . "\"" .
(defined $input->{uri} ? "; uri = \"" . $input->{uri} . "\"" : "") .
(defined $input->{revNumber} ? "; rev = " . $input->{revNumber} . "" : "") .
(defined $input->{revision} ? "; rev = \"" . $input->{revision} . "\"" : "") .
(defined $input->{revCount} ? "; revCount = " . $input->{revCount} . "" : "") .
(defined $input->{gitTag} ? "; gitTag = \"" . $input->{gitTag} . "\"" : "") .
(defined $input->{shortRev} ? "; shortRev = \"" . $input->{shortRev} . "\"" : "") .
(defined $input->{version} ? "; version = \"" . $input->{version} . "\"" : "") .
(defined $input->{outputName} ? "; outputName = \"" . $input->{outputName} . "\"" : "") .
(defined $input->{drvPath} ? "; drvPath = builtins.storePath " . $input->{drvPath} . "" : "") .
";}";
}
return $result;
}

sub inputsToArgs {
my ($inputInfo, $exprType) = @_;
my @res = ();

foreach my $input (sort keys %{$inputInfo}) {
push @res, "-I", "$input=$inputInfo->{$input}->[0]->{storePath}"
if scalar @{$inputInfo->{$input}} == 1
&& defined $inputInfo->{$input}->[0]->{storePath};
foreach my $alt (@{$inputInfo->{$input}}) {
if ($alt->{type} eq "string") {
push @res, "--argstr", $input, $alt->{value};
}
elsif ($alt->{type} eq "boolean") {
push @res, "--arg", $input, booleanToString($exprType, $alt->{value});
}
elsif ($alt->{type} eq "nix") {
die "input type ‘nix’ only supported for Nix-based jobsets\n" unless $exprType eq "nix";
push @res, "--arg", $input, $alt->{value};
}
elsif ($alt->{type} eq "eval") {
die "input type ‘eval’ only supported for Nix-based jobsets\n" unless $exprType eq "nix";
my $s = "{ ";
# FIXME: escape $_. But dots should not be escaped.
$s .= "$_ = builtins.storePath ${\$alt->{jobs}->{$_}}; "
foreach keys %{$alt->{jobs}};
$s .= "}";
push @res, "--arg", $input, $s;
}
else {
push @res, "--arg", $input, buildInputToString($exprType, $alt);
}
}
}

return @res;
}

sub evalJobs {
my ($inputInfo, $exprType, $nixExprInputName, $nixExprPath) = @_;

my $nixExprInput = $inputInfo->{$nixExprInputName}->[0]
or die "cannot find the input containing the job expression\n";
die "multiple alternatives for the input containing the Nix expression are not supported.\n"
if scalar @{$inputInfo->{$nixExprInputName}} != 1;
my $nixExprFullPath = $nixExprInput->{storePath} . "/" . $nixExprPath;

my $evaluator = ($exprType eq "guile") ? "hydra-eval-guile-jobs" : "hydra-eval-jobs";

my @cmd = ($evaluator, $nixExprFullPath, "--gc-roots-dir", getGCRootsDir, "-j", 1, inputsToArgs($inputInfo, $exprType));

if (defined $ENV{'HYDRA_DEBUG'}) {
sub escape {
my $s = $_;
$s =~ s/'/'\\''/g;
return "'" . $s . "'";
}
my @escaped = map escape, @cmd;
print STDERR "evaluator: @escaped\n";
}

(my $res, my $jobsJSON, my $stderr) = captureStdoutStderr(21600, @cmd);
die "$evaluator returned " . ($res & 127 ? "signal $res" : "exit code " . ($res >> 8))
. ":\n" . ($stderr ? decode("utf-8", $stderr) : "(no output)\n")
if $res;

print STDERR "$stderr";

return (decode_json($jobsJSON), $nixExprInput);
}

# Return the most recent evaluation of the given jobset (that
# optionally had new builds), or undefined if no such evaluation
# exists.
sub getPrevJobsetEval {
my ($db, $jobset, $hasNewBuilds) = @_;
my ($prevEval) = $jobset->jobsetevals(
($hasNewBuilds ? { hasnewbuilds => 1 } : { }),
{ order_by => "id DESC", rows => 1 });
return $prevEval;
}

# Check whether to add the build described by $buildInfo.
sub checkBuild {
my ($db, $jobset, $inputInfo, $nixExprInput, $buildInfo, $buildMap, $prevEval, $jobOutPathMap, $plugins) = @_;

my @outputNames = sort keys %{$buildInfo->{outputs}};
die unless scalar @outputNames;

# In various checks we can use an arbitrary output (the first)
# rather than all outputs, since if one output is the same, the
# others will be as well.
my $firstOutputName = $outputNames[0];
my $firstOutputPath = $buildInfo->{outputs}->{$firstOutputName};

my $jobName = $buildInfo->{jobName} or die;
my $drvPath = $buildInfo->{drvPath} or die;

my $build;

txn_do($db, sub {
my $job = $jobset->jobs->update_or_create({ name => $jobName });

# Don't add a build that has already been scheduled for this
# job, or has been built but is still a "current" build for
# this job. Note that this means that if the sources of a job
# are changed from A to B and then reverted to A, three builds
# will be performed (though the last one will probably use the
# cached result from the first). This ensures that the builds
# with the highest ID will always be the ones that we want in
# the channels. FIXME: Checking the output paths doesn't take
# meta-attributes into account. For instance, do we want a
# new build to be scheduled if the meta.maintainers field is
# changed?
if (defined $prevEval) {
# Only check one output: if it's the same, the other will be as well.
my $firstOutput = $outputNames[0];
my ($prevBuild) = $prevEval->builds->search(
# The "project" and "jobset" constraints are
# semantically unnecessary (because they're implied by
# the eval), but they give a factor 1000 speedup on
# the Nixpkgs jobset with PostgreSQL.
{ project => $jobset->project->name, jobset => $jobset->name, job => $jobName,
name => $firstOutputName, path => $firstOutputPath },
{ rows => 1, columns => ['id'], join => ['buildoutputs'] });
if (defined $prevBuild) {
#print STDERR " already scheduled/built as build ", $prevBuild->id, "\n";
$buildMap->{$prevBuild->id} = { id => $prevBuild->id, jobName => $jobName, new => 0, drvPath => $drvPath };
return;
}
}

# Prevent multiple builds with the same (job, outPath) from
# being added.
my $prev = $$jobOutPathMap{$jobName . "\t" . $firstOutputPath};
if (defined $prev) {
#print STDERR " already scheduled as build ", $prev, "\n";
return;
}

my $time = time();

sub null {
my ($s) = @_;
return $s eq "" ? undef : $s;
}

# Add the build to the database.
$build = $job->builds->create(
{ timestamp => $time
, description => null($buildInfo->{description})
, license => null($buildInfo->{license})
, homepage => null($buildInfo->{homepage})
, maintainers => null($buildInfo->{maintainers})
, maxsilent => $buildInfo->{maxSilent}
, timeout => $buildInfo->{timeout}
, nixname => $buildInfo->{nixName}
, drvpath => $drvPath
, system => $buildInfo->{system}
, nixexprinput => $jobset->nixexprinput
, nixexprpath => $jobset->nixexprpath
, priority => $buildInfo->{schedulingPriority}
, finished => 0
, iscurrent => 1
, ischannel => $buildInfo->{isChannel}
});

$build->buildoutputs->create({ name => $_, path => $buildInfo->{outputs}->{$_} })
foreach @outputNames;

$buildMap->{$build->id} = { id => $build->id, jobName => $jobName, new => 1, drvPath => $drvPath };
$$jobOutPathMap{$jobName . "\t" . $firstOutputPath} = $build->id;

print STDERR "added build ${\$build->id} (${\$jobset->project->name}:${\$jobset->name}:$jobName)\n";
});

return $build;
};
edit in src/lib/Hydra/Helper/Nix.pm at line 13

[13.122]

[17.127]

use IPC::Run;
replacement in src/script/hydra-eval-jobset at line 5

[17.151]→[17.473:492](∅→∅),[17.1363]→[17.473:492](∅→∅),[17.492]→[17.19854:19873](∅→∅),[17.19873]→[17.1746:1770](∅→∅),[17.492]→[17.1746:1770](∅→∅)

use Hydra::Schema;
use Hydra::Plugin;
use Hydra::Helper::Nix;

[17.151]

[17.10916]

use Config::General;
use Data::Dump qw(dump);
use Digest::SHA qw(sha256_hex);
use Encode;
use File::Slurp;
edit in src/script/hydra-eval-jobset at line 11

[17.10946]

[17.2430]

use Hydra::Helper::CatalystUtils;
edit in src/script/hydra-eval-jobset at line 13

[17.2456]

[17.688]

use Hydra::Helper::Nix;
replacement in src/script/hydra-eval-jobset at line 15

[17.710]→[17.2013:2045](∅→∅),[17.10946]→[17.2013:2045](∅→∅),[17.396]→[17.396:417](∅→∅),[17.417]→[17.2426:2451](∅→∅),[17.2451]→[17.2457:2472](∅→∅)

use Digest::SHA qw(sha256_hex);
use Config::General;
use Data::Dump qw(dump);
use Try::Tiny;

[17.710]

[17.0]

use Hydra::Plugin;
use Hydra::Schema;
use JSON;
edit in src/script/hydra-eval-jobset at line 19

[17.17]

[16.0]

use Nix::Store;
replacement in src/script/hydra-eval-jobset at line 21

[16.51]→[17.4394:4421](∅→∅),[17.67]→[17.4394:4421](∅→∅)

use JSON;
use File::Slurp;

[16.51]

[17.418]

use Try::Tiny;
edit in src/script/hydra-eval-jobset at line 33

[17.500]

[17.5400]

sub parseJobName {
# Parse a job specification of the form `<project>:<jobset>:<job>
# [attrs]'. The project, jobset and attrs may be omitted. The
# attrs have the form `name = "value"'.
my ($s) = @_;
our $key;
our %attrs = ();
# hm, maybe I should stop programming Perl before it's too late...
$s =~ / ^ (?: (?: ($projectNameRE) : )? ($jobsetNameRE) : )? ($jobNameRE) \s*
(\[ \s* (
([\w]+) (?{ $key = $^N; }) \s* = \s* \"
([\w\-]+) (?{ $attrs{$key} = $^N; }) \"
\s* )* \])? $
/x
or die "invalid job specifier `$s'";
return ($1, $2, $3, \%attrs);
}

sub attrsToSQL {
my ($attrs, $id) = @_;

my $query = "1 = 1";

foreach my $name (keys %{$attrs}) {
my $value = $attrs->{$name};
$name =~ /^[\w\-]+$/ or die;
$value =~ /^[\w\-]+$/ or die;
# !!! Yes, this is horribly injection-prone... (though
# name/value are filtered above). Should use SQL::Abstract,
# but it can't deal with subqueries. At least we should use
# placeholders.
$query .= " and exists (select 1 from buildinputs where build = $id and name = '$name' and value = '$value')";
}

return $query;
}

sub fetchInputBuild {
my ($db, $project, $jobset, $name, $value) = @_;

my $prevBuild;

if ($value =~ /^\d+$/) {
$prevBuild = $db->resultset('Builds')->find({ id => int($value) });
} else {
my ($projectName, $jobsetName, $jobName, $attrs) = parseJobName($value);
$projectName ||= $project->name;
$jobsetName ||= $jobset->name;

# Pick the most recent successful build of the specified job.
$prevBuild = $db->resultset('Builds')->search(
{ finished => 1, project => $projectName, jobset => $jobsetName
, job => $jobName, buildStatus => 0 },
{ order_by => "me.id DESC", rows => 1
, where => \ attrsToSQL($attrs, "me.id") })->single;
}

return () if !defined $prevBuild || !isValidPath(getMainOutput($prevBuild)->path);

#print STDERR "input `", $name, "': using build ", $prevBuild->id, "\n";

my $pkgNameRE = "(?:(?:[A-Za-z0-9]|(?:-[^0-9]))+)";
my $versionRE = "(?:[A-Za-z0-9\.\-]+)";

my $relName = ($prevBuild->releasename or $prevBuild->nixname);
my $version = $2 if $relName =~ /^($pkgNameRE)-($versionRE)$/;

my $mainOutput = getMainOutput($prevBuild);

my $result =
{ storePath => $mainOutput->path
, id => $prevBuild->id
, version => $version
, outputName => $mainOutput->name
};
if (isValidPath($prevBuild->drvpath)) {
$result->{drvPath} = $prevBuild->drvpath;
}

return $result;
}

sub fetchInputSystemBuild {
my ($db, $project, $jobset, $name, $value) = @_;

my ($projectName, $jobsetName, $jobName, $attrs) = parseJobName($value);
$projectName ||= $project->name;
$jobsetName ||= $jobset->name;

my @latestBuilds = $db->resultset('LatestSucceededForJob')
->search({}, {bind => [$projectName, $jobsetName, $jobName]});

my @validBuilds = ();
foreach my $build (@latestBuilds) {
push(@validBuilds, $build) if isValidPath(getMainOutput($build)->path);
}

if (scalar(@validBuilds) == 0) {
print STDERR "input `", $name, "': no previous build available\n";
return ();
}

my @inputs = ();

foreach my $prevBuild (@validBuilds) {
my $pkgNameRE = "(?:(?:[A-Za-z0-9]|(?:-[^0-9]))+)";
my $versionRE = "(?:[A-Za-z0-9\.\-]+)";

my $relName = ($prevBuild->releasename or $prevBuild->nixname);
my $version = $2 if $relName =~ /^($pkgNameRE)-($versionRE)$/;

my $input =
{ storePath => getMainOutput($prevBuild)->path
, id => $prevBuild->id
, version => $version
, system => $prevBuild->system
};
push(@inputs, $input);
}

return @inputs;
}

sub fetchInputEval {
my ($db, $project, $jobset, $name, $value) = @_;

my $eval;

if ($value =~ /^\d+$/) {
$eval = $db->resultset('JobsetEvals')->find({ id => int($value) });
die "evaluation $eval->{id} does not exist\n" unless defined $eval;
} elsif ($value =~ /^($projectNameRE):($jobsetNameRE)$/) {
my $jobset = $db->resultset('Jobsets')->find({ project => $1, name => $2 });
die "jobset ‘$value’ does not exist\n" unless defined $jobset;
$eval = getLatestFinishedEval($jobset);
die "jobset ‘$value’ does not have a finished evaluation\n" unless defined $eval;
} elsif ($value =~ /^($projectNameRE):($jobsetNameRE):($jobNameRE)$/) {
$eval = $db->resultset('JobsetEvals')->find(
{ project => $1, jobset => $2, hasnewbuilds => 1 },
{ order_by => "id DESC", rows => 1
, where =>
\ [ # All builds in this jobset should be finished...
"not exists (select 1 from JobsetEvalMembers m join Builds b on m.build = b.id where m.eval = me.id and b.finished = 0) "
# ...and the specified build must have succeeded.
. "and exists (select 1 from JobsetEvalMembers m join Builds b on m.build = b.id where m.eval = me.id and b.job = ? and b.buildstatus = 0)"
, [ 'name', $3 ] ]
});
die "there is no successful build of ‘$value’ in a finished evaluation\n" unless defined $eval;
} else {
die;
}

my $jobs = {};
foreach my $build ($eval->builds) {
next unless $build->finished == 1 && $build->buildstatus == 0;
# FIXME: Handle multiple outputs.
my $out = $build->buildoutputs->find({ name => "out" });
next unless defined $out;
# FIXME: Should we fail if the path is not valid?
next unless isValidPath($out->path);
$jobs->{$build->get_column('job')} = $out->path;
}

return { jobs => $jobs };
}

sub fetchInput {
my ($plugins, $db, $project, $jobset, $name, $type, $value, $emailresponsible) = @_;
my @inputs;

if ($type eq "build") {
@inputs = fetchInputBuild($db, $project, $jobset, $name, $value);
}
elsif ($type eq "sysbuild") {
@inputs = fetchInputSystemBuild($db, $project, $jobset, $name, $value);
}
elsif ($type eq "eval") {
@inputs = fetchInputEval($db, $project, $jobset, $name, $value);
}
elsif ($type eq "string" || $type eq "nix") {
die unless defined $value;
@inputs = { value => $value };
}
elsif ($type eq "boolean") {
die unless defined $value && ($value eq "true" || $value eq "false");
@inputs = { value => $value };
}
else {
my $found = 0;
foreach my $plugin (@{$plugins}) {
@inputs = $plugin->fetchInput($type, $name, $value, $project, $jobset);
if (defined $inputs[0]) {
$found = 1;
last;
}
}
die "input `$name' has unknown type `$type'." unless $found;
}

foreach my $input (@inputs) {
$input->{type} = $type;
$input->{emailresponsible} = $emailresponsible;
}

return @inputs;
}

sub booleanToString {
my ($exprType, $value) = @_;
my $result;
if ($exprType eq "guile") {
if ($value eq "true") {
$result = "#t";
} else {
$result = "#f";
}
$result = $value;
} else {
$result = $value;
}
return $result;
}

sub buildInputToString {
my ($exprType, $input) = @_;
my $result;
if ($exprType eq "guile") {
$result = "'((file-name . \"" . ${input}->{storePath} . "\")" .
(defined $input->{revision} ? "(revision . \"" . $input->{revision} . "\")" : "") .
(defined $input->{revCount} ? "(revision-count . " . $input->{revCount} . ")" : "") .
(defined $input->{gitTag} ? "(git-tag . \"" . $input->{gitTag} . "\")" : "") .
(defined $input->{shortRev} ? "(short-revision . \"" . $input->{shortRev} . "\")" : "") .
(defined $input->{version} ? "(version . \"" . $input->{version} . "\")" : "") .
")";
} else {
$result = "{ outPath = builtins.storePath " . $input->{storePath} . "" .
"; inputType = \"" . $input->{type} . "\"" .
(defined $input->{uri} ? "; uri = \"" . $input->{uri} . "\"" : "") .
(defined $input->{revNumber} ? "; rev = " . $input->{revNumber} . "" : "") .
(defined $input->{revision} ? "; rev = \"" . $input->{revision} . "\"" : "") .
(defined $input->{revCount} ? "; revCount = " . $input->{revCount} . "" : "") .
(defined $input->{gitTag} ? "; gitTag = \"" . $input->{gitTag} . "\"" : "") .
(defined $input->{shortRev} ? "; shortRev = \"" . $input->{shortRev} . "\"" : "") .
(defined $input->{version} ? "; version = \"" . $input->{version} . "\"" : "") .
(defined $input->{outputName} ? "; outputName = \"" . $input->{outputName} . "\"" : "") .
(defined $input->{drvPath} ? "; drvPath = builtins.storePath " . $input->{drvPath} . "" : "") .
";}";
}
return $result;
}

sub inputsToArgs {
my ($inputInfo, $exprType) = @_;
my @res = ();

foreach my $input (sort keys %{$inputInfo}) {
push @res, "-I", "$input=$inputInfo->{$input}->[0]->{storePath}"
if scalar @{$inputInfo->{$input}} == 1
&& defined $inputInfo->{$input}->[0]->{storePath};
foreach my $alt (@{$inputInfo->{$input}}) {
if ($alt->{type} eq "string") {
push @res, "--argstr", $input, $alt->{value};
}
elsif ($alt->{type} eq "boolean") {
push @res, "--arg", $input, booleanToString($exprType, $alt->{value});
}
elsif ($alt->{type} eq "nix") {
die "input type ‘nix’ only supported for Nix-based jobsets\n" unless $exprType eq "nix";
push @res, "--arg", $input, $alt->{value};
}
elsif ($alt->{type} eq "eval") {
die "input type ‘eval’ only supported for Nix-based jobsets\n" unless $exprType eq "nix";
my $s = "{ ";
# FIXME: escape $_. But dots should not be escaped.
$s .= "$_ = builtins.storePath ${\$alt->{jobs}->{$_}}; "
foreach keys %{$alt->{jobs}};
$s .= "}";
push @res, "--arg", $input, $s;
}
else {
push @res, "--arg", $input, buildInputToString($exprType, $alt);
}
}
}

return @res;
}
edit in src/script/hydra-eval-jobset at line 332

[17.5401]

[17.1424]

sub evalJobs {
my ($inputInfo, $exprType, $nixExprInputName, $nixExprPath) = @_;
edit in src/script/hydra-eval-jobset at line 335

[17.1425]

[17.483]

my $nixExprInput = $inputInfo->{$nixExprInputName}->[0]
or die "cannot find the input containing the job expression\n";
die "multiple alternatives for the input containing the Nix expression are not supported.\n"
if scalar @{$inputInfo->{$nixExprInputName}} != 1;
my $nixExprFullPath = $nixExprInput->{storePath} . "/" . $nixExprPath;

my $evaluator = ($exprType eq "guile") ? "hydra-eval-guile-jobs" : "hydra-eval-jobs";

my @cmd = ($evaluator, $nixExprFullPath, "--gc-roots-dir", getGCRootsDir, "-j", 1, inputsToArgs($inputInfo, $exprType));

if (defined $ENV{'HYDRA_DEBUG'}) {
sub escape {
my $s = $_;
$s =~ s/'/'\\''/g;
return "'" . $s . "'";
}
my @escaped = map escape, @cmd;
print STDERR "evaluator: @escaped\n";
}

(my $res, my $jobsJSON, my $stderr) = captureStdoutStderr(21600, @cmd);
die "$evaluator returned " . ($res & 127 ? "signal $res" : "exit code " . ($res >> 8))
. ":\n" . ($stderr ? decode("utf-8", $stderr) : "(no output)\n")
if $res;

print STDERR "$stderr";

return (decode_json($jobsJSON), $nixExprInput);
}

# Return the most recent evaluation of the given jobset (that
# optionally had new builds), or undefined if no such evaluation
# exists.
sub getPrevJobsetEval {
my ($db, $jobset, $hasNewBuilds) = @_;
my ($prevEval) = $jobset->jobsetevals(
($hasNewBuilds ? { hasnewbuilds => 1 } : { }),
{ order_by => "id DESC", rows => 1 });
return $prevEval;
}

# Check whether to add the build described by $buildInfo.
sub checkBuild {
my ($db, $jobset, $inputInfo, $nixExprInput, $buildInfo, $buildMap, $prevEval, $jobOutPathMap, $plugins) = @_;

my @outputNames = sort keys %{$buildInfo->{outputs}};
die unless scalar @outputNames;

# In various checks we can use an arbitrary output (the first)
# rather than all outputs, since if one output is the same, the
# others will be as well.
my $firstOutputName = $outputNames[0];
my $firstOutputPath = $buildInfo->{outputs}->{$firstOutputName};

my $jobName = $buildInfo->{jobName} or die;
my $drvPath = $buildInfo->{drvPath} or die;

my $build;

txn_do($db, sub {
my $job = $jobset->jobs->update_or_create({ name => $jobName });

# Don't add a build that has already been scheduled for this
# job, or has been built but is still a "current" build for
# this job. Note that this means that if the sources of a job
# are changed from A to B and then reverted to A, three builds
# will be performed (though the last one will probably use the
# cached result from the first). This ensures that the builds
# with the highest ID will always be the ones that we want in
# the channels. FIXME: Checking the output paths doesn't take
# meta-attributes into account. For instance, do we want a
# new build to be scheduled if the meta.maintainers field is
# changed?
if (defined $prevEval) {
# Only check one output: if it's the same, the other will be as well.
my $firstOutput = $outputNames[0];
my ($prevBuild) = $prevEval->builds->search(
# The "project" and "jobset" constraints are
# semantically unnecessary (because they're implied by
# the eval), but they give a factor 1000 speedup on
# the Nixpkgs jobset with PostgreSQL.
{ project => $jobset->project->name, jobset => $jobset->name, job => $jobName,
name => $firstOutputName, path => $firstOutputPath },
{ rows => 1, columns => ['id'], join => ['buildoutputs'] });
if (defined $prevBuild) {
#print STDERR " already scheduled/built as build ", $prevBuild->id, "\n";
$buildMap->{$prevBuild->id} = { id => $prevBuild->id, jobName => $jobName, new => 0, drvPath => $drvPath };
return;
}
}

# Prevent multiple builds with the same (job, outPath) from
# being added.
my $prev = $$jobOutPathMap{$jobName . "\t" . $firstOutputPath};
if (defined $prev) {
#print STDERR " already scheduled as build ", $prev, "\n";
return;
}

my $time = time();

sub null {
my ($s) = @_;
return $s eq "" ? undef : $s;
}

# Add the build to the database.
$build = $job->builds->create(
{ timestamp => $time
, description => null($buildInfo->{description})
, license => null($buildInfo->{license})
, homepage => null($buildInfo->{homepage})
, maintainers => null($buildInfo->{maintainers})
, maxsilent => $buildInfo->{maxSilent}
, timeout => $buildInfo->{timeout}
, nixname => $buildInfo->{nixName}
, drvpath => $drvPath
, system => $buildInfo->{system}
, nixexprinput => $jobset->nixexprinput
, nixexprpath => $jobset->nixexprpath
, priority => $buildInfo->{schedulingPriority}
, finished => 0
, iscurrent => 1
, ischannel => $buildInfo->{isChannel}
});

$build->buildoutputs->create({ name => $_, path => $buildInfo->{outputs}->{$_} })
foreach @outputNames;

$buildMap->{$build->id} = { id => $build->id, jobName => $jobName, new => 1, drvPath => $drvPath };
$$jobOutPathMap{$jobName . "\t" . $firstOutputPath} = $build->id;

print STDERR "added build ${\$build->id} (${\$jobset->project->name}:${\$jobset->name}:$jobName)\n";
});

return $build;
};
edit in tests/Setup.pm at line 7

[17.22]→[17.208:238](∅→∅),[17.208]→[17.208:238](∅→∅)

use Hydra::Helper::AddBuilds;
edit in tests/evaluation-tests.pl at line 4

[17.197]→[17.894:924](∅→∅),[17.894]→[17.894:924](∅→∅)

use Hydra::Helper::AddBuilds;